Hello,
Unfortunately it is getting more an der more difficult to understand the other end while talking. I would assume this is a problem with the speaker.
Is there any way to solve this problem by repair or by replacement? Would the top module cover the Speaker as well?
Have you had a look at Fairphone’s Troubleshooting Tool?
It will basically suggest to first remove the top module out of your FP and screw it back into place at least once. If that doesn’t help, you might need a new top module. Details in the link. 
Thanks Urs, right now the problem does not occur, but I will give it a try if the issue comes back.