Microphone is out of order
In loudspeaker mode a different microphone is used, so it could be that your primary microphone is broken. Sometimes removing the bottom module and then putting it back again solves the problem, in other cases Fairphone support has been known to replace parts under warranty. You can contact support via their form or by phone (see also here).
For more information see the topic linked below. Please continue there as well if you have other questions (this topic will be closed as a duplicate of the linked topic).
Duplicate of the topic linked in the post above, please continue there.