Welcome to the community forum 
Your problem seems similar to this:
except that yours persists when using the loudspeaker, if I understand rightly, but you don’t have the problem with WA calls.
I suggest
- Perform the hardware tests either using the service menu as described in the other topic and / or My Fairphone app > Support > Audio
- Could you indicate your current OS version. Go to Settings > About phone > Build number