Hi and welcome to the forum
You mean the FP3
Speaker mode uses the top mic, so you may want to get hold of a top module.
The Fairphone 3 top module includes an 8 megapixel front camera to make the best selfies. It also contains everything you need for clear calls, like the receiver, 3.5mm earphone jack, secondary microphone and proximity sensor.
https://shop.fairphone.com/en/spare-parts/fairphone-3-top-module
See if you can find a fairphoneangel near you that may have one to test.
Meanwhile disassemble the phone and check the top modules contacts.
You can even dismantle the actual top module for checking, cleaning etc.
There is always the option of buying a new module if you assess that the current one is indeed faulty.