It is probably a problem with your headphone jack. Try to disassemble your top module, clean the contacts and put your phone back together. If that doesn’t work, try another headset with 4-contact jack and microphone, if it doesn’t work, too, ask support for a new top module.
If you know another Fairphoner or are willing to use our Fairphone Angels, you could insert another top module for testing before contacting support.