Very strange. Maybe a cleaning of the headphone port is in order - dust can build up in the bottom of the port, preventing proper contact.
If there’s lint or other, bigger grime in the bottom of the port you can get it out with a wooden toothpick. To clean the surfaces I usually use a pipe cleaner soaked in isopropyl alcohol. If you don’t smoke a pipe and so do not have pipe cleaners laying around, you can use a cotton bud with most of the cotton removed, or you can dip your headphone plug in alcohol and insert, rotate, and remove it a couple of times.
Let’s see if that works. If it doesn’t, you can try swapping out the top module to ascertain whether or not this is in fact a hardware issue - you’ll need a second module, of course, but maybe we can make that happen with the help of a Fairphone Angel, if there’s one near you.