Have you checked if it makes a difference when you plug it in with the case removed?
We had a similar problem like yours with someone on the FP3 as well and in her case it actually took a harder push until the plug (different headphones though) was properly in its place.
I myself used the new Fairphone earphones in an FP2 without such problems.