It sounds like it might be broken and it’s not something that is probably easily repairable on it’s own as it’s attached to the main motherboard (one of the expensive parts to buy!).
You could contact Fairphone directly to see whether they can repair on warranty otherwise you’ll need to buy a replacement motherboard.
The other thing you could try at your own risk would be a DIY repair job. If you follow the link here you can see someone repairing their USB socket. Might give some clues how to go about doing a DIY repair if that’s what you wanted to do.