I’ve been experimenting a bit with a confirmed working USB-C port and the broken one I have. So far I’ve found three failure modes.
Lint in the port
Causes the usb cable connector to not be inserted fully. Fixed by removing lint with a needle or similar.
Loudspeaker unit is not pressing on the USB-C port hard enough
The USB-C port is only held in place by its connector, a single screw, and the pressure of the loudspeaker unit which is screwed in on top of it. The loudspeaker needs to be screwed in very tightly, otherwise I could easily dislodge the USB-C port enough for it to start constantly loosing its connection by putting pressure on the connected headphone cable in a few different directions. I could definitely see that amount of pressure occur if you have the phone in your pocket and the headphone connector is pressed into your leg when you are walking up some stairs for example.
This was fixed by reinstalling the USB-C port completely, and attaching the loudspeaker screws like your life depended on it. I reproduced this twice.

Unknown Issue
I have yet to determine what my broken USB-C port suffers from. The symptoms are very similar to the one with the loosely installed loudspeaker, it works but disconnects constantly. I can’t get it to work by applying the fix described there.
The pins in the internal connector looks fine, but I will have a closer look with a microscope next week to see if they are bent. Other alternatives are the flat internal cable that runs between the port and connector, or the contact board in the external part of the port. Not sure if I can test these with probes somehow.
Since I have another working USB-C port I can be somewhat certain that the issue is not with the phone itself.
It would be very interesting to hear if others that have the same problem can place their issue in the categories above.