Apart from debris that can cause the force used to insert distorting the port, and or the plug, generally people are not careful and the cable may hang from port at an angle and cause distortion.
I have in the past physically bend both ports and plugs to overcome the issue, now I use a magnetic adapter so as to avoid such, plus it keeps the dust out.