I just did a clean install of the app, and tried this video*. I don’t seem to be able to reproduce what you’re describing. I haven’t figured out how to swipe around, but if I move the phone around (i.e. turn left/right/up/down whilst holding the phone), the video pans to follow my movement, and I can go round 360 degrees (and again, and again until I get dizzy) this way, so the Fairphone is able to handle these videos. Does the video I linked above also not work for you, or is this issue specific to only some videos?
* apologies for the marketing on this specific video - I just took one of the first ones at random.