GPS/Location accuracy unstable

I have been having issues with my GPS location since purchase, but never realized it was specific to the device and not the software used.

The behavior seems to be that even though the device stays at the same location, it (seemingly) randomly drops in accuracy and jumps around the map as if it loses GPS and uses telephone pole locations instead (which seem inaccurate). But that also means WIFI locations are not used either.

I have seen the behavior in various apps, which include: HomeAssistant location tracking, Google Maps and PhoneTrack.

This is the mapping of PhoneTrack for two hours, those inaccurate points map out to wrong coordinate locations as well:

If anyone has an idea that should improve accuracy, or what to test more please let me know.