I experience the same problem and did some Google research.
My conclusions:
- The problem is the Mediatek 65xx CPU. (You can find not only Fairphone users with this problem, but also users with other phones having the Mediatek CPU).
- Therefore, as already mentioned here, the problem is “inside” the internal audio player. (The problem persits even after several factory resets and with lots of different apps).
- Workaround 1: Use an podcast app which allows to use a different 3rd party audio player.
- Workaround 2: The podcast app “Podcast Addict” has a special option for MT65xx users.
Thanks for the tip for using playback speed 1.1!