Wild guess: Are you using one of the two updated OS versions that were developed for the FP1U? Fairphone’s own unfinished Macadamia 1.9.9 or @z3ntu’s Android 4.4.4?
I remember that the problem you are describing happened to me when I had one of these two installed on my own FP1U. Back then, just a reboot helped (have you tried to keep holding the power button to force a reboot?).
The problem will return after a while (I never figured out if it’s a certain number of presses or a time span independent of that), but I am somewhat sure it did not get worse (i.e. it will not return ever-quicker).