So this is what I did to get me out of the loop and fix it:
- remove the battery to turn the FP2 off (there might be a more subtle way)
- got into recovery mode by pressing volume up and power
- pressed install
- did not find de zip of GAPPS so I had to download it again here (ARM/5.1/pico) and put it on the SD-card by cable
- installed it
- rebooted the phone
All seems to be well. So the only thing I yet need to do is to thank @freibadschwimmer for his kind sharing of his update guide. Thank you!