Hello
Ok maybe you do not see the forest because of the trees. 
“I” have done that, just start all over again.
Yess we … ääh you can 
But just the “normal” way . . .
- Remove any USB-C cable and turn off your Fairphone 4. If you cannot turn your device off, remove the battery for about 5 seconds, then put it back in.
- Press and hold the Volume Down button.
- Insert a USB-C cable connected to the power (can either be a power outlet or a computer).
- Release the Volume Down button as soon as you see the FAIRPHONE logo.
- Use the Volume buttons to select the option . . .
And only exactly like this.
If now the /e/ recovery comes, be careful !
Either adb “or” fastboot, you can’t mix in the console anymore!
Try adb devices and fastboot devices, then you know in which “mode” you are (seems fastboot).
And always allow debugging, the stupid phone does not remember that, . . . despite check mark 
Anyway, that’s how I did it . . . like this or similar 