Yes, you probably killed your recovery. Unfortunately Fairphone does not provide a stock recovery for now. The only option to get the stock recovery back is to compile FP-OSOS, but that would probably take a lot of time and effort. What I would suggest now is to install TWRP recovery from this threat:
I have the version that @Max_S compiled on my phone right now and it works fine.
It would however also be interesting to know what happens if you have a boot image in your recovery partition… It might just boot your normal OS when trying to enter recovery 
Fastboot mode should however not be impacted by a destroyed recovery partition. We had one case here in the Forum where a user could not boot his OS, could not enter recovery but could still enter fastboot mode in order to flash back working images…
So far for now. Regards,
Moritz