Even in the script provided by Fairphone to restore the Stock ROM, I had to comment out the regarding lines:
flash_image_or_abort xbl_a "${IMAGES_DIR}/xbl.elf"
# flash_image_or_abort xbl_b "${IMAGES_DIR}/xbl.elf"
flash_image_or_abort xbl_config_a "${IMAGES_DIR}/xbl_config.elf"
# flash_image_or_abort xbl_config_b "${IMAGES_DIR}/xbl_config.elf"