Which OS file exactly did you use for the downgrade?
Assuming the correct file was used, the support pages for the manual install state that user data will not be erased, while in general it seems Android doesn’t react too well to a data partition which was in use by a higher Android version, so a missing data wipe while downgrading would probably lead to what you describe.
You could try a factory reset in recovery mode, or a clean install from scratch … 🇬🇧 🇩🇪 ✏ Installing the Fairphone 2 from scratch in case of failure / Komplette Neuinstallation des Fairphone 2 im Fehlerfall