Custom ROMs are not necessarily more secure. Especially if you have to disable secure boot due to a missing signed boot image. You also have to fully trust these devs. Qualcomm updates are often out of date on old phones as well.
Pixel devices are the most secure Android devices because of the Titan M chip. Also because they get prompt security updates (Pixel 6 seems to have some issues with that, for now) .
Fairphone has the Qualcomm trusted execution environment, which is not bad either. And the updates don’t seem to be that much out of sync with other vendors. Fairphone also tends to support phones longer than Qualcomm does, leaving you without proper security support after a few years. This was also mentioned in the recent panel discussion on YouTube.
However, for some reason FIPS 140-2 is not available on the FP4. Do you have this enabled in your custom ROM?