Why don’t you start from a known working - selfcompiled - linux/dtb/dts combination as published by fairphone (with a minimal userspace /ramdisk for debugging) and then just figure out the diff to the mainline linux kernel of the same kernel version? that should make it more easy to apply the same patches + device tree changes to your kernel version of choice
also you can check the git log - using these kernel sources as suggested in