FP3 custom rom development based on released source code

I made the exact same observation yesterday about the dsp.img, Not sure exactly how that happened.
I will upload a corrected version of the installer today.
And I also created an installer for 110.
Your version of payload-dumper works great.
I had some issues, with product-partition apparently being mounted by TWRP and update failing because of that.
There will be a new TWRP as well.

6 Likes

stock firmware link to A.0101 (Nov 2019) - for anyone who needs it :wink:

https://www.androidfilehost.com/?fid=4349826312261713361
Filename: Fairphone_FP3_8901.2.A.0101.20191115_11150009_user-release-keys.zip
md5sum: 051eb5375d1412f77c9b83c51e1c6e42
sha256sum: 5384b8bc71d4dd284f21626b59004e5b2b5cebf8ce855bca7852b2be086a4ac4

Independent Verification: You can cross check against vendor check sums
using payload.bin from https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/eab40c208c6bd6899b51e7103822972d7a40336c.zip and https://github.com/CorvusCorax/payload_dumper in --diff mode (will ignore trailing zero bytes).
Applying the payload.bin will get you to A.0105. Furher applying https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/996d9c3cd83bf55bcbf4674a7074885e2d66e0f8.zip payload bin will get you to A.0110

$ sha256sum *.img
845924ee31a3ad8ca86e122299679cc7d2cd192b64060b9e4009a784f6889056  aboot.img
c8695bc30189a80ca0f70fda81d34d821c9215420c0c22226d774a2b700f37cf  boot.img
e67326e2050a9174a9d437e70e4954f97385ff22fd8e40c51115c582ffa7c13d  cmnlib64.img
be941fd8a054104ed8fb2ebfefa00b742d379f4e784c9d5f3a1638e4e7329355  cmnlib.img
c5e571383664f1704a033a4055e4dfffaa1d60d07fe15e7616590d129dda939a  devcfg.img
feaa15a42cb63bc81fffe93e79eaa9262c8c1818abe40f43ba713c5184ac5839  dsp.img
cade61b50b2ba8a3c866b8ea928162bafe8c33386d712a9a78cf0857b1ef6833  dtbo.img
1982e8cec6f9fe3984da7d1184346e1a437013e47be2bf5fcc72b91e6e24efe4  keymaster.img
3fdd919985501069641d59b16f7034609018186b2f33a6850e7f113a0fd0383e  lksecapp.img
c47821e639220f11a04e6d03dc2f6e67fe73dcb8738ffd1968462ea98a21c7d6  mdtp.img
0933af7ceadc3cdda1e2a6ca15f8b354ab38aa15dcebb74019c822584562e84b  modem.img
4e097e334c1231bc1d967fd8256211ef7c7d055b5f0defbd69306868df8857d9  product.img
95994f93282b372173057b045f9be0f29262ca982f787e331bb16678409a0540  rpm.img
8139a8f922287f0a6e6da6c627503bf72910f591429a815d918ed4dffdf966eb  sbl1.img
6ac7ba7e3bd58f4c80b388a3812bafa78f90e19b6474b7a36e913cf0b2437cca  system.img
e9ce4855657b7ecf63b777a644a6ce1bde0da616826ae3e1bd3e18f1334fe4f3  tz.img
3248b79a1f79a4648cdad89398a41f8d8ac6ec11e6b7d9ca235b659d2839ea9c  vbmeta.img
c95a77493d4a79a303ce388bde11c172a64642f9e12dd2546a9f4026e64aa5cd  vendor.img

verification:

$ python payload_dumper.py --diff payload.bin
Processing modem partition... Checking original Sha256, should be CTOvfOrcPN2h4qbKFfizVKs4qhXc67dAGcgiWEVi6Es=
 ... passed... 
extracting... truncating... Checking new Sha256, should be uHW4LbgFtXmxV1+teceezEVABCib10a2i3B2gW75+HY=
 ... passed... done
Processing sbl1 partition... Checking original Sha256, should be kar6XUqbymOV72Qh/oFfe9mUupFuPu75/WabRFQm1D0=
 ... passed... 
extracting... truncating... Checking new Sha256, should be 3iRULfan90ZPWy2QHzeaIywOUEqkuyYgBWKyf9Hk6PU=
 ... passed... done
Processing rpm partition... Checking original Sha256, should be EwlUjVBzt+j66Mg3/KT+nURNTxjfRSCP0cKE6mJ+jRQ=
 ... passed... 
extracting... truncating... Checking new Sha256, should be kikeLNG4xKlPqaQqzPcj3JQyfwiwen9nj+S1MgcpjEM=
 ... passed... done
Processing tz partition... Checking original Sha256, should be 7w0fA6NRXHIuw0m2HE7A2q4IeQDjm19oxtOc6sq3ou8=
 ... passed... 
extracting... truncating... Checking new Sha256, should be qXgGb2NgGOUVRV+CxVvB8kX0EpI62133/4ytrQZg/X8=
 ... passed... done
Processing devcfg partition... Checking original Sha256, should be a5OHX+pL9/cL11bqSNz41MQp6HuQBJ3ZJGBZIGAbdQg=
 ... passed... 
extracting... truncating... Checking new Sha256, should be T35U136bWdVNyZuJcvQTzJrj1CgmTMCO5kLFYvIAK/0=
 ... passed... done
Processing dsp partition... Checking original Sha256, should be /qoVpCy2O8gf/+k+eeqpJiyMGBir5A9DunE8UYSsWDk=
 ... passed... 
extracting... truncating... Checking new Sha256, should be /qoVpCy2O8gf/+k+eeqpJiyMGBir5A9DunE8UYSsWDk=
 ... passed... done
Processing aboot partition... Checking original Sha256, should be wKA7kN0CoGNQcfbg7Bpn+DydHQlqmSnWtrParETMhII=
 ... passed... 
extracting... truncating... Checking new Sha256, should be KXmQci8cU5P9IUKXol2o4qw54kUE6y4mzB4TFL6/Pww=
 ... passed... done
Processing dtbo partition... Checking original Sha256, should be yt5htQsrqKPIZrjqkoFiuv6MMzhtcSqaeM8IV7HvaDM=
 ... passed... 
extracting... truncating... Checking new Sha256, should be kEUcBPYvu+pz5cUCb7qx9KmLjkeGlqpWlvtKHID7itw=
 ... passed... done
Processing vbmeta partition... Checking original Sha256, should be mTSdCq7aC8ZIqODfX4Z02wvTqs6E75OSY7jw97vxARo=
 ... passed... 
extracting... truncating... Checking new Sha256, should be SuaZ/8GipVzd68nhGlpPPFDJXgQNNeMc5NYMYfOW//c=
 ... passed... done
Processing boot partition... Checking original Sha256, should be yGlbwwGJqAyg9w/agdNNghySFUIMDCIibXdKK3APN88=
 ... passed... 
extracting... truncating... Checking new Sha256, should be /I7hvKKxvGXORU+CCdH2epRBtbdoySCkpxJrv80xfgo=
 ... passed... done
Processing system partition... Checking original Sha256, should be ase6fjvVj0yAs4ijgSuvp4+Q4ZtkdLejbpE88LJDfMo=
 ... passed... 
extracting... truncating... Checking new Sha256, should be Liw8mKN2aLsNgVHs/tu5ziCLclop6dynwnxFSqg1H7U=
 ... passed... done
Processing vendor partition... Checking original Sha256, should be yVp3ST1KeaMDzjiL3hHBcqZGQvnhLdJUap9AJuZKpc0=
 ... passed... 
extracting... truncating... Checking new Sha256, should be PpYt1c0pixRKlOlQYBmbNO1ti1aBCBptkvtqACiDDx4=
 ... passed... done
Processing mdtp partition... Checking original Sha256, should be 3UmJGHRUbFJowuS3vjyVsTd33YzQpt/ER9vY/Fkn4rk=
 ... passed... 
extracting... truncating... Checking new Sha256, should be D9f4qXZyV2tkFGfhr30tZ6Aj6ErFTQSry8Ni6OGybwI=
 ... passed... done
Processing lksecapp partition... Checking original Sha256, should be OS0HWbkaczBKrVZ+7difoybiOeHzZInqO7vHL5hXHB4=
 ... passed... 
extracting... truncating... Checking new Sha256, should be 4C0eklWxwA5qnPYXWMWBcluzIrqHv5CnMdnINYDMZWM=
 ... passed... done
Processing cmnlib partition... Checking original Sha256, should be m/VLgHlSQt/NukkbIPFpn0lqUWB3EzLYnLMYK8epIRg=
 ... passed... 
extracting... truncating... Checking new Sha256, should be JZxlbOinSAO23eysX0R64jpHv/xluM+BOSssri7+BiI=
 ... passed... done
Processing cmnlib64 partition... Checking original Sha256, should be 4y8JsNsSKktjJxJq/iWEzy0Inrtvi4V8gLJTifw4mGo=
 ... passed... 
extracting... truncating... Checking new Sha256, should be +VrbC5/Ccalf+WQ6TpcncvLC4tWF2+1qwyRm3idU+5U=
 ... passed... done
Processing keymaster partition... Checking original Sha256, should be dXiXiRgYC5pFJV8EqSnhoeBajQcl45yhufrFrXm1dRw=
 ... passed... 
extracting... truncating... Checking new Sha256, should be Vd6R61DSpG34I+nHkogdpmx8krdbAlUzZkuIS2JD+2I=
 ... passed... done
Processing product partition... Checking original Sha256, should be Tgl+M0wSMbwdln/YJWIR73x9BVtfDe+9aTBoaN+IV9k=
 ... passed... 
extracting... truncating... Checking new Sha256, should be 0I32cRejEBKI2z60Ht03+8Jd4+nNstHlzT8//VFzEk0=
 ... passed... done
2 Likes

Do you have any doc to do this kind of zip?

no, and i don’t think you can actually use it for direct sideloading - I just duplicated the format @k4y0z used in his images. Only content are the partitions (format <partition_name>.img ) that are included in every OTA update, which are dumped from the phone. I dumped them like this (example for slot A):

fastboot boot twrp.img
adb shell
$ cat /dev/block/by-name/system_a >/external_sd/system.img

for each partition name in slot a, then i got the images down with adb pull
the metadata I added by analogy by copying info from the one in the differential update for the same version, I don’t know if that’s good for anything.

then just zip it all together with maximum compression

the list of partitions I simply used the same set that the FP OTA update updates (payload_updater in fakediff mode will list them)

1 Like

Ok, and did you try the zip?

1 Like

as in flashing it on the phone again? no, not yet. since it’d be a downgrade, that likely wouldn’t boot without erasing userdata, which is something i want to avoid right now since funny enough I actually need the phone for everyday use right now :wink: that’s also why i haven’t rooted or installed LOS yet.

I don’t know any better way to re-install those partitions than you described in TWRP for Fairphone 3 which would be one by one, using fastboot.
I know I was previously able to boot from that slot (which got overwritten in the latest update) but only if I erased userdata (mentioned upthread)

edit: as I understand it, if you installed the latest version using OTA, you can always go back to the previous version (if the phone is unlocked) by simply switching slots and erasing userdata. the OTA update will install the new version in the currently unused slot, then switch slots, but it doesn’t erase or alter the old one (if it did, my images would have failed checksum tests)

2 Likes

Actually, the flashable zip is a really good way to re install the partitions :). For the fastboot cmds, you can do a little .sh

I understand, that’s my daily phone too ^^. I have done too many times the erase userdata haha ! And the backup of the data with TWRP works pretty well. You can loose your documents, your photos and you have to re-set your password but everything else works. But I’m not sure, and I won’t test, that it works with different ROMs.

3 Likes

As far as I understand it, the zip in the format we have it is not directly flashable because the system partition is too large. @k4y0z overcame that to create a “sparse” image, but while that might install via sideloading I don’t know how to create, unpack or verify that. (I looked into the zip, but couldn’t figure out the file formats)
I can’t make one of those without further documentation, but most likely anyone can make a sparse image from the full image, later.

Edit: found this:

2 Likes

I am using img2simg first to create a sparse image from the ext4 (that’s available as a package in my debian).
Then I use img2sdat to create the dat-files.
and then i pass it through brotli -q 6 (also available in debian) to create the brotli-files.
I do this for system, vendor and product.
The rest of the partitions are straight dd-images.

You can do the reverse process for confirming they result in the same images.

Then update META-INF/com/android/metadata and META-INF/com/google/android/updater-script with the ROM-information.

Should probably create a sript that does all this automatically.

4 Likes

Do you have the source for that update-binary that is used to unpack it all? all other packages I found so far seem to use a shell script in its place.

2 Likes

Should be this one (It’s part of recovery sources):

2 Likes

thanks. I found the source of the weird strings. It’s google’s “brotli”

this is still weird, but at least now I know where the weirdness comes from.

2 Likes

anyone here interested into looking into this?

the FP3’s chipset is supposed to include a radio chip. The driver is in the kernel, but doesn’t show up in dmesg, so it might be a device tree / configuration issue. Since the “chip” is actually an integral part of the system-on-a-chip it might be possible to copy paste the config from another device with the same hardware (linked in linked thread)

3 Likes

tl;dr
I read a lot. but I couldn’t find an answer to: WHAT was the solution in the end? How did you get the kernel running? What was not working?

2 Likes
3 Likes

From https://source.codeaurora.org/quic/la/kernel/lk/tree/platform/msm_shared/avb/libavb/avb_slot_verify.c?h=LA.UM.7.6.2.r1-09200-89xx.0 line 694 :

  if (vbmeta_header.flags & AVB_VBMETA_IMAGE_FLAGS_VERIFICATION_DISABLED) {
    AvbSlotVerifyResult sub_ret;
    avb_debugv(
        full_partition_name, ": VERIFICATION_DISABLED bit is set.\n", NULL);
    /* If load_requested_partitions() fail it is always a fatal
     * failure (e.g. ERROR_INVALID_ARGUMENT, ERROR_OOM, etc.) rather
     * than recoverable (e.g. one where result_should_continue()
     * returns true) and we want to convey that error.
     */
    sub_ret = load_requested_partitions(
        ops, requested_partitions, ab_suffix, slot_data);
    if (sub_ret != AVB_SLOT_VERIFY_RESULT_OK) {
      ret = sub_ret;
    }
    goto out;
  }

I guess that’s load_requested_partitions which is broken

3 Likes

the answer is chaos-magic :wink:

1 Like

please elaborate in more detail

1 Like

spell that @k4y0z - magic

@k4y0z managed to get it to work. Main changes:

  1. instead of compiling the kernel using the kernel build scripts, they used an android-build-environment which assembles and packages the kernel, dtb and dtbo containers using the scripts in

in combo with @k4y0z own kernel modifications which in turn are based on @z3ntu kernel modifications of the fairphone released kernel sources which were needed to get the kernel to actually compile, boot in general and with TWRP in particular.

But a big factor was probably using the android build environment instead of kernel make files to package the image with device tree in the way aboot wanted it.

for more details you have to ask the lords of chaos :slight_smile:

6 Likes

Actually using the android build environment ist just way more convenient, when building TWRP.
Im quite certain with my modifications and config it would also work using the build-scripts.

The main things to get it working where enabling (probably only needed for fastboot boot, not when flashing the kernel)
CONFIG_BUILD_ARM64_DT_OVERLAY=y
Which got rid of the “dtb not found” error, but requires using an external device tree compiler (DTC_EXT).
And patching the kernel to ignore the skip_initramfs flag the bootloader sets. (So it actually loads the TWRP initramfs instead of booting straight into the OS)
My config also enables the serial console (UART) which was usefull for debugging, but not required for it to work now.

7 Likes