If I well understood, your new version of payload_dumper reached to do the new images ? If so, I’ll post my script to get the OTA update without the phone here : FP3 : MITM update engine and we’ll be able to have any new version of the stock rom 
Also, it would be nice to add a link to the bsdiff and puffin repo on the readme. OK, i’ll use git clone --recursive next time 
bsdiff and puffin are included as sub-repositories, just say git submodule init, git submodule update . or git clone --recursive on the whole thing, and then use the top level makefile to build it all
Edit: I updated the Readme. It was still from the original repo I forked and heavily outdated
Does anyone think that true FOSS OS’s will ever exist for the FP3? I hope to be able to flash real Linux like; UBports, Plasma Mobile and PostmarketOS, I know FP2 had this so I hope FP3 has this.
we already have a self compiled linux kernel with a ramdisk running. going to a linux distribution from there should be straightforward, as long as you don’t want to use the modem or wifi or any other hardware that needs proprietary qualcomm stuff. we can probably use the binaries from the /vendor partition to get some of it to run.
Question: are there ready to use “full fledged linux” GSI roms we can just put on ?
Edit: Another question
Ah by the way, maybe a stupid question, but would an Android10 GSI (respectively Lineage 17) image work on FP3 despite the vendor partition meant for Android9 ?
I don’t think so, since GSI and Linux dont really work together, however I think halium is the way forward, if anyone can compile halium for FP3 then I believe you would also be able to run WiFi + Modem because it uses the kernel that came with the phone.
Android 10 GSI should work, as it does on most treble compatible phones
As @sm01man said, GSI are android related, so I don’t think it 'll be usable as it is for linux mobile distribution.
What you did on payload_dumper is awesome! I now have the stock A0110 images. I’m going to upload my script to have the OTA update. But I don’t know if there is personnal phone information in it, and I don’t like that much
Almost. Yes only difference is filesize if the partition is bigger than the image, rest is zero padded.
The payload.bin includes patches, checksums for the images pre-patching and checksums post-patching. Given the correct original partitions, “my” tool (well, more or less code from android_update_engine scripts folder with some mods to ease compilation) will patch and pass the post-patching checksums.
The issue is, for most partition these post-patching images don’t match the images taken from a phone post update, so when doing an OTA update something else happens after the payload.bin is applied - either before or after the reboot.
I compared the output images of payload dumper with what I had on my phone post OTA:
The following partitions were correct/identical:
boot.img
dsp.img
dtbo.img
modem.img
product.img
system.img
vendor.img
The following partitions were NOT identical:
aboot.img
cmnlib64.img
cmnlib.img
devcfg.img
keymaster.img
lksecapp.img
mdtp.img
rpm.img
sbl1.img
tz.img
vbmeta.img
question is, what changes these during the update process, and how?
Edit: It could be something really stupid like the patched images not fully filling the respective partitions, so theres zero bytes at the end in a raw dump from the phone… trying to verify that 
<-<-that
Damned.
Could you give one of these patched (from phone) partition please? vbmeta for example
Nice job !
OK I’m stupid. its only zero bytes. The partitions on Fairphone are bigger than the image size, with the remainder being zero filled. Of course additional zeroes still change the checksum, that’s why the file checksums and the partition dump checksums don’t match. If I zero pad the images to partition size it all works
Edit: I just double verified:
Taken: System image A.0101 from November
I made the exact same observation yesterday about the dsp.img, Not sure exactly how that happened.
I will upload a corrected version of the installer today.
And I also created an installer for 110.
Your version of payload-dumper works great.
I had some issues, with product-partition apparently being mounted by TWRP and update failing because of that.
There will be a new TWRP as well.
stock firmware link to A.0101 (Nov 2019) - for anyone who needs it 
https://www.androidfilehost.com/?fid=4349826312261713361
Filename: Fairphone_FP3_8901.2.A.0101.20191115_11150009_user-release-keys.zip
md5sum: 051eb5375d1412f77c9b83c51e1c6e42
sha256sum: 5384b8bc71d4dd284f21626b59004e5b2b5cebf8ce855bca7852b2be086a4ac4
Independent Verification: You can cross check against vendor check sums
using payload.bin from https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/eab40c208c6bd6899b51e7103822972d7a40336c.zip and https://github.com/CorvusCorax/payload_dumper in --diff mode (will ignore trailing zero bytes).
Applying the payload.bin will get you to A.0105. Furher applying https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/996d9c3cd83bf55bcbf4674a7074885e2d66e0f8.zip payload bin will get you to A.0110
$ sha256sum *.img
845924ee31a3ad8ca86e122299679cc7d2cd192b64060b9e4009a784f6889056 aboot.img
c8695bc30189a80ca0f70fda81d34d821c9215420c0c22226d774a2b700f37cf boot.img
e67326e2050a9174a9d437e70e4954f97385ff22fd8e40c51115c582ffa7c13d cmnlib64.img
be941fd8a054104ed8fb2ebfefa00b742d379f4e784c9d5f3a1638e4e7329355 cmnlib.img
c5e571383664f1704a033a4055e4dfffaa1d60d07fe15e7616590d129dda939a devcfg.img
feaa15a42cb63bc81fffe93e79eaa9262c8c1818abe40f43ba713c5184ac5839 dsp.img
cade61b50b2ba8a3c866b8ea928162bafe8c33386d712a9a78cf0857b1ef6833 dtbo.img
1982e8cec6f9fe3984da7d1184346e1a437013e47be2bf5fcc72b91e6e24efe4 keymaster.img
3fdd919985501069641d59b16f7034609018186b2f33a6850e7f113a0fd0383e lksecapp.img
c47821e639220f11a04e6d03dc2f6e67fe73dcb8738ffd1968462ea98a21c7d6 mdtp.img
0933af7ceadc3cdda1e2a6ca15f8b354ab38aa15dcebb74019c822584562e84b modem.img
4e097e334c1231bc1d967fd8256211ef7c7d055b5f0defbd69306868df8857d9 product.img
95994f93282b372173057b045f9be0f29262ca982f787e331bb16678409a0540 rpm.img
8139a8f922287f0a6e6da6c627503bf72910f591429a815d918ed4dffdf966eb sbl1.img
6ac7ba7e3bd58f4c80b388a3812bafa78f90e19b6474b7a36e913cf0b2437cca system.img
e9ce4855657b7ecf63b777a644a6ce1bde0da616826ae3e1bd3e18f1334fe4f3 tz.img
3248b79a1f79a4648cdad89398a41f8d8ac6ec11e6b7d9ca235b659d2839ea9c vbmeta.img
c95a77493d4a79a303ce388bde11c172a64642f9e12dd2546a9f4026e64aa5cd vendor.img
verification:
$ python payload_dumper.py --diff payload.bin
Processing modem partition... Checking original Sha256, should be CTOvfOrcPN2h4qbKFfizVKs4qhXc67dAGcgiWEVi6Es=
... passed...
extracting... truncating... Checking new Sha256, should be uHW4LbgFtXmxV1+teceezEVABCib10a2i3B2gW75+HY=
... passed... done
Processing sbl1 partition... Checking original Sha256, should be kar6XUqbymOV72Qh/oFfe9mUupFuPu75/WabRFQm1D0=
... passed...
extracting... truncating... Checking new Sha256, should be 3iRULfan90ZPWy2QHzeaIywOUEqkuyYgBWKyf9Hk6PU=
... passed... done
Processing rpm partition... Checking original Sha256, should be EwlUjVBzt+j66Mg3/KT+nURNTxjfRSCP0cKE6mJ+jRQ=
... passed...
extracting... truncating... Checking new Sha256, should be kikeLNG4xKlPqaQqzPcj3JQyfwiwen9nj+S1MgcpjEM=
... passed... done
Processing tz partition... Checking original Sha256, should be 7w0fA6NRXHIuw0m2HE7A2q4IeQDjm19oxtOc6sq3ou8=
... passed...
extracting... truncating... Checking new Sha256, should be qXgGb2NgGOUVRV+CxVvB8kX0EpI62133/4ytrQZg/X8=
... passed... done
Processing devcfg partition... Checking original Sha256, should be a5OHX+pL9/cL11bqSNz41MQp6HuQBJ3ZJGBZIGAbdQg=
... passed...
extracting... truncating... Checking new Sha256, should be T35U136bWdVNyZuJcvQTzJrj1CgmTMCO5kLFYvIAK/0=
... passed... done
Processing dsp partition... Checking original Sha256, should be /qoVpCy2O8gf/+k+eeqpJiyMGBir5A9DunE8UYSsWDk=
... passed...
extracting... truncating... Checking new Sha256, should be /qoVpCy2O8gf/+k+eeqpJiyMGBir5A9DunE8UYSsWDk=
... passed... done
Processing aboot partition... Checking original Sha256, should be wKA7kN0CoGNQcfbg7Bpn+DydHQlqmSnWtrParETMhII=
... passed...
extracting... truncating... Checking new Sha256, should be KXmQci8cU5P9IUKXol2o4qw54kUE6y4mzB4TFL6/Pww=
... passed... done
Processing dtbo partition... Checking original Sha256, should be yt5htQsrqKPIZrjqkoFiuv6MMzhtcSqaeM8IV7HvaDM=
... passed...
extracting... truncating... Checking new Sha256, should be kEUcBPYvu+pz5cUCb7qx9KmLjkeGlqpWlvtKHID7itw=
... passed... done
Processing vbmeta partition... Checking original Sha256, should be mTSdCq7aC8ZIqODfX4Z02wvTqs6E75OSY7jw97vxARo=
... passed...
extracting... truncating... Checking new Sha256, should be SuaZ/8GipVzd68nhGlpPPFDJXgQNNeMc5NYMYfOW//c=
... passed... done
Processing boot partition... Checking original Sha256, should be yGlbwwGJqAyg9w/agdNNghySFUIMDCIibXdKK3APN88=
... passed...
extracting... truncating... Checking new Sha256, should be /I7hvKKxvGXORU+CCdH2epRBtbdoySCkpxJrv80xfgo=
... passed... done
Processing system partition... Checking original Sha256, should be ase6fjvVj0yAs4ijgSuvp4+Q4ZtkdLejbpE88LJDfMo=
... passed...
extracting... truncating... Checking new Sha256, should be Liw8mKN2aLsNgVHs/tu5ziCLclop6dynwnxFSqg1H7U=
... passed... done
Processing vendor partition... Checking original Sha256, should be yVp3ST1KeaMDzjiL3hHBcqZGQvnhLdJUap9AJuZKpc0=
... passed...
extracting... truncating... Checking new Sha256, should be PpYt1c0pixRKlOlQYBmbNO1ti1aBCBptkvtqACiDDx4=
... passed... done
Processing mdtp partition... Checking original Sha256, should be 3UmJGHRUbFJowuS3vjyVsTd33YzQpt/ER9vY/Fkn4rk=
... passed...
extracting... truncating... Checking new Sha256, should be D9f4qXZyV2tkFGfhr30tZ6Aj6ErFTQSry8Ni6OGybwI=
... passed... done
Processing lksecapp partition... Checking original Sha256, should be OS0HWbkaczBKrVZ+7difoybiOeHzZInqO7vHL5hXHB4=
... passed...
extracting... truncating... Checking new Sha256, should be 4C0eklWxwA5qnPYXWMWBcluzIrqHv5CnMdnINYDMZWM=
... passed... done
Processing cmnlib partition... Checking original Sha256, should be m/VLgHlSQt/NukkbIPFpn0lqUWB3EzLYnLMYK8epIRg=
... passed...
extracting... truncating... Checking new Sha256, should be JZxlbOinSAO23eysX0R64jpHv/xluM+BOSssri7+BiI=
... passed... done
Processing cmnlib64 partition... Checking original Sha256, should be 4y8JsNsSKktjJxJq/iWEzy0Inrtvi4V8gLJTifw4mGo=
... passed...
extracting... truncating... Checking new Sha256, should be +VrbC5/Ccalf+WQ6TpcncvLC4tWF2+1qwyRm3idU+5U=
... passed... done
Processing keymaster partition... Checking original Sha256, should be dXiXiRgYC5pFJV8EqSnhoeBajQcl45yhufrFrXm1dRw=
... passed...
extracting... truncating... Checking new Sha256, should be Vd6R61DSpG34I+nHkogdpmx8krdbAlUzZkuIS2JD+2I=
... passed... done
Processing product partition... Checking original Sha256, should be Tgl+M0wSMbwdln/YJWIR73x9BVtfDe+9aTBoaN+IV9k=
... passed...
extracting... truncating... Checking new Sha256, should be 0I32cRejEBKI2z60Ht03+8Jd4+nNstHlzT8//VFzEk0=
... passed... doneDo you have any doc to do this kind of zip?
no, and i don’t think you can actually use it for direct sideloading - I just duplicated the format @k4y0z used in his images. Only content are the partitions (format <partition_name>.img ) that are included in every OTA update, which are dumped from the phone. I dumped them like this (example for slot A):
fastboot boot twrp.img
adb shell
$ cat /dev/block/by-name/system_a >/external_sd/system.img
for each partition name in slot a, then i got the images down with adb pull
the metadata I added by analogy by copying info from the one in the differential update for the same version, I don’t know if that’s good for anything.
then just zip it all together with maximum compression
the list of partitions I simply used the same set that the FP OTA update updates (payload_updater in fakediff mode will list them)
Ok, and did you try the zip?
as in flashing it on the phone again? no, not yet. since it’d be a downgrade, that likely wouldn’t boot without erasing userdata, which is something i want to avoid right now since funny enough I actually need the phone for everyday use right now that’s also why i haven’t rooted or installed LOS yet.
I don’t know any better way to re-install those partitions than you described in TWRP for Fairphone 3 which would be one by one, using fastboot.
I know I was previously able to boot from that slot (which got overwritten in the latest update) but only if I erased userdata (mentioned upthread)
edit: as I understand it, if you installed the latest version using OTA, you can always go back to the previous version (if the phone is unlocked) by simply switching slots and erasing userdata. the OTA update will install the new version in the currently unused slot, then switch slots, but it doesn’t erase or alter the old one (if it did, my images would have failed checksum tests)
Actually, the flashable zip is a really good way to re install the partitions :). For the fastboot cmds, you can do a little .sh
I understand, that’s my daily phone too ^^. I have done too many times the erase userdata haha ! And the backup of the data with TWRP works pretty well. You can loose your documents, your photos and you have to re-set your password but everything else works. But I’m not sure, and I won’t test, that it works with different ROMs.
As far as I understand it, the zip in the format we have it is not directly flashable because the system partition is too large. @k4y0z overcame that to create a “sparse” image, but while that might install via sideloading I don’t know how to create, unpack or verify that. (I looked into the zip, but couldn’t figure out the file formats)
I can’t make one of those without further documentation, but most likely anyone can make a sparse image from the full image, later.
Edit: found this:
I am using img2simg first to create a sparse image from the ext4 (that’s available as a package in my debian).
Then I use img2sdat to create the dat-files.
and then i pass it through brotli -q 6 (also available in debian) to create the brotli-files.
I do this for system, vendor and product.
The rest of the partitions are straight dd-images.
You can do the reverse process for confirming they result in the same images.
Then update META-INF/com/android/metadata and META-INF/com/google/android/updater-script with the ROM-information.
Should probably create a sript that does all this automatically.
Do you have the source for that update-binary that is used to unpack it all? all other packages I found so far seem to use a shell script in its place.
Should be this one (It’s part of recovery sources):