FP3 custom rom development based on released source code

corvuscorax · January 29, 2020, 1:21pm

Yeah, for the majority of users, Qualcomm’s spying is a privacy nuisance which can be mitigated by simply disabling their apps and preventing the network traffic. The default installation will (most hopefully) not include “trustlets” that directly send user data to third parties.
The main security risk comes from the way QSEEOS is implemented.

It sits above the kernel with elevated privileges.
It, itself is a proprietary security nightmare which - unlike the Linux kernel - doesn’t get regular audits, which makes exploiting much more easy. (See linked blog series about the complete lack of security in 2016. Knowing proprietary security approaches, they will have applied the minimum patches to render the proof of concepts non-working without fixing the underlying architectural issues.)
The issue is as such not so much zero-days in Adroid/AOSP, but zero-days in the proprietary QSEE environment, as they are much more obscure which translates to “will remain open for a much longer time” - and will affect any phone with qualcomm hardware regardless of android version and patchlevel. In fact, because of the CPU-level privilege escalation there is no kernel level mitigation (other than disabling QSEECOM drivers, which as @k4y0z noted will render the modem inoperational)

JeroenH · January 29, 2020, 2:11pm

We practically all use proprietary firmware somewhere though, sometimes on ring 0 or lower. If it isn’t an implementation of TrustZone or SGX, it is Intel ME or AMD PSP or a WLAN driver. Practically everyone with a LTE or 5G modem suffers the same fate. The way I see it, if you don’t want to be vulnerable, stick to wired and ensure physical security. You can lower your attack surface by using less radios. For example, not use LTE/5G, or avoid using Bluetooth for anything serious.

There was a good talk about this on 36c3, All wireless communication stacks are equally broken.

Another alternative is using audited, FOSS implementations such as i.MX series, Talos II, RISC-V, etc. Do come with either less performance, less compatibility, or higher price tag. AFAIK there are no reproducible builds for such firmware though.

corvuscorax · January 30, 2020, 10:51pm

where should we upload/share stock ROM images and in which format? I have orig images of the phones partitions for A.0101 from November, A.0105 from December and the latest A.0110 from Jan 2020

Edit: as I understood some additional work is needed to change the raw image into correctly formatted .zip for sideloading

pigpig · January 30, 2020, 11:59pm

Maybe you can upload them on androidfilehost. I know that @k4y0z was looking for the A.0110

corvuscorax · January 31, 2020, 11:46am

working on it.

Ah by the way the image @Claymore1297 ( uploaded by @k4y0z ) provided above for A.0105 is corrupt. When I compared them to mine, I noticed a mismatch in the “dsp” partition.
So I tried to manually apply the January 2020 update to the full image, with active checksum tests using the checksums in the signed payload.bin turned out my image for A.0105 was correct, while the one from the zipfile fails its Sha256 check.

The over the air update expects dsp partition to be
/qoVpCy2O8gf/+k+eeqpJiyMGBir5A9DunE8UYSsWDk=
the one in

reads K4bYXDmSrwhYxLh4EeoJRoyOzggEi1fJuwj7hoO/Je8=

Interesting note: Applying the payload.bin via bindiff is not enough to get from an old full image to a new valid image. The update procedure on the phone does some funky stuff on the phone after applying the bindiff. So while the checksum tests integrated in the payload applier succeed, the resulting images are not identical to the full image after the update has been fully completed (including booting from the new version) <-<-that ended up being just padding zeroes. Manually patched image files are correct.

Edit: I updated GitHub - CorvusCorax/payload_dumper: Android OTA payload dumper to correctly patch and print checksums (both before and after patch) This allows verification of a system image if you have an OTA update that is based on it, since this includes signed checksums of each partition.

Edit: I requested dev access to androidfilehost, lets see if they approve it

In the meantime here is the correct dsp.img for A.105 dsp.img.zip.not_a.pdf (3.2 MB) (zipped and renamed to pdf to allow upload here)

pigpig · January 31, 2020, 12:29pm

If I well understood, your new version of payload_dumper reached to do the new images ? If so, I’ll post my script to get the OTA update without the phone here : FP3 : MITM update engine and we’ll be able to have any new version of the stock rom

~~Also, it would be nice to add a link to the bsdiff and puffin repo on the readme.~~ OK, i’ll use git clone --recursive next time

corvuscorax · January 31, 2020, 12:34pm

bsdiff and puffin are included as sub-repositories, just say git submodule init, git submodule update . or git clone --recursive on the whole thing, and then use the top level makefile to build it all

Edit: I updated the Readme. It was still from the original repo I forked and heavily outdated

sm01man · January 31, 2020, 1:43pm

Does anyone think that true FOSS OS’s will ever exist for the FP3? I hope to be able to flash real Linux like; UBports, Plasma Mobile and PostmarketOS, I know FP2 had this so I hope FP3 has this.

corvuscorax · January 31, 2020, 1:51pm

we already have a self compiled linux kernel with a ramdisk running. going to a linux distribution from there should be straightforward, as long as you don’t want to use the modem or wifi or any other hardware that needs proprietary qualcomm stuff. we can probably use the binaries from the /vendor partition to get some of it to run.

Question: are there ready to use “full fledged linux” GSI roms we can just put on ?

Edit: Another question
Ah by the way, maybe a stupid question, but would an Android10 GSI (respectively Lineage 17) image work on FP3 despite the vendor partition meant for Android9 ?

sm01man · January 31, 2020, 2:00pm

I don’t think so, since GSI and Linux dont really work together, however I think halium is the way forward, if anyone can compile halium for FP3 then I believe you would also be able to run WiFi + Modem because it uses the kernel that came with the phone.

pigpig · January 31, 2020, 2:12pm

Android 10 GSI should work, as it does on most treble compatible phones
As @sm01man said, GSI are android related, so I don’t think it 'll be usable as it is for linux mobile distribution.

What you did on payload_dumper is awesome! I now have the stock A0110 images. I’m going to upload my script to have the OTA update. But I don’t know if there is personnal phone information in it, and I don’t like that much

corvuscorax · January 31, 2020, 2:24pm

~~Almost.~~ Yes only difference is filesize if the partition is bigger than the image, rest is zero padded.

The payload.bin includes patches, checksums for the images pre-patching and checksums post-patching. Given the correct original partitions, “my” tool (well, more or less code from android_update_engine scripts folder with some mods to ease compilation) will patch and pass the post-patching checksums.

The issue is, for most partition these post-patching images don’t match the images taken from a phone post update, so when doing an OTA update something else happens after the payload.bin is applied - either before or after the reboot.

~~I compared the output images of payload dumper with what I had on my phone post OTA:~~

~~The following partitions were correct/identical:~~
~~boot.img~~
~~dsp.img~~
~~dtbo.img~~
~~modem.img~~
~~product.img~~
~~system.img~~
~~vendor.img~~

~~The following partitions were NOT identical:~~
~~aboot.img~~
~~cmnlib64.img~~
~~cmnlib.img~~
~~devcfg.img~~
~~keymaster.img~~
~~lksecapp.img~~
~~mdtp.img~~
~~rpm.img~~
~~sbl1.img~~
~~tz.img~~
~~vbmeta.img~~

~~question is, what changes these during the update process, and how?~~

Edit: It could be something really stupid like the patched images not fully filling the respective partitions, so theres zero bytes at the end in a raw dump from the phone… trying to verify that <-<-that

pigpig · January 31, 2020, 2:29pm

Damned.
Could you give one of these patched (from phone) partition please? vbmeta for example
Nice job !

corvuscorax · January 31, 2020, 2:35pm

OK I’m stupid. its only zero bytes. The partitions on Fairphone are bigger than the image size, with the remainder being zero filled. Of course additional zeroes still change the checksum, that’s why the file checksums and the partition dump checksums don’t match. If I zero pad the images to partition size it all works

Edit: I just double verified:
Taken: System image A.0101 from November

patch with payload_dumper and OTA zip to A.0105 (all checks succeeded)
patch with payload_dumper and OTA zip to A.0110 (all checks succeeded)
pad with zero bytes to size of raw partition and compare to raw dump from phone after both OTA were applied (exact match)

k4y0z · January 31, 2020, 3:01pm

I made the exact same observation yesterday about the dsp.img, Not sure exactly how that happened.
I will upload a corrected version of the installer today.
And I also created an installer for 110.
Your version of payload-dumper works great.
I had some issues, with product-partition apparently being mounted by TWRP and update failing because of that.
There will be a new TWRP as well.

corvuscorax · January 31, 2020, 5:52pm

stock firmware link to A.0101 (Nov 2019) - for anyone who needs it

https://www.androidfilehost.com/?fid=4349826312261713361
Filename: Fairphone_FP3_8901.2.A.0101.20191115_11150009_user-release-keys.zip
md5sum: 051eb5375d1412f77c9b83c51e1c6e42
sha256sum: 5384b8bc71d4dd284f21626b59004e5b2b5cebf8ce855bca7852b2be086a4ac4

Independent Verification: You can cross check against vendor check sums
using payload.bin from https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/eab40c208c6bd6899b51e7103822972d7a40336c.zip and https://github.com/CorvusCorax/payload_dumper in --diff mode (will ignore trailing zero bytes).
Applying the payload.bin will get you to A.0105. Furher applying https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/996d9c3cd83bf55bcbf4674a7074885e2d66e0f8.zip payload bin will get you to A.0110

$ sha256sum *.img
845924ee31a3ad8ca86e122299679cc7d2cd192b64060b9e4009a784f6889056  aboot.img
c8695bc30189a80ca0f70fda81d34d821c9215420c0c22226d774a2b700f37cf  boot.img
e67326e2050a9174a9d437e70e4954f97385ff22fd8e40c51115c582ffa7c13d  cmnlib64.img
be941fd8a054104ed8fb2ebfefa00b742d379f4e784c9d5f3a1638e4e7329355  cmnlib.img
c5e571383664f1704a033a4055e4dfffaa1d60d07fe15e7616590d129dda939a  devcfg.img
feaa15a42cb63bc81fffe93e79eaa9262c8c1818abe40f43ba713c5184ac5839  dsp.img
cade61b50b2ba8a3c866b8ea928162bafe8c33386d712a9a78cf0857b1ef6833  dtbo.img
1982e8cec6f9fe3984da7d1184346e1a437013e47be2bf5fcc72b91e6e24efe4  keymaster.img
3fdd919985501069641d59b16f7034609018186b2f33a6850e7f113a0fd0383e  lksecapp.img
c47821e639220f11a04e6d03dc2f6e67fe73dcb8738ffd1968462ea98a21c7d6  mdtp.img
0933af7ceadc3cdda1e2a6ca15f8b354ab38aa15dcebb74019c822584562e84b  modem.img
4e097e334c1231bc1d967fd8256211ef7c7d055b5f0defbd69306868df8857d9  product.img
95994f93282b372173057b045f9be0f29262ca982f787e331bb16678409a0540  rpm.img
8139a8f922287f0a6e6da6c627503bf72910f591429a815d918ed4dffdf966eb  sbl1.img
6ac7ba7e3bd58f4c80b388a3812bafa78f90e19b6474b7a36e913cf0b2437cca  system.img
e9ce4855657b7ecf63b777a644a6ce1bde0da616826ae3e1bd3e18f1334fe4f3  tz.img
3248b79a1f79a4648cdad89398a41f8d8ac6ec11e6b7d9ca235b659d2839ea9c  vbmeta.img
c95a77493d4a79a303ce388bde11c172a64642f9e12dd2546a9f4026e64aa5cd  vendor.img

verification:

$ python payload_dumper.py --diff payload.bin
Processing modem partition... Checking original Sha256, should be CTOvfOrcPN2h4qbKFfizVKs4qhXc67dAGcgiWEVi6Es=
 ... passed... 
extracting... truncating... Checking new Sha256, should be uHW4LbgFtXmxV1+teceezEVABCib10a2i3B2gW75+HY=
 ... passed... done
Processing sbl1 partition... Checking original Sha256, should be kar6XUqbymOV72Qh/oFfe9mUupFuPu75/WabRFQm1D0=
 ... passed... 
extracting... truncating... Checking new Sha256, should be 3iRULfan90ZPWy2QHzeaIywOUEqkuyYgBWKyf9Hk6PU=
 ... passed... done
Processing rpm partition... Checking original Sha256, should be EwlUjVBzt+j66Mg3/KT+nURNTxjfRSCP0cKE6mJ+jRQ=
 ... passed... 
extracting... truncating... Checking new Sha256, should be kikeLNG4xKlPqaQqzPcj3JQyfwiwen9nj+S1MgcpjEM=
 ... passed... done
Processing tz partition... Checking original Sha256, should be 7w0fA6NRXHIuw0m2HE7A2q4IeQDjm19oxtOc6sq3ou8=
 ... passed... 
extracting... truncating... Checking new Sha256, should be qXgGb2NgGOUVRV+CxVvB8kX0EpI62133/4ytrQZg/X8=
 ... passed... done
Processing devcfg partition... Checking original Sha256, should be a5OHX+pL9/cL11bqSNz41MQp6HuQBJ3ZJGBZIGAbdQg=
 ... passed... 
extracting... truncating... Checking new Sha256, should be T35U136bWdVNyZuJcvQTzJrj1CgmTMCO5kLFYvIAK/0=
 ... passed... done
Processing dsp partition... Checking original Sha256, should be /qoVpCy2O8gf/+k+eeqpJiyMGBir5A9DunE8UYSsWDk=
 ... passed... 
extracting... truncating... Checking new Sha256, should be /qoVpCy2O8gf/+k+eeqpJiyMGBir5A9DunE8UYSsWDk=
 ... passed... done
Processing aboot partition... Checking original Sha256, should be wKA7kN0CoGNQcfbg7Bpn+DydHQlqmSnWtrParETMhII=
 ... passed... 
extracting... truncating... Checking new Sha256, should be KXmQci8cU5P9IUKXol2o4qw54kUE6y4mzB4TFL6/Pww=
 ... passed... done
Processing dtbo partition... Checking original Sha256, should be yt5htQsrqKPIZrjqkoFiuv6MMzhtcSqaeM8IV7HvaDM=
 ... passed... 
extracting... truncating... Checking new Sha256, should be kEUcBPYvu+pz5cUCb7qx9KmLjkeGlqpWlvtKHID7itw=
 ... passed... done
Processing vbmeta partition... Checking original Sha256, should be mTSdCq7aC8ZIqODfX4Z02wvTqs6E75OSY7jw97vxARo=
 ... passed... 
extracting... truncating... Checking new Sha256, should be SuaZ/8GipVzd68nhGlpPPFDJXgQNNeMc5NYMYfOW//c=
 ... passed... done
Processing boot partition... Checking original Sha256, should be yGlbwwGJqAyg9w/agdNNghySFUIMDCIibXdKK3APN88=
 ... passed... 
extracting... truncating... Checking new Sha256, should be /I7hvKKxvGXORU+CCdH2epRBtbdoySCkpxJrv80xfgo=
 ... passed... done
Processing system partition... Checking original Sha256, should be ase6fjvVj0yAs4ijgSuvp4+Q4ZtkdLejbpE88LJDfMo=
 ... passed... 
extracting... truncating... Checking new Sha256, should be Liw8mKN2aLsNgVHs/tu5ziCLclop6dynwnxFSqg1H7U=
 ... passed... done
Processing vendor partition... Checking original Sha256, should be yVp3ST1KeaMDzjiL3hHBcqZGQvnhLdJUap9AJuZKpc0=
 ... passed... 
extracting... truncating... Checking new Sha256, should be PpYt1c0pixRKlOlQYBmbNO1ti1aBCBptkvtqACiDDx4=
 ... passed... done
Processing mdtp partition... Checking original Sha256, should be 3UmJGHRUbFJowuS3vjyVsTd33YzQpt/ER9vY/Fkn4rk=
 ... passed... 
extracting... truncating... Checking new Sha256, should be D9f4qXZyV2tkFGfhr30tZ6Aj6ErFTQSry8Ni6OGybwI=
 ... passed... done
Processing lksecapp partition... Checking original Sha256, should be OS0HWbkaczBKrVZ+7difoybiOeHzZInqO7vHL5hXHB4=
 ... passed... 
extracting... truncating... Checking new Sha256, should be 4C0eklWxwA5qnPYXWMWBcluzIrqHv5CnMdnINYDMZWM=
 ... passed... done
Processing cmnlib partition... Checking original Sha256, should be m/VLgHlSQt/NukkbIPFpn0lqUWB3EzLYnLMYK8epIRg=
 ... passed... 
extracting... truncating... Checking new Sha256, should be JZxlbOinSAO23eysX0R64jpHv/xluM+BOSssri7+BiI=
 ... passed... done
Processing cmnlib64 partition... Checking original Sha256, should be 4y8JsNsSKktjJxJq/iWEzy0Inrtvi4V8gLJTifw4mGo=
 ... passed... 
extracting... truncating... Checking new Sha256, should be +VrbC5/Ccalf+WQ6TpcncvLC4tWF2+1qwyRm3idU+5U=
 ... passed... done
Processing keymaster partition... Checking original Sha256, should be dXiXiRgYC5pFJV8EqSnhoeBajQcl45yhufrFrXm1dRw=
 ... passed... 
extracting... truncating... Checking new Sha256, should be Vd6R61DSpG34I+nHkogdpmx8krdbAlUzZkuIS2JD+2I=
 ... passed... done
Processing product partition... Checking original Sha256, should be Tgl+M0wSMbwdln/YJWIR73x9BVtfDe+9aTBoaN+IV9k=
 ... passed... 
extracting... truncating... Checking new Sha256, should be 0I32cRejEBKI2z60Ht03+8Jd4+nNstHlzT8//VFzEk0=
 ... passed... done

pigpig · January 31, 2020, 6:43pm

Do you have any doc to do this kind of zip?

corvuscorax · January 31, 2020, 6:53pm

no, and i don’t think you can actually use it for direct sideloading - I just duplicated the format @k4y0z used in his images. Only content are the partitions (format <partition_name>.img ) that are included in every OTA update, which are dumped from the phone. I dumped them like this (example for slot A):

fastboot boot twrp.img
adb shell
$ cat /dev/block/by-name/system_a >/external_sd/system.img

for each partition name in slot a, then i got the images down with adb pull
the metadata I added by analogy by copying info from the one in the differential update for the same version, I don’t know if that’s good for anything.

then just zip it all together with maximum compression

the list of partitions I simply used the same set that the FP OTA update updates (payload_updater in fakediff mode will list them)

pigpig · January 31, 2020, 7:02pm

Ok, and did you try the zip?

corvuscorax · January 31, 2020, 7:14pm

as in flashing it on the phone again? no, not yet. since it’d be a downgrade, that likely wouldn’t boot without erasing userdata, which is something i want to avoid right now since funny enough I actually need the phone for everyday use right now that’s also why i haven’t rooted or installed LOS yet.

I don’t know any better way to re-install those partitions than you described in TWRP for Fairphone 3 which would be one by one, using fastboot.
I know I was previously able to boot from that slot (which got overwritten in the latest update) but only if I erased userdata (mentioned upthread)

edit: as I understand it, if you installed the latest version using OTA, you can always go back to the previous version (if the phone is unlocked) by simply switching slots and erasing userdata. the OTA update will install the new version in the currently unused slot, then switch slots, but it doesn’t erase or alter the old one (if it did, my images would have failed checksum tests)