The bsdiff from the google repo is doing the patch file and not the new one. I’m going to change it bspatch patch an old file to a new one with a patch
1 Like
I finally got aversion of both psdiff and puffdiff compiled. Ill try to pretty that up a bit ( needs heavymakefile patching ) and put it back on github, maybe with a matching, patched payload_dumper. That way we can at least extract all available data from the payload.bin.
edit:
this version has support for the compressed psdiff variants.
there is a Makefile to compile them (there’s a number of dependencies, currently not listed, but you can install all them easily with apt-get -dev if make complains. tested on Ubuntu18
sbatch works correctly both with compressed and uncompressed diffs
I currently have an issue where the puffin “puffdiff” binary complains about a corrupted compression string when trying to do patch the system image
ah and there’s a commandline option “–fakediff” that allows to attempt an differential update simulation without having the original files - it simplay assumes the original files are “all zeroes” - so it still extracts all the data from the payload diff, but of course it cannot restore information that is not there in the first place.
feel free to try it out
3 Likes
I don’t understand most of what you are talking about, but I’d like to say that I’m watching this closely and encourage you to carry on. I’d love to see AOSP or a Custom ROM on the FP3 and I think that I am not alone. 
PS.: You might even earn the Developer badge. 
6 Likes
For the BSDF2 diff, the one from google has a binary called bspatch. It can extract everything from the BSDF2 patch but use some bytes from the sources (like the headers, most of the binary is still patched). Trying to figure out how is it possible to get the boot.img.
I got a corrupted zImage for the moment.
% file zImage.gz
zImage.gz: gzip compressed data, max compression, from Unix, original size modulo 2^32 514204299 gzip compressed data, unknown method, was "", has comment, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 514204299
It looks like vbmeta is just updating its certificates/hashs.
1 Like
% python payload_dumper.py --fake-diff ../OTA_1909/payload.bin
Traceback (most recent call last):
File "payload_dumper.py", line 16, in <module>
from update_payload import update_metadata_pb2 as um
ImportError: cannot import name 'update_metadata_pb2' from 'update_payload'
I guess “update_payload.py” is missing in the commit
2 Likes
About the boot.img :
% ./abootimg/abootimg -i boot.img
boot.img: kernel size : 0x00aeb4f8
boot.img: ramdisk size : 0x0073bc81
boot.img: Image page size : 0x00000800
boot.img: reserved memory for kernel : 0x00aeb800
boot.img: reserved memory ramdisk : 0x0073c000
boot.img: reserved memory second : 0x00000000
boot.img: total size (addition) : 0x01228000
(info from a built one)
The kernel start after one page size (we can see the gunzip magic header from zImage) :
% xxd boot.img| grep -A1 -B1 "00000800:"
000007f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000800: 1f8b 0800 0000 0000 0203 ec5b 0b70 14e7 ...........[.p..
00000810: 91ee 9959 3d10 0f09 adc4 4a82 781f 924d ...Y=.....J.x..M
It finishes at @ beginning + size :
>>> hex(0x800+0x00aeb4f8)
'0xaebcf8'
% xxd boot.img| grep -A1 -B1 "00aebcf0:"
00aebce0: ed07 b403 0000 c0bb c44b 0b2d f16e f23f .........K.-.n.?
00aebcf0: e838 2e34 00f2 f101 0000 0000 0000 0000 .8.4............
00aebd00: 0000 0000 0000 0000 0000 0000 0000 0000 ................
The ramdisk starts at the next page :
% xxd boot.img| grep -A1 -B1 "00aec000:"
00aebff0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00aec000: 1f8b 0800 0000 0000 0003 c4bd 095c 54e5 .............\T.
00aec010: fe3f fe9c 390c cc0c 8a0b 082e 2983 5402 .?..9.......).T.
And it finishes @ beginning + size
>>> hex(0x00aec000+0x0073bc81)
'0x1227c81'
% xxd boot.img| grep -A1 -B1 "01227c80:"
01227c70: 9fff fca7 23fe f7ff 008e 3310 0800 05ed ....#.....3.....
01227c80: 0000 0000 0000 0000 0000 0000 0000 0000 ................
01227c90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
Then comes the second etc.
Why all of this? We don’t have the headers of the boot.img created. If we can figure out what their size, we will be able to create a fake old boot.img (and then see what’s missing)
Some more information :
% xxd 12.bin | head -n3 # 12.bin is done from an empty old image + the patch
00000000: 0000 0000 0000 0000 0508 0000 0000 0000 ................
00000010: d801 0000 0000 0000 0000 0000 0000 0000 ................
00000020: 0000 0000 0000 0000 0000 0000 0200 0000 ................
The first 8 bytes are the boot image magic ANDROID!, the next word is the kernel size (0x???0805), then its map address, then the ramdisk size (0x???01d8), then its map address etc. cf the headers.
Then, I watched all the addresses matching 0x???0805+0x800 to know the kernel size. (there should be 0* until the next page, or little more). I didn’t find any pattern matching it.
edit: 0x0805 and 0x01d8 are parts of the patch which need the source bytes to do the regular dest bytes
If someone as any clue… I’m waiting to try @corvuscorax 's payload_dumper
1 Like
I’ll have a look asap
and fixed. the entire folder was missing, stupid .gitignore ;/
I changed the behaviour slightly. when doing a fakediff, the missing parts of the image are now filled with “0xef” instead of “0x00” - this makes it possible to distinguish between missing diff blocks and areas that have deliberately been zero-filled
you can also change that in update_payload/applier.py
function _ReadExtents
edit: I still have issues with the “puffin” executable. it builds but doesn’t parse the patches correctly !?
2 Likes
I guess you’re using python2. I changed update_metadata_pb2.py from the last google one. I also had to modify common.py (weird).
diff --git a/update_payload/common.py b/update_payload/common.py
index 9061a75..f17e310 100644
--- a/update_payload/common.py
+++ b/update_payload/common.py
@@ -25,7 +25,7 @@ from update_payload.error import PayloadError
#
# Constants.
#
-PSEUDO_EXTENT_MARKER = (1L << 64) - 1 # UINT64_MAX
+PSEUDO_EXTENT_MARKER = (1 << 64) - 1 # UINT64_MAX #py3 all int are long
SIG_ASN1_HEADER = (
'\x30\x31\x30\x0d\x06\x09\x60\x86'
@@ -146,7 +146,7 @@ def Read(file_obj, length, offset=None, hasher=None):
try:
data = file_obj.read(length)
- except IOError, e:
+ except IOError as e:
I still have a strange error :
% python payload_dumper.py --fakediff ../OTA_1912/payload.bin :(
Processing modem partitionDone
Traceback (most recent call last):
File "payload_dumper.py", line 124, in <module>
dump_part(apobj,part)
File "payload_dumper.py", line 70, in dump_part
apobj._ApplyOperations(part.operations, part.partition_name, old_file,
File "./FP3/OTA_update/payload_dumper/update_payload/applier.py", line 555, in _ApplyOperations
self._ApplySourceCopyOperation(op, op_name, old_part_file,
File "./FP3/OTA_update/payload_dumper/update_payload/applier.py", line 378, in _ApplySourceCopyOperation
in_data = _ReadExtents(old_part_file, op.src_extents, block_size)
File "./FP3/OTA_update/payload_dumper/update_payload/applier.py", line 111, in _ReadExtents
data = array.array('c')
ValueError: bad typecode (must be b, B, u, h, H, i, I, l, L, q, Q, f or d)
1 Like
2 images patches from fake boot.img. One from a file full of ‘a’ the other full of ‘b’ :
% xxd 12_from_b.bin | head -n 100 | grep "00000240:"
00000240: dc29 115d 5c4e beb8 44c2 53b0 6c34 8e32 .).]\N..D.S.l4.2
% xxd 12_from_a.bin | head -n 100 | grep "00000240:"
00000240: db28 105c 5b4d bdb7 43c1 52af 6b33 8d31 .(.\[M..C.R.k3.1
% xxd 12_from_a.bin | head -n 200 | grep "00000810:"
00000810: 6161 6161 6161 6161 4827 7408 4877 2728 aaaaaaaaH't.Hw'(
% xxd 12_from_b.bin | head -n 200 | grep "00000810:"
00000810: 6262 6262 6262 6262 4827 7408 4877 2728 bbbbbbbbH't.Hw'(
A part of the patch is an addition to the source bytes.
## usual bin patched
% cat dump_from_a | wc -l
1307904
## where the 16 bytes of the lines aren't similar (1 source byte is enough)
% diff dump_from_a dump_from_b | grep -e "^>" | wc -l
288965
## without the lines of 16 sources bytes
% diff dump_from_a dump_from_b | grep -v "6262 6262 6262 6262 6262 6262 6262 6262" | grep -e "^>" | wc -l
39786
More than 96,9% is similar, and we have 77,9% of the boot partition… Maybe if we cross the patches we would be able to find out the kernel size etc
1 Like
thats a python inbuilt function afaik, looks almost as if your python version is incompatible. which python are you running? this is python-2.7 code
1 Like
python3. I’ll try with python2
1 Like
% python run.py
[+] Patch patch_9.bin
[+] Nb ok: 15064450
[+] %ok: 72
[+] Patch patch_10.bin
[+] Nb ok: 15864338
[+] %ok: 79
[+] Patch patch_12.bin
[+] Nb ok: 16965786
[+] %ok: 81
I hope we’ll be able to have the zImage with some tricks. I didn’t look deeper, but it seems the ramdisk is the part with the most lost bytes. I couldn’t find any dtb header (“d00d feed”).
If I missed a patch, please tell me
note: a string coming often : qcom,4184448_Arima_8902EU
1 Like
I updated the code again - I think I figured out what PUFFDIFF really does.
PUFFDIFF is a binary diff, much like the other bspatch variants, but unlike them it patches compressed data directly. Internally the code deflates the original image content of selected chunks, applies binary patches to them and then re-compresses them.
This has the weird effect that the data can have a different size after patching.
This of course fails with “fake” input, since the decompression fails, so the data size ends up being incorrect. So I modified the code to just fill these chunks with 0xef.
so large parts of /system are actually compressed - interesting
1 Like
That’s part of the compiled device tree specification. The source is
arch/arm64/boot/dts/qcom/qg-batterydata-Kayo-3000mah-Nov4th2019-pmi632.dtsi:qcom,4184448_Arima_8902EU_3000mAh
So this is actually the FP3’s battery 
If its there more than once, this means there is more than one device configuration attached. I’ll check later if I can split them apart and decompile them. It would be interesting to see if its a plain concatenation or if there is a database header with an index of all supported chips.
of course decompilation will only work if the descriptors have no holes 
1 Like
It’s in a huge hole…
Actually I’m running out of idea. Unless we have more updates or a full update, we can’t built back the image
edit:
I couldn’t find any “Arima” in the Image kernel I built. Maybe shall we go to the compilation again?
edit2 :
Do we have any news from the users who were stock in boot loop and has contacted the support ?
1 Like
Arima is not in the compiled kernel image, only in the device tree. The device tree is obviously part of the boot image (that we know now) which was expected anyway. The question is is only one (correct) device tree descriptor in there, or are there many (looks like there’s many) and if they are just concatenated or if they are wrapped in a database (https://github.com/LineageOS/android_system_tools_dtbtool)
if they are complete, its possible to extract the info and the individual files from the database and the .dtb files can be decompiled back into .dts (which could be used in our kernel if they differ from the ones coming with the kernel sources) but of course only if there are no gaps 
@pigpig - do we have a complete list of links of all differential updates available so far? if we apply all patches one after the other on the same image it should be considerably more complete in the end than with just a single one - of course we need the correct order
1 Like
We won’t be able to reverse it. It is in a hole. The only thing that can help is a date with it Nov4th2019, and other part of strings coming periodically, as Arima. (I’ll post them in a few minutes on this post)
qcom,4184448_Arima_8902EUaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaNov4th2019aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
The parts of strings were : chg-ter and g-ranges
For the updates, I did that with run.py taking two different old files and saving only the same bytes after a patch (the others depend on the source bytes) and patching it again with the other patch etc.
1 Like
Security patch in order :
September 2019, v. A.0096, from A.0081 : https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/39ce8b1b568149d8ecb8b4b7fc0d37beefc388fb.zip
September 2019, v. A.0096, from A.0095 : https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/d1e85d55938fdd545fcdc4b4b11098c5d183636f.zip
October 2019, v. A.0101, from A.0096 : https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/a1b56846df6bb6c656a511eeae9f732f5b35fbf9.zip
December 2019, v. A.0105, from A.0101 : https://android.googleapis.com/packages/ota-api/fairphone_fp3_fp3/eab40c208c6bd6899b51e7103822972d7a40336c.zip
So. There must be an update for A.0095 somewhere. useless
2 Likes
since they are incremental, you need to look from what to what they update:
e9ce8: from 8901.2.A.0081.20190816 to 8901.2.A.0096.20191001/10011803
d1e85: from 8901.2.A.0095.20190926/09261600 to 8901.2.A.0096.20191001/10011803
a1b56: from 8901.2.A.0096.20191001 to 8901.2.A.0101.20191115
eab40 from 8901.2.A.0101.20191115 to 8901.2.A.0105.20191217
i think 8901.2.A.0105.20191217 is still the latest
patching the tiny one from 95 to 96 is probably not going to be very useful, there is little data in it and it patches to the same revision
I think 95 was a broken update that most users didn’t even get, as it was pulled back before it got to everyone
1 Like
I was editing it.
Actually with my script I did A.0081 -> A.0096 -> A.0101 -> A.0105.
But still, we can’t do that much without many more patches.
Is there anywhere we can have the date of coming update? It might be possible that some few people got the January one (if there is one).
1 Like