FP3 custom rom development based on released source code

Participate Development
, ,
41

Also relevant, bugreport dmesg contains the kernel command line. Only issue is a buffer overrun, its truncated after 4 lines, but still includes valuable info

Kernel command line: kpti=0 console=ttyMSM0,115200,n8 androidboot.console=ttyMSM0 androidboot.hardware=qcom msm_rtb.filter=0x237 ehci-hcd.park=3 lpm_levels.sleep_disabled=1 androidboot.bootdevice=7824900.sdhci earlycon=msm_serial_dm,0x78af000 firmware_class.path=/vendor/firmware_mnt/image androidboot.usbconfigfs=true loop.max_part=7 buildvariant=user androidboot.emmc=true androidboot.pcbastage=Reserved androidboot.verifiedbootstate=orange androidboot.keymaster=1 dm="…" root=/dev/dm-0 …

interesting is this ttyMSM0 – that’s a serial console, isn’t it?
when you open the phone, there’s several debug pads, I’d almost bet some of them are connected to a debugging UART. does android start a getty/root-shell on console by any chance?

Edit: it looks like there isn’t anything running - at least that I can see in “adb ps a” and “adb lsof” – now it could be hidden from shell user, but I fear all we would see is printk output, and maybe maybe a bit of advanced interfacing with the bootloader.

Edit:

@_tmp
regarding sdm632-mtp-s3.dts ( qcom,board-id = <8 3>; )
versus sdm632-ext-codec-mtp-s4.dts ( qcom,board-id = <8 4>; )

although the CPU reports as “S3”, the only difference between these two device descriptions is that sdm632-ext-codec-mtp-s4.dts (via sdm632-ext-codec-mtp-s4.dtsi ) loads sdm632-ext-audio-mtp.dtsi

sdm632-ext-audio-mtp.dtsi has no further includes, and it defines an
wsa881x amplifier connected to the wcd9335 audio circuit - and makes it available for the “msm8953-tasha-snd-card” audio driver

if you search for these chip ids further down in dmesg, they are actually present (that’s exactly the sound chip the FP3 uses)

my guess is the S3 description should be bootable (if the bootloader would let it) but you’d have no sound/microphone :wink:

1 Like
42

Could someone give the output of getprop ? It is needed to do the prop.default of the ramdisk.

Also, we might have more information about the device with :

adb logcat -d -b events | grep -i boot
adb logcat -d -b events | grep -i preload
### and more : 
adb logcat -d -b events
adb logcat -d -b system
adb logcat -d

link : https://elinux.org/Android_Booting

edit:
Once we will able to boot with our kernel, changing these lines in prop.default should grant a adb root access. Then we will be able to dump averything we need :

ro.secure=0
ro.debuggable=1
persist.sys.usb.config=adb # to get an access in recovery mode
ro.adb.secure=0
1 Like
43

The flag CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE is just adding the dtb files to Image.gz and saving it to Image.gz-dtb. (source : https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/arch/arm64/Makefile)

And here are the dtb files added in the kernel (I built) :

sdm632-pm8004-qrd-sku4.dtb
apq8053-lite-dragon-v1.0.dtb
apq8053-lat-concam-dev.dtb  or  apq8053-lat-concam-proto.dtb
sdm632-ext-codec-mtp-s4.dtb
sdm450-pmi8937-mtp.dtb
apq8053-ext-audio-mtp.dtb
apq8053-lite-dragon-v2.2.1.dtb  or  apq8053-lite-dragon-v2.2.dtb
msm8953-ipc.dtb
sdm632-qrd.dtb
apq8053-lite-dragon-v2.1.dtb
sda450-cdp.dtb  or  sdm450-cdp.dtb  or  sdm450-rcm.dtb
sda450-mtp.dtb  or  sdm450-mtp.dtb
sda450-cdp.dtb  or  sdm450-cdp.dtb  or  sdm450-rcm.dtb
msm8916-mtp.dtb
msm8953-pmi8937-ext-codec-mtp.dtb
sda450-pmi632-mtp-s3.dtb  or  sdm450-pmi632-mtp-s3.dtb
msm8953-pmi8937-mtp.dtb
msm8953-iot-mtp.dtb
sdm632-cdp-s2.dtb
sda450-pmi632-cdp-s2.dtb  or  sdm450-pmi632-cdp-s2.dtb
sdm632-mtp-s3.dtb
sdm450-qrd.dtb
sda450-pmi632-mtp-s3.dtb  or  sdm450-pmi632-mtp-s3.dtb
sdm450-iot-mtp.dtb
msm8953-pmi8937-cdp.dtb
msm8953-pmi8940-mtp.dtb

(script based on the header)

It seems that base_addr, kernel_offset, ramdisk_offset don’t have to be the same than the original boot.img. It just needs to have enough space to not be at the same addresses. I gave a look, and they are ok : kernel_addr + kernel_size = 0x80af34f8 < ramdisk_addr .
link : https://stackoverflow.com/questions/29803996/repack-existing-boot-img-using-mkbootimg-cyanogenmod-12-1-lollipop

Some people had experienced the ‘dtb not found’ problem because of an incompatible bootloader : https://discuss.96boards.org/t/build-kernel-image-for-dragon-board/906/11 .
Maybe we should try to boot with boot image header version 0, and without dtb. Also, it can be a good idea to tell which image was tried for each test (Image.gz, or Image.gz-dtb).

edit :
I have built an Image with the option CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE=n , if someone want to try it.
https://framadrop.org/r/5Ur1LXzM70#LJa+1rhMMa7yGp6VIS7Q3Mmz/tSVhVeZkLuS+ZmWJmk=

1 Like
44

Doesn’t boot

Wohoo, I didn’t know this existed for giving useful information :smile:

2 Likes
45

OK new approach:

@z3ntu posted in

a link to the OTA update. which is a zip file
the zip includes a payload.bin
payload.bin can be unpacked with

this reveals: the following partition images:
-rw-rw-r-- 1 android android 1450 Jan 7 19:01 aboot.img
-rw-rw-r-- 1 android android 16484865 Jan 7 19:01 boot.img
-rw-rw-r-- 1 android android 1297 Jan 7 19:01 cmnlib64.img
-rw-rw-r-- 1 android android 1021 Jan 7 19:01 cmnlib.img
-rw-rw-r-- 1 android android 1562 Jan 7 19:01 devcfg.img
-rw-rw-r-- 1 android android 0 Jan 7 19:01 dsp.img
-rw-rw-r-- 1 android android 433 Jan 7 19:01 dtbo.img
-rw-rw-r-- 1 android android 972 Jan 7 19:01 keymaster.img
-rw-rw-r-- 1 android android 4096 Jan 7 19:01 lksecapp.img
-rw-rw-r-- 1 android android 0 Jan 7 19:01 mdtp.img
-rw-rw-r-- 1 android android 11211446 Jan 7 19:01 modem.img
-rw-rw-r-- 1 android android 1151468 Jan 7 19:01 product.img
-rw-rw-r-- 1 android android 2505 Jan 7 19:01 rpm.img
-rw-rw-r-- 1 android android 4898 Jan 7 19:01 sbl1.img
-rw-rw-r-- 1 android android 67613036 Jan 7 19:01 system.img
-rw-rw-r-- 1 android android 883 Jan 7 19:01 tz.img
-rw-rw-r-- 1 android android 819 Jan 7 19:01 vbmeta.img
-rw-rw-r-- 1 android android 17725364 Jan 7 19:01 vendor.img

file *
aboot.img: data
boot.img: data
cmnlib64.img: data
cmnlib.img: data
devcfg.img: data
dsp.img: empty
dtbo.img: data
keymaster.img: data
lksecapp.img: data
mdtp.img: empty
modem.img: data
product.img: data
rpm.img: data
sbl1.img: data
system.img: Linux rev 1.0 ext2 filesystem data, UUID=9bbab6c6-a59c-5e6e-beb3-6d4b8baef1ed (extents) (large files) (huge files)
tz.img: data
vbmeta.img: data
vendor.img: Linux rev 1.0 ext2 filesystem data, UUID=2ac8771d-1ea8-5fe6-b1c8-64f9852f01b0, volume name “vendor” (extents) (large files) (huge files)

obvious remarks:
dtbo is only 400 bytes - that can’t be a huge device table overlay, but there’s something in it

“file” can’t identfy it, but most of the images are in what I’d call “BSDF2” format

head -c 5 *
==> aboot.img <==
BSDF2
==> boot.img <==
BSDF2
==> cmnlib64.img <==
BSDF2
==> cmnlib.img <==
BSDF2
==> devcfg.img <==
BSDF2
==> dsp.img <==

==> dtbo.img <==
BSDF2
==> keymaster.img <==
BSDF2
==> lksecapp.img <==
|�
==> mdtp.img <==

==> modem.img <==
BSDF2
==> product.img <==
BSDF2
==> rpm.img <==
BSDF2
==> sbl1.img <==
BSDF2
==> system.img <==

==> tz.img <==
BSDF2
==> vbmeta.img <==
BSDF2
==> vendor.img <==

wtf is BSDF2 ?

edit: the system.img is useful, since thats ext2 data. some useful contents should be gained from that. who knows maybe it can even be patched - some binaries replaced to have a shell and a ssh server - and flashed and still boots with stock kernel?

edit2: BSDF2 is apparently a “bsdiff” - a binary diff, so instead of providing the whole image, it’s only a list of differences. doesn’t help us without the original version.
edit3: I tried to decode the bsdiff file and realized it has the wrong header. it should read “BSDIFF40” but it says “BSDF2” – that’s not a standard bsdiff file, what exactly is it?

edit: from @_tmp 's post and some chinese logfiles and educated guessing and an old press announcement regarding A/B OTA updates, android compresses bsdiff with brotli (their own lz77 derivate) - but I haven’t found any file format description or code to parse that.

edit: and I can’t mount system.img either. it looks like an ext2 partition but its corrupt. I don’t get whats going on there

edit: now i get it, the version of payload_dumper I used was too old. it did not support some of the operations used. apparently the payload is in chunks with different “operations” defined on them, (compressed, non-compressed) and there are some which are completely unsupported creating a file with holes.
does anyone have a better payload decryptor?

edit: the one I used was this thing payload_dumper which I found in another forum, the zip contains python, but its patched to silently ignore decoding errors when it encounters unknown types, which is how it “successfully” generated corrupt images.

apparently android9 introduced new block types in the payload which encode data differently and the public decoders don’t support

6 Likes
47

You are able to mount it now? I don’t know if the system partition is sent entirely.
If needed, this can help with img files : GitHub - anestisb/android-simg2img: Tool to convert Android sparse images to raw images . But system.img doesn’t look like compressed.

note: system should be ext4 fs.

1 Like
48

Technically /system can’t be a,“diff” because if it were, updates would always fail.on any rooted phone or any changes to /system - which afaik even the update process itself makes.
a binary diff requires the guarantee that the original file is unchanged.
so /system must be complete in that payload.bin, just in unsupported encoding. Do you know if the AOSP android sourcecode has tools to manipulaze that?

I still need to check if that"sparse image converter" can help. Currently I’m on a bus.

4 Likes
49

Well, that’s perfect if we are able to have our own system (then we will be able to dump everything).
I’m looking into the payload_dumper and the payload.bin if we can have a workaround.

I guess you did python payload_dumper.py --diff ../OTA_1912/payload.bin with empty files in out ?
When it comes to system.img, I have this error : Unsupported type = 9
edit : I don’t this error with a payload.bin from a lineage zip, and I’m able to mount it

2 Likes
50

In googling i found3 different versions of that payload_dumper each supporting different "type"s - which i figured,are chunks of data in thepayload.bin with a header ( image name, size, offset, type ) that differ in compression, wether or not its a binary diff, wether or not a large chunk getscopied from the original image, etc…
theremust be a library in stock google android that does that … question - is it in the AOSP sources or would that be proprietary?

1 Like
51

Tried this, no luck:

qcom,msm-id = <349 0x0>;
qcom,board-id = <8 3>;
qcom,pmic-id = <0x010016 0x0 0x0 0x0>;
1 Like
52

The library handling the update process is platform/system/update_engine and the file describing the chunk types is here: https://android.googlesource.com/platform/system/update_engine/+/refs/heads/master/update_metadata.proto

I have run a modified payload_extractor.py on all the ota packages but so far I did not find any full partition image, mostly SOURCE_COPY, BROTLI_BSDIFF and ZERO chunks (several thousand chunks in one update package). And this doesn’t help without access to the original partitions.

1 Like
53

I think ZERO chunks probably just mean “insert X number of zeroes here” - question is how many exactly :wink: is there documentation anywhere?
btw in a stock android image (for a google phone)
system/product/priv-app/FactoryOta/FactoryOta.apk
handles the payload.bin (did a string search) - maybe adb pull can look at that?

1 Like
54

FactoryOta will deal with full payload. TWRP does the same too, and it’s open source (that how it installs the zip).

Edit:
from twrp source

./installcommand.cpp
45:static constexpr const char* AB_OTA_PAYLOAD_PROPERTIES = "payload_properties.txt";
46:static constexpr const char* AB_OTA_PAYLOAD = "payload.bin";
177:    // For A/B updates we extract the payload properties to a buffer and obtain
178:    // the RAW payload offset in the zip file.
183:    std::vector<unsigned char> payload_properties(
185:    if (!zip->ExtractToBuffer(AB_OTA_PAYLOAD_PROPERTIES, payload_properties.data())) {
194:    long payload_offset = zip->GetEntryOffset(AB_OTA_PAYLOAD);
197:        android::base::StringPrintf("--payload=file://%s", path),
198:        android::base::StringPrintf("--offset=%ld", payload_offset),
199:        "--headers=" + std::string(payload_properties.begin(),
200:                                   payload_properties.end()),

edit2: ignoring type 9, and I was able to mount system.img… without anything in it of course (corrupted I guess). But it sounds well to get a working system.img :slight_smile:

1 Like
55

extract_android_ota_payload can parse the relevant metadata and with some extra debug statements print something like:

op type: ZERO
dst_extents {
    start_block: 2052
    num_blocks: 512
}
op type: SOURCE_COPY
src_extents {
  start_block: 8287
  num_blocks: 2
}
src_extents {
  start_block: 8288
  num_blocks: 1
}
src_extents {
  start_block: 8288
  num_blocks: 1
}
src_extents {
  start_block: 8291
  num_blocks: 1
}
dst_extents {
  start_block: 8287
  num_blocks: 5
}

op type: BROTLI_BSDIFF
data_offset: 27741614
data_length: 42466
src_extents {
  start_block: 626078
  num_blocks: 1
}
src_extents {
  start_block: 626600
  num_blocks: 12
}
dst_extents {
  start_block: 8292
  num_blocks: 1
}
dst_extents {
  start_block: 8814
  num_blocks: 12
}

There are also some checksums in the metadata for all the chunks and source blocks.

But applying the operations is not implemented but it I also don’t see how this would help.

1 Like
56

+1
Unless we are able to find out the source data with the 2 others updates, we need a full OTA update.
The S stands for “SOURCE COPY”

Processing system partitionX.S.B.0.X.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.R.S.B.S.B.S.B.S.B.S.B.R.R.R.S.B.S.S.B.S.B.R.R.S.R.S.S.S.S.S.R.S.S.S.R.S.S.S.R.S.S.S.S.R.S.S.S.S
.R.S.R.S.B.S.R.S.R.S.R.S.R.S.R.S.R.S.R.R.S.R.S.S.S.R.S.R.S.S.R.S.R.S.R.S.R.S.R.S.R.S.R.S.R.S.R.S.B.R.S.R.S.R.S.R.S.R.S.R.R.S.R.S.R.S.B.S.B.R.S.R.S.R.S.S.S.S.R.S.R.S
.R.S.S.S.S.S.S.R.S.S.S.S.R.S.S.S.S.S.R.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.S.R.S.R.S.S.S.R.S.S.S.S.S.S.S.R.S.X.0.S.S.S.S.S.S.S.S.S.R.S
1 Like
57

Few interesting things : (R for replace, there is no source copy)

Processing vbmeta partitionR.Done
Processing boot partitionR.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.Done

edit : Still very interesting, but what I wrote after was wrong.

1 Like
58

BSDF2 is some weird binary diff format passed on to psdiff. The operation here is:

  1. parse the BSDF2 blob somehow (it has its headers stripped and replaced somehow so it wont decompress)
  2. uncomress the BSDF2 blob using the “brotli” (de)compression algorithm
  3. the result is a BSDIFF40 binary diff now behave as if type was SOURCE_BSDIFF

the system has the same also with type 9 which is PUFFDIFF
its treated the same as BROTLI_PSDIFF except that the header is PUF1 instead of PSDF2 and the compression algorithm is a zlib stream

I am currently stuck at step 1 for both. neither zlib nor brotli will accept the blob as is, nor the blob with the first bytes stripped. I think with type 9 i might be able to treat is as a “headerless zlib stream” but with type10 I don’t know since I don’t know what the brotli deflator expects

Edit: finally something better.

LineageOS has code to deal with this mess, and its in python:

this applier can apparently deal with all 10 types – it needs the source of course for any of the PSDIFF (including compressed versions) or SOURCE_COPY - but at least it can apply it

Edit: no it can’t… it uses external tools to apply both puffdiff and bspatch, but maybe i can finally find the repos where those come from. puff seems to come from

but for pspatch i only find

and that does not support PUFFDIFF nor BROTLI :frowning:

3 Likes
59

Try with the AOSP sources :
https://android.googlesource.com/platform/external/
https://android.googlesource.com/platform/external/bsdiff/

bsdiff_main.cc:

void PrintUsage(const std::string& proc_name) {
  std::cerr << "usage: " << proc_name
            << " [options] oldfile newfile patchfile\n";
  std::cerr << "  --format <legacy|bsdiff40|bsdf2|endsley>  The format of the"
               " bsdiff patch.\n"
            << "  --minlen LEN                       The minimum match length "
               "required to consider a match in the algorithm.\n"
            << "  --type <bz2|brotli|nocompression>  The algorithm to compress "
               "the patch, bsdf2 format only. Multiple supported compressors "
               "should be split by ':', e.g. bz2:brotli.\n"
            << "  --brotli_quality                   Quality of the brotli "
               "compressor.\n";
}

It comes from here : http://www.daemonology.net/bsdiff/ , juste like the lineage one.

2 Likes
60

and here is the other one:

https://android.googlesource.com/platform/external/puffin/

except the source is broken. it looks like not too long ago the “patch” ability was integrated directly into

which deprecated both the external bspatch and puffin libs, which in turn had functionality removed. the patch code in puffin does no longer compile.

I’m stopping work on this. I had hoped that we could get at least a system.img

the bad news out of this is, the “update” function in Fairphone apparently applies binary patch operations on the /system partition.

This means, once you flash /system with your own stuff, or even just mount /system writeable, the next OTA update will likely not just fail, but could in the worst case brick the device.

(OTA updates also update the bootloader)

not good. not good at all.

1 Like
62

You don’t care, once we have a root access (via system or boot), we will be able to dump every partitions and do a custom ROM :slight_smile:

1 Like