English

FP3 custom rom development based on released source code

Tags: #<Tag:0x00007f05da2ad398> #<Tag:0x00007f05da2ad258> #<Tag:0x00007f05da2ad118>

good links. some scary things there regarding the qcom bootloader. we’re going to have to be careful to not brick our phones :wink:

what I meant is:
with a regular boot sequence from slot A, the bootloader

  1. loads the boot image from boot partition A
  2. extracts the kernel, ramdisk, and dtb from that boot image (basically reads it all into RAM and reads the header where what sits)
  3. reads the dtbo from the dtbo partition A
  4. patches the dtb in memory to apply the changes to dtb described in the overlay
  5. extracts the kernel and starts it - feeding it the address of ramdisk and the (now patched) device tree inc. overlay.

now if you say "boot <myimage.img>

  1. fastboot will push a boot image to the phone into RAM
  2. bootloader reads out header for addresses of image, ramdisk, dtb
  3. <<< this is the question - does the bootloader read out a dtbo? if so which one? dtbo A, dtbo B, whichever is the current one?
  4. bootloader might or might not try to patch the bootimg supplied dtb device tree with whatever overlay info it got from the partition or not
  5. bootloader aborts with “error: no dtb found”
  6. instead of starting the kernel as in step 5 above :wink:
3 Likes

Too late :grin: (waiting for stock rom or a rooted phone)

If the bootloader read the dtbo partition, it is on the active slot (pretty sure, from the test I did, which bricked my phone…). You can know which one is active with “getvar all”, or change it with “–set-active=b”

Maybe we should contact one of the devs who’ve ported twrp to moto g7

2 Likes

I didn’t manage to compile overlay dtbo 's
It’s possible to start compilation by setting

CONFIG_BUILD_ARM64_DT_OVERLAY=y

and

export EXT_DTC=$( which dtc )

(assuming you have a system wide dtc installed which supports the -@ directive. apt-get device-tree-compiler does the trick

however the actual compilation hits a lot of syntax errors in the device description files. @z3ntu I have no idea if this is something easily fixable or a sign fairphone never compiled any overlays, I can’t tell.
If only fastboot had a “pull” option to inspect partitions :wink:

1 Like

Interesting find: Slot A was active. I set it to slot B and tried to boot.

I still cannot boot anything I give it with

fastboot boot boot.img

Same as before (dtb not found)

But the phone is unable to boot into android from slot B

It can boot into recovery and after trying to boot into android and failing for 3 times, it enters into a minimal recovery menu instead. (only options are “try again” and “Factory data reset”)

warning: this decreased a counter, and set an “unbootable” flag getvars all now reports:
(bootloader) slot-retry-count:b:0
(bootloader) slot-unbootable:b:Yes

as far as I understood this situation from the links above, this will prevent slot b from booting at all in the future (including “fastboot boot” commands) and including if a good image is later flashed into that slot.
I don’t know if there’s a way to reset that.

OK, this is harmless.
The bootloader behaves exactly as described in
https://source.android.com/devices/bootloader/flashing-updating
if you force change the active slot with fastboot --set-active the “unbootable” flag is reset and the retry count is set back to 7

it still won’t boot though :wink:

it’s possible that manual boot overrides could be affected by an “unbootable” flag on the currently active slot, as suggested by ((sorry can#t find the link anymore))

1 Like

Actually, you can change its state :

~ % sudo fastboot getvar all
[...]
(bootloader) current-slot:a
(bootloader) slot-retry-count:b:0
(bootloader) slot-success:b:No
(bootloader) slot-active:b:No
(bootloader) slot-unbootable:b:Yes
(bootloader) slot-retry-count:a:1
(bootloader) slot-success:a:No
(bootloader) slot-active:a:Yes
(bootloader) slot-unbootable:a:No
[...]

~ % sudo fastboot --set-active=b
Setting current slot to 'b'                        OKAY [  0.023s]
Finished. Total time: 0.025s

~ % sudo fastboot reboot bootloader
Rebooting into bootloader                          OKAY [  0.002s]
Finished. Total time: 0.203s

~ % sudo fastboot getvar all
[...]
(bootloader) current-slot:b
(bootloader) slot-retry-count:b:7
(bootloader) slot-success:b:No
(bootloader) slot-active:b:Yes
(bootloader) slot-unbootable:b:No
(bootloader) slot-retry-count:a:1
(bootloader) slot-success:a:No
(bootloader) slot-active:a:No
(bootloader) slot-unbootable:a:No
[...]

But that’s very weird that you can’t boot from slot b. (I tried to flash system -it should be treble compatible-, without telling which slot… I guess the first time was on the slot b, and the 2nd -terrible- one on slot a. I had the same boot into recovery the first time. It looks like the slots are differents??)

1 Like

I don’t want to quote the whole post, but do I smell possible solution vectors here for the following? …

1 Like

I found some more info on the hardware:
The closest dts seems to be sdm632-mtp-s3.dts because a bugreport (adb bugreport something.zip) contains a full dmesg from boot:

<5>[    0.000000] Linux version 4.9.112-perf+ (build@15e77c5c4e08) (gcc version 4.9.x 20150123 (prerelease) (GCC) ) #2 SMP PREEMPT Tue Dec 17 14:35:21 CST 2019
<6>[    0.000000] Boot CPU: AArch64 Processor [51af8014]
<6>[    0.000000] Machine: Qualcomm Technologies, Inc. SDM632 PMI632 MTP S3

also:

<6>[    0.287704] socinfo_print: v0.10, id=349, ver=1.0, raw_id=186, raw_ver=0, hw_plat=8, hw_plat_ver=65536
<6>[    0.287704]  accessory_chip=0, hw_plat_subtype=3, pmic_model=65558, pmic_die_revision=65536 foundry_id=2 serial_number=XXX
  • id=349 is {MSM_CPU_SDM632, "SDM632"} (from socinfo.c)
  • hw_plat=8 and hw_plat_subtype=3 might correspond to qcom,board-id = <8 3>;
  • pmic_model=65558 is pmic-id 0x010016
  • if the pmic_die_revision corresponds to the other entries in the pmic-id then there is no matching dts…
4 Likes

Actually, that kinda makes sense, from
https://source.android.com/devices/bootloader/flashing-updating

The way an A/B update works is – assume working system is booted from slot A: (according to how I understand https://source.android.com/devices/tech/ota/ab/#post-installation )

  1. data of the OTA update is downloaded - and potentially streamed directly to the respective slot B partitions
  2. after writing to the partitions the new system partition is temporarily mounted and a post-install script is run which verifies some stuff
  3. after succesful update (with the old sys still running) the system is eventually rebooted
  4. the system now boots from slot B - using the NEW system
  5. the new system makes additional integrity checks - if they fail, phone boots back into working slot A
  6. (my guess) the old system in slot A is rendered non-operational
    (this could be implicit thanks to updated version numbers in the data partition – if that IS the case, then a device reset might actually render the old system bootable again)

#6 is testable :wink: I shall test that

#6 is indeed the case - after a factory reset, I was able to boot from slot b, and indeed the phone now says it has
Linux localhost 4.9.112-perf+ #2 SMP PREEMPT Fri Nov 15 01:22:55 CST 2019 aarch64
Build Number 8901.2.A.0101.20191115
and it says it has an update available

wiping /data again and booting from slot a I get
8901.2.A.0105.20191217
Linux localhost 4.9.112-perf+ #2 SMP PREEMPT Tue Dec 17 14:35:21 CST 2019 aarch64

so indeed both slots work - the non-active one should hold the previous to latest system update, and it will not boot unless you factory-reset / wipe data first.

2 Likes

There’s some more info which fields are used for what:

In the event that a given device tree is applicable to all hardware versions matching a given Platform Type / Subtype ID, the major/minior platform version fields in the board_id property shall both be specified as 0xff.

and

1 Like

Also relevant, bugreport dmesg contains the kernel command line. Only issue is a buffer overrun, its truncated after 4 lines, but still includes valuable info

Kernel command line: kpti=0 console=ttyMSM0,115200,n8 androidboot.console=ttyMSM0 androidboot.hardware=qcom msm_rtb.filter=0x237 ehci-hcd.park=3 lpm_levels.sleep_disabled=1 androidboot.bootdevice=7824900.sdhci earlycon=msm_serial_dm,0x78af000 firmware_class.path=/vendor/firmware_mnt/image androidboot.usbconfigfs=true loop.max_part=7 buildvariant=user androidboot.emmc=true androidboot.pcbastage=Reserved androidboot.verifiedbootstate=orange androidboot.keymaster=1 dm="…" root=/dev/dm-0 …

interesting is this ttyMSM0 – that’s a serial console, isn’t it?
when you open the phone, there’s several debug pads, I’d almost bet some of them are connected to a debugging UART. does android start a getty/root-shell on console by any chance?

Edit: it looks like there isn’t anything running - at least that I can see in “adb ps a” and “adb lsof” – now it could be hidden from shell user, but I fear all we would see is printk output, and maybe maybe a bit of advanced interfacing with the bootloader.

Edit:

@_tmp
regarding sdm632-mtp-s3.dts ( qcom,board-id = <8 3>; )
versus sdm632-ext-codec-mtp-s4.dts ( qcom,board-id = <8 4>; )

although the CPU reports as “S3”, the only difference between these two device descriptions is that sdm632-ext-codec-mtp-s4.dts (via sdm632-ext-codec-mtp-s4.dtsi ) loads sdm632-ext-audio-mtp.dtsi

sdm632-ext-audio-mtp.dtsi has no further includes, and it defines an
wsa881x amplifier connected to the wcd9335 audio circuit - and makes it available for the “msm8953-tasha-snd-card” audio driver

if you search for these chip ids further down in dmesg, they are actually present (that’s exactly the sound chip the FP3 uses)

my guess is the S3 description should be bootable (if the bootloader would let it) but you’d have no sound/microphone :wink:

1 Like

Could someone give the output of getprop ? It is needed to do the prop.default of the ramdisk.

Also, we might have more information about the device with :

adb logcat -d -b events | grep -i boot
adb logcat -d -b events | grep -i preload
### and more : 
adb logcat -d -b events
adb logcat -d -b system
adb logcat -d

link : https://elinux.org/Android_Booting

edit:
Once we will able to boot with our kernel, changing these lines in prop.default should grant a adb root access. Then we will be able to dump averything we need :

ro.secure=0
ro.debuggable=1
persist.sys.usb.config=adb # to get an access in recovery mode
ro.adb.secure=0
1 Like

The flag CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE is just adding the dtb files to Image.gz and saving it to Image.gz-dtb. (source : https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/arch/arm64/Makefile)

And here are the dtb files added in the kernel (I built) :

sdm632-pm8004-qrd-sku4.dtb
apq8053-lite-dragon-v1.0.dtb
apq8053-lat-concam-dev.dtb  or  apq8053-lat-concam-proto.dtb
sdm632-ext-codec-mtp-s4.dtb
sdm450-pmi8937-mtp.dtb
apq8053-ext-audio-mtp.dtb
apq8053-lite-dragon-v2.2.1.dtb  or  apq8053-lite-dragon-v2.2.dtb
msm8953-ipc.dtb
sdm632-qrd.dtb
apq8053-lite-dragon-v2.1.dtb
sda450-cdp.dtb  or  sdm450-cdp.dtb  or  sdm450-rcm.dtb
sda450-mtp.dtb  or  sdm450-mtp.dtb
sda450-cdp.dtb  or  sdm450-cdp.dtb  or  sdm450-rcm.dtb
msm8916-mtp.dtb
msm8953-pmi8937-ext-codec-mtp.dtb
sda450-pmi632-mtp-s3.dtb  or  sdm450-pmi632-mtp-s3.dtb
msm8953-pmi8937-mtp.dtb
msm8953-iot-mtp.dtb
sdm632-cdp-s2.dtb
sda450-pmi632-cdp-s2.dtb  or  sdm450-pmi632-cdp-s2.dtb
sdm632-mtp-s3.dtb
sdm450-qrd.dtb
sda450-pmi632-mtp-s3.dtb  or  sdm450-pmi632-mtp-s3.dtb
sdm450-iot-mtp.dtb
msm8953-pmi8937-cdp.dtb
msm8953-pmi8940-mtp.dtb

(script based on the header)

It seems that base_addr, kernel_offset, ramdisk_offset don’t have to be the same than the original boot.img. It just needs to have enough space to not be at the same addresses. I gave a look, and they are ok : kernel_addr + kernel_size = 0x80af34f8 < ramdisk_addr .
link : https://stackoverflow.com/questions/29803996/repack-existing-boot-img-using-mkbootimg-cyanogenmod-12-1-lollipop

Some people had experienced the ‘dtb not found’ problem because of an incompatible bootloader : https://discuss.96boards.org/t/build-kernel-image-for-dragon-board/906/11 .
Maybe we should try to boot with boot image header version 0, and without dtb. Also, it can be a good idea to tell which image was tried for each test (Image.gz, or Image.gz-dtb).

edit :
I have built an Image with the option CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE=n , if someone want to try it.
https://framadrop.org/r/5Ur1LXzM70#LJa+1rhMMa7yGp6VIS7Q3Mmz/tSVhVeZkLuS+ZmWJmk=

1 Like

Doesn’t boot

Wohoo, I didn’t know this existed for giving useful information :smile:

2 Likes

OK new approach:

@z3ntu posted in


a link to the OTA update. which is a zip file
the zip includes a payload.bin
payload.bin can be unpacked with

this reveals: the following partition images:
-rw-rw-r-- 1 android android 1450 Jan 7 19:01 aboot.img
-rw-rw-r-- 1 android android 16484865 Jan 7 19:01 boot.img
-rw-rw-r-- 1 android android 1297 Jan 7 19:01 cmnlib64.img
-rw-rw-r-- 1 android android 1021 Jan 7 19:01 cmnlib.img
-rw-rw-r-- 1 android android 1562 Jan 7 19:01 devcfg.img
-rw-rw-r-- 1 android android 0 Jan 7 19:01 dsp.img
-rw-rw-r-- 1 android android 433 Jan 7 19:01 dtbo.img
-rw-rw-r-- 1 android android 972 Jan 7 19:01 keymaster.img
-rw-rw-r-- 1 android android 4096 Jan 7 19:01 lksecapp.img
-rw-rw-r-- 1 android android 0 Jan 7 19:01 mdtp.img
-rw-rw-r-- 1 android android 11211446 Jan 7 19:01 modem.img
-rw-rw-r-- 1 android android 1151468 Jan 7 19:01 product.img
-rw-rw-r-- 1 android android 2505 Jan 7 19:01 rpm.img
-rw-rw-r-- 1 android android 4898 Jan 7 19:01 sbl1.img
-rw-rw-r-- 1 android android 67613036 Jan 7 19:01 system.img
-rw-rw-r-- 1 android android 883 Jan 7 19:01 tz.img
-rw-rw-r-- 1 android android 819 Jan 7 19:01 vbmeta.img
-rw-rw-r-- 1 android android 17725364 Jan 7 19:01 vendor.img

file *
aboot.img: data
boot.img: data
cmnlib64.img: data
cmnlib.img: data
devcfg.img: data
dsp.img: empty
dtbo.img: data
keymaster.img: data
lksecapp.img: data
mdtp.img: empty
modem.img: data
product.img: data
rpm.img: data
sbl1.img: data
system.img: Linux rev 1.0 ext2 filesystem data, UUID=9bbab6c6-a59c-5e6e-beb3-6d4b8baef1ed (extents) (large files) (huge files)
tz.img: data
vbmeta.img: data
vendor.img: Linux rev 1.0 ext2 filesystem data, UUID=2ac8771d-1ea8-5fe6-b1c8-64f9852f01b0, volume name “vendor” (extents) (large files) (huge files)

obvious remarks:
dtbo is only 400 bytes - that can’t be a huge device table overlay, but there’s something in it

“file” can’t identfy it, but most of the images are in what I’d call “BSDF2” format

head -c 5 *
==> aboot.img <==
BSDF2
==> boot.img <==
BSDF2
==> cmnlib64.img <==
BSDF2
==> cmnlib.img <==
BSDF2
==> devcfg.img <==
BSDF2
==> dsp.img <==

==> dtbo.img <==
BSDF2
==> keymaster.img <==
BSDF2
==> lksecapp.img <==
|�
==> mdtp.img <==

==> modem.img <==
BSDF2
==> product.img <==
BSDF2
==> rpm.img <==
BSDF2
==> sbl1.img <==
BSDF2
==> system.img <==

==> tz.img <==
BSDF2
==> vbmeta.img <==
BSDF2
==> vendor.img <==

wtf is BSDF2 ?

edit: the system.img is useful, since thats ext2 data. some useful contents should be gained from that. who knows maybe it can even be patched - some binaries replaced to have a shell and a ssh server - and flashed and still boots with stock kernel?

edit2: BSDF2 is apparently a “bsdiff” - a binary diff, so instead of providing the whole image, it’s only a list of differences. doesn’t help us without the original version.
edit3: I tried to decode the bsdiff file and realized it has the wrong header. it should read “BSDIFF40” but it says “BSDF2” – that’s not a standard bsdiff file, what exactly is it?

edit: from @_tmp 's post and some chinese logfiles and educated guessing and an old press announcement regarding A/B OTA updates, android compresses bsdiff with brotli (their own lz77 derivate) - but I haven’t found any file format description or code to parse that.

edit: and I can’t mount system.img either. it looks like an ext2 partition but its corrupt. I don’t get whats going on there

edit: now i get it, the version of payload_dumper I used was too old. it did not support some of the operations used. apparently the payload is in chunks with different “operations” defined on them, (compressed, non-compressed) and there are some which are completely unsupported creating a file with holes.
does anyone have a better payload decryptor?

edit: the one I used was this thing https://www.mediafire.com/file/pslxh616isribx6/payload_dumper.zip/file which I found in another forum, the zip contains python, but its patched to silently ignore decoding errors when it encounters unknown types, which is how it “successfully” generated corrupt images.

apparently android9 introduced new block types in the payload which encode data differently and the public decoders don’t support

6 Likes

You are able to mount it now? I don’t know if the system partition is sent entirely.
If needed, this can help with img files : https://github.com/anestisb/android-simg2img . But system.img doesn’t look like compressed.

note: system should be ext4 fs.

1 Like

Technically /system can’t be a,“diff” because if it were, updates would always fail.on any rooted phone or any changes to /system - which afaik even the update process itself makes.
a binary diff requires the guarantee that the original file is unchanged.
so /system must be complete in that payload.bin, just in unsupported encoding. Do you know if the AOSP android sourcecode has tools to manipulaze that?

I still need to check if that"sparse image converter" can help. Currently I’m on a bus.

4 Likes

Well, that’s perfect if we are able to have our own system (then we will be able to dump everything).
I’m looking into the payload_dumper and the payload.bin if we can have a workaround.

I guess you did python payload_dumper.py --diff ../OTA_1912/payload.bin with empty files in out ?
When it comes to system.img, I have this error : Unsupported type = 9
edit : I don’t this error with a payload.bin from a lineage zip, and I’m able to mount it

2 Likes

In googling i found3 different versions of that payload_dumper each supporting different "type"s - which i figured,are chunks of data in thepayload.bin with a header ( image name, size, offset, type ) that differ in compression, wether or not its a binary diff, wether or not a large chunk getscopied from the original image, etc…
theremust be a library in stock google android that does that … question - is it in the AOSP sources or would that be proprietary?

1 Like

Tried this, no luck:

qcom,msm-id = <349 0x0>;
qcom,board-id = <8 3>;
qcom,pmic-id = <0x010016 0x0 0x0 0x0>;
1 Like

The library handling the update process is platform/system/update_engine and the file describing the chunk types is here: https://android.googlesource.com/platform/system/update_engine/+/refs/heads/master/update_metadata.proto

I have run a modified payload_extractor.py on all the ota packages but so far I did not find any full partition image, mostly SOURCE_COPY, BROTLI_BSDIFF and ZERO chunks (several thousand chunks in one update package). And this doesn’t help without access to the original partitions.

1 Like