English

FP2: Compiling a blob-free variant of Fairphone OS

The tar ball contains a lot of stuff. More than just some binary blobs. If you look at it, it also contains config files, data files, libs (.so). All also “protected” with funny headers “Qualcomm Technologies Proprietary and Confidential.” The “real” proprietary firmware (.fw,.bin,.b*) mostly ends up in “product/FP2/system/etc/firmware”.

I don’t see a problem with the last part (/etc/firmware), but I’m not so sure about the rest. I think the images would have to be opened and “repacked” on the users machine to add the libs again I think. And even linking the code … I’m not so sure about that, but I would like to see an official answer on this. I guess it’s a no, but a automatic build service could still be provided by FP if they find the manpower.

Or to put it very simple:

a) Build working code without "bin blobs"
b) Build working code with “bin blobs”

Make a diff/compare checksums. Check if you can make “a” match “b” just by copying files from the “bin blob” tarball into the images.

2 Likes

Thanks for the input (I am just too busy these days to investigate the details myself).

Provided you can build the system without the blobs (and the proprietary config files with the funny headers) and without linking to these parts during compilcation - why do you think the image would need to be unpacked and repacked? The files normally just end up on the system partition, so having just a flashable ZIP which will copy them there should do the job? Of course, to produce this flashable ZIP we would need need to build some script/whatever which either uses the files present on the device already or the archive which is available on the web.

Not sure if I understand the question. The system files in the out directory are packed up into an image. You will see something like that during the build. This also includes the famous “vendor/lib” directory.

mkuserimg.sh -s out/target/product/FP2/system out/target/product/FP2/obj/PACKAGING/systemimage_intermediates/system.img ext4 system 2147483648 out/target/product/FP2/root/file_contexts

The script is located here:
out/host/linux-x86/obj/EXECUTABLES/mkuserimg.sh_intermediates/mkuserimg.sh

I think the idea was to pack the blobs and any other non-distributable files into their own flashable zip. Then you could flash the blob-free system image first and add the blobs by installing another zip just like you could do with e.g. Open GApps.

2 Likes

Yes exactly and thank you - gapps is a perfect example for illustrating how I suppose this to work for the end-user.

Perhaps one could try to persuade Fairphone to make a flashable blob zip available on code.fairphone.com. :slight_smile:

3 Likes

It would be nice if fairphone’s dev could participate to this thread. Maybe they can bring some useful information or suggestion…

Maybe @keesj can help ?

3 Likes

Hello,

We are willing to cooperate and help find solutions to the problems found in this area. First I want to give a little context that might help guide the discussion:

  1. The code as published on code.fairphone.com is based on publicly available that can be found on https://www.codeaurora.org/ (from Qualcomm). The very interesting property of that code is that it already compiles without blobs. The result is not a fully working System … but it kinda does what this thread suggest Fairphone should be doing.
  2. @FOSDEM I learned that the msm kernel mainline already has support for some peripherals where we still use blobs for (e.g. the Graphics driver) It would be a good thing to start working on getting the mainline kernel to work on FP2
  3. CM and Firefox OS have solutions to the blobs problem (by fetching them from an official build or an existing device) and what the current code.fairphone.com is using is not much different. We are able to do a more grained deliveries.

We are internally discussing the possibilities to work around the blobs problem but have so far not come up with a solution to all the problems. One thing we can do is build something (say B2G) based on open source code hosted elsewhere and offer that as a user installable OS but that still does not make it possible for our community members to hosts builds themselves.

A pure open source version(with blobs stored on the device) might be the sweet spot

7 Likes

Sounds interesting. Can you give feedback on the code that is currently hosted on code.fairphone.com?

Will it even compile without the bin blobs/tar-ball-files? (Is there more to it than an extra QC libart and some other libs?). I think cleaning up the misunderstanding what code.fairphone.com is intended to be would already help a lot. Is it “just” there to build with the ODM blobs or is it also meant to support work like codeaurora and a current msm kernel? Or are both projects even the same? I have no idea what QC is sharing/not sharing.

Currently I read it like this: Without the blob-tar-ball your are not able to build images that would boot and show graphics on the FP2. Is that correct? I only tried building without the bin-blob-tar-ball a few times and to me it “only” looked like were some problems switching to the plain libartd correctly, but I always thought there would be enough working code to run&boot the phone.

Would a “fix it later” by just copying libs/configs/blobs around after the compilation approach even work at all? So are most QC libs just a “snap-in” replacement of already existing ones?

This would be interesting to know!

3 Likes

It’s in his first bullet point:

That’s the interesting question. Has anyone tried this yet?

1 Like

Not for for me the last time I tried, but maybe that’s just my problem. I have not looked into this, I wanted to know this from the project. I removed vendor/ resync’ed and hoped for the best, but I ran into a few problems, but they looked fixable (quoting myself here):

But if they would work, I would start comparing both builds out-dirs using checksums.

I guess I will try again and report back later.

Update: I hope I’m not too tried, but I did:

$ make clean
$ rm -fr vendor/
$ repo sync -c  # (for fixing vendor/ again)
$ . build/envsetup.sh
$ choosecombo 1 FP2 2
$ make -j8

It went through clean. I will now backup the out/ dir and start the next build, but this time with unpacking the bin blobs.

$ mv out out-without-blob-tar-ball # backup out/

2nd blob-out:

$ make clean
$ yes | sh fp2-sibon-2.0.1-blobs.sh
$ . build/envsetup.sh
$ choosecombo 1 FP2 2
$ make -j8

Done, both compiled successfully.

Very short report for now: Everything java/libart differs (=different libart). And of course a lot of libs will be added to FP2/system/vendor/lib. I will post a full comparison online, once I’ve figure out the best way to do the diff.

The diff is too long for a pastebin, I will have to clean it up a bit.

Warning: I modified the file to safe space and also to hide the most important proprietary config-files that could show up in diffs. I also removed all java/jar/apk/txt/img diffs, because they differ anyway. And some libs (to safe space).

http://pastebin.ca/3375084

If you have other ideas, questions, let me know!

5 Likes

I guess the next step from here is: What files are neither in the blob tar ball nor in the “blob” free version? Are there files that get “created”? And what files are in what folders. I try to figure this out if I find the time.

Update: What’s a good hosting service for a 23Mb diff file? I sorted and uniq’ed it, so only the files that differ between the two versions should show.

2 Likes

OT, but I recommend Hidrive if you are okay that every 50 downloads you have to update the link: https://www.free-hidrive.com/

So if I get this right, the “only blob files” you listed there are files which are not contained in the blob archive but get created from the files in the blob archive when you build with blobs included?

Okay, cleaning this up a bit. The following files can be found in the vendor tar ball, but end up modified in the blob build image. I’m not worried about the .apk/apps (signature, libart), but it looks like the .so’s below get modified during the build.

- 49a74eba45617e5c98d876d26d6ca1e6 lib/libaudioroute.so
- 53c7e8992d7218c5c202b9c494a7609f xbin/antradio_app
- 6229f6d968476067590cd519d06c2f54 lib/soundfx/libqcompostprocbundle.so
- b3e2e7b7de9c596975c9399b247c02f9 app/AntHalService/AntHalService.apk
- 77f758abeb54b1e7c67fc7b621d02920 lib/libqcomfm_jni.so
- ef7dc142542234807f92a01c6d89aada lib/hw/audio.primary.msm8974.so
- 497e52f88615c4ad9bedf962239e00df framework/qcom.fmradio.jar
- 43c0cd12d6213174e0a995e257c9f882 lib/libgps.utils.so
- 6ccab7b0c4170620efee0b1644f63921 lib/hw/copybit.msm8974.so
- a5366bb1f6fe5c831ded8e3b68476e36 lib/libantradio.so
- 0f6e7bc7315128bef7f9b528bcb07a90 app/FMRecord/FMRecord.apk
- 9678c20f056e044d235884f407586720 app/FM2/FM2.apk

Let’s have a closer look: Just GPS and audio/radio and, oh joy … “fitness-tracker” stuff (“ant”)?

- 43c0cd12d6213174e0a995e257c9f882 lib/libgps.utils.so  
- 49a74eba45617e5c98d876d26d6ca1e6 lib/libaudioroute.so
- 6229f6d968476067590cd519d06c2f54 lib/soundfx/libqcompostprocbundle.so
- 6ccab7b0c4170620efee0b1644f63921 lib/hw/copybit.msm8974.so
- 77f758abeb54b1e7c67fc7b621d02920 lib/libqcomfm_jni.so
- a5366bb1f6fe5c831ded8e3b68476e36 lib/libantradio.so
- ef7dc142542234807f92a01c6d89aada lib/hw/audio.primary.msm8974.so

Most modified (different file size) between the versions are:

lib/libart-compiler.so  (snap in replacement)
lib/libwifi-service.so  (snap in replacement)
vendor/lib (all added from the tar ball, except the stuff mentioned above.)
bin/wcnss_service (snap in replacement)

And of course a lot of new stuff gets added to /etc (certs, firmware, permissions, debug-levels, calibration, wifi). Around 60 apps go into /app/ and bin/ gets around 100 new files as well. (All tar ball based not newly created).

4 Likes

So to put it briefly “just copying the files” will not work, as I feared. Just proving this took some time. Do you guys agree? @jftr @kuleszdl @keesj.

2 Likes

Yes, I agree. At least partially. I don’t know which functionality the blobs provide for the phone. My guess was that the phone can work in a limited fashion without them (although fundamental things like WiFi might not work).

What I don’t believe to work is compiling without the blobs and then just copying files in at a later time. I’ve just had a look at the files included in the archive. These are not only binaries and firmware themselves but also makefiles (with the .mk extension) which are part of the configuration of the build. They likely contain crucial information at compile time.

Libraries (*.so) are also likely necessary at compile time if something is compiled that links to them (which is likely).

What I think to be likely to not cause any problems are firmware blobs (e.g. the *.bin or *.fw files) and APKs. Of course, one still needs to know where they need to be copied to.

In other cases I’m unsure: My guess is that configuration files (e.g. with .cfg or .xml extension) are not necessary when compiling. It’s only a guess though.

Then there are also many scripts which could do something when building the script, however they might also be meant for running at the device or even for the update-process (i.e. when the image is flashed as an OTA package). One has to look at each of them separately to find out.

For some other file types I don’t have a clue.

To summarize: Many of the files are probably necessary at compile time or at least special precautions have to be taken to make it work without them.

3 Likes

Yep, some get linked (see the libs above). But I can send you a list of the “only added” file locations if you want to create a “cp” script for the rest (.bin, b*, fw*, xml, config, apk, …). But I’m not sure if it is worth the work.

I also doubt it. IMHO a more fundamental change is necessary to make it work. More decoupling would help. The best solution probably includes someone working at inclusion of the free parts that can replace some non-free parts (as @keesj mentioned already):

That means trying to build a mainline kernel for Fairphone 2 is a good next step.

2 Likes

I totally agree and that’s all I tried to show here anyway :slightly_smiling:

Also I don’t like the idea of linking code.

One of the Nemo guys is investigating this stuff that Sony has been working on with mainline kernels on Xperia devices with some mixed results. So it will be interesting to see how this goes with the Fairphone but I would suggest that perhaps its a little early just yet and all the parts arent there yet.

http://developer.sonymobile.com/knowledge-base/open-source/open-devices/how-to-build-and-flash-a-linux-kernel/how-to-build-mainline-linux-for-xperia-devices/

1 Like