FP2: Compiling a blob-free variant of Fairphone OS

Participate Development
,
29

Sounds interesting. Can you give feedback on the code that is currently hosted on code.fairphone.com?

Will it even compile without the bin blobs/tar-ball-files? (Is there more to it than an extra QC libart and some other libs?). I think cleaning up the misunderstanding what code.fairphone.com is intended to be would already help a lot. Is it “just” there to build with the ODM blobs or is it also meant to support work like codeaurora and a current msm kernel? Or are both projects even the same? I have no idea what QC is sharing/not sharing.

Currently I read it like this: Without the blob-tar-ball your are not able to build images that would boot and show graphics on the FP2. Is that correct? I only tried building without the bin-blob-tar-ball a few times and to me it “only” looked like were some problems switching to the plain libartd correctly, but I always thought there would be enough working code to run&boot the phone.

Would a “fix it later” by just copying libs/configs/blobs around after the compilation approach even work at all? So are most QC libs just a “snap-in” replacement of already existing ones?

This would be interesting to know!

3 Likes
30

It’s in his first bullet point:

That’s the interesting question. Has anyone tried this yet?

1 Like
31

Not for for me the last time I tried, but maybe that’s just my problem. I have not looked into this, I wanted to know this from the project. I removed vendor/ resync’ed and hoped for the best, but I ran into a few problems, but they looked fixable (quoting myself here):

But if they would work, I would start comparing both builds out-dirs using checksums.

I guess I will try again and report back later.

Update: I hope I’m not too tried, but I did:

$ make clean
$ rm -fr vendor/
$ repo sync -c  # (for fixing vendor/ again)
$ . build/envsetup.sh
$ choosecombo 1 FP2 2
$ make -j8

It went through clean. I will now backup the out/ dir and start the next build, but this time with unpacking the bin blobs.

$ mv out out-without-blob-tar-ball # backup out/

2nd blob-out:

$ make clean
$ yes | sh fp2-sibon-2.0.1-blobs.sh
$ . build/envsetup.sh
$ choosecombo 1 FP2 2
$ make -j8

Done, both compiled successfully.

Very short report for now: Everything java/libart differs (=different libart). And of course a lot of libs will be added to FP2/system/vendor/lib. I will post a full comparison online, once I’ve figure out the best way to do the diff.

The diff is too long for a pastebin, I will have to clean it up a bit.

Warning: I modified the file to safe space and also to hide the most important proprietary config-files that could show up in diffs. I also removed all java/jar/apk/txt/img diffs, because they differ anyway. And some libs (to safe space).

http://pastebin.ca/3375084

If you have other ideas, questions, let me know!

5 Likes
32

I guess the next step from here is: What files are neither in the blob tar ball nor in the “blob” free version? Are there files that get “created”? And what files are in what folders. I try to figure this out if I find the time.

Update: What’s a good hosting service for a 23Mb diff file? I sorted and uniq’ed it, so only the files that differ between the two versions should show.

2 Likes
33

OT, but I recommend Hidrive if you are okay that every 50 downloads you have to update the link: https://www.free-hidrive.com/

So if I get this right, the “only blob files” you listed there are files which are not contained in the blob archive but get created from the files in the blob archive when you build with blobs included?

34

Okay, cleaning this up a bit. The following files can be found in the vendor tar ball, but end up modified in the blob build image. I’m not worried about the .apk/apps (signature, libart), but it looks like the .so’s below get modified during the build.

- 49a74eba45617e5c98d876d26d6ca1e6 lib/libaudioroute.so
- 53c7e8992d7218c5c202b9c494a7609f xbin/antradio_app
- 6229f6d968476067590cd519d06c2f54 lib/soundfx/libqcompostprocbundle.so
- b3e2e7b7de9c596975c9399b247c02f9 app/AntHalService/AntHalService.apk
- 77f758abeb54b1e7c67fc7b621d02920 lib/libqcomfm_jni.so
- ef7dc142542234807f92a01c6d89aada lib/hw/audio.primary.msm8974.so
- 497e52f88615c4ad9bedf962239e00df framework/qcom.fmradio.jar
- 43c0cd12d6213174e0a995e257c9f882 lib/libgps.utils.so
- 6ccab7b0c4170620efee0b1644f63921 lib/hw/copybit.msm8974.so
- a5366bb1f6fe5c831ded8e3b68476e36 lib/libantradio.so
- 0f6e7bc7315128bef7f9b528bcb07a90 app/FMRecord/FMRecord.apk
- 9678c20f056e044d235884f407586720 app/FM2/FM2.apk

Let’s have a closer look: Just GPS and audio/radio and, oh joy … “fitness-tracker” stuff (“ant”)?

- 43c0cd12d6213174e0a995e257c9f882 lib/libgps.utils.so  
- 49a74eba45617e5c98d876d26d6ca1e6 lib/libaudioroute.so
- 6229f6d968476067590cd519d06c2f54 lib/soundfx/libqcompostprocbundle.so
- 6ccab7b0c4170620efee0b1644f63921 lib/hw/copybit.msm8974.so
- 77f758abeb54b1e7c67fc7b621d02920 lib/libqcomfm_jni.so
- a5366bb1f6fe5c831ded8e3b68476e36 lib/libantradio.so
- ef7dc142542234807f92a01c6d89aada lib/hw/audio.primary.msm8974.so

Most modified (different file size) between the versions are:

lib/libart-compiler.so  (snap in replacement)
lib/libwifi-service.so  (snap in replacement)
vendor/lib (all added from the tar ball, except the stuff mentioned above.)
bin/wcnss_service (snap in replacement)

And of course a lot of new stuff gets added to /etc (certs, firmware, permissions, debug-levels, calibration, wifi). Around 60 apps go into /app/ and bin/ gets around 100 new files as well. (All tar ball based not newly created).

4 Likes
35

So to put it briefly “just copying the files” will not work, as I feared. Just proving this took some time. Do you guys agree? @jftr @kuleszdl @keesj.

2 Likes
36

Yes, I agree. At least partially. I don’t know which functionality the blobs provide for the phone. My guess was that the phone can work in a limited fashion without them (although fundamental things like WiFi might not work).

What I don’t believe to work is compiling without the blobs and then just copying files in at a later time. I’ve just had a look at the files included in the archive. These are not only binaries and firmware themselves but also makefiles (with the .mk extension) which are part of the configuration of the build. They likely contain crucial information at compile time.

Libraries (*.so) are also likely necessary at compile time if something is compiled that links to them (which is likely).

What I think to be likely to not cause any problems are firmware blobs (e.g. the *.bin or *.fw files) and APKs. Of course, one still needs to know where they need to be copied to.

In other cases I’m unsure: My guess is that configuration files (e.g. with .cfg or .xml extension) are not necessary when compiling. It’s only a guess though.

Then there are also many scripts which could do something when building the script, however they might also be meant for running at the device or even for the update-process (i.e. when the image is flashed as an OTA package). One has to look at each of them separately to find out.

For some other file types I don’t have a clue.

To summarize: Many of the files are probably necessary at compile time or at least special precautions have to be taken to make it work without them.

3 Likes
37

Yep, some get linked (see the libs above). But I can send you a list of the “only added” file locations if you want to create a “cp” script for the rest (.bin, b*, fw*, xml, config, apk, …). But I’m not sure if it is worth the work.

38

I also doubt it. IMHO a more fundamental change is necessary to make it work. More decoupling would help. The best solution probably includes someone working at inclusion of the free parts that can replace some non-free parts (as @keesj mentioned already):

That means trying to build a mainline kernel for Fairphone 2 is a good next step.

2 Likes
39

I totally agree and that’s all I tried to show here anyway :slightly_smiling:

Also I don’t like the idea of linking code.

40

One of the Nemo guys is investigating this stuff that Sony has been working on with mainline kernels on Xperia devices with some mixed results. So it will be interesting to see how this goes with the Fairphone but I would suggest that perhaps its a little early just yet and all the parts arent there yet.

http://developer.sonymobile.com/knowledge-base/open-source/open-devices/how-to-build-and-flash-a-linux-kernel/how-to-build-mainline-linux-for-xperia-devices/

1 Like
41

Did someone actually try flashing/booting the blobfree build? Which devices work, which don’t?

42

Can’t we upload it anyway? I have the blob-free images. I’m sure someone will test it :slightly_smiling:

I assume that a lot of things will not work. There are around 477 libs in the tar ball. Eight of those include the term “adreno” that’s the name of QC’s graphics chip … And over 30 .bin/.fw files. And around 70 config files … I’m sure I missed a lot of other important file types.

I hate to say it, but copying Sony is the way to go. :frowning:

But keep in mind only the few libs mentioned above are really tricky (gps, audio, radio, fitness stuff), the rest just gets copied anyway.

1 Like
43

I think most of the stuff that needs to be modified could be found here:

/kernel/arch/arm/boot/dts/ (dts for the chip, maybe already in the main kernel)

The qualcomm bin blob make files, I guess, can be found here:
cat device/qcom/*/*.mk

#FM
FM := qcom.fmradio
FM += libqcomfm_jni

#GPS
GPS_HARDWARE := gps.conf
GPS_HARDWARE += gps.default
GPS_HARDWARE += gps.mahimahi
GPS_HARDWARE += libloc_adapter
GPS_HARDWARE += libgps.utils
GPS_HARDWARE += libloc_eng
GPS_HARDWARE += libloc_api_v02

#ANT stack
PRODUCT_PACKAGES += \
    AntHalService \
    libantradio \
    antradio_app


#LIBCOPYBIT
LIBCOPYBIT := copybit.msm8660
LIBCOPYBIT += copybit.msm8960
LIBCOPYBIT += copybit.msm8974
LIBCOPYBIT += copybit.msm8226
LIBCOPYBIT += copybit.msm8610
LIBCOPYBIT += copybit.apq8084
44

Was there any effort since February?

1 Like
45

I tried to compile without blobs and got this:

$ make -j4
clang: error: linker command failed with exit code 1 (use -v to see invocation)
build/core/host_shared_library_internal.mk:44: recipe for target 'out/host/linux-x86/obj32/lib/libnativehelper.so' failed
make: *** [out/host/linux-x86/obj32/lib/libnativehelper.so] Error 1
make: *** Waiting for unfinished jobs....

#### make failed to build some targets (03:42 (mm:ss)) ####

Could you build it correctly on a clean repo sync without blobs infection, @fp1_wo_sw_updates?

The desirable would be a blob-free flashable ZIP, to which add blobs from a running device with a neat script. B2G does exactly this (through a web interface that I think wouldn’t be hard to adapt), so it should be possible!

1 Like
46

Hey, I haven’t tried yet ;(

If I remember correctly the blob libs are linked during the build. But I guess is possible to use the standard much slower Android based ones. It would be interesting if one could diff the Mozilla build script with the one FP provides. Or just ask the Mozilla coders :wink:

Reading about clang makes me a bit suspicious. I’m not sure if Android 5 already uses clang. Maybe something is using your systems linker and not the one provided with the code?

47

Yup, Lollipop+ use it, and the issue seems to be something about host clang, acording to CM’s gerrit. Trying to workaround this by WITHOUT_HOST_CLANG=true make systemimage -j4 doesn’t work for FPoOS. I’ll have to learn how to setup well my host clang then, :confounded:

48

Okay, was just a guess! I don’t remember anymore, but accidentally mixing provided tools and compiler stuff from the host could look like this. Maybe you can use a verbose make to see all the tool paths during the build? I would try to avoid using tools from the host as much as possible.

1 Like