Kernel hackers: upstream kernel for FP2 OpenOS?

This is a #wiki post, feel free to edit your findings and information in!

Introduction

Following up on an earlier thread for FP3 brainstorming, I am curious whether people are up for the challenge of building an FP Open OS build around the upstream kernel. Why?

  • Upstream has security fixes first.
  • Upstream drivers have a bigger chance of being audited: you know what you run!
  • Upstream gets most attention from ARM (through their Linaro people) in terms of code optimisation and arch-wide bugfixes.
  • Upstream has more open source drivers thanks to Qualcomm and Red Hat. You’ll have a more open experience and make FP a showcase for the success of SoC vendor upstream efforts, hopefully incentivising other SoC vendors to push through with this too, contributing to the long-term security of the Android platform…

Qualcomm and Red Hat have made significant progress for upstream Linux support of the Qualcomm Snapdragon 801 and most of its devices (including the GPU). They’re quite helpful too (say “hi!” in the Freenode ##linux-msm IRC channel ;-))!

Sources

@z3ntu has provided a fork of the 4.12-rc5 kernel on GitHub, which boots and contains the following additions:

Challenges left include:

  • Adding device-specific device tree bindings (anything you can extract from Fairphone’s kernel tree) to the kernel. For display, see this page, panel information here.
  • Configuring the kernel for an Android environment rather than desktop Linux.
  • Replacing the closed source user-space OpenGL (ES) driver with Mesa freedreno.
  • Making the adsp (Hexagon DSP) do useful stuff with upstream gstreamer, perhaps by interacting with the (closed) firmware files.
  • Configuring the bootloader to load using device tree.
  • Get Qualcomm to release an (open source or otherwise closed source) baseband driver that at least works.

Build instructions

Please visit https://github.com/z3ntu/mainline_fairphone_2

16 Likes

Quite big challenges - and I am afraid that there are way too few people with the skill and the time to dive into this.

1 Like

By “upstream kernel”, do you mean CodeAurora’s one or mainline Linux kernel? I’m not aware of the current state, but mainline Linux was historicaly very reluctant to Android’s customizations for embedded devices (wakelocks and stuff).

Anyway, I personally lack the skills. Kernel hacking (done well) is really hard.

1 Like

Upstream would be mainline Linux. :slight_smile:

They still are, because code in Android kernels is perceived a bit “hacky”. Looking at the Android drivers from some vendors, I can understand why… The problem with codeaurora-esque Android kernels is that it allows SoC vendors to circumvent quality checks. Any upstream kernel maintainer would have NACKed code like the linked backdoor instantly, but for Android kernels the responsibility for quality assurance lies entirely with the SoC vendor. Their mileage seems to vary.

Oh I’m not saying it’s easy… but it depends on how you tackle it. :slight_smile: I bet that booting an upstream kernel on the device is not a major challenge, and just takes a few days of messing about. I have already booted a regular Linux distro with an upstream kernel on a Qualcomm 600 device in the past. Graphics worked using Freedreno almost out-of-the-box back then, and many improvements have been made since. Collabora seems to have done a lot of heavy lifting on bridging the gap between Android and open source graphics stack. Considering most bits of code are already in place I expect that the first step of booting a working kernel takes:

  • Altering a few devicetree definitions (starting for example from the existing devicetree file for Sony XPeria). Hopefully if we ask Fairphone nicely they’d be able to provide board documentation for their GPIOs, regulators, touchscreen and SDHCI configuration.
  • A bit of fiddling with cross-compiling a kernel configured from the qcom default config.
  • Figuring out how to compile mesa (open source 3D acceleration) on Android. Their build system supports it, but you’ll need a compiler that can link against the Fairphone Open OS libraries.
  • Enabling features one device at a time. For wifi for instance, probably you need to add semi-public firmware files to an initrd.

None of this would require actual development, unless you count devicetree definition!

The challenging part would be the features that haven’t been developed upstream. Eg. the GSM modem and hardware audio decoding using the DSP. Arguably the first one is quite an important one and I think this is where Qualcomm might need to be nudged. But then, I wouldn’t try to solve all problems in one go!

3 Likes

I wonder: is there any smartphone atm using an upstream kernel?

Used by the vendor no, but some people managed to get a mainline kernel running on devices like (limited hardware support):

I also played with the mainline kernel on the Galaxy S3 and I was able to boot Replicant 6.0 on top of the Linux 4.8 kernel with a few patches so that it was usable with very limited functionality.

Source: Replicant 6.0 development updates | Replicant

Something I was particularly proud of being able to demo last week at Linaro Connect, was the Nexus7 running a mainline kernel, which has been part of our form-factor enablement effort.

Source: Google Workspace Updates: New community features for Google Chat and an update on Currents

Arch Linux running on a Google Pixel C (kernel v4.11) [in collaboration with /u/samt434]

Source: https://www.reddit.com/r/archlinux/comments/6bckvm/arch_linux_running_on_a_google_pixel_c_kernel/

5 Likes

2nd link: Still ~50 patches… John Stultz says though, this should work with all Nexus devices… :slight_smile:
Replicant is still my favourit, but they are having a hard time too to get newer devices working.

Not to mention Sony is doing cool stuff. From 2016:

Mind you, the MSM8974 is the same SoC as you find in the Fairphone (hence I proposed taking their device tree and modifying it for the Fairphone initially).

4 Likes

Woa, that’s the same SoC, yeah! I was aware of the Sony Open Devices initiative (and useful docs), in fact. One of the most interesting thing for vendors to achieve kernels closest to mainline is the following:

Every new Android release comes with added requirements and continued development that essentially restarts the entire process. This process of re-implementing features and bug fixes to be able to upgrade devices is costly and does not scale, which makes it difficult to provide upgrades to older devices.

(Emphasis by me)

What’s even more, this upstream pushing clearly reflects the objectives of the Open Source movement started with Linux (among others) of universal collaboration that has being obscured by the vast amount of uncommon Linux devices (Android) launched in recent years (and which has generated a lot of dangerous non up-to-date devices with big security flaws).

6 Likes

FYI Snapdragon 835 devices are using kernel version 4.4.
Screenshots: https://www.reddit.com/r/Android/comments/6gabrz/comment/diq4y3j

2 Likes

Yep! There’s a tendency of forking off a longterm kernel release because they get the most important upstream fixes backported without meddling with internal APIs. Makes it easier to copy-paste non-upstream drivers in. I presume at the time of developing the Snapdragon 835, the latest longterm kernel was 4.4 first released January 2016. In December 2016, the 4.9 kernel was the next one to be labelled longterm. Both have been updated just last week with various fixes (whereas the 3.4 kernel shipped for Fairphone OS first released 20th of May 2012 hasn’t seen a new official release in 9 months now - just non-upstream backports that Google/Qualcomm cluncked on top with their monthly security updates).

Got something for you :wink:

https://pastebin.com/BjbcPZUB

4 Likes

You legend! That’s an excellent starting point there. When I get back I’ll turn the opening post into a wiki post (if I can). Can I ask you to share what you have (a github repo with fleshed out DTB, links to how you got the UART working, that sort of stuff)? :smiley:

UART stuff is in this thread: Information about the "debug connector" on the FP2
Source: https://github.com/z3ntu/linux

5 Likes

Updated OP with build instructions and more. Have fun :slight_smile:

Is there any progress with this project? As Linux 3.4 is EOL since April, a more recent Kernel with LTS would be interesting.

I don’t think we would ever use a mainline kernel for running Android just because of the incompatibilities and performance improvements made by Qualcomm.
But for other projects such as postmarketOS a mainline kernel would be great.

And no, I’m not working on it currently because apparently there will be some changes to how eg USB and display gets handled on mainline and without these changes, stuff won’t really work great.

1 Like

Doesn’t Google require a way more up to date kennel for oreo? Will this open up the possibility of a more modern kernel?

No, it isn’t a technical requirement but one set by Google and is probably just enforced by the Google CTS. (but yes they “require” a new kernel version for Oreo)

1 Like

Let me correct, 3.4.0 alone does not work with Oreo but 3.4 with backported patches works fine on several devices already. (eg “ambient capabilities” is needed with Oreo and other patches probably too)