[Toybox] [landley/toybox] Help building toybox with the NDK/bionic (#43)

[Rob Landley]

On 12/27/2016 02:49 PM, enh wrote:
> On Tue, Dec 27, 2016 at 1:21 AM, Rob Landley <rob at landley.net> wrote:
>> I'm trying to improve upon "Elliott will tell me if I screwed up the
>> android build", which seems impolite somehow...
> 
> although i'd rather triage Android build issues myself rather than
> never hear of them because of dodgy workarounds committed upstream,
> which is what happens in some other projects!

True, but "works on musl and glibc, doesn't work on bionic" is the kind
of thing I generally just poke you and make puppy eyes about anyway.

I'm thinking more along the lines of "finding the sed problem earlier
would have been nice".

>> Long-term I'm hoping to make it easier to build system images. (I'm
>> aware installing them on actual hardware with "verified boot" is its own
>> can of worms, ala http://www.youtube.com/watch?v=OfWFmhucWlg but
>> building and running under qemu should be easier than I currently find it.)
> 
> or you can just flash your own AOSP -eng build to a Nexus device, "adb
> root" and do what you like.

I thought they'd renamed it Pixel?

I've got a Nexus 5. It maxed out at Android M and you stopped shipping
security updates for it in October. I'm reluctant to buy another device
that may be discontinued before I get it home.

(The whole "two year support horizon" combined with the implosion of
cyanogenmod kinda undermines my whole "if you're selling a billion
phones a year then there are a billion _old_ phones a year available for
the next generation of developers to play with and learn on instead of
just landfilling them" theory. Making it easier to build and install new
system images might work around that, but there's a lot of work to do to
get from here to there. I need to figure out how to carve out more time
to work on this again. Endless travel between time zones got me out of
the "set the alarm at 5am and get a couple hours in before work" habit...)

> qemu is... pretty awful.

It's slow, but using it I can script automated regression tests for a
half-dozen architectures from a coffee shop without having to plug extra
hardware into my netbook.

The best part is since an emulator image starts in a known state, the
mount or ifconfig tests failing to clean up after themselves (or the ps
tests --bind mounting test data over /proc and getting interrupted
halfway through so it didn't unmount it) don't screw up the next round
of tests in a new emulator instance.

> you _can_ use host bionic to run stuff on a regular Linux box, but
> none of the "interesting" stuff (the stuff in toys/android/) will work
> there anyway.

Hence booting an android kernel under qemu. But that's something I need
to work up to...

>> I was thinking that the NDK is the thing changing right now, so it would
>> get a new symbol to indicate the build environment it provides differs
>> from the existing AOSP. But you know your build system design better
>> than I do.
> 
> it's the other way round: you tell the NDK what API level you're
> targeting. (because you might legitimately want to say "don't let me
> use anything that wasn't available before ICS, because i want to run
> on everything >= ICS".)

Ah, selectable API levels. Hmmm.

>>>> Yay patch. I look forward to it.
>>>
>>> sent.
>>
>> $ git am 0001-Switch-to-bionic-sys-system_properties.h.patch
>> Applying: Switch to bionic <sys/system_properties.h>.
>> error: patch failed: toys/android/getprop.c:17
>> error: toys/android/getprop.c: patch does not apply
>> error: patch failed: toys/android/setprop.c:17
>> error: toys/android/setprop.c: patch does not apply
>> Patch failed at 0001 Switch to bionic <sys/system_properties.h>.
>>
>> The last commit toy toys/android that wasn't from you was on August 4th?
>> Hmmm... I removed the #ifdef __ANDROID__ back in commit 5b493dc48db0 in
>> 2015. (And then fixed a screwup in in e5fb6a28ffb0 the next day.)
>>
>> We appear to be out of sync? (Speaking of "Elliott will tell me if I
>> screwed up the android build..." :)
> 
> that kind of rings a bell, that you did something years ago that
> caused a merge conflict in toys/android/ and i just skipped it. it
> used to be useful to me that i could build the Android configuration
> on non-Android, but i gave up fighting you and now i have the opposite
> problem to you: i don't/can't test the Android toybox build against
> glibc.

I'm attempting to fix this, but being able to build an android version
myself is a prerequisite to me being able to make both work. :)

> new patch attached.
> 
> you still won't be able to build getprop.c with the NDK, though,
> because its use of libselinux is quite ingrained.

It's still an improvement. Thanks.

>> So a patch on this would cc: Rich Felker, David Woodhouse, David
>> Howells, James Bottomley, Doug Gilbert, and Christoph Hellwig. Plus
>> maybe Al Viro and Andrew Morton.
>>
>> But the first part would be coming up with the suggested patch, which
>> involves comparing musl's and bionic's version of these files (bonic's
>> at
>> https://github.com/android/platform_bionic/tree/master/libc/kernel/android/scsi
> 
> (that's not us. https://android.googlesource.com/platform/bionic/ is us.)

It claims to be a mirror of that, but
https://android.googlesource.com/platform/bionic/+/master/libc/kernel/android/scsi/
then.

>> which has a fourth (scsi_proto.h) which git log in the kernel sources
>> says was introduced in 2015 by Bart an Assche and Hannes Reinecke (never
>> heard of either of them)...) And then compare them with the kernel
>> version, try to come up with a set of changes, and cc you and Rich on
>> the patch before sending it to the list...
> 
> ours just are the kernel files, scrubbed by machine. actual source
> (and readme) here:
> 
> https://android.googlesource.com/platform/external/kernel-headers/+/master/original/

I wonder what packages actually _use_ these headers? (Real world testing
of the headers proposed for new deployment would be nice...)

>> Why does spending time working through my todo list never result in a
>> shorter todo list?
>>
>>> the uapi headers aren't great for what i
>>> assume is their intended purpose. they're missing stuff that should be
>>> exposed to userspace, still include stuff that shouldn't, don't map
>>> well to the should-be corresponding POSIX headers, et cetera.
>>
>> There are corresponding posix headers?

I checked, susv4 doesn't include the word "scsi" in the tarball I
downloaded.

> what i mean is cases where you have <sys/whatever.h> #include
> <linux/whatever.h>. most of the time you get too much or too little
> (or both at the same time).

The musl headers have a policy of not ever using/including the linux/*
headers at all, and I had to make puppy eyes at Rich to have
musl-cross-make even install them.

(I fought the "cut and paste the linux header bits into your own code"
wars back on busybox, from
http://lists.busybox.net/pipermail/busybox/2004-November/013128.html to
http://lists.busybox.net/pipermail/busybox/2010-March/071874.html and
the answer was "don't go there if you want to support more than one
architecture".)

>> Yes, but the clang wrapper isn't prefixed. If I'm building for mips it
>> distinguishes host toolchain from target toolchain with a prefix, not
>> with compiler flags. (Because that's the way other packages that cross
>> compile did it.)
>>
>> How do you cross compile your kernels with clang, for example? I'm happy
>> to hit my build with a rock until it's in a new shape, just dunno what
>> that new shape should look like. Don't want to invent something unique
>> if I can help it.
> 
> building kernels with clang is still a work in progress.

I've been following that for years:

http://lists.landley.net/pipermail/aboriginal-landley.net/2011-July/001862.html

I suspect the main reason https://www.youtube.com/watch?v=TiLPL-zMndM
hasn't made much progress in the 3 years since that video is they
allowed the Linux Foundation to get involved. When the money ran out,
http://llvm.linuxfoundation.org stopped updating. (Most recent status
I've seen is
https://www.phoronix.com/scan.php?page=news_item&px=Clang-The-Kernel-2016)

But my opinion of the Linux Foundation hasn't changed since
http://landley.net/notes-2010.html#18-07-2010 and was more or less
confirmed by Eben Moglen's explanation at
https://www.softwarefreedom.org/blog/2016/apr/11/lf/ about how the Linux
Foundation is a "trade association" and not a 501c3.

I worked on building the kernel under another compiler myself ten years
ago, when I maintained my own tinycc fork and tried to build the kernel
with it. Didn't get too far (some ELF tool I hadn't implemented yet like
objdump or nm was parsing a .o file to produce ascii output that was run
through sed or awk to produce a header file with constants for structure
offsets). But it wasn't that hard, either.

Fabrice Bellard got tinycc to build a bootable linux kernel in tccboot
back in 2004, so we know it's doable. (Ok
http://blog.smartbear.com/careers/fabrice-bellard-portrait-of-a-super-productive-programmer/
is not a good yardstick for mere mortals but I was doing ok at it, the
_work_ was all on the compiler side with tinycc still missing a bunch of
basic features like constant propagation and dead code elimination,
stuff either required by c99 or present back in Turbo C for DOS. If
you're willing to push patches upstream into the kernel instead of
laboriously improve your tools to handle the existing kernel code, it
gets a lot easier.)

The trick is to start with a minimal config and add features. I'm told
they got a minimal config building ages ago, so defend it with
regression testing each release and work on new features one at a time...

> but i'm not
> sure what you're talking about. make-standalone-toolchain generates
> little wrapper scripts for exactly this:
>
> /tmp/arm64-unified$ cat ./bin/aarch64-linux-android-clang

Ah, I'd missed that. (I just saw the "clang*" in that directory. Hmmm...
CC=clang CROSS_COMPILE=aarch64-linux-android- should work then. Thanks,
I'll give it a try once I carve out some more toybox development time.

(Email can be composed in 2 minute increments. Coding requires a bit
more sustained focus, for me at least...)

>>> and i'd still like a
>>> hermetic build by having a toybox-linked-with-host-bionic for use in
>>> the build itself, replacing all the stuff that usually comes from the
>>> host /bin.
>>
>> See "make install_airlock".
...
>> http://lists.landley.net/pipermail/aboriginal-landley.net/2016-November/002591.html).
>>
>> It doesn't support weight yet, but I'm working on it. Still a week to
>> New Year's...

See $DAYJOB Perpetual Crisis last message. This got hit too. :(

>> Under AOSP it's "select your hardware platform" and then it builds a
>> thousand packages for it and I haven't figured out how to tell it NOT
>> to. I'd like to break that down into chunks a bit, starting with "build
>> a base android system that boots to a shell prompt under stock QEMU"
>> (like the embedded world's doing anyway), then "add android's init, the
>> system manager, and all that SELINUX stuff", then maybe enough to run a
>> single fullscreen Java app (which I think ChromeOS factored out?
> 
> not afaik. i think they just run lightly-modified "full" android in a container.

So a todo item then.

>> There's
>> a natural "everything spawned from zygote" layer but I'm not sure where
>> that starts) and then the full desktop stuff. (And then the Play Store
>> can of worms, which the rest of the world has zero visibility into China
>> and India are ignoring anyway according to Clay Shirky's interview about
>> his year teaching at NYU's Shanghai campus at
>> http://supchina.com/sinica/hour-clay-shirky/)
>>
>> I.E. Orthogonal Layers would be nice. But last I checked, tackling that
>> is a huge undertaking. And the "make a web server or router or build
>> machine or drone flight system or 3d printer using android plumbing"
>> layer isn't there at all that I can find.
> 
> brillo always includes Java now it's s/brillo/things/:
> https://developer.android.com/things/sdk/index.html

I await Oratroll's lawsuits:
http://www.businessinsider.com/oracle-starts-to-audit-java-customers-2016-12

(Does this have an AOSP build target? Ironically, the new name is
_slightly_ less easy to google for.)

> one prerequisite to a secure IoT is

Airtight container infrastructure?

> that everyone stops shooting at
> the nuclear missiles^W^W^W^W^Wwriting unnecessary native code.

You mean like qualcomm's baseband binary blob?

[deletes 100 line rant]

How should I phrase this... "I'm not sure your experience here is
universal."

Rob