[Toybox] [landley/toybox] Help building toybox with the NDK/bionic (#43)

[Rob Landley]

On 12/26/2016 03:48 PM, enh wrote:
> On Mon, Dec 26, 2016 at 10:31 AM, Rob Landley <rob at landley.net> wrote:
>> Limiting what I can test by building against the NDK. :(
> 
> well, toybox is in this weird position of being both a project in its
> own right but also part of the system. so it gets its own selinux
> label that lets it do things a regular app couldn't (like read other
> process' info in /proc, say, for ps(1) and friends), and it also gets
> a dependency on libcutils for decoding the Android-specific extra
> scheduler info that toybox ps can show on Android.

Indeed. But that also puts toybox in a weird position with regard to
regression testing, and AOSP hasn't got a "build a minimal root
filesystem that boots to a command line under stock qemu" option. (That
I've found yet, anyway.)

I'm trying to improve upon "Elliott will tell me if I screwed up the
android build", which seems impolite somehow...

>>> i think the right fix here is to have a probe for libcutils
>>> (like there already is for libselinux)?
>>
>> I can do that, but this means I still don't have a "success" case to
>> test against. (I can't even test that the probe works.)
> 
> i don't think the existing probes work in AOSP either; Android's build
> system doesn't believe in a separate "configure" step.

Yes, but I'm not trying to fix that one this week. :)

>> Sigh. I'm finally back in Austin and my big box has a terabyte disk in
>> it, and cable modem instead of phone tethering, so I can presumably
>> clean off 200 gigs from that and try to install AOSP again...
>>
>> (That said asking most people to do that remains a big ask.)
> 
> if you're not building your own system image, the ps you can build
> would be pretty useless to you anyway, because it won't have the right
> selinux label. not having the Android scheduling priority field is the
> least of their worries.

Long-term I'm hoping to make it easier to build system images. (I'm
aware installing them on actual hardware with "verified boot" is its own
can of worms, ala http://www.youtube.com/watch?v=OfWFmhucWlg but
building and running under qemu should be easier than I currently find it.)

But that's somewhere on my todo list _after_ implementing awk, vi, and
the rest of the shell.

>>> or -- since the platform build
>>> doesn't use your build system at all -- you can just change
>>> __ANDROID__ here to anything you like, and i'll set it in the Android
>>> build system's build system for toybox.
>>
>> The logical thing to do would be to change it in the _new_ thing, ala
>> defining an __ANDROID_NDK__ or similar in the NDK, if that has a
>> significantly different API than the existing AOSP build.
> 
> it's more the other way round though: the public API exposed in the
> NDK is a subset of the full platform API.

I was thinking that the NDK is the thing changing right now, so it would
get a new symbol to indicate the build environment it provides differs
from the existing AOSP. But you know your build system design better
than I do.

>> But a compile time probe works too, and I can just do that for now.
> 
> as long as there's something i can -D in the Android build system for
> toybox to say "we can link against libcutils"...

Sure.

>>>> toys/android/getprop.c:20:31: fatal error: cutils/properties.h: No such
>>>> file or directory
>>>>  #include <cutils/properties.h>
>>>>
>>>> I'm guessing that was the libselinux thing you were talking about,
>>>> maybe? Anyway, I can switch that app off.
>>>
>>> yeah, i think for now just turning off the android apps makes the most
>>> sense.
>>
>> You are aware of the irony of the android apps being the main thing I
>> _can't_ test under the android NDK, right?
> 
> yes and no. see above.

I didn't say there weren't reasons for it. Just... ironic.

>>> (these ones are slightly different to the one above in that we
>>> can rewrite these to just use bionic's lower-level primitives. i'll
>>> send a patch for that when we get closer to being able to build with
>>> the NDK.)
>>
>> Yay patch. I look forward to it.
> 
> sent.

$ git am 0001-Switch-to-bionic-sys-system_properties.h.patch
Applying: Switch to bionic <sys/system_properties.h>.
error: patch failed: toys/android/getprop.c:17
error: toys/android/getprop.c: patch does not apply
error: patch failed: toys/android/setprop.c:17
error: toys/android/setprop.c: patch does not apply
Patch failed at 0001 Switch to bionic <sys/system_properties.h>.

The last commit toy toys/android that wasn't from you was on August 4th?
Hmmm... I removed the #ifdef __ANDROID__ back in commit 5b493dc48db0 in
2015. (And then fixed a screwup in in e5fb6a28ffb0 the next day.)

We appear to be out of sync? (Speaking of "Elliott will tell me if I
screwed up the android build..." :)

>> How do I get new versions to test?
> 
> we do have nightly builds:
> 
> https://plus.google.com/+ElliottHughes/posts/ixutWK8A5nz?sfc=false
> 
> and betas:
> 
> https://github.com/android-ndk/ndk/wiki
> 
> but you'll need to wait for
> https://github.com/android-ndk/ndk/issues/271 to be fixed first :-)

Ok.

>> Yeah, musl substitutes in its own too:
>>
>> http://git.musl-libc.org/cgit/musl/tree/include/scsi
>>
>> Possibly somebody should poke the kernel guys. :)
> 
> if you know who, please do.

First I'd cc: Rich Felker, since he knows what he did for musl and how
the kernel stuff would differ. (He's the linux arch/sh maintainer too,
so he does kernel stuff too, although this is partially my fault for
dragging him into it.)

The current scripts/headers_install.sh was written by me in 2013 as part
of my 2013 patch series to remove perl as a kernel build dependency. But
I don't have to cc: myself if it's my patch. :)

The "make headers_install" documentation I wrote in 2007
(https://www.kernel.org/doc/Documentation/kbuild/headers_install.txt)
still says David Woodhouse <dwmw2 at infradead.org> maintains it at the
end; nobody's changed that and he posted to linux-kernel from that email
earlier this month, so that's somebody else to cc.

David Howells <dhowells at redhat.com> introduced the uapi directory in the
first place. ("git log --no-merges include/uapi | cat"  and look at the
last commit.)

James Bottomley <James.Bottomley at HansenPartnership.com> is the SCSI
SUBSYSTEM maintainer in MAINTAINERS. (There's a co-maintainer but James
has done it forever and the other guy's at Oratroll, still, voluntarily,
in 2016.)

The most recent patch to include/uapi/scsi was signed-off-by Christoph
Hellwig <hch at lst.de> at the bottom, so he's presumably the relevant
lieutenant. (Although James probably counts on that score too.)

Next we look at the three actual files: sg.h says the SCSI SG DRIVER
maintainer (Doug Gilbert <dgilbert at interlog.com>) should get a cc.

All the changes to scsi_ioctl.h since 1998 are from Christoph Hellwig,
James Bottomley, and Al Viro. (everything before that in my "unified
linux git repo going back to 0.0.1" at
http://landley.net/kdocs/local/linux-fullhist.tar.bz2 are attributed to
Linus Torvalds. and that far back it's just the various dot-releases
with the commit messsages being the release notes, but it still lets you
annotate and go "this was introduced in 1.2.7 around this date" and then
check the mail archives for a possible reason.) Technically Al maintains
"filesystems" but he's also "The Locking Guy" and like 12 oer things.
He's the guy _Linus_ bounces stuff off of to see if he got it right. Not
sure whether to add him to the cc: pile or not.

And when nobody explicitly maintains something, I cc: Andrew Morton on
it. He takes the patches that don't have another tree to go through.

So a patch on this would cc: Rich Felker, David Woodhouse, David
Howells, James Bottomley, Doug Gilbert, and Christoph Hellwig. Plus
maybe Al Viro and Andrew Morton.

But the first part would be coming up with the suggested patch, which
involves comparing musl's and bionic's version of these files (bonic's
at
https://github.com/android/platform_bionic/tree/master/libc/kernel/android/scsi
which has a fourth (scsi_proto.h) which git log in the kernel sources
says was introduced in 2015 by Bart an Assche and Hannes Reinecke (never
heard of either of them)...) And then compare them with the kernel
version, try to come up with a set of changes, and cc you and Rich on
the patch before sending it to the list...

Why does spending time working through my todo list never result in a
shorter todo list?

> the uapi headers aren't great for what i
> assume is their intended purpose. they're missing stuff that should be
> exposed to userspace, still include stuff that shouldn't, don't map
> well to the should-be corresponding POSIX headers, et cetera.

There are corresponding posix headers?

> still,
> better than having to maintain all that stuff ourselves, for the most
> part.
> 
> (you'd probably be horrified at how much glibc-specific crap is in
> them too, which i've never really understood because glibc doesn't use
> them.)

Sigh. Horrified, but not surprised.

>>>> And it's doing pretty well through the rest of the commands. Lots of
>>>> warnings about implicit declarations (gethostid, crypt) wandering by. I
>>>> wonder if I can -Werror just _that_ error? (That's going to come back
>>>> and bite me at link time, I just know it...)
>>>
>>> -Werror=implicit-function-declaration
>>
>> Ah, very nice. I wonder if llvm supports that? Hmmm... seems to.
>>
>> This brings up another point: llvm is _not_ prefixed,  instead it uses
>> --target=blah runtime output flag selector thingies. I've only ever
>> really tested clang on x86 native, but I'd like to use the ndk to add
>> that to my standard regression tests (at least before each release).
>>
>> But I don't quite understand how your standalone toolchain thing's
>> automation is supposed to work:
>>
>>   This operation also installs two wrapper scripts, named clang and
>>   clang++, under <install-dir>/bin. These scripts invoke the clang
>>   binary with the correct target architecture flags. In other words,
>>   they should work without any modification, and you should be able to
>>   use them in your own builds by just setting the CC and CXX
>>   environment variables to point to them.
>>
>> The problem I have cross compiling is that I need a native compiler to
>> build kconfig and instlist and such with. Traditionally, CROSS_COMPILE
>> is a prefix, and then the "cc" binary (which was the standard name in
>> posix in the SUSv2 days, and then c99 came out and they went "clearly
>> you need to switch the binary name to c99 the same way you rename the ls
>> binary to show it's posix-2008 instead of posix-2001!" and nobody did
>> that, and we all stuck with "cc" and waited for posix to admit it made a
>> mistake. Given that the loudest member of that committee is still
>> proclaiming Solaris the One True Unix, my personal strategy has been to
>> wait for somebody to start a better standards body. LSB was making a
>> stab at it but the Linux Foundation put a stop to that.)
>>
>> Anyway, the $CC variable lets you say gcc instead of cc (the FSF can't
>> hear anything outside of the confines of its own ass either, but then it
>> never could), but there isn't a standard/portable way I'm aware of to
>> say "the cross compiler has a different $CC name than the host
>> compiler". I can stick a prefix on it, but not independently rename just
>> the compiler.
> 
> use clang on the host too? we don't use gcc for anything for current
> devices, and moved off gcc even earlier for the host.

Yes, but the clang wrapper isn't prefixed. If I'm building for mips it
distinguishes host toolchain from target toolchain with a prefix, not
with compiler flags. (Because that's the way other packages that cross
compile did it.)

How do you cross compile your kernels with clang, for example? I'm happy
to hit my build with a rock until it's in a new shape, just dunno what
that new shape should look like. Don't want to invent something unique
if I can help it.

>> I'll wait for a fix. Lemme know when there's a new version to try. :)
> 
> will do.
> 
> hopefully soon we'll be at a point where we can start adding building
> various projects out of the box to our testing. strace and tcpdump are
> more obvious candidates, but a toybox that we can take back in time
> would help with NDK testing on old releases too.

It works as a single static binary for a reason. :)

> and i'd still like a
> hermetic build by having a toybox-linked-with-host-bionic for use in
> the build itself, replacing all the stuff that usually comes from the
> host /bin.

See "make install_airlock".

I'm also redoing my http://landley.net/aboriginal/about.html project as
a much simpler http://github.com/landley/mkroot which I might eventually
merge into toybox (under scripts/root implementing "make install_root"
target) once I've implemented the remaining commands the script
currently builds busybox for. In the meantime, I can use this to put
together qemu images and test LFS builds under toybox again (which got a
bit stalled by
http://lists.landley.net/pipermail/aboriginal-landley.net/2016-May/002567.html
leading to
http://lists.landley.net/pipermail/aboriginal-landley.net/2016-November/002591.html).

It doesn't support weight yet, but I'm working on it. Still a week to
New Year's...

>>> crypt(3) is another deliberate "please stop and thing about what
>>> you're doing" omission.
>>
>> Indeed. I was trying for legacy compatibility with existing Linux
>> systems. (And the $1$ and $5$ stuff isn't as bad; you can stick in an
>> arbitrary algorithm there. And _no_ hash is going to survive having
>> /etc/shadow leaked; the attack brute forces the password space there.)
>>
>> But it falls under the same "android treats users differently than the
>> method Linux inherited from Bell Labs and only lightly modified".
>> There's design work pending there.
>>
>>> Android code that wants this kind of
>>> functionality should probably be using BoringSSL.
>>
>> Does it provide a crypt()? It can add $8$ and so on.
> 
> no.
> 
> weird that they chose arbitrary integers rather than just using the
> name of the algorithm.

It was glibc in the Ulrich Drepper era.

>> I am often torn between "I haven't implemented this yet because I'm not
>> sure what it should look like" and "users are submitting code to me that
>> I haven't merged yet and I'm being a bottleneck, lemme put it in
>> pending, oh people are using stuff out of pending when I dunno if what
>> it implements is the best approach to take"...
> 
> i'm not saying mkpasswd and friends don't make sense for traditional
> Unix systems; just that they don't make sense for Android.

Indeed. And I don't know what a replacement should look like in the
context of "I have a posix chroot/container within which to run builds".

>> (I carve out as much time as I can, but it's never enough to keep up.
>> And I've never managed to do much design work in 15 minute increments
>> between higher priority interrupts. Oh well. Christmas break, time to
>> shovel out the code backlog a bit...)
> 
> yeah, i haven't been able to get through as much as i'd hoped this
> year either. i had hoped to switch over dd, getevent, and grep this
> year (basically everything except newfs_msdos), but out of those only
> had time for a few small changes to dd.

I have large pending changes to dd and grep that I got 80% done and then
got pulled away before I finished them to the point where it's easier to
redo them from scratch because thinking carefully through the design
changes and verifying that everything works is half the point of the
exercise and reverse engineering my own code after a month away does not
provide the same guarantees. Grrr. (In the case of dd this is at least
the 3rd time.

(The grep changes winding up extensive are because --color is basically
the -o logic with extra bits, and genericizing that turns out to be
intrusive. The result's better code but I have to re-verify everything
to get there.)

> on the bright side, at least
> i'm ending 2016 without any open toybox bugs (counting "switch to..."
> as feature requests rather than bugs)!
> 
> thanks for all your work in 2016!

Thanks for yours. I'm trying to get my testing back up to a level I'm
not actively ashamed of, which means using it to build dozens of
packages under current kernels (like aboriginal linux used to) _and_
testing on bionic as well as musl and glibc so I break you guys less.

Long-term I want to make hobbyist android development a lot easier. Back
in the Linux days there was "build a base system that boots to a shell
prompt", then "add development tools until it's a usable server and
build machine", then "add x11 and boot an xterm", then "install a
desktop/browser/word processor and so on".

Under AOSP it's "select your hardware platform" and then it builds a
thousand packages for it and I haven't figured out how to tell it NOT
to. I'd like to break that down into chunks a bit, starting with "build
a base android system that boots to a shell prompt under stock QEMU"
(like the embedded world's doing anyway), then "add android's init, the
system manager, and all that SELINUX stuff", then maybe enough to run a
single fullscreen Java app (which I think ChromeOS factored out? There's
a natural "everything spawned from zygote" layer but I'm not sure where
that starts) and then the full desktop stuff. (And then the Play Store
can of worms, which the rest of the world has zero visibility into China
and India are ignoring anyway according to Clay Shirky's interview about
his year teaching at NYU's Shanghai campus at
http://supchina.com/sinica/hour-clay-shirky/)

I.E. Orthogonal Layers would be nice. But last I checked, tackling that
is a huge undertaking. And the "make a web server or router or build
machine or drone flight system or 3d printer using android plumbing"
layer isn't there at all that I can find. (Well, not there in AOSP. I've
worked at more than one company that _did_ that, but they used an
extensively locally hacked version of a vendor fork of a stale version
of android to do it. For example Polycom made a videoconferencing system
that ran half of Ice Cream Sandwich using a TI fork of Android for the
Netra SOC, which the locally modified to do something between NUMA and
clustering via PCIe. There ware DSPs involved. I doubt any of that code
ever made it back to you. I'm sure a historyless tarball of
undifferentiated source was posted on some obscure website for legal
reasons back around 2012, but nobody cared at the time, let alone now.)

Heck, ELC has a workshop on doing that sort of thing every year or two:

https://www.youtube.com/watch?v=dEKYZUgorWQ

(Watching that version is on my todo list. I caught about half of 2010's
version in person but had to leave for another panel...)

Rob