[Toybox] [landley/toybox] Help building toybox with the NDK/bionic (#43)

[enh]

On Tue, Dec 27, 2016 at 1:21 AM, Rob Landley <rob at landley.net> wrote:
> On 12/26/2016 03:48 PM, enh wrote:
>> On Mon, Dec 26, 2016 at 10:31 AM, Rob Landley <rob at landley.net> wrote:
>>> Limiting what I can test by building against the NDK. :(
>>
>> well, toybox is in this weird position of being both a project in its
>> own right but also part of the system. so it gets its own selinux
>> label that lets it do things a regular app couldn't (like read other
>> process' info in /proc, say, for ps(1) and friends), and it also gets
>> a dependency on libcutils for decoding the Android-specific extra
>> scheduler info that toybox ps can show on Android.
>
> Indeed. But that also puts toybox in a weird position with regard to
> regression testing, and AOSP hasn't got a "build a minimal root
> filesystem that boots to a command line under stock qemu" option. (That
> I've found yet, anyway.)
>
> I'm trying to improve upon "Elliott will tell me if I screwed up the
> android build", which seems impolite somehow...

although i'd rather triage Android build issues myself rather than
never hear of them because of dodgy workarounds committed upstream,
which is what happens in some other projects!

>>>> i think the right fix here is to have a probe for libcutils
>>>> (like there already is for libselinux)?
>>>
>>> I can do that, but this means I still don't have a "success" case to
>>> test against. (I can't even test that the probe works.)
>>
>> i don't think the existing probes work in AOSP either; Android's build
>> system doesn't believe in a separate "configure" step.
>
> Yes, but I'm not trying to fix that one this week. :)
>
>>> Sigh. I'm finally back in Austin and my big box has a terabyte disk in
>>> it, and cable modem instead of phone tethering, so I can presumably
>>> clean off 200 gigs from that and try to install AOSP again...
>>>
>>> (That said asking most people to do that remains a big ask.)
>>
>> if you're not building your own system image, the ps you can build
>> would be pretty useless to you anyway, because it won't have the right
>> selinux label. not having the Android scheduling priority field is the
>> least of their worries.
>
> Long-term I'm hoping to make it easier to build system images. (I'm
> aware installing them on actual hardware with "verified boot" is its own
> can of worms, ala http://www.youtube.com/watch?v=OfWFmhucWlg but
> building and running under qemu should be easier than I currently find it.)

or you can just flash your own AOSP -eng build to a Nexus device, "adb
root" and do what you like.

qemu is... pretty awful.

you _can_ use host bionic to run stuff on a regular Linux box, but
none of the "interesting" stuff (the stuff in toys/android/) will work
there anyway.

> But that's somewhere on my todo list _after_ implementing awk, vi, and
> the rest of the shell.
>
>>>> or -- since the platform build
>>>> doesn't use your build system at all -- you can just change
>>>> __ANDROID__ here to anything you like, and i'll set it in the Android
>>>> build system's build system for toybox.
>>>
>>> The logical thing to do would be to change it in the _new_ thing, ala
>>> defining an __ANDROID_NDK__ or similar in the NDK, if that has a
>>> significantly different API than the existing AOSP build.
>>
>> it's more the other way round though: the public API exposed in the
>> NDK is a subset of the full platform API.
>
> I was thinking that the NDK is the thing changing right now, so it would
> get a new symbol to indicate the build environment it provides differs
> from the existing AOSP. But you know your build system design better
> than I do.

it's the other way round: you tell the NDK what API level you're
targeting. (because you might legitimately want to say "don't let me
use anything that wasn't available before ICS, because i want to run
on everything >= ICS".)

>>> But a compile time probe works too, and I can just do that for now.
>>
>> as long as there's something i can -D in the Android build system for
>> toybox to say "we can link against libcutils"...
>
> Sure.
>
>>>>> toys/android/getprop.c:20:31: fatal error: cutils/properties.h: No such
>>>>> file or directory
>>>>>  #include <cutils/properties.h>
>>>>>
>>>>> I'm guessing that was the libselinux thing you were talking about,
>>>>> maybe? Anyway, I can switch that app off.
>>>>
>>>> yeah, i think for now just turning off the android apps makes the most
>>>> sense.
>>>
>>> You are aware of the irony of the android apps being the main thing I
>>> _can't_ test under the android NDK, right?
>>
>> yes and no. see above.
>
> I didn't say there weren't reasons for it. Just... ironic.
>
>>>> (these ones are slightly different to the one above in that we
>>>> can rewrite these to just use bionic's lower-level primitives. i'll
>>>> send a patch for that when we get closer to being able to build with
>>>> the NDK.)
>>>
>>> Yay patch. I look forward to it.
>>
>> sent.
>
> $ git am 0001-Switch-to-bionic-sys-system_properties.h.patch
> Applying: Switch to bionic <sys/system_properties.h>.
> error: patch failed: toys/android/getprop.c:17
> error: toys/android/getprop.c: patch does not apply
> error: patch failed: toys/android/setprop.c:17
> error: toys/android/setprop.c: patch does not apply
> Patch failed at 0001 Switch to bionic <sys/system_properties.h>.
>
> The last commit toy toys/android that wasn't from you was on August 4th?
> Hmmm... I removed the #ifdef __ANDROID__ back in commit 5b493dc48db0 in
> 2015. (And then fixed a screwup in in e5fb6a28ffb0 the next day.)
>
> We appear to be out of sync? (Speaking of "Elliott will tell me if I
> screwed up the android build..." :)

that kind of rings a bell, that you did something years ago that
caused a merge conflict in toys/android/ and i just skipped it. it
used to be useful to me that i could build the Android configuration
on non-Android, but i gave up fighting you and now i have the opposite
problem to you: i don't/can't test the Android toybox build against
glibc.

new patch attached.

you still won't be able to build getprop.c with the NDK, though,
because its use of libselinux is quite ingrained. i've also attached a
patch to just make getprop depend on selinux so an NDK "defconfig"
won't include it.

>>> How do I get new versions to test?
>>
>> we do have nightly builds:
>>
>> https://plus.google.com/+ElliottHughes/posts/ixutWK8A5nz?sfc=false
>>
>> and betas:
>>
>> https://github.com/android-ndk/ndk/wiki
>>
>> but you'll need to wait for
>> https://github.com/android-ndk/ndk/issues/271 to be fixed first :-)
>
> Ok.
>
>>> Yeah, musl substitutes in its own too:
>>>
>>> http://git.musl-libc.org/cgit/musl/tree/include/scsi
>>>
>>> Possibly somebody should poke the kernel guys. :)
>>
>> if you know who, please do.
>
> First I'd cc: Rich Felker, since he knows what he did for musl and how
> the kernel stuff would differ. (He's the linux arch/sh maintainer too,
> so he does kernel stuff too, although this is partially my fault for
> dragging him into it.)
>
> The current scripts/headers_install.sh was written by me in 2013 as part
> of my 2013 patch series to remove perl as a kernel build dependency. But
> I don't have to cc: myself if it's my patch. :)
>
> The "make headers_install" documentation I wrote in 2007
> (https://www.kernel.org/doc/Documentation/kbuild/headers_install.txt)
> still says David Woodhouse <dwmw2 at infradead.org> maintains it at the
> end; nobody's changed that and he posted to linux-kernel from that email
> earlier this month, so that's somebody else to cc.
>
> David Howells <dhowells at redhat.com> introduced the uapi directory in the
> first place. ("git log --no-merges include/uapi | cat"  and look at the
> last commit.)
>
> James Bottomley <James.Bottomley at HansenPartnership.com> is the SCSI
> SUBSYSTEM maintainer in MAINTAINERS. (There's a co-maintainer but James
> has done it forever and the other guy's at Oratroll, still, voluntarily,
> in 2016.)
>
> The most recent patch to include/uapi/scsi was signed-off-by Christoph
> Hellwig <hch at lst.de> at the bottom, so he's presumably the relevant
> lieutenant. (Although James probably counts on that score too.)
>
> Next we look at the three actual files: sg.h says the SCSI SG DRIVER
> maintainer (Doug Gilbert <dgilbert at interlog.com>) should get a cc.
>
> All the changes to scsi_ioctl.h since 1998 are from Christoph Hellwig,
> James Bottomley, and Al Viro. (everything before that in my "unified
> linux git repo going back to 0.0.1" at
> http://landley.net/kdocs/local/linux-fullhist.tar.bz2 are attributed to
> Linus Torvalds. and that far back it's just the various dot-releases
> with the commit messsages being the release notes, but it still lets you
> annotate and go "this was introduced in 1.2.7 around this date" and then
> check the mail archives for a possible reason.) Technically Al maintains
> "filesystems" but he's also "The Locking Guy" and like 12 oer things.
> He's the guy _Linus_ bounces stuff off of to see if he got it right. Not
> sure whether to add him to the cc: pile or not.
>
> And when nobody explicitly maintains something, I cc: Andrew Morton on
> it. He takes the patches that don't have another tree to go through.
>
> So a patch on this would cc: Rich Felker, David Woodhouse, David
> Howells, James Bottomley, Doug Gilbert, and Christoph Hellwig. Plus
> maybe Al Viro and Andrew Morton.
>
> But the first part would be coming up with the suggested patch, which
> involves comparing musl's and bionic's version of these files (bonic's
> at
> https://github.com/android/platform_bionic/tree/master/libc/kernel/android/scsi

(that's not us. https://android.googlesource.com/platform/bionic/ is us.)

> which has a fourth (scsi_proto.h) which git log in the kernel sources
> says was introduced in 2015 by Bart an Assche and Hannes Reinecke (never
> heard of either of them)...) And then compare them with the kernel
> version, try to come up with a set of changes, and cc you and Rich on
> the patch before sending it to the list...

ours just are the kernel files, scrubbed by machine. actual source
(and readme) here:

https://android.googlesource.com/platform/external/kernel-headers/+/master/original/

> Why does spending time working through my todo list never result in a
> shorter todo list?
>
>> the uapi headers aren't great for what i
>> assume is their intended purpose. they're missing stuff that should be
>> exposed to userspace, still include stuff that shouldn't, don't map
>> well to the should-be corresponding POSIX headers, et cetera.
>
> There are corresponding posix headers?

what i mean is cases where you have <sys/whatever.h> #include
<linux/whatever.h>. most of the time you get too much or too little
(or both at the same time).

>> still,
>> better than having to maintain all that stuff ourselves, for the most
>> part.
>>
>> (you'd probably be horrified at how much glibc-specific crap is in
>> them too, which i've never really understood because glibc doesn't use
>> them.)
>
> Sigh. Horrified, but not surprised.
>
>>>>> And it's doing pretty well through the rest of the commands. Lots of
>>>>> warnings about implicit declarations (gethostid, crypt) wandering by. I
>>>>> wonder if I can -Werror just _that_ error? (That's going to come back
>>>>> and bite me at link time, I just know it...)
>>>>
>>>> -Werror=implicit-function-declaration
>>>
>>> Ah, very nice. I wonder if llvm supports that? Hmmm... seems to.
>>>
>>> This brings up another point: llvm is _not_ prefixed,  instead it uses
>>> --target=blah runtime output flag selector thingies. I've only ever
>>> really tested clang on x86 native, but I'd like to use the ndk to add
>>> that to my standard regression tests (at least before each release).
>>>
>>> But I don't quite understand how your standalone toolchain thing's
>>> automation is supposed to work:
>>>
>>>   This operation also installs two wrapper scripts, named clang and
>>>   clang++, under <install-dir>/bin. These scripts invoke the clang
>>>   binary with the correct target architecture flags. In other words,
>>>   they should work without any modification, and you should be able to
>>>   use them in your own builds by just setting the CC and CXX
>>>   environment variables to point to them.
>>>
>>> The problem I have cross compiling is that I need a native compiler to
>>> build kconfig and instlist and such with. Traditionally, CROSS_COMPILE
>>> is a prefix, and then the "cc" binary (which was the standard name in
>>> posix in the SUSv2 days, and then c99 came out and they went "clearly
>>> you need to switch the binary name to c99 the same way you rename the ls
>>> binary to show it's posix-2008 instead of posix-2001!" and nobody did
>>> that, and we all stuck with "cc" and waited for posix to admit it made a
>>> mistake. Given that the loudest member of that committee is still
>>> proclaiming Solaris the One True Unix, my personal strategy has been to
>>> wait for somebody to start a better standards body. LSB was making a
>>> stab at it but the Linux Foundation put a stop to that.)
>>>
>>> Anyway, the $CC variable lets you say gcc instead of cc (the FSF can't
>>> hear anything outside of the confines of its own ass either, but then it
>>> never could), but there isn't a standard/portable way I'm aware of to
>>> say "the cross compiler has a different $CC name than the host
>>> compiler". I can stick a prefix on it, but not independently rename just
>>> the compiler.
>>
>> use clang on the host too? we don't use gcc for anything for current
>> devices, and moved off gcc even earlier for the host.
>
> Yes, but the clang wrapper isn't prefixed. If I'm building for mips it
> distinguishes host toolchain from target toolchain with a prefix, not
> with compiler flags. (Because that's the way other packages that cross
> compile did it.)
>
> How do you cross compile your kernels with clang, for example? I'm happy
> to hit my build with a rock until it's in a new shape, just dunno what
> that new shape should look like. Don't want to invent something unique
> if I can help it.

building kernels with clang is still a work in progress. but i'm not
sure what you're talking about. make-standalone-toolchain generates
little wrapper scripts for exactly this:

/tmp/arm64-unified$ cat ./bin/aarch64-linux-android-clang
#!/bin/bash
if [ "$1" != "-cc1" ]; then
    `dirname $0`/clang38 -target aarch64-none-linux-android --sysroot
`dirname $0`/../sysroot -D__ANDROID_API__=24 "$@"
else
    # target/triple already spelled out.
    `dirname $0`/clang38 "$@"
fi
/tmp/arm64-unified$

>>> I'll wait for a fix. Lemme know when there's a new version to try. :)
>>
>> will do.
>>
>> hopefully soon we'll be at a point where we can start adding building
>> various projects out of the box to our testing. strace and tcpdump are
>> more obvious candidates, but a toybox that we can take back in time
>> would help with NDK testing on old releases too.
>
> It works as a single static binary for a reason. :)
>
>> and i'd still like a
>> hermetic build by having a toybox-linked-with-host-bionic for use in
>> the build itself, replacing all the stuff that usually comes from the
>> host /bin.
>
> See "make install_airlock".
>
> I'm also redoing my http://landley.net/aboriginal/about.html project as
> a much simpler http://github.com/landley/mkroot which I might eventually
> merge into toybox (under scripts/root implementing "make install_root"
> target) once I've implemented the remaining commands the script
> currently builds busybox for. In the meantime, I can use this to put
> together qemu images and test LFS builds under toybox again (which got a
> bit stalled by
> http://lists.landley.net/pipermail/aboriginal-landley.net/2016-May/002567.html
> leading to
> http://lists.landley.net/pipermail/aboriginal-landley.net/2016-November/002591.html).
>
> It doesn't support weight yet, but I'm working on it. Still a week to
> New Year's...
>
>>>> crypt(3) is another deliberate "please stop and thing about what
>>>> you're doing" omission.
>>>
>>> Indeed. I was trying for legacy compatibility with existing Linux
>>> systems. (And the $1$ and $5$ stuff isn't as bad; you can stick in an
>>> arbitrary algorithm there. And _no_ hash is going to survive having
>>> /etc/shadow leaked; the attack brute forces the password space there.)
>>>
>>> But it falls under the same "android treats users differently than the
>>> method Linux inherited from Bell Labs and only lightly modified".
>>> There's design work pending there.
>>>
>>>> Android code that wants this kind of
>>>> functionality should probably be using BoringSSL.
>>>
>>> Does it provide a crypt()? It can add $8$ and so on.
>>
>> no.
>>
>> weird that they chose arbitrary integers rather than just using the
>> name of the algorithm.
>
> It was glibc in the Ulrich Drepper era.
>
>>> I am often torn between "I haven't implemented this yet because I'm not
>>> sure what it should look like" and "users are submitting code to me that
>>> I haven't merged yet and I'm being a bottleneck, lemme put it in
>>> pending, oh people are using stuff out of pending when I dunno if what
>>> it implements is the best approach to take"...
>>
>> i'm not saying mkpasswd and friends don't make sense for traditional
>> Unix systems; just that they don't make sense for Android.
>
> Indeed. And I don't know what a replacement should look like in the
> context of "I have a posix chroot/container within which to run builds".
>
>>> (I carve out as much time as I can, but it's never enough to keep up.
>>> And I've never managed to do much design work in 15 minute increments
>>> between higher priority interrupts. Oh well. Christmas break, time to
>>> shovel out the code backlog a bit...)
>>
>> yeah, i haven't been able to get through as much as i'd hoped this
>> year either. i had hoped to switch over dd, getevent, and grep this
>> year (basically everything except newfs_msdos), but out of those only
>> had time for a few small changes to dd.
>
> I have large pending changes to dd and grep that I got 80% done and then
> got pulled away before I finished them to the point where it's easier to
> redo them from scratch because thinking carefully through the design
> changes and verifying that everything works is half the point of the
> exercise and reverse engineering my own code after a month away does not
> provide the same guarantees. Grrr. (In the case of dd this is at least
> the 3rd time.
>
> (The grep changes winding up extensive are because --color is basically
> the -o logic with extra bits, and genericizing that turns out to be
> intrusive. The result's better code but I have to re-verify everything
> to get there.)
>
>> on the bright side, at least
>> i'm ending 2016 without any open toybox bugs (counting "switch to..."
>> as feature requests rather than bugs)!
>>
>> thanks for all your work in 2016!
>
> Thanks for yours. I'm trying to get my testing back up to a level I'm
> not actively ashamed of, which means using it to build dozens of
> packages under current kernels (like aboriginal linux used to) _and_
> testing on bionic as well as musl and glibc so I break you guys less.
>
> Long-term I want to make hobbyist android development a lot easier. Back
> in the Linux days there was "build a base system that boots to a shell
> prompt", then "add development tools until it's a usable server and
> build machine", then "add x11 and boot an xterm", then "install a
> desktop/browser/word processor and so on".
>
> Under AOSP it's "select your hardware platform" and then it builds a
> thousand packages for it and I haven't figured out how to tell it NOT
> to. I'd like to break that down into chunks a bit, starting with "build
> a base android system that boots to a shell prompt under stock QEMU"
> (like the embedded world's doing anyway), then "add android's init, the
> system manager, and all that SELINUX stuff", then maybe enough to run a
> single fullscreen Java app (which I think ChromeOS factored out?

not afaik. i think they just run lightly-modified "full" android in a container.

> There's
> a natural "everything spawned from zygote" layer but I'm not sure where
> that starts) and then the full desktop stuff. (And then the Play Store
> can of worms, which the rest of the world has zero visibility into China
> and India are ignoring anyway according to Clay Shirky's interview about
> his year teaching at NYU's Shanghai campus at
> http://supchina.com/sinica/hour-clay-shirky/)
>
> I.E. Orthogonal Layers would be nice. But last I checked, tackling that
> is a huge undertaking. And the "make a web server or router or build
> machine or drone flight system or 3d printer using android plumbing"
> layer isn't there at all that I can find.

brillo always includes Java now it's s/brillo/things/:
https://developer.android.com/things/sdk/index.html

one prerequisite to a secure IoT is that everyone stops shooting at
the nuclear missiles^W^W^W^W^Wwriting unnecessary native code.

> (Well, not there in AOSP. I've
> worked at more than one company that _did_ that, but they used an
> extensively locally hacked version of a vendor fork of a stale version
> of android to do it. For example Polycom made a videoconferencing system
> that ran half of Ice Cream Sandwich using a TI fork of Android for the
> Netra SOC, which the locally modified to do something between NUMA and
> clustering via PCIe. There ware DSPs involved. I doubt any of that code
> ever made it back to you. I'm sure a historyless tarball of
> undifferentiated source was posted on some obscure website for legal
> reasons back around 2012, but nobody cared at the time, let alone now.)
>
> Heck, ELC has a workshop on doing that sort of thing every year or two:
>
> https://www.youtube.com/watch?v=dEKYZUgorWQ
>
> (Watching that version is on my todo list. I caught about half of 2010's
> version in person but had to leave for another panel...)
>
> Rob



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Switch-to-bionic-sys-system_properties.h.patch
Type: text/x-patch
Size: 3439 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20161227/5556d725/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Admit-that-getprop-requires-selinux.patch
Type: text/x-patch
Size: 1047 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20161227/5556d725/attachment-0003.bin>