[Toybox] Android sitrep

[enh]

On Sun, Oct 6, 2019 at 11:38 AM Rob Landley <rob at landley.net> wrote:
>
> I have soooo many open but unfinished email reply windows. This one's a month
> old! (You can tell how much time I've had to "clean" by how many half-finished
> open windows my laptop's got...)
>
> On 9/1/19 12:57 PM, enh wrote:
> >> You listed lsof out of the second list but not the others?
> >
> > correct. lsof is (odd thought it may seem) used in the build --- but
> > only if something goes wrong; the resulting bug reporting includes
> > things like lsof and pstree output. so they're third-class citizens as
> > far as i'm concerned:
> >
> > 1. non-pending, used in successful build
> > 2. pending, used in successful build
> > 3. pending, not used normally
>
> Ok.
>
> >>>> The android directory's your baby, but could you explain a bit more what you
> >>>> were using them for before, and where you're using them now?
> >>>
> >>> i used to just have one .config so i'd have to jump through a few
> >>> hoops to be able to use that on the linux host too. now i have
> >>> separate configs, Android stuff only needs to build on Android, so
> >>> portability shmortability.
> >>
> >> If they're not needed during the AOSP build and only have to be present on the
> >> target, then sure.
> >
> > if anyone starts using traceroute as part of the build, we're going to
> > have to have words with them :-)
> >
> > (i assume the nsjail they're run in prevents that kind of silliness anyway.)
> >
> >> The reason toybox can do ps is it has magic selinux attributes only created
> >> during the AOSP build, not available in the NDK build. I take it all the selinux
> >> attributes in the new filesystem are created during the filesystem packaging step?
> >
> > yes.
>
> Where is this packaging done in AOSP? (I can dig my way to it eventually, but
> you'd probably know off the top of your head. :)

system/sepolicy/private/file_contexts

> >> (I tried to dig into AOSP recently. The Makefile sourced build/make/core/main.mk
> >> which started with ANDROID_BUILD_SHELL and I went off to look up what that was
> >> and wound up reading about https://developer.android.com/ndk/guides/wrap-script
> >> instead and never got back... :)
> >>
> >>>> Yay. In theory I'm doing that here, in practice it's sometimes just every couple
> >>>> weeks so I can take a while to spot regressions. And I'm testing vanilla, not
> >>>> android kernel, and not your config...
> >>>
> >>> (i don't actually know _which_ kernel they're building. it could
> >>> actually be upstream, or it could be the "android common kernel", or
> >>> it could be -- and this is actually the least likely -- a device
> >>> kernel.)
> >>
> >> I love reproducible bug reports. Sending real world data through the commands is
> >> how you find real world problems, I'm all for it.
> >
> > i can tell you already that one of the issues is going to be xargs -P
> > --- i almost sent you a patch last week that made -P a no-op (given
> > that the docs say "up to ... processes"); i don't know whether it
> > actually makes a difference to build times.
>
>   While  xargs  is
>   running,  you  can send its process a SIGUSR1 signal to increase
>   the number of commands to run simultaneously, or  a  SIGUSR2  to
>   decrease  the number.
>
> Seriously?
>
> Sigh, I'm ok with managing parallelism, you may notice that I'm doing so from a
> bash script in scripts/make.sh:
>
>   https://github.com/landley/toybox/blob/master/scripts/make.sh#L314
>
> Ooh, when did wait -n show up? I really _wanted_ that when I was writing this,
> and it wasn't available. Hmmm... looks like bash 4.3. Released Feb 2014. Hmmm.
> Not quite 7 years (5.5 at this point). But once I can build toybox under toysh I
> might make an exception. (Assuming it doesn't take that long to complete toysh.
> Grrr.)
>
> Anyway, I might whip up an xargs -P if it's quick and I get a gap, but this
> SIGUSR increment/decrement thing is kind of _conceptually_ silly? (What... why...?)

i don't know of anyone who wants the signal nonsense. actually, if you
think you can get the hard-coded -P 8 removed from the kernel, i think
we could all forget about this. (the scary part is that searching all
the source available to me internally, -P 8 is basically the only use
of -P. there's the odd pointless -P 1, and one -P 100, but presumably
everyone's copy & pasted 8 from somewhere? too high for my MacBook Pro
which starts to melt around 4, and too low for my desktop, which can't
count that low.)

i do wonder how long it would take for anyone to notice if we made -P
a no-op (and am very disappointed with myself that i didn't do that
long ago).

> >> Presumably the response is "oh the markdown command is broken, github does it
> >> magically right so you can only see what your changes look like after you upload
> >> them to an external website"... which has kinda slowed my acceptance of this
> >> patch. :P
> >
> > so much as i think markdown is the least worst option for markup ...
> > the biggest problem is that there is no One True Markdown. aiui, the
> > original one is so anemic that no-one uses it, and there are a bunch
> > of independent reimplementations with no conformance test suite.
> > github and gitiles are different, for example, which are the two i
> > dealt with regularly.
> >
> > _but_ there is a common subset that you can rely on.
>
> How do I test what the output looks like locally?
>
> [I got pointed at a thing since and have an open terminal window for that...]
>
> > i suspect here you want to use ``` to blockquote.
> >
> > (but i can tell you that what i sent you does work with github because
> > i edited it in the github bug tracker. you can just switch between
> > "write" and "preview" there. and github at least uses the same
> > markdown implementation everywhere :-) )
>
> Well that's something. Hmmm...
>
> >>>>> * unrelated, i built a static toybox binary.
> >>>>
> >>>> Yay! (Against bionic?)
> >>>
> >>> correct. the current (R, not Q!) libc.a.
> >>
> >> I'm all for it.
> >
> > the NDK guy is too, so even though i'm a bit unsure about the
> > usefulness without adbd fixes on old devices (which aren't possible)
> > and a mksh static binary (which i could also do), i'll get this done.
> > "something is better than nothing", after all.
>
> Yay!
>
> >>>>>   * the transitive dependencies are pretty gross thanks to the cgroup
> >>>>> stuff. i wonder if anyone uses that enough for it to be really worth
> >>>>> it, or whether we could drop it without anyone noticing?
> >>>>
> >>>> Huh, I did this in nsenter:
> >>>>
> >>>> #define unshare(flags) syscall(SYS_unshare, flags)
> >>>> #define setns(fd, nstype) syscall(SYS_setns, fd, nstype)
> >>>>
> >>>> Precisely to _reduce_ dependencies. Do you mean the CLONE flags being defined,
> >>>> or...?
> >>>
> >>> no, the Android cgroup stuff. grep ps.c for ANDROID. basically so ps
> >>> can say "bg" or "fg" or whatever rather than an int.
> >>
> >> In all caps there's just the one?
> >>
> >>   else if (CFG_TOYBOX_ON_ANDROID)
> >>     sprintf(not_o = toybuf+128,
> >>             "USER,%%sPPID,VSIZE,RSS,WCHAN:10,ADDR:10,S,%s",
> >>             FLAG(T) ? "CMD" : "NAME");
> >>
> >> The other case insensitive hits are about scheduling policy and a todo about
> >> field size?
> >>
> >> bg and fg... the shell commands?
> >
> > no, the background and foreground cgroups used for scheduling policy.
> > (the first of the case-insensitive matches you found.)
>
> I need to get an AOSP qemu instance building from source so I can test this stuff...
>
> >> I've missed a curve here.
> >>
> >>>>>   * ran tests (with some difficulty) on jellybean, and had about 100
> >>>>> failures out of about 1200 tests. (`adb shell` didn't return exit
> >>>>> status that far back, so this is grep for PASS and FAIL rather than
> >>>>> the usual output from my little test runner.)
> >>>>
> >>>> Statically linked against a newer bionic, but running on the older kernel?
> >>>
> >>> correct.
> >>
> >> Seems pretty reasonable then. Do you have a dump of what the failures are so we
> >> can categorize them?
> >
> > not yet, i was doing it in parallel with a bunch of other stuff. i'll
> > have a proper run though now everyone seems in favor of doing this.
>
> Did that ever happen? I've had a crazy month and totally lost context of what I
> was doing before...

it's still lying around in a tree somewhere, once i've popped off all
the current problems.

> >>>> You sent a link to https://developer.android.com/about/dashboards which says
> >>>> Jellybean is 3.2% of machines checking into the play store or some such. Even
> >>>> adding in the 0.6% from the previous 2 releases, that means 96% of all android
> >>>> devices are running kitkat or newer.
> >>>
> >>> aye, but our self-imposed albatross is that we wait until the share is < 1%.
> >>
> >> Works for me. (And technically the 7 year horizon agrees with you. :)
>
> So the jellybean ABI will still be in the NDK until J drops below 1%,

yes.

> and the
> static toybox binary on developer.android.com should be built with the _oldest_ ABI?

no, the *only* libc.a is the latest one, whatever that is at any given
time. you could build with the 16 headers and link with the current
libc.a, but it'd be a bit of a waste of time!

> >>>> While the 7 year horizon is a bit over Kit Kat's introduction 6 years and 1
> >>>> month ago, historical static binary thing is a new feature. I'm ok anchoring it
> >>>> to kitkat if you are. That's still going back 2 releases before toybox was
> >>>> _introduced_.
> >>>
> >>> ...which is exactly why those devices need a static toybox the most :-)
> >>>
> >>> i wish i had a good idea of "oldest release anyone's doing any serious
> >>> testing on", but i don't. (and like i say, the shell and adb are awful
> >>> that far back too. i don't think adb was fixed until N :-( )
> >>
> >> I'm in the process of writing a proper shell you can |& with, which can handle
> >> its own job control and such. I have:
> >>
> >>   $ cat boing |& sed s/boing/walrus/
> >>   cat: walrus: No such file or directory
> >>
> >> In my test pile. Alas, it is by _no_ means there yet, but I am at least working
> >> on it. :)
> >
> > the shell's actually the easy part: it's easier to package that up
> > than a static toybox (because fewer dependencies, and especially not
> > that awkward libcutils one). i pity anyone who has to do any automated
> > testing on old releases. (this is why the world is full of weird stuff
> > like parsing ls(1) output to test whether files exist... <movie
> > trailer voice>i a world where you can't rely on exit status...)
>
> Circa 2006 reiserfs didn't set d_type in getdents() so I had to change something
> in busybox to do an extra stat(). Only filesystem that _ever_ got that wrong, as
> far as I know...
>
> (Before I knew the context I mentally pronounced that re-i-serfs, and after the
> trial I went _back_ to mentally pronouncing it that way. I don't think I've ever
> named a software project after myself, and even Linus tried to call Linux
> "freax", it was sysadmin Ari Lemmke who gave him an ftp directory called "linux"
> to publish it in...)
>
> >>>> I'm happy to do that, and from the website's nav bar. And yes, I need to catch
> >>>> up on patches this weekend, that's one in the stack for this release.
>
> [a month later, still no release out. Yesterday I plugged toybox into the old
> aboriginal linux build again and this time I got most of the way through the gcc
> build, at which point the tar "no ../ files outside this directory" check is
> false positiving on "./" for some reason, gotta track that down. All the corner
> cases, gotta constantly regression test, old builds are a good way to do that.
> They test stuff I'd never think off...]

releases have to be time-based or feature-based, you can't have both :-)

> >>> it's certainly easier than folks trying to build toybox themselves...
> >>
> >> I need to get the FAQ updated with build instructions for various targets. (It's
> >> in the README, but more docs doesn't hurt...)
> >
> > toybox is an example in the NDK docs,
>
> For the old https://developer.android.com/ndk/guides/standalone_toolchain
> toolchain type with the python script. The new one at
> https://developer.android.com/ndk/guides/other_build_systems uses libpng and
> bzip, but the principle's the same. :)
>
> I need to add a FAQ entry on building toybox with the NDK, I have to work it out
> again each time I do it...
>
> > but i'm not sure how many people
> > actually build it themselves rather than just suffer whatever happened
> > to be on whatever device they have.
>
> Given how many people are still sticking busybox on devices, I'd expect there's
> a demand for newer toybox binaries on older systems. (Especially if I get the
> rest of the 1.0 roadmap sorted.)

aren't busybox users folks who've rooted their devices? so they can
just use the latest toybox anyway?

> Rob