[Toybox] Android .config pondering.

[enh]

On Sat, May 7, 2022 at 4:38 AM Rob Landley <rob at landley.net> wrote:
>
> On 5/5/22 12:24, enh wrote:
> > On Wed, May 4, 2022 at 8:15 PM Rob Landley <rob at landley.net> wrote:
> >>
> >> So I thought there should probably be a "make android_defconfig" target even if
> >> android isn't using it, just so I have an easier time tracking stuff. Except
> >> android has three configs: device, linux, and mac.
> >
> > meh, like i always say, i think it's a sensible division of
> > responsibilities for you to worry about "does it work on debian?" and
> > android (and to a lesser, more "good guy greg" extent, mac) is my
> > problem?
>
> Oh sure. I occasionally find myself asking the question "is this command enabled
> on android" and then I have to go fish for it, but my main motivation for
> looking at this is I'm doing documentation videos and trying to explain stuff
> highlights holes. The default defconfig is basically linux_defconfig, and then
> there's make macos_defconfig and make bsd_defconfig... but no make
> android_defconfig. Seemed less awkward to just add one than explain why it's not
> there. My macos_defconfig isn't what AOSP builds for mac either. :)
>
> >> The android .config-mac the output of "make macos_defconfig" differing seems
> >> like a "me" problem (I should update my macos_defconnfig, and ideally I'd get an
> >> ssh into a mac box so I can see what actually builds and passes its tests. I
> >> could kinda hack the github build stuff in a fork to do that maybe? Rather a lot
> >> of iteration required though, unless maybe "make change" works on mac...)
> >
> > iirc the reason they differ is that macos_defconfig is "everything
> > that passes the tests on a mac [the last time enh ran all the tests on
> > a mac]" whereas the android mac config is basically "the intersection
> > of macos_defconfig and android's _linux_ config".
>
> The android one had a couple commands that weren't in macos_defconfig, so I
> added them. :)

yeah, that certainly seems like a mistake :-)

> >> But the other two are... kinda confusing? The device (android) config has a lot
> >> of stuff the linux (build) one does not: acpi, base64, blkdiscard, blkid,
> >> blockdev, cal, chattr, chcon, chgrp, chown, chroot, chrt, cksum, clear, devmem,
> >> df, dmesg, expand, fallocate, false, file, flock, fmt, freeramdisk, free,
> >> fsfreeze, fsync, getenforce, getfattr, groups, gunzip, help, hwclock, i2c*,
> >> iconv, ifconfig, inotifyd, insmod, ionice, iorenice, iotop, kill, killall,
> >> load_policy, logname, log, losetup, lsattr, lsmod, lsof, lspci, lsusb, makedevs,
> >> mkfifo, mknod (twice?), mkswap, modinfo, modprobe, more, mount, mountpoint,
> >> nbd_client, netcat, netstat, nice, nohup, nsenter, partprobe, pidof, ping,
> >> ping6, pivot_root, pmap, printenv, pwdx, readelf, renice, restorecon, rev,
> >> rfkill, rmmod, rtcwake, runcon, sendevent, setenforce, setfattr, sha224sum,
> >> sha384sum, split, strings, stty, swapoff, swapon, sync, sysctl, tac, taskset
> >> (twice), time, top, traceroute, tty, tunctl, uclampset, ulimit, umount, unlink,
> >> unshare, uptime, usleep, uudecode, uuencode, uuidgen, vconfig, vi, vmstat,
> >> watch, and yes.
> >
> > yeah, so for "android" my policy (to the extent there's any kind of
> > policy!) is basically "does it pass the tests on android?". if it
> > does, meh, ship it! someone might find it useful.
>
> Woo! I'm all for it.
>
> > there's a slightly
> > higher bar for whether or not it gets a symlink (which is actually a
> > separate list in the .bp file) --- but that's a lot more vague,
> > something like "does enh think this is either (a) in a good enough
> > state that we can realistically support it effectively or (b)
> > sufficiently useful that 'half an eye is better than no eye' and we
> > should make it easily available anyway, and deal with issues as they
> > arrive?".
>
> If it's not in pending, send me the bug reports.
>
> (I'm simultaneously happy somebody is looking at sh.c and sending me patches,
> and going "it's not ready for company..." The next test_sh that fails is the one
> checking that pipeline segments are properly backgrounded, because right now if
> you pipe a while loop into another while loop the second one won't start running
> until the first one finishes because they're happening int the same shell
> context. One of the largeish reworks I got distracted from many moons ago is
> most of the way to making that work, but it's nontrivial...)

yeah, i did tell him, but he couldn't resist poking it with a stick to
see what kind of condition it was in.

> >> In fact only two symbols are enabled in the linux host but NOT the device:
> >> CONFIG_TOYBOX_LSM_NONE just means selinux support was NOT selected, and
> >> CONFIG_TOYBOX_SUID... can't work in the AOSP build? (Not only does the android
> >> build not run as root, but it doesn't CLONE as root either, meaning the toybox
> >> binary can't be installed suid root, so never has any privileges to drop.)
> >
> > those will just be historical accidents. i can't actually remember
> > whether i started the two host configs from the android config, or
> > from separate defconfig/macos_defconfig runs. (there's probably stuff
> > in the files that doesn't exist any more, like the recent CATV
> > removals. a script to check for such things wouldn't be useless :-) )
>
> $ cat blah.sh
> #!/bin/bash
>
> x() {
>   sed -n 's/.*\(CONFIG_[^ =]*\).*/\1/p' "$1" | sort
> }
>
> egrep "^($(diff -u <(x "$1") <(x "$2") |sed -n 's/^+C/C/p' | xargs | tr ' '
> '|'))" "$2"
> $ ./blah.sh ~/toybox/clean/.config .config-linux
> CONFIG_TOYBOX_I18N=y
> CONFIG_CAT_V=y
> CONFIG_CP_PRESERVE=y

well, i meant "checked in" :-)

https://android-review.googlesource.com/c/platform/external/toybox/+/2093104
cleans up the current set, but if you're not going to check that in
yourself, i'll probably add it to our "post-update" script so we'll at
least see regular warnings.

> >> I guess this means I should use the device .config?
> >
> > if your aim is "build and test what android ships on the device", yes.
>
> That's more or less what I'm trying to track with this exercise.
>
> > if your aim is "build and test what android uses on the host to
> > _build_ android", you want the linux config.
>
> I'm testing toybox defconfig for the build environment stuff.
>
> > (and if you're really not-so-secretly interested in "what i'd need to
> > build android on android", you'd still just need the linux config :-)
> > )
> >
> >> (It sounds like what
> >> .config-linux wants is everything built but only certain symlinks populated?
> >> Which I guess is a dependency tracking thing?)
> >
> > no, historically the linux config is "what does the AOSP build need?".
> > that's a much smaller set than "full" toybox.
> >
> > that said, i am quite tempted to just add "all the stuff from posix/"
> > to the linux set. but the rate at which i get asked to add missing
> > stuff (usually by the folks building the _kernel_ than folks building
> > AOSP, because AOSP is locked down by default, and you can't just
> > "accidentally" add a dependency on a new host tool) is so low that
> > it's just not been worth it.
>
> I point you at PENDING= in scripts/install.sh, which is everything I've needed
> for mkroot.sh to build kernels for:
>
> https://landley.net/toybox/downloads/binaries/mkroot/latest/
>
> The relevant section is:
>
> # The following are commands toybox should provide, but doesn't yet.
> # For now symlink the host version. This list must go away by 1.0.
>
> PENDING="dd diff expr git tr vi bash sh xzcat bc ar gzip   ftpd less awk unxz
> bison flex make nm"
>
> # "gcc" can go away if the kernel guys merge my patch:
> # http://lkml.iu.edu/hypermail/linux/kernel/2202.0/01505.html
> TOOLCHAIN="as cc ld gcc objdump"

for _Android_ kernels (an "extended subset") i'm told all we need right now is:

1. `tar --transform` (because of
https://android.googlesource.com/kernel/build/+/d68a8336a396a98820de2b3432ce5206fe70c854/build.sh#668)

2. `realpath --relative-to` (which i think we've talked about before,
but i've lost the link right now)

(i've asked them to get back to me on whether toybox install is
sufficient for their install(1) needs. i don't think they've actually
tried it since i added the symlink.)

> > (and especially not worth dealing with
> > the can of worms that is "aye, but you only have one directory of
> > symlinks for prebuilts/build-tools and if you add something there
> > 'just because' then it [aiui] automatically gets made available for
> > build rules, which is probably something i shouldn't just unilaterally
> > do" :-) )
>
> Back in the day, every busybox command I added to aboriginal's linux from
> scratch build I had to rebuild all the packages to see if any of the configure
> steps would decide to go crazy.
>
> You at least have much less gnu autoconf exposure to deal with. (I hope.)

sort of. autoconf is definitely a problem for us (see external/**) but
-- for better *and* worse -- we check in specific autoconf results
rather than actually run autoconf for multiple architectures in every
build.

> >> Rob
>
> Rob
>
> P.S. Because the powerpc64le build calls "bash" out of the $PATH rather than as
> #!/bin/bash, that's why it needs the bash symlink in the $PATH. Only
> architecture I've seen do that so far...
>
> P.P.S. I don't remember why vi and diff need to be in the $PATH, I can try the
> builds without them and see...