[Toybox] Android .config pondering.

Tue May 10 10:04:19 PDT 2022

On Tue, May 10, 2022 at 7:45 AM Rob Landley <rob at landley.net> wrote:
>
> On 5/9/22 19:37, enh wrote:
> > On Sat, May 7, 2022 at 4:38 AM Rob Landley <rob at landley.net> wrote:
> >> > iirc the reason they differ is that macos_defconfig is "everything
> >> > that passes the tests on a mac [the last time enh ran all the tests on
> >> > a mac]" whereas the android mac config is basically "the intersection
> >> > of macos_defconfig and android's _linux_ config".
> >>
> >> The android one had a couple commands that weren't in macos_defconfig, so I
> >> added them. :)
> >
> > yeah, that certainly seems like a mistake :-)
>
> Things that directly call Linux syscalls or mess with /proc or /sys probably
> won't work on mac, but most of the rest should? I mean looking at toys/other/*.c
>
> $ sed -n 's/^config /CONFIG_/p' toys/other/*.c | egrep -v "^$(sed -n 's/=y//p'
> kconfig/macos_miniconfig  | xargs | tr ' ' '|')\$"
>
> There's no reason bzcat shouldn't work. It looks like shred should work? pwgen?
> mcookie? fsync? Dunno about chroot because I haven't opened the suid can of
> worms on macos yet. blkid is basically another variant of "file". base64 is
> enabled but not base32? ascii but not unicode?
>
> In posix egrep and fgrep should probably be switched on... In lsb/ gunzip and
> sync? In net/ host, httpd, and sntp probably aren't too far from workability?
>
> Ahem. Mac support is WAAAAAAAAY down the todo list. If Darwin worked and could
> boot under qemu that would be another matter, but "you must buy an expensive
> proprietary development environment to play, and trying to author software
> without paying tribute to us is illegal" puts me off the whole platform...

right now i think the "can't bootstrap without an existing toybox
binary" is the worst mac problem. (i think there's already a thread
about how your sed skills are too much for BSD sed...) by the time
you've installed homebrew to get GNU sed to build toybox ... you've
installed homebrew and can just install GNU everything, which is
probably fine for most mac users...

sadly my sed skills weren't enough to make that portable. (and i still
haven't installed homebrew on my current mac.)

> >> (I'm simultaneously happy somebody is looking at sh.c and sending me patches,
> >> and going "it's not ready for company..." The next test_sh that fails is the one
> >> checking that pipeline segments are properly backgrounded, because right now if
> >> you pipe a while loop into another while loop the second one won't start running
> >> until the first one finishes because they're happening int the same shell
> >> context. One of the largeish reworks I got distracted from many moons ago is
> >> most of the way to making that work, but it's nontrivial...)
> >
> > yeah, i did tell him, but he couldn't resist poking it with a stick to
> > see what kind of condition it was in.
>
> I'm not complaining! I'm happy! And I've gotten similar pokes from other people
> (including at least one Apple developer). I'm just... perpetually disappointed
> with myself for not having gotten more done faster.
>
> Working on it... :)
>
> >> >> In fact only two symbols are enabled in the linux host but NOT the device:
> >> >> CONFIG_TOYBOX_LSM_NONE just means selinux support was NOT selected, and
> >> >> CONFIG_TOYBOX_SUID... can't work in the AOSP build? (Not only does the android
> >> >> build not run as root, but it doesn't CLONE as root either, meaning the toybox
> >> >> binary can't be installed suid root, so never has any privileges to drop.)
> >> >
> >> > those will just be historical accidents. i can't actually remember
> >> > whether i started the two host configs from the android config, or
> >> > from separate defconfig/macos_defconfig runs. (there's probably stuff
> >> > in the files that doesn't exist any more, like the recent CATV
> >> > removals. a script to check for such things wouldn't be useless :-) )
> >>
> >> $ cat blah.sh
> >> #!/bin/bash
> >>
> >> x() {
> >>   sed -n 's/.*\(CONFIG_[^ =]*\).*/\1/p' "$1" | sort
> >> }
> >>
> >> egrep "^($(diff -u <(x "$1") <(x "$2") |sed -n 's/^+C/C/p' | xargs | tr ' '
> >> '|'))" "$2"
> >> $ ./blah.sh ~/toybox/clean/.config .config-linux
> >> CONFIG_TOYBOX_I18N=y
> >> CONFIG_CAT_V=y
> >> CONFIG_CP_PRESERVE=y
> >
> > well, i meant "checked in" :-)
>
> I have a tool to create miniconfig files from full .config files. There's a
> certain amount of history there (https://lwn.net/Articles/160497/ is just the
> tip of the iceberg), but it does work:
>
>   https://github.com/landley/aboriginal/blob/master/more/miniconfig.sh
>
> I haven't migrated that over to toybox yet, nor did I transplant
> https://landley.net/aboriginal/FAQ.html#dev_miniconfig to the new FAQ yet. But
> possibly I should...
>
> In theory the current kernel config plumbing has a "make savedefconfig" that
> writes out a similarly terse format, but unfortunately they decided to diff
> against "defconfig" rather than against "allnoconfig", which I consider
> significantly less useful from an "understanding what's going on" perspective.
> That said, I've already had to add "switch a symbol off" support FROM
> ALLNOCONFIG because the kernel devs are insane:
>
>   https://landley.net/notes-2014.html#13-08-2014
>
> But since I'm A) using a kconfig fork from the dawn of time, B) planning to
> rewrite it all anyway... I've just left that on the todo list for now. :)
>
> > https://android-review.googlesource.com/c/platform/external/toybox/+/2093104
> > cleans up the current set, but if you're not going to check that in
> > yourself, i'll probably add it to our "post-update" script so we'll at
> > least see regular warnings.
>
> The current plumbing can already do it:
>
> $ make allnoconfig KCONFIG_ALLCONFIG=.config-linux
> kconfig/conf -n Config.in > /dev/null
> .config-linux:13:warning: trying to assign nonexistent symbol TOYBOX_FALLOCATE
> .config-linux:21:warning: trying to assign nonexistent symbol TOYBOX_I18N
> .config-linux:26:warning: trying to assign nonexistent symbol
> TOYBOX_MUSL_NOMMU_IS_BROKEN
> .config-linux:29:warning: trying to assign nonexistent symbol TOYBOX_PEDANTIC_ARGS
> .config-linux:61:warning: trying to assign nonexistent symbol CATV
> .config-linux:62:warning: trying to assign nonexistent symbol CAT_V
> .config-linux:80:warning: trying to assign nonexistent symbol CP_PRESERVE
> .config-linux:149:warning: trying to assign nonexistent symbol HELP_EXTRAS
> .config-linux:192:warning: trying to assign nonexistent symbol LOGWRAPPER
> .config-linux:198:warning: trying to assign nonexistent symbol LSPCI_TEXT
> .config-linux:231:warning: trying to assign nonexistent symbol NETCAT_LISTEN
> .config-linux:356:warning: trying to assign nonexistent symbol XARGS_PEDANTIC

ah, great. i've cleaned all that up too, and taught our post-update
script to run this every time:
https://android-review.googlesource.com/c/platform/external/toybox/+/2093104

> >> The relevant section is:
> >>
> >> # The following are commands toybox should provide, but doesn't yet.
> >> # For now symlink the host version. This list must go away by 1.0.
> >>
> >> PENDING="dd diff expr git tr vi bash sh xzcat bc ar gzip   ftpd less awk unxz
> >> bison flex make nm"
> >>
> >> # "gcc" can go away if the kernel guys merge my patch:
> >> # http://lkml.iu.edu/hypermail/linux/kernel/2202.0/01505.html
> >> TOOLCHAIN="as cc ld gcc objdump"
> >
> > for _Android_ kernels (an "extended subset") i'm told all we need right now is:
> >
> > 1. `tar --transform` (because of
> > https://android.googlesource.com/kernel/build/+/d68a8336a396a98820de2b3432ce5206fe70c854/build.sh#668)
>
> Ok.

(this morning i had them ask "does toybox tar support $TAR_OPTIONS?"
wrt to https://android-review.googlesource.com/c/kernel/build/+/2090303
where they'd like to be able to factor out the various "reproducible
tarball please" options [like in the toybox tar tests].)

> > 2. `realpath --relative-to` (which i think we've talked about before,
> > but i've lost the link right now)
>
> I've got multiple links on that, and it's halfway done in my tree. Currently a
> GIANT MESS but I'll try to put the bits back together this week. (I think there
> are also a couple cases where cp -s with relative paths does the wrong thing for
> the same reason?)
>
> $ git diff toys/*/readlink.c | wc
>     123     521    3617
> $ git diff tests/readlink.test | wc
>      31     241    1560
>
> (There were just too many half-finished projects to tidy them all up for the
> release, I ground away at it for a month and then punted...)
>
> > (i've asked them to get back to me on whether toybox install is
> > sufficient for their install(1) needs. i don't think they've actually
> > tried it since i added the symlink.)
>
> Happy to field bug reports...
>
> >> > (and especially not worth dealing with
> >> > the can of worms that is "aye, but you only have one directory of
> >> > symlinks for prebuilts/build-tools and if you add something there
> >> > 'just because' then it [aiui] automatically gets made available for
> >> > build rules, which is probably something i shouldn't just unilaterally
> >> > do" :-) )
> >>
> >> Back in the day, every busybox command I added to aboriginal's linux from
> >> scratch build I had to rebuild all the packages to see if any of the configure
> >> steps would decide to go crazy.
> >>
> >> You at least have much less gnu autoconf exposure to deal with. (I hope.)
> >
> > sort of. autoconf is definitely a problem for us (see external/**) but
> > -- for better *and* worse -- we check in specific autoconf results
> > rather than actually run autoconf for multiple architectures in every
> > build.
>
> Ow.
>
> I need a whiteboard just to unpack the layers of wrong going on there, but as
> with double negatives NOT making a positive (and yet "yeah right" means you
> don't believe somewhere), I think this needs calculus to determine whether or
> not it's net the right thing...?

chromeos went the opposite route and just have a different set of
problems. it's not clear that either is better or worse (though it is
clear that _other_ problems are much bigger :-) ).

> Ow.
>
> ("I say we take off and nuke the entire site from orbit. Only way to be sure." -
> Ellen Ripley.)
>
> Rob
>
> (See, with aboriginal linux I was making my automated Linux From Scratch build
> work for whatever host architecture you ran it on, x86, arm, mips, powerpc, sh4,
> sparc, and so on. 95% of what autoconf dies boils down to 1) I was unaware of
> all the symbols "cc -E -dM - < /dev/null" shows you, 2) #if
> __has_include(<file>) hadn't been invented yet. But unfortunately, if you
> snapshot the output it tries to use the arm answers on sparc, and you have to
> preprepare versions for each target architecture in which case you might as well
> just ship binaries? So I put in the work to make it actually perform its stupid
> dance and get the right answers, so that when I added m68k or s390x it would
> mostly Just Work. Not having autoconf at all is, of course, the much better
> option...)

aka "the only winning move is not to play" :-)

+1 to that!