[Toybox] Android .config pondering.

[Rob Landley]

On 5/9/22 19:37, enh wrote:
> On Sat, May 7, 2022 at 4:38 AM Rob Landley <rob at landley.net> wrote:
>> > iirc the reason they differ is that macos_defconfig is "everything
>> > that passes the tests on a mac [the last time enh ran all the tests on
>> > a mac]" whereas the android mac config is basically "the intersection
>> > of macos_defconfig and android's _linux_ config".
>>
>> The android one had a couple commands that weren't in macos_defconfig, so I
>> added them. :)
> 
> yeah, that certainly seems like a mistake :-)

Things that directly call Linux syscalls or mess with /proc or /sys probably
won't work on mac, but most of the rest should? I mean looking at toys/other/*.c

$ sed -n 's/^config /CONFIG_/p' toys/other/*.c | egrep -v "^$(sed -n 's/=y//p'
kconfig/macos_miniconfig  | xargs | tr ' ' '|')\$"

There's no reason bzcat shouldn't work. It looks like shred should work? pwgen?
mcookie? fsync? Dunno about chroot because I haven't opened the suid can of
worms on macos yet. blkid is basically another variant of "file". base64 is
enabled but not base32? ascii but not unicode?

In posix egrep and fgrep should probably be switched on... In lsb/ gunzip and
sync? In net/ host, httpd, and sntp probably aren't too far from workability?

Ahem. Mac support is WAAAAAAAAY down the todo list. If Darwin worked and could
boot under qemu that would be another matter, but "you must buy an expensive
proprietary development environment to play, and trying to author software
without paying tribute to us is illegal" puts me off the whole platform...

>> (I'm simultaneously happy somebody is looking at sh.c and sending me patches,
>> and going "it's not ready for company..." The next test_sh that fails is the one
>> checking that pipeline segments are properly backgrounded, because right now if
>> you pipe a while loop into another while loop the second one won't start running
>> until the first one finishes because they're happening int the same shell
>> context. One of the largeish reworks I got distracted from many moons ago is
>> most of the way to making that work, but it's nontrivial...)
> 
> yeah, i did tell him, but he couldn't resist poking it with a stick to
> see what kind of condition it was in.

I'm not complaining! I'm happy! And I've gotten similar pokes from other people
(including at least one Apple developer). I'm just... perpetually disappointed
with myself for not having gotten more done faster.

Working on it... :)

>> >> In fact only two symbols are enabled in the linux host but NOT the device:
>> >> CONFIG_TOYBOX_LSM_NONE just means selinux support was NOT selected, and
>> >> CONFIG_TOYBOX_SUID... can't work in the AOSP build? (Not only does the android
>> >> build not run as root, but it doesn't CLONE as root either, meaning the toybox
>> >> binary can't be installed suid root, so never has any privileges to drop.)
>> >
>> > those will just be historical accidents. i can't actually remember
>> > whether i started the two host configs from the android config, or
>> > from separate defconfig/macos_defconfig runs. (there's probably stuff
>> > in the files that doesn't exist any more, like the recent CATV
>> > removals. a script to check for such things wouldn't be useless :-) )
>>
>> $ cat blah.sh
>> #!/bin/bash
>>
>> x() {
>>   sed -n 's/.*\(CONFIG_[^ =]*\).*/\1/p' "$1" | sort
>> }
>>
>> egrep "^($(diff -u <(x "$1") <(x "$2") |sed -n 's/^+C/C/p' | xargs | tr ' '
>> '|'))" "$2"
>> $ ./blah.sh ~/toybox/clean/.config .config-linux
>> CONFIG_TOYBOX_I18N=y
>> CONFIG_CAT_V=y
>> CONFIG_CP_PRESERVE=y
> 
> well, i meant "checked in" :-)

I have a tool to create miniconfig files from full .config files. There's a
certain amount of history there (https://lwn.net/Articles/160497/ is just the
tip of the iceberg), but it does work:

  https://github.com/landley/aboriginal/blob/master/more/miniconfig.sh

I haven't migrated that over to toybox yet, nor did I transplant
https://landley.net/aboriginal/FAQ.html#dev_miniconfig to the new FAQ yet. But
possibly I should...

In theory the current kernel config plumbing has a "make savedefconfig" that
writes out a similarly terse format, but unfortunately they decided to diff
against "defconfig" rather than against "allnoconfig", which I consider
significantly less useful from an "understanding what's going on" perspective.
That said, I've already had to add "switch a symbol off" support FROM
ALLNOCONFIG because the kernel devs are insane:

  https://landley.net/notes-2014.html#13-08-2014

But since I'm A) using a kconfig fork from the dawn of time, B) planning to
rewrite it all anyway... I've just left that on the todo list for now. :)

> https://android-review.googlesource.com/c/platform/external/toybox/+/2093104
> cleans up the current set, but if you're not going to check that in
> yourself, i'll probably add it to our "post-update" script so we'll at
> least see regular warnings.

The current plumbing can already do it:

$ make allnoconfig KCONFIG_ALLCONFIG=.config-linux
kconfig/conf -n Config.in > /dev/null
.config-linux:13:warning: trying to assign nonexistent symbol TOYBOX_FALLOCATE
.config-linux:21:warning: trying to assign nonexistent symbol TOYBOX_I18N
.config-linux:26:warning: trying to assign nonexistent symbol
TOYBOX_MUSL_NOMMU_IS_BROKEN
.config-linux:29:warning: trying to assign nonexistent symbol TOYBOX_PEDANTIC_ARGS
.config-linux:61:warning: trying to assign nonexistent symbol CATV
.config-linux:62:warning: trying to assign nonexistent symbol CAT_V
.config-linux:80:warning: trying to assign nonexistent symbol CP_PRESERVE
.config-linux:149:warning: trying to assign nonexistent symbol HELP_EXTRAS
.config-linux:192:warning: trying to assign nonexistent symbol LOGWRAPPER
.config-linux:198:warning: trying to assign nonexistent symbol LSPCI_TEXT
.config-linux:231:warning: trying to assign nonexistent symbol NETCAT_LISTEN
.config-linux:356:warning: trying to assign nonexistent symbol XARGS_PEDANTIC

>> The relevant section is:
>>
>> # The following are commands toybox should provide, but doesn't yet.
>> # For now symlink the host version. This list must go away by 1.0.
>>
>> PENDING="dd diff expr git tr vi bash sh xzcat bc ar gzip   ftpd less awk unxz
>> bison flex make nm"
>>
>> # "gcc" can go away if the kernel guys merge my patch:
>> # http://lkml.iu.edu/hypermail/linux/kernel/2202.0/01505.html
>> TOOLCHAIN="as cc ld gcc objdump"
> 
> for _Android_ kernels (an "extended subset") i'm told all we need right now is:
> 
> 1. `tar --transform` (because of
> https://android.googlesource.com/kernel/build/+/d68a8336a396a98820de2b3432ce5206fe70c854/build.sh#668)

Ok.

> 2. `realpath --relative-to` (which i think we've talked about before,
> but i've lost the link right now)

I've got multiple links on that, and it's halfway done in my tree. Currently a
GIANT MESS but I'll try to put the bits back together this week. (I think there
are also a couple cases where cp -s with relative paths does the wrong thing for
the same reason?)

$ git diff toys/*/readlink.c | wc
    123     521    3617
$ git diff tests/readlink.test | wc
     31     241    1560

(There were just too many half-finished projects to tidy them all up for the
release, I ground away at it for a month and then punted...)

> (i've asked them to get back to me on whether toybox install is
> sufficient for their install(1) needs. i don't think they've actually
> tried it since i added the symlink.)

Happy to field bug reports...

>> > (and especially not worth dealing with
>> > the can of worms that is "aye, but you only have one directory of
>> > symlinks for prebuilts/build-tools and if you add something there
>> > 'just because' then it [aiui] automatically gets made available for
>> > build rules, which is probably something i shouldn't just unilaterally
>> > do" :-) )
>>
>> Back in the day, every busybox command I added to aboriginal's linux from
>> scratch build I had to rebuild all the packages to see if any of the configure
>> steps would decide to go crazy.
>>
>> You at least have much less gnu autoconf exposure to deal with. (I hope.)
> 
> sort of. autoconf is definitely a problem for us (see external/**) but
> -- for better *and* worse -- we check in specific autoconf results
> rather than actually run autoconf for multiple architectures in every
> build.

Ow.

I need a whiteboard just to unpack the layers of wrong going on there, but as
with double negatives NOT making a positive (and yet "yeah right" means you
don't believe somewhere), I think this needs calculus to determine whether or
not it's net the right thing...?

Ow.

("I say we take off and nuke the entire site from orbit. Only way to be sure." -
Ellen Ripley.)

Rob

(See, with aboriginal linux I was making my automated Linux From Scratch build
work for whatever host architecture you ran it on, x86, arm, mips, powerpc, sh4,
sparc, and so on. 95% of what autoconf dies boils down to 1) I was unaware of
all the symbols "cc -E -dM - < /dev/null" shows you, 2) #if
__has_include(<file>) hadn't been invented yet. But unfortunately, if you
snapshot the output it tries to use the arm answers on sparc, and you have to
preprepare versions for each target architecture in which case you might as well
just ship binaries? So I put in the work to make it actually perform its stupid
dance and get the right answers, so that when I added m68k or s390x it would
mostly Just Work. Not having autoconf at all is, of course, the much better
option...)