[Toybox] Android .config pondering.

[enh]

On Wed, May 11, 2022 at 7:16 AM Rob Landley <rob at landley.net> wrote:
>
> On 5/10/22 12:04, enh wrote:
> > right now i think the "can't bootstrap without an existing toybox
> > binary" is the worst mac problem. (i think there's already a thread
> > about how your sed skills are too much for BSD sed...)
>
> It has a SED= environment variable so you can point it at gsed on mac, but
> GETTING gsed on the mac is outside my expertise...

yeah, that's the "homebrew" i was talking about. (for all i know, it
might actually be easier to just download and build gnu sed alone, but
"if you're planning on using a mac for development, you'll want
homebrew sooner or later" has meant i've never yet not given in and
installed the whole thing.)

> > by the time
> > you've installed homebrew to get GNU sed to build toybox ... you've
> > installed homebrew and can just install GNU everything, which is
> > probably fine for most mac users...
> >
> > sadly my sed skills weren't enough to make that portable. (and i still
> > haven't installed homebrew on my current mac.)
>
> The point of generated/build.sh is to have at least a start at being able to
> compile a binary without running any build plumbing, so you can compile on a
> system that doesn't even have "make", or a nontrivial shell, ala:
>
>   cc *.c -o toybox -I existing_generated_snapshot
>
> (Shipping a generated/* snapshot is a bit like shipping ./configure in an
> autoconf system or kconfig/*_shipped in toybox. (Which the kernel did until they
> made menuconfig turing complete in 2018 so "make oldconfig" can "rm -rf ~".)
>
> I want a proper solution to that, a "make airlock" providing a
> no-external-libraries version of the commands needed to build toybox defconfig,
> with the supplied config assuming that all of the scripts/genconfig.sh probes
> failed and none of the extra shared libraries were available.
>
> But that comes _after_ getting toybox to build under toybox on the todo list.
>
> And unfortunately it needs to work with the bsd and mac defconfigs because "ps
> won't work on mac" is a seperate problem from "the build environment can't run
> the plumbing without help". Which means it needs to be able to create a new
> generated/configs.h but I can do that with posix sed. Let's see...
>
> $ sed -n -e 's/# CONFIG_\(.*\) is not set/CFG_\1 1 at USE_\1(...) __VA_ARGS__/p' -e
> 's/^CONFIG_\(.*\)=y/CFG_\1 0 at USE_\1(...)/p' .config | tr @ '\n'
>
> That should work fine with Posix-2001 tools from 20 years ago, and the rest of
> the generated/ files can be a snapshot. (At least for commands with no
> sub-options, which is almost all of them now. Otherwise generated/flags.h with
> the opststr substitutions would get out of sync with config changes, but
> "command exsits" vs "command does not exist" isn't problematic.)
>
> So being able to do a 'make airlock' and then 'PATH="$PWD/airlock:$PATH" make
> clean defconfig toybox tests install' under that isn't a huge stretch. Possibly
> 'make airlock' should install in generated/airlock and the rest of the build
> automatically stick that at the start of the $PATH. (Note that mkroot does this
> already, it just doesn't try to hunker down and defend itself from an
> insufficient initial build environment. It uses its airlock step to avoid
> cross-compiling host/target contamination...)
>
> > ah, great. i've cleaned all that up too, and taught our post-update
> > script to run this every time:
> > https://android-review.googlesource.com/c/platform/external/toybox/+/2093104
>
> Cool.
>
> >> >> The relevant section is:
> >> >>
> >> >> # The following are commands toybox should provide, but doesn't yet.
> >> >> # For now symlink the host version. This list must go away by 1.0.
> >> >>
> >> >> PENDING="dd diff expr git tr vi bash sh xzcat bc ar gzip   ftpd less awk unxz
> >> >> bison flex make nm"
> >> >>
> >> >> # "gcc" can go away if the kernel guys merge my patch:
> >> >> # http://lkml.iu.edu/hypermail/linux/kernel/2202.0/01505.html
> >> >> TOOLCHAIN="as cc ld gcc objdump"
> >> >
> >> > for _Android_ kernels (an "extended subset") i'm told all we need right now is:
> >> >
> >> > 1. `tar --transform` (because of
> >> > https://android.googlesource.com/kernel/build/+/d68a8336a396a98820de2b3432ce5206fe70c854/build.sh#668)
> >>
> >> Ok.
> >
> > (this morning i had them ask "does toybox tar support $TAR_OPTIONS?"
>
> $ man tar | grep TAR_OPTIONS
> $
>
> I don't know what that is?

i was about to celebrate (because i'd already said to them that i
personally _hate_ `GREP_OPTIONS` _because_ it messes with hermetic
builds unless you know about it and explicitly clobber it, and the
idea of having random other commands grow similar warts doesn't
exactly fill me with joy) ... but then i noticed you only said "man",
and this is a gnu thing, so _of course_ the man page won't mention it.
how else could they make you use their stupid "info" crap?

anyway, checking whether this is a real thing the One True Way:

$ strings `which tar` | grep OPTION
TAR_OPTIONS
cannot split TAR_OPTIONS: %s
 [OPTION...]

it's also described on the web:
https://www.gnu.org/software/tar/manual/html_section/using-tar-options.html

(but i still think it's a bad idea, personally.)

> > wrt to https://android-review.googlesource.com/c/kernel/build/+/2090303
> > where they'd like to be able to factor out the various "reproducible
> > tarball please" options [like in the toybox tar tests].)
>
> It supports --owner and --group and I made it so you can specify the numeric IDs
> for both with the :123 syntax so you can specify a user that isn't in
> /etc/passwd. (Commit 690526a84ffc.)

yeah, that's what they want to not have to keep repeating.

> >> >> You at least have much less gnu autoconf exposure to deal with. (I hope.)
> >> >
> >> > sort of. autoconf is definitely a problem for us (see external/**) but
> >> > -- for better *and* worse -- we check in specific autoconf results
> >> > rather than actually run autoconf for multiple architectures in every
> >> > build.
> >>
> >> Ow.
> >>
> >> I need a whiteboard just to unpack the layers of wrong going on there, but as
> >> with double negatives NOT making a positive (and yet "yeah right" means you
> >> don't believe somewhere), I think this needs calculus to determine whether or
> >> not it's net the right thing...?
> >
> > chromeos went the opposite route and just have a different set of
> > problems. it's not clear that either is better or worse (though it is
> > clear that _other_ problems are much bigger :-) ).
>
> Yeah, back in the day I made autoconf work with busybox. And making it work
> SUCKED and took years. And I wound up stuck on an old version of Linux From
> Scratch because updating was such a pain, although half of that was staying with
> the last GPLv2 release of all the toolchain packages because the FSF went
> aggressively malignantly nuts.
>
> But my goal was "same behavior on as many build targets as possible". You've got
> multiple vendor forks to a bunch of hardware targets, but there' almost all
> arm64 with bionic and llvm.
>
> Meanwhile I was hitting
> https://lkml.iu.edu/hypermail/linux/kernel/1002.2/02231.html regularly. Right
> now I'm trying to add a coldfire toolchain to to mkroot and it's all
> https://www.denx.de/wiki/U-Boot/ColdFireNotes
>
> > Since gcc team seems to keep m68k issues in a very low priority, these
> > toolchains have the libgcc.a, libgcov.a and multilibs copied from an old
> > toolchain.
>
> Thank you Wolfgang. Thanks EVER SO MUCH. Embedded guys just stop engaging with
> "upstream" and keep using 10 year old kernels and toolchains because they got it
> to work once and don't care what the crazy people are off doing. I'm nuts for
> trying to get current stuff to work on the full range of theoretically supported
> thingies, including NATIVE COMPILING on them.
>
> Sigh.

could be worse ... could be a _proprietary_ toolchain from a decade
ago. not that _that_ ever happens...

> >> (See, with aboriginal linux I was making my automated Linux From Scratch build
> >> work for whatever host architecture you ran it on, x86, arm, mips, powerpc, sh4,
> >> sparc, and so on. 95% of what autoconf dies boils down to 1) I was unaware of
> >> all the symbols "cc -E -dM - < /dev/null" shows you, 2) #if
> >> __has_include(<file>) hadn't been invented yet. But unfortunately, if you
> >> snapshot the output it tries to use the arm answers on sparc, and you have to
> >> preprepare versions for each target architecture in which case you might as well
> >> just ship binaries? So I put in the work to make it actually perform its stupid
> >> dance and get the right answers, so that when I added m68k or s390x it would
> >> mostly Just Work. Not having autoconf at all is, of course, the much better
> >> option...)
> >
> > aka "the only winning move is not to play" :-)
> >
> > +1 to that!
>
> I had a rant years ago about how configure/make/install needed to be replaced
> the way git replaced CVS. Here's a 2-part version, I'm sure I didn't better
> writeups but can't find them....
>
> http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000859.html
> http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000860.html
>
> Unfortunately, all I'd seen when I wrote that was a lot of svn and perforce, and
> not a real proper "everybody moves to the new thing and universally agrees its'
> better" complete rethink the way git finally rendered cvs properly irrelevant.
> And sadly, that's STILL the case. (Otherwise we wouldn't have this
> cmake/ninja/kaiju cycling every 5 years with the kernel still using gmake.)

i think the trouble is that no-one's found the "big thing" here that
git was able to offer. i don't think we're in the git/bk/bzr/hg/...
phase, i think we're still in the cvs/svn phase.

version control also had the advantage that you could use the same one
for all languages; every individual language community seems to have a
strong preference for "their" build system, even if/though it's
useless for everyone else.

i wouldn't hold my breath for this getting any better before we're all retired.

(i'll let the reader decide for themselves whether rpi pico
introducing embedded folks to cmake is a positive step or not :-) )

> Rob

P.S. since i had a few minutes before my next meeting, i gave in and
built gnu sed from source ... it took literally _minutes_ to run
configure on this M1 mac, and then a couple of _seconds_ to actually
build. so so wrong...