[Toybox] Android .config pondering.

[Rob Landley]

On 5/10/22 12:04, enh wrote:
> right now i think the "can't bootstrap without an existing toybox
> binary" is the worst mac problem. (i think there's already a thread
> about how your sed skills are too much for BSD sed...)

It has a SED= environment variable so you can point it at gsed on mac, but
GETTING gsed on the mac is outside my expertise...

> by the time
> you've installed homebrew to get GNU sed to build toybox ... you've
> installed homebrew and can just install GNU everything, which is
> probably fine for most mac users...
>
> sadly my sed skills weren't enough to make that portable. (and i still
> haven't installed homebrew on my current mac.)

The point of generated/build.sh is to have at least a start at being able to
compile a binary without running any build plumbing, so you can compile on a
system that doesn't even have "make", or a nontrivial shell, ala:

  cc *.c -o toybox -I existing_generated_snapshot

(Shipping a generated/* snapshot is a bit like shipping ./configure in an
autoconf system or kconfig/*_shipped in toybox. (Which the kernel did until they
made menuconfig turing complete in 2018 so "make oldconfig" can "rm -rf ~".)

I want a proper solution to that, a "make airlock" providing a
no-external-libraries version of the commands needed to build toybox defconfig,
with the supplied config assuming that all of the scripts/genconfig.sh probes
failed and none of the extra shared libraries were available.

But that comes _after_ getting toybox to build under toybox on the todo list.

And unfortunately it needs to work with the bsd and mac defconfigs because "ps
won't work on mac" is a seperate problem from "the build environment can't run
the plumbing without help". Which means it needs to be able to create a new
generated/configs.h but I can do that with posix sed. Let's see...

$ sed -n -e 's/# CONFIG_\(.*\) is not set/CFG_\1 1 at USE_\1(...) __VA_ARGS__/p' -e
's/^CONFIG_\(.*\)=y/CFG_\1 0 at USE_\1(...)/p' .config | tr @ '\n'

That should work fine with Posix-2001 tools from 20 years ago, and the rest of
the generated/ files can be a snapshot. (At least for commands with no
sub-options, which is almost all of them now. Otherwise generated/flags.h with
the opststr substitutions would get out of sync with config changes, but
"command exsits" vs "command does not exist" isn't problematic.)

So being able to do a 'make airlock' and then 'PATH="$PWD/airlock:$PATH" make
clean defconfig toybox tests install' under that isn't a huge stretch. Possibly
'make airlock' should install in generated/airlock and the rest of the build
automatically stick that at the start of the $PATH. (Note that mkroot does this
already, it just doesn't try to hunker down and defend itself from an
insufficient initial build environment. It uses its airlock step to avoid
cross-compiling host/target contamination...)

> ah, great. i've cleaned all that up too, and taught our post-update
> script to run this every time:
> https://android-review.googlesource.com/c/platform/external/toybox/+/2093104

Cool.

>> >> The relevant section is:
>> >>
>> >> # The following are commands toybox should provide, but doesn't yet.
>> >> # For now symlink the host version. This list must go away by 1.0.
>> >>
>> >> PENDING="dd diff expr git tr vi bash sh xzcat bc ar gzip   ftpd less awk unxz
>> >> bison flex make nm"
>> >>
>> >> # "gcc" can go away if the kernel guys merge my patch:
>> >> # http://lkml.iu.edu/hypermail/linux/kernel/2202.0/01505.html
>> >> TOOLCHAIN="as cc ld gcc objdump"
>> >
>> > for _Android_ kernels (an "extended subset") i'm told all we need right now is:
>> >
>> > 1. `tar --transform` (because of
>> > https://android.googlesource.com/kernel/build/+/d68a8336a396a98820de2b3432ce5206fe70c854/build.sh#668)
>>
>> Ok.
> 
> (this morning i had them ask "does toybox tar support $TAR_OPTIONS?"

$ man tar | grep TAR_OPTIONS
$

I don't know what that is?

> wrt to https://android-review.googlesource.com/c/kernel/build/+/2090303
> where they'd like to be able to factor out the various "reproducible
> tarball please" options [like in the toybox tar tests].)

It supports --owner and --group and I made it so you can specify the numeric IDs
for both with the :123 syntax so you can specify a user that isn't in
/etc/passwd. (Commit 690526a84ffc.)

>> >> You at least have much less gnu autoconf exposure to deal with. (I hope.)
>> >
>> > sort of. autoconf is definitely a problem for us (see external/**) but
>> > -- for better *and* worse -- we check in specific autoconf results
>> > rather than actually run autoconf for multiple architectures in every
>> > build.
>>
>> Ow.
>>
>> I need a whiteboard just to unpack the layers of wrong going on there, but as
>> with double negatives NOT making a positive (and yet "yeah right" means you
>> don't believe somewhere), I think this needs calculus to determine whether or
>> not it's net the right thing...?
> 
> chromeos went the opposite route and just have a different set of
> problems. it's not clear that either is better or worse (though it is
> clear that _other_ problems are much bigger :-) ).

Yeah, back in the day I made autoconf work with busybox. And making it work
SUCKED and took years. And I wound up stuck on an old version of Linux From
Scratch because updating was such a pain, although half of that was staying with
the last GPLv2 release of all the toolchain packages because the FSF went
aggressively malignantly nuts.

But my goal was "same behavior on as many build targets as possible". You've got
multiple vendor forks to a bunch of hardware targets, but there' almost all
arm64 with bionic and llvm.

Meanwhile I was hitting
https://lkml.iu.edu/hypermail/linux/kernel/1002.2/02231.html regularly. Right
now I'm trying to add a coldfire toolchain to to mkroot and it's all
https://www.denx.de/wiki/U-Boot/ColdFireNotes

> Since gcc team seems to keep m68k issues in a very low priority, these
> toolchains have the libgcc.a, libgcov.a and multilibs copied from an old
> toolchain.

Thank you Wolfgang. Thanks EVER SO MUCH. Embedded guys just stop engaging with
"upstream" and keep using 10 year old kernels and toolchains because they got it
to work once and don't care what the crazy people are off doing. I'm nuts for
trying to get current stuff to work on the full range of theoretically supported
thingies, including NATIVE COMPILING on them.

Sigh.

>> (See, with aboriginal linux I was making my automated Linux From Scratch build
>> work for whatever host architecture you ran it on, x86, arm, mips, powerpc, sh4,
>> sparc, and so on. 95% of what autoconf dies boils down to 1) I was unaware of
>> all the symbols "cc -E -dM - < /dev/null" shows you, 2) #if
>> __has_include(<file>) hadn't been invented yet. But unfortunately, if you
>> snapshot the output it tries to use the arm answers on sparc, and you have to
>> preprepare versions for each target architecture in which case you might as well
>> just ship binaries? So I put in the work to make it actually perform its stupid
>> dance and get the right answers, so that when I added m68k or s390x it would
>> mostly Just Work. Not having autoconf at all is, of course, the much better
>> option...)
> 
> aka "the only winning move is not to play" :-)
> 
> +1 to that!

I had a rant years ago about how configure/make/install needed to be replaced
the way git replaced CVS. Here's a 2-part version, I'm sure I didn't better
writeups but can't find them....

http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000859.html
http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000860.html

Unfortunately, all I'd seen when I wrote that was a lot of svn and perforce, and
not a real proper "everybody moves to the new thing and universally agrees its'
better" complete rethink the way git finally rendered cvs properly irrelevant.
And sadly, that's STILL the case. (Otherwise we wouldn't have this
cmake/ninja/kaiju cycling every 5 years with the kernel still using gmake.)

Rob