[Toybox] [mkroot] Issue with mcm-buildall.sh

[Rob Landley]

On 3/28/21 3:06 AM, Riccardo Paolo Bestetti wrote:
> On Sat Mar 27, 2021 at 10:43 AM CET, Rob Landley wrote:
>>> This:
>>>
>>> 1) afaics successfully builds the i686 bootstrap compiler in
>>>    $MCM/host-i686-linux-musl
>>
>> Not unless you ask it to? (Or it's left over from a previous run.) Let's
>> see...
>>
>> https://github.com/landley/toybox/blob/master/scripts/mcm-buildall.sh#L235
>>
>> Looks like the two tuple indicators these days would be:
>>
>> scripts/mcm-buildall.sh i686:: x86_64::--with-mtune=nocona
>>
> 
> As far as I can see, the bootstrap toolchain is always built:
> 
> https://github.com/landley/toybox/blob/master/scripts/mcm-buildall.sh#L224

Oh right. (Forgot I added that. :)

>>> 2) fails to build x86_64-linux-musl-cross (see below)
>>
>> Presumably because its ./configure falls back to using your host gcc,
>> because it
>> can't find the i686:: cross compiler that didn't get built.
> 
> As it turns out, that's correct, even though the i686 cross compiler
> /did/ get built. However, for some reason I'm not corageous enough to
> investigate, it got placed into $MCM/ instead of $MCM/output.
>
> Manually copying it to the output directory (and renaming it) was
> sufficient to get step 2, and step 3 after that, to succeed.

That was with the old build script, correct? Is the new one getting the pathing
right? (It is for me, but I haven't rebuilt toolchains in a while.)

>> And the actual error is in the file "config.log", not in the console
>> output,
>> because gcc was written by the FSF and was thus actively user-hostile.
>> (Only
>> special people who proved their worthiness could be allowed inside
>> Richard
>> Stallman's Cathedral of white beardy men. This is why LLVM happened.)
> 
> Ugh.

He currently appears intent on destroying the FSF:

  https://twitter.com/cstross/status/1376630269818630146

(The logic seems to be the "if I can't have her no man shall" line from ladyhawke.)

>> However what we _do_ know is that the i686 cross compiler is borked, and
>> possibly what it's complaining about is your host toolchain can't create
>> static
>> binaries (because libc.a isn't installed by your distro, because it
>> listened to
>> glibc maintainer Ulrich Drepper being loudly wrong several years back.)
>> Building
>> the i686 compiler first should fix it, the rest try to use that as their
>> host
>> compiler because:
>>
>> https://github.com/landley/toybox/blob/master/scripts/mcm-buildall.sh#L18
>>
>> says to.
> 
> My distribution ships libc.a, apparently. So it must have been something
> different.

I vaguely recall something being broken here but "build a musl compiler to
static link against" was the solution. (It generally is for static linking,
Ulrich DrPepper intentionally broke static linking sometime after he wrote the
deeply insane https://www.akkadia.org/drepper/no_static_linking.html by
sprinkling "dlopen()" all through the code so things like DNS lookups simply
wouldn't happen in static linked code. Oh, and he made hello world static link
to most of a megabyte.)

He then left Red Hat to go work for Goldman Sachs in the aftermath of the
mortgage crisis (when they were being extra-evil), to do HFT microoptimization
(more or less conventional finance's version of bitcoin).

>>> Riccardo
>>
>> Rob.
> 
> Nice to meet you! I appreciate your help. I'm attempting kernel
> development for the first time & I'm currently trying to get a hang of
> the avaiable tooling.

If you look at mkroot, it has the kernel configs for a dozen different targets,
capable of booting and running under qemu. Yes, in a ~250 line bash script. The
trick is this bit:

https://github.com/landley/toybox/blob/bc46bc619fcd/scripts/mkroot.sh#L222

That's a list of common config symbols and the loop/sed to expand them. The
explicitly listed symbols there are common to all platforms, and at the end of
the line is $KCONF which is set ala:

https://github.com/landley/toybox/blob/bc46bc619fcd/scripts/mkroot.sh#L193

And then $KERNEL_CONFIG adds any symbols that don't expand to "CONFIG_NAME=y".

This creates what's called a "miniconfig", and I REALLY should migrate
https://landley.net/aboriginal/FAQ.html#dev_miniconfig over to the toybox faq.
But basically, the theory behind miniconfig is "if I start with make
allnoconfig, then fire up make menuconfig and switch on a checklist of symbols
letting the dependency resolver do its thing each time, what is the list of
symbols I would need to manually switch on?" It puts that list of symbols in a
file and lets the plumbing switch them on automatically, the magic invocation
for which is:

  make allnoconfig KCONFIG_ALLCONFIG=filename

And the NICE thing about that list is every symbol in it is necessary (the
resulting kernel behavior changes very noticeably without it), so you can look
them all up in the menuconfig help text (forward slash is "search for symbol
name", and yes it accepts partial strings) and see what they do, and even look
in the code to see what C functions are depending on that symbol.

So, in mkroot.sh two compressed NAME1,NAME1,NAME1 lists gets turned into:

  CONFIG_NAME1=y
  CONFIG_NAME2=y
  CONFIG_NAME3=y

which then gets expanded to a full 2000+ symbol .config (most of them switched
_off_ because kconfig), and then the script invokes the kernel build with that
config using the appropriate cross compiler. (Oh and the $VMLINUX entry in each
arch type says which file to save from the kernel build, to feed to qemu's
built-in bootloader later.)

And that's how mkroot.sh can store all those kernel configs and still be 250
lines. (Yes, I cheated a bit with the long lines. I should go in and add line
breaks and just take the line count hit...)

> I also appreciate the references to the socio-historical
> (socio-hysterical? :) context of software development. I found some of
> your conferences interesting for this reason, as well as the technical
> content.

Thanks. Long ago I had more time to do computer history reasearch and writeups:

  https://landley.net/notes-2010.html#17-07-2010

  https://landley.net/notes-2010.html#19-07-2010

> Riccardo

Rob