[Toybox] Any interest in patches to make the build process friendlier to hermetic builds?

[Rob Landley]

On 10/11/24 22:20, John Millikin wrote:
> Hello,
> 
> I'm currently investigating the use of toybox to provide POSIX shell builtins
> for a hermetic build environment (Bazel + remote build execution), with the
> goal of bootstrapping past Autoconf-style configuration scripts without a
> dependency on GNU coreutils (due to the circular dependency).
> 
> The structure of `scripts/make.sh' is causing me a bit of trouble because of
> how the generated files are produced by inline calls to sed. My current
> approach is to hardcode a minimal toybox build (including synthetic minimal
> `generated/*.h') that's just enough to run `make.sh' and bootstrap from there,
> but this is more than slightly unpleasant and I have concerns about the
> maintainability between toybox releases.

I need to write a FAQ entry about scripts/prereq/build.sh and maybe a 
section of the README. It's regenerated each release by 
scripts/recreate-prereq.sh (yes I'm checking in a generated file) and 
the current documentation is the commit messages on the initial commit:

https://landley.net/toybox/git/commit/d1acc6e88be5

And then the update where I got it working under MacOS (_without_ homebrew):

https://landley.net/toybox/git/commit/3bbc31c78b41

> If I were to prepare patches to refactor how the toybox build produces
> `generated/*.h', would there be any interest from upstream in
> reviewing / merging them?

I'm interested, but the approach I took was to have toybox provide its 
own prerequisites.

Also, scripts/make.sh currently line 247 stops the build after header 
generation but before actually compiling anything when NOBUILD=1 is set. 
There used to be dependency checking that wouldn't regenerate headers 
that were already there and newer than the corresponding stuff, but I 
simplified most of the header generation so it was faster to just always 
do it.

> My specific proposal is to convert some or all of the $SED processing into
> C code and either put it in its own binary, or unify it with
> `scripts/mkflags.c' / `scripts/config2help.c' / etc.

I was actually looking to get away from those and move it _more_ towards 
sed. (And maybe awk now that we have one of those but I have to read 
through it and promote it out of pending first.)

That said, I'm working on replacing kconfig/* into a new 
scripts/config.c so maybe it will inherit some of those functions, not 
sure yet...

> Then it's straightforward
> for hermetic build systems to compile that into a binary and execute it to
> produce `generated/*.h'. without having to get a working POSIX shell set up
> first (it being easier and lower-maintenance to synthesize the build process's
> inputs than its outputs).

Alas the one thing the build still needs is /bin/bash because toysh 
isn't quite ready yet. I'm working on that too, but this year's kind of 
gotten away from me. (I sold my house and moved, my wife graduated and 
got a full time job, I went back to work for the j-core guys...)

Rob