[Toybox] Any interest in patches to make the build process friendlier to hermetic builds?

[John Millikin]

On 2024-10-13 01:28, Rob Landley <rob at landley.net> wrote:
> I need to write a FAQ entry about scripts/prereq/build.sh and maybe a
> section of the README. It's regenerated each release by
> scripts/recreate-prereq.sh (yes I'm checking in a generated file) and
> the current documentation is the commit messages on the initial commit:

That's a very useful pointer, thank you! The `scripts/prereq/generated/*.h'
files were exactly what I was looking for.

> > My specific proposal is to convert some or all of the $SED processing into
> > C code and either put it in its own binary, or unify it with
> > `scripts/mkflags.c' / `scripts/config2help.c' / etc.
>
> I was actually looking to get away from those and move it _more_ towards
> sed. (And maybe awk now that we have one of those but I have to read
> through it and promote it out of pending first.)
>
> That said, I'm working on replacing kconfig/* into a new
> scripts/config.c so maybe it will inherit some of those functions, not
> sure yet...

Ah, in that case, would an acceptable middle ground be to move the sed
invocations into a separate `.sh' script that can be invoked directly?

For example, `scripts/make.sh' currently contains this code to produce
`generated/newtoys.h':

    if isnewer newtoys.h toys
    then
      # The multiplexer is the first element in the array
      echo "USE_TOYBOX(NEWTOY(toybox, 0, TOYFLAG_STAYROOT|TOYFLAG_NOHELP))" \
        > "$GENDIR"/newtoys.h
      # Sort rest by name for binary search (copy name to front, sort,
remove copy)
      $SED -n 's/^\(USE_[^(]*(.*TOY(\)\([^,]*\)\(,.*\)/\2 \1\2\3/p' toys/*/*.c \
        | sort -s -k 1,1 | $SED 's/[^ ]* //'  >> "$GENDIR"/newtoys.h
      [ $? -ne 0 ] && exit 1
    fi

It's difficult to get there without having the rest of `make.sh' tag along
(the hostcmp and environment probing), but if the code were adjusted to
something like this:

    # make-generated.sh
    gen_newtoys_h() {
      # The multiplexer is the first element in the array
      echo "USE_TOYBOX(NEWTOY(toybox, 0, TOYFLAG_STAYROOT|TOYFLAG_NOHELP))" \
        > "$GENDIR"/newtoys.h
      # Sort rest by name for binary search (copy name to front, sort,
remove copy)
      $SED -n 's/^\(USE_[^(]*(.*TOY(\)\([^,]*\)\(,.*\)/\2 \1\2\3/p' toys/*/*.c \
        | sort -s -k 1,1 | $SED 's/[^ ]* //'  >> "$GENDIR"/newtoys.h
      [ $? -ne 0 ] && exit 1
    }

    # scripts/make.sh
    source scripts/make-generated.sh
    # [...]
    if isnewer newtoys.h toys
    then
      gen_newtoys_h
    fi

This would let a hermetic build system handle the C compilation of the helper
tools, then call into `make-generated.sh' for the sedding.

> Alas the one thing the build still needs is /bin/bash because toysh
> isn't quite ready yet. I'm working on that too, but this year's kind of
> gotten away from me. (I sold my house and moved, my wife graduated and
> got a full time job, I went back to work for the j-core guys...)

I did some light testing and found that the generated code portions of
`scripts/make.sh' are mostly portable. There were two minor Bash-isms that
were easy to replace with POSIX equivalents, and I could successfully run the
build using either dash-0.5.12 or mksh-R59c (which are both much easier to
build in an isolated chroot than Bash).

Do you have any interest in patches to make `scripts/make.sh' (and/or an
extracted `make-generated.sh') POSIX-compatible-er?