[Toybox] [PATCH] toysh: fix -Wuse-after-free

[Rob Landley]

On 3/24/24 10:15, Oliver Webb wrote:
> On Sunday, March 24th, 2024 at 04:09, Rob Landley <rob at landley.net> wrote:
>> On 3/24/24 01:00, Oliver Webb wrote:
> 
>> This isn't the hard part. To me, the hard part is wanting to share lib/.c code
>> with this new binary, which implies it would live in toys/example/.c, which
>> means in the NEW design it would be a normal command that's "default n"... and
>> maybe depends on TOYBOX_BUILD or some such? Except moving stuff from scripts/.c
>> to toys/.c is conceptually ugly. But if we're getting rid of the
>> subdirectories... Maybe make.sh needs to be able to build commands that DON'T
>> live in toys/ but then...
> 
> There is a chicken and egg problem with the build infrastructure and kconfig being a toy,

Yes, I know. That's why I've avoided it up until now.

> We need a .config file to build toys, and parsing the help text requires some kconfig
> parser, But we can't make a .config file until we have kconfig.

You don't need a .config file to build lib/*.c (policy, and why lib/lib.h is
separate from toys.h).

I'm talked before about doing packaged minimal headers to build "sed" and "sh"
standalone, as part of toybox airlock stuff. (Possibly the full airlock command
list needed to build toybox.) Ones that assume all the config probes failed and
$LIBRARIES is empty and so on.

The EASY way to do that is to have a scripts/shipped/generated with handcrafted
headers files, and then stick -I scripts/shipped at the start of $CFLAGS.

The hard way involves more cleanup so there are fewer header entry points, and I
could have a single "hairball.h".

The individual toys/*/*.c files only #include toys.h and generated/flags.h
(which could be a re-include of toys.h with a little #ifdef cleverness). The
rest are all included from toys.h and main.c.

The above #ifdef cleverness could wrap the generated/ includes in toys.h in a
__has_include("hairball.h") or similar, so I could provide a single replacement
file with the collated stuff I need for specific commands to build in a way that
assumes the host system has no brain, and then generated/build.sh it. The
problem is the includes are in two places: "generated/config.h" comes before
lib/portability.h (which comes before everything else so it can override
standard header #includes), and then the rest of the generated/*.h are after a
#define NEWTOY() and OLDTOY() so there's some reordering to do to combine them
all into one header.

(I don't think any of the generated/*.h files care about stuff in portability.h?
There would be various structs used before they were defined in
generated/globals.h if it got moved up, but that _should_ be ok? Nothing takes
the sizeof() them or similar that early. Eh, I should be able to work it out,
just haven't sat down to try yet. Far too many already open cans of worms...)

And then there's the #includes in main.c, the other half of the
declaration/definition pair for various global data: that needs newtoys.h,
help.h, and zhelp.h. One of which is already chopped out by a config option, the
second of which just needs a way to stub it to "", and which leaves newtoy.h to
address...

> The solution I thought of was to use the infrastructure that we will have to have to remove
> bash and gsed dependencies to build kconfig as a early step in the process.

No.

>  But then we will
> still need to extract the help text.

config TOYBOX_HELP
        bool "Help messages"
        default y
        help
          Include help text for each command.

You can configure help out entirely. (This isn't CONFIG_HELP the command, this
is "the help subsystem" in the toybox general settings menu.) This was intentional.

> Do you plan on not keeping 2 different kconfig parsers or moving scripts/*.c to toys/example

Look up at the first paragraph of mine you quoted in this email.

It's an open question, but stripping down a "cc -I scripts/prebuilts main.c
lib/*.c toys/*/{abc,def,ghi,jkl}.c"  build so it could provide commands with
nothing but a compiler would be a step towards that.

Modulo that "cc *.c" doesn't parallelize across processors because C++
developers took over compiler development about 2 years after the Core Duo hit
the market and brought SMP to the cheap retail mainstream, at which point making
compilers better rather than merely more complicated hit a sudden brick wall.

And thus even on my ancient 4x laptop:

$ time make clean defconfig toybox
...
real	0m16.170s
$ time generated/build.sh
...
real	0m27.474s

I don't want to significatly slow down the build by compiling prerequisites? In
theory:

$ time gcc -I . main.c lib/*.c -o blah
...
real	0m1.780s

(Yeah exits with a link error but that's not the point.)

And I mean yeah, 2 seconds, not that big a deal. But I'd prefer it be optional...

> (We could
> make toys/build tho, which would make slightly more sense),

See the recent discussion of removing the toys/subdirectories once pending's
cleared out, rather than adding more of them.

> There's circular dependency nightmares
> that would HAVE to be worked around at one point or another,

My horribly jetlagged "building the simplest possible linux system" talk started
out by talking about circular dependencies. It's circular dependencies all the
way down. This is why toybox (and busybox) try to have no external dependencies,
while providing everything* you need to build linux, a C library, and a
compiler. Because the 4 packages at the base of the 50 year old unix circle are
the kernel, libc, command line utilities, and compiler. That can rebuild itself
under itself from source, which SHOULD be enough to bootstrap up to arbitrary
complexity. (Except for the loons who went "let's rewrite everything in C++, and
then let's rewrite THAT in rust" so now your trusting trust auditing that
produced a working system has to cope with new incompatible ecosystems of binary
blob being imported. But I can't stop people from shooting themselves in the
foot and then "aiming higher"...)

* Try is a weasel word, I need to write a "make" at some point.

> a lot of stuff in the infrastructure relies
> on kconfig so making kconfig rely on the infrastructure would require
> bootstrapping it in a weird way,

That would be why I called it "the hard part" above? (Not unsolvable, pretty
sure I know how, but that's the bit that required design effort.)

> but there are also problems with that (main.c includes generated/*help.h
> files, toys.h includes generated/config.h),

I know, yes.

> So if the actual solution is to not make it a toy and build it with lib/*
> and toys.h to codeshare with lib/, that creates problems too...

Such as the "functions copied from lib/*.c" section of scripts/config2help.c
which I've been trying to sunset. Yes, I know.

> Does any of lib/ depend on those functions in toys.h that live in main,

xexec() calls toy_exec() out of main.c, "toys" is initialized by toy_init() in
main.c:

$ grep -w toys lib/*.c | wc -l
104

> grepping
> through, of course there are parts, which means more ifdefs if we wanna pry it off the infrastructure,

Hell no. #ifdefs belong in lib/portability.[ch] (Although I made an exception
for toys/*/getconf.c because Elliott made puppy eyes at me via email.)

Rob