[Toybox] [PATCH] Generate all build files with single program.

[Rob Landley]

On 03/18/2012 04:17 PM, Georgi Chorbadzhiyski wrote:
> On 3/18/12 6:55 PM, Rob Landley wrote:
>> On 03/18/2012 11:14 AM, Georgi Chorbadzhiyski wrote:
>>> On 3/17/12 4:57 AM, Rob Landley wrote:
>>>> On 03/16/2012 05:05 PM, Georgi Chorbadzhiyski wrote:
>>>>> +void generate_headers(int argc, char **argv) {
>>>>> +    // Parse toys
>>>>> +    strbuf_add(&newtoys_h, "NEWTOY(toybox, NULL, 0)\n");
>>>>> +    strbuf_add(&globals_union, "extern union global_union {\n");
>>>>
>>>> It matters that these be in alphabetical order (for the binary search),
>>>> and we can't depend on the filesystem doing it for us.  Where are you
>>>> sorting this?
>>>
>>> I didn't sort them. I depend on the shell doing the right thing. The
>>> program
>>> is called like this generated/genconf toys/*.c so it should be OK. Of
>>> course
>>> the parameters can be sorted but it seemed unnecessary since the
>>> shell is
>>> doing that for us.
>>
>> Actually the shell isn't.  The filesystem is.  (I've hit this one
>> before, it's not reliable.)
>>
>> Also, sorting the .c files isn't sufficient, you have to sort the
>> _output_ array, because one .c file can produce more than one array
>> entry (sh and toysh, netcat and nc...)
> 
> I'll send an updated patch tomorrow.

The C command I was working on had several goals:

1) Remember when Daniel Walter did groups and whoami as reskins of id?
The current infrastructure supports aliases (two commands with the same
behavior), but does _not_ support two commands in the same file with
different main() functions and help text.

2) I'm trying to add the nbd-client.c I wrote back in 2010, and with
config NBD_CLIENT the build is trying to include "nbd.c".

3) I want to combine the help text and usage: lines, as explained here
earlier.

4) I'd like to be able to build individual commands without necessarily
going through the config infrastructure.  (That way I could build the
toybox sed/sort/shell commands to run the rest of the build even if the
host tools are strange, but it's also nice to be able to build a
non-swiss-army-knife version fo the binaries.  I did "make individual"
for busybox long ago for similar reasons.)

5) I was hoping the config-parsing infrastructure I made could also act
as the start of a kconfig replacement.  If the help text merging bits
have to understand the kconfig language thoroughly enough to parse "XXX
&& YYY" style dependencies (and without dependency tracking the rules on
what's an option to a command and what's a new command get REALLY
brittle in the presence of a single file with multiple commands that can
be named "pivot_root" and "newfs_msdos" and so on.)

Merging this patch means I abandoning the (admittedly unfinished) work
I've done towards those goals, which would be fine if it addressed any
of them, but it doesn't.

It replaces it with a build infrastructure I've read through twice and
still not quite wrapped my head around yet (your buffer infrastructure
is doing what again: seems kind a overkill...).  It brings back bad
associations to the old busybox "makedep" command.  Its primary goal
seems to be making the project compile faster, which isn't exactly a
priority for me. (The current build doesn't even support smp
compilation, that's lower-hanging fruit than rewriting the shell scripts
in C.)

If you'd done a patch that was _just_ changing the help text and leaving
the rest of it alone, I'd already have merged it and worried about
fixing up the rest I need later.  I am _very_ interested in making the
build dependency on python go away, since it screws up my main
real-world test case for toybox (the aboriginal linux build, followed by
the native linux from scratch build).

But this isn't that. This is a complete rewrite of all the scripts in C,
doing pretty much what the current ones do. The Linux kernel developers
talk about "small patches that do one thing", and have sometimes used a
"trail of breadcrumbs" analogy. (I think Al Viro had a good explanation,
but haven't got net access right now.)  The problem with a giant
take-it-or-leave-it lump that re-architects the an existing subsystem in
one go is that there's nothing you can bisect through if something
breaks later. If you try to track down a bug, debugging hits the
megapatch and then stops.  You can't do the "it used to work, now it
doesn't, here's what changed" thing because _everything_ changed. The
fact it used to work is irrelevant.

That's not necessarily a showstopper, but it is a downside.

The build system has a design issues, as mentioned above.  It doesn't
currently do what I want it to do.  But this doesn't address any design
issues other than "the python dependency is inconvenient". And I'm not
sure the new design is an improvement: it manages to be a code
consolidation without being a code cleanup.

If we're going to redo how the build works and replace scripts/*.sh with
a single C file, I'd move it to the top level directory and call it
something like "generate.c" (since it creates generated/*), and then
clean out the scripts directory and possibly remove it.  Move the
remaining stub script text back into the Makefile.  End result: less
complexity. (I'd also want the generate.c file to be something I can
easily understand, which the current one isn't, but that could easily be
my fault.  Bott times I read it I was pretty tired at the end of a long
day.)

By the way, the reason I keep getting hung up on the backlog is due to
architectural issues, like "three different dirtree() variants", "should
whoami be a wrapper or a separate command", and "is mentioning any
feature test macros for the headers a good idea or should I just have a
policy of never #defining _any_ of them and declaring build environments
that require them broken".

I also have pending cleanups: I haven't been able to add the extra ls
options so aboriginal can use it, or add -f to tail.  Let alone doing
sed, mount, or converting nbd-client.c...

I haven't been able to focus on this because there's been an endless
stream of new code to review, introducing more architectural issues.  I
am trying to keep complexity down, which means that the new pieces I add
to the puzzle have to be _right_.  I feel really _bad_ about not merging
the code you're submitting, but I also have to look through Daniel
Walker's mode parser...

Rob
-- 
GNU/Linux isn't: Linux=GPLv2, GNU=GPLv3+, they can't share code.
Either it's "mere aggregation", or a license violation.  Pick one.

 1332164116.0