[Toybox] [PATCH] Generate all build files with single program.

[Georgi Chorbadzhiyski]

On 19.3.2012 г. 15:35, Rob Landley wrote:
> The C command I was working on had several goals:
>
> 1) Remember when Daniel Walter did groups and whoami as reskins of id?
> The current infrastructure supports aliases (two commands with the same
> behavior), but does _not_ support two commands in the same file with
> different main() functions and help text.

I think the current scheme of using different files for different
commands is extra easy to understand. Adding two or more commands
in a single C file is only asking for trouble. If there is shared functionality
it can easily be put in lib/blah.{c,h} and used by both commands.

> 2) I'm trying to add the nbd-client.c I wrote back in 2010, and with
> config NBD_CLIENT the build is trying to include "nbd.c".

Using genconf.c and toys/nbd_client.c (the dash is problematic) results
in "working" nbd_client. Granted there should be a way for a command
to have dash in the name, so using "all-in" C file brings nothing
extra here (except that nbd_client.c builds and acts like a toy).

> 3) I want to combine the help text and usage: lines, as explained here
> earlier.

Could you elaborate more on that, I can't find anything in the ML from
the last month or so. I searched mails for "usage" and "help". It seems
I have missed that.

> 4) I'd like to be able to build individual commands without necessarily
> going through the config infrastructure.  (That way I could build the
> toybox sed/sort/shell commands to run the rest of the build even if the
> host tools are strange, but it's also nice to be able to build a
> non-swiss-army-knife version fo the binaries.  I did "make individual"
> for busybox long ago for similar reasons.)

It seems this depends only on generated/config.h. Should be easy to use
dummy config.h with only the needed command enabled and then copy resulting
toybox binary as generated/CMD.

> 5) I was hoping the config-parsing infrastructure I made could also act
> as the start of a kconfig replacement.  If the help text merging bits
> have to understand the kconfig language thoroughly enough to parse "XXX
> &&  YYY" style dependencies (and without dependency tracking the rules on
> what's an option to a command and what's a new command get REALLY
> brittle in the presence of a single file with multiple commands that can
> be named "pivot_root" and "newfs_msdos" and so on.)

Thats good but seeing how big kconfig is, it is no small task.

> Merging this patch means I abandoning the (admittedly unfinished) work
> I've done towards those goals, which would be fine if it addressed any
> of them, but it doesn't.
>
> It replaces it with a build infrastructure I've read through twice and
> still not quite wrapped my head around yet (your buffer infrastructure
> is doing what again: seems kind a overkill...).  It brings back badig.h
> associations to the old busybox "makedep" command.  Its primary goal
> seems to be making the project compile faster, which isn't exactly a
> priority for me. (The current build doesn't even support smp
> compilation, that's lower-hanging fruit than rewriting the shell scripts
> in C.)

My primary goal was to simplify the build. Admittedly genconf.c can
be smaller but since it does couple of things intermixed it probably looks
scarier that it really is. The buffer infrastructure is very simple auto
growing string buffer.

If the parsing for different functionalities is separated it'll be
very easy to follow.

getconf.c does this:
   1. Parse toys files and build generated/globals.h
   2. Parse toys files and build generated/newtoys.h
   3. Parse toys files and build generated/Config.in
   4. Parse Config.in and generated/Config.in and build generated/help.h
   5. Parse .config and build generated/config.h and generated/build_files

Version 2 that I have send should be easier to read:
https://github.com/gfto/toybox/commit/0a68d9cbdb77426d3383af92542e4ea0c62a102d#diff-3

> If you'd done a patch that was _just_ changing the help text and leaving
> the rest of it alone, I'd already have merged it and worried about
> fixing up the rest I need later.  I am _very_ interested in making the
> build dependency on python go away, since it screws up my main
> real-world test case for toybox (the aboriginal linux build, followed by
> the native linux from scratch build).

I can easily leave only this functionality but it will not simplify the
build, it'll just replace python dependency with small C program and as
long as a C program have to be made why don't we do the other things
that current mix of shell, sed, sort, tr, etc does. It can be done step
by step of course.

> But this isn't that. This is a complete rewrite of all the scripts in C,
> doing pretty much what the current ones do. The Linux kernel developers
> talk about "small patches that do one thing", and have sometimes used a
> "trail of breadcrumbs" analogy. (I think Al Viro had a good explanation,
> but haven't got net access right now.)  The problem with a giant
> take-it-or-leave-it lump that re-architects the an existing subsystem in
> one go is that there's nothing you can bisect through if something
> breaks later. If you try to track down a bug, debugging hits the
> megapatch and then stops.  You can't do the "it used to work, now it
> doesn't, here's what changed" thing because _everything_ changed. The
> fact it used to work is irrelevant.

I can replace each step in a different patch and if that is what will
help to get genconf to be merged I'll find the time to do it.

> That's not necessarily a showstopper, but it is a downside.
>
> The build system has a design issues, as mentioned above.  It doesn't
> currently do what I want it to do.  But this doesn't address any design
> issues other than "the python dependency is inconvenient". And I'm not
> sure the new design is an improvement: it manages to be a code
> consolidation without being a code cleanup.

The cleanup was the part where bunch of tools was replaced with one for
the specific need. A nice side effect was the build speedup (not important),
the ability to build toybox with just shell and C compiler.

> If we're going to redo how the build works and replace scripts/*.sh with
> a single C file, I'd move it to the top level directory and call it
> something like "generate.c" (since it creates generated/*), and then
> clean out the scripts directory and possibly remove it.  Move the
> remaining stub script text back into the Makefile.  End result: less
> complexity. (I'd also want the generate.c file to be something I can
> easily understand, which the current one isn't, but that could easily be
> my fault.  Bott times I read it I was pretty tired at the end of a long
> day.)

The naming is not a problem let it be generate.c. I left shell scripts
because I thought that they are there in order for build to not depend on
make. Put .config file in the root, run scripts/make.sh and you have toybox
binary.

> By the way, the reason I keep getting hung up on the backlog is due to
> architectural issues, like "three different dirtree() variants", "should
> whoami be a wrapper or a separate command", and "is mentioning any
> feature test macros for the headers a good idea or should I just have a
> policy of never #defining _any_ of them and declaring build environments
> that require them broken".

There a two ways here, merge and fix later (which will happen someday,
/the fix part/ possibly, may be not...), or sit on top of a pile and try
to merge it "when it is ready (tm)". It is up to you. I would merge everything
and cleanup later (or never :-) but that is just me.

> I also have pending cleanups: I haven't been able to add the extra ls
> options so aboriginal can use it, or add -f to tail.  Let alone doing
> sed, mount, or converting nbd-client.c...

IMHO you should write new code since this brings toybox closer 1.0
If we try to make every new toy close to perfect there will never be
enough time. And with new code there will be new issues with build system,
lib/, portability, etc. but the code will live in the tree and if possible
cleanups, feature additions, wish lists are documented perhaps somebody will
fix them. It doesn't have to be you.

> I haven't been able to focus on this because there's been an endless
> stream of new code to review, introducing more architectural issues.  I
> am trying to keep complexity down, which means that the new pieces I add
> to the puzzle have to be _right_.  I feel really _bad_ about not merging
> the code you're submitting, but I also have to look through Daniel
> Walker's mode parser...

I have slowed down now, the easy stuff is mostly done /and I'm good at
easy/, it would be great if I can do something to ease your burden.

-- 
Georgi Chorbadzhiyski
http://georgi.unixsol.org/

 1332369392.0