[Toybox] Kernighan's awk repo

[Andy Chu]

>> My patches already solved that problem.  That's why there is a test.sh script.
>
> Your top level script (there are no other top level scripts) calls make.
> None of my scripts ever call make. Make is a purely optional front-end
> to those scripts. Except for the kconfig stuff (which is slated for a
> rewrite), you should be able to build, install, and run the test suite
> on a system that hasn't even got make installed.

The reason for that is so you can just do:

$ git clone ...
$ ./test.sh ...   # depending on what flags you pass, it invokes
"make" on a different build target

So you don't have to remember all the dependencies yourself, which I
think is a good thing.  It's silly if you have to remember to build
the ASAN variant before running ASAN, build the coverage variant
before running coverage, etc.  It's error prone.

Admittedly it's a bit unusual for open source projects... it's a bit
like the Google build system, where tests know what binaries and
libraries they depend on.  If you don't have this, then you get weird
problems like trying to run coverage against ASAN binaries, which will
likely result in horrible error messages or mystifying runtime
behavior.

Likewise, PLEASE make sure the binaries have different names, like
toybox and toybox_asan, toybox_msan, etc.  Otherwise it's too easy to
use it incorrectly.  If you have objects that were built with ASAN
flags, but the makefile invokes the linker without ASAN flags, you
will get link errors.

I also hate superstitious "make clean"  ... the makefile should be set
up so that people aren't trained to do this.  If you have to make
clean after every ASAN run, it will not be used as much.  (I think I
fixed some stuff with respect to this in my patches but don't recall
exactly.)

In ideal world I want to do ./test.sh -asan and ./test.sh -perf
(against -O3 or -O4 binary) without worrying about builds... otherwise
it's easy to accidentally publish performance results with a debug
build, etc.

>> Like I said in the original messages, I wrote the patches once in make
>> and again in shell, and they were much better in shell.  (That is, the
>> test support is in shell, and the build support is in make, rather
>> than everything in make with .PHONY targets.)
>>
>> Specifically, the originals had asan_test_sed, etc. targets with
>> generated .mk code, and then I changed it to be an argument to a shell
>> script.
>
> The makefile I looked at still had targets for each, and I started to
> reverse engineer "what are you trying to accomplish" from what you'd
> implemented, so I started looking up what these various sanitizers were
> and how you'd use them, which is where I needed an installed version of
> llvm to start playing with, and the version in ubuntu's repo was too old

I'm using the Clang binaries here: http://llvm.org/releases/

You just download the Ubuntu ones and untar and run -- no installation
necessary.

> I'm aware it's a big bottleneck, yes. I regret this enormously, and it's
> why I added the "pending" directory so people had the option to route
> around me.
>
> But I want to be able to _support_ all this code. You see asan as
> important, I see code coverage and testing root stuff under a stable
> qemu environment as more important. There are large pending changes to
> this infrastructure (and semi-constant small ones that don't always get
> checked in promptly because seeing "+# cp -a rootfile userfile # should
> have no error" in git diff for tests/cp.test lets me know to add the
> test, whereas if I check that comment in I may not see it again any time
> soon. Local diffs act as yet another form of todo item for me.)

I don't really think it's sustainable to rely on one person to support
a 70K line or eventually 100K+ line project indefinitely, but that's
probably a long discussion that isn't high priority now...

My patches definitely made progress toward tests as root and QEMU...
but no need to argue that now either.

Andy