[Toybox] Kernighan's awk repo

[Rob Landley]

On 07/20/2016 11:20 PM, Andy Chu wrote:
>> yeah, i'd like to see asan support in toybox's makefile; i'm starting
>> to get a small pile of bug reports from the folks running asan Android
>> system images, but it's going to be inconvenient to have to deal with
>> them on the device rather than just on the host.
> 
> How are they generating test input?  When you say "folks running", is
> that an automated or manual process?  Are the bug reports public, and
> are they in toybox or elsewhere on the system image?
> 
> In case it got lost, these patches I sent out in March added ASAN
> support to Toybox's Makefile, as well as scripts for running tests
> under the ASAN-instrumented binaries (and other LLVM sanitizers)
> 
> http://lists.landley.net/pipermail/toybox-landley.net/2016-March/008147.html
> 
> I think Rob didn't agree with it being a big lump, which is some what
> understandable.

More that my review got a half hour for work in then the window got
buried then got lost in a crash, as usual. (My workflow has a certain
aspect of "compost heap".)

The above patch doesn't seem to understand the infrastructure that's
there. For example, you can already set CFLAGS yourself from the command
line. The file "configure" will supply a default value (-Wall -Wundef
-Wno-char-subscripts) if you don't, but the build-important stuff
(-ffunction-sections and such) got broken out into $OPTIMIZE. So adding
new makefile targets to set different command line options is weird,
nothing else in the makefile was setting compiler options before this
and that was a design decision because different toolchains may need
different flags.

Adding a NOSTRIP target to the makefile: you can already set NOSTRIP=1
as an environment variable and scripts/make.sh will skip calling strip,
and even when it _does_ the original unstripped version goes in
generated/unstripped. You add a wrapper script, but it requires what
sounds like really extensive infrastructure changes to the makefile...

(In general complicating the makefile enough that you needed to add more
than an 80x25 screen full of tree layout information doesn't seem
particularly useful for a false positive generator I will never
personally use.)

People have sent me the output of false positive generators periodically
for years. The first big one was this:

http://lists.landley.net/pipermail/toybox-landley.net/2014-February/003280.html

Which is page after page of "not a bug".

And since then they keep coming, although I asked other people to look
at the output before sending it to me to wee out the obviously useless hits:

http://lists.landley.net/pipermail/toybox-landley.net/2014-August/006782.html
http://lists.landley.net/pipermail/toybox-landley.net/2014-August/006775.html
http://lists.landley.net/pipermail/toybox-landley.net/2014-October/003646.html

While they do find stuff, the quality of the bug reports tends to be
significantly lower than average, and what they fix tends to be things
like "assigning to a variable that never gets used", ala:

http://lists.landley.net/pipermail/toybox-landley.net/2015-September/004428.html

> In my defense, if something is in good shape, I will
> be very careful and make the minimum changes.  If something is in bad
> shape, as the toybox tests are, I don't feel bad about rewriting it --
> it deserves it.

The reason many toybox tests are in bad share (rather than merely
incomplete) is that for a long time when people asked me how they could
help I pointed them at the test suite, and never had tests/pending
because I didn't think I needed to. (I learned better.)

> In other words, if I find 5 bugs in the process of adding a feature,
> I'm going to take more liberties in re-architecting it... and those
> patches fixed way more than 5 bugs in the test harness, some of which
> I'm sure are still there.

Wiring up C level static analysis tools is not the same as having a set
of command line invocations with controlled input and output that
reproduce known behavior from the tools. You are fundamentally changing
what the test suite sets out to accomplish, and saying "there were bugs"
seems orthogonal to that.

> Andy

Rob