[Toybox] Kernighan's awk repo

[enh]

On Thu, Jul 21, 2016 at 1:46 AM, Rob Landley <rob at landley.net> wrote:
>
>
> On 07/20/2016 11:20 PM, Andy Chu wrote:
>>> yeah, i'd like to see asan support in toybox's makefile; i'm starting
>>> to get a small pile of bug reports from the folks running asan Android
>>> system images, but it's going to be inconvenient to have to deal with
>>> them on the device rather than just on the host.
>>
>> How are they generating test input?  When you say "folks running", is
>> that an automated or manual process?  Are the bug reports public, and
>> are they in toybox or elsewhere on the system image?
>>
>> In case it got lost, these patches I sent out in March added ASAN
>> support to Toybox's Makefile, as well as scripts for running tests
>> under the ASAN-instrumented binaries (and other LLVM sanitizers)
>>
>> http://lists.landley.net/pipermail/toybox-landley.net/2016-March/008147.html
>>
>> I think Rob didn't agree with it being a big lump, which is some what
>> understandable.
>
> More that my review got a half hour for work in then the window got
> buried then got lost in a crash, as usual. (My workflow has a certain
> aspect of "compost heap".)
>
> The above patch doesn't seem to understand the infrastructure that's
> there. For example, you can already set CFLAGS yourself from the command
> line. The file "configure" will supply a default value (-Wall -Wundef
> -Wno-char-subscripts) if you don't, but the build-important stuff
> (-ffunction-sections and such) got broken out into $OPTIMIZE. So adding
> new makefile targets to set different command line options is weird,
> nothing else in the makefile was setting compiler options before this
> and that was a design decision because different toolchains may need
> different flags.
>
> Adding a NOSTRIP target to the makefile: you can already set NOSTRIP=1
> as an environment variable and scripts/make.sh will skip calling strip,
> and even when it _does_ the original unstripped version goes in
> generated/unstripped.

(having both is really handy. Android does this too. i make use of it
all the time, for both projects.)

> You add a wrapper script, but it requires what
> sounds like really extensive infrastructure changes to the makefile...
>
> (In general complicating the makefile enough that you needed to add more
> than an 80x25 screen full of tree layout information doesn't seem
> particularly useful for a false positive generator I will never
> personally use.)
>
> People have sent me the output of false positive generators periodically
> for years. The first big one was this:
>
> http://lists.landley.net/pipermail/toybox-landley.net/2014-February/003280.html
>
> Which is page after page of "not a bug".
>
> And since then they keep coming, although I asked other people to look
> at the output before sending it to me to wee out the obviously useless hits:
>
> http://lists.landley.net/pipermail/toybox-landley.net/2014-August/006782.html
> http://lists.landley.net/pipermail/toybox-landley.net/2014-August/006775.html
> http://lists.landley.net/pipermail/toybox-landley.net/2014-October/003646.html
>
> While they do find stuff, the quality of the bug reports tends to be
> significantly lower than average, and what they fix tends to be things
> like "assigning to a variable that never gets used", ala:
>
> http://lists.landley.net/pipermail/toybox-landley.net/2015-September/004428.html

asan doesn't produce false positives. it's run-time checks, and if it
says something just went wrong, it actually did.

the closest it gets is where you _are_ reading uninitialized data but
you happen to not surface that to the user. (toybox ls had this bug.)
but that's still a bad idea, and not fixing bad style like that (which
you did fix) means you can't use asan to find all the other bugs. that
would definitely be throwing the baby out with the bathwater. i think
it's an open question how security-critical we should consider the
various things implemented by toybox. my personal opinion is toward
the "not at all" end of the spectrum, but i _do_ care about /bin being
reliable, and asan helps correctness. even if there's no reason for
users to panic when ps segfaults or shows incorrect data, it's still
annoying for developers.

security (lack of), particularly memory safety (lack of) is the
asteroid that will kill a lot of our beloved C dinosaurs (such as the
Linux kernel). tools like asan and kasan buy Barney and friends some
extra time.

(unlike valgrind, the asan overhead is low enough that you can often
actually use asan-sanitized binaries for real work. mileage may vary
on a netbook :-) )

>> In my defense, if something is in good shape, I will
>> be very careful and make the minimum changes.  If something is in bad
>> shape, as the toybox tests are, I don't feel bad about rewriting it --
>> it deserves it.
>
> The reason many toybox tests are in bad share (rather than merely
> incomplete) is that for a long time when people asked me how they could
> help I pointed them at the test suite, and never had tests/pending
> because I didn't think I needed to. (I learned better.)
>
>> In other words, if I find 5 bugs in the process of adding a feature,
>> I'm going to take more liberties in re-architecting it... and those
>> patches fixed way more than 5 bugs in the test harness, some of which
>> I'm sure are still there.
>
> Wiring up C level static analysis tools is not the same as having a set
> of command line invocations with controlled input and output that
> reproduce known behavior from the tools. You are fundamentally changing
> what the test suite sets out to accomplish, and saying "there were bugs"
> seems orthogonal to that.
>
>> Andy
>
> Rob



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.