[Toybox] Toybox test image / fuzzing

[Rob Landley]

On 03/13/2016 02:13 PM, Samuel Holland wrote:
> On 03/13/2016 01:18 PM, Rob Landley wrote:
>> On 03/13/2016 03:34 AM, Andy Chu wrote:
>>> FWIW I think the test harness is missing a few concepts:
>>>
>>> - exit code
>>
>> blah; echo $?
>>
>>> - stderr
>>
>> 2>&1
> 
> I think the idea here was the importance of differentiating between
> stdout and stderr, and between text output and return code.

You can do this now. "2>&1 > /dev/null" gives you only stdout, or
2>file... There are many options.

> simple as having a separate output variable for each type of output.
> 
> Granted, it will usually be unambiguous as to the correctness of the
> program,

Yes, adding complexity to every test that isn't usually needed.

> but having the return code in the output string can be
> confusing to the human looking at the test case. Plus, why would you not
> want to verify the exit code for every test?

Because science is about reducing variables and isolating to test
specific things? Because "we can so clearly we should" is the gnu
approach to things? Because you're arguing for adding complexity to the
test suite to do things it can already do, and in many cases is already
doing? Because we've already got 5 required arguments for each test and
you're proposing adding a couple more, and clearly that's going to make
the test suite easier to maintain and encourage additional contributions?

I'm not saying it's _not_ a good extension, but you did ask. Complexity
is a cost, spend it wisely. You're saying each test should test more
things, which means potential false positives and more investigation
about why a test failed (and/or more complex reporting format).

Also, "the return code" implies none of the tests are pipelines, or
multi-stage "do thing && examine thing" (which _already_ fails if do
thing returned failure, and with the error_msg() stuff would have said
why to stderr already). Yesterday I was poking at mv tests which have a
lot of "mv one two && [ -e two ] && [ ! -e one ] && echo yes" sort of
constructs. What is "the exit code" from that?

> It's a lot of duplication
> to write "echo $?" in all of the test cases.

I don't. Sometimes I go "blah && yes" or "blah || yes", when the return
code is specifically what I'm testing, and sometimes checking the return
code and checking the output are two separate tests.

Keep in mind that error_msg() and friends produce output, and the tests
don't catch stderr by default but pass it through. If we catch stderr by
default and a test DOESN'T check it, then it's ignored instead of
visibile to the caller.

Also, keep in mind I want the host version to pass most of these tests
too, and if there are gratuitous differences in behavior I don't WANT
the test to fail based on something I don't care about and wasn't trying
to test. You're arguing for a tighter sieve with smaller holes when I've
already received a bunch of failing tests that were written against gnu
and never _tried_ against the toybox version, and failed for reasons
that aren't real failures.

> As for stdout/stderr, it helps make sure diagnostic messages are going
> to the right stream when not using the helper functions.

Right now diagnostic messages are visible in the output when running the
test. There shouldn't be any by default, when there is it's pretty
obvious because those lines aren't colored.

I'm all for improving the test suite, but "what I think the test suite
should be trying to do differs from what you think the test suite should
be trying to do, therefore I am right" is missing some steps.

Rob

- All syllogisms have three parts, therefore this is not a syllogism.

 1457901168.0