[Toybox] Testing: how to check return code

[Rob Landley]

On 10/30/19 11:34 AM, enh via Toybox wrote:
> as someone who likes to see the specific exit value (because it's less
> ambiguous when you see it in a failure than "yes" or "ok" or
> whatever)...

Yes but the question then becomes what you're testing for. Is "returns 37 in
this case" part of the spec? (Where is that documented? It's not in posix, does
the test suite _become_ the documentation fo rthat?)

I also try not to test _specific_ error messages because toybox does "bad $INFO"
wherever possible (because non-english speakers), whereas the gnu stuff...

  http://landley.net/notes-2019.html#16-08-2019

(This is a disagreement between Elliott and myself: he wants distinct error
messages for each case, and yeah he's got a point...)

> i suspect the question you have to ask yourself here is "when does
> that $? get evaluated?".
> 
> as a stronger hint, "do you get different results if you use \$? or if
> you use 'quoting' instead of "quoting"?"
> 
> this, of course, is the argument in favor of || and && --- you might
> have to think more to understand a _failure_ but you don't have to
> think much when you're writing it. $? is the opposite.

The other advantage is to avoid testing a detail that may not matter, and thus
if you throw busybox or a BSD implementation through the same tests it may
gratuitously fail for a reason that doesn't actually impact anyone. (TEST_HOST
runs on what host?)

> (some tests will always need $? --- the ones that test specific exit
> values like 126 vs 127 or whatever.)

Signal exits are 128+signal, which implies that the range of exit codes you can
return to userspace, while represented as an int, should actual be unsigned 7
bits in any normal case.

What are the ones that test just _below_ 128? (At the high end of that range?)
We're at the 50th anniversary of Unix this year and this is documented _nowhere_
that I'm aware of. (A couple commands have specific exit codes in their man
page, but you wanted TOYFLAG_ARGFAIL(2) in things like sort, env, and chroot
where the man page doesn't mention it. And the timeout error code shenanigans
lying about which signal we died with seem an awful lot like a historical bug
being carefully preserved...)

Rob