[Toybox] Shell tests for reference.

[Rob Landley]

On 12/17/19 12:10 PM, enh wrote:
> this sounds like a good time to mention what i've been up to with
> testing. you may recall i've talked about the mksh test suite before,
> and that -- despite being a thousand lines of perl -- that what they
> had was the least worst solution i'd seen for this problem?
> 
>   https://cs.android.com/android/platform/superproject/+/master:external/mksh/src/check.t

If that's a file format you prefer, sure?

But this doesn't seem to address interlacing. Testing line continuations boils
down to "wait for $, emit stuff and a newline, wait for >, emit next blob". The
$ vs > prompt (or having run stuff) tells out whether or not it recognized the
need for more data given what it had digested. (And then there's the "some
output on stdout, some on stderr" issue, although I have yet to come up with a
case where I care _what_ the output on stderr is, it's always just that it
produced some so far...)

> here's all the options mksh needed for testing mksh:
> 
>   https://cs.android.com/android/platform/superproject/+/master:external/mksh/src/check.pl;l=50

A thousand lines of perl.

> i've recently reimplemented an extended subset of that for easier
> testing of Android command-line tools:
> 
>   https://cs.android.com/android/platform/superproject/+/master:system/core/cli-test/?q=cli-test

In C++.

> here's the example unzip tests, which i think are quite a bit more readable:
> 
>   https://cs.android.com/android/platform/superproject/+/master:system/core/libziparchive/cli-tests/unzip.test

If that's a better file format, we can move towards something like that. The
data representation for this sort of thing is the hard part, the plumbing
implementing it is replacable.

> the implementation is pretty minimal (and significantly shorter than
> the Perl original), but the tests are actually surprisingly readable
> and expressive. (i've been particularly pleased with the way `after:`
> subsumes a lot of different "does file f exist?"/"is d a directory?"
> type of stuff.)
> 
> i'll doubtless need some regex matching at some point, but just the
> freedom from quoting is worth the entry price. as is being able to say
> one thing per line, rather than trying to cram every assertion into
> one line. (those have been the two most problematic parts of the
> toybox tests, i think.)

It wasn't really designed to scale. I was pondering something with HERE
documents, or perhaps:

IN="one
two
three
"
OUT="one
two
three
"

testing "" "blah" "$IN" "$OUT" "" ""

But I've mentioned before that the test suite could easily eat a year of going
after corner cases with flashlight and a toothbrush.

> the "Future Directions" in the cli-test
> README.md lists some of the more obvious gaps, but none of them are
> hard to add if/when i actually need them, and what i have is already
> pretty useful.

I've got _categories_ of infrastructure I need to add to the test suite. For
example, ls needs a pty master/slave driver if you're going to test -x and -c
properly, and ps needs one to test its word wrapping, as do top and watch, vi,
shell interactive command editing/history...

But right now, I'm trying to use the holidays to get toysh usable enough it
isn't blocking mkroot anymore.

Rob