[Toybox] Test design issue.

[Rob Landley]

On 04/26/2017 04:56 PM, enh wrote:
> On Wed, Apr 26, 2017 at 2:41 PM, Rob Landley <rob at landley.net
> <mailto:rob at landley.net>> wrote:
> 
>     On 04/25/2017 12:28 PM, enh wrote:
>     > note that the existing SKIP_HOST mechanism is very error prone. iirc
>     > none of the sed tests actually run because one near the top needs
>     > SKIP_HOST and it never gets unset.
> 
>     I thought that "VARIABLE=value command" worked the same for shell
>     functions as it does for external commands? Hmmm...
> 
>     $ make test_sed > one.txt
>     $ TEST_HOST=1 make test_sed > two.txt
>     $ diff -u one.txt two.txt | grep '^[+-]'
>     ...
>     -PASS: sed sed - - twice
>     -PASS: sed match \n literal n
>     -PASS: sed c empty continuation
>     -PASS: sed 'y/a\bc/de\f/'
>     +FAIL: sed 'y/a\bc/de\f/'
>     -PASS: sed skip start of range
>     +FAIL: sed skip start of range
> 
>     The first three are the ones we skip, the second the host version is
>     failing but the target version isn't so possibly they should be
>     annotated too. (Or at least have a comment...)
> 
>     What shell are you using to run this? (Not bash?)
> 
> hmm, maybe this got fixed since i last saw it?

Not that I recall, and commit log of sed.test isn't saying so either.

 with TEST_HOST=1 make
> test_sed i'm not seeing any "SKIP" output, whereas when i ran in to this
> i saw a whole wall of yellow "SKIP" lines.

Hmmm... I just tried "BLAH=42 echo hello; echo $BLAH" on the android M
shell on my phone and it didn't persistently set it, but I don't have an
O test environment lying around... (I really need to work out how to
build individual AOSP packages outside of AOSP. I'd like a static
version of mksh and such to compare things against. And a bionic
toolchain I can build toybox against on an ubuntu host...)

I don't recall fixing this here, and am not seeing anything obvious in
the logs, but that doesn't mean much. Remember my recent "I should teach
tests to ignore whitespace" and fixing ls -m output to space
differently? A year ago I did:

commit fcbf55ad565861e22e1cb297e6954b9d82f5094b
Author: Rob Landley <rob at landley.net>
Date:   Thu Jun 30 10:39:23 2016 -0500

    Add NOSPACE=1 to allow tests to pass with diff -b

Which is in news.html but not code.html. I should add a section
documenting the test suite...

One of the reasons there's a whole lotta "squirrel!" in my development
process is when I get a reproducible bug I tend to _stop_ and chase that
sucker. Reproducible bugs are gold, and tend to submerge again if you
let the environment they reproduce in drift. Submerge is NOT the same as
get fixed, it's like roaches. You see it, you kill it, then you spray
for it. Do not let it breed. So fixing reproducible bugs can take
priority of implementing features, even halfway through...

>     > SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
>     > distinguish "host tool is just plain broken" from "we deliberately chose
>     > to diverge [because ...]".
> 
>     Which is why I added comments before them, ala:
> 
>     # This segfaults ubuntu 12.04's sed. No really.
>     SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"
> 
>     But I see I haven't kept up with that. Blah. (Presumably git annotate
>     should give a commit comment, and/or let me know what date to look for a
>     mailing list entry or blog post. I'll throw it on the todo list.)
> 
> exactly: having the skip mechanism require a string would help keep us
> honest.

In theory you could do SKIP_HOST="this is why" now, it only cares that
it's not an empty string. It'd be a bit _more_ error prone though
because you'd have to do:

  SKIP_HOST="Tea not hot enough to drive atomic vector plotter" \
  testing "improbability drive" "whale" "petunias" "" ""

And if you miss the \ at the end of the line you get the trailing
SKIP_HOST again. (That said, you'd presumably _notice_ that rather a
lot. But only if you remember to TEST_HOST as well as target. :)

If I get the test suite in a reasonable shape, I can add a line to my
release.txt checklist, which is currently:

news.html - release notes
Update version in main.c
  - git commit main.c www/news.html
tag repo
  - git tag $VER
  - git push --tags
source tarball
  git archive --prefix=toybox-$VER/ $VER | gzip -9 > toybox-$VER.tar.gz
  scp toybox-$VER.tar.gz landley.net:
  scp www/news.html landley.net:
binaries
  cd ../clean
  git pull ../toy3
  ../relbuild.sh
update help.html
  ./toybox help -ah > www/help.html
  scp www/help.html landley.net:landley.net/toybox/
scripts/mkstatus.py -> status.html
  - scp www/status.gen landley.net:landley.net/toybox/
roadmap.html
code.html
Inform mailing list

This _used_ to have "build linux from scratch under aboriginal linux" on
all targets, but aboriginal fell by the wayside because
(http://lists.landley.net/pipermail/aboriginal-landley.net/2016-May/002567.html)
and I'm finally doing mkroot to replace it now...

(You may have noticed I'm a big believer in testing with real-world data. :)

>     Oh toysh is going to need quite possibly a thousand lines of tests. The
>     tests I have _now_ are just ones that I had lying around because I was
>     poking at something that was doing this...
> 
> 
> mksh has a thousand line perl script to run 13k lines of tests that look
> like

I am totally stealing that if you'll let me. :)

> name: xtrace-2
> description:
>         Check that "set -x" is off during PS4 expansion
> stdin:
>         f() {
>                 print -n "(f1:$-)"
>                 set -x
>                 print -n "(f2:$-)"
>         }
>         PS4='[(p:$-)$(f)] '
>         print "(o0:$-)"
>         set -x -o inherit-xtrace
>         print "(o1:$-)"
>         set +x
>         print "(o2:$-)"
> expected-stdout:
>         (o0:sh)
>         (o1:shx)
>         (o2:sh)
> expected-stderr:
>         [(p:sh)(f1:sh)(f2:sh)] print '(o1:shx)'
>         [(p:sh)(f1:sh)(f2:sh)] set +x

Given the difference in file formats (mine has a lot of \n in it), i
expect my ballkpark of a thousand tests and your 13k lines may be a
similar amount of tests. :)


> 
>     > and if you "fix" it by running sh, bash-as-sh fails too. for now i just have an
>     > explicit case to skip sh.test in my Android toybox test runner, but in
>     > an ideal world it would be more fine-grained than that.
> 
>     My infrastructure checks .config and should skip tests for commands that
>     are disabled?
> 
> huh. i hadn't thought of that (though i had thought of its moral
> equivalent: checking to see whether the tool in question is a symlink to
> `toybox`). that's an option, though i'm potentially still interested in
> running the toybox tests against a non-toybox implementation. (but maybe
> my runner should check against .config and explicitly say when we're not
> running toybox.)

I think TEST_HOST skips the .config check already. (Haven't looked in a
while.)

Properly doing the test suite has been on my todo list for a while as
one of the Big Projects between here and the 1.0 release. The tests I
have there are the tests I ran during development (or when fixing a bug)
that I want to memorialize, but that's not the same as scrutinizing the
spec, man page, help text, and code to come up with full coverage.

>     In the meantime, I can add a check for "bash" in the $PATH and skip if
>     not". Does that help?
>  
> if you hadn't committed it already i'd say "don't worry about it":
> there's not enough in sh.test at the moment to worry about. what i
> _ought_ to be doing for shell testing is building a perl binary to run
> the mksh suite, but the thought of building perl in 2017 really
> stretches my pragmatism to its limit!

Ouch. I've done that
(https://github.com/landley/control-images/blob/master/images/lfs-bootstrap/mnt/build/perl.sh)
but it's way stale and kinda sad and I've spent a lot more time porting
things _off_ of perl (such as the perl removal work I did for the linux
kernel build).

Point me at your shell tests and maybe I can port them to something that
doesn't require perl. I need proper toybox shell tests anyway, and
making toysh pass the test suite you're already using would be an
important step in implementing something you could potentially move to
in a future release. (It's almost a hard prerequisite...)

> i'm more interested in seeing the gzip/gunzip/zcat tests go in so we can
> start arguing about how to upstream my toybox rewrite of the toolbox
> zlib-based gzip/gunzip/zcat. (i did the port on monday, so i do have a
> toybox gzip/gunzip/zcat that passes the tests, but i didn't even try to
> integrate the half-finished gz code from toys/pending/compress.c.)

Eh, send me what you've got and I can do that. I'm going to try to
devote this weekend to it.

I _did_ have gunzip and zcat working fine at one point, I just screwed
it up since. (Somebody actually sent a bug report with fix to github,
but it was the wrong fix and I had a window open and my netbook battery
died and the todo item got shuffled down the list in the reboot because
"open window" trumps "need to read one of my ~5 active todo.txt files".)

>     > here though, is there a strong reason to prefer \e? otherwise why not
>     > just do what the host does in this case?
> 
>     Because "man echo" says \e and my escape/unescape plumbing is
>     symmetrical. I'd have to special case it to _not_ understand this.
> 
>     I suppose I can also have the test run the output through sed to replace
>     "\e" with "\033" and then it'd except either?

This todo item meanwhile is recorded via a local modification to my
tests/ls.test adding two comment lines, which aren't checked in but
which show up in "git diff". (I also added a line to code.html about the
test suite and NOSPACE=1, same trick.)

It's not so bad if I keep on top of them. And if I don't, the
mechanism(s) by which I record them probably don't help. :P

Rob