[Toybox] Test design issue.

[Rob Landley]

On 04/25/2017 12:28 PM, enh wrote:
> note that the existing SKIP_HOST mechanism is very error prone. iirc
> none of the sed tests actually run because one near the top needs
> SKIP_HOST and it never gets unset.

I thought that "VARIABLE=value command" worked the same for shell
functions as it does for external commands? Hmmm...

$ make test_sed > one.txt
$ TEST_HOST=1 make test_sed > two.txt
$ diff -u one.txt two.txt | grep '^[+-]'
...
-PASS: sed sed - - twice
-PASS: sed match \n literal n
-PASS: sed c empty continuation
-PASS: sed 'y/a\bc/de\f/'
+FAIL: sed 'y/a\bc/de\f/'
-PASS: sed skip start of range
+FAIL: sed skip start of range

The first three are the ones we skip, the second the host version is
failing but the target version isn't so possibly they should be
annotated too. (Or at least have a comment...)

What shell are you using to run this? (Not bash?)

> SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
> distinguish "host tool is just plain broken" from "we deliberately chose
> to diverge [because ...]".

Which is why I added comments before them, ala:

# This segfaults ubuntu 12.04's sed. No really.
SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"

But I see I haven't kept up with that. Blah. (Presumably git annotate
should give a commit comment, and/or let me know what date to look for a
mailing list entry or blog post. I'll throw it on the todo list.)

> cases like sh.test are awkward too...

Yeah shellit() tries to compensate a bit but I haven't opened the toysh
can of worms yet. (Tim Bird asked me what it would take to do toysh, and
my estimate was 6 months full-time devoted to it, but he hasn't got a
budget to buy my time from my employer. Meanwhile $DAYJOB is focused on
winning solar/wind grid integration monitoring contracts. The market
opportunity for that is likely to explode in about 3 years and we need
to deploy stuff people can buy before then and be ready to scale up, so
getting any time to spend on toybox takes a bit of pushing...)

On the bright side I got https://github.com/landley/mkroot a bit more
fluffed out this week. Once I get all the musl-cross-make targets
booting, I can coordinate with Rich to get him to tag and ship an actual
release with binaries you can download (maybe he'll even put the
mcm-buildall.sh script in _his_ repo instead of mine).

Building kernels with mkroot needs toybox commit 9be140ebb29f so I'll
have to cut a new release here to coincide with the mkroot release, but
it's coming up on time for that anyway...

> exponentiation is a bash extension
> to the more widespread arithmetic extension (which is all the test
> tests).

Oh toysh is going to need quite possibly a thousand lines of tests. The
tests I have _now_ are just ones that I had lying around because I was
poking at something that was doing this...

I thought it was bc/dc but:

  $ dc "1+2"
  dc: Could not open file 1+2
  $ bc "1+2"
  File 1+2 is unavailable.

Those are both _way_ too stupid to use as a quick and dirty calculator
from the command line. (I usually do echo $((1+2)) when I don't need
floating point, and fire up python when I do. But there should be a
better way, possibly I should have toysh $(()) support floating point?
Hmmm...)

> so although it's "sh.test", it explicitly invokes "bash".

Because toysh should provide the bash alias along with bash functionality.

Ubuntu's epically stupid https://wiki.ubuntu.com/DashAsBinSh broke the
#!/bin/sh symlink to point to a broken shell (seriously, at the time it
segfaulted and got signal handling wrong and...), so scripts explicitly
say #!/bin/bash instead. I intend to provide a reasonable bash
replacement that runs those scripts.

Bash was the first program linux ever ran (and running bash was the
reason Linus implemented system calls in his term program), so it was
the standard shell of Linux from 1991 to 2006. Ubuntu's stated reason
for changing this (speeding up init scripts) didn't work (they then
implemented upstart to get parallelism), but they never admitted the
mistake and reverted it. They said that changing the #! at the start of
each init script was too intrusive a change, so they broke the kernel
build instead.

That's why all my scripts say #!/bin/bash and not #!/bin/sh. I need to
replace _that_.

> and if you "fix" it by running sh, bash-as-sh fails too. for now i just have an
> explicit case to skip sh.test in my Android toybox test runner, but in
> an ideal world it would be more fine-grained than that.

My infrastructure checks .config and should skip tests for commands that
are disabled?

Alas, I don't have a "pending" for the tests directory. In an ideal
world I'd be making progress fast enough this wouldn't be an issue. In
practice we're coming up on the one year anniversary of $DAYJOB's
"funding knothole" where we've been in perpetual crisis mode ever since.
(It's compelling technology I really want to see ship, but the pay sucks
and it's eating my life. Doing what I can here...)

In the meantime, I can add a check for "bash" in the $PATH and skip if
not". Does that help?

> here though, is there a strong reason to prefer \e? otherwise why not
> just do what the host does in this case?

Because "man echo" says \e and my escape/unescape plumbing is
symmetrical. I'd have to special case it to _not_ understand this.

I suppose I can also have the test run the output through sed to replace
"\e" with "\033" and then it'd except either?

Rob