[Toybox] test flakiness sitrep

[Rob Landley]

On 8/12/19 3:33 PM, enh via Toybox wrote:
> last week i turned on the toybox tests in presubmit for changes to
> toybox in AOSP, and postsubmit otherwise. so we have a fair bit of
> data now (sadly no statistics because non-gtest tests aren't that well
> integrated yet).

Yay! Thanks for improving the test suite.

> bad news:
> * i can't merge my own find(1) fix because the test i added is so
> flaky i can't get it in. (but really, that's a feature, and should
> count as good news.)

Which find fix?

> good news:
> * very little flakiness so far. i've only seen a couple of tests (one
> cpio, one tar) failing, and both make me suspicious that there's still
> some bad behavior in dd somewhere.

It's in pending for a reason. :)

> (i haven't had time to investigate
> at all yet, but i can say that i haven't seen any failures running
> these tests on the desktop with GNU dd. so, yes, i also did this to
> myself.)

I have an ELC talk scheduled on the 22nd, and I'm trying to get at least an
excuse for toysh running by then.

Writing a new shell is...

  $ ((1+1
  > <2)); echo $?
  1
  $ ((echo hello)|cat)
  hello

Is (( a special token or is it two ) characters? The answer is:

  $ ((echo hello))
  bash: ((: echo hello: syntax error in expression (error token is "hello")
  $ ((echo hello) )
  hello

It's a special token that retroactively BECOMES two ( characters if the
parentheses count goes negative, but is matched by )) on the same level to say
whatever's in between is math.

MEANWHILE, THIS nonsense (in bash, not in my thing yet!):

  $ for((;
  > ;));do echo hello;done

BUT cursor up and hit enter, and you get infinite hello. The syntax checker
accepts both newline and whitespace but the execution behaves differently for
them, and when the history glues it back together you get the version that isn't
broken. Oh, and it's a newline in the middle there, you can have it before the
first ; and after the second ; and that's fine.

Oh, and ;; is a special token but AGAIN, in this special case:

  $ for((;;));do echo hello;done

It turns into two separate ; (presumably retroactively again).

I am adding tests to the giant test file and grimly implementing. Alas since the
syntax checking pass and the running pass are different (I'm sharing as much
code as I can between them but they fundamentally do different things), I have a
lot more syntax parsing than executing yet. :)

> but that's not bad, i reckon. and even the flake is helping find bugs
> in dd, it seems :-)

I'm unlikely to have spare cycles after getting the shell to a good point
because I need to go get another job and refill the bank account, but I agree dd
needs to be more loadbearing...

Rob