[Toybox] test flakiness sitrep

[enh]

On Mon, Aug 12, 2019 at 8:52 PM Rob Landley <rob at landley.net> wrote:
>
> On 8/12/19 3:33 PM, enh via Toybox wrote:
> > last week i turned on the toybox tests in presubmit for changes to
> > toybox in AOSP, and postsubmit otherwise. so we have a fair bit of
> > data now (sadly no statistics because non-gtest tests aren't that well
> > integrated yet).
>
> Yay! Thanks for improving the test suite.
>
> > bad news:
> > * i can't merge my own find(1) fix because the test i added is so
> > flaky i can't get it in. (but really, that's a feature, and should
> > count as good news.)
>
> Which find fix?

the dangling symlinks one. you already merged the test fix, but i just
sent you another fix now i've run it on Android, not just my laptop
:-/

> > good news:
> > * very little flakiness so far. i've only seen a couple of tests (one
> > cpio, one tar) failing, and both make me suspicious that there's still
> > some bad behavior in dd somewhere.
>
> It's in pending for a reason. :)

yeah, i learned my lesson though --- nothing in pending is used for
the build :-)

> > (i haven't had time to investigate
> > at all yet, but i can say that i haven't seen any failures running
> > these tests on the desktop with GNU dd. so, yes, i also did this to
> > myself.)
>
> I have an ELC talk scheduled on the 22nd, and I'm trying to get at least an
> excuse for toysh running by then.
>
> Writing a new shell is...
>
>   $ ((1+1
>   > <2)); echo $?
>   1
>   $ ((echo hello)|cat)
>   hello
>
> Is (( a special token or is it two ) characters? The answer is:
>
>   $ ((echo hello))
>   bash: ((: echo hello: syntax error in expression (error token is "hello")
>   $ ((echo hello) )
>   hello
>
> It's a special token that retroactively BECOMES two ( characters if the
> parentheses count goes negative, but is matched by )) on the same level to say
> whatever's in between is math.
>
> MEANWHILE, THIS nonsense (in bash, not in my thing yet!):
>
>   $ for((;
>   > ;));do echo hello;done
>
> BUT cursor up and hit enter, and you get infinite hello. The syntax checker
> accepts both newline and whitespace but the execution behaves differently for
> them, and when the history glues it back together you get the version that isn't
> broken. Oh, and it's a newline in the middle there, you can have it before the
> first ; and after the second ; and that's fine.
>
> Oh, and ;; is a special token but AGAIN, in this special case:
>
>   $ for((;;));do echo hello;done
>
> It turns into two separate ; (presumably retroactively again).
>
> I am adding tests to the giant test file and grimly implementing. Alas since the
> syntax checking pass and the running pass are different (I'm sharing as much
> code as I can between them but they fundamentally do different things), I have a
> lot more syntax parsing than executing yet. :)

i don't know how you have the stomach for the shell... just thinking
about it makes me miss
[rc](http://doc.cat-v.org/plan_9/4th_edition/papers/rc) which -- like
so much of Plan 9 -- sadly wasn't better _enough_ to make a
difference. utf8 being the obvious exception.

i had a look at escape characters at the weekend, thinking i'd clean
up some of the duplication, mainly so i could remove the
even-more-duplication in my hexedit patch that's the reason you
haven't seen that yet. (but also so i could add \u without adding it
multiple times) ... and all i ended up with was a bunch of tests
showing how insane and different echo and printf are. the differences
between printf's direct interpretation of escapes and its %b
interpretation particularly made my day. i'll come back to that at
some point when my stomach's recovered...


> > but that's not bad, i reckon. and even the flake is helping find bugs
> > in dd, it seems :-)
>
> I'm unlikely to have spare cycles after getting the shell to a good point
> because I need to go get another job and refill the bank account, but I agree dd
> needs to be more loadbearing...
>
> Rob