[Toybox] avoiding testing shell builtins

[enh]

On Fri, Jun 28, 2019 at 11:38 AM Rob Landley <rob at landley.net> wrote:
>
> On 6/28/19 11:43 AM, enh wrote:
> > On Fri, Jun 28, 2019 at 9:31 AM Rob Landley <rob at landley.net> wrote:
> >>
> >> On 6/27/19 1:10 PM, enh wrote:
> >>> this also fixed the echo test failure. now at 68 passed, 11 failed
> >>> (though that includes the split test failure that's really just "mksh
> >>> doesn't support that bash extension").
> >>
> >> I'm actively working on a shell that _should_ support the bash extension (which
> >> is why I used it in the tests), but that's got another month to go before it's
> >> usable and you're not using it this year even if it's perfect, so...
> >
> > yeah, sorry about that.
>
> No plan survives contact with the real world. It's good to have users.
>
> > i did wonder about adding a "# TODO: remove when toysh is done".
>
> tests/sh.test needs a test for it, that should cover it.
>
> And at a certain point, I need to dogfood this stuff. Not quite sure how that's
> gonna work, since I need to stay current with what other interesting contexts
> are doing to stay compatible and add new things people expect.

that's part of why i'm trying to get to where we can pass all the
tests --- then i can flip the presubmit switch and not commit any
changes that break existing tests/add new failing tests. (which might
need some kind of EXPECTED_FAILURE support to check in "TODO" tests.)

not looking too bad right now... down to just these:

blkid
failures: output differences between toybox and “real” blkid we use
state: patch sent, but not sure what way you want to go there (whether
i should just ignore test failures from non-toybox implementations for
now).

du
failures: all tests fail, file system specific assumptions
state: still thinking about this one.

grep
failures: 2 tests fail, empty regex passed to regcomp()
state: tests broken on all BSD-derived regex implementations, not sure
whether to

ifconfig
failures: 2 tests fail “pointopoint”
state: patch sent to disable "pointopoint" tests because i don't have
a way to test pointopoint anywhere.

killall
failures: incorrect name-to-pid behavior
state: patch sent

tar
failures: (broken on host too: assumes /dev/loop0). broken on device
in several ways. one test exposes toybox dd issue.
state: patch sent (fixes most issues, disables dd-dependent test for toybox dd).

xargs
failues: “argument too long”
state: looking at this one today.

> Then again, once I can get android working _as_ a development environment, I can
> use it as such. (Really what I want is to build AOSP but disable 2/3 of it
> that's not relevant to me. That's gonna take some surgery and a whole lot of
> research. Ideally I'd prefer one of the existing "android x86" groups to do it
> instead, but the ones I've looked at so far seem... politically frought.)
>
> > this
> > actually reminds me of the issues i have running tests on macOS: the
> > problem there is that the /bin tools are so anemic that many of the
> > tests fail just because of them.
>
> In the non TEST_HOST version it should use the toybox commands for everything,
> including toysh to run the shell plumbing. The _easy_ way to do this is "make
> root" and then TEST_HOST inside it...
>
> I also want to do a minimal bootstrap mode where make builds a static version of
> toysh and toybox sed _without_ needing to use the host sed or fancy shell stuff,
> and uses those to run the rest of the build (generated/build.sh is initial
> doodling at that idea). But there's all sorts of chicken-and-egg problems with
> that. (Which android gets around by shipping prebuilt static binaries, but I'm
> thinking "build toybox is weird foreign environments where I don't even know
> what processor you're using". Who <strike>watches the watchers</strike> builds
> the prebuilts?)
>
> But that's a bit down the road...
>
> > i don't currently have a better idea
> > than having a "known good" toybox and symlinks, and having that
> > directory on my path (so the command being tested is coming from the
> > new toybox, but i can still use toybox rather than the host for the
> > commands used while testing --- stat being the most obvious example).
>
> I need to move my testing infrastructure closer to what you posted yesterday
> (it's simpler), it's just... juggling a lot of balls and both the documentation
> and the test suite have been starved for cycles (along with the pending
> directory)...
>
> Working on it. :)
>
> > i've not been thinking very hard about it though because macOS -- like
> > "switching to toysh" -- is a problem for another year...
>
> I'm thinking about it and working on it, but can only get so much done before
> the end of August when I gotta find a new $DAYJOB. And what I'm _trying_ to get
> done is a route command and enough of toysh so I can merge mkroot into "make
> root" and get the test suite using it.
>
> (To promote toybox it needs to run the toybox build, meaning scripts/*.sh and
> whatever make is calling out to. I can keep adding stuff after that, but that's
> the minimum for me.)
>
> Rob