[Toybox] avoiding testing shell builtins

[Rob Landley]

On 6/28/19 11:43 AM, enh wrote:
> On Fri, Jun 28, 2019 at 9:31 AM Rob Landley <rob at landley.net> wrote:
>>
>> On 6/27/19 1:10 PM, enh wrote:
>>> this also fixed the echo test failure. now at 68 passed, 11 failed
>>> (though that includes the split test failure that's really just "mksh
>>> doesn't support that bash extension").
>>
>> I'm actively working on a shell that _should_ support the bash extension (which
>> is why I used it in the tests), but that's got another month to go before it's
>> usable and you're not using it this year even if it's perfect, so...
> 
> yeah, sorry about that.

No plan survives contact with the real world. It's good to have users.

> i did wonder about adding a "# TODO: remove when toysh is done".

tests/sh.test needs a test for it, that should cover it.

And at a certain point, I need to dogfood this stuff. Not quite sure how that's
gonna work, since I need to stay current with what other interesting contexts
are doing to stay compatible and add new things people expect.

Then again, once I can get android working _as_ a development environment, I can
use it as such. (Really what I want is to build AOSP but disable 2/3 of it
that's not relevant to me. That's gonna take some surgery and a whole lot of
research. Ideally I'd prefer one of the existing "android x86" groups to do it
instead, but the ones I've looked at so far seem... politically frought.)

> this
> actually reminds me of the issues i have running tests on macOS: the
> problem there is that the /bin tools are so anemic that many of the
> tests fail just because of them.

In the non TEST_HOST version it should use the toybox commands for everything,
including toysh to run the shell plumbing. The _easy_ way to do this is "make
root" and then TEST_HOST inside it...

I also want to do a minimal bootstrap mode where make builds a static version of
toysh and toybox sed _without_ needing to use the host sed or fancy shell stuff,
and uses those to run the rest of the build (generated/build.sh is initial
doodling at that idea). But there's all sorts of chicken-and-egg problems with
that. (Which android gets around by shipping prebuilt static binaries, but I'm
thinking "build toybox is weird foreign environments where I don't even know
what processor you're using". Who <strike>watches the watchers</strike> builds
the prebuilts?)

But that's a bit down the road...

> i don't currently have a better idea
> than having a "known good" toybox and symlinks, and having that
> directory on my path (so the command being tested is coming from the
> new toybox, but i can still use toybox rather than the host for the
> commands used while testing --- stat being the most obvious example).

I need to move my testing infrastructure closer to what you posted yesterday
(it's simpler), it's just... juggling a lot of balls and both the documentation
and the test suite have been starved for cycles (along with the pending
directory)...

Working on it. :)

> i've not been thinking very hard about it though because macOS -- like
> "switching to toysh" -- is a problem for another year...

I'm thinking about it and working on it, but can only get so much done before
the end of August when I gotta find a new $DAYJOB. And what I'm _trying_ to get
done is a route command and enough of toysh so I can merge mkroot into "make
root" and get the test suite using it.

(To promote toybox it needs to run the toybox build, meaning scripts/*.sh and
whatever make is calling out to. I can keep adding stuff after that, but that's
the minimum for me.)

Rob