[Toybox] Shell Compatibility Reports from Oils - ~800 tests passing

[Rob Landley]

On 6/28/25 21:42, Andy Chu wrote:
>> (Coding is easy. Figuring out what it's supposed to DO has always been
>> the hard part.)
> 
> I agree!
> 
> And to repeat, that's what I'm saying OSH has already done.  It's the
> most bash-compatible shell, by far -
> https://pages.oils.pub/spec-compat/2025-06-26/renamed-tmp/spec/compat/PASSING.html
> 
> So I think sharing some labor rather than producing 4 different shells
> is a good idea ... e.g. in particular OSH has some holes on the
> interactive side, and we need to test it building real distros.

I.E. "Stop working on your project, come work on my project".

> And we have a budget of 50 K euros for the next 6-12 months.  I
> realized that reducing test cases from real distro shell scripts
> (Alpine, Nix) is now a bottleneck.  That is, we have plenty of
> isolated tests, but it's better to PRIORITIZE those by what actually
> happens in the wild.

I have SO MANY TESTS, all of which I isolated to a minimal reproduction 
sequence but didn't bother to wire up into the automated regression test 
script because I have enough tests that don't pass already.

I mentioned needing to dig through my blog, I've also got multiple text 
files full of boring stuff like:

$ thingy ()
 > ^C
$ echo (
bash: syntax error near unexpected token `newline'
$ function
bash: syntax error near unexpected token `newline'
$ function name () {
 > }
bash: syntax error near unexpected token `}'
$ function name () { ;}
bash: syntax error near unexpected token `;'
$ if ; then; echo hello; fi
bash: syntax error near unexpected token `;'
$ func (); { echo hello; }
bash: syntax error near unexpected token `;'
$ if true |
 > then
bash: syntax error
echo one && if true; then echo hello; fi
one
hello


Basically, if I needed to run it from the command line to figure out 
what bash would do with a given input, I cut and pasted it to a file. If 
I needed to know, the test suite needs to know. And then I ask Chet 
about some of the really weird ones that make no sense like "why does 
if; syntax error but if\n prompts for line continuation", and sometimes 
he "fixes" a corner case, and then TEST_HOST has version skew and I step 
away from the keyboard...

But yay more tests. I'm always happy to find more tests. I'm trying to 
organize the tests I have into some sort of coherent order, and I need 
to do a close reading of both posix-2024 and the bash man page at some 
point to ensure that every concrete statement they make has a test, but 
that's... later. :)

It would be great if there was something like the old automated posix 
test suite that stopped mattering because they charged an arm and a leg 
for it so the Linux community went "up yours" and everybody stopped 
caring what Posix said after Linux's famous "212% annually growth" in 
1998 because all the Java devs flooeded over (which I wrote about a bit 
in http://www.catb.org/esr/writings/taoup/html/ch02s03.html and probably 
more in a blog entry)...

Sigh, I'm pretty sure Linux Journal had an article on Linus's rejection 
of the offer of explicit Posix conformance testing for Linux, published 
back in the 1990s when it was happening, but Google no longer indexes 
their archives. https://liw.fi/linux-news/issue03/ was back before Posix 
noticed Linux existed, and the later interviews don't mention posix 
because it was irrelevant. By the time Red Hat had its IPO, FIPS-151-2 
had been withdrawn, ala 
https://www.opengroup.org/testing/fips/general_info.html and 
https://www.federalregister.gov/documents/2000/02/25/00-4512/announcing-approval-of-withdrawal-of-thirty-three-federal-information-processing-standards-fips

Anyway, tests are nice but rigid tests aren't nice. Testing is a whole 
discipline of its own, and mostly what you're testing for is that your 
behavior is consistent, intentional, and expected. "Right" is... 
complicated.

> Andy

Rob