[Toybox] Shell Compatibility Reports from Oils - ~800 tests passing

[Rob Landley]

On 7/8/25 14:32, enh wrote:
> On Tue, Jul 1, 2025 at 1:52 PM Rob Landley <rob at landley.net> wrote:
>> That's not exactly how AOSP upstream does it, but was provably
>> equivalent. :)

Ahem, how LFS upstream did it...

>> separate issue. The above phone (not even one the cheap chinese ones) is
>> faster than my laptop and I do real work on it. (Although I did upgrade
>> my laptop's ram to 16tb, because I could.)

Ahem, 16gb.

(You can tell how sleep deprived I am sometimes. But at least I wasn't 
saying busybox when I meant toybox and vice versa...)

>>>>> And between that, c11's __has_include(), and features.h you can
>>>>
>>>> it's been implemented by clang/gcc for a long time, but if you're
>>>> going to be a standards lawyer about it, __has_include() is c++17 and
>>>> c23.
>>
>> Sigh. Did one of the standards finally include gcc's empty ? : middle
>> argument? (Which I use all the time...)
> 
> c23 does not include this, and i don't think i've seen a proposal to
> standardize it. +Alejandro Colomar pays more attention to wg14 though,
> so he might know better...

This is the problem I have with standards. In the GOOD ones things that 
seem really obvious aren't included (posix still hasn't got mount and 
refuses to acknowledge that removing "tar" was a mistake), and the bad 
ones are full of things that clearly shouldn't be there. (LSB died 
because Red Hat paying the Linux Foundation to mandate RPM cost it every 
non-RPM distro from Debian to Arch.)

I first noticed the Linux kernel depending on "a ? : b" over 20 years 
ago (and it wasn't new then, that's just when I noticed it), in part 
because other compiler projects called it out as something they had to 
add to get their non-gcc compiler to build and run Linux. (Generally one 
of the easier things to add since it had just calculated the value and 
had it right there, but still.)

In 2004 tccboot built linux 2.4.26 with tinycc 
(https://bellard.org/tcc/tccboot_readme.html), Intel's ICC built Linux 
somewhere around the same time (I miss when Google could find stuff: 
https://lwn.net/Articles/320795/ is from 2009 but the comments are full 
of people saying this is 5 years old, and I mentioned it as one of the 
three known-to-work compilers in my 2008 BOF talk from ottawa linux 
symposium a year earlier and I _think_ ICC got Linux running before 
tinycc?), and LLVM built a working Linux kernel 2.6.36 at least by 2010 
(https://lwn.net/Articles/411654).

All of that was before C11 came out in 2011, which is probably why I 
assumed they were _aware_ of "a ? : b" existing and being necessary to 
build Linux. Let alone C23.

This isn't even the kind of falling between the cracks of the "how do I 
ask a FILE * how much readahead data is waiting in its buffer so I can 
grab it without sucking more input from a potentially unseekable 
filehandle I'm going to hand off to a syscall" problem, where posix said 
"we don't have the concept of FILE * we're only filehandles" and the C 
standards committee said "we don't have the concept of filehandles we're 
only FILE *" and each side pointed fingers at the other to produce 
conversion functions. Which is the main reason I haven't tackled "read" 
in toysh yet:

$ echo $'one\ntwo\nthree\nfour\nfive' > seekable
$ { read i; echo a=$i; head -n 2; } < seekable
a=one
two
three
$ echo $'one\ntwo\nthree\nfour\nfive' | { read i; echo b=$i; head -n 2;}
b=one
two
three

If getline() sucks extra data into the FILE * buffer but I can't 
fseek(ftell()) to rewind the underlying filehandle before doing an 
exec("head") because the input is a pipe, how does the child command NOT 
skip starting input? Bash is apparently getting this right, without 
doing single byte reads to implement "read". Alas strace output on bash 
is so thoroughly chaffed (thanks glibc) I haven't a clue how it's doing 
that. I think I _did_ ask Chet and he said to do the fseek(ftell) thing, 
except my notes say fseeko() and ftello() because large files... but how 
does that work on a PIPE? Did the kernel do something REALLY WEIRD when 
I wasn't looking? (Please tell me the parent process isn't SPINNING 
READING the FILE * and writing to a second pipe. Urgh, now I've gotta 
dig through my old notes and see where I left off on this, I thought 
"support doing it right for seekable and drop data for pipes" was the 
compromise bash was using but the test I just ran above contradicts 
that, which may be because I'm misremembering and may be because it's a 
newer bash version that changed its behavior...)

>> Sadly, standards remain a frame of reference to diverge from...

Grumble grumble...

For toybox the systems "real" enough to be worth supporting are 
Linux/Android and FreeBSD/MacOS, plus stuff like QNX has done its own 
support. But it doesn't matter how "posix" aix or solaris are (let alone 
the kind of "posix" support S/390 and Windows NT once had) if they can't 
keep up with the group. There is a de-facto standard going on, which 
would be really nice if it was explicit, but isn't. (And when you look 
closely the edges are all blurry judgement calls about "should" and 
shift over time because they're not written down and everybody has 
different opinions and implementing ps or route just sucks.)

The C compiler world is kind of the same, which is sad but not 
surprising. (Did you know https://github.com/PortableCC/pcc moved to 
github recently?) I've kinda been doing the same thing with busybox and 
toybox vs util-linux and friends (and Rich has been doing with musl): 
figure out what a critical mass of things actually use, support what you 
can't come up with good workarounds for, and eventually if your 
implementation gets widely used enough third parties start pushing sharp 
edge removals out into the larger ecosystem with a convincing "this is a 
big enough section of the potential userbase to care about".

Some years back I was eyeing man7.org as vaguely standard shaped, but it 
fell into the "include everything" trap SO hard that right now 
https://man7.org/linux/man-pages/dir_section_1.html reads along the top: 
abicompat (funky obscure gnuism), gitweb (not part of ANY "base os" 
definition), and pmdaoracle (for a proprietary database). The 
interesting commands are drowned out in noise, and what's in even the 
commands also often errs on the side of "kitchen sink".

As I said, "grumble". I would LOVE to conform to standards more, but 
*gestures wildly above*. And I would love to PRODUCE something 
standard-shaped, but that's a full time job just to do it wrong and a 
red queen's race to stay merely out of date.

Rob