[Toybox] is orthodoxy a strong value of toybox?

[Rob Landley]

On 9/1/22 15:33, Marc Chantreux wrote:
> this is because to me, choosing a shell about features and weight.
> 
> which dash mksh oksh yash zsh bash|xargs realpath|xargs ls -s|sort -nr
> 
> 1208 /bin/bash
>  864 /bin/zsh
>  400 /usr/bin/yash
>  288 /bin/mksh
>  284 /bin/oksh
>  124 /bin/dash

That kinda needs context. A dynamic i686 build against musl-libc of toysh:
$ ls -l sh
-r-xr-xr-x 1 landley landley 80032 Sep  2 06:53 sh

A static x86-64 musl build of the same code:
$ ls -l sh
-r-xr-xr-x 1 landley landley 166696 Sep  2 06:54 sh

Dynamic x86-64 version against glibc:

$ ls -l sh
-r-xr-xr-x 1 landley landley 94440 Sep  2 06:55 sh

And of course that's toysh standalone, not the amount of space enabling it in
menuconfig adds to a toybox defconfig binary.

> * there is a lot of lovely things in zsh: excellent as interactive shell
> * all the others are at least 50% the size of bash and good enough for
>   99% of the scripts
> 
> so bash is something I just avoided for 2 decades now

Bash was not just the standard shell of Linux for its first 15 years, and was
literally the first program Linux ever ran in 1991:

"So instead of starting init, the first thing my kernel did was to start the
shell. I had implemented about twenty-five system calls and as I mentioned this
was the first real program I was trying to run. The shell wasn't something I had
written myself. I had downloaded onto a disk a clone of the Bourne Shell, which
was one of the original Unix shells. It was available over the Internet as free
software, and its name was derived from a bad pun. The guy who wrote the
original was named Bourne, so was the clone Bourne-Again Shell -- commonly
referred to as bash." - Linus Torvalds, "Just For Fun", Page 65-66.

When Ubuntu switched its /bin/sh to point to dash in 2006, they broke the kernel
build. (They made the change to try to speed up Ubuntu's init scripts as
explained in https://wiki.ubuntu.com/DashAsBinSh and the failure of this to
accomplish their goal led to the creation of https://upstart.ubuntu.com/ which
also failed. Kind of a theme for Ubuntu...) Until then, bash was "the shell of
linux" pretty much everywhere.

> and I'm just happy
> this way. plus: it needs compound command to declare a function ;)

  $ bash -c 'x() { echo hello;}; x'
  hello

>> > If I want that feature, I need a first external dependency (i'll
>> > probably choose rc or [om]ksh) but is orthodoxy a strong value of toybox?
>> I don't know what you mean by orthodoxy in this context.
> 
> orthodoxy meant "keep staying straight to the doxa"

Hmmm... https://en.wikipedia.org/wiki/Doxa says it means "common belief or
popular opinion". Maybe you're not wrong? At the top of
https://landley.net/toybox it says:

> Toybox combines the most common Linux command line utilities together into a
> single BSD-licensed executable that's simple, small, fast, reasonably
> standards-compliant, and powerful enough to turn Android into a development
> environment.

But what specifically those commands are and how they should behave is...
squishy? The best standard is Posix but that's _intentionally_ insufficient
(lots of holes and obsolete sections). At the other end man7.org is too much
(describes many commands and features we should NOT implement). The "Linux
Standard Base" effort failed, in part because it ceased being a community
project after the Linux Foundation bureaucracy took control of it, eventually
leading to https://lwn.net/Articles/658809/

The roadmap section tries to explain the reasoning for the selection at length.
Other webpages such as https://landley.net/toybox/design.html tries in the
design goals->features and portability->standards sections, and the "about" page
has "What commands are planned/implemented" as well... but it all boils down to
"eh, it's a judgement call".

I initially got into busybox development because I built Linux From Scratch and
then tried to replace all the "gnu" packages with busybox. Linux From Scratch
release 3.0 chapter 5 created a temporary system (at a special /tools path)
which you would then chroot into to build the final version, a sort of "airlock
step" to isolate it from variations in the host system and prevent unrecorded
files or settings from leaking through verbatim. I reasoned that if I could do a
toybox based chapter 5, you could build the whole of LFS from there, and then
bootstrap up to arbitrary complexity (starting with the Beyond Linux From
Scratch packages, and even reproducing full distributions like Fedora or Debian
or SuSE or Gentoo under the result).

Unfortunately, distro bootstrapping turned out to be a lot harder than I
expected (as explained at https://landley.net/aboriginal/about.html#hairball and
I also wrote a more detailed version of the above history at
https://landley.net/aboriginal/history.html which I really SHOULD finish
someday. Or at least edit to end more gracefully...)

The other problem is once I _have_ such a system... well technically nothing in
any of the package builds requires "ps" to be there, let alone "top" or "vi".
But if I ssh into a system that can't list available processes or edit a text
file, I'm unhappy. (This leads into a discussion of "build environment" vs
"development environment", the latter of which should probably have X11 or
simiilar and the first of which doesn't need it, which is a LONG DIGRESSION and
does not result in clear lines between one and the other.)

Even before I joined Busybox, that project was a previous attempt at coming up
with a reasonably clean "base" system (including things like watch/top/vi), and
at this point incorporates almost a quarter century of end-user feedback about
what people requested and could live without. So I do sometimes check what they
have and haven't implemented when unsure about whether toybox should have
something. (Of course given that I maintained that project for a while 15 years
ago last I checked circa 2010 or so I'd written something like 1/3 of what was
there myself from scratch, it's mostly a question of what I don't remember or
decisions they've made recently. But toybox has "wget" but not "curl" because
that's the decision busybox made. Them having traceroute goes about 80% of the
way to convincing me toybox should have traceroute. That said neither busybox
nor toybox having "mesg" or "sum" has convinced me we should have that yet, and
I moved "hostid" to the examples directory because it just silly these days...)

Other decisions like the current "toybox diff should only support unified diff
output, the others are obsolete" are based on years of usage rather than any
specific published standard. I can point to influences like "git diff does diff
-u" but that's not a _standard_. That's anecdotal evidence.

And these days, Android's the biggest user of toybox and their needs are more
immediate and represent more users than my original goals. I'm still _pursuing_
my original goals, but "distro bootstrapping" has turned into "building AOSP
under a stock android system".

> (french people use this word
> in its etymological sense, sorry if it's not the case in english) and I
> though yours was "POSIX only things" (false memory from your "busybox vs
> toybox" talk).  I was wrong: sorry about that.

Wikipedia got that impression too somehow. It's not that simple.

>> It's not hard to add, I literally didn't because bash didn't. No deeper reason.
> 
> Because of bash, many people don't know the fact that every other shell
> does support it but it seems everyone I shown the tips just liked it.

It's not that hard to add to toysh either. Sometimes I go "this would be a
trivial extension" but don't do it because I'm assuming there's a reason for it
I just don't get? I don't want to have significantly less error checking. I also
want to avoid tempting people into using toybox-isms that WON'T work on other
implementations. (Or at least keep that down to a dull roar.) I still don't
understand why ";" on its own without a statement in front of it is an error,
leading to:

  $ if true; then; echo hello; fi
  bash: syntax error near unexpected token `;'
  $ if true; then
  > echo hello; fi
  hello

I mean, LOGICALLY the ; and the line break should be the same, right? But no.
You can have a line break in places you can't have a ; and... ok then?

Rob