[Toybox] is orthodoxy a strong value of toybox?

[Rob Landley]

On 9/6/22 14:18, Marc Chantreux wrote:
>> Bash was not just the standard shell of Linux for its first 15 years, and was
>> literally the first program Linux ever ran in 1991
> 
> I really see bash as a bloated korn shell that becames standard on linux
> shell just because it came as a part of the GNU project

Linux has nothing to do with the Gnu project. Never did. The Gnu project latched
onto Linux the way a drowning swimmer latches on to anyone nearby, but Stallman
announced gnu in 1983 and when Linux first shipped 7 years later gnu still
didn't have anything usable. (For comparison, the Mark Williams company created
Coherent Unix from scratch in 3 years, including their own compiler, kernel,
shell, and complete set of command line utilities:
https://en.wikipedia.org/wiki/Coherent_(operating_system) . Started 1977, first
version shipped 1980.)

The FSF has never been good at making software, they're a political lobbying
organization. Saying the FSF's purpose is to make software is like saying the
Girl Scouts' purpose is to bake cookies. It's a sideline to support their real
goal of license prosthelytization. (I say that having driven to FSF headquarters
in Boston in February 2001 to interview Richard Stallman in person for the
computer history book I intended to write.)

> and lot of
> people (including myself) used bash because we thought the choice was
> made after curration.

The Linux kernel was literally developed to run Bash. The test load its initial
system calls were implemented against was an a.out bash binary.

In 1990 Linus saved his birthday money and sold his Sinclair QL to buy a 386 PC
which he installed Minix on, including the huge accumulated patch stack people
maintained to convert it from 16 to 32 bits (among other things which meant
recompiling everything with gcc which supported the 386, instead of minix's own
8086-only compiler). None of those patches could go upstream because Tanenbaum
wanted a simple teaching tool to go with the textbook he'd written, and didn't
accept any external contributions from people trying to use it as a real OS, so
it took Linus weeks to convert his Minix install over from 16 to 32 bits.

But even a 32-bit Minix was a microkernel OS, which meant it did a process
context switch through the scheduler to handle every interrupt in a userspace
process, which meant it couldn't keep up with a 2400 baud modem without dropping
characters. Linus had been dialing into the univerity's Microvax with his
Sinclair QL to read usenet (including comp.os.minix) and upload and download
programs, but he'd sold that machine to buy the PC and now had to leave his dorm
room to go to the library to read usenet, which sucked in the middle of winter
in Finland.

So Linus wrote a terminal program that booted from a floppy and ran on the bare
metal of his PC (bypassing Minix's interrupt handling limitations). In order to
upload and download stuff with it, he taught it to read and write the minix
filesystem. But he had to boot back into minix in order to mkdir or rename files
or delete files on that filesystem (he only had a 40 meg hard drive and needed
to delete stuff often to free up space).

So he taught his terminal program to run a shell, and from that commands like
"mv" and "rm". And the shell he used was "bash", because it compiled cleanly
with the gcc he was using for his 32 bit Minix port, and other shell sources
needed adapting to gcc. (As he said, "my terminal program grew legs".)

Then one day Linus accidentally set his terminal program autodialing /dev/hda
instead of /dev/sda, and by the time he figured out the university's phone line
wasn't busy the dialer had overwritten his minix partition with hayes ATDT and
+++ATH commands, and he either had to reinstall Minix and convert it back to 32
bits (which had taken WEEKS the first time around because it was a long
elaborate patch stack requiring a lot of manual adjustment), or turn Linux
(which still booted from floppy, and his /home partition with the source was
intact) into a usable system. He was forced, by accident, to dogfood his project.

Bash was central to that narrative.

Most of the above was in "Just for Fun", but some of it was in the early posts
from the "linux-activists" list (where development discussion moved to when they
got kicked off comp.os.minix, I collected posts I found interesting from the
first two years at https://landley.net/history/mirror/linux/1991.html and
https://landley.net/history/mirror/linux/1992.html ), and a little is from the
book "open sources" specifically
https://www.oreilly.com/openbook/opensources/book/linus.html and
https://www.oreilly.com/openbook/opensources/book/appa.html and for context you
can get the BSD perspective
https://www.oreilly.com/openbook/opensources/book/kirkmck.html
and stallman's on self-serving gnu narrative
https://www.oreilly.com/openbook/opensources/book/stallman.html

Really, the whole book's a good read...

> But I see a trend to replace bash by

Because it went GPLv3. People aren't really trying to replace bash, they're
trying to replace GPLv3 software.

> * zsh when the main goal is to provide a confortable shell with a lot of features
>   (and really: bash can't compete with zsh on it)

That's the decision Apple made when they rejected GPLv3 and stayed on the last
GPLv2 release of every package (including shipping gcc 4.2.1 and binutils 2.17
in xcode for 5 years until the llvm/clang work they sponsored was ready to
replace it). After many years stuck on the last GPLv2 release of bash, they
switched to a different shell because bash would never have another non-GPLv3
release.

They similarly abandoned GPLv3 Samba and wrote their own replacement (
https://appleinsider.com/articles/11/03/23/inside_mac_os_x_10_7_lion_server_apple_replaces_samba_for_windows_networking_services
) and Jeremy Allison the Samba maintainer has publicly lamented his decision to
go to GPLv3 ( https://archive.org/details/copyleftconf2020-allison ) and said
he'd undo it if he could, but he can't.

http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/

Don't attribute a general move away from GPLv3 to a desire to move away from
bash specifically.

> * dash when the goal is to provide the minimal setup for simple scripts
>   and most of the scripts I reviewed don't even use the features from
>   dash also provides so most of them can be converted to dash
>   scripts with minor changes (just removing useless bashisms).
>   Also "The Debian policy manual has long mandated that "shell scripts
>   specifying '/bin/sh' as interpreter must only use POSIX features"

$ grep '[#][!]/bin' toybox/scripts/*.sh
toybox/scripts/change.sh:#!/bin/bash
toybox/scripts/findglobals.sh:#!/bin/bash
toybox/scripts/genconfig.sh:#!/bin/bash
toybox/scripts/install.sh:#!/bin/bash
toybox/scripts/make.sh:#!/bin/bash
toybox/scripts/make.sh:  echo '#!/bin/sh'
toybox/scripts/mcm-buildall.sh:#!/bin/bash
toybox/scripts/mkgetconf.sh:#!/bin/bash
toybox/scripts/mkroot.sh:#!/bin/bash
toybox/scripts/mkroot.sh:#!/bin/sh
toybox/scripts/single.sh:#!/bin/bash
toybox/scripts/test.sh:#!/bin/bash

The two that DON'T say "bash" specifically are:

1) In make.sh it creates build.sh, which ships on an unknown target system
(which could be bsd, which for some reason puts bash in /usr/local or something
and wants you to "#!/usr/bin/env bash" to run it, which is INSANE because if you
trust /usr/bin/env to be installed in THAT SPECIFIC PLACE why can't you trust
bash to be in /bin/bash, how is that qualitatively different? Yes I'm aware
posix is stupid here, it has to do with IBM and Microsoft wanting fips-151-2
money back in the 1990s.)

2) The bin/sh in mkroot.sh is because it's writing an init script to run in the
new root filesystem, where I'm providing the shell, and I know that points to toysh.

> so
>   dash is just enough." (https://wiki.ubuntu.com/DashAsBinSh)

I linked to that article on Ubuntu's switch to the Defective Annoying SHell
(which was segfault city back when they did it, because in it they explained
that the REASON the switched was they were trying to speed up init scripts, and
felt that changing every init script to say #!/bin/dash instead of #!/bin/sh at
the top was too intrusive a change, so they changed where /bin/sh pointed to
instead (and in the process broke the kernel build, very strongly implying
nobody else had ever been that stupid or someone would have noticed before).

I note that at the time, dash was utterly CRAPTACULAR. Obviously missing
features, I managed to segfault it multiple ways... it even got terminal control
wrong: https://landley.net/notes-2007.html#:~:text=wrong%20with%20dash

Of course the sad part was that the change DIDN'T speed up their shell scripts
as much as they thought it would, so they wound up creating "upstart" to run
them in parallel. An idea they copied (without credit) from an IBM developer who
used "make" a couple years earlier to run his init scripts in parallel:

https://lwn.net/Articles/50115/

Of course upstart went the way of every new technology Ubuntu introduced: they
pushed bzr source control in launchpad: everybody went git. They pushed mir,
everybody went wayland. The "unity" desktop was just SPECTACULARLY unusable. So
of course when Ubuntu pushed upstart, its failure gave us systemd. How do you
faceplant so badly you LOSE TO SYSTEMD?

But what makes it especially sad is WHEN they admitted defeat and wrote upstart,
THEY DIDN'T POINT BIN/SH BACK TO BASH! Because that would be acknowledging they
made a mistake.

> I added the mksh variants originated from openBSD (mksh, oksh) because
> they are very good at providing a good balance between feature and
> weight and are saddly underrated in the linux world.

That was the shell Android used to avoid shipping GPLv3 code.

The FSF crapped out GPLv3 the year before the first commercially available
Android phone (the G1) shipped, which seems to have led to Android's "no GPL in
userspace" policy where nothing GPLv3 is ever on the device, the grandfathered
in GPLv2 packages tend to be slowly replaced with new non-GPL packages (such as
the Bluetooth stack), and adding additional GPL packages to the system violated
the trademark licensing guidelines back when they were public. (Dunno if that's
still the case, the terms weren't online anymore last I checked.)

Apple and Android aren't remotely unique in this, LOTS of people refuse GPLv3.
Not just corporations and weirdos like me (https://lwn.net/Articles/202106/)
either. Here's the "hell no" the Linux kernel developers collectively put out
about GPLv3:

https://lwn.net/Articles/200422/

Which was a follow-up to Linus Torvalds saying his code in the kernel was all
GPLv2-only and could NOT be used under GPLv3 all the way back in 2000, 7 years
before Stallman played "chicken" with him and lost:

https://lkml.iu.edu/hypermail/linux/kernel/0009.1/0096.html

Sadly, there's been a lot of throwing the GPLv2 baby out with the GPLv3
bathwater, but that's understandable. I even talked about it in my 2013 toybox talk:

  https://landley.net/toybox/about.html#why

> I put yash because I see a recent buzz about it. I don't know it that
> much but noticed it is half the size of zsh but yet provide process
> expansions so once again another compromise.

*shrug* People wave a new shell at me every year or so. (Is "fish" still around?
How about oil shell?)

>> > and I'm just happy
>> > this way. plus: it needs compound command to declare a function ;) 
> 
>>   $ bash -c 'x() { echo hello;}; x'
>>   hello
> 
> $ dash -c 'x() echo hello; x'
> hello
> 
> why should I bet on a 10 times bigger, considered slow shell to do less
> than what I want to write?

*shrug* You came to me. I'm implementing a bash clone. Always was. The first
version of www/roadmap.html checked in back in 2012 (commit 8f90d3aa019f)
included the text:

  This means that toysh needs to supply several bash extensions _and_ work
  when called under the name "bash".

> About the rest of your mail: thanks for the context about all your
> links. I'll read them carefully.

Link dump's an occupational hazard around here. :)

  https://landley.net/history/mirror

>> file, I'm unhappy. (This leads into a discussion of "build environment" vs
>> "development environment", the latter of which should probably have X11 or
>> simiilar and the first of which doesn't need it, which is a LONG DIGRESSION and
>> does not result in clear lines between one and the other.)
> 
> I don't understand way X11 would be necessary in a "development
> environment"

Shorthand for "a GUI". The "or similar" could be wayland or opengl or gtk on the
linux framebuffer or android's renderer or sdl or...

A build environment can be a headless build server you remotely submit jobs to,
or is driven by jenkins to checkout out and build each new git commit, or does
nightly builds, or...

A development environment is where a human writes new code and creates tests,
which generally means multiple terminal windows and cut and paste with a mouse
and colored syntax highlighting and an email client and a web browser and chat
client (slack/signal/skype/teams/discord/line/hexchat...) and...

The thing is, even a fairly sparse build server should have ASPECTS of a
development environment because you're probably going to ssh into it and poke at
stuff from time to time when the build goes weird there in a way it doesn't on
your personal laptop. The build environment shouldn't be so bare that it's
unfriendly.

Toybox by policy does not do graphical anything (with the possible exception of
fbsplash), but we don't want to interfere with it either. It should be possible
to cleanly install a graphical IDE on TOP of a build system. If you're going to
convert a build environment into a development environment by installing more
packages, then the build environment should at least potentially be a strict
subset of the development environment without requiring anything to be replaced.

This means if I'm going to provide a "vi" I want at least the OPTION for people
to use it as a real load bearing vi that doesn't need to be replaced with a
"better" one. Because if I _don't_, I'm inviting horrible version skew and
complete lack of testing for the thing that gets replaced with a "real one"
instead of actually used.

In the case of android, anything graphical is probably going to be implemented
at least partly in Java and be an android application running in its own ID with
weird GUI stuff and that I'm not touching the high voltage lines there. Render
unto Android Studio that which is Android Studio's, my job is to provide
commands it can cleanly call out of the $PATH.

> but assuming awk and make are available from the build env,
> I would think about vi, ctags and maybe cscope and a debugger.

See, when you run configure/make/install or a given package's equivalent, it's
likely to call awk. It's unlikely to call vi. That's why awk is part of a build
environment, and vi is part of a development environment.

>> implementations. (Or at least keep that down to a dull roar.) I still don't
>> understand why ";" on its own without a statement in front of it is an error,
>> leading to:
>> 
>>   $ if true; then; echo hello; fi
>>   bash: syntax error near unexpected token `;'
>>   $ if true; then
>>   > echo hello; fi
>>   hello
>> 
>> I mean, LOGICALLY the ; and the line break should be the same, right? But no.
>> You can have a line break in places you can't have a ; and... ok then?
> 
> to me those kind of flaws are not important

A) Is it a flaw? It's a design choice. I don't know WHY. Sometimes there's a
subtle reason or weird history I'm unaware of.

B) I'm writing a shell. I have to care.

I'm vaguely aware of posix, but my actual SPEC is "what bash does". (And when
what bash does isn't a match for what the bash man page says, I am sad. When I
inform chet and he fixes the man page, I am happy. When he changes bash's
behavior instead, I have introduced more version skew and categorically cannot
implement "what bash does" without picking a version, and if I take the new
stuff it's not what anybody has deployed yet and I get very tired...)

> because what you really want
> to write is indeed `then echo hello`. there are more things that are
> problematic to me. for example:
> 
> those are ok
> 
>>&2 echo error
> echo error >&2
> {
> 	echo error
> 	echo error
> 	echo error
> } >&2
> 
> but this one isn't
> 
>>&2 {
> 	echo error
> 	echo error
> 	echo error
> }

Because { is a flow control statement. Flow control statements have to be the
first thing on the line. (Well, the logical line after the semicolon or
similar.) You can't "A=B if true; then echo $A; fi" either.

> another think I hate in *sh is that a glob with no match returns the
> glob itself and some examples use this as as feature.

  $ shopt -s nullglob
  $ echo walrus*
  $

(It's described in the bash man page. You can also "help shopt" and run shopt
with no arguments to get a list of the options and their current status, but I'm
unaware of a "shopt help option" outside the man page? I haven't implemented
shopt yet...)

> yet another? why the hack can't I use read at the end of a pipe the
> way bash and zsh does?

Because if any pipeline segment isn't a child process you can't suspend the
pipeline with ctrl-z, or background it with & or bg.

Fun test case in the other direction:

  while true; do echo potato; done | head -n 5

Even though "echo" and "true" are both shell builtins, if the chunk of shell
script before the pipe DOESN'T get an implicit ( ) around it to become a child
shell, you get a hang because the while loop runs endlessly (well blocks writing
to a full pipe) before spawning the "head" process. Trying to make a hybrid
"child and not child" approach work is a recipe for deadlocks, I say this having
tried it. :)

> this is ok
> 
> 	read h < /etc/hostname
> 	echo $h
> 
> this is not
> 
> 	hostname | read h
> 	echo ${h?can't read the output of a pipe}

hostname | {
  read h
  echo ${h?can't read the output of a pipe}
}

> I really would like a shell to fix all those known problems and with
> a syntax inspired by rc and the alternative zsh but I fear there would
> be no adoption.

I am _writing_ a shell. From scratch. Have been for a while now. And I blog
about this stuff when not posting about it here:

  https://landley.net/notes-2022.html#22-06-2022

Bash has been around longer than Linux. It's had 8 gazillion people pound on it
and find weird corner cases. It still has some unquestioned assumptions and
weird evolutionary leftovers (the programming equivalent of the recurrent
laryngeal nerves), but it's got noticeably more "aha, THAT'S why that's like
that" than "there's no reason for that to be like that" so far.

Rob