[Toybox] Endless bash questions...

[Rob Landley]

On 6/30/22 23:39, Ryan Prichard wrote:
> On Mon, Jun 6, 2022 at 8:50 AM Chet Ramey <chet.ramey at case.edu
> <mailto:chet.ramey at case.edu>> wrote:
> 
>     On 6/4/22 7:12 PM, Rob Landley wrote:
>     >
>     > P.S. Currently I'm trying to work out the sequencing of redirects vs
>     > $(subshells) because these work:
>     >
>     > $ bash -c $'if true; then cat; fi << EOF\none two\nEOF'
>     > one two
>     > $ bash -c $'[[ $(cat) == "one two" ]] <<EOF\none two\nEOF'
>     > $
>     > $ bash -c $'[[ $(cat) == "one two" ]] <<EOF\none two\nEOF\n[ $? -eq 0 ] &&
>     echo yes'
>     > yes
>     >
>     > But this hangs (cat is reading stdin, not the HERE document):
>     > $ bash -c $'[ $(cat) == "one two" ] <<EOF\none two\nEOF'
>     > ^C
> 
>     It's an order of operations question. One of the things about compound
>     commands is that the redirections apply to the entire command, so they
>     are performed before the compound command is executed. So what happens is
>     the command gets parsed into a tree, like always, then executed, and
>     redirections happen before the command is executed. For `[[' the operand
>     expansion is part of the execution, so the redirection has already taken
>     place.
> 
>     Now, `[' is a simple command, not a compound command. Bash performs the
>     word expansions before performing redirections. It always has, and it
>     makes things like handling null commands with redirections slightly
>     easier. It's not strictly Posix-conformant.
> 
>     (And having the redirections precede the simple command on the input line
>     only affects the parser -- it ends up getting parsed to the same structure
>     and executed the same way as putting them after the command.)
> 
> I noticed a situation where bash is behaving differently from other shells I've
> tested, and it looked related, maybe? If the $(...) subshell is assigned to a
> variable, then the redirection doesn't apply to it:
> 
> $ bash -c 'out=$(echo err >&2) 2>/dev/null'
> err

try: bash -c '{ out=$(echo err >&2);} 2>/dev/null'

Assignments happen before redirections. This is why you can't quote or escape
the variable name before the equals sign, it basically just does a for (x=0;
varend(arg[x])=='='; x++); on the arguments, peels the assignments off the
front, and handles them first (and separately) before addressing the rest of the
command.

The main difference between prefix assignments and non-prefix assignments is
whether it creates a temporary function context for the assignments to be local
(but exported) within and pops it back off afterwards. Otherwise they're
assigned into the current function context.

This is complicated by it only being a prefix assignment if the EXPANDED
arguments provide a nonzero command line:

  $ abc=def $POTATO
  $ echo $abc
  def

So yeah, there's a sequencing issue. (Bash does variable expansions and
redirections in two passes, toysh does them in one, so this is a hard hair for
me to split. But I've got a TODO item for it...)

> $ zsh -c 'out=$(echo err >&2) 2>/dev/null'
> $ dash -c 'out=$(echo err >&2) 2>/dev/null'
> $ adb shell 'out=$(echo err >&2) 2>/dev/null'   # Android uses mksh
> $ /x/toybox/toybox sh -c 'out=$(echo err >&2) 2>/dev/null'

I'm mildly curious if posix explicitly expreses an opinion, but don't really
care what the other shells do. I test what bash does, and sometimes I test what
older versions of bash do (usually 2.05b), and I read the bash man page, and
when all else fails I make puppy eyes at Chet.

The last time I read the whole shell part of the posix spec through was back
when the 2008 version came out.

> I see the same result if I replace $(...) with `...`.

They're functionally equivalent, backticks just don't nest as easily.

> FWIW, I noticed this behavior while writing code to capture stdout and stderr
> separately, then print one after another. I'm not actually redirecting FD2 to
> /dev/null, but rather to FD1, just in case FD1 and FD2 unexpectedly point to
> different files/pipes. I only actually need the code on old versions of Android
> (prior to N, which added the shell_v2 adb feature).

See above curly bracket trick to force ordering.

> Surely this syntax is POSIX-specified,

Posix is full of giant holes big enough to drive Windows NT and OS/360 through.
It does not mention "init" or "mount", meaning you can't BOOT a system that
doesn't go beyond posix.

> but I wonder if the semantics are specified.

The standard for Linux isn't posix, the standard is bash. Linux was literally
created to run bash.

(Sorry, pet peeve of mine. Rant incoming.)

In 1991 Linus Torvalds expanded a bare metal terminal program he'd written so it
could run bash, creating Linux. Not "a shell", specifically _bash_. He wrote the
term program in the first place because minix was a microkernel which meant its
interrupt processing was so expensive it couldn't keep up with a 2400 baud
modem. So Linus wrote his own terminal program that booted from a floppy to run
on the bare metal of his 386 pc, then he taught it to read and write the minix
filesystem on his IDE hard drive so it could upload and download files, and then
he made it run bash so he could mkdir/mv/rm without rebooting into minix, which
turned out to be 95% of the system calls gcc needed to run so his system was
self-hosting and he didn't need to boot into minix anymore. (Which was lucky
when he autodialed /dev/hda instead of /dev/sda and overwrote the start of his
minix partition with repeated ATDT strings commands before noticing the
university's vax was busy for an unusually long time. He still had his data
partition, and Linux DID boot from a floppy...)

This means bash wasn't just the standard shell of Linux even before 0.0.1 was
posted to comp.os.minix, running bash specifically is why Linux became a unix
kernel over 30 years ago. Anything bash has been doing all that time is by
definition the correct behavior for a Linux shell to do, and anything posix says
that contradicts that is kind of irrelevant. I may implement it if it doesn't
comflict, but I'm assuming Chet already read posix and tweaked bash to be as
compliant as necessary.

For most commands I'll weigh what posix says against what the gnu/dammit version
is doing and often side with posix: but not here. If you want to say that
current bash has a problem and what bash 2.05b was doing was better I'll listen,
but if posix contradicts bash then posix is wrong. (About "what a linux shell
should do". Bash is DEFINITIONALLY right there.)

I've ranted about Ubuntu's switch from bash to dash before:

http://lists.landley.net/pipermail/toybox-landley.net/2018-February/009353.html

Remember that Ubuntu only existed because post-IPO Red Hat exited the retail
Linux market in 2003 (abandoning 50% market share in desktop linux to take over
Sun's market niche instead), and after a couple years where SuSE was going
bankrupt and Slackware's founder was slowly dying from electric toothbrush
disease and I was running knoppix manually "installed" onto my hard drive via a
hand-written init script doing a loopback mount, a white male south african
crypto billionaire bought his way into the market vacuum. (No, the other one.)

And yes ubuntu switched first and debian picked it up from them, in part because
(I'm told) Shuttleworth hired one or more full-time debian developers to get
"debian stale" unblocked before the base distro his project was sitting on top
of died of constipation: 3.0 came out in 2002, 4.0 came out in 2008, and in
between enough developers fled debian to crush gentoo. (The debian users fled
the endless flamewars for a distro where communication was civil and focused on
tech. They rapidly outnumbered gentoo's original developers and quickly started
complaining about how there were so many flamewars here at gentoo and they must
have been misled about that civil tech-focused culture they'd been promised.
When debian finally got a release out they went back, at which point what was
left of the Gentoo development community... imagine the Avengers scene with Loki
lying in smashed concrete after encountering the Hulk, except the escaping
wheeze says the word "Funtoo". The project never did entirely recover if you ask
me...)

I.E. the switch from bash to dash was a billionaire imposing a decision on a
larger community in a time of crisis, and then refusing to admit a mistake and
undo the mess when it failed. (A failure which let Red Hat, with $2 billion
revenue in 2015, shove systemd down everybody's throats because Enterprise.)

I do not, and never have, considered dash interesting. I consider mksh
interesting to the extent that toysh should be able to run existing shell
scripts in android, but my strategy there is "wait for bug reports to provide
test cases"...

Rob

P.S. To be honest, the most FRAUGHT part of me asking Chet questions about weird
corner cases is when he goes "oh, I'll change it". But then it's incompatible
with previous versions! I dowanna make compatibility with bash a moving target.
Project's old enough to have defined behavior, whether or not it's RIGHT...