[Toybox] [PATCH] sh: pass "\" to the later app

[Rob Landley]

On 6/12/23 19:40, Chet Ramey wrote:
>> and they have a list of "special built-in utilities" that does NOT include cd
>> (that's listed in normal utilities: how would one go about implementing that
>> outside of the shell, do you think?)
> 
> That's not what a special builtin means. alias, fg/bg/jobs, getopts, read,
> and wait are all regular builtins, and they can't be implemented outside
> the shell either.
> 
> Special builtins are defined that way because of their effect:
> 
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_14
> 
> It's really a useless concept, by the way.

It's not that simple: kill has to be built-in or it can't interface with job
control...

Wait, assignments before these magic utilities are NOT prefix assignments
limited to the duration of the command?

  $ abc=123 true
  $ echo $abc
  $ abc=123 :
  $ echo $abc
  $ abc=123 eval 'echo $abc'
  123
  $ echo $abc
  $

Nope, even bash doesn't do that. (A prefix assignment... on continue? I can't
even do a prefix assignment on "if", and I have _use_cases_ for that. I had that
implemented and then backed it out again because it's an error in bash. I
remember I did make "continue&" work, but don't remember why...)

>> Anyway, I found the third shall retain" in V3_chap02, and... it's wrong?
> 
> No.
> 
>> 
>>> (see Escape Character (Backslash)) only when followed by one of the
>>> following characters when considered special:
>>>
>>>       $   `   "   \   <newline>"
>>>
>>> So the backslash-newline gets removed, but, say, a \" only has the
>>> backslash removed.
>> 
>> Because when you put a backslash in front of another char:
>> 
>>    $ echo \x
>>    x
>>    $ basename \x
>>    x
> 
> The text I quoted was from the Double Qoutes section. The additional
> reference to (Escape Character...) gives it away.
> 
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02_03

And I need parsing that eats \$ and \\n but leaves \x alone, great. (Which is a
new status flag for expand_arg(), can't be handled in a preparsing pass nor is
NO_QUOTE gonna do it right...)

Why is only the second of these an error in bash?

  $ unset ABC; echo ${ABC::'1+2'}
  $ ABC=abcdef; echo ${ABC::'1+2'}
  bash: ABC: '1+2': syntax error: operand expected (error token is "'1+2'")

>> And my approach of handling HERE document lines one at a time probably came from
>> "...the lines in the here-document are not expanded. If word is unquoted, all
>> lines of the here-document are subjected to  parameter  expansion, command
>> substitution, and arithmetic expansion, the character sequence \<newline> is
>> ignored, and \ must be used to quote the characters \, $, and `."
>> 
>> Except... \<newline> is ignored when the EOF _is_ quoted? It glues lines
>> together when it's not quoted? (It's late and I'm not sure I'm reading this
>> clearly. Need test cases...)
> 
> When the EOF is not quoted, the here-document body is essentially double-
> quoted.

Working on it...

> "In this case, the <backslash> in the input behaves as the <backslash> 
> inside double-quotes (see Double-Quotes)." (POSIX again.)
> 
> The backslash-newline gets removed just like it does in double quotes.
> 
> When the EOF is quoted, the here-document body is essentially single-quoted
> (not exactly, but you get the idea). The backslash-newline gets preserved.

I think when the EOF is quoted the HERE body has no processing, and when it's
not quoted then $VARS \$ and \<newline> are the only special... Nope, \\ is too.

>>>>     $ bash -c $'cat<<"";echo X\n\necho Z'
>>>>     X
>>>>     Z
>>>
>>> This is dodgy behavior to rely on: a null delimiter is matched by the next
>>> blank line, since that's technically "a line containing only the delimiter
>>> and a <newline>, with no <blank> characters in between."
>> 
>> I'm trying to match what bash does, which means figuring _out_ what bash does. I
>> respect posix, but I expect to diverge from it a lot because so much of what I'm
>> trying to be compatible with already does. :(
> 
> I understand, but what I wrote explains what bash currently does. It's just
> not a good idea for anyone to rely on the behavior of a null here-document
> delimiter. I can't imagine anyone does.

If I can think of a corner case, I'm trying to make it match bash, because stuff
depends on bugs ALL THE TIME:

  https://github.com/landley/toybox/commit/32b3587af261

>> I'm basically abusing function contexts, because that's what I attach local
>> variables to, and $LINENO resets but persists in the same way as local vars:
> 
> I don't think that makes any sense. You can use `return' in a `.' script as
> a special case, but dot doesn't make local variables work. LINENO gets
> reset because you're using a new input source, and reverts to its previous
> value when you go back to the previous input source. LINENO's not good for
> much more than error messages, and it's good to have the current input
> source and the current line number match up. It's not a local variable, per se.

When you create a new local variable it does so in the most recent named
function context (or the root context if it reaches it), skipping unnamed
function contexts. When you resolve or modify an existing variable (or unset it,
which creates a whiteout entry) it iterates back through all existing function
contexts to find a matching entry (then puts one in the root context if you were
assigning without declaring it local).

So "local blah" won't bind to an anonymous function context, and errors out if
it reaches the root context. I _think_ it works...

>>    $ bash -c $'echo $LINENO;. <(echo echo \\$LINENO);echo $LINENO'
>>    0
>>    1
>>    0
> 
> The real question is what value LINENO should have when using -c command,
> even though it's only defined for a script or function.

We've gone over that one before. You decided you were going to initialize it to
1 instead of 0, but I haven't updated to the version that does that yet so I'm
still matching the behavior in my devuan install. (Still devuan bronchitis,
haven't updated to devuan cholera yet. Um, the web page says devuan B matches
debian "buster" and devuan C matches "bullseye", if that helps.)

>> Basically any time I call back into do_source() it stacks a new function
>> context, but the ones with a NULL pointer for the function name behave slightly
>> differently so things like:
> 
> So the NULL function name tells you which aspects of function behavior to
> ignore?

Yes. One list, multiple uses.

>> so I'm not sure _how_ you'd remove it from input before tokenization, since
>> resolving quotes is part of tokenization...?
> 
> I do it as part of the routine that fetches the next character. If you're
> in a context where the backslash-newline is going to be removed, the
> tokenizer never sees it.

I was naieve enough to write the variable resolution logic with the design
assumption that unbalanced quoting contexts had already been caught before the
data was passed to us. Kinda biting me now, although I think I'm most of the way
through it.

(The other place that's gonna suck for me is I have a kind of a structural
problem handling $(case word in...) and friends. My word parsing handles nested
quoting contexts (saving it as a string to re-parse later) by counting quote
spans. It doesn't handle nested logical contexts, and "case" logic has unmatched
ending parentheses that can end the $() span prematurely...)

>> Mostly I'm reading the bash man page, pondering many years of
>> writing and editing bash scripts, and doing LOTS of tests...
> 
> And pointing out places where the man page isn't clear or doesn't describe
> the shell's behavior, which I appreciate.

Happy to help. At the same time, trying not to spam you too badly...

>>> The current edition is from 2018.
>> 
>> Except they said 2008 was the last feature release and everying since is
>> bugfix-only, and nothing is supposed to introduce, deprecate, or significantly
>> change anything's semantics.
> 
> That's clearly not true. If things are specified incorrectly, or don't
> reflect what the majority of shell implementations actually do, they're
> going to change.
> 
> If new needs arise, things are going to get added (e.g., gettext, or the
> ferocious arguments over strlcpy, etc.).
> 
>> That's why it's still "Issue 7". The new stuff is
>> all queued up for Issue 8, which has been coming soon now since the early Obama
>> administration.
> 
> Oh, I was there.

I was lurking on the posix list since... 2006 I think?

I pestered people at Atlanta Linux Showcase 1999 to explain HOW one builds a
Linux system entirely from source since the greybeards apparently knew but it
was undocumented, then got pointed at Linux From Scratch in email a couple
months later, compared the 1.7 megabyte "tomsrtbt" utility floppy with the 110
megabyte "Linux From Scratch" build result and went "hang on", started trying to
replace gnu packages with busybox and uClibc one command at a time somewhere
around 2002, spent years fixing problems with busybox as each new command didn't
work right in an actual build, helpfully assembled a busybox 1.01 candidate
release in 2005 when the old maintainer got too busy to work on the project
anymore, accidentally wound up officially maintaining busybox until GPLv3
happened, got my https://landley.net/aboriginal/about.html project building
itself under itself (and building Linux From Scratch under the result) in...
something like 2011? I had it down to 7 packages (busybox, uclibc, linux, gcc,
binutils, make, and bash.) But four of those packages were stuck on the last
GPLv2 release, the uClibc project was years dead, and I'd moved on from busybox
to toybox, so I did mkroot...

Sigh, I already gave a talk on all this history:

  https://www.youtube.com/watch?v=MkJkyMuBm3g#t=1m18s

Anyway, now I maintain Android's command line utilities, and am trying to get my
new shell in shape, and I hope Android can eventually use it instead of mksh.
(Working on it...)

>> They SUSv2 in 1997 (https://pubs.opengroup.org/onlinepubs/7990989775/), SUSv3 in
>> 2001 (https://pubs.opengroup.org/onlinepubs/009695399/), SUSv4 in 2008
>> (https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/) and SUSv5 isn't
>> out yet. A 4 year gap, a 6 year gap, and now 15 years and counting...
> 
> I don't think you get to count the interim (20xx edition) releases as not
> existing. They may not satisfy your exact definition of an update, but
> updates they are. You have to include them as updates, since those updates
> incorporate defect resolutions, which possibly modify required behavior.

The project isn't dead, but those are defined as bugfix releases. Adding new
libc functions or command line options, or "can you please put cpio and tar back
in the command list", are out of scope for them.

The kind of things they change are the kind of things I side with man7.org over
opengroup.org already. I was _already_ locally fixing the kind of thing they're
changing, because I view posix as a frame of reference to diverge from.

> You're on the list, there's plenty of "let's try and figure out what they
> were thinking in 1992 and then go from there."

"Chesterton's fence" is kind of annoying, but I do try to root cause things so I
can confidently REMOVE them, ala
https://www.linux.com/news/ever-changing-linux-filesystems-merging-directoris-usr/

(I also tentatively remove or skip a lot of stuff I _haven't_ root caused, and
wait for people to complain. But "this is why it was put in" is as close to
proving a negative as you can get...)

"A dozen white beardy men working on a PDP-11 funded by the patent and licensing
department of a fortune 500 corporation in New Jersey had a reason to do this in
1976" is not the same as there being a good reason to have it now. It _can_ be,
but it's not inherent in the proposition. Ken or Dennis having a reason means a
lot to me because those guys were really smart. The Programmers Workbench guys,
not so much. "Bill Joy decided" is a coin flip at best...

> The only thing that does is
> satisfy people who want all existing implementations to be non-conforming
> on some minor point. You have to look at what the actual implementations do
> and make the standard match those. Otherwise, how would users know what to
> expect?

My aboriginal linux project involved running package builds through the Linux
>From Scratch tools and the busybox tools, and doing side by side comparisons of
the resulting log output (and I made a tool to intercept every command out of
the $PATH and append the command line that got called to a file, which mostly
works deterministically in order on UP builds so it's easy-ish to diff the results).

My goal was to make the build behavior with busybox and the build behavior with
the corresponding linux from scratch tool come out identically. (I was also
trying to strip dates and uninitialized ELF fields and such out of the binaries
so I could compare them, long before the "deterministic build" people started
doing that. Some of my bug reports at them may have helped get that ball rolling...)

I switched from busybox to toybox a decade in 2011 (oh WOW were a bunch of
completely unrelated people mad, https://lwn.net/Articles/478308/), and
eventually replaced my set of system build scripts with a single ~300 line bash
script that builds Linux bootable to a shell prompt under qemu on a dozen
different architectures:

  https://github.com/landley/toybox/blob/master/mkroot/mkroot.sh

I only started trying to get _this_ project to build Linux From Scratch again a
few months ago, though. I still haven't even started the "awk" implementation yet...

  https://landley.net/toybox/roadmap.html#goals

Working on it. (Well in busybox somebody else had already written an awk, I just
sent them bug reports and made puppy eyes. This time I have to learn how to use
"awk". And I have to write a "make". And a shell, which is in progress... :)

>>> The only way to guarantee a resolution is to file an interpretation
>>> request, which has to be acted on. Otherwise, unless we get to some kind
>>> of consensus on the list, shells keep doing their thing.
>> 
>> As I said, I respect the work the posix guys are doing, but it's not the
>> standard I'm implementing the shell against. 
> 
> Second-hand, it is. If there's something in bash that isn't posix-
> conformant, and there's not an extremely good reason to keep it non-
> conformant, I'm going to make it conform. If you want to keep pace with
> bash, that requirement will eventually hit your code base.

Oh it does. But I'm happy to receive it second hand.

If I can get to the end of the shell TODO pile and the end of the bash man page
and run out of stuff I already know I'm doing wrong, THEN I go compare
everything against posix...

>> That's another reason I'm reluctant to start threads with you: it's very easy to
>> talk shop with somebody who knows MORE about this than I do, but I have a bit of
>> homework still to do before approaching the teacher about a lot of this stuff. :)
> 
> I wish you were not so reluctant. Look at how many things you've discovered
> that I decided were bugs based on our discussions.

But I'm taking up your valuable time.

I often link to Pascal's Apology ala
https://www.npr.org/sections/13.7/2014/02/03/270680304/this-could-have-been-shorter
because a big failure mode of mine is to blather on when I'm sleep deprived or
busy or similar. Editing stuff down to be concise is 2/3 the work. I try to keep
the stream of consciousness stuff to my blog.

But since you asked, today's new question I wrestled with was "what's the error
logic for slice fields"?

  $ echo ${one:!}
  bash: !}: event not found
  $ echo ${one:+}
  $ echo ${one:+:}
  $ echo ${one:]} two
  two
  $ echo ${one:0/0}
  $ echo ${PATH::1+2}
  /ho
  $ echo ${PATH::0/0}
  bash: PATH: 0/0: division by 0 (error token is "0")

It's doing math, but only _sometimes_ even reporting division by zero as an error?

(I still check the busybox list every month or so to see if they've come up with
interesting new test cases or features I missed, and
http://lists.busybox.net/pipermail/busybox/2023-June/090359.html got me to
notice https://github.com/landley/toybox/commit/577b4d35ca96 which is a fix for
the obvious problem, but there were non-obvious adjacent questions...)

This is my average day. I break everything.

>>> Bash-5.1 switched to using pipes for the here-document if the document size
>>> is smaller than the pipe buffer size (and hence won't block), keeping the
>>> temporary file for documents larger than that.
>> 
>> I hate having multiple codepaths to do the same thing without a good reason.
> 
> I understand. You'd be surprised (or not) at how vocal people were about
> "hitting the file system" and the horrible security consequences that has.

The name "tmpfs" comes from its original use on /tmp, mkstemp() is defined to
create with permission 0600 and O_EXCL, and the Linux kernel at least _used_ to
take an inode link count of 0 as an indicator not to schedule any flushes the
data to backing store (it would still do it for memory pressure but memory gets
written to the swap partition for the same reason) so it should mostly stay in
the page cache if you delete it BEFORE writing the data to it...

If you can read other people's /tmp files why can't you read their /proc/$$/fd
entries? (Did you know those aren't really symlinks? It's a synthetic
filesystem, if you can open them the kernel does a cross-process dup() of the
file descriptor into your new process's file descriptor table, a bit like that
old "hand a file descriptor across a linux domain socket" magic...) Either
filesystem permissions are safe or they aren't. (Unless this is a "panic the
kernel, rip out the hard drive, do forensics on the disk image" state actor
level threat mitigation attempt...?)

*shrug* I know security people, but I'm not NEARLY professionally paranoid
enough for that crowd. Yeah there's a band-aid over my laptop camera, but I
can't stop somebody pointing a yagi antenna at my screen through the window and
listening to my keypresses via the laptop's microphone. And I _always_ assume my
phone's listening to me. (Darn it, my electrical tape over the forward facing
camera came off...)

> In this case, adding the pipe was a couple of dozen lines of code.
> 
>> 
>> But doing pipes here seems like a microoptimization?
> 
> It's not the only consideration. Here-documents before any writable file
> systems are mounted, for instance. Look at the dueling requirements from
> my message and your own experience. IYKYK.

Indeed, the failure to open was how I first figured out how it was implemented. :)

But there's always a failure mode. File descriptor exhaustion. PID exhaustion.
memory exhaustion. I try to have error messages, but "test all your error paths"
remains a todo item for a lot of that. (In theory, with containers, you could
kinda rig up something EXTREMELY BRITTLE... but if it fails is it useful info or
a bad test? Always balancing conflicting goals...)

>>> Single quotes: preserved. Double quotes: removed when special. For
>>> instance, the double quotes around a command substitution don't make the
>>> characters in the command substitution quoted.
>> 
>> Quotes around $() retain whitespace that would otherwise get IFS'd. 
> 
> Correct, but that's behavior that affects how the output of the command
> substitution is treated, not how the substitution itself is parsed or
> executed.

They're the same thing for me: my parsing produces a result.

>> And command
>> substitution quoting contexts NEST:
> 
> Sure, in the sense that each command substitution has its own quoting
> context that starts out as `unquoted'. The original Bourne shell
> implementation of stuff like that (and stuff like double quotes inside
> ${...}) was just horribly broken, and ksh88 only fixed a couple of cases,
> so it took POSIX a while to reconcile all that.

I'm collecting test cases. I spent a number of months doing the "yes but what
if" dance when implementing variable expansion, and I blogged most of it because
I'd forget otherwise.

Seriously, my blog is my external brain. I keep meaning to go back and review my
blog entries to find all the design issues and test cases that got dropped along
the way. It's not _hard_, just tedious.

According to "git log toys/*/sh.c" my restart on the shell was
https://github.com/landley/toybox/commit/7fceed5f75c9 (replacing a stub shell
I'd written that was more or less a port of my old
https://git.busybox.net/busybox/commit/shell/bbsh.c?id=ef08184d9e0c4217 back in
the busybox days, which was roughly me reading lash.c and then the posix spec
and then printing out "the advance bash scripting guide" and getting like 20
pages in on the bus commute...) and that's dated June 25. Looking backwards in
my blog from there (which was the first checkin, not the start of work), it
looks like I embarassed myself about the state of the shell after proposing the
2019 ELC talk:

  https://landley.net/notes-2019.html#26-05-2019

And then got my nose rubbed in "adb shell" behavior shortly afterwards:

  https://landley.net/notes-2019.html#30-05-2019

And between the two of those, that got me to reopen the can of worms that is
implementing a shell on June 2:

  https://landley.net/notes-2019.html#02-06-2019

And then I got the bit caught in my teeth and kept grinding away at it, as you
do. More than half my blog entries for the rest of that month are about shell
design (mostly "oh hey, bash does THIS thing I didn't expect" from the point of
view of someone who knew just enough about bash to get it to do stuff), and it
continues on from there...

>>    echo -n "$(echo "hello $(eval $'echo -\\\ne \'world\\n \'')")"
>>    hello world
>> 
>> When I can't puzzle through it I just run lots of tests against all the corner
>> cases I can think of and try to retcon a general rule from the results...
> 
> The problem with doing that is that there were so many special cases in the
> original Bourne shell, Korn felt he had to preserve a lot of them for
> backwards compatibility. By the time POSIX came along, there were so many
> that special cases snuck into the standard. Sometimes trying to puzzle out
> a general rule works, sometimes it doesn't.

Indeed. I'm on year 4 of trying. :P (Albeit distracted by the other ~230
commands in toybox, plus the project has users who send me bug reports.)

>>> That's the `special' part.
>>> There's also the case of double quotes around the `new' word expansions
>>>
>>> ${parameter[#]#word}
>>> ${parameter[%]%word}
>> 
>> This part I don't know about, it looks like that's the prefix/suffix removal syntax?
> 
> Yes. `new' is relative, of course -- they were in ksh88. They just weren't
> in the SVR4 sh. But they're special, in that double quoting those 
> expansions doesn't mean the patterns are quoted, and you have an
> independent quoting context after the operator.

$ echo ${PATH//":"/xxx}
/home/landley/binxxx/usr/local/binxxx/usr/binxxx/binxxx/usr/local/gamesxxx/usr/games
$ echo "${PATH//':'/xxx}"
/home/landley/binxxx/usr/local/binxxx/usr/binxxx/binxxx/usr/local/gamesxxx/usr/games
$ echo "${PATH//"/"/xxx}"
xxxhomexxxlandleyxxxbin:xxxusrxxxlocalxxxbin:xxxusrxxxbin:xxxbin:xxxusrxxxlocalxxxgames:xxxusrxxxgames

Quoting contexts nest...

>> I note that I have yet to open the can of worms that is bash array variables,
>> although I've reserved plumbing for them in like five different places. (This is
>> mostly because I have not historically used them much, and thus don't have a
>> good handle on how to test it. But multiple people have said that's the biggest
>> feature they're looking forward to...)
>> 
>> (And "$@" is kind of array variable-ish already...)
> 
> Kind of, but it's not sparse. Support for very large sparse arrays is one
> thing that informs your implementation.

Oh goddess. (Adds note to sh.tests, which is my text file of cut and paste
snippets to look at later. Yes, my todo lists nest.) Is sparse array a new type
or are all arrays sparse?

The variable types I've currently got are:

// Assign one variable from malloced key=val string, returns var struct
// TODO implement remaining types
#define VAR_NOFREE    (1<<10)
#define VAR_WHITEOUT  (1<<9)
#define VAR_DICT      (1<<8)
#define VAR_ARRAY     (1<<7)
#define VAR_INT       (1<<6)
#define VAR_TOLOWER   (1<<5)
#define VAR_TOUPPER   (1<<4)
#define VAR_NAMEREF   (1<<3)
#define VAR_EXPORT    (1<<2)
#define VAR_READONLY  (1<<1)
#define VAR_MAGIC     (1<<0)

I think you know what INT, TOLOWER, TOUPPER, NAMEREF, and READONLY mean. (When
you make LINENO readonly, bash gets quite chatty!)

NOFREE means the "name=value" string supplied by the caller should NOT be freed
when the variable is, WHITEOUT is when you unset a local variable so the
enclosing scope may have an unchanged definition but variable resolution needs
to stop there and get the ${x:=} vs ${x=} part right), EXPORT means this gets
included when filtering the array returned by visible_vars() to assemble the
envp passed to child processes), and MAGIC means this variable's value is
calculated every time you resolve it. (But still has to be in the table because
you can unset or redefine a MAGIC. Or make them readonly. Or hide them behind a
local. You can also export them, and I have a number of blog entries about how
exported magics not getting updated between child process launches is user-visible:

  $ export SECONDS; while true; do sleep 1; env | grep SECONDS; done
  SECONDS=23
  SECONDS=23
  SECONDS=23
  SECONDS=23
  SECONDS=23
  SECONDS=23
  ...
  $ export RANDOM; while true; do sleep 1; env | grep RANDOM; done
  RANDOM=25047
  RANDOM=25047
  RANDOM=25047
  RANDOM=25047
  RANDOM=25047

And then the whole "export a local" business, which we talked about before...

Anyway, that leaves VAR_ARRAY, and VAR_DICT (for associative arrays). I take it
a sparse array is NOT a dict? (Are all VAR_ARRAY sparse...?)

Glancing at my notes for any obvious array todo bits, it's just things like "WHY
does unsetting elements of BASH_ALIASES not remove the corresponding alias, does
this require two representations of the same information? It's not just a normal
VAR_MAGIC? Do I need WHITEOUTS here? AAAAAHHHHHH."

(My notes often have variants of AAAAHHHHHH in them. It's my working style. The
Sam Vimes "vibrating with the internal anger of a man who wants to arrest the
gods for not doing it right" approach, except I'm too old to stay that mad
anymore and it often dulls to a vindictive grudge aimed at software, but eh,
close enough. Spite: it keeps you going.)

>> I remember being deeply confused by ${X at Q} when I was first trying to implement
>> it, but it seems to have switched to a much cleaner $'' syntax since? 
> 
> The @Q transformation has preferred $'...' since I introduced the
> parameter transformations in bash-4.4. I'm not sure when you were looking
> at it?

I stuck with the last GPLv2 release for longer than Apple did:

  https://news.ycombinator.com/item?id=18852887

At some point after I noticed Apple had replaced bash with zsh (I'm not a mac
user but some co-workers were), I switched to testing against and using the man
page of the version of bash preinstalled in my distro, and just acknowledged
that I didn't feel comfortable even glancing at the source code in github web
view anymore. (Not under that license.)

Did you know the move to GPLv3 essentially killed Samba? Jeremy Allison gave a
talk about it in 2020 at https://archive.org/details/copyleftconf2020-allison
where he mentioned regretting the move. Apple wrote their own
(https://www.osnews.com/story/24572/apple-ditches-samba-in-favour-of-homegrown-replacement/)
and the Linux kernel wrote its own in-kernel server
(https://www.kernel.org/doc/html/latest/filesystems/smb/ksmbd.html).

(Part of the reason I ended Aboriginal Linux in 2017, ala
https://landley.net/aboriginal/ was half the packages were stuck on the last
GPLv2 release of stuff and patching the linux kernel to still build with them
was getting awkward. And building newer Linux From Scratch under the result
couldn't skip the "build the toolchain" part because half the packages would
need patching to work with 10 year old tools. My general sunsetting horizon is 7
years and I was already well beyond that...)

>>>> Echo isn't processing any of these backslashes. Both bash and toybox echo need
>>>> -e to care about backslashes in their arguments. (Again, posix-2008 says
>>>> "implementations shall not support any options", which seems widely ignored.)
>>>
>>> They're not options, per se, according to POSIX. It handles -n as an
>>> initial operand that results in implementation-defined behavior. The next
>>> edition extends that treatment to -e/-E.
>> 
>> An "initial operand", not an argument.
> 
> That's the same thing. There are no options to POSIX echo. Everything is
> an operand. If a command has options, POSIX specifies them as options, and
> it doesn't do that for echo.

Hence the side-eye. In general use, echo has arguments. But posix insists it
does not have arguments. To so save face, they've created an "argument that
isn't an argument", and they want us to pretend that's not what they did.

"All options must come before non-option arguments" is a common use pattern,
echo isn't special in this regard. "Unrecognized options are passed through" is
another common pattern.

Heck, you want funky: "kill -stop" vs "kill -s top". It's passing through
unrecognized arguments to a later processing pass, and retroactively declaring
-s as unrecognized because -t isn't a thing. Or "ls --color auto" vs "ls
--color=auto"... But again, there _is_ a regular pattern behind it...

>> Right. So they're going from "wrong" to "wrong" then:
>> 
>>    $ echo -n -e 'hey\nthere'
>>    hey
>>    there$
> 
> Yeah, echo is a lost cause. Too many incompatible implementations, too much
> existing code. That's why everything non-trivial (like the above) is
> implementation-defined. POSIX recommends that everyone use printf.

  $ printf abc\n
  abcn$

Oh yeah, that'll happen.

>> Maybe posix should eventually break down and admit this is a thing? "ls . -l"
>> has to work, but "ssh user at server -t ls -l" really really REALLY needs that
>> second -l going to ls not ssh.
> 
> Why do you think they don't acknowledge this today?

  https://landley.net/notes-2016.html#11-03-2016

Oddly enough, you can just about pull that thread out of:

https://web.archive.org/web/20170408052208/http://comments.gmane.org/gmane.comp.standards.posix.austin.general/12181

(Yes, I'm aware of recent changes. That's why I re-engaged with Posix, felt I
owed it to them since the condition under which I said I'd come back
unexpectedly happened. But having already written them off, my heart really
wasn't in it. I _should_, but I'm juggling too many other balls...)

> Options only exist as
> such if they come before the first non-option argument.

  $ cat <(echo hello) -E
  hello$
  $ diff <(echo one) <(echo two) -u | sort - -r | wc
      5      16     124

> Options have to
> begin with `-'.

  tar tvzf blah.tgz
  ps ax
  ar t /usr/lib/libsupp.a

> So in your example, the -t isn't an option to ssh; it's
> ssh that breaks the POSIX guidelines by accepting it as an option instead
> of a command name.

Very little DOESN'T break the posix guidelines. A rule that's almost never
honored isn't a very good rule, and toybox handles all the above examples with
generic logic. (The option position one is SO generic it's the default, and
there's a character to switch it _off_.)

> In the POSIX world, the -t is not an option -- it's an
> operand that ssh happens to treat like an option. If POSIX were to take
> ssh under consideration for standardization, it would probably make it an
> application requirement that ssh options appear before operands (since
> that's an existing utility syntax guideline).

You can chain "ssh xargs strace nice unshare setsid timeout prlimit chroot"
arbitrarily deep, and each command has its own arguments and then a command line
it execs, which can itself contain arguments. That's usually WHY a command cares
about argument position.

Meanwhile "tar Cx .. README -Ofblah.tgz"  exists, hasn't changed how it works
since I learned it in the 90's, and the rules are standard enough that I got
generic plumbing to handle it.

> If you really want to go
> hardcore, require that the application (user) supply a `--' before the
> remote command and its arguments if you want to use it in this way.

But what's already there works, and has for decades.

A good standards body should document, not legislate.

>> And yes, my echo parses initial -- the same way
>> every other command that parses any arguments does, 
> 
> OK. Bash doesn't. POSIX doesn't.

I know. :(

>> Is it more important for the toybox
>> commands to be consistent with each OTHER, or for them to be consistent with
>> other implementations? Navigating conflicting wossnames. It's a thing.
> 
> Right. Is internal or external consistency more important? You get to make
> that call.

Sometimes I try for both! A couple years back I added -O, -D, and -F options to
toybox cut (partly because I don't have awk implemented yet, but also cut SHOULD
be able to do that), so you can go:

  $ echo one two three four five | toybox cut -DF 2-4,3
  two three four three

The -F is regex search (defaulting to runs of whitespace ala IFS), the -D
disables the sort/uniq on the arguments, and then -O is a cleanup (lets you
change the output delimiter, -d changes the input delimiter and yes -F listens
to -d which is not limited to a single character).

Elliott (the Android base OS maintainer) asked if I could get that pushed into
other implementations (so if their scripts started depending on it, it was more
widely available), so I submitted a patch and explanation to the Denys (the
busybox maintainer), who merged it after some minor discussion:

  http://lists.busybox.net/pipermail/busybox/2021-July/088983.html
  https://git.busybox.net/busybox/commit/?id=0068ce2fa0e3

And then I submitted a feature request to coreutils:

  https://lists.gnu.org/archive/html/coreutils/2022-01/msg00004.html

Which resulted in a lot of discussion, and an eventual decision to include it,
and some patches were discussed:

  https://lists.gnu.org/archive/html/coreutils/2022-01/msg00048.html

And then when it wasn't in the next release, they said it was "still in
development":

  https://lists.gnu.org/archive/html/coreutils/2022-04/msg00010.html

And then a year later it was still on their todo list:

  https://lists.gnu.org/archive/html/coreutils/2023-02/msg00012.html

This sort of thing consumes my "engaging with bureaucracy" meter. Whether it's
wrangling with linux-kernel or posix or gnu: I'm fine with talking about the
code and fixing the code and testing the code, but when all that's a small part
of the work I get... very tired. See "not good at the politics"...

>> For the shell though, I plan to document my deviations from BASH. Because that's
>> my standard.
> 
> Sure, I appreciate that. You should pick a particular version for your
> compatibility target, because then you can refer to bash and its
> documentation of how it differs from POSIX.

Reality is a moving target. :)

>> I have not committed to implementing 100% of what bash does. It's beyond 80/20,
>> but whether it's 2 iterations of 8/20 (96%) or 3 (99.2%) I dunno yet. Somewhere
>> between, probably... (https://landley.net/notes-2021.html#30-09-2021 was
>> promising 3 but it's not an exact science.)
> 
> That is, of course, your call.
> 
>> 
>> Implementing -p mode might be in the "two iterations" part. 
> 
> Behavior differences when invoked setuid? Or `-o posix'?

Blah, I have a strong tropism for short opts and misremembered -p as short for
--posix but it isn't. (There's also POSIXLY_CORRECT= or "set -o posix".)

>> Actually passing the
>> posix test suite (where does one even GET that? Do you have to pay for it?)
> 
> I asked for it. I have a limited-use, limited-time license. I'm probably in
> a different position vis a vis The Open Group than you are, though.

http://www.opengroup.org/testing/downloads.html says there's a no-fee license.
Maybe closer to the 1.0 release I'll jump through the hoops to help me document
my deviations?

But posix compliance doesn't really _get_ me anything, except bragging rights.
"It successfully ran several thousand package builds and a large user base has
had opportunity to complain" does.

I have a bunch of stuff on my todo list like checking if
https://linux-test-project.github.io/metadata/metadata.stable.html has any
relevant command line tests in it, but really that falls under the "does this
package run in a toybox+musl+kernel-I-built environment" as just another package
to try to make work...

>>> This is completely unspecified behavior.
>> 
>> The standard is not complete, yes.
> 
> A different interpretation. There's plenty of unspecified and
> implementation-defined behavior.

Bash is an implementation, defining behavior. There may be version skew, but it
does something specific. I just have to think of what questions to ask.

>>> POSIX shell scripts are text
>>> files, which consist of lines, and lines end with newlines. It's up to each
>>> shell implementor to decide how to handle it. You can push for an extension
>>> to that, but I would not hold my breath.
>> 
>> I don't care what posix says, I care what bash does.
> 
> I understand that. I'm telling you what might happen if you went for an
> interpretation request.

A thing I have done from time to time, but... an expensive thing, as far as
spoons go:

  https://en.wikipedia.org/wiki/Spoon_theory

>>> but any command
>>> substitution is always going to remove trailing newlines, quoted or not.
>>> We had a pretty good argument about that trick on the POSIX list, too, but
>>> most of the objections are theoretical.
>> 
>> I'm not trying to make it work on solaris or AIX. And only a subset works on BSD
>> or MacOS...
> 
> It's a locale thing, not necessarily an OS thing. It works in UTF-8
> encodings, so it's basically universal enough.

My toupper and tolower stuff is doing utf8, but my isspace() isn't skipping UTF8
spaces yet. And my command history editing plumbing was hung up on working out
the unicode space traversal stuff (now hopefully resolved but not implemented).

Is there a way to hurt the unicode developers for putting combining characters
AFTER the characters they combine with? UTF-8 is brilliant, unicode is just
PAINFUL. In order to flush you have to read PAST what you want to display and
then ungetc(). When you read and it ends with a combining character you NEVER
KNOW IF YOU ARE DONE.

(See "AAAAAAAAHHHHHHH", above.)

>>>>> There's genuine disagreement between shells here. The ash-based shells
>>>>> (dash, the BSD sh, gwsh) preserve the backslash. Bash through bash-5.2,
>>>>> yash, mksh, ksh93 all remove it.
>>>>
>>>> See "pining for standards", above.
>>>
>>> File an interpretation request. I'm going to do what I think makes sense.
>>> Be prepared to have it rejected, though.
>> 
>> I'm treating bash as my standard here, not posix.
> 
> Sure, then you get to be pissed off when I decide it's a bug and change it.

I'm used to it. :)

> If you're pining for a standard, file an interpretation request against
> something that claims to be one. Or stick a stake in the ground and say
> bash-x.y is your standard. But "bash" is going to be a moving target.

It's the 80/20 thing again. The vast majority of bash is the same as it was in
2.05b. Some stuff's been added and is unlikely to be removed (ala =~). Some
things changed clearly for the better (declare -p outputting $'whole-thing'
escaping).

And then there's some noise and jitter. Happens.

(The more I engage with bash the more stuff I thought I was going to exclude
winds up on the todo list...)

>> I would like there to BE a standard, but do not believe Posix can ever become it
> 
> Maybe not, but there's no other alternative.

Currently. Posix didn't always exist, the Linux Standard Base was making good
progress until the Linux Foundation's accretion disk swallowed it, man7.org was
decent until Michael Kerrisk retired and handed off to a guy who doesn't
maintain a current web version...

Did you know I mantained https://kernel.org/doc as my day job for 6 months in
2007? The downside of maintaining standards is you need expertise that provides
coverage, which is hard to do. I couldn't do it for the kernel because I don't
KNOW enough about the kernel (even Linus doesn't). But I've just about gotten to
the point where I can describe what toybox is doing. :)

> There's not going to be a lot
> of momentum among other shell authors to "do it like bash does." For a
> long time, the opposite was true, especially among the austin-group
> members.

For years the de-facto spreadsheet standard was Microsoft Excel and the word
processing file format standard was Microsoft Word. They SUCKED, but had vastly
dominant market share. And every weird corner cases of their behavior was part
of that standard.

Then Star Division cloned compatible versions that could read and write those
files in Star Office, which Sun Microsystems purchased renaming it Open Office,
which Oratroll purchased and the people who fled screaming renamed Libre Office.
And the files they WROTE could be read by anything, and the closer those got to
being able to read any arbitrary output of microsoft's versions, the more
pressure there was on people not to trigger the remaining corner cases. And also
a certain amount of "don't use the new versions, use the old versions" because
compatibility. This allowed Writely (which Google purchased and renamed Google
Docs) to implement reasonably compatible read and write support for the file format.

This is the same way that llvm (and intel's icc, and for a brief while tinycc)
got to build the linux kernel: most of it was llvm implementing gcc extensions,
a little bit was linux filing off a few rough edges. A similar thing happened
with all the "optimized for IE" and "optimized for netscape" pages on the web.
Standards emerged documenting the reliable subset, but plenty of people went
past them (and the whole "flash" nonsense). Konqueror cloned enough that both
chrome and safari are descended from it...

The point is, once you have two independent implementations, the subset both
support becomes a lot more standard-shaped. This was the IETF way back in the
day, "rough consensus and running code". The bake-offs were to get multiple
interoperable implementations. You NEED two, or this doesn't work. :)

Some academic document produced by a committee so that everyone everywhere has
to adjust to keep up with it is not very interesting to me.

>> I've engaged with the posix guys when there was obvious missing info, ala:
>> 
>> https://landley.net/notes-2014.html#02-12-2014
>> 
>> But one of my big pet peeves about them is they haven't got an actual stable web
>> archive so discussions that happen on the mailing list are NOT A RELIABLE RECORD
>> of anything.
> 
> They never have been. Mantis is the only official record.

So they spend their time trying to figure out what people actually said way back
when, and then refuse to retain their OWN history.

> I don't know whether or not the austingroup-bugs list has a reliable
> archive.

They do not. There were third party ones, but they went down. :(

> I keep my own of things I'm concerned about. But I've been at it
> longer than you have.

Quite likely. I went commodore 64 -> amiga -> DOS -> OS/2 -> linux in the Red
Hat 5 era. I remember the SLS disk images coming across fidonet and couldn't
understand why anyone would clone a sun workstation. It couldn't run DOS
programs (yet), and no computer my friend and I tried to install it on could get
greater than 320x200 resolution out of x11. That was... 1993? Maybe 94?

I didn't cycle back to Linux seriously until 1998...

>>>> (The downside to using bash as a standard is when I ask you about corner cases,
>>>> half the time you fix things. Not a downside for YOU, but I'm left with a moving
>>>> target. https://threeplusone.com/quotes/pratchett/ .)
>>>
>>> We talked about this before. Pick a fixed target (e.g., bash-5.1) and write
>>> to that, then move forward if you like.
>> 
>> I did, it was bash-2.05b and I had to move forward to run "emerge".
> 
> What are you using now?

$ bash --version
GNU bash, version 5.0.3(1)-release (x86_64-pc-linux-gnu)

If I can get the Linux From Scratch build automated, I can presumably drop in
source release tarballs and stay a bit more current. My tests all do
"TEST_HOST=1 make test_sh" so I can regression test against different bash
versions...

>> P.S. I'd really hoped I could get a reasonable shell in 3500 lines. The version
>> I checked in today is 4762 lines. Not exactly dunning-kruger, but there's always
>> a certain amount of learning on the job. If I knew what I was doing I'd be done.
> 
> It took Bourne over 5000, but with a considerably less functional C library.

I've implemented my own glob() but am using libc's regex. I'm testing on glibc,
musl, and bionic (android's libc), which MOSTLY agree these days. (There's also
some testing on freebsd and macos, but only a subset of stuff is ever likely to
work there.)

> Chet

Rob