[Toybox] bash continues to confuse me.

[Chet Ramey]

On 6/18/20 7:48 PM, Rob Landley wrote:
> On 6/18/20 1:46 PM, Chet Ramey wrote:
>> On 6/17/20 1:22 PM, Rob Landley wrote:
>>> Trying to figure out when spaces are and aren't allowed in ${blah} led to asking
>>> why echo ${!a* } is an error but ${!a@ } isn't (when there are no variables
>>> starting with a), and I eventually worked out that:
>>>
>>>   $ X=PWD
>>>   $ echo ${!X at Q}
>>>   '/home/landley/toybox/clean'
>>>
>>> Is going on? 
>>
>> It's variable transformation. It's introduced by `@' and uses single-letter
>> operators. I cribbed the idea from mksh and extended it. The syntax is kind
>> of loose because I'm still experimenting with it.
> 
> I know what the @Q part does:

OK, sorry.


> It's the circumstances under which ${!abc***} falls back to processing the ***
> part that I'm trying to work out.
> 
>   $ ABC=123; echo ${!ABC:1:2}
>   21
> 
> I _think_ what's happening is when you do ${!ABC@} it specifically checks for
> "@}" after the variable name, and if so it lists prefixes. (And [@] does the
> array thingy, haven't checked if that needs the } yet.) But if it doesn't end
> with THAT SPECIFIC SEQUENCE, it does the !ABC substitution FIRST and takes
> whatever's left after the original variable name (using normal "$WALRUS-x" logic
> to figure out where it ended) and does further slicing to do on _that_ result.

More or less. The ${!var@} expansion is a special case, no question.  What
it does is pretty much what I described in my previous message, which you
end up quoting below:

Start at the character following `{' and read to the end of the parameter.
Look at the character you're on, and dispatch depending on that. If it's
`@', do one more character of lookahead to see if you've got a prefix
operator or a possible variable transformation.

Once it figures out what the parameter part is, it does just what the man
page says:

"If the first character of parameter is an exclamation  point  (!),  and
 parameter is not a nameref, it introduces a level of indirection.  Bash
 uses the value formed by expanding the rest of parameter as the new pa-
 rameter;  this  is  then expanded and that value is used in the rest of
 the expansion, rather than the expansion  of  the  original  parameter."

> (I understand the implementation reason. I don't understand the _design_ reason.
> As a language... why?)

I don't really care about the design reason. Whatever happened did so more
than 40 years ago, and it's a waste of effort to worry about it.

> 
>>>   $ echo ${PWD:1:3 at Q}
>>>   bash: PWD: 3 at Q: value too great for base (error token is "3 at Q")
>>>   $ echo ${PWD:1 at Q}
>>>   bash: PWD: 1 at Q: value too great for base (error token is "1 at Q")
>>
>> What's wrong with that? `@' and `Q' are valid characters in numeric
>> constants when the base is large enough to need them. You can use
>> bases up to 64 in base#constant.
> 
> I.E. most expansions do not nest. The fact ${!x} does nest is an exception, and
> ${!x@} is a special case within that exception.

Yes, variable indirection is an exception.

>>> Hmmm...
>>>
>>>   $ echo ${!potato at walrus}
>>>
>>>   $ echo ${!P at walrus}
>>
>> Invalid transformations just expand to nothing rather than being errors.
>> That's part of what I'm still experimenting with.
> 
> What is and isn't an error is not consistent, yes. I STILL haven't found an
> answer to my first question, which is what was the difference between:
> 
>   $ echo ${!potato* }
>   bash: ${!potato* }: bad substitution
>   $ echo ${!potato@ }
> 
>   $

It really is the variable transformation. I'm not sure why you won't
believe the answer. The debatable part is whether or not to short-circuit
when the variable transformation code sees that the value it's being
asked to transform is NULL, but that's what mksh does and one of the
things I picked up when I experimented with the feature.


>>> Ok, when X exists and points to another variable, then !X becomes the contents
>>> of that other variable and THEN the @ cares about trailing garbage. But * is a
>>> different codepath from @...?
>>
>> It's a different expansion.
> 
> In my code they're the same codepath with different $IFS behavior.

OK. That's probably going to turn out to be insufficient.

 There are 3
> places in IFS expansion that check for '*' to distinguish it from '@' during
> array-style IFS expansion:
> 
> https://github.com/landley/toybox/blob/master/toys/pending/sh.c#L998
> https://github.com/landley/toybox/blob/master/toys/pending/sh.c#L1024
> https://github.com/landley/toybox/blob/master/toys/pending/sh.c#L1041
> 
> And that third one should really just be checking *sep set by the first one.
> (And possibly the second one should be too. If IFS starts with an invalid utf8
> sequence it won't _set_ sep, which affects the second but not the third check,
> but I should fix that in the first check's body...)
> 
> Anyway, my code's in flux by trying hard to use common codepaths. Which is much
> easier when there's consistent behavior.
> 
>>> And when the ${!var} doesn't have contents pointing to another existing
>>> variable, it falls back to a codepath that tries the prefix stuff (and/or the
>>> array stuff, in some order?) and that codepath tries to parse trailing @Q and
>>> friends, but DOESN'T error out if it can't recognize the trailing stuff after the @?
>>
>> Not really, no. It grabs the parameter, which may begin with `!', looks at
>> the character following it, does one character of lookahead if that
>> character is `@', and branches to the appropriate thing: variable prefix
>> expansion or indirect expansion with the result being used as the parameter
>> for the rest of the expansion.
> 
>   $ chicken() { echo ${!*};}; fruit=123; chicken fruit
>   123
>   $ xx() { echo "${!@:2: -1}";}; yy=abcdef xx yy
>   cde

The character that ends the parameter is `@' and the character after that,
which determines the expansion, is `:'.

> 
> The tricky part is being sure:
> 
>   $ xx() { echo "${!@}";}; yy=abcdef xx yy
>   abcdef
> 
> is doing the "indirection through $@" thing not the "list all variables with an
> empty prefix" thing. (I.E. getting the special case checking in the right order.)

There is no such thing as a variable with an empty prefix. The parameter
is !@; the ! introduces indirection, and the shell is left to do what it
can with the `@'. What it does is expand it in a context in which word
splitting does not take place, like on the rhs of an assignment statement.



>> Invalid transformation operators just expand to nothing.
> 
> Some things error, some things expand to nothing, If there's a pattern I don't
> understand it yet. 

In this case, it's how mksh treats it and whether or not I wanted that much
compatibility when I implemented it. So far, I've come down on the side of
compatibility.

On the other hand, if my code handles cases yours errors on,
> I'm not exactly being incompatible, am I?
> 
> I mean I'm assuming:
> 
>   $ ABC=def:1
>   $ def=12345
>   $ echo ${!ABC}
>   bash: def:1: bad substitution
>   $ echo ${def:1}
>   2345
> 
> not working is maybe some sort of security thing,

`def:1' is not a valid parameter name. In the second `echo', `def' is the
parameter name.

 although it makes:
> 
>   $ x=?; echo ${!x/0/q}
>   q
>   $ x=^; echo ${!x/0/q}
>   bash: ^: bad substitution
> 
> a bit harder to get right since as I said, my code for recognizing $? is in
> expand_arg() and not in varlen() or getvar(). 

There are valid parameters, and there are invalid variable parameters.

But it expand_arg seems to be
> where it belongs because there's a bunch of:
> 
>   $ declare -p 'x'
>   declare -- x="^"
>   $ declare -p '?'
>   bash: declare: ?: not found

Do you think declare should do syntax checking on whether or not its
arguments are valid identifiers? Even given that function names don't
have to be valid identifiers?


> incompatible sets of plumbing for doing the same thing:
> 
>   $ walrus='?'
>   $ declare -n walrus
>   bash: declare: `?': invalid variable name for name reference

Yeah, that's what ksh93 does there. It rejects the attempt to make walrus
a nameref, but it's still exists as a variable.

>   $ echo ${!walrus}
>   1

So the variable indirection works.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet at case.edu    http://tiswww.cwru.edu/~chet/