[Toybox] Would someone please explain what bash is doing here?
Rob Landley
rob at landley.net
Sun Mar 8 07:53:01 PDT 2020
On 3/7/20 3:38 PM, Chet Ramey wrote:
> On 3/6/20 9:05 PM, Rob Landley wrote:
>> It turns into echo "=a" "" "bf=" and when I do that from the command line I get
>> a space before and a space after the NULL argument? But with bash there are
>> three? Is it turning into two NULL entries? Ah, it SHOULD do that because there
>> are two consecutive non-whitespace IFS separators that don't bind to an existing
>> string (like the first one does). So why _isn't_ mine doing that...
>
> Your analysis is pretty much spot on.
>
> The first thing, as you already noted, is that the argument to `echo' is
> also being split on $IFS. If you quote that, you see that what you get out
> of the for loop is
>
> =axyxbf=
> ==
> =abc=
>
> That's because the $@ expands to three arguments, concatenated to what
> comes before and after, as described in
>
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_02
>
> which can be further split, under certain circumstances. POSIX isn't
> exactly precise about which expansions the `word list' following `in'
> undergoes ("the list...shall be expanded"), but those don't include word
> splitting, so this isn't one of those circumstances.
I read through the posix shell bits long enough ago it was probably SUSv3 rather
than v4, but at the moment I'm taking bash as my standard and just doing
whatever that does. I'm not looking at the bash source but I'm _scrutinizing_
the bash man page and testing all the corner cases I can think of under both
4.4.12 and 2.05b. (I still need to collate the attached notes with
https://github.com/landley/toybox/blob/master/tests/sh.test, and together
they're about 1/3 of the tests I need).
I should do another pass reading posix afterwards, but after
https://landley.net/notes-2016.html#11-03-2016 I've been much less interested in
interacting with the posix committee due to the risk of another Schilling, and
have pretty much backed up to
https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/ in much the same
way Debian backed up to LSB 4.1 ala https://lwn.net/Articles/658809/
I still _sort_ of care about newer posix, but I got {bracket,expansion} working
last year and last month taught my $IFS splitting to understand utf8 characters
(and have a TODO item that if IFS is an array it should understand strings), and
I honestly don't expect to live long enough for either NOT to be a divergence
from Posix.
Sorry to bother you, I should have (and eventually did) figure this out for myself.
Rob
-------------- next part --------------
https://bash.cyberciti.biz/guide/Select_loop
/*
alias bg cd command fc fg getopts hash jobs kill read type ulimit umask unalias wait exit if while for case export set unset trap exec function source
TOY_MAYFORK: echo test printf
no memory allocations, filehandles open... if signaled, no cleanup.
maybe: help ulimit
.toysh_history file
{abc,def}
$LINENO $RANDOM
"source" and "." check current dir after $PATH
job control: CTRL-Z, CTRL-C, fg, bg, kill %jobspec, wait
$(command)
$((math)) # integer attribute for variables, assignment does that
$((abc"def)) the " doesn't take effect so )) is recognized?
if then elif else fi
for while until do done
case esac
function () { commands ; }
local variables in functions
set -x (and -x command line option) to trace
export -n (unexports)
readonly
read set unset umask
time (whole pipeline)
command trap type times getopts
todo: echo | (cat) # as far as cat's concerned previous end was NULL
What to do about builtin vs $PATH:
check $PATH first _then_ builtin?
config opt?
if files exist use 'em, otherwise:
/dev/fd/fd stdin stdout stderr tcp/host/port udp/host/port
exec redirects in current shell
n<word
n>word
n>>word (append)
&>word # stdout and stderr, same as >word 2>&1
&>>word (append)
>&word # note: >&$BLAH when $BLAH is number or -
n<&word # number = duplicate, - = close, number- = move
n>&word # special case: no n and word not number or -, redirect stderr & out.
n<<word # only quote removal, no expansion
# unquoted: parameter ex, command subst, arith ex, \\ \$ \` \newline
<<-word # strips leading tabs
<<<word # all expansions _except_ pathname expansion and word splitting
n<>word # open file word for reading & writing, create if necessary, n def 0
// &> or >& = >word 2>&1
// &>> append stdout & stderr == >> word 2>&1
// <&# copy input fd (error if # not open for read)
// >&# copy output fd (error if # not open for write)
// <&#- move fd# to n
// >&#- move fd# to n
What does this mean?
env {abcdef}>&2 | grep abcdef
Instead of number, precede with {varname}?
brace expansion, tilde expansion, parameter and variable expansion, command substitution,
arithmetic expansion, quote removal, pathname expansion, and word
splitting.
<< no path ex or word split
test should work as a nofork, handle [[
what does $(( ))
handle ( ( echo hello ) ) | cat and (echo)|cat
if (then)
else elif (then)
fi
for select while until (do)
done
case (esac)
( )
{ }
[[ ]]
}
// function
// HERE document
// ( ) - ) is always last argument of line
// | & && || |&
// "if", "for", "while", "until", "select"
// "do" : "then";
// then else elif fi
// continue break case esac return do done function
// in time { } ]] ]]
// ! coproc
// function ()
// . source [ test [[ ( { ((
// [ pwd ulimit -- kill time
// cd exit -- pushd popd logout umask getopts eval exec
// fg bg jobs disown wait suspend
// alias unalias set export unset let local readonly read shift trap
# Because of HERE documents we can't re-parse, have to retain state.
# -- either that or our caller has to know about here document servicing?
# -c input must be split into lines for HERE documents
< > >> <<< WORD
<< HERE [HERE] # eat lines until HERE as _only_ word on line
if [then]
for while until select [do]
case [esac]
( [)] # funky because ) not first word
{ [}]
[[ []]]
function NAME ( ) { # note expect { doesn't eat { be because need to expect }
/*
4 cases:
redirect now: from -> to, saving displaced to (+, +) = hfd to
explicitly close (now), saving displaced to (-1, +) = hfd to
close in child only: (+, -1)
redirect saving to, close saving from: (+, +) = hfd to then (-1, +) = hfd to
{var} leak - nothing to save? Except the from open is deferred (+, +) but to == hfd
*/
// TODO job control: & backgrounding
// TODO && ||
// TODO | |&
// TODO add redir segment, redir pipes[1] to pipes[0] with close, pop again later
// TODO pipe segments are subshells ala ( )
// TODO: don't () around single process pipeline, shell otherwise
// this turns into exec for sh -c "echo"
// single pipe segments that _aren't_ builtins run directly,
// compound pipe segments and builtins run via subshell.
// I.E. subshell implicit exec
// TODO we're not running a command, we're running a block stack?
// echo one two three | while read i; do echo hello; done
// TODO free/cleanup partial pipeline on NULL return?
// TODO -o pipefail
// TODO pipes don't connect commands, they connect arbitrary function chunks.
*/
# can't have space before first : but arguments can have lead/trail
$ BLAH=abcdefghi; echo ${BLAH: 1 : 3 }
# leading assignments don't affect current command line
$ VAR=12345 echo $VARa
$ bash -c 'ls $('
bash: -c: line 0: unexpected EOF while looking for matching `)'
bash: -c: line 1: syntax error: unexpected end of file
$ ;
bash: syntax error near unexpected token `;'
$ ABC= ; env | grep ABC= ; unset ABC ; env | grep ABC=
ABC=
$ cat blah<(echo hello)thing
cat: blah/dev/fd/63thing: No such file or directory
$ "AB"="CD" echo $AB
bash: AB=CD: command not found
landley at driftwood:~/toybox/toybox$ AB="CD" echo $AB
$ 'AB'="CD" echo $AB
bash: AB=CD: command not found
$ \AB="CD" echo $AB
bash: AB=CD: command not found
$ \AB="CD" echo $AB
$ if echo hello; then fi
bash: syntax error near unexpected token `fi'
$ echo )
bash: syntax error near unexpected token `)'
$ ( echo hello ) | cat
hello
$ if
> true
> then
> echo hello
> fi
hello
$ if true; then echo hello; fi
hello
$ if false; then echo hello; fi
$ false; X=47; echo $?
0
$ if;
bash: syntax error near unexpected token `;'
$ if
> true
> echo
> hello
> fi
bash: syntax error near unexpected token `fi'
$ echo )))
bash: syntax error near unexpected token `)'
$ echo &&&
bash: syntax error near unexpected token `&'
$ echo |||
bash: syntax error near unexpected token `|'
$ echo hello | if true; then read i; echo i=$i; fi
i=hello
$ if true && false; then echo hi; fi
while true ; do echo hello; done | tee blah
if { echo hello; }; then echo hi; fi
if if true; then true; fi; then echo hello; fi
boom(){ echo hello; }; boom
( ( echo; thingy ) ) | cat
$ echo hello | X=y env | grep -w X; echo $X
X=y
;|x
;x|
(echo)also
(echo)(echo)
$ if true esac; then echo hi; fi
hi
# note: exits top level shell anyway!
$ (exit walrus)
bash: exit: walrus: numeric argument required
$ echo $?
2
$ exit walrus 2>blah.txt
$ echo ${abc:?error m essage}
bash: abc: error m essage
$ if blah () { echo bang; true; }; blah; then echo hello; fi; blah
bang
hello
bang
$ meep() { echo hello; klarg () { echo helloier; }; klarg; }
$ meep
hello
helloier
$ meep() { echo hello; klarg() { echo helloier; } ; }
$ klarg
bash: klarg: command not found
$ meep
hello
$ klarg
helloier
$
landley at driftwood:~/toybox/toybox$ thingy ()
> ^C
landley at driftwood:~/toybox/toybox$ echo (
bash: syntax error near unexpected token `newline'
landley at driftwood:~/toybox/toybox$ function
bash: syntax error near unexpected token `newline'
landley at driftwood:~/toybox/toybox$ function name () {
> }
bash: syntax error near unexpected token `}'
landley at driftwood:~/toybox/toybox$ function name () { ;}
bash: syntax error near unexpected token `;'
landley at driftwood:~/toybox/toybox$ if ; then; echo hello; fi
bash: syntax error near unexpected token `;'
landley at driftwood:~/toybox/toybox$ func (); { echo hello; }
bash: syntax error near unexpected token `;'
$ if true |
> then
bash: syntax error
echo one && if true; then echo hello; fi
one
hello
$ echo one two \
> three
$ bash -c "$(echo -e 'cat << HERE\none two\nHERE')"
one two
$ cat << HERE; echo hello
> boing
> HERE
boing
hello
$ cat << one << two
> abc
$ ls > file 2>&1 #redirects both
$ ls 2>&1 >file # redirects only stdout, stderr goes to original stdout
$ echo hello < nowhere
bash: nowhere: No such file or directory
$ cat << E"O"F
> $PATH
> EOF
$PATH
$ cat << HERE filename
> HERE
cat: filename: No such file or directory
$ cat <<< here filename
cat: filename: No such file or directory
landley at driftwood:~/toybox/toybox$ bash -c "cat <<< HERE; echo hello; HERE"
HERE
hello
bash: HERE: command not found
landley at driftwood:~/toybox/toybox$ bash -c "cat << HERE; echo hello; HERE"
bash: warning: here-document at line 0 delimited by end-of-file (wanted `HERE')
hello
bash: HERE: command not found
landley at driftwood:~/toybox/toybox$ bash -c "$(echo 'cat << HERE\necho hello\nHERE')"
bash: warning: here-document at line 0 delimited by end-of-file (wanted `HEREnecho')
cat: hellonHERE: No such file or directory
landley at driftwood:~/toybox/toybox$ bash -c "$(echo -e 'cat << HERE\necho hello\nHERE')"
echo hello
landley at driftwood:~/toybox/toybox$ X=3; echo $((X))
3
landley at driftwood:~/toybox/toybox$ X=3; echo $((X))^C
landley at driftwood:~/toybox/toybox$ echo |
>
> cat
landley at driftwood:~/toybox/toybox$ echo &&
> &&
bash: syntax error near unexpected token `&&'
landley at driftwood:~/toybox/toybox$ echo &&
> ;
bash: syntax error near unexpected token `;'
landley at driftwood:~/toybox/toybox$ if
> true
> then
> echo hello
> fi
hello
landley at driftwood:~/toybox/toybox$ << EOF
> echo hello
> EOF
$ echo &<2
[1] 17850
bash: 2: No such file or directory
landley at driftwood:~/toybox/toybox$
[1]+ Done echo
landley at driftwood:~/toybox/toybox$ { echo hello; }
hello
landley at driftwood:~/toybox/toybox$ { { echo hello; };}
hello
landley at driftwood:~/toybox/toybox$ {{ echo hello; };}
bash: syntax error near unexpected token `}'
landley at driftwood:~/toybox/toybox$ { {echo hello; };}
bash: syntax error near unexpected token `}'
landley at driftwood:~/toybox/toybox$ { { echo hello; };}
hello
landley at driftwood:~/toybox/toybox$ {
> {
> echo hello
> }
> }
hello
$ if cat << EOF ; then
> one two three
> EOF
> echo hello
> fi
one two three
hello
cat <(ls "$PWD"/a{b,c}*) &> /dev/tcp/127.0.0.1/80
$ while grep -q << EOF walrus; do
> walrus
> EOF
> echo hello; done | head -n 3
hello
hello
hello
$ if echo hello; then echo also; fi | tee and.txt # both get redirected
hello
also
$ cat - << EOF <(echo
> hello)
> boing
> EOF
bash: hello: command not found
boing
$ ( ( ( echo abc ) echo ) )
bash: syntax error near unexpected token `echo'
$ ( ( ( echo abc )
> echo also ) )
abc
also
$ ( ( echo one ) > file )
$ ( ( echo one ) > two ) > three
$ ( echo ) >> blah
$ if true; then echo; fi >> blah
$ cat - /proc/self/fd/3 << BOING 3<<MEEP
> one
> BOING
> aha
> MEEP
one
aha
$ cat - << BOING $(
echo hello)
abc
BOING
abc
blah
$ ( cat << EOF
> hello
> EOF
> )
hello
$ if cat << EOF
> hello
> EOF
> then
> echo fi
> fi
hello
fi
$ echo 2>&1
$ echo 2 >&1
$ echo 2 >& 1
$ if while true; do echo hello; done; then echo hi; fi
$ (echo &&)
$ (echo ;)
$ if true; then echo $X; fi {X}</dev/null
10
$ X=2; {X}<&-; boing # does not close stderr?
bash: boing: command not found
$ ls /proc/$$/fd
$ exec 10<& -
# >&; ;&>
# prompts before echoing
$ echo hello; if
> echo two; then echo three
> fi
hello
two
three
$ echo )hello
bash: syntax error near unexpected token `)'
$ echo {abc
{abc
$ echo {abc</dev/null
{abc
# innermost redirect wins
$ if cat <<< moo ; then cat <<< also; fi <<< potato
moo
also
$ (echo &&)
bash: syntax error near unexpected token `)'
$ funkiness() { env ; }
$ POTATO=blah funkiness | grep POTATO
POTATO=blah
$ echo $POTATO
$ chicken() { echo hello; chicken() { echo also ;}; chicken;}
$ chicken
hello
also
$ chicken
also
$ thingy() { echo hello; } | cat
$ thingy
bash: thingy: command not found
$ echo hello > >(sed 's/.*/abc&def/') > >(sed 's/.*/ghi&jkl/')
abcghihellojkldef
$ echo hello > one > two
$ yes | head -n 3 > >(wc -l >two) > >(wc -l >one)
landley at driftwood:~$ cat one
3
landley at driftwood:~$ cat two
0
$ </dev/null echo hello
hello
$ echo {+}</dev/null
{+}
$ echo {}</dev/null
{}
$ echo <
bash: syntax error near unexpected token `newline'
$ murgle() { echo > "$@" ;}
$ murgle one two three
bash: "$@": ambiguous redirect
$ murgle one
$ cat << " "
> abc
>
abc
$ cat << \"
> boing
> "
boing
$ echo hello &>> blah.txt
$ potato() { echo "$@"; }; IFS=1234 potato one two three
one two three
$ cat <<< {one,two,three}
{one,two,three}
$ cat <<< one two three
$ echo hello >&2-x && ls 2-x
2-x
$ while true; do sleep 1; read a; echo $a; [ -z "$a" ] && break; done << EOF &
> one
> two
> three
> EOF
[1] 30590
$ (set -o noclobber; echo > /dev/null)
$ (ln -s /dev/null blah; set -o noclobber; echo > blah)
$ (touch walrus; ln -s walrus penguin; set -o noclobber; echo > penguin)
$ funky() { A="$@" ;}; funky one two three; echo $A
one two three
$ export "()=42"
bash: export: `()=42': not a valid identifier
$ set hello="$@" > walrus
$ X=$(false) || echo true
true
$ for((i=1;i<5;i++));do echo $i;done
1
2
3
4
# when is (( a token vs ( (?
$ ((echo a) | sed s/a/b/)
b
$ ((1
> +2<1)); echo $?
1
$ X=ii;for $X in a b c; do echo $ii; done
bash: `$X': not a valid identifier
$ for i in 1 2 3 & do echo hello; done
bash: syntax error near unexpected token `&'
$ if true; then false; else true; elif true; then false; fi
bash: syntax error near unexpected token `elif'
$ while false; true; do echo hello; done | head -n 3
hello
hello
hello
$ せ=42
bash: せ=42: command not found
$ function abc=def () {echo hello;} ; "abc=def"
hello
$ ((echo hello) )
hello
$ abc() { for name; do echo $name; done;}; abc X Y Z
X
Y
Z
# In a for loop ;; breaks down to ; ;
$ for((i=0;;i++)); do echo $i; done | head -n 30
1
2
$ (()); echo $?
1
$ ((x=12)); echo $x
12
$ (echo 123) # redir prefix not used for non-redir end tokens
123
$ x=42; ! ((x<3)) && ((x<43)) && echo yes
yes
# redirects aren't in math
$ echo $((2<3))
1
$ echo $((2&3))
2
$ ((3<2)); echo $?
1
$ X=X; echo $((X+2))
bash: X: expression recursion level exceeded (error token is "X")
$ X=3 ((3<2)) || echo hello
bash: syntax error near unexpected token `('
$ echo $((`echo 1`))
1
$ X=3; (($X<3)) || echo no; (($X<4)) && echo yes
no
yes
$ cat <(echo "$(((1<2)) && echo "hello")")
hello
$ THINGY=whoami; echo "$("${THINGY}")"
landley
$ burble() #comment
> { echo hello;};burble
hello
$ for i=3
> do echo hello; done
bash: `i=3': not a valid identifier
$ ((1<2)) > blat
$ boing () { for name do echo $name; done }; boing one two three
one
two
three
$ cat <( "(" )
bash: (: command not found
$ break > potato 2>/dev/null || ls potato
potato
# too many error messages
$ break 1 37
bash: break: only meaningful in a `for', `while', or `until' loop
$ for i in 1; do break walrus; done
bash: break: walrus: numeric argument required
$ for i in 1; do break 37 walrus; done
bash: break: too many arguments
$ for i in 1; do break 0; done
bash: break: 0: loop count out of range
$ if true; then if false; then echo one; elif ! echo two; then echo three; else echo four; fi; fi
two
four
$ cat boing |& sed s/boing/walrus/
cat: walrus: No such file or directory
# runs printing "hi" until segfault
bash -c "abc() { echo hi; def() { abc;echo lo;};def;}; abc"
# word splitting test
$ X="one two"; printf '%s\n' $X 2 3 | while read a; do echo $a; done
one
two
2
3
$ fff() { echo abc$*def;}; fff one two three | xargs -n 1 echo
abcone
two
threedef
$ ABC=abcdefg
$ echo ${ABC:+"}"}
}
$ for i in a b c d e; do break & done
[1] 25533
[2] 25534
[3] 25535
[4] 25536
[5] 25537
$ walrus=1; echo ${walrus+blah}
blah
$ unset walrus; echo ${walrus+blah}
$
$ echo echo hello | sh
hello
$ { { { echo hello;} ;} >blat ;} ; echo one; cat blat
one
hello
$ ABC=1
$ echo ${"AB"C}
bash: ${"AB"C}: bad substitution
$ echo ${"ABC"}
bash: ${"ABC"}: bad substitution
$ X='*'; echo $X
$ <&2-; ls boing # redirect undone at end of command
ls: cannot access 'boing': No such file or directory
$ ls boing <&2- # redirect applies to command
$ X=42 </dev/null ; echo $X # local assignment persists even with redirect
42
$ bash -c 'read i >/dev/null <<< boing; echo $i'
boing
# The expansions unquoted << _doesn't_ do.
$ cat << TEST
> ~landley
> *
> potato/{one,two,three}
> "hello"
> TEST
~landley
*
potato/{one,two,three}
"hello"
$ X="|"; echo hello $X and
hello | and
$ for i in a b c; do for j in d e f; do for k in g h i; do echo $i $j $k; continue 3; done done done
a d g
b d g
c d g
$ sh -c '<&3-'
sh: 1: Syntax error: Bad fd number
$ {abc}<&2- # leaves $abc open even when 2- unwound
$ {def}<&2 # doesn't close 2
$ {bcd}<walrus
$ {cde}<<EOF
hello
EOF
$ echo {fgh}<<<potato another; echo hi; cat <&$fgh
another
hi
potato
$ <&2- <&37 ; echo $?
1
$ echo hello {var}<nonexistent # doesn't set var on error
$ var=wurble; echo hello {var}<nope # doesn't blank either
$ X="echo hello"; $X
hello
$ > /does/not/exist && echo $?
1
$ { echo -e "one\ntwo" ;} | { if true; then read i; echo a=$i;fi;if true; then read i; echo b=$i;fi;}
a=one
b=two
$ if false; then if false; then echo one; else echo two; else echo three; fi
# disabled on block entry disables entire block: else doesn't revive
$ if false; then if false; then echo one; else echo two; fi; else echo three; fi
three
$ { { echo hello ;} | }
bash: syntax error near unexpected token `}'
$ {|{ echo hello;};}
$ echo hello | if false; then cat; fi | cat
$ blah() { echo hello; } | echo ha; blah
ha
bash: blah: command not found
$ X=42 | true; echo $X
$ if echo hello; then true; fi | tr l x
hexxo
# assignment persists, redirect does not
$ X=1 >woot; echo did not persist; echo $X
did not persist
1
$ echo <(if)
bash: command substitution: line 22: syntax error near unexpected token `)'
bash: command substitution: line 22: `if)'
# this shows error immediately but doesn't exit for 10 seconds...
$ sleep 10 | cat < notit
bash: notit: No such file or directory
$ echo {"on,t"w}e
{on,tw}e
$ func() { echo $0,$1; shift; echo $0,$1;}; func one two three
bash,one
bash,two
$ X=123 cat <(echo $X)
$ env -i bash --norc --noprofile -c env | sort
PWD=$(pwd -P)
SHLVL=1
_=/usr/bin/env
$ echo {~,~root}/pwd
/home/landley/pwd /root/pwd
$ echo \{~,~root}/pwd
{~,~root}/pwd
$ echo ""{~,~root}/pwd
~/pwd ~root/pwd
$ A= ABC=123; echo $A{BC,""}
123
$ A=; echo $A''C
C
$ echo $((x=3)); echo $x
3
3
$ y=-4; echo $((x=y)); echo $y
-4
-4
$ x=x; echo $((x+1))
bash: x: expression recursion level exceeded (error token is "x")
$ expr 1 + 2x
expr: non-integer argument
$ echo $((1+_)) # because $_
bash: 2x: value too great for base (error token is "2x")
$ echo hello | cat <(read i; echo $i)
hello
# errors propagate up but return code doesn't
$ echo -n thingy $(if true)
bash: command substitution: line 2: syntax error near unexpected token `)'
bash: command substitution: line 2: `if true)'
$ echo $?
1
$ echo -n $(false)
$ echo $?
0
$ echo -n $(true < walroid)
bash: walroid: No such file or directory
landley at driftwood:~/toybox/toybox$ echo $?
0
$ walrus=42; readonly walrus; walrus=7; echo $?
bash: walrus: readonly variable
1
$ walrus=42; readonly walrus; walrus=7 echo hello; echo $?
bash: walrus: readonly variable
hello
0
$ ./bash -c 'echo $_'
./bash
$ PATH=$PWD:$PATH bash -c 'echo $_'
$PWD/bash
$ PATH=$PWD:$PATH PWD=/bin bash -c 'echo $_'
$OLDPWD/bash
$./sh -c 'cat <(echo hello 2>&1)'
hello
$
$ chicken() { for i in a"$@"b;do echo =$i=;done;}; chicken 123 456 789
=a123=
=456=
=789b=
# I hate IFS
$ THINGY=$'potatoxsalad\tand this\n\n\n' IFS='x '; for i in $THINGY; do echo =$i=; done
=potato=
=salad and=
=this
=
More information about the Toybox
mailing list