[Toybox] Would someone please explain what bash is doing here?

[Rob Landley]

On 5/27/20 2:31 PM, Chet Ramey wrote:
>> Right. So it CAN include more stuff, it just doesn't. And instead the stuff
>> RESUMES. (Is that posix what posix says to do? Does a shell script differ from
>> interactive behavior here?)
> 
> Yes, that's how you have to handle it. And it doesn't depend on whether or
> not the shell is interactive, in the sense that you can do some job control
> stuff when the `-m' option is set.

I'm mostly distinguishing between "I can haz command prompt" and "running a
shell script" since those are the two common cases.

> But what are you going to do with a shell script when you suspend a
> pipeline in it? Keep on going with reading commands in the script? That's
> rarely going to be what the user wants. Most people expect the shell
> running the script to get suspended when you hit it with SIGTSTP.

To be honest I'm leaning towards doing that and seeing who complains.

>> Right. Am I gonna have to go read posix again? I dowanna read posix again.
> 
> POSIX doesn't say anything about the former (unless you want to go read the
> discussion around bug 1254). The latter is ambiguous; the current bash
> behavior is what users want and expect.

Technically my thorough reading of posix, circa 2007, was susv3. So when I hit
this sort of thing I have no idea if A) I forgot in the decade since, B) it
changed, C) the standard didn't mention it.

>> (And THIS is why I've held off on implementing job control because I have no
>> current idea how to REGRESSION TEST this mess. 
> 
> It's hard. You can test the separate process group aspects of job control,
> and you can hit things with SIGTSTP and SIGSTOP from other processes and
> test fg/bg/jobs, but it's hard to simulate keyboard signals.

I need to write a scriptable pty master. I need that to test "top" and "vi" and
so on too. It's on the todo list.

Hey, I finally wrote my half-assed expect. I need to add regex support for it so
the "read and discard line" stuff can instead be "read line, succeed if it
contains this string". It's on the todo list.

"It's on the todo list" is distantly etymologically related to "nuts to your
white mice", only with the addition of sad resignation that someday I'm going to
have to care. But today is not that day.

>> The man page says "delayed suspend character" (CTRL-Y) is a thing, but as far as
>> I can tell it isn't. This was some GNU Hurd feature maybe?
> 
> No, it's really part of the tty driver going all the way back to 4.1 BSD.
> Readline turns it off because it uses ^Y for other things. Plus it's
> rarely useful.

I think I'm pretty ok with ignoring it.

>> The man page has more "written by somebody who already knows it" stuff:
>>
>>        referred to as %n.  A job may also be referred to using a prefix of the
>>        name used to start it, or using a substring that appears in its command
>>        line.  For example, %ce refers to  a  stopped  ce  job.   If  a  prefix
>>        matches  more  than one job, bash reports an error.  Using %?ce, on the
>>        other hand, refers to any job containing the string ce in  its  command
>>        line.   If  the  substring  matches  more than one job, bash reports an
>>
>> That "%?ce" is a specific example, not an explanation of what's happening here.
> 
> It's a specific example of a program named `ce' being suspended. It could
> be any program. That's why the man page has `ce' in bold.

It's a program name _starting_ with ce. I got that part. (Doesn't seem to be a
way to do an exact match, I'd guess curly braces but probably not?)

But no, you don't need the ? for it to be a prefix:

  $ sleep 999&
  [1] 12293
  $ kill %sl
  $
  [1]+  Terminated              sleep 999

You need the ? to match an argument instead of a command name. And the FUN part is:

  $ ABC=def sleep 999&
  [1] 12368
  $ kill %sl
  bash: kill: %sl: no such job
  $ kill %A
  $
  [1]+  Terminated              ABC=def sleep 999

Which combined with the "ctrl-Z continues past ;" thing makes me just REALLY TIRED.

> Since the first sentence in that section says that `%' starts a job spec,
> it's difficult to see how `%ce' doesn't refer to `the name used to start
> it'.
> 
> 
>> Is it because "?" is a wildcard, or because %? is a special prefix? 
> 
> It's because `%' introduces the job spec, and `?' is the notation to refer
> to the substring of the job name, so the latter.

Special prefix, yay.

>> That
>> question mark isn't going to act as a wildcard and match a filesystem character
>> during normal wildcard processing, right?
> 
> It could. These are invariably used as arguments to builtins, and undergo
> all the usual word expansions.

So it's %\?ce

>> If there's a %xce in the current
>> directory, it won't resolve to that? 
> 
> It could. Such filenames are extremely rare. I've never actually had a bug
> report complaining about this.

I BREAK EVERYTHING. Trust me, if I started using this feature, I would hit this.

>> The % is parsed _before_ wildcards and
>> suppresses normal wildcard expansion? 
> 
> No. fg/bg/jobs/kill/wait/disown are not special in that regard.

Despite being shell builtins. Ok.

>> If a $variable expands to %?ce does it
>> still count? 
> 
> Yes, when it's used as an argument to a shell builtin.

Alright, it's not parsed by the shell it's just consumed by commands so I should
document %\? as the prefix.

>> How does this interact with quoting? Is kill "%?ce" different from
>> kill %?ce or are they the same?
> 
> As long as `kill' sees the job spec, it treats it the same.

Got it.

>> Ordinarily I would start assembling a list of regression tests to add to sh.test
>> to determine/demonstrate the behavior I need to implement, but... backgrounding!
> 
> It's not easy. `bg' is not that hard once you have a job id. But getting
> a job to stop by hitting it with a signal from a separate process is the
> hard part.

I used to think I need containers so background processes spawned by tests and
_not_ killed because a test failed could be cleaned up, but then I did the
mkroot stuff to run under qemu and a vm can be my container here. :P

I have SOOOOOOOO much test suite stuff to do at some point...

>> Asynchronous! Not really designed to be scripted!
> 
> Asynchronous jobs are easy. Once you have something in the background, and
> you know its job number or how to refer to it using %name, you can run bg
> and fg to your heart's content, use `kill' to kill it, and use `wait' to
> clean up.

A test suite that assumes all the tests _succeed_ is easy to write.

Did I mention I fixed (the first of):

  # Race condition (in bash, but not in toysh) can say 43.
  testing 'SECONDS' 'SECONDS=41; sleep 1; echo $SECONDS' '42\n' '' ''
  # dash has an error message, bash silently fails
  testing 'SECONDS2' 'readonly SECONDS; SECONDS=0; echo $SECONDS' '' '' ''
  # declare says -i, but it doesn't take integer assignment.
  testing 'SECONDS3' 'SECONDS=123+456; echo $SECONDS' '0\n' '' '' #bash!!

By having my TT.SECONDS offset be a 64 bit millisecond count so when you assign
a time into SECONDS setvar() does:

  } else if (flags&VAR_MAGIC) {
    if (*s == 'S') TT.SECONDS = millitime() - 1000*do_math(s+len-1);

(My "it's millitime()" function returns current unix time in 64 bit milliseconds.)

I need to add a proper test for it reliablish showing the failure:

$ SECONDS=42; sleep .25; echo $SECONDS; sleep .25; echo $SECONDS; sleep .25;
echo $SECONDS
42
43
43
$ SECONDS=42; sleep .25; echo $SECONDS; sleep .25; echo $SECONDS; sleep .25;
echo $SECONDS
42
42
43
$ SECONDS=42; sleep .25; echo $SECONDS; sleep .25; echo $SECONDS; sleep .25;
echo $SECONDS
43
43
43

But although mine will say 42 for all 3 each time, consistently, that test
consistently fails on TEST_HOST rather than "just occasionally enough to cause
spurious unreproducable test suite failures of the kind that makes Elliott's
guys deeply unhappy". So I haven't added it yet because I'm still MOSTLY testing
TEST_HOST as I add stuff I mean to implement...

> Job control, at least the terminal-based aspects of it, is not really
> designed for use by non-interactive shells.  That's just the reality of it.

https://www.youtube.com/watch?v=yBiZgE9-FYc

I have a black belt in disgusting solutions to impossible problems, and then a
LOT of cleanup experience to mitigate the worst to the disgusting. The three
stages are:

  1) It can't be done.
  2) It can't practically be done.
  3) Everyone's always done it.

In this case, the problem is to do a pty master I have to knock something
together in C, and it's awkward to figure out where to PUT it. The test suite
doesn't require a toolchain to be there (TEST_HOST can run on a toybox system,
that's part of the reason for mkroot), I can add a binary to toys/examples but
then it would require that to be enabled in the config to run the test, and I
don't want a test program to be part of normal defconfig...

In theory implementing my own version of:
https://android.googlesource.com/platform/system/core/+/android-4.4_r1/toolbox/ioctl.c

would let me do openpty() in shell (since tcsetattr is just ioctl with a struct
and I can do up to sizeof(toybuf)=4k of hex coding if I want to), but A) ew, B)
TEST_HOST still can't expect an ioctl command to be there...

(Yes I've heard of socat, no it doesn't help here.)

> Chet

Rob