[Toybox] [Nommu] Week ending June 27ish.

[Rich Felker]

On Sun, Jun 28, 2015 at 06:15:26PM -0500, Rob Landley wrote:
> > Are there perhaps any other shells that work on nommu? It would be
> > nice to be able to put off toysh until you have time to do a really
> > good job on it rather than rushing it because there's nothing else you
> > can use...
> 
> Eh, I can do it in stages. And setting up another shell is effort (and
> natural test/use cases) that's _not_ going into toysh development, so
> I'd rather do it right than do other things that get thrown away.
> 
> That said, sure, http://www.cod5.org/archive/ links to es and pdksh for
> example, and uclibc had like 5 of them (all kinda crappy if I recall).
> None really an improvement on hush in terms of actually building stuff.

I'm pretty sure pdksh conforms to POSIX (at least the 1992 version) so
it's probably sufficient for running portable scripts including all
configure scripts. I know your goal is much higher (bash
compatibility) but it sounds like pdksh could get you a working
environment until you finish toysh, with little or no effort spent on
the temporary solution.

> >>>> 2) I'm teaching toybox to probe for the existence of fork(), set a nommu
> >>>> symbol if it's not there, and use sort of a fakeroot version of fork
> >>>> (re-exec yourself with some way of transferring state to the child,
> >>>> maybe an environment variable, maybe a pipe...).
> >>>
> >>> BTW toybox does a lot of expensive probing at startup already (before
> >>> even entering the applet's main function) that makes simple commands
> >>> take about twice the time of their busybox versions.
> >>
> >> Could you please be a little more vague?
> >>
> >> I'd prefer to respond to this on the toybox list, but since you felt it
> >> should be asked here instead...
> > 
> > Sorry, it just came up as part of the overhead for self-exec topic. I
> > agree the toybox list would have been a better place.
> > 
> >> Or do you mean something other by "probing" than "system calls"? (Memory
> >> usage? CPU usage? Could I have some sort of metric here, or at least the
> >> axis of interest?)
> > 
> > I meant system calls. I have:
> > 
> > # strace toybox true
> > set_tid_address(0x15c9eeb4)             = 161
> > rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER|SA_RESTART, 0x15c4cb48}, {SIG_DFL, [], 0}, 8) = 0
> > getuid32()                              = 0
> > geteuid32()                             = 0
> > umask(0)                                = 0755
> > umask(0755)                             = 0
> > getuid32()                              = 0
> > geteuid32()                             = 0
> > umask(0)                                = 0755
> > umask(0755)                             = 0
> 
> Ah, that's two instances of setup. The setup for toybox_main() and the
> setup for true.
> 
> It can normally skip one of those, but you're explicitly calling
> "toybox" so it treats it as a command and does setup for it.
> 
> > exit_group(0)                           = ?
> > +++ exited with 0 +++
> > 
> > It looks like they're all repeated twice when the toybox command is
> > used to invoke a command by name (rather than with symlinks).
> 
> Yes. If you call toybox as a command name, it gets the normal command
> setup done for it. I'm not optimizing for that case. Pilot error.

OK. This was the source of the duplicates. I would still like to see
no extra syscalls at all (much like my feeling about glibc startup),
but the situation isn't nearly as bad as I thought.

BTW what happens with the suid check/drop being done twice? Do
commands that need suid work when you call them as "toybox cmdname"?
Are they supposed to?

> > It's not
> > as bad without the duplication, but that's still 4 unnecessary
> > round-trips to kernelspace for a lot of commands. Maybe the get[e]uid
> > stuff is hard to remove when the suid support is compiled-in, but
> > getauxval() could be used instead on systems that have it.
> 
> There was talk a few years back of putting getuid and friends in vdso
> and the result was nobody bothered because the overhead wasn't high
> enough for anybody to care. (People tend to call gettimeofday() in tight
> loops because it constantly changes.)

I could cache it in libc too, but I feel the risk of returning a stale
value outweighs the performance benefits, so I didn't do it. But if
you're checking it at startup (without having done uid changes),
getauxval is reliable.

> >>>> I don't want to replace fork _entirely_ because fork takes about 5% as
> >>>> long as exec on systems that _do_ support it, so the toybox shell being
> >>>> able to fork and run commands internally is a big potential speedup for
> >>>> shell scripts. But codepaths that are _not_ performance critical should
> >>>> use the long way round so it gets testing.
> >>>
> >>> This seems like a good approach, but I have a suggestion that will
> >>> perform even better: use pthread_create instead of fork for
> >>> implementing shell builtins
> >>
> >> Suppressing the gag refex for involving pthreads in something for no
> >> reason, I actually looked at that a long time ago (using clone()
> >> directly to create shared process contexts), and didn't like it. Random
> >> blog post (one of several over a multi-month period):
> >>
> >> http://landley.net/notes-2007.html#20-01-2007
> > 
> > The difference with using clone() directly is that __thread variables
> > are not going to work and it might not even be safe to call any libc
> > functions. glibc doesn't document what is or isn't safe; I could go
> > into detail on the topic with musl if anyone cares to hear.
> 
> Feel free, but if threads need to do something non-obvious that clone()
> doesn't, I don't want to get threading on me.

What they do is specifically setting up TLS, both visible TLS that
belongs to the application/libraries and libc-internal stuff. In
principle the former could be skipped if you know your code is not
using TLS, but the latter could be dangerous not to have setup right.
I *think* (this needs checking) the new thread created manually by
clone will use the same TLS pointer as the thread that called clone.
In this case, libc internals will potentially be reading and writing
the same data, without any synchronization. The most obvious case that
_necessarily_ breaks is that they're both using the same errno, so you
can never reliably check errno, and anything libc-internal that checks
errno as part of its operation could get the wrong value and badly
break.

I know glibc's pthread library is big and heavy, but in musl there's
really nothing you "don't want to get on you". Even if you use all the
threads functions, it's under half the size of stdio, and typical
usage (like what I suggested toybox could so) will just be a few kb.
The real cost, as you've noted, would be making the toys so that they
clean up sufficiently, and sufficiently avoid breaking global state
like fd 0/1/2, to be usable as threads, and I agree that's probably
prohibitive without a significantly different design.

> >> Commands that _can't_ run without fork() generally have a reason, such
> >> as they don't free all their memory on error paths, they've changed
> >> signal handlers, they may leave memory mappings, there may be opendir()
> >> state, there could be exit handlers... And no, beefing up the error and
> >> exit handling to make TOYBOX_CLEANUP_TO_HUMOR_VALGRIND be mandatory
> >> doesn't fix it because ctrl-c can come in anywhere.
> > 
> > The obvious case where you need fork not to work around implementation
> > limitations like the above, but for an actual semantic reason, is for
> > pipelines and () subshells.
> 
> You can't intercept sigstop so you'd have to write some sort of filter
> intercepting I/O and faking your own PTY to implement ctrl-z and I am
> so, so, so not going there. Plus fg, bg, &.

^Z sends SIGTSTP, not SIGSTOP. I agree it would be some work to make
fake job control where no processes exist, but I would just use real
processes for interactive shells with job control. Performance
difference is not noticable there anyway. The case where you get a
measurable benefit from avoiding forking and execing is in scripts,
where you don't have job control to worry about.

Rich

 1435546620.0