[Toybox] [Nommu] Week ending June 27ish.

[Rich Felker]

On Mon, Jun 29, 2015 at 03:26:13PM -0500, Rob Landley wrote:
> On 06/28/2015 09:57 PM, Rich Felker wrote:
> > On Sun, Jun 28, 2015 at 06:15:26PM -0500, Rob Landley wrote:
> >>> Are there perhaps any other shells that work on nommu? It would be
> >>> nice to be able to put off toysh until you have time to do a really
> >>> good job on it rather than rushing it because there's nothing else you
> >>> can use...
> >>
> >> Eh, I can do it in stages. And setting up another shell is effort (and
> >> natural test/use cases) that's _not_ going into toysh development, so
> >> I'd rather do it right than do other things that get thrown away.
> >>
> >> That said, sure, http://www.cod5.org/archive/ links to es and pdksh for
> >> example, and uclibc had like 5 of them (all kinda crappy if I recall).
> >> None really an improvement on hush in terms of actually building stuff.
> 
> I should have clarified these were other shells to look at with
> licensing I didn't mind distributing, not that I'd confirmed they work
> with nommu. (Except the uclinux shells, which are _crap_.)

Yes, I noticed. Whatever that shell from uclinux I was using at first
was, it was utterly awful.

> > I'm pretty sure pdksh conforms to POSIX (at least the 1992 version) so
> 
> Uh-huh. 1992 was 23 years ago. That would mean the shell's active
> development predates Linux. That standard is a decade older than the
> stale version of bash I'm trying to move off of because stuff keeps
> breaking when I use it.

I'm not aware of important changes in the shell spec since then, but I
may be mistaken.

> And on those grounds you expect it to be better than hush?

I recall reading that hush is known to be missing some important
functionality, though I don't remember what. It worked fine for my
interactive use and simple scripts though.

> > it's probably sufficient for running portable scripts including all
> > configure scripts.
> 
> You think all configure scripts are portable?
> 
> Really?

Anything produced by autotools that's not using custom shell script
level code (just the standard m4 macros, etc.) is theoretically
completely portable, and in practice very much so. I've run these
scripts on all kinds of crazy proprietary unices and when I've
encountered trouble, it hasn't been at the shell interpreter level.

> But sure, let's take a look at the actual pdksh code, in jobs.c it says:
> 
>  * The interface to the rest of the shell should probably be changed
>  * to allow use of vfork() when available but that would be way too much
>  * work :)
> 
> ...
> 
>         /* create child process */
>         forksleep = 1;
>         while ((i = fork()) < 0 && errno == EAGAIN && forksleep < 32) {
>                 if (intrsig)     /* allow user to ^C out... */
>                         break;
>                 sleep(forksleep);
>                 forksleep <<= 1;
>         }
> 
> So no, that doesn't work with nommu either. (And I have no idea what
> weird historical bug that was trying to work around back in 1992.)

OK, so that idea is out. Guess you need to write toysh. :-)

> >> Yes. If you call toybox as a command name, it gets the normal command
> >> setup done for it. I'm not optimizing for that case. Pilot error.
> > 
> > OK. This was the source of the duplicates. I would still like to see
> > no extra syscalls at all (much like my feeling about glibc startup),
> > but the situation isn't nearly as bad as I thought.
> > 
> > BTW what happens with the suid check/drop being done twice? Do
> > commands that need suid work when you call them as "toybox cmdname"?
> > Are they supposed to?
> 
> in scripts/make.sh:
> 
> echo "USE_TOYBOX(NEWTOY(toybox, NULL, TOYFLAG_STAYROOT))" > \
>   generated/newtoys.h
> 
> The stayroot tells the multiplexer not to drop privs. (Commit 15a8d71674b4.)

Looks good.

> >>> It's not
> >>> as bad without the duplication, but that's still 4 unnecessary
> >>> round-trips to kernelspace for a lot of commands. Maybe the get[e]uid
> >>> stuff is hard to remove when the suid support is compiled-in, but
> >>> getauxval() could be used instead on systems that have it.
> >>
> >> There was talk a few years back of putting getuid and friends in vdso
> >> and the result was nobody bothered because the overhead wasn't high
> >> enough for anybody to care. (People tend to call gettimeofday() in tight
> >> loops because it constantly changes.)
> > 
> > I could cache it in libc too, but I feel the risk of returning a stale
> > value outweighs the performance benefits, so I didn't do it. But if
> > you're checking it at startup (without having done uid changes),
> > getauxval is reliable.
> 
> $ man getauxval | grep CONFORMING -A 1
> CONFORMING TO
>        This function is a nonstandard glibc extension.
> 
> So you just suggested I replace a posix function with a nonstandard
> glibc extension in the name of saving a system call that does an
> unlocked fetch on a single integer.
> 
> Well that's certainly a point of view.

While glibc was the first to add it, it's conceptually something nice
to have on any ELF-based platform or platform with ELF-like process
startup semantics, and I worked with the glibc team on making it suck
less (inability to distinguish no-value from a value equal to the
no-value return code, etc.) and once we agreed on behavior,
implemented the same in musl.

You can take or leave it, but if you're doing Linux-specific stuff in
toybox already, it doesn't 'feel' terribly 'dirty' to me, and it cuts
down on the syscall overhead.

> >>> The difference with using clone() directly is that __thread variables
> >>> are not going to work and it might not even be safe to call any libc
> >>> functions. glibc doesn't document what is or isn't safe; I could go
> >>> into detail on the topic with musl if anyone cares to hear.
> >>
> >> Feel free, but if threads need to do something non-obvious that clone()
> >> doesn't, I don't want to get threading on me.
> > 
> > What they do is specifically setting up TLS, both visible TLS that
> > belongs to the application/libraries and libc-internal stuff.
> 
> And activate cancellation point stuff and locking in libc that was a
> _fun_ source of subtle bugs in uClibc,

That's active even without pthread_create being called, but it won't
be acted upon unless your program calls pthread_cancel somewhere.

> and then make errno stop being a
> global variable,

errno is thread-local whether or not there's more than one thread, but
accessing it won't work right if you call clone() yourself.

> and they open a can of worms of asynchronous race
> conditions for debugging...

Only if you're accessing the same data. Most of the good uses of
threads have no shared data at all.

> > In principle the former could be skipped if you know your code is not
> > using TLS,
> 
> Which would mean not using errno?

Yes, that was in the text that immediately followed where you split
the quote. :-)

> > The most obvious case that
> > _necessarily_ breaks is that they're both using the same errno, so you
> > can never reliably check errno,
> 
> Actually, there are horrible, horrible, horrible ways to make it work.
> (Did you now you can implement your own page fault handler in userspace?
> Or you can suspend all the other clones with SIGSTOP, do your thing and
> check errno, and resume all the other clones. There are a _bunch_ of bad
> ways to do this.) But I'm already deep into "not going there"...

None of this is safe or reliable. Even if you manage to make it work
on one particular setup, there are all kinds of ways it can and will
break under changes to libc that you shouldn't be poking at..

BTW you can't suspend threads (CLONE_THREAD) separately with SIGSTOP.
You could do it with CLONE_VM without CLONE_THREAD, making processes
that share memory, but these don't work at all under qemu-user.

> > Performance difference is not noticable there anyway. The case where
> > you get a measurable benefit from avoiding forking and execing is in
> > scripts, where you don't have job control to worry about.
> 
> You're once again recommending I have two codepaths implementing the
> same functionality. (And possibly basing it on the assumption that shell
> scripts aren't going to do crazy asynchronous stuff like scripts/make.sh
> does, let alone try to implement "disown"...)

disown is for job control, and thus irrelevant to scripts.

I'm not pressing you to do this, and it's quite clear that you don't
want to, so that's fine. But I think it would be possible (and IMO
interesting and useful) to make a nommu-friendly shell that uses
threads and thereby is fast at running complex scripts that would
otherwise involve A LOT of re-exec on nommu. It's certainly an
interesting task to leave open for the future of nommu, and that's
part of why I brought it up for discussion here. But please don't
think I'm trying to pressure you to implement it -- I know you don't
want to.

Rich

 1435631100.0