[Toybox] [Nommu] Week ending June 27ish.

[Rob Landley]

On 06/28/2015 09:57 PM, Rich Felker wrote:
> On Sun, Jun 28, 2015 at 06:15:26PM -0500, Rob Landley wrote:
>>> Are there perhaps any other shells that work on nommu? It would be
>>> nice to be able to put off toysh until you have time to do a really
>>> good job on it rather than rushing it because there's nothing else you
>>> can use...
>>
>> Eh, I can do it in stages. And setting up another shell is effort (and
>> natural test/use cases) that's _not_ going into toysh development, so
>> I'd rather do it right than do other things that get thrown away.
>>
>> That said, sure, http://www.cod5.org/archive/ links to es and pdksh for
>> example, and uclibc had like 5 of them (all kinda crappy if I recall).
>> None really an improvement on hush in terms of actually building stuff.

I should have clarified these were other shells to look at with
licensing I didn't mind distributing, not that I'd confirmed they work
with nommu. (Except the uclinux shells, which are _crap_.)

Those are other things I could go look at instead of writing something I
mean to write anyway, spending the effort advancing a project I _know_
would do what I need it to do instead of debugging another dead end.

> I'm pretty sure pdksh conforms to POSIX (at least the 1992 version) so

Uh-huh. 1992 was 23 years ago. That would mean the shell's active
development predates Linux. That standard is a decade older than the
stale version of bash I'm trying to move off of because stuff keeps
breaking when I use it.

And on those grounds you expect it to be better than hush?

> it's probably sufficient for running portable scripts including all
> configure scripts.

You think all configure scripts are portable?

Really?

> I know your goal is much higher (bash compatibility)

Well, some bash extensions. Full bug for bug bash compatability is a
moving target. I'm basically thinking bash 2.x plus stuff people
actually use as they complain about it. (-o pipefail for one thing, and
the ~= comparator for another; portage used that).

> but it sounds like pdksh could get you a working
> environment until you finish toysh, with little or no effort spent on
> the temporary solution.

Having never put together your own distro build system, you believe
there's no effort involved. Having done this sort of thing repeatedly
for about 15 years, I don't even want to go there.

But sure, let's take a look at the actual pdksh code, in jobs.c it says:

 * The interface to the rest of the shell should probably be changed
 * to allow use of vfork() when available but that would be way too much
 * work :)

...

        /* create child process */
        forksleep = 1;
        while ((i = fork()) < 0 && errno == EAGAIN && forksleep < 32) {
                if (intrsig)     /* allow user to ^C out... */
                        break;
                sleep(forksleep);
                forksleep <<= 1;
        }

So no, that doesn't work with nommu either. (And I have no idea what
weird historical bug that was trying to work around back in 1992.)

>> Yes. If you call toybox as a command name, it gets the normal command
>> setup done for it. I'm not optimizing for that case. Pilot error.
> 
> OK. This was the source of the duplicates. I would still like to see
> no extra syscalls at all (much like my feeling about glibc startup),
> but the situation isn't nearly as bad as I thought.
> 
> BTW what happens with the suid check/drop being done twice? Do
> commands that need suid work when you call them as "toybox cmdname"?
> Are they supposed to?

in scripts/make.sh:

echo "USE_TOYBOX(NEWTOY(toybox, NULL, TOYFLAG_STAYROOT))" > \
  generated/newtoys.h

The stayroot tells the multiplexer not to drop privs. (Commit 15a8d71674b4.)

>>> It's not
>>> as bad without the duplication, but that's still 4 unnecessary
>>> round-trips to kernelspace for a lot of commands. Maybe the get[e]uid
>>> stuff is hard to remove when the suid support is compiled-in, but
>>> getauxval() could be used instead on systems that have it.
>>
>> There was talk a few years back of putting getuid and friends in vdso
>> and the result was nobody bothered because the overhead wasn't high
>> enough for anybody to care. (People tend to call gettimeofday() in tight
>> loops because it constantly changes.)
> 
> I could cache it in libc too, but I feel the risk of returning a stale
> value outweighs the performance benefits, so I didn't do it. But if
> you're checking it at startup (without having done uid changes),
> getauxval is reliable.

$ man getauxval | grep CONFORMING -A 1
CONFORMING TO
       This function is a nonstandard glibc extension.

So you just suggested I replace a posix function with a nonstandard
glibc extension in the name of saving a system call that does an
unlocked fetch on a single integer.

Well that's certainly a point of view.

>>> The difference with using clone() directly is that __thread variables
>>> are not going to work and it might not even be safe to call any libc
>>> functions. glibc doesn't document what is or isn't safe; I could go
>>> into detail on the topic with musl if anyone cares to hear.
>>
>> Feel free, but if threads need to do something non-obvious that clone()
>> doesn't, I don't want to get threading on me.
> 
> What they do is specifically setting up TLS, both visible TLS that
> belongs to the application/libraries and libc-internal stuff.

And activate cancellation point stuff and locking in libc that was a
_fun_ source of subtle bugs in uClibc, and then make errno stop being a
global variable, and they open a can of worms of asynchronous race
conditions for debugging...

> In principle the former could be skipped if you know your code is not
> using TLS,

Which would mean not using errno?

Me, I just declared a structure in main() so it was at the base of the
stack and passed a darn "that" pointer around to it (or bouncing off
linked list of pid/pointer pairs to look it up if I hadn't wanted to
pass the pointer around, which could be a hash if there were enough
threads but there never were in anything I played with back when I
cared) so get_that() was a lightweight enough function not to hugely care...

I have built my own infrastructure for this sort of thing from first
principles more than once over the years, but my first question these
days is "do we really need to go there".

> but the latter could be dangerous not to have setup right.
> I *think* (this needs checking) the new thread created manually by
> clone will use the same TLS pointer as the thread that called clone.
> In this case, libc internals will potentially be reading and writing
> the same data, without any synchronization.

See "decided to just use fork(), which means re-exec /proc/self/exe as
necessary because exec(NULL) doesn't re-exec yourself despite multiple
proposals over the year that the kernel just DO that since the kernel
knows the right inode even if proc isn't mounted"...

Ahem.

> The most obvious case that
> _necessarily_ breaks is that they're both using the same errno, so you
> can never reliably check errno,

Actually, there are horrible, horrible, horrible ways to make it work.
(Did you now you can implement your own page fault handler in userspace?
Or you can suspend all the other clones with SIGSTOP, do your thing and
check errno, and resume all the other clones. There are a _bunch_ of bad
ways to do this.) But I'm already deep into "not going there"...

> and anything libc-internal that checks
> errno as part of its operation could get the wrong value and badly
> break.
> 
> I know glibc's pthread library is big and heavy, but in musl there's
> really nothing you "don't want to get on you".

Did you miss the part where I DID THREADED PROGRAMMING FOR YEARS IN
COLLEGE AND LATER FOR MY DAY JOB? (And then did Java as my main
programming language for about 3 more years, which was all threads
because they didn't wrap poll() or select(), but that's another story.)

Ok, I'm throwing this topic in the "religion" bucket and doing the
hiding behind the sofa pretending to not be home when you ring the
doorbell thing on the topic from now on.

>>>> Commands that _can't_ run without fork() generally have a reason, such
>>>> as they don't free all their memory on error paths, they've changed
>>>> signal handlers, they may leave memory mappings, there may be opendir()
>>>> state, there could be exit handlers... And no, beefing up the error and
>>>> exit handling to make TOYBOX_CLEANUP_TO_HUMOR_VALGRIND be mandatory
>>>> doesn't fix it because ctrl-c can come in anywhere.
>>>
>>> The obvious case where you need fork not to work around implementation
>>> limitations like the above, but for an actual semantic reason, is for
>>> pipelines and () subshells.
>>
>> You can't intercept sigstop so you'd have to write some sort of filter
>> intercepting I/O and faking your own PTY to implement ctrl-z and I am
>> so, so, so not going there. Plus fg, bg, &.
> 
> ^Z sends SIGTSTP, not SIGSTOP. I agree it would be some work to make
> fake job control where no processes exist, but I would just use real
> processes for interactive shells with job control.

As would I. Not threads.

> Performance difference is not noticable there anyway. The case where
> you get a measurable benefit from avoiding forking and execing is in
> scripts, where you don't have job control to worry about.

You're once again recommending I have two codepaths implementing the
same functionality. (And possibly basing it on the assumption that shell
scripts aren't going to do crazy asynchronous stuff like scripts/make.sh
does, let alone try to implement "disown"...)

> Rich

Rob

 1435609573.0