[Toybox] Thoughts on seperating shell dependencies and MAYFORK commands?

[enh]

On Wed, Feb 21, 2024 at 6:08 AM Rob Landley <rob at landley.net> wrote:
>
> On 2/20/24 19:41, enh wrote:
> >> The SHELL won't do it, because the shell cares fairly deeply about what is and
> >> isn't a child process. In fact pipelines have implicit ( ) around each entry
> >> because each one is a subshell to avoid pipelines blocking on full pipe buffers.
> >> (And yes that includes the last one in the list, for consistency. And thus "echo
> >> | x=47; echo $x" isn't going to remember the 47, same as bash.)
> >>
> >> But on an mmu system, fork() is like 5% of the expense and exec() is 95%.
> >
> > (in case you ever get quoted out of context, i'll add "for a process
> > like toybox that doesn't have threads, large numbers of VMAs, or large
> > numbers of open fds" here. because those things do make fork()
> > expensive in large systems. but, no, not for toybox.)
>
> Oh sure. https://landley.net/notes-2018.html#26-06-2018 for example. :)
>
> >> There are a couple exceptions, like true/false which are SO cheap that the extra
> >> overhead from fork() is noticeable in "while true; do blah; done" loops. And
> >> "echo" is historically a bash builtin so a standalone shell on a system with no
> >> $PATH might want that. But as you've pointed out, even cat didn't get that
> >> annotation.
> >
> > (fwiw, mksh _does_ have a builtin cat. that's something i disabled on
> > Android because it was confusing to users. for "fully toybox" systems
> > though, with just one binary and loads of symlinks, you wouldn't be
> > able to tell the difference as easily. though iirc it was weird
> > signal-related behavior that was the first time someone noticed cat
> > was a builtin?)
>
> If there's an actual _need_ for cat to be a builtin it can be, but I hadn't
> noticed a demand?

me neither. (like i said --- i disabled this in Android's copy of
mksh, and it's really rare i'll keep a local patch in a third-party
open source project in AOSP --- i'm usually the guy going around
undoing such things!)

> There used to be a blog that handed out the "unnecessary use of cat" award,
> because redirects are cheaper. Once upon a time I had a todo item to try to work
> splice(2) into more stuff, which gave over to sendfile(2) instead which
> xsendfile() and friends hopefully cover today, but the fundamental problem is
> the process context can't go away even after we shorted two filehandles together
> so data goes straight from one to the other. Even when the process context no
> longer has anything to do, it can't exit. (Zombie Prime.)
>
> I asked about this on lkml long ago, noting that technically you can pass a
> filehandle across a unix domain socket and see what another process's file
> descriptors are connected to in /proc so you could KIND of implement this in
> userspace with ptrace if you really tried, so it shouldn't be THAT hard for the
> kernel to do, and the resulting screams of conceptual design-level anguish
> dissuaded me from pursuing it further...
>
> (Alas, the old "there is no way to do this" -> "here's an utterly terrible way
> to do it" -> "don't ask questions post errors" -> "better fix ASAP to prevent
> THAT" pipeline stopped being acceptable on linux-kernel roughly around the 2008
> mortgage crisis. Also, in this case, something about reference counting?)
>
> I _could_ also have the shell special case recognize "cat singlefile |" as an
> input redirect, maybe that's what mksh is doing? But I really dowanna unwind
> "cat <(potato) |" and friends. Hmmm, speaking of which...
>
> $ bash -c 'cat <<< <(echo hello)'
> /dev/fd/63
> $ toybox sh -c 'cat <<< <(echo hello)'
> <(echo hello)
> $ toybox sh -c 'echo <(echo hello)'
> /proc/self/fd/3
>
> Ok, I don't even know what "fixing" that would look like. Throw it on the todo heap.
>
> >> Seriously, benchmark it. The expense isn't clone(2), it's execve(2). (Now maybe
> >> selinux nonsense makes that go weird, couldn't say...)
> >
> > (as a libc maintainer, "the expense is all in your ELF constructors"
> > :-)
>
> As we discovered, MacOS homebrew launch time is horrific. But even there, you
> only really notice this in loops. We're talking fractions of a second here.
>
> But "sed 'pattern' long list of input files" is gonna be faster than "for i in
> long list of input files; do sed $i; done" even if the shell overhead was
> literally free, because of the repeated setup code in both OS and the command.
>
> > last i looked, most of the "wasted" time in a trivial toybox
> > invocation on Android is some front-loaded work in the networking code
> > that makes sense for the zygote but would be unfortunate for someone
> > running lots of random unix commands!)
>
> I thought the zygote ran some setup code before it spins forking itself?
> (https://developer.android.com/topic/performance/memory-overview says "common
> framework code and resources" which could mean anything).

that part's basically "the java ui stuff" --- so you don't pay [in
either time or dirty pages] to load all the same bitmaps and fonts on
every single app launch.

> I wonder why that
> network setup's in libc instead of the zygote's main()...

i didn't think anyone would be paying enough attention for it to
warrant me being more precise :-) so, no, the zygote isn't the
_reason_ we do this --- the zygote is the reason that we've
historically not really cared about front-loading work in libc. the
_reason_ why it's a good idea to get all the network stuff started
early is because you don't want to leave it all until the networking
functions are actually called because that's error-prone for folks in
jails or whatever. (and, yes, really that shouldn't be our problem,
but when you have a system built by the independent work of multiple
teams at multiple companies on multiple continents, coordination is
very hard and very expensive.) which leads in to the last reason
(addressing the "but why is some of the networking code in a separate
.so anyway?!"), which is that "our org chart is showing". this
particular instance was my fault, and i'm still not sure whether it
was a good idea or not. i think if we'd had the time and people it
would have been cleaner to have everything in bionic just talk to the
netd socket, but that wasn't what we inherited, and even if we had
someone to work on cleaning up that tech debt, i'm not sure how i'd
argue that it was worth the potential for app compat pain. (which is
the other "distributed" problem --- when you have more than 3 billion
users, there's nothing you can change that won't break _someone_...)

> Rob