[Toybox] 2019 May sitrep

[Rob Landley]

On 5/24/19 6:09 PM, enh via Toybox wrote:
> tar
> 
> good news on the tar front: other than one caller (accidentally) using
> --format=gnu, which i've fixed, i think we're good to go switching tar
> over for the AOSP build.

Yay!

> (though unfortunately i can no longer easily
> do my "build almost everything" builds because a full crosshatch
> checkbuild takes so long that the build servers time out.)

I don't have the context to evaluate this statement, I can only do builds on my
machines here.

I haven't set up the fastest one to try to do an AOSP build yet
(https://twitter.com/landley/status/996039436801593344 can only be a server but
it still has systemd on it because
https://twitter.com/landley/status/1104942661763510272 and is thus useless). I
haven't let an AOSP build run to completion on my new laptop yet because after
running overnight I kind of what to do other things with it. (I wound up using
the stock Google image my new pixel 3xl came with, just like my last 3 android
phones.)

P.S. I installed terminal on the pixel, did an "ls -l", and got "access denied"
because of the open("/") issue, and I think I need more O_PATH in there
somewhere, haven't poked at it yet. I haven't figured out yet how to package up
a binary I build to sideload onto this thing in a way it would have permissions
to do anything. There's a can of worms, there's whole youtube videos on this but
the bookmarks I have are circa Marshmallow...

> if you're
> still working on tar, though, let me know and i'll hold off switching
> the build over until the dust settles :-)

The dust has settled for the moment, have at. My next big tar can of worms is
xattr support and I haven't looked into that at all yet.

What I'm really trying to do now is get toysh to not be a useless stub anymore
before I have to vanish back into $DAYJOB land because none of the android
ecosystem is actually paying me to do this. (I took 3 months off before looking
for my next gig, and there's one week left of that third month. I did a lot more
catching up on sleep and housework and finally renewing my passport and such
than the intense toybox development I expected to do (I was pretty badly burned
out when I started), but I finally submitted some talks to a conference's CFP
for the first time in over a year and of the 4 topics I submitted to ELC's
August event in San Diego they decided they want to hear "busybox vs toybox".

I do _not_ want to say "busybox has a useable shell and toybox still doesn't" in
said talk, so... working on it. :)

> grep
> 
> aiui you still had some rewrite you wanted to do, so i'm holding off
> on grep (both for the device and for the host) for now.

Two actually, but they should both just be performance.

I think I know how to redo do_lines() so it mmap()s or read()s a large chunk of
data all at once and hands pointer+length off to the callback. But there's some
design fiddliness: It's tempting to mmap() because then I can do huge chunks of
the file for free (the memory is page cache, no second copy!) and basically
never have to worry about line continuations, plus I kinda get I/O threading for
free with the OS faulting the data in behind the process's back). The downside
is A) I still need to implement a read() codepath for pipes, B) 32 bit address
space exhaustion with large files so it still needs a line continuation
codeapth, C) if I don't NULL terminate each line of input I need to audit all
the users and if I _do_ I'm dirtying pages and breaking copy-on-write with the
page cache and causing the copying I was trying to avoid and the common case is
everything gets copied anyway.

So... Doing mmap() at all isn't necessarily a win unless I can figure out how to
be clever about it, which is why I haven't started that one yet.

Over on the write site it's the line buffering thing, which is a larger issue
and not just grep and is really a continuation of "xprintf() shouldn't flush" so
much. You may not care about "grep | less" providing smooth interactive updates
but I do, and it turns out to involve violating posix with a spoon.

> sed
> 
> i haven't yet flipped back to the toybox sed because of I/O and time
> off, but i'll do that as soon as i get back again.

I got the performance work done there, my pending todo things there are stuff
like teaching it about:

  sed -ne "/blah/,+3p"

> date/timeout
> 
> i believe these are both ready for the build, but i need to update the
> host prebuilt first. i'll do that as soon as i'm back too.

Nothing springs to mind as todo items for these. (There's usually something if I
dig deep enough for that's "collate todo lists" territory...)

> find
> 
> i've just sent you a patch adding -printf. i haven't actually tried
> building AOSP with that yet, and i've only tested it manually, but it
> looks plausible. like i said in the commit message, feel free to sign
> me up for any additional work there.

Working on that one. (Merged what you did, but want to redo bits of it...)

Rob