[Toybox] Project progress for 0.8.3.

[Rob Landley]

On 5/29/20 2:00 PM, enh wrote:
> On Sun, May 24, 2020 at 1:55 PM Rob Landley <rob at landley.net> wrote:
>>
>> Two weeks since the release and Wikipedia[citation needed] hasn't noticed yet,
>> which is fine and normal (I just don't want them to be actively _wrong_), but
>> their "Project progress" section explains that "in 2015"... which was 5 years
>> ago now? And they still link to http://www.landley.net/toybox/todo.txt from 2011
>> which is PURELY HISTORICAL...
> 
> wikipedia doesn't update itself, you know :-)
> 
> (if you were actually just trying to get someone to do it for you: done.)

Editing wikipedia is against my religion.

>> That leaves getopt, which is legitimately stuck in pending due to being a
>> design-level nightmare: the one that's there works fine but pulls in a whole
>> second set of option parsing logic from lib.c that nothing else uses.
> 
> that's kind of the _point_ of that command :-)

Maybe I can move it to toys/examples so it's not part of defconfig? No?

Who _does_ use getopt (but not getopts)? I current need to implement getopts as
part of the shell stuff, and can cycle back to looking at this after that.
(Hopefully with enough information about what it either does that I could
understand why anyone would want to use it.)

>> sha256sum and sha512sum are tangled into the "sha-3" todo item (see also
>> sha224sum sha384sum and sha3sum). I did my own md5sum and sha1sum
>> implementations way back when (which are merged and share code) and I want to
>> see if I can fit the rest in there, but haven't sat down to really focus on it
>> yet. (It's mathy, there's research. Like the compression algorithms. I do NOT
>> wanna be distracted from chewing on it halfway through, so big chunka time.)
> 
> for anyone happy with using libcrypto's optimized code, though, these
> are all done. count them as 50%? :-)

Oh sure. I think I covered that later.

This isn't necessarily for "general use", this is for "auditable base" which
must be self-contained with no external dependencies.

Longer term I'd also like to do qcc and get the system building itself with that
so the toolchain is _also_ part of the auditable base. The weakness in this
theory is that I'd then use llvm's C backend to compile llvm to C and build that
under qcc, to natively bootstrap up into an environment that can run insane C++
specification du jour (no really, we've fixed it all THIS time, and if not try
next year!)

And the problem with building the C version of LLVM is how would you spot a
trusting trust attack in LLVM? If it compiled itself to C code containing the
exploit, which was faithfully compiled by qcc into a runnable binary (which you
then rebuild with itself to get a 3x efficiency boost)... it's still a trusting
trust exploit injection vector.

You'd have to audit the C version of LLVM produced by the LLVM backend to spot
trusting trust exploits in there, because LLVM isn't written in C and needs
itself to compile itself. (Which is doable, but moderately horrific. Or you
could audit the resulting binary, which at least nothing _else_ has been built
with yet so still no circular dependencies in the base OS...)

>> I started cleaning up "man" once and other people were changing it while I was
>> changing it, so I deleted my changes and merged theirs, and haven't looked at it
>> again since. Possibly it's quiet enough now, but I haven't cycled back to it.
> 
> ugh, yeah, i need to dig out my man changes too. unfortunately they're
> currently in a bit of a local optimum where i need to rewrite quite a
> bit _again_ now i've uncovered a few more worms.

No rush, but I'm not touching it until it stops moving. :)

>> stdbuf is sort of a wrapper that intercepts a command's stdin/stdout and does
>> reads/writes of different sizes at it. I keep meaning to give it a closer look
>> to see if it's worth bothering.
> 
> does this even work for libcs other than glibc? i thought it just sets
> an environment variable and execs, with glibc doing the actual work?

Yes and no. At least _some_ of what it does seems to be it reads data and sends
it along at a different granularity? It's sort of like a line oriented "dd" that
potentially handles input _and_ output for a child process.

But yeah, there's magic:

  $ env -i /usr/bin/stdbuf -i 0 -o 0 -e 0 $(which toybox) env
  _STDBUF_I=0
  _STDBUF_O=0
  _STDBUF_E=0
  LD_PRELOAD=/usr/lib/x86_64-linux-gnu/coreutils/libstdbuf.so

I'm not doing an LD_PRELOAD, and if you can't without one I don't think this
belongs in toybox. :P

>> runcon is also already implemented. It's more selinux nonsense and I haven't got
>> the selinux libraries installed on my system so the compile time probes make it
>> drop out, that's why it's in the list. (It's not in _my_ defconfig, but that's a
>> false negative. :)
>>
>> getevent is an android thing: on the one hand elliott hasn't removed it from the
>> android part of the roadmap (which he maintains at this point), on the other
>> there isn't one in toybox. *shrug*?
> 
> i've considered removing it from the list, given that i don't plan on
> writing a toybox version, and wouldn't necessarily even switch to a
> toybox version if there was one --- the Android version effectively
> self-updates based on the kernel headers it's built with, and that's a
> really nice feature i wouldn't want to lose (and i don't think you
> want "point toybox to a set of kernel headers" in toybox). we used to
> update manually, and it was always out of date. (you'd be surprised
> how many keys/switches/buttons still get added.)

It's not in the debian base install, and the description at:

  https://source.android.com/devices/input/getevent

really sounds like a debugging tool. Removing it might make sense, doesn't seem
like something toybox really needs.

What it REALLY sounds like is there should be a module you can load to get a
/proc /sys or /dev thing you listen to and get events from (or netlink, or a
hotplug handler), and userspace doesn't need huge amounts of plumbing to parse a
text result. (The kernel has lots of config options for human readable output, I
think the first was KALLSYMS.) "Need to parse version-specific kernel headers to
build userspace tool" is halfway to "just have the kernel do it already".

*shrug* Not my area...

>> (Of course "make" isn't in the 1.0 roadmap, that's qcc. Getting android
>> self-hosting through a minimal binary-auditable native build environment is the
>> NEXT big can of worms. It would be nice if AOSP pulled the NDK as a build
>> prerequisite, but they're not there yet...)
> 
> there are inherent circular dependency issues with AOSP and the NDK
> and LLVM. at the moment the basic flow is LLVM -> AOSP -> NDK (with
> some of these feeding back too).

Someday, I hope to be in a position to help shovel out the AOSP build
dependencies, but my plate's overflowing at present.

Rob