[Toybox] Did I mention the release?

[Rob Landley]

On 03/15/2015 08:47 PM, enh wrote:
> this is _slightly_ misleading. you really want to look at the second
> list in that file. the first list tells you what's built into our
> toybox binary. the second list tells you what gets a symlink in
> /system/bin. so it's stuff from pending that's on the second list we
> _really_ need to worry about because those are the things people are
> going to notice.
> 
> that said... anything on the first list but not the second is probably
> particularly interesting to you because -- for one reason or another
> -- they don't seem quite ready yet.

I'm still grinding away at ps.c, which is slowly turning into a complete
rewrite and bringing up some really interesting corner cases where what
posix says and what ubuntu does do not match...

>> So these are my current cleanup priorities. I'm currently poking at ps
>> cleanup. (I also promised someone else to work on mdev this month.)
> 
> as a particularly observant follower of my AOSP gerrit activity might
> discern, from the fact i committed a small improvement to the
> _tool_box ps last week, i'm not expecting to switch to the toybox ps
> in the near future.

I wouldn't switch to what we've got right now either. That's why I'm
fixing it. :)

That said, I should look at toolbox ps and see what _it_ does, because
it's at least as much an example as ubuntu.

> there are two slightly conflicting -- because available time is small
> and finite -- goals to choose from here, and it's perhaps worth
> stating them explicitly. goal 1 is "move the maximum number of toolbox
> tools over to toybox toys" and goal 2 is "have those toys that have
> replaced tools be highly polished". for me, goal 1 is the primary
> goal. a lot of the toolbox tools were really bad and almost any change
> would have been an improvement; others (touch, say) had one thing they
> did better that was easily ported to toybox, some had very unusual
> behaviors (ps, say), and relatively few (dd being the sole remaining
> example in toolbox) were of pretty high quality (in dd's case because
> it's the NetBSD dd).

I'm all for it. My own domain expertise leans towards #2, but that's why
it's good to have multiple people contributing to a project. :)

Sigh. I did half of a dd rewrite last year, which derailed into "all
this comma separated option parsing is _almost_ common code with mount
-o, I should write mount first but I need to write umount before that
and for mount -o loop and umount -d I need a way for those to call
losetup and read its stdout but not factoring the common code out or
writing into toybuf in a horrible way, this should be infrastructure
that'll serve toysh's needs when the time comes, how do I design that..."

I should get back to that. But right _now_ what I really should be doing
is working on the ELC talk slides that are theoretically due today. Of
course writing documentation about a thing you maintain always turns
into a fixing spree because "no, I"m not explaining the broken bit here,
I'll just clean it up" and then 4 hours later you're rewriting
xpopen_both() for nommu support...

(Back when I played world of warcraft, my sidequest slots were always
full. This should come as a surprise to nobody)

> so for me i'd like to get the remaining SELinux tools turned into toys
> and merged (last week's patches cover two of the three missing
> commands, and then there are just the missing -Zs in existing
> commands, such as ls), i'd like to fix the one glaring bug in dd (the
> human-readable output bug i sent a patch for) and switch us over so
> we've got as much time as possible to find any remaining bugs, and i
> need to find the time to work out whether the toybox mount/unmount can
> replace the toolbox ones or whether there's anything subtle but
> important in the toolbox ones.
> 
> here's what's still in toolbox right now...
> 
> various things that toybox has, but not [necessarily] in a form we can
> switch over to today:
> 
> toolbox/df.c
>   missing -h; column alignment broken. (but no need to support old
> Android output format.)

I was addressing that at one point. Got distracted. I'll bump it up the
todo list.

> toolbox/upstream-netbsd/usr.bin/du/du.c
>   our default block size was 512 (but people will probably be pleased
> by that change); -h output slightly different. [patch sent upstream]

I have a dozen tabs with patches open, and can do a new scan of the list
history after that. (So nice to have the web archive working again.
Really hope it sticks.)

> toolbox/ls.c
>   missing -Z (-h would be nice, but toolbox doesn’t have that either)

I really want to genericize lib/lib.c:human_readable() so it can
reproduce the variants we need, and factor out the column spacing code.

What would really help here is test cases. (Extract this tarball, cd in
and ls -h, and the output should look like this textfile...)

> toolbox/ps.c

Workin' on it.

> toolbox/top.c
>   output, options, and behavior all differ enough that any switchover
> is going to be interesting. punt for now, worry about these later.

I would very much like to unify top, htop, and iotop (but haven't looked
into how the third is implemented). Also there's the possibility of
common code with vmstat and ps.

The other thing is that "top" does terminal control similar to "less".
And "watch" should do the same, and ps does a _little_ of the same.
(They query the terminal size and truncate output at screen width and
screen height.) So my plan was to genericize _that_ code, in a way that
works with shell command line history editing and vi.

None of which is directly relevant to you, you just want it to work. But
this is why promoting it is harder than it looks...

> toolbox/renice.c
>   ours has nonstandard -r, -t (equivalent to -n?), -g (“get”); do we need these?

Android is as least a much a standard as gnu. If you've got existing
users who will miss the options, patches welcome.

> toolbox/mount.c

I think my mount implementation should be able to hold weight... as of
monday. :P

(I.E. when I fixed the darn mount -a segfault. That was embarassing. I
tested mount -a at some length... and then implemented -O in a way that
if you _don't_ provide -O it dereferences a null pointer. I.E. missing
regression tests to find where stuff that used to work stops working,
which means I need to implement the "test commands as root"
infrastructure using aboriginal linux. That used to be hard because the
root filesystem was a read only squashfs, but now that aboriginal is
switched over to using initramfs I can edit / easily without screwing up
a shared image in a persistent way, so that's unblocked and I should go
do it.)

> toolbox/route.c

I should clean that up, ifconfig is kinda useless without it...

> toolbox/umount.c
>   work out whether toybox is superset.

If it's not, let me know so I can fix it.

> three SELinux commands:
> 
> toolbox/load_policy.c (patch sent upstream)
> toolbox/restorecon.c
> toolbox/runcon.c (patch sent upstream)

Indeed.

> the NetBSD dd implementation (patch sent upstream for the one obvious
> bug i found in manual testing of the toybox dd):
> 
> toolbox/upstream-netbsd/bin/dd/args.c
> toolbox/upstream-netbsd/bin/dd/conv.c
> toolbox/upstream-netbsd/bin/dd/dd.c
> toolbox/upstream-netbsd/bin/dd/dd_hostops.c
> toolbox/upstream-netbsd/bin/dd/misc.c
> toolbox/upstream-netbsd/bin/dd/position.c
> toolbox/upstream-netbsd/lib/libc/gen/getbsize.c
> toolbox/upstream-netbsd/lib/libc/gen/humanize_number.c
> toolbox/upstream-netbsd/lib/libc/stdlib/strsuftoll.c
> toolbox/upstream-netbsd/lib/libc/string/swab.c
> toolbox/upstream-netbsd/lib/libutil/raise_default_signal.c

Possibly I should have a "how to make testcases" section in my toybox
talk on wednesday. A good test suite really helps development. (Alas, as
I've learned, what counts as good is another thing apparently requiring
some explanation...)

> toolbox/uptime.c
>   Android's output format is completely different (all about time
> spent idle/sleeping, not concurrent users or processes); not clear
> that merging makes sense.

I should take a look. (Is there documentation on any of the android
commands? Each toybox command at least produces --help output, and the
test suite is gives a gazillion usage examples, or at least _should_...)

> various common-enough Linux tools that toybox doesn't have (should
> look into turning these into toys instead, but that can wait):
> 
> toolbox/iftop.c
> toolbox/ionice.c
> toolbox/lsof.c

I see that those are in my roadmap _because_ they're in android. Two of
them I use, the third I was unfamiliar with but it's nice to know that's
a thing. (I use the xubuntu toolbar widget to do that...)

> plus Android-specific things that i plan on leaving in a stub toolbox
> for the time being (other than the fact i'd like to use the toybox
> infrastructure in some of these, i'm not sure either side gains much
> by having Android-specific stuff in toybox, but that's something we
> can worry about when we've sorted out the things above):

Toybox was always intended to work alongside other packagesin the same
$PATH, but toybox is also explicitly designed to allow forks to add new
commands unobtrusively. Not only is each command self-contained (a trick
I had to explain in person Denys Vlasenko at the 2010 ELC before he
started doing it in busybox), but the directories are self-contained too.

So if you want a locally-maintained directory of android commands, all
you have to do is:

1) mkdir toys/android
2) echo "Android" > toys/android/README
3) cp mycommand.c toys/android
4) make distclean
5) make defconfig
6) make

(Once upon a time there was a directory list in scripts/genconfig.sh to
manually control the ordering, but I ripped it back out and just did a
"sort -r" to put posix first and examples last and called it good enough.)

As for additions to the lib directory, the build just grabs lib/*.c and
then lets "-ffunction-sections -fdata-sections -Wl,--gc-sections" sort
it out, so you can add a lib/android.c and as long as the names don't
conflict life is good. (I'm aware your build doesn't, but presumably
adjusting your build is easier for you than for me. :)

Adding stuff to lib/lib.h is bad (merge conflicts), but you could
#include "toys/android/lib.h" after the #include "toys.h" in your local
commands, and have the library header live there.

(You could even build with just the android commands enabled, mv toybox
toolbox, and point symlinks at that during the install if all you wanted
was the toybox infrastructure but wanted to keep it separate. Or use
scripts/single.sh or the new "make change" target, although I dunno how
that translates to your build infrastructure.)

So if you do want locally maintained toybox commands that are only of
interest to android (maybe as a git subrepo or repo equivalent), it
should be reasonably straightforward.

That said, some of your commands:

> toolbox/getevent.c
> toolbox/getprop.c
> toolbox/ioctl.c
> toolbox/log.c
> toolbox/nandread.c
> toolbox/newfs_msdos.c
> toolbox/prlimit.c
> toolbox/r.c
> toolbox/schedtop.c
> toolbox/sendevent.c
> toolbox/setprop.c
> toolbox/smd.c
> toolbox/start.c
> toolbox/stop.c
> toolbox/watchprops.c

I'm doing a mkdosfs that should cover newfs_msdos, nandread is a
perennial not-quite-request, ioctl looks useful, prlimit and sendevent
look generally linuxy (which implies getevent is too but it's too big
for me to glance over). I've seen like 8 different implementations of r.
(The one at $DAYJOB is "readmem".)

I see you've yanked schedtop and smd.

I agree the getprop/setprop/watchprops stuff is android-specific,
although it looks like setprop.c wants to iterate over its arguments,
and then start and stop become scripts calling setprop? (Why doesn't
setprop have a key=value syntax? I am behind on my reading for android.
Possibly these commands have different permissions... selinux extended
attributes? Or maybe setprop is only a debuging command or something?)

> i also plan on sticking with BSD grep for now (we can worry about the
> quality of the BSD regex implementation in bionic and fleshing out the
> missing grep features in toybox when everything else is settled):

No rush. The "regex|regex" thing with multiple grep -e is something I
need to figure out how to deal with anyway. (The problem is | only means
something in extended regexes. I had a patch to musl to allow \| to work
in non-extended regexes, but then they rewrote their regex plumbing. I
may have to break down and do multiple passes over the data, or maybe
extend and factor out the ghostwheel() function in sed.c.)

(I finally did have somebody email me about the naming convention in
sed.c, whereupon I pointed them at Roger Zelazny's "nine princes in
amber". After enough repetitions of "pattern" you throw in a "logrus"
out of self defense and it sort of spirals from there. Ordinarily I
clean up my naming conventions before checking things in, but xpipe()
uses cestnepasun as a variable and in the ps cleanup I currently have a
struct fields {}; instance named strawberry. I do that sort of thing
while programming all the time, but generally don't check it in...)

> toolbox/upstream-netbsd/usr.bin/grep/fastgrep.c
> toolbox/upstream-netbsd/usr.bin/grep/file.c
> toolbox/upstream-netbsd/usr.bin/grep/grep.c
> toolbox/upstream-netbsd/usr.bin/grep/queue.c
> toolbox/upstream-netbsd/usr.bin/grep/util.c
> 
> so -- though you didn't ask -- dd/df/du seem like priorities to me
> because they seem close to being done, and the SELinux stuff because
> they're just ports. there's a perverse incentive at play here too that
> for me adding a new command (even if only to pending) will always be
> the most urgent thing because it's always going to be easier for me to
> cherrypick a fix than it is for me to switch to something "new". the
> other incentive is that making a toy the default implementation means
> we'll get to find what, if anything, actually causes trouble for
> developers and can prioritize based on feedback.

Sounds like fun.

>  --elliott

Rob

 1426816366.0