[Toybox] Did I mention the release?

[enh]

On Thu, Mar 19, 2015 at 6:52 PM, Rob Landley <rob at landley.net> wrote:
>
> On 03/15/2015 08:47 PM, enh wrote:
> > this is _slightly_ misleading. you really want to look at the second
> > list in that file. the first list tells you what's built into our
> > toybox binary. the second list tells you what gets a symlink in
> > /system/bin. so it's stuff from pending that's on the second list we
> > _really_ need to worry about because those are the things people are
> > going to notice.
> >
> > that said... anything on the first list but not the second is probably
> > particularly interesting to you because -- for one reason or another
> > -- they don't seem quite ready yet.
>
> I'm still grinding away at ps.c, which is slowly turning into a complete
> rewrite and bringing up some really interesting corner cases where what
> posix says and what ubuntu does do not match...
>
> >> So these are my current cleanup priorities. I'm currently poking at ps
> >> cleanup. (I also promised someone else to work on mdev this month.)
> >
> > as a particularly observant follower of my AOSP gerrit activity might
> > discern, from the fact i committed a small improvement to the
> > _tool_box ps last week, i'm not expecting to switch to the toybox ps
> > in the near future.
>
> I wouldn't switch to what we've got right now either. That's why I'm
> fixing it. :)
>
> That said, I should look at toolbox ps and see what _it_ does, because
> it's at least as much an example as ubuntu.

i think the most important thing is to implement -o well. other than
-Z which Ubuntu has too, i think the only oddity that we wouldn't be
better off without is --abi. right now developers often care about
whether a process is 32 or 64 bit. the main behavioral oddity is that
it defaults to showing all processes (because people are usually doing
"adb shell ps" rather than "ps") and the last unrecognized argument
that don't start '-' or with a digit is taken as a "name filter" ---
only processes whose COMM or NAME exactly matches are shown. i suspect
that this is an exact match means it's not used all that much, but
it's hard to know without taking it away and seeing if anyone
complains. i might be reduced to that :-(

it might be worth my while introducing toybox's pgrep as soon as
possible and encourage people to switch over.

> > there are two slightly conflicting -- because available time is small
> > and finite -- goals to choose from here, and it's perhaps worth
> > stating them explicitly. goal 1 is "move the maximum number of toolbox
> > tools over to toybox toys" and goal 2 is "have those toys that have
> > replaced tools be highly polished". for me, goal 1 is the primary
> > goal. a lot of the toolbox tools were really bad and almost any change
> > would have been an improvement; others (touch, say) had one thing they
> > did better that was easily ported to toybox, some had very unusual
> > behaviors (ps, say), and relatively few (dd being the sole remaining
> > example in toolbox) were of pretty high quality (in dd's case because
> > it's the NetBSD dd).
>
> I'm all for it. My own domain expertise leans towards #2, but that's why
> it's good to have multiple people contributing to a project. :)
>
> Sigh. I did half of a dd rewrite last year, which derailed into "all
> this comma separated option parsing is _almost_ common code with mount
> -o, I should write mount first but I need to write umount before that
> and for mount -o loop and umount -d I need a way for those to call
> losetup and read its stdout but not factoring the common code out or
> writing into toybuf in a horrible way, this should be infrastructure
> that'll serve toysh's needs when the time comes, how do I design that..."
>
> I should get back to that. But right _now_ what I really should be doing
> is working on the ELC talk slides that are theoretically due today. Of
> course writing documentation about a thing you maintain always turns
> into a fixing spree because "no, I"m not explaining the broken bit here,
> I'll just clean it up" and then 4 hours later you're rewriting
> xpopen_both() for nommu support...
>
> (Back when I played world of warcraft, my sidequest slots were always
> full. This should come as a surprise to nobody)
>
> > so for me i'd like to get the remaining SELinux tools turned into toys
> > and merged (last week's patches cover two of the three missing
> > commands, and then there are just the missing -Zs in existing
> > commands, such as ls), i'd like to fix the one glaring bug in dd (the
> > human-readable output bug i sent a patch for) and switch us over so
> > we've got as much time as possible to find any remaining bugs, and i
> > need to find the time to work out whether the toybox mount/unmount can
> > replace the toolbox ones or whether there's anything subtle but
> > important in the toolbox ones.
> >
> > here's what's still in toolbox right now...
> >
> > various things that toybox has, but not [necessarily] in a form we can
> > switch over to today:
> >
> > toolbox/df.c
> >   missing -h; column alignment broken. (but no need to support old
> > Android output format.)
>
> I was addressing that at one point. Got distracted. I'll bump it up the
> todo list.
>
> > toolbox/upstream-netbsd/usr.bin/du/du.c
> >   our default block size was 512 (but people will probably be pleased
> > by that change); -h output slightly different. [patch sent upstream]
>
> I have a dozen tabs with patches open, and can do a new scan of the list
> history after that. (So nice to have the web archive working again.
> Really hope it sticks.)
>
> > toolbox/ls.c
> >   missing -Z (-h would be nice, but toolbox doesn’t have that either)
>
> I really want to genericize lib/lib.c:human_readable() so it can
> reproduce the variants we need, and factor out the column spacing code.
>
> What would really help here is test cases. (Extract this tarball, cd in
> and ls -h, and the output should look like this textfile...)

a tarball or function for all tests to use to test with is a great
idea. it would make it easier to have interesting cases available for
all the commands to be tested against. (dangling symlinks, empty and
non-empty directories, non-regular files, ...)

> > toolbox/ps.c
>
> Workin' on it.
>
> > toolbox/top.c
> >   output, options, and behavior all differ enough that any switchover
> > is going to be interesting. punt for now, worry about these later.
>
> I would very much like to unify top, htop, and iotop (but haven't looked
> into how the third is implemented). Also there's the possibility of
> common code with vmstat and ps.
>
> The other thing is that "top" does terminal control similar to "less".
> And "watch" should do the same, and ps does a _little_ of the same.
> (They query the terminal size and truncate output at screen width and
> screen height.) So my plan was to genericize _that_ code, in a way that
> works with shell command line history editing and vi.
>
> None of which is directly relevant to you, you just want it to work. But
> this is why promoting it is harder than it looks...

for me "as good as the desktop" is a long-term goal that can take
several releases. "better than toolbox" is enough for right now. it
pains me to waste time on things like WCHAN for toolbox ps when i
could have been doing that for toybox ps instead, but until we're
switched over...

> > toolbox/renice.c
> >   ours has nonstandard -r, -t (equivalent to -n?), -g (“get”); do we need these?
>
> Android is as least a much a standard as gnu. If you've got existing
> users who will miss the options, patches welcome.

things like renice are tricky because they're not heavily used. i need
to work out first whether the extra stuff is useful or just random.
(one particular problem is that people have added things that weren't
quite what they really wanted because it was easier to do that and
good enough for what they needed to do, than it was to implement the
full standard option they'd have used if it had been available.)

> > toolbox/mount.c
>
> I think my mount implementation should be able to hold weight... as of
> monday. :P
>
> (I.E. when I fixed the darn mount -a segfault. That was embarassing. I
> tested mount -a at some length... and then implemented -O in a way that
> if you _don't_ provide -O it dereferences a null pointer. I.E. missing
> regression tests to find where stuff that used to work stops working,
> which means I need to implement the "test commands as root"
> infrastructure using aboriginal linux. That used to be hard because the
> root filesystem was a read only squashfs, but now that aboriginal is
> switched over to using initramfs I can edit / easily without screwing up
> a shared image in a persistent way, so that's unblocked and I should go
> do it.)
>
> > toolbox/route.c
>
> I should clean that up, ifconfig is kinda useless without it...
>
> > toolbox/umount.c
> >   work out whether toybox is superset.
>
> If it's not, let me know so I can fix it.
>
> > three SELinux commands:
> >
> > toolbox/load_policy.c (patch sent upstream)
> > toolbox/restorecon.c
> > toolbox/runcon.c (patch sent upstream)
>
> Indeed.
>
> > the NetBSD dd implementation (patch sent upstream for the one obvious
> > bug i found in manual testing of the toybox dd):
> >
> > toolbox/upstream-netbsd/bin/dd/args.c
> > toolbox/upstream-netbsd/bin/dd/conv.c
> > toolbox/upstream-netbsd/bin/dd/dd.c
> > toolbox/upstream-netbsd/bin/dd/dd_hostops.c
> > toolbox/upstream-netbsd/bin/dd/misc.c
> > toolbox/upstream-netbsd/bin/dd/position.c
> > toolbox/upstream-netbsd/lib/libc/gen/getbsize.c
> > toolbox/upstream-netbsd/lib/libc/gen/humanize_number.c
> > toolbox/upstream-netbsd/lib/libc/stdlib/strsuftoll.c
> > toolbox/upstream-netbsd/lib/libc/string/swab.c
> > toolbox/upstream-netbsd/lib/libutil/raise_default_signal.c
>
> Possibly I should have a "how to make testcases" section in my toybox
> talk on wednesday. A good test suite really helps development. (Alas, as
> I've learned, what counts as good is another thing apparently requiring
> some explanation...)

(my [lame] excuse is that even with my patch the output still isn't
_exactly_ the desktop output, and i'm not sure how anal we should be
about that.)

> > toolbox/uptime.c
> >   Android's output format is completely different (all about time
> > spent idle/sleeping, not concurrent users or processes); not clear
> > that merging makes sense.
>
> I should take a look. (Is there documentation on any of the android
> commands? Each toybox command at least produces --help output, and the
> test suite is gives a gazillion usage examples, or at least _should_...)

no. most of the toolbox commands don't/didn't even have _usage_
output. the built-in help is one of my favorite things about toybox.
(it was when i was trying to show it off this morning that i found the
"top --help" bug i sent the patch for. so that was embarrassing.)

Android's uptime looks like this:

$ adb shell uptime
up time: 04:05:17, idle time: 08:00:53, sleep time: 00:00:00

how would you offer both formats? or were you just thinking that we'd
have a big if (ANDROID)? about the only code the two formats would
share would be the duration formatting. and Android doesn't have
anything like utmp, so it's not obvious we could show the regular form
output even if we wanted to.

> > various common-enough Linux tools that toybox doesn't have (should
> > look into turning these into toys instead, but that can wait):
> >
> > toolbox/iftop.c
> > toolbox/ionice.c
> > toolbox/lsof.c
>
> I see that those are in my roadmap _because_ they're in android. Two of
> them I use, the third I was unfamiliar with but it's nice to know that's
> a thing. (I use the xubuntu toolbar widget to do that...)
>
> > plus Android-specific things that i plan on leaving in a stub toolbox
> > for the time being (other than the fact i'd like to use the toybox
> > infrastructure in some of these, i'm not sure either side gains much
> > by having Android-specific stuff in toybox, but that's something we
> > can worry about when we've sorted out the things above):
>
> Toybox was always intended to work alongside other packagesin the same
> $PATH, but toybox is also explicitly designed to allow forks to add new
> commands unobtrusively. Not only is each command self-contained (a trick
> I had to explain in person Denys Vlasenko at the 2010 ELC before he
> started doing it in busybox), but the directories are self-contained too.
>
> So if you want a locally-maintained directory of android commands, all
> you have to do is:
>
> 1) mkdir toys/android
> 2) echo "Android" > toys/android/README
> 3) cp mycommand.c toys/android
> 4) make distclean
> 5) make defconfig
> 6) make
>
> (Once upon a time there was a directory list in scripts/genconfig.sh to
> manually control the ordering, but I ripped it back out and just did a
> "sort -r" to put posix first and examples last and called it good enough.)

oh, great. i hadn't actually looked; i'd just assumed there was a
hard-coded list.

> As for additions to the lib directory, the build just grabs lib/*.c and
> then lets "-ffunction-sections -fdata-sections -Wl,--gc-sections" sort
> it out, so you can add a lib/android.c and as long as the names don't
> conflict life is good. (I'm aware your build doesn't, but presumably
> adjusting your build is easier for you than for me. :)

yes, given that i'd prefer not to have differences from upstream
(which is why i'm not committing any of my own changes locally; i'm
merging them back if/when they make it upstream) it's useful to me
that your build will just pick up extra lib/*.c files too. (i usually
test on the host first.)

> Adding stuff to lib/lib.h is bad (merge conflicts), but you could
> #include "toys/android/lib.h" after the #include "toys.h" in your local
> commands, and have the library header live there.
>
> (You could even build with just the android commands enabled, mv toybox
> toolbox, and point symlinks at that during the install if all you wanted
> was the toybox infrastructure but wanted to keep it separate. Or use
> scripts/single.sh or the new "make change" target, although I dunno how
> that translates to your build infrastructure.)

no, there's no particular benefit to keeping things separate. although
if long term you plan on having a toys/android upstream i might want
to go with toys/android-pending or something so i can keep the two
sets straight.

> So if you do want locally maintained toybox commands that are only of
> interest to android (maybe as a git subrepo or repo equivalent), it
> should be reasonably straightforward.
>
> That said, some of your commands:
>
> > toolbox/getevent.c
> > toolbox/getprop.c
> > toolbox/ioctl.c
> > toolbox/log.c
> > toolbox/nandread.c
> > toolbox/newfs_msdos.c
> > toolbox/prlimit.c
> > toolbox/r.c
> > toolbox/schedtop.c
> > toolbox/sendevent.c
> > toolbox/setprop.c
> > toolbox/smd.c
> > toolbox/start.c
> > toolbox/stop.c
> > toolbox/watchprops.c
>
> I'm doing a mkdosfs that should cover newfs_msdos, nandread is a
> perennial not-quite-request, ioctl looks useful, prlimit and sendevent
> look generally linuxy (which implies getevent is too but it's too big
> for me to glance over). I've seen like 8 different implementations of r.
> (The one at $DAYJOB is "readmem".)

in the short term i'm moving r out into system/extras, since only the
bringup guys need it (and they're the only ones who can use it
anyway).

> I see you've yanked schedtop and smd.
>
> I agree the getprop/setprop/watchprops stuff is android-specific,
> although it looks like setprop.c wants to iterate over its arguments,
> and then start and stop become scripts calling setprop? (Why doesn't
> setprop have a key=value syntax?

probably because just passing argv[1] and argv[2] to a function is
easier than doing anything with the arguments first.

> I am behind on my reading for android.
> Possibly these commands have different permissions... selinux extended
> attributes? Or maybe setprop is only a debuging command or something?)

no. everything in toolbox is in one binary (with the exception of r),
so they're all the same command as far as SELinux is concerned.
setprop works by talking to the property service over a socket, and
that uses SO_PEERCRED, and then there's some kind of SELinux lookup,
and there are extra checks for stuff like the "ro.*" properties, and
...

> > i also plan on sticking with BSD grep for now (we can worry about the
> > quality of the BSD regex implementation in bionic and fleshing out the
> > missing grep features in toybox when everything else is settled):
>
> No rush. The "regex|regex" thing with multiple grep -e is something I
> need to figure out how to deal with anyway. (The problem is | only means
> something in extended regexes. I had a patch to musl to allow \| to work
> in non-extended regexes, but then they rewrote their regex plumbing. I
> may have to break down and do multiple passes over the data, or maybe
> extend and factor out the ghostwheel() function in sed.c.)

(BSD just tries them all in order, separately.)

> (I finally did have somebody email me about the naming convention in
> sed.c, whereupon I pointed them at Roger Zelazny's "nine princes in
> amber". After enough repetitions of "pattern" you throw in a "logrus"
> out of self defense and it sort of spirals from there. Ordinarily I
> clean up my naming conventions before checking things in, but xpipe()
> uses cestnepasun as a variable and in the ps cleanup I currently have a
> struct fields {}; instance named strawberry. I do that sort of thing
> while programming all the time, but generally don't check it in...)
>
> > toolbox/upstream-netbsd/usr.bin/grep/fastgrep.c
> > toolbox/upstream-netbsd/usr.bin/grep/file.c
> > toolbox/upstream-netbsd/usr.bin/grep/grep.c
> > toolbox/upstream-netbsd/usr.bin/grep/queue.c
> > toolbox/upstream-netbsd/usr.bin/grep/util.c
> >
> > so -- though you didn't ask -- dd/df/du seem like priorities to me
> > because they seem close to being done, and the SELinux stuff because
> > they're just ports. there's a perverse incentive at play here too that
> > for me adding a new command (even if only to pending) will always be
> > the most urgent thing because it's always going to be easier for me to
> > cherrypick a fix than it is for me to switch to something "new". the
> > other incentive is that making a toy the default implementation means
> > we'll get to find what, if anything, actually causes trouble for
> > developers and can prioritize based on feedback.
>
> Sounds like fun.
>
> >  --elliott
>
> Rob

 1426830170.0