[Toybox] ps and top (and Android)

[Rob Landley]

On 04/23/2016 11:57 AM, enh wrote:
> On Sat, Apr 23, 2016 at 3:06 AM, Rob Landley <rob at landley.net> wrote:
>> On 04/21/2016 08:59 PM, enh wrote:
>>> On Thu, Apr 21, 2016 at 1:06 AM, Rob Landley <rob at landley.net> wrote:
>>>> On 04/20/2016 08:47 PM, enh wrote:
>>>>> re: https://github.com/landley/toybox/commit/529d5db29a32db565d2ee0aeac133e47ec045e12,
>>>>> note that we're not switched over because neither ps nor top have
>>>>> thread support. everything on Android is threaded except init, so
>>>>> people would notice :-)
>>>>
>>>> I was tracking whether or not it was in pending, but you're right. My bad.
>>>
>>> well, it's your roadmap --- you get to choose what it means :-) i just
>>> wanted to point out that we might have been interpreting it
>>> differently.
>>
>> Indeed, but you maintaining a toybox roadmap provides me with
>> information I wouldn't otherwise have. (I can scribble my plans down in
>> a textfile, and they're subject to change anyway.)
>>
>> (I still occasionally check https://wiki.tizen.org/wiki/Toybox but other
>> than typo fixes in January it hasn't been touched in two years. I
>> haven't figured out how to follow cyanogenmod's toybox discussions yet...)
> 
>  i don't know about discussions, but they certainly know to submit bug
> fixes here instead of AOSP (because we've told them a few times), so i
> doubt you'll miss out on much.

I grabbed their git repo for the flight. If I fix the same issues they
never told me about in a different way it'll cause conflicts, but
presumably if I did anything _else_ with those files it would to, so I
can't feel too guilty about it...


>> (Woo, 149 commands!)
> 
> note that there's a second-level distinction between "in toybox" and
> "gets a symbolic link".

I know. And the 149 number is wrong because I disabled selinux to build
that (headers not installed on my local box) and that ripped out half a
dozen commands I put back for the rest of the analysis...

Still, if somebody wants to set up a development environment on a stock
android system, they can add the symlinks to the $PATH themselves
without having to build anything. That's just a minor config tweak...

> (aiui from a few bug reports that came to us by mistake, cyanogen
> _doesn't_ make that distinction --- they have a symlink for everything
> that's compiled in, which causes trouble/confusion sometimes.)

I'm happy to fix any issues they have, if they let me know about them.
(Modulo the todo list making exactly _when_ those things get fixed
kinda... fuzzy.)

> looks like there are 128 symlinks.

Woot!

>> Pending commands included in the build:
>>
>> $ make list_pending | tr ' ' "\n" | \
>> egrep "^($(./toybox | tr '\n ' '||'))\$" | xargs
>> dd expr lsof more netstat route tar tr traceroute traceroute6
> 
> the dd symlink is still to toolbox: we still use the BSD dd.

I did half the cleanup to this once (sometime before
http://landley.net/notes-2014.html#08-07-2014), but it got blocked on
needing comma separated string handling infrastructure, which turned
into implementing "mount" that used that, and then "ps" expanded it
more, by the time I was happy with the infrastructure I'd lost that
working directory in the shuffle before I got back to it. (I do remember
deciding NOT to do ebcdic support, and head-scratching at the separate
input and output blocking requirements, and people in IRC seriously
trying to convince me that the posix dd spec was an elaborate practical
joke on the part of IBM and never meant to be implemented by anyone...)

So I need to do a fresh dd cleanup...

> i'm removing route in O at the request of the networking folks..

It was in pending for a reason. Large Company That Prefers To Remain
Anonymous sent it to me, and although I cleaned up ifconfig I never got
around to cleaning up route because I needed to do research on missing
features (possibly the exact things your networking guys were
complaining about. :)

Plus I wanted an iptables implementation, but now it looks like that's
going the way of ipchains and being replaced with this funky bpf stuff
and I'm not enough of a sysadmin to be up to date on this...

> lsof is a port of the old toolbox lsof, and netstat is a superset of
> the old toolbox netstat.

All for it. I should prioritize cleaning them up. (lsof does an awful
lot of stuff, without a spec. there are all sorts of little thingslike

> more i've worked a lot on (and you have two more patches from me to
> apply) and is used a fair bit.

Indeed. That's why I'm looking at promoting the one that's there instead
of just doing less, but as I've posted close examination brings out a
number of issues. (Maybe nobody else cares, but I do.)

> expr is apparently not in good shape judging by all the recent
> patches, but no one ever complained to me so either it's not really
> been used yet or it's only used for trivial stuff so far.

I can fix the allocation lifetime stuff, but really I want to rewrite
this so the same plumbing can implement $(( )) for the shell. (That was
ALWAYS my plan, but I got an external contribution and didn't want to
block an existing impementation with an idea I didn't have time to work
on yet...)

> tar and tr work well enough for a few scripts that folks were trying
> to use (though i think the cyanogen folks had trouble with tar and
> long filenames).

tar, zip, gene2fs, genvfatfs, and so on all required basically the same
 plumbing. The dirtree stuffis reasonably in shape now (although I need
to change it so rm -rf can handle infinite recursion depth, which means
not keeping a filehandle open per level, which means recovering closed
filehandles via openat(dirfd, "..") and then comparing the stat values
(we'll have to keep filehandles for symlinks we recursed into, which rm
should never do anyway, that's -L behavior in various commands, but we
can close directory ones and then just have a recovery mechanism for the
"somebody did a mv on an ancestor directory while we were traversing"
case. For rm we can just call it pilot error and ignore everything we
can't re-traverse down from the top to reach, or just error out and go
"traversal error" and have them re-run rm if they still need to delete
stuff. Not sure what that recovery mechanism should look like yet.)

Anyway, the thing about gene2fs (as an archiver) is it needs to grab all
the metadata before it writes out the superblock, so it needs to travere
and snapshot the entire hierarchy, THEN output. (If you "gene2fs dir |
gzip" you can't seek the output file, and ext2 has the metadata before
the file contents, with lots of directories and allocation tables and
such that need to know where the data will eventually go. I can probably
cheat to find sparse files using the block allocation stuff though... :)

Anyway, that's why I hadn't tackled tar yet. (And zip needs the deflate
compression side finished, which I haven't had time to work on in ages.)

> the traceroutes don't have symbolic links and are really just
> long-term TODO items for me; maybe one day we'll be able to replace
> the iputils one.

Again, external contributions I havent' ahd a change to look at. There
was an externally contributed ping too, but I really wanted to do it
another way (using the non-root api)...

>> The four that jump out at me are link, unlink, uuencode, and uudecode,
>> which are in posix. I'm aware ln and rm cover the first two, and base64
>> more or less covers the rest, and yet:
>>
>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/link.html
>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/unlink.html
>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/uuencode.html
>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/uudecode.html
> 
> it turns out no-one born later than about 1985 has even heard of
> uuencode/uudecode. it has to be explained as "a historic form of
> base64". yagni.

Understandable. (You're going farther trimming the posix list than I did.)

I primarily use this to cut and paste files through a serial console,
but you can use base64 just as well...

> (bionic also doesn't have the posix a64l/l64a for
> similar reasons, although the fact that they explicitly operate on a
> static buffer is the biggest nail in their coffin.)
> 
> link/unlink was a surprise to even the old-timers, and seem to offer
> nothing other than confusion beyond the well-known ln/rm. yagni.

I did them because they were in tiny, in posix, and I'm not sure there
are any package builds that don't use them. That said, if I find a
package that does it can be patched. I agree that ln and rm are the
modern ways to do both.

>> eject: is that actually used for anything but CD/DVD drives these days?
>> I suppose it's really a special case of the ioctl...
> 
> yeah, and i no longer have any device other than my car that still has
> an optical drive.

Indeed.

I have now bought two weird al yankovic albums that I can't listen to
because no cd drive (except the game consoles hooked up to the tv with
unimpressive speakers). I just listened to the relevant videos on
youtube, but was recently blindsided by one I missed:

  http://landley.net/notes-2016.html#01-04-2016

>> hexedit: perhaps an acquired taste. (And I still need to work out how to
>> make it insert and delete, and let you cursor over into the ASCII
>> display part and type stuff in there...)
> 
> no one has asked for a hex editor yet. vi is what the people really want :-)

Working on it.

>> hostid: maybe not relevant in context? Large Company's developer
>> submitted it and it should REALLY be showing ipv6 addresses these days
>> instead of ipv4. Its standardness is dubious...
> 
> bionic doesn't have gethostid(3), and it's not obvious it makes any
> sense in 2016 anyway.

Indeed.

I should have a roadmap section of commands android has explicitly
chosen NOT to enable.

>> iotop: If the android kernel doesn't have CONFIG_IRQ_TIME_ACCOUNTING
>> enabled, this won't do you a whole lot of good. :)

I note that if you DO have it enabled, it's kind of nice. :)

(Requires suid bit for full process coverage, though. Otherwise it just
shows you the I/O bandwidth used by _your_ processes...)

>> lspci: most android devices don't have pci busses, but android gets used
>> in the embedded space sometimes... *shrug* (I worked on one that did in
>> 2012, but the product was doomedish and the company recently got acquired.)
> 
> what we really don't have is /usr/share/misc/pci.ids.

Indeed, but it does produce output without that, and you can point it at
a different location with -i. (All the toybox stuff needs to work
standalone if you drop a static binary on a system.)

I suspect lsusb would be more useful to you, and I think you've got that
one. (Same lack of id's file with that by the way. That doesn't even
have the capability to support it, although I should add it. The
corresponding file is http://www.linux-usb.org/usb.ids .)

>> nproc: prints number of CPUs. Might be useful, and is in modern
>> coreutils, but you can get the info out of /proc or /sys if you need to.
> 
> "what does number of CPUs mean?" is one of the most popular Android
> questions. nproc only gives one of the possible answers, so probably
> more harmful than useful.

Two, actually. The default says how many processors this task can bind
to (according to the current affinity mask), and --all gives
sysconf(_SC_NPROCESSORS_CONF).

If there's more options than that... ouch.

(I didn't invent this command, it's in coreutils.)

>> nsenter/unshare: Someday, opening the containers can of worms would be a
>> really nice thing for Android to do. But we've talked about how there's
>> Much Design Work to do to make that happen. Still, if the kernel
>> plumbing's enabled, maybe for O?
>>
>> readahead: was requested. Seems like a redhat-ism to me.
> 
> for the most part i took the view of "if it's not in the base goobuntu
> install, no one cares".

I dunno where to find google's ubuntu. In xubuntu taskset and unshare
are there by default. :)

> (not foolproof, because what's useful on a
> developer's workstation and what's useful on Android aren't
> necessarily the same, but it's a reasonable place to start.)

*shrug* Readahead was requested. I personally don't use it, and am
unlikely to miss it. :)

>> shred: was also requested. Probably useless on log based flash
>> filesystems, or even stuff like btrfs that can snapshot (let alone have
>> multiple heads).
>>
>>> i'm on the opposite end of my
>>> release cycle now, where now's a great time to make big changes in
>>> AOSP master, for O next year. which is why my mind's back on ps and
>>> top.
>>>
>>> dd too, though as i said last week i plan on waiting for that to leave
>>> pending because (unlike expr) it's heavily used.
>>
>> Gotit: ps, top, and dd are android todo priorities. I also have mtools,
>> genvfatfs, mke2fs, and gene2fs on the request list. Plus the stuff used
>> out of pending.
> 
> netcat used to be on that list, but it stopped building at some point.
> i can't remember whether i bothered to send you a patch. but long-term
> it would be nice to remove external/netcat too. (toybox is missing -4,
> -6, and -U.)

Indeed. All three are on my todo list. (Adding -u lets you do netconsole.

Networking in general is a can of worms I should dive into in one go.
The tcpsvd implementation I got came in _after_ netcat had a server
mode, and they should _really_ share code. I need to clean up wget and
integrate the pending https support, I'm using ftpget and ftpput out of
busybox in the aboriginal linux build stuff, I'd just started writing
tftp and tftpd when external versions came in...

It's a question of which can of worms to focus on...

(Right now, for example, I'm trying to close tabs so I can take my
second battery out of the box and charge it for the international
flight. Alas, swapping batteries powers off the machine...)

Rob