[Toybox] ps and top (and Android)

[enh]

On Sat, Apr 23, 2016 at 3:06 AM, Rob Landley <rob at landley.net> wrote:
> On 04/21/2016 08:59 PM, enh wrote:
>> On Thu, Apr 21, 2016 at 1:06 AM, Rob Landley <rob at landley.net> wrote:
>>> On 04/20/2016 08:47 PM, enh wrote:
>>>> re: https://github.com/landley/toybox/commit/529d5db29a32db565d2ee0aeac133e47ec045e12,
>>>> note that we're not switched over because neither ps nor top have
>>>> thread support. everything on Android is threaded except init, so
>>>> people would notice :-)
>>>
>>> I was tracking whether or not it was in pending, but you're right. My bad.
>>
>> well, it's your roadmap --- you get to choose what it means :-) i just
>> wanted to point out that we might have been interpreting it
>> differently.
>
> Indeed, but you maintaining a toybox roadmap provides me with
> information I wouldn't otherwise have. (I can scribble my plans down in
> a textfile, and they're subject to change anyway.)
>
> (I still occasionally check https://wiki.tizen.org/wiki/Toybox but other
> than typo fixes in January it hasn't been touched in two years. I
> haven't figured out how to follow cyanogenmod's toybox discussions yet...)

 i don't know about discussions, but they certainly know to submit bug
fixes here instead of AOSP (because we've told them a few times), so i
doubt you'll miss out on much.

>>> Yesterday I found out I'm going to Japan on Sunday, so the release I was
>>> trying to prepare at the end of the month got bumped up. I have a 9 hour
>>> layover in San Francisco on Sunday, and I'm hoping to upload it then.
>>>
>>> I have like 6 bugs I need to fix between now and then (the find
>>> segfault, the new bzcat segfault, expr memleak, etc). Plus I'm in the
>>> process of cleaning up another half dozen commands out of pending, and I
>>> should get at least a couple of them closed out and promoted this release.
>>
>> barring anything huge enough to warrant a cherrypick, what you see in
>> N DP2 (http://developer.android.com/preview/index.html) should be what
>> ships in N as far as toybox goes.
>
> Going by tags that's commit 9fcaca8434ec for android-n-preview-2 (modulo
> the build system churn). Check that out in the android repo, make
> oldconfig, switch off selinux support, build, and:
>
> acpi base64 basename blkid blockdev bunzip2 bzcat cal cat chattr chgrp
> chmod chown chroot cksum clear cmp comm cp cpio cut date dd df dirname
> dmesg dos2unix du echo egrep env expand expr fallocate false fgrep
> find flock free freeramdisk fsfreeze grep groups head help hostname
> hwclock id ifconfig inotifyd insmod install ionice iorenice kill killall
> ln logname losetup ls lsattr lsmod lsof lsusb makedevs md5sum mkdir
> mkfifo mknod mkswap mktemp modinfo more mount mountpoint mv nbd-client
> nc netcat netstat nice nl nohup od partprobe paste patch pgrep pidof
> pivot_root pkill pmap printenv printf ps pwd pwdx readlink realpath
> renice rev rfkill rm rmdir rmmod route sed seq setsid sha1sum sleep
> sort split stat strings swapoff swapon sync sysctl tac tail tar taskset
> tee time timeout top touch tr traceroute traceroute6 true truncate
> tty ulimit umount uname uniq unix2dos uptime usleep vconfig vmstat
> wc which whoami xargs xxd yes
>
> (Woo, 149 commands!)

note that there's a second-level distinction between "in toybox" and
"gets a symbolic link".

(aiui from a few bug reports that came to us by mistake, cyanogen
_doesn't_ make that distinction --- they have a symlink for everything
that's compiled in, which causes trouble/confusion sometimes.)

looks like there are 128 symlinks.

> Pending commands included in the build:
>
> $ make list_pending | tr ' ' "\n" | \
> egrep "^($(./toybox | tr '\n ' '||'))\$" | xargs
> dd expr lsof more netstat route tar tr traceroute traceroute6

the dd symlink is still to toolbox: we still use the BSD dd.

i'm removing route in O at the request of the networking folks..

lsof is a port of the old toolbox lsof, and netstat is a superset of
the old toolbox netstat.

more i've worked a lot on (and you have two more patches from me to
apply) and is used a fair bit.

expr is apparently not in good shape judging by all the recent
patches, but no one ever complained to me so either it's not really
been used yet or it's only used for trivial stuff so far.

tar and tr work well enough for a few scripts that folks were trying
to use (though i think the cyanogen folks had trouble with tar and
long filenames).

the traceroutes don't have symbolic links and are really just
long-term TODO items for me; maybe one day we'll be able to replace
the iputils one.

> Non-pending commands _not_ included in the build:
>
> $ make list_working | tr ' ' "\n" | \
>   egrep -v "^($(./toybox | tr '\n ' '||'))\$" | xargs
>
> $ make list_working | tr ' ' "\n" | \
>   egrep -v "^($(./toybox | tr '\n ' '||'))\$" | xargs
> catv chvt count eject factor fstype fsync halt hello hexedit hostid
> iotop killall5 link login lspci mix mkpasswd nfsmount nproc nsenter
> oneit passwd poweroff readahead reboot reset shred skeleton
> skeleton_alias su switch_root test_many_options unlink unshare uudecode
> uuencode w who
>
> what have we skipped:
>
> You have your own init/login and toybox hasn't completed that circle yet
> anyway (halt/oneit/poweroff/reboot/reset/switch_root/killall5), you
> don't use standard uids (login/mkpasswd/passwd/su/w/who), no vga ttys
> (chvt), I should filter the examples directory out of "list_working"
> (hello/skeleton/skeleton_alias/test_many_options), nfsmount is just a
> todo item in mount (it needs a command that can prompt for password),
> and you have your own non-oss audio layer (mix).
>
> We can also say that count/factor aren't important, blkid covers fstype,
> sync covers fsync (albeit less efficiently, and I'm trying to remember
> if the "fsync /dev/sdb" to flush pending I/O to a specific block device
> is actually supported upstream or was an out of tree patch...), and you
> have the standard cat -v (catv).
>
> That leaves:
>
> eject hexedit hostid iotop link lspci nproc nsenter readahead shred
> unlink unshare uudecode uuencode
>
> The four that jump out at me are link, unlink, uuencode, and uudecode,
> which are in posix. I'm aware ln and rm cover the first two, and base64
> more or less covers the rest, and yet:
>
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/link.html
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/unlink.html
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/uuencode.html
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/uudecode.html

it turns out no-one born later than about 1985 has even heard of
uuencode/uudecode. it has to be explained as "a historic form of
base64". yagni. (bionic also doesn't have the posix a64l/l64a for
similar reasons, although the fact that they explicitly operate on a
static buffer is the biggest nail in their coffin.)

link/unlink was a surprise to even the old-timers, and seem to offer
nothing other than confusion beyond the well-known ln/rm. yagni.

> eject: is that actually used for anything but CD/DVD drives these days?
> I suppose it's really a special case of the ioctl...

yeah, and i no longer have any device other than my car that still has
an optical drive.

> hexedit: perhaps an acquired taste. (And I still need to work out how to
> make it insert and delete, and let you cursor over into the ASCII
> display part and type stuff in there...)

no one has asked for a hex editor yet. vi is what the people really want :-)

> hostid: maybe not relevant in context? Large Company's developer
> submitted it and it should REALLY be showing ipv6 addresses these days
> instead of ipv4. Its standardness is dubious...

bionic doesn't have gethostid(3), and it's not obvious it makes any
sense in 2016 anyway.

> iotop: If the android kernel doesn't have CONFIG_IRQ_TIME_ACCOUNTING
> enabled, this won't do you a whole lot of good. :)
>
> lspci: most android devices don't have pci busses, but android gets used
> in the embedded space sometimes... *shrug* (I worked on one that did in
> 2012, but the product was doomedish and the company recently got acquired.)

what we really don't have is /usr/share/misc/pci.ids.

> nproc: prints number of CPUs. Might be useful, and is in modern
> coreutils, but you can get the info out of /proc or /sys if you need to.

"what does number of CPUs mean?" is one of the most popular Android
questions. nproc only gives one of the possible answers, so probably
more harmful than useful.

> nsenter/unshare: Someday, opening the containers can of worms would be a
> really nice thing for Android to do. But we've talked about how there's
> Much Design Work to do to make that happen. Still, if the kernel
> plumbing's enabled, maybe for O?
>
> readahead: was requested. Seems like a redhat-ism to me.

for the most part i took the view of "if it's not in the base goobuntu
install, no one cares". (not foolproof, because what's useful on a
developer's workstation and what's useful on Android aren't
necessarily the same, but it's a reasonable place to start.)

> shred: was also requested. Probably useless on log based flash
.> filesystems, or even stuff like btrfs that can snapshot (let alone have
> multiple heads).
>
>> i'm on the opposite end of my
>> release cycle now, where now's a great time to make big changes in
>> AOSP master, for O next year. which is why my mind's back on ps and
>> top.
>>
>> dd too, though as i said last week i plan on waiting for that to leave
>> pending because (unlike expr) it's heavily used.
>
> Gotit: ps, top, and dd are android todo priorities. I also have mtools,
> genvfatfs, mke2fs, and gene2fs on the request list. Plus the stuff used
> out of pending.

netcat used to be on that list, but it stopped building at some point.
i can't remember whether i bothered to send you a patch. but long-term
it would be nice to remove external/netcat too. (toybox is missing -4,
-6, and -U.)

>>> Android's -t (which conflicts with ubuntu's -t ttylist) uses /proc/task
>>> to fetch stat and attr/current, blanks cmdline, and hardwires ppid to
>>> the thread parent (which it already seems to be in stat). So not a big
>>> change.
>>
>> the name of the option doesn't matter much. there's one hard-coded
>> callsite that i'll have to change when we switch ps and top anyway (a
>> bugreport includes both by default).
>>
>> i don't think there's any ppid hard-wiring. the top-level ps loop in
>> Android relies on the /proc weirdness you mention later: it iterates
>> through the visible /proc/\d+ entries, outputs the line for that pid,
>> then looks in its tasks subdirectory and calls the "show a line of
>> info about pid x" function for each tid in there.
>>
>>> Ubuntu's man page says says threads are ps -eLf. So -e is -A, -L is
>>> "show threads", and -f is full format and adds LWP and NLWP columns when
>>> used with -L. (And it DOESN'T seem to document that -L adds LWP
>>
>> here are the GNU options i alluded to:
>>
>> ~$ ps --help threads
>>
>> Usage:
>>  ps [options]
>>
>> Show threads:
>>   H                   as if they where processes
>
> where? (Typo for were? What?)
>
> Hmmm... according to the man page, that's process hierarchy (I.E.
> forest): group by parent/child and indent the cmdline.
>
> According to ls /proc/*/task pid 9922 is a chromium fiddly bit that has
> multiple threads (9923 to 9940... Huh, I wonder if I can use presence of
> child threads to work around the darn chromium command line truncation
> bug in Ubuntu 14.04? http://landley.net/notes-2016.html#29-03-2016 .)
>
> anyway, ps -H shows pid 9922 but nothing under it, and does NOT show pid
> 9923.
>
>>  -L                   possibly with LWP and NLWP columns
>
> ps -LH gives me a usage message.
>
>>  -m, m                after processes
>
> There are two instances of -m, both lower case? (So... -mm means
> something different than -m?)
>
> $ ps -m
>   PID TTY          TIME CMD
> 19869 pts/121  00:00:00 bash
>     - -        00:00:00 -
> 22293 pts/121  00:00:00 ps
>     - -        00:00:00 -
>
> Well that's nice...?
>
> $ ps -mH
>
> 290 lines of output, the header line of which is
> PID TTY      MAJFLT MINFLT   TRS   DRS  SIZE  SWAP   RSS  SHRD   LIB
> DT COMMAND
>
> And yes it's longer than 80 chars.
>
>>  -T                   possibly with SPID column
>
> ps -TH shows the two processes attached to this terminal (bash and ps
> -T), ps -TH does the same, and ps -ATH gives me a usage message...
>
> $ ps -Tm
>   PID  SPID TTY          TIME CMD
> 19869     - pts/121  00:00:00 bash
>     - 19869 -        00:00:00 -
> 22425     - pts/121  00:00:00 ps
>     - 22425 -        00:00:00 -
>
> This is just STUNNINGLY well designed, isn't it?
>
> Right, it looks like -T to show threads is the least _ILLOGICAL_ option,
> and then the
>
>>> Android's top is showing thread count as field #5, and "PCY" (scheduling
>>> policy) as field #8. And that's fetched with infrastructure that doesn't
>>> exist on Ubuntu: get_sched_policy() and get_sched_policy_name(). Um,
>>> does this need to be fixed up too? (My top has "pr" and "ni" in the
>>> default output. You can add SCHED with -o, but it's numeric, and the
>>> help text doesn't match the android fields (audio?).)
>>
>> yeah, don't worry about that stuff --- i'll send you patches for those
>> things later. (it's more cutils cruft. it's hairy enough and changes
>> from release to release that i think it would be a mistake for toybox
>> to try to do it itself.)
>
> Works for me. Lemme see what I can do with ps...
>
> Rob



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.