[Toybox] ps and top (and Android)

[Rob Landley]

On 04/21/2016 08:59 PM, enh wrote:
> On Thu, Apr 21, 2016 at 1:06 AM, Rob Landley <rob at landley.net> wrote:
>> On 04/20/2016 08:47 PM, enh wrote:
>>> re: https://github.com/landley/toybox/commit/529d5db29a32db565d2ee0aeac133e47ec045e12,
>>> note that we're not switched over because neither ps nor top have
>>> thread support. everything on Android is threaded except init, so
>>> people would notice :-)
>>
>> I was tracking whether or not it was in pending, but you're right. My bad.
> 
> well, it's your roadmap --- you get to choose what it means :-) i just
> wanted to point out that we might have been interpreting it
> differently.

Indeed, but you maintaining a toybox roadmap provides me with
information I wouldn't otherwise have. (I can scribble my plans down in
a textfile, and they're subject to change anyway.)

(I still occasionally check https://wiki.tizen.org/wiki/Toybox but other
than typo fixes in January it hasn't been touched in two years. I
haven't figured out how to follow cyanogenmod's toybox discussions yet...)

>> Yesterday I found out I'm going to Japan on Sunday, so the release I was
>> trying to prepare at the end of the month got bumped up. I have a 9 hour
>> layover in San Francisco on Sunday, and I'm hoping to upload it then.
>>
>> I have like 6 bugs I need to fix between now and then (the find
>> segfault, the new bzcat segfault, expr memleak, etc). Plus I'm in the
>> process of cleaning up another half dozen commands out of pending, and I
>> should get at least a couple of them closed out and promoted this release.
> 
> barring anything huge enough to warrant a cherrypick, what you see in
> N DP2 (http://developer.android.com/preview/index.html) should be what
> ships in N as far as toybox goes.

Going by tags that's commit 9fcaca8434ec for android-n-preview-2 (modulo
the build system churn). Check that out in the android repo, make
oldconfig, switch off selinux support, build, and:

acpi base64 basename blkid blockdev bunzip2 bzcat cal cat chattr chgrp
chmod chown chroot cksum clear cmp comm cp cpio cut date dd df dirname
dmesg dos2unix du echo egrep env expand expr fallocate false fgrep
find flock free freeramdisk fsfreeze grep groups head help hostname
hwclock id ifconfig inotifyd insmod install ionice iorenice kill killall
ln logname losetup ls lsattr lsmod lsof lsusb makedevs md5sum mkdir
mkfifo mknod mkswap mktemp modinfo more mount mountpoint mv nbd-client
nc netcat netstat nice nl nohup od partprobe paste patch pgrep pidof
pivot_root pkill pmap printenv printf ps pwd pwdx readlink realpath
renice rev rfkill rm rmdir rmmod route sed seq setsid sha1sum sleep
sort split stat strings swapoff swapon sync sysctl tac tail tar taskset
tee time timeout top touch tr traceroute traceroute6 true truncate
tty ulimit umount uname uniq unix2dos uptime usleep vconfig vmstat
wc which whoami xargs xxd yes

(Woo, 149 commands!)

Pending commands included in the build:

$ make list_pending | tr ' ' "\n" | \
egrep "^($(./toybox | tr '\n ' '||'))\$" | xargs
dd expr lsof more netstat route tar tr traceroute traceroute6

Non-pending commands _not_ included in the build:

$ make list_working | tr ' ' "\n" | \
  egrep -v "^($(./toybox | tr '\n ' '||'))\$" | xargs

$ make list_working | tr ' ' "\n" | \
  egrep -v "^($(./toybox | tr '\n ' '||'))\$" | xargs
catv chvt count eject factor fstype fsync halt hello hexedit hostid
iotop killall5 link login lspci mix mkpasswd nfsmount nproc nsenter
oneit passwd poweroff readahead reboot reset shred skeleton
skeleton_alias su switch_root test_many_options unlink unshare uudecode
uuencode w who

what have we skipped:

You have your own init/login and toybox hasn't completed that circle yet
anyway (halt/oneit/poweroff/reboot/reset/switch_root/killall5), you
don't use standard uids (login/mkpasswd/passwd/su/w/who), no vga ttys
(chvt), I should filter the examples directory out of "list_working"
(hello/skeleton/skeleton_alias/test_many_options), nfsmount is just a
todo item in mount (it needs a command that can prompt for password),
and you have your own non-oss audio layer (mix).

We can also say that count/factor aren't important, blkid covers fstype,
sync covers fsync (albeit less efficiently, and I'm trying to remember
if the "fsync /dev/sdb" to flush pending I/O to a specific block device
is actually supported upstream or was an out of tree patch...), and you
have the standard cat -v (catv).

That leaves:

eject hexedit hostid iotop link lspci nproc nsenter readahead shred
unlink unshare uudecode uuencode

The four that jump out at me are link, unlink, uuencode, and uudecode,
which are in posix. I'm aware ln and rm cover the first two, and base64
more or less covers the rest, and yet:

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/link.html
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/unlink.html
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/uuencode.html
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/uudecode.html

eject: is that actually used for anything but CD/DVD drives these days?
I suppose it's really a special case of the ioctl...

hexedit: perhaps an acquired taste. (And I still need to work out how to
make it insert and delete, and let you cursor over into the ASCII
display part and type stuff in there...)

hostid: maybe not relevant in context? Large Company's developer
submitted it and it should REALLY be showing ipv6 addresses these days
instead of ipv4. Its standardness is dubious...

iotop: If the android kernel doesn't have CONFIG_IRQ_TIME_ACCOUNTING
enabled, this won't do you a whole lot of good. :)

lspci: most android devices don't have pci busses, but android gets used
in the embedded space sometimes... *shrug* (I worked on one that did in
2012, but the product was doomedish and the company recently got acquired.)

nproc: prints number of CPUs. Might be useful, and is in modern
coreutils, but you can get the info out of /proc or /sys if you need to.

nsenter/unshare: Someday, opening the containers can of worms would be a
really nice thing for Android to do. But we've talked about how there's
Much Design Work to do to make that happen. Still, if the kernel
plumbing's enabled, maybe for O?

readahead: was requested. Seems like a redhat-ism to me.

shred: was also requested. Probably useless on log based flash
filesystems, or even stuff like btrfs that can snapshot (let alone have
multiple heads).

> i'm on the opposite end of my
> release cycle now, where now's a great time to make big changes in
> AOSP master, for O next year. which is why my mind's back on ps and
> top.
> 
> dd too, though as i said last week i plan on waiting for that to leave
> pending because (unlike expr) it's heavily used.

Gotit: ps, top, and dd are android todo priorities. I also have mtools,
genvfatfs, mke2fs, and gene2fs on the request list. Plus the stuff used
out of pending.

>> Android's -t (which conflicts with ubuntu's -t ttylist) uses /proc/task
>> to fetch stat and attr/current, blanks cmdline, and hardwires ppid to
>> the thread parent (which it already seems to be in stat). So not a big
>> change.
> 
> the name of the option doesn't matter much. there's one hard-coded
> callsite that i'll have to change when we switch ps and top anyway (a
> bugreport includes both by default).
> 
> i don't think there's any ppid hard-wiring. the top-level ps loop in
> Android relies on the /proc weirdness you mention later: it iterates
> through the visible /proc/\d+ entries, outputs the line for that pid,
> then looks in its tasks subdirectory and calls the "show a line of
> info about pid x" function for each tid in there.
> 
>> Ubuntu's man page says says threads are ps -eLf. So -e is -A, -L is
>> "show threads", and -f is full format and adds LWP and NLWP columns when
>> used with -L. (And it DOESN'T seem to document that -L adds LWP
> 
> here are the GNU options i alluded to:
> 
> ~$ ps --help threads
> 
> Usage:
>  ps [options]
> 
> Show threads:
>   H                   as if they where processes

where? (Typo for were? What?)

Hmmm... according to the man page, that's process hierarchy (I.E.
forest): group by parent/child and indent the cmdline.

According to ls /proc/*/task pid 9922 is a chromium fiddly bit that has
multiple threads (9923 to 9940... Huh, I wonder if I can use presence of
child threads to work around the darn chromium command line truncation
bug in Ubuntu 14.04? http://landley.net/notes-2016.html#29-03-2016 .)

anyway, ps -H shows pid 9922 but nothing under it, and does NOT show pid
9923.

>  -L                   possibly with LWP and NLWP columns

ps -LH gives me a usage message.

>  -m, m                after processes

There are two instances of -m, both lower case? (So... -mm means
something different than -m?)

$ ps -m
  PID TTY          TIME CMD
19869 pts/121  00:00:00 bash
    - -        00:00:00 -
22293 pts/121  00:00:00 ps
    - -        00:00:00 -

Well that's nice...?

$ ps -mH

290 lines of output, the header line of which is
PID TTY      MAJFLT MINFLT   TRS   DRS  SIZE  SWAP   RSS  SHRD   LIB
DT COMMAND

And yes it's longer than 80 chars.

>  -T                   possibly with SPID column

ps -TH shows the two processes attached to this terminal (bash and ps
-T), ps -TH does the same, and ps -ATH gives me a usage message...

$ ps -Tm
  PID  SPID TTY          TIME CMD
19869     - pts/121  00:00:00 bash
    - 19869 -        00:00:00 -
22425     - pts/121  00:00:00 ps
    - 22425 -        00:00:00 -

This is just STUNNINGLY well designed, isn't it?

Right, it looks like -T to show threads is the least _ILLOGICAL_ option,
and then the

>> Android's top is showing thread count as field #5, and "PCY" (scheduling
>> policy) as field #8. And that's fetched with infrastructure that doesn't
>> exist on Ubuntu: get_sched_policy() and get_sched_policy_name(). Um,
>> does this need to be fixed up too? (My top has "pr" and "ni" in the
>> default output. You can add SCHED with -o, but it's numeric, and the
>> help text doesn't match the android fields (audio?).)
> 
> yeah, don't worry about that stuff --- i'll send you patches for those
> things later. (it's more cutils cruft. it's hairy enough and changes
> from release to release that i think it would be a mistake for toybox
> to try to do it itself.)

Works for me. Lemme see what I can do with ps...

Rob