[Toybox] ps and top (and Android)

[enh]

On Thu, Apr 21, 2016 at 1:06 AM, Rob Landley <rob at landley.net> wrote:
> On 04/20/2016 08:47 PM, enh wrote:
>> re: https://github.com/landley/toybox/commit/529d5db29a32db565d2ee0aeac133e47ec045e12,
>> note that we're not switched over because neither ps nor top have
>> thread support. everything on Android is threaded except init, so
>> people would notice :-)
>
> I was tracking whether or not it was in pending, but you're right. My bad.

well, it's your roadmap --- you get to choose what it means :-) i just
wanted to point out that we might have been interpreting it
differently.

> Yesterday I found out I'm going to Japan on Sunday, so the release I was
> trying to prepare at the end of the month got bumped up. I have a 9 hour
> layover in San Francisco on Sunday, and I'm hoping to upload it then.
>
> I have like 6 bugs I need to fix between now and then (the find
> segfault, the new bzcat segfault, expr memleak, etc). Plus I'm in the
> process of cleaning up another half dozen commands out of pending, and I
> should get at least a couple of them closed out and promoted this release.

barring anything huge enough to warrant a cherrypick, what you see in
N DP2 (http://developer.android.com/preview/index.html) should be what
ships in N as far as toybox goes. i'm on the opposite end of my
release cycle now, where now's a great time to make big changes in
AOSP master, for O next year. which is why my mind's back on ps and
top.

dd too, though as i said last week i plan on waiting for that to leave
pending because (unlike expr) it's heavily used.

> And then there's the "build LFS with it" test, which I'm now trying to
> migrate to LFS 7.8. Busy week. :)
>
>> there are a bunch of little things too that i can send patches for,
>
> Sure, throw 'em on the pile. :)
>
>> but the thread support seems like a more architectural thing so i was
>> waiting to see what you were going to do. (not least because GNU ps
>> has three or four different modes for showing threads, their 'H' being
>> closest to the Android -t.)
>
> That's pretty much why I haven't done it yet: figuring out the right
> thing to do is always the hard part, coding it's generally much easier.
>
> I used threads quite extensively on OS/2, then saw pthreads and went
> "ew" and never really got back into 'em. Given that unix can trivially
> map shared memory and synchronize via pipes, I've just done that when I
> needed to. (Java kinda forced threads down people's throats because they
> refused to implement poll/select or nonblocking I/O for ages...)
>
> So I'm not hugely familiar with what the output should look like for
> threads. Let's see... what has multiple entries under /proc/*/task...
> chromium of course. What the heck is "whoopsie"? (The ubuntu crash
> reporter, which does not have a man page.)
>
> Alright, it looks like all the threads show up in /proc but don't report
> assocation to a tty, so get filtered out. (/proc/816/task/819 has a
> subset of the same data in /proc/819, seems to be a strict subset? Do
> ALL the threads exist at the top level?
>
>   $ for i in /proc/*/task/*; do [ ! -d /proc/$(basename $i) ] && \
>     echo $i; done
>
> Yes, they all show up in /proc. So in theory all I have to do is adjust
> the filtering parameters and possibly the display fields.
>
> Android's -t (which conflicts with ubuntu's -t ttylist) uses /proc/task
> to fetch stat and attr/current, blanks cmdline, and hardwires ppid to
> the thread parent (which it already seems to be in stat). So not a big
> change.

the name of the option doesn't matter much. there's one hard-coded
callsite that i'll have to change when we switch ps and top anyway (a
bugreport includes both by default).

i don't think there's any ppid hard-wiring. the top-level ps loop in
Android relies on the /proc weirdness you mention later: it iterates
through the visible /proc/\d+ entries, outputs the line for that pid,
then looks in its tasks subdirectory and calls the "show a line of
info about pid x" function for each tid in there.

> Ubuntu's man page says says threads are ps -eLf. So -e is -A, -L is
> "show threads", and -f is full format and adds LWP and NLWP columns when
> used with -L. (And it DOESN'T seem to document that -L adds LWP

here are the GNU options i alluded to:

~$ ps --help threads

Usage:
 ps [options]

Show threads:
  H                   as if they where processes
 -L                   possibly with LWP and NLWP columns
 -m, m                after processes
 -T                   possibly with SPID column

For more details see ps(1).

> LWP an alias for tid and NLWP is an alias for thcount (thread count). So
> crazy gnu names for things that already have common names.
>
> Just confirming: the posix ps page does not use the word "thread", and
> there isn't a posix top page. Of course not, that would be far too useful...
>
> So. If /proc/$PPID/tasks/$PID exists, then this is a thread: don't
> display it. Except my ps -A isn't currently displaying 819, but when I
> ls /proc/819 it's there...?
>
> Oh you bastard. Whoever wrote proc is cheating, the synthetic filesystem
> isn't showing the PID in the directory listing, but the thread is there
> at the top level if you try to access it. So my test was invalid because
> I expected SANITY from this filesystem. Grrr. (Dear Linux: your wires
> are showing.)
>
> Right, ok, I need to iterate through each pid's tasks subdirectory to
> enumerate threads. I can do that. The top level view has several extra
> files that task view doesn't, which is /proc being cute again: if I look
> at /proc/816/tasks/819 I get a different set of files than /proc/819. It
> looks like task view has a "children" that top level doesn't, and the
> top level view adds autogroup, coredump_filter, map_files, mountstats,
> task, and timers. None of which the current infrastructure is reading,
> so yes I can just look at the tasks subdir with -L. Then have -L add TID
> after PID, and have -f add TCNT.
>
> Right, that's probably what ps should do. Now, what does TOP do? Rummage
> rummage... H toggles thread view. It turns "Tasks:" into "Threads:" in
> the header and makes the S column width 1 instead of width 2? Huh...
>
> Android's top is showing thread count as field #5, and "PCY" (scheduling
> policy) as field #8. And that's fetched with infrastructure that doesn't
> exist on Ubuntu: get_sched_policy() and get_sched_policy_name(). Um,
> does this need to be fixed up too? (My top has "pr" and "ni" in the
> default output. You can add SCHED with -o, but it's numeric, and the
> help text doesn't match the android fields (audio?).)

yeah, don't worry about that stuff --- i'll send you patches for those
things later. (it's more cutils cruft. it's hairy enough and changes
from release to release that i think it would be a mistake for toybox
to try to do it itself.)

> Next question: do I have to make pgrep and pkill thread aware? Let's
> see... pkill doesn't care, pgrep has -w to show tid instead of pid.
> Sounds straightforward enough...
>
> Rob



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.