[Toybox] CPU% collation in top -H...

[enh]

On Sat, Mar 2, 2019 at 2:06 PM Rob Landley <rob at landley.net> wrote:
>
> On 2/26/19 11:57 AM, enh wrote:
> > On Tue, Feb 26, 2019 at 1:08 AM Rob Landley <rob at landley.net> wrote:
> >>> i know, you list these every time this comes up :-)
> >>
> >> Hopefully not the same ones? I like to think I'm making _some_ progress...
> >
> > well, the `top -H` bug is new :-) (i have looked at that a couple of
> > times, but don't have a fix yet. on the bright side, i'm a lot more
> > confident about everything else we show in top right now --- seems to
> > match traditional top on my desktop at least.)
>
> Hmmm... For non-threaded programs, diff /proc/$PID/stat and
> /proc/$PID/task/$PID/stat are identical, but for threaded programs they differ.
>
> $ diff -u /proc/12744/stat /proc/12744/task/12744/stat
> --- /proc/12744/stat    2019-03-01 12:26:03.290671451 -0600
> +++ /proc/12744/task/12744/stat 2019-02-28 11:14:14.125850040 -0600
> @@ -1 +1 @@
> -12744 (chromium-browse) S 23217 1568 1568 0 -1 1077960768 1921759 0 6 0 1479649
> 58352 0 0 20 0 37 0 368618601 3328770048 121098 18446744073709551615
> 140663802400768 140663941864172 140728669358448 140728669356000 140663797983166
> 0 0 4098 1073826341 0 0 0 17 2 0 0 15 0 0 140663943963800 140663949468376
> 140663973699584 140728669362290 140728669362347 140728669362347 140728669364173 0
> +12744 (chromium-browse) S 23217 1568 1568 0 -1 1077960768 1613496 0 6 0 65932
> 5089 0 0 20 0 37 0 368618601 3328770048 121098 18446744073709551615
> 140663802400768 140663941864172 140728669358448 140728669356000 140663797983166
> 0 0 4098 1073826341 1 0 0 17 2 0 0 15 0 0 140663943963800 140663949468376
> 140663973699584 140728669362290 140728669362347 140728669362347 140728669364173 0
>
> Hmmm... Unless I'm counting columns wrong the 1479649 in the first and the 65932
> in the second are the "utime jiffies" field that %CPU ultimately derives from,
> and that first number is way way higher than the second, so yes. Reading the
> data from the two different locations gives different info. The kernel's doing
> the collating here.
>
> Now the parent process is _intentionally_ reading from the proc entry rather
> than the task entry because for everything else it wants the process-global
> view... but in this case we _don't_ want it? Hmmm...
>
> Should -H make us read the thread stat for _everything_? What would that change?
> Fire up the 4 spinner subthreads test and...
>
> $ echo 16213 | (read i; diff -u <(tr ' ' '\n' < /proc/$i/stat | cat -n ) <(tr '
> ' '\n' < /proc/$i/task/$i/stat | cat -n))
> --- /dev/fd/63  2019-03-02 15:20:07.447570138 -0600
> +++ /dev/fd/62  2019-03-02 15:20:07.447570138 -0600
> @@ -7,11 +7,11 @@
>       7 34835
>       8 16213
>       9 1077960704
> -    10 290
> +    10 286
>      11 0
>      12 0
>      13 0
> -    14 1783
> +    14 0
>      15 0
>      16 0
>      17 0
> @@ -32,7 +32,7 @@
>      32 0
>      33 0
>      34 0
> -    35 0
> +    35 1
>      36 0
>      37 0
>      38 17
>
> Ok, entries 10, 14, and 35 accoding to tale 1-4 in
> linux/Documentation/filesystems/proc.txt are... 10 = min_flt (number of minor
> faults, sure I'll buy that), 14 = utime (user mode jiffies, case in point), and
> 35 = placeholder, used to be wchan address. (What? I... what? Why does it... what?)
>
> Right. In any case, yes it looks like always reading from the task structure in
> -H mode should give us what we asked for, I guess? (Unless we want the option to
> _have_ a collated mode? Except that wouldn't be -H...)

tested both on Android and on my desktop (where it seems to give
results consistent with Debian's top). lgtm, thanks!

> >>> but even if you can't solve the whole problem, anything you can do to reduce
> >>> your https://en.wikipedia.org/wiki/Bus_factor helps...
> >>
> >> You mean like these?
> >>
> >> http://landley.net/toybox/code.html
> >> http://landley.net/toybox/design.html
> >> http://landley.net/toybox/cleanup.html
> >> https://github.com/landley/toybox/commit/e2191095c3c6
> >>
> >> (I've been told Google's management considers Android's command line "good
> >> enough", and were uninterested in funding me to spend time working through my
> >> todo list. Not even to focus on the test suite or on documentation. So they
> >> can't be _that_ worried...)
> >
> > _they_ don't care, but you and i do :-)
>
> I gave JCI 2 weeks' notice a week ago. I'm aware it's not a smart move my most
> metrics but it's been 6 years since the 2013 talk (and 8 since the 2011
> relicensing) and I need to get this sucker DONE...

you're a braver man than me!

> (Oddly enough, JCI can use toybox ar and toybox tar if I can get 'em both
> finished/promoted this weekend. So I'm trying to do that, but have spent so far
> today closing tabs so the precarious pile of todo items doesn't collapse on me.)
>
> > i think there's also the cookie-licking aspect of this... we both
> > know, for example, that dd needs a lot of work but it doesn't get it
> > because of the potential for collision. i assume the TAGGED_ARRAY
> > change is a step in the right direction, but didn't want to do
> > anything i didn't have to do in the same patch. (i'm assuming, for
> > example, that you'd rather have TT.ibs rather than TT.in.bs, but
> > because your in-progress state is invisible...)
>
> Ask me again in a couple weeks. :)
>
> > this is one thing i like about gerrit: it's a good place to post
> > in-progress [but not ready to commit] work. that and leaving TODOs in
> > the code rather than out of band seems more conducive to having other
> > folks pitch in. (github also has a "help wanted" label for bugs that
> > the owners aren't actually working on.)
>
> In theory blogging is where I blathered about in progress stuff like that. In
> practice I've fallen way behind on posting it, although I've been catching up a
> bit recently. Except that's full of entries like the next one I'm blocked on:
>
> <a name="05-10-2018"><hr><h2><a href="#05-10-2018">October 5, 2018</a></h2>
> <span id=programming>
> <p>Made another attempt to get qemu to run a Hello World kernel. In theory
> qemu's -kernel loader can handle ELF files (that's what vmlinux is),
> and you can write to the serial port with a two line loop, so if your
> entry point is a loop writing characters to the serial port...</p>
>
> <p>In reality, I haven't gotten it to work yet. I've tried:</p>
>
> <blockquote><p>
> echo "void _start(void) { *(volatile unsigned char *)0x81093025 = 42; for(;;);
> }" > bork.c<br />
> powerpc-linux-musl-cc bork.c --static -nostartfiles -nostdlib
> -Wl,-Ttext=0xc0000000<br >
> qemu-system-ppc -M g3beige -m 256 -nographic -kernel a.out
> </p></blockquote>
>
> <p>I should see an asterisk, but I'm not seeing an asterisk.</p>
>
> <p>I'm using ppc because that's the mkroot target I've got using a vmlinux
> kernel already. According to "readelf -a vmlinux" that kernel's
> entry point is
>
>
> ... and it stops there, abruptly, as I got distracted. And the next day, just
> started a new entry. I think I tweeted the data? If not I can reproduce t, but I
> have to go do work to finish the entry. (Or trim off the last paragraph I wrote
> and call it good...)
>
> >> Just the toybox _license_ can eat weeks, between SPDX approval, _defending_ the
> >> SPDX approval from OSI, arguing about it on github, a whole second round getting
> >> OSI to agree with SPDX, then this month it's back to Github and trying _not_ to
> >> reopen OSI or SPDX cans of worms...
> >>
> >> https://github.com/github/choosealicense.com/issues/464#issuecomment-438718643
> >> https://github.com/github/choosealicense.com/pull/643
> >> https://github.com/spdx/license-list-XML/pull/768
> >>
> >> I don't have the heart to face wikipedia[citation needed]. I convinced OSI to
> >> stop misnaming the license (buried the hatched with Bruce Perens, even got Kirk
> >> McKusick to publicly endorse the name), and wikipedia[citation needed] is STILL
> >> deadnaming the license at
> >> https://en.wikipedia.org/wiki/BSD_licenses#0-clause_license_(%22Zero_Clause_BSD%22
> >> . Sigh. At least the part of
> >> https://en.wikipedia.org/wiki/Public-domain-equivalent_license where they were
> >> doing the same has been turned into a very awkward "OSI says X. SPDX also says X
> >> and said it first so I dunno why OSI is even mentioned unless you look at the
> >> history of this wiki page..."
>
> I should cycle back to that...
>
> >> And gmail/dreamhost's unsubscribed everybody again. (Wrestles with web gui,
> >> clicks through individual letter pages to uncheck the B checkboxes but not the U
> >> checkboxes and click "submit" to reload each page before clicking on the next
> >> letter...) And dreamhost wants its renewal check. And I haven't checked my own
> >> gmail spam filter web page to see what it's false positived on in weeks... yup,
> >> dozens from qemu-devel and buildroot and musl and linux-kernel... It is now
> >> sending dreamhost "bounce action notification" messages to spam. That's almost
> >> meta. Aha, I didn't see
> >> http://lists.landley.net/pipermail/toybox-landley.net/2019-February/010129.html
> >> (and missed it in the web archive), sorry about that...)
> >
> > yeah, i didn't see that patch either.
>
> Applied now. :)
>
> >> *shrug* The usual. (And my blog's up on https://landley.net/notes.html through
> >> the end of August! Maybe I'll go edit and post more of that... Editing another
> >> week's worth of entries took an hour and 3 minutes. Adding links and finishing a
> >> sentence that stopped abruptly turned into adding several more paragraphs on the
> >> topic, with even more links, as usual. But at least they're up now...)
> >
> > anyway, getting back to grep --line-buffered... what's the next step?
> > merge this and do the x* flush-removal later? upload another patch
> > that's like this but removes the x* flushes rather than switches away
> > from x*? something else?
>
> I haven't had time to get back to it, but have a tab open.
>
> > (also, if you're not happy with the dd patch you'll need to tell me
> > what you don't like :-) )
>
> I haven't had time to get back to it, but have a tab open.
>
> > note that we're also entering peak TPS season again, so i'll be even
> > more evenings-and-weekends than usual.
>
> As in... TPS reports from Office Space?

yeah, i use it generically for anything that causes no work to get
done because of paperwork.

> >> Rob
>
> Rob