[Toybox] CPU% collation in top -H...

[Rob Landley]

On 2/26/19 11:57 AM, enh wrote:
> On Tue, Feb 26, 2019 at 1:08 AM Rob Landley <rob at landley.net> wrote:
>>> i know, you list these every time this comes up :-)
>>
>> Hopefully not the same ones? I like to think I'm making _some_ progress...
> 
> well, the `top -H` bug is new :-) (i have looked at that a couple of
> times, but don't have a fix yet. on the bright side, i'm a lot more
> confident about everything else we show in top right now --- seems to
> match traditional top on my desktop at least.)

Hmmm... For non-threaded programs, diff /proc/$PID/stat and
/proc/$PID/task/$PID/stat are identical, but for threaded programs they differ.

$ diff -u /proc/12744/stat /proc/12744/task/12744/stat
--- /proc/12744/stat	2019-03-01 12:26:03.290671451 -0600
+++ /proc/12744/task/12744/stat	2019-02-28 11:14:14.125850040 -0600
@@ -1 +1 @@
-12744 (chromium-browse) S 23217 1568 1568 0 -1 1077960768 1921759 0 6 0 1479649
58352 0 0 20 0 37 0 368618601 3328770048 121098 18446744073709551615
140663802400768 140663941864172 140728669358448 140728669356000 140663797983166
0 0 4098 1073826341 0 0 0 17 2 0 0 15 0 0 140663943963800 140663949468376
140663973699584 140728669362290 140728669362347 140728669362347 140728669364173 0
+12744 (chromium-browse) S 23217 1568 1568 0 -1 1077960768 1613496 0 6 0 65932
5089 0 0 20 0 37 0 368618601 3328770048 121098 18446744073709551615
140663802400768 140663941864172 140728669358448 140728669356000 140663797983166
0 0 4098 1073826341 1 0 0 17 2 0 0 15 0 0 140663943963800 140663949468376
140663973699584 140728669362290 140728669362347 140728669362347 140728669364173 0

Hmmm... Unless I'm counting columns wrong the 1479649 in the first and the 65932
in the second are the "utime jiffies" field that %CPU ultimately derives from,
and that first number is way way higher than the second, so yes. Reading the
data from the two different locations gives different info. The kernel's doing
the collating here.

Now the parent process is _intentionally_ reading from the proc entry rather
than the task entry because for everything else it wants the process-global
view... but in this case we _don't_ want it? Hmmm...

Should -H make us read the thread stat for _everything_? What would that change?
Fire up the 4 spinner subthreads test and...

$ echo 16213 | (read i; diff -u <(tr ' ' '\n' < /proc/$i/stat | cat -n ) <(tr '
' '\n' < /proc/$i/task/$i/stat | cat -n))
--- /dev/fd/63	2019-03-02 15:20:07.447570138 -0600
+++ /dev/fd/62	2019-03-02 15:20:07.447570138 -0600
@@ -7,11 +7,11 @@
      7	34835
      8	16213
      9	1077960704
-    10	290
+    10	286
     11	0
     12	0
     13	0
-    14	1783
+    14	0
     15	0
     16	0
     17	0
@@ -32,7 +32,7 @@
     32	0
     33	0
     34	0
-    35	0
+    35	1
     36	0
     37	0
     38	17

Ok, entries 10, 14, and 35 accoding to tale 1-4 in
linux/Documentation/filesystems/proc.txt are... 10 = min_flt (number of minor
faults, sure I'll buy that), 14 = utime (user mode jiffies, case in point), and
35 = placeholder, used to be wchan address. (What? I... what? Why does it... what?)

Right. In any case, yes it looks like always reading from the task structure in
-H mode should give us what we asked for, I guess? (Unless we want the option to
_have_ a collated mode? Except that wouldn't be -H...)

>>> but even if you can't solve the whole problem, anything you can do to reduce
>>> your https://en.wikipedia.org/wiki/Bus_factor helps...
>>
>> You mean like these?
>>
>> http://landley.net/toybox/code.html
>> http://landley.net/toybox/design.html
>> http://landley.net/toybox/cleanup.html
>> https://github.com/landley/toybox/commit/e2191095c3c6
>>
>> (I've been told Google's management considers Android's command line "good
>> enough", and were uninterested in funding me to spend time working through my
>> todo list. Not even to focus on the test suite or on documentation. So they
>> can't be _that_ worried...)
> 
> _they_ don't care, but you and i do :-)

I gave JCI 2 weeks' notice a week ago. I'm aware it's not a smart move my most
metrics but it's been 6 years since the 2013 talk (and 8 since the 2011
relicensing) and I need to get this sucker DONE...

(Oddly enough, JCI can use toybox ar and toybox tar if I can get 'em both
finished/promoted this weekend. So I'm trying to do that, but have spent so far
today closing tabs so the precarious pile of todo items doesn't collapse on me.)

> i think there's also the cookie-licking aspect of this... we both
> know, for example, that dd needs a lot of work but it doesn't get it
> because of the potential for collision. i assume the TAGGED_ARRAY
> change is a step in the right direction, but didn't want to do
> anything i didn't have to do in the same patch. (i'm assuming, for
> example, that you'd rather have TT.ibs rather than TT.in.bs, but
> because your in-progress state is invisible...)

Ask me again in a couple weeks. :)

> this is one thing i like about gerrit: it's a good place to post
> in-progress [but not ready to commit] work. that and leaving TODOs in
> the code rather than out of band seems more conducive to having other
> folks pitch in. (github also has a "help wanted" label for bugs that
> the owners aren't actually working on.)

In theory blogging is where I blathered about in progress stuff like that. In
practice I've fallen way behind on posting it, although I've been catching up a
bit recently. Except that's full of entries like the next one I'm blocked on:

<a name="05-10-2018"><hr><h2><a href="#05-10-2018">October 5, 2018</a></h2>
<span id=programming>
<p>Made another attempt to get qemu to run a Hello World kernel. In theory
qemu's -kernel loader can handle ELF files (that's what vmlinux is),
and you can write to the serial port with a two line loop, so if your
entry point is a loop writing characters to the serial port...</p>

<p>In reality, I haven't gotten it to work yet. I've tried:</p>

<blockquote><p>
echo "void _start(void) { *(volatile unsigned char *)0x81093025 = 42; for(;;);
}" > bork.c<br />
powerpc-linux-musl-cc bork.c --static -nostartfiles -nostdlib
-Wl,-Ttext=0xc0000000<br >
qemu-system-ppc -M g3beige -m 256 -nographic -kernel a.out
</p></blockquote>

<p>I should see an asterisk, but I'm not seeing an asterisk.</p>

<p>I'm using ppc because that's the mkroot target I've got using a vmlinux
kernel already. According to "readelf -a vmlinux" that kernel's
entry point is


... and it stops there, abruptly, as I got distracted. And the next day, just
started a new entry. I think I tweeted the data? If not I can reproduce t, but I
have to go do work to finish the entry. (Or trim off the last paragraph I wrote
and call it good...)

>> Just the toybox _license_ can eat weeks, between SPDX approval, _defending_ the
>> SPDX approval from OSI, arguing about it on github, a whole second round getting
>> OSI to agree with SPDX, then this month it's back to Github and trying _not_ to
>> reopen OSI or SPDX cans of worms...
>>
>> https://github.com/github/choosealicense.com/issues/464#issuecomment-438718643
>> https://github.com/github/choosealicense.com/pull/643
>> https://github.com/spdx/license-list-XML/pull/768
>>
>> I don't have the heart to face wikipedia[citation needed]. I convinced OSI to
>> stop misnaming the license (buried the hatched with Bruce Perens, even got Kirk
>> McKusick to publicly endorse the name), and wikipedia[citation needed] is STILL
>> deadnaming the license at
>> https://en.wikipedia.org/wiki/BSD_licenses#0-clause_license_(%22Zero_Clause_BSD%22
>> . Sigh. At least the part of
>> https://en.wikipedia.org/wiki/Public-domain-equivalent_license where they were
>> doing the same has been turned into a very awkward "OSI says X. SPDX also says X
>> and said it first so I dunno why OSI is even mentioned unless you look at the
>> history of this wiki page..."

I should cycle back to that...

>> And gmail/dreamhost's unsubscribed everybody again. (Wrestles with web gui,
>> clicks through individual letter pages to uncheck the B checkboxes but not the U
>> checkboxes and click "submit" to reload each page before clicking on the next
>> letter...) And dreamhost wants its renewal check. And I haven't checked my own
>> gmail spam filter web page to see what it's false positived on in weeks... yup,
>> dozens from qemu-devel and buildroot and musl and linux-kernel... It is now
>> sending dreamhost "bounce action notification" messages to spam. That's almost
>> meta. Aha, I didn't see
>> http://lists.landley.net/pipermail/toybox-landley.net/2019-February/010129.html
>> (and missed it in the web archive), sorry about that...)
> 
> yeah, i didn't see that patch either.

Applied now. :)

>> *shrug* The usual. (And my blog's up on https://landley.net/notes.html through
>> the end of August! Maybe I'll go edit and post more of that... Editing another
>> week's worth of entries took an hour and 3 minutes. Adding links and finishing a
>> sentence that stopped abruptly turned into adding several more paragraphs on the
>> topic, with even more links, as usual. But at least they're up now...)
> 
> anyway, getting back to grep --line-buffered... what's the next step?
> merge this and do the x* flush-removal later? upload another patch
> that's like this but removes the x* flushes rather than switches away
> from x*? something else?

I haven't had time to get back to it, but have a tab open.

> (also, if you're not happy with the dd patch you'll need to tell me
> what you don't like :-) )

I haven't had time to get back to it, but have a tab open.

> note that we're also entering peak TPS season again, so i'll be even
> more evenings-and-weekends than usual.

As in... TPS reports from Office Space?

>> Rob

Rob