[Toybox] [PATCH] test.tar: fix tar tests on Android.

[Rob Landley]

On 7/9/19 4:42 PM, Gavin Howard wrote:
>> I cut and paste these things to a big text file I need to turn into a much, much
>> longer tests/sh.test. My _code_ may not be getting longer fast, but that file is...
>
> I see. I apologize for bothering you.

No, it's fine. You want to help. I'm glad you want to help. It's my failure as
an open source project maintainer that I can't easily figure out how to take
advantage of that desire to help.

This is a chronic problem. I created the pending directory to be less of a
bottleneck, wrote the cleanup.html page (and gave a talk on it) in hopes other
people could learn to do that stuff, and wrote the code.html and design.html
pages, used to point people at the test suite in hopes they could do that part...

Unfortunately, writing code I'm happy with is hard for _me_ to do. Teaching
other people to write code I'm happy with is harder than writing it. And the
problem isn't really writing code, it's _design_.

Random example: at some point I want to break up toys/*/ps.c so common
infrastructure is in lib/ps.c and then ps.c top.c and pkill.c can become
separate files. This doesn't _sound_ hard but there's a missing interface
abstraction: right now the function get_ps() uses the SLOT_* enum and typos[]
array to parse data into struct procpid, and then show_ps() gets it back out
again to display it using a struct ofields list containing the "ps -o
blah,blah.blah" field names (and associated :size and =rename information) to
select what and how to display, calling string_field() to get a string
represenation of each field in a given order that it pads and trims to line up
right.

I want to move all the above into a lib file, but it's an incestous hairball of
shared types and global data (well, TT.global data). It's concise and memory
efficient (although top is 3x slower than it should be and I should profile it
and figure out why), but can only really be understood holistically. You've got
to wrap your head around the whole thing to follow what any of it is doing, and
that's... not ideal? I'd like to change it to be easier to understand in smaller
chunks, but it's all tied together.

If I could explain how to do it, I'd already have done it. The hard part isn't
the code, it's figuring out what I want the result to look like and how the data
should flow across clear documentable interfaces.

When I try to deal with that sort of thing, even having _written_ it and thus
being more familiar with it as anyone on the planet (I know what I _meant_ to
accomplish, and were I fell short), I wind up trying things and then deleting
them repeatedly as I work out the inherent contradictions or unmet implicit
assumptions of a new approach. If I told somebody else to try, I'd almost
certainly wind up discarding what they wrote because I discard what _I_ write.
In my case, it's "having learned from it". (Given that reading code is harder
than writing code, and documentation is time consuming, for me to read someone
else's code, work out the design issues being implicitly addressed, and then
write up a clear english explanation for them of aining what the issue is
generally takes longer than having tried it myself.

I'm working on sh instead of ps cleanup right now because the payoff is higher.
The ps we have right now works and from a _technical_ standpoint mostly doesn't
need to be fixed. It's too long (almost 2000 lines and I try to keep toybox
commands under 1000 when possible) but that's 5 commands so 400 lines per
command. (And very roughtly: it's 1100 lines of shared infrastructure a lot of
which is large comment blocks, 300 lines of ps, 400 lines of top and iotop, and
100 lines of pgrep/pkill.)

The issue I want to address isn't that what we have doesn't work (although
adding bsd support remains a todo item), it's that the cognitive load of
maintenance is higher than I'd like. You have to keep too much in your head to
safely patch it. But all that stuff is currently serving a purpose and can't be
removed without breaking it, so it has to be... reordered? I tried documenting
it, but just helps "getting back up to speed" be faster, it doesn't reduce the
size of the hairball you have to keep in your head to reason about it. I keep
hoping if I sort things better they'll collapse together, but I kinda already
_did_ that to get where I am...

I held off on working on ps for as long as I did because it's an even _bigger_
hairball. Not so much shell syntax (although that's proving to be quite the
pain) but things like "cd isn't a normal command you shell out to, it affects
the current PID's state so has to execute in the current PID's context" and
"echo | X=Y" is a NOP because the X=Y runs in a new process context so it can
have a separate input, and thus can't add variables to the parent shell's
envionment, but "echo && X=Y" works fine. just working out the lifetime rules
for stuff... I didn't know "echo $((x=42); echo $x" assigned to the variable,
and now I do.

Oh, here's a fun one:

  $ meep() { echo hello; klarg() { echo helloier; } ; }
  $ klarg
  bash: klarg: command not found
  $ meep
  hello
  $ klarg
  helloier
  $

Yes, running the function defined another function.

This has been my month. I'm trying to finish the prompt continuation logic (I.E.
is the next line's prompt PS1="$ " or PS2="> ") and I keep asking "yeah but what
happens if the input looks like THIS" and having long tangents into the bash man
page and writing code like the above and going "but what does THAT mean...".

I'm sorry I'm not good enough to break off parts of this task other people can
do for me. It's a serious failing on my part. I barely know how to do this task.
More accurately: the parts I know how to do, I already did. I barely know how to
work _out_ how to do the rest. Again, if I knew how to work out how to do it,
I'd have done _that_ by now too, so I'm stuck with:

  https://www.youtube.com/watch?v=TYM4QKMg12o

I am aware this development methodology sucks. Sorry.

Rob

(Today I didn't write much C, instead I tried to put my cut-and-paste tests file
into tests/sh.test so I could run TEST_HOST=1 and confirm I've got them all down
right, and I went "this looks crazy, I just need sh -c "$TEST" for each one,
rewrote it to be simpler, proved that doesn't work (because eval sh -c "test
with quotes in it"), realized I already solved this last year and THAT'S why the
test code isn't using a _simple_ wrapper for the normal testing function but
instead doing that EVAL= thing... wasted 4 hours re-learning what I already knew
because I tried to throw out code I'd forgotten the reason for. I'm not even
that good at working with _myself_ when the gap's long enough...)