[Toybox] Kernighan's awk repo

[Rob Landley]

On 07/21/2016 05:01 PM, Andy Chu wrote:
> What's annoying to me is that you spent more time writing about things
> than trying them.  You've spilled so many words about it, whereas just
> trying it would have answered all the questions.  I wrote up a lot of
> detailed instructions.

For me, writing email is a break from programming. I wrote the text
you're complaining about while I compiling clang+llvm so I had the
sanitizer to try, which was also during breaks of staring at sed and
scripts/sed.test to make sed -f -e -f maintain sequence (because
somebody finally had a real-world use case), and chipping away at grep
--color and expanding tests/grep.test (I've commented about some of the
design issues this raises here), and glancing at
https://github.com/hifi/toybox/commits/vi to see what that guy's doing,
and izabera sent a bugfix for "file -" that clashes with the
xopen_nostd() rewrite I've got pending in my tree and I need to figure
out WHAT is being fixed exactly and whether the rewrite already handles
it, and I set up a test environment for that arm stat segfault but
haven't managed to reproduce it yet (QEMU may not be catching all
unaligned accesses that the android emulator does), and of course doing
j-core stuff for $DAYJOB and so on.

Writing text can be done in 3 minute increments. Writing code requires
more concentration and larger contiguous time chunks. I write
documentation as a break from coding, and long emails fall in that
bucket. :)

(The fact this _is_ done in small chunks means I tend to trail off and
forget to finish it. I just hit send on two unfinished emails from
earlier today. I.E. I wrote a LOT more than the email you complained
about. And that's not counting the all other text I wrote today. *shrug*
I type fast.)

> And it is also annoying to have trivial bugs linger in the tree for
> months, and to have bugs introduced on top of my patches.  IMO expr
> was done long ago -- I know you want to rewrite it because you don't
> understand the algorithm and want to combine it with $(()).  I haven't
> been pressing the issue for the last few months, but it seems obvious
> that you could just ship it and combine it with $(()) later (note that
> awk also has an expression evaluator, and awk and shell also have
> boolean expressions, so this thing never ends).
> 
> I won't really be surprised if in a year expr is still in the same
> state it's in now -- shipping out of pending on Android with bugs,
> even though I already fixed them.

I'm currently trying to go through the list of commands android is using
out of pending, clean them up, and promote them. But first I'm going
through Elliott's suggestions from his toolbox status report a couple
weeks back. And that's interrupted by bug reports of real issues on
things not in pending (ala the arm stat segfault I still haven't
reproduced).

I'd like to get everything android is using out of pending before O
freezes, but expr is not one of the ones I"m currently looking at and if
I tackle too many topics at once I swap thrash. (And if I get
interrupted on something I'm doing too long I have to backtrack a lot to
reestablish context.)

(Part of the reason opening the llvm can of worms now makes sense is
I've been meaning to switch aboriginal linux over to it, and between
http://lists.landley.net/pipermail/aboriginal-landley.net/2016-May/002567.html
http://lists.landley.net/pipermail/aboriginal-landley.net/2016-July/002578.html,
the guy poking at adding https://j-core.org support to llvm, and the
fact AOSP is migrating over to an llvm build anyway, if I get momentum
here it's useful in multiple contexts. So that was already near the top
of the todo list, just not AT the top.)

>> I downloaded it, looked at the README to see it was still unchanged, ran
>> ./run.sh and it produced now output, "ran ./run.sh help" and got the
>> bash help output, and put it in ~/toybox/pending so that when I get
>> around to implementing awk I can get back to it.
> 
> The instructions are on the front page!!!   Scroll down here:
>
> https://github.com/andychu/bwk

Ah, there's actual stuff about the test suite below the bit about
building a specific awk implementation with your address sanitizer.
Hadn't made it that far down.

It's still not a can of worms I'm looking to open today. I'm trying to
close tabs.

> And in README.md (I didn't clobber the original README on purpose).

So I spent too much time writing email (a break from coding), and not
enough time trying to make an awk test suite work (which comes out of
the "coding" bucket) when I haven't started on an awk implementation yet
and wouldn't be able to _use_ this until I did.

(This awk implementation is under a BSD license with advertising clause,
and believe I asked its author a while ago if it was relicenseable to
0BSD and he said no because Lucent owns copyrights on it. My history
with IP enforcement has made me very _very_ careful about licensing
issues. Being able to _almost_ take this code is like CDDL being almost
compatible with GPL version du jour.)

>> (The problem is I don't wanna add asan_sed and asan_test_sed and so on
>> ad ifinitum. This is a shift key, not its own button. And really, I
> 
> My patches already solved that problem.  That's why there is a test.sh script.

Your top level script (there are no other top level scripts) calls make.
None of my scripts ever call make. Make is a purely optional front-end
to those scripts. Except for the kconfig stuff (which is slated for a
rewrite), you should be able to build, install, and run the test suite
on a system that hasn't even got make installed.

> Like I said in the original messages, I wrote the patches once in make
> and again in shell, and they were much better in shell.  (That is, the
> test support is in shell, and the build support is in make, rather
> than everything in make with .PHONY targets.)
> 
> Specifically, the originals had asan_test_sed, etc. targets with
> generated .mk code, and then I changed it to be an argument to a shell
> script.

The makefile I looked at still had targets for each, and I started to
reverse engineer "what are you trying to accomplish" from what you'd
implemented, so I started looking up what these various sanitizers were
and how you'd use them, which is where I needed an installed version of
llvm to start playing with, and the version in ubuntu's repo was too old
to run https://github.com/nickg/nvc so I'd already installed a newer
llvm from source but although I'd enabled linux-rt I hadn't copied clang
into the right place so had to run the build again (on a slow machine)...

> I guess you want to retread all the same steps so you understand
> everything in the repo.  I get that viewpoint, but I'm sure you also
> understand that it's a big bottleneck.

I'm aware it's a big bottleneck, yes. I regret this enormously, and it's
why I added the "pending" directory so people had the option to route
around me.

But I want to be able to _support_ all this code. You see asan as
important, I see code coverage and testing root stuff under a stable
qemu environment as more important. There are large pending changes to
this infrastructure (and semi-constant small ones that don't always get
checked in promptly because seeing "+# cp -a rootfile userfile # should
have no error" in git diff for tests/cp.test lets me know to add the
test, whereas if I check that comment in I may not see it again any time
soon. Local diffs act as yet another form of todo item for me.)

> Andy

Rob