[Toybox] What should I do with my boxes generic editor code?

[Rob Landley]

On Tue, May 26, 2015 at 12:08 AM, David Seikel <onefang at gmail.com> wrote:
> This has been most frustrating.
>
> tl,dr - Getting boxes into toybox has entirely failed for four years,
> which has blocked the progress of both.  Let's do something about that.

I suspect the best thing to do is just release it as an independent package.
I've been consistently overwhelmed with stuff, merging as fast as I can.

I worked through the weekend trying to get ready for LinuxCon Japan, and
yesterday I found out my house flooded again while I'm away (via twitter,
https://twitter.com/fadeaccompli/status/602955075292528640
but it's always something.) Today I find out that someone "needs my reply!"
the same way https://twitter.com/roytam1/status/603142482725965825

I'm in the process of migrating machines from my netbook to the linux vm
on the shiny mac laptop work got me, which involves closing lots of tabs
and making a list of them so I can reopen them on the other machine.
Opening my toybox desktop I see I was in the middle of modifying stat.c
(add %T support), cp.c (make --preserve work), test.c (there's a public
domain implementation from uclinux I'm looking at), more.c (I promised
enh), sed.c (new feature + support in ranges, somebody asked), swapon.c
(support -deLsU), expr.c (priority groups now that the posix thing's
resolved), netcat (commit f492fccc9ceb was wrong), and ps.c
(still about 2/3 done, got distracted). Plus I never did add an undo
buffer to hexedit.c. And of course the smack patch, and I think I've
got a couple patches from enh pending, and of course the giant patch
stack from Ashwini Sharma that I _still_ have sitting in a folder to
look at.

But sure, it's a crisis now, let's do this.

I just sat down to look at the boxes code again. started reading from the top,
two thousand five hundred lines of boxes.c. I remember at one point I
asked if there was a way to chip off a hundred or so self-contained lines
to add as a first step, I believe I suggested the escape key sequence
parsing (ala the stuff I eventually implemented myself for hexedit) might
be self-containable? I do not remember a patch to toybox lib/ forthcoming
in response, but maybe I missed it.

Let's see, single config option for "boxes" saying it can be used in a bunch
of different modes but they don't have their own config or help entries.
Hand #defining FLAG_ macros even though the infrastructure's
done this for us for at least a year. Big comment, "current code is a
work in progress, design may change"... midnight commander, design thinks
like curses does, "I plan on splitting these things up a bit so they can be
used separately" but hasn't happened, apparently not going to because crisis...
"justification of design" the need for which is always a good sign, binary tree
of boxes, root box is "a global" (variable? no way to tell from here, I'm a
hundred lines in and haven't found code yet), content has context (I'm sure
I won't confuse that)... Ok, giving up and scrolling down a bit to try to
find code. "I assume there won't be a terribly large number of boxes" which
is clearly why there's need for a binary tree for them.

"The edit line is basically a movable readline"... not really, no. For command
history I've been pondering how to handle "I scrolled up and got a long
multiline command history thing and then scrolled past that to get a shorter
one, so even if I scroll back down it's now several lines _up_ from the bottom
of the screen because the transient long line moved everything". Lines that
wrap can vary in size, they're no more _one_ vertical line than horizontal
characters are one character with utf-8. Speaking of which, searching this
file for "mbr" or "wcwidth" didn't find any hits? I added the basic
fontmetrics dance to hexedit's filename display in commit 5ea14bd1c246
a couple weeks ago, not because it's important for a hex editor but because
I'm using a simple command to work out the infrastructure I want to
genericize. Your version has a lot of very complicated behavior but doesn't
address fundamental issues I'm interested in getting right, ones which aren't
necessarily easier to retrofit on _top_ of all that complexity.

Right, LOOKING FOR CODE, skip giant blob of text, paged down through a
bunch of TODO lines, through "Rob's Lament" and "Rob's Contradiction"
(quoting me)... and I'm finding static global arrays (ouch),
typedefs (ouch), structure definitions...

Ok, switching to the bottom of the file (last page also has a TODO),
cursoring up to find the start of main()... and none of this is legible on
an 80 column screen (your comments are about half a screen over).
Lots of lines starting at the left edge (when I do that it means debug code,
but this is checked into source control). At line 2400 there's two screens
full of what looks like a man page cut and pasted into the middle of the code?
(Why not just reference the man page?)

Ok, backwards search for "main(" and... it's got a multiplexer for "mode",
with emacs, joe, less, mcedit, more, nano, and vi. But you can't submit one
at a time, because this code cannot be broken up into smaller pieces. Right.

Ok, let's look at vi, which is the editor required by posix. Case insensitive
search for "vi " from the top of the file... help text, help text,
comment, comment... ok, the eleventh hit is still in yet another comment
(right after yet another TODO) but it's right in front of "simpleEditCommands"
which sounds like fun, and which seems to have built-in help text? "Back space
last character." Ok, let's see what displays this help text. Forward
searching from simpleEditCommands it's in simpleLess (is there a non-simple
less?) and in simpleMore and in simpleMcedit... ok, "simple" here seems to
be an assertion rather than actually informative. I was looking for vi so
presumably that's simple too... yup, simpleViCommands has more help text
("Switch to insert mode."), and simpleViCommands is used in simpleVi which
is used to set struct context *context in main()... so "struct context *" is
used as a struct but at the top there was a typedef for box? Right.
Ok, where is "context" used after this assignment:

  rootBox = addBox("root", context, toys.optargs[0], 0, 0, W, H - 1);
  currentBox = rootBox;

That's a lot of arguments, one of which is a global. Ok, so where is rootBox
used:

  commandLine = addView("command", rootBox->view->content->context,
".boxes.history", 0, H, W, 1);

Those string constants made me raise an eyebrow. The three levels of
dereferincing in an argument called from main() are also a "huh?" moment.
And the next search for rootBox wraps around to the top of the file where I
find it's a global that's not in GLOBALS()..

Ok, how about currentBox:

  calcBoxes(currentBox);
  drawBoxes(currentBox);
  // Do the first cursor update.
  updateLine(currentBox->view);

  // Run the main loop.
  handle_keys((long) currentBox->view, handleEvent);

I'd think this was the "display loop" except that's not in a loop, that's
just all inline in main called once. (Why is that (long) there? I have no
idea. Seems a minor thing in context...)

My previous question "what, if anything, is displaying this help text" has
vanished into "some sort of callback through structures that point to other
structures, maybe?" When addView() returned commandLine (?) it took an
instance of context as its argument so maybe it stashed that in the
structure it returned, but it traversed from a global to _get_ it so maybe
it's reconstructed that way by later code so... I guess I have to read
through everything before I can understand anything? (That's the
conclusion I came to the last few times I looked at this...)

What are the _other_ files here? README.md: "Please don't actually include
this in toybox." BOXES.txt: a series of disconnected notes-to-self apparently
not meant to be read by anybody other than its author.

handlekeys.c is 445 lines of code. my lib/interstingtimes.c and
toys/*/hexedit.c combined are 133+286 = 419 lines, I.E. smaller than that.
I'm not saying mine's better (you handle sigwinch, I implement a hex editor),
I'm just saying that in terms of breaking off manageable chunks of code from
boxes to add to toybox I've never had a clue where to _start_.

> I've been trying for years (since 2011) to get my generic editor code
> (boxes) into toybox, so far with zero movement on that front.  Now Rob
> has added two editors he wrote himself,

I got tired of waiting for the project that still describes itself in git as
"a work in progress, proof of concept, playground, packaged up as
one big toy, to be self contained until the mess is cleaned up." to send
me small intelligible patches I can review in the context of what I already
know, rather than expecting me to go on some sort of mining
expedition for useful code in thousands of lines full of global variables,
typedefs, and TODOs.

(One advantage of GLOBALS() is that all global variables are prefixed TT
so you can tell at a glance it's a global. That's not part of C, but this is
nominally toybox code that doesn't even do that much.)

> and from what I can tell wants
> to just write it all himself.

I want to understand the code I add to the project. This has always been
a failing of mine, but I don't see it changing.

Right now I'm trying to understand smack and selinux. (I'm still not sure
what to do about the "mkdir world writeable then add a label to it after
the fact" race condition, for example.)

> Not to mention a few other things that my
> generic code covers, where written by others, and have been included
> already.  Boxes covers emacs, joe (Wordstar), less, mcedit (the editor
> I actually use), more, nano, and vi.  It also covers the ed style
> modes of those things that include one.  It could cover a few other
> things as well, for instance I wrote an example shell that just
> demonstrates how it could be used in toysh.  It could share code with
> sed, and now hexedit.

Failure to be big enough and complicated enough to attract my attention was
not, in fact, the problem.

> It's time for me to make a decision about what to do with boxes.  Four
> years with no movement is ridiculous.  People keep wanting me to finish
> it, "Rob wont put it into toybox" just isn't an acceptable response for
> me to give after four years.  I'm putting this on the public mailing
> list so that those who are actually interested can provide some
> feedback to help inform my decision.  Some actual useful feedback from
> Rob that progresses this instead of just sitting on it forever would be
> great to.

I've just spent an hour looking at it. I still have no idea how it works.

> So far the only actual objection you have given to me Rob is that it's
> too big a blob for you to grok.  So I cut out a bit of it, to see if you
> could at least handle a small part of it, even though it was too small
> to be actually useful.  You ignored that to.  That's a problem, a
> generic editor is just too big, even if it's cut down to "barely usable
> editor".  Sure it can be cut down into bite size bits, but none of
> those bits will be useful at all.  So this has been the big blocker,
> Rob wants bite sized bits, this doesn't break down into bite sized
> bits, and Rob aint putting it in, not even to pending, unless it's
> bite sized bits.

Sorry.

> Me completing it by fleshing out the missing editing
> functions wont make it any more bite sized, that has to wait until
> it's in toybox.  Once it is in, fleshing it out CAN be done in bite
> sized bits.

Once I install emacs you can write small lisp snippets on top of it?

> Every now and then Rob you say "I want to write this", and I usually
> reply "I've written this for you already", sometimes you respond with
> "we'll talk about it later", then later never comes.

I don't know what to say.

> Mostly you just ignore it.

When my previous posts on the subject get catalogued as "Rob's Lament"
and "Rob's Contradiction", I may become reticient. A character flaw.

> Despite the fact that it covers a lot of things that you
> keep saying you want to add.

There are existing implementations of all this stuff out there. I used to
object when people would nail an entire external project onto the
side of busybox, e2fsprogs comes to mind (now e2fsprogs_old), or the
three thousand lines of fdisk.c that _still_ have a function called
is_ide_cdrom_or_tape() and support for sun and sgi disk label formats, but
at least they put an #if 0 around:

    if (dos_changed)
        printf(
        "\nWARNING: If you have created or modified any DOS 6.x\n"
        "partitions, please see the fdisk manual page for additional\n"
        "information\n");

So I guess that's something. But hey, they have a working fdisk without
having to install an external package! (Or write a new one, apparently.)
Toybox does not, so they're better than toybox. QED.

> So, what are my choices?
>
> 0) Try to convince Rob to just dump it into pending.  Then we can all
> work on it to make it great.

Has this worked for the rest of pending? (Also, I haven't got a pending
for lib or tests, which is a bit of a structural problem I haven't thought
of a good way to fix yet.)

> 1) Keep waiting another four years in the forlorn hope that Rob
> actually does something with it.

It's not a question of the amount of time. I do not know how to proceed
with the existing codebase, and trying to guilt me into accepting code
in its present form isn't a lot of fun from this end either.

> 2) Fork toybox and just add it myself.

Go for it.

> 3) Write it as a drop in module for toybox.

I thought that was the original plan?

> 4) Rewrite it to be entirely separate from toybox.

That would work too.

> 5) Just forget about it, and wait four or more years for Rob to write
> his versions of a bunch of things.
>
> 6) Insert choices I haven't thought of here.

I don't know what to tell you.

> Another thing to bring up is that I had discussed on this mailing list
> before my desire to implement Midnight Commander (mc) as part of toybox.
> Something that Rob agreed to in principle at least.

Yeah, it's one of the things I was thinking might be a logical next step
after hexedit. The problem is I haven't used xtgold since dos, and never
really used mc (a clone of xtree, which I believe was originally for Sun
or SGI or some such).

It's also not a priority since it's not required by any standards or build
enviornments, and many other remaining todo items are. Adding hexedit let
me work out display infrastructure, which basically took a couple of
weekends. There isn't similar collateral upside with mc, it would just
be a second user of that infrastructure and I could do command history
editing or "less" instead, both of which have aspects of those wordwrap
variable line height issues.

(Leaning towards less since more is on the near-term hit list anyway and
they should probably be combined. The shell needs a "back up and
rethink" kind of rewrite, and that's something I need to close tabs to clear
mental space for.)

Rob

 1432672317.0