[Toybox] [PATCH] more.c: More stuff, down cursor key scrolls down. Also stuff about less

[Rob Landley]

On 3/21/24 06:52, Jarno Mäkipää wrote:
> On Thu, Mar 21, 2024 at 1:08 AM Rob Landley <rob at landley.net> wrote:
>> >> > There is also a testing problem. vi.c doesn't do TEST_HOST because it needs a -s option
>> >> > to pass in scripts to test with.
>> >>
>> >> Which is an issue I need to figure out how to address. What does a test that
>> >> only toybox passes actually prove? (That it hasn't changed since we last
>> >> looked at it?)
>> >
>> > There is vi -c which preforms a ex command which we could implement
> 
> I took -s from vim, so toybox vi could be tested comparing to vim,
> since vi itself does not have -s. And I was not interested in -c since
> ex was out of the scope of implementation at that time.

I'm not saying it's bad, I'm saying it's not sufficient. (The toysh tests have
_both_ "testcmd" and "shxpect" tests.)

Also, I'm not UPSET that someone's been making vi usable. Something is better
than nothing and I'm thankful. I'm just really annoyed at myself for not having
been able to get to it myself in a reasonable amount of time.

The vi that's there has users, and at some point I _do_ need to go through and
digest it all and wrap my head around it and take ownership of the thing, but i
haven't even managed to reboot my laptop for months to install the new devuan
version and put the 16 gig memory sticks back in, because I've been opening tabs
as fast as I've been closing them and trying to close them turns into "let me
fix this one thing real quick"... (It's like trying to pack bookshelves and
winding up reading books, which I also spent too much of last month doing.)

>> I leave vi to the people who are maintaining that vi. I got out of way for that
>> command.
>>
> 
> Well im not sure who is "maintaining" vi.c at this point, I wrote base
> implementation years ago, Elliott extended it with few commands,
> because he had some use case for it. But mostly development has been
> dormant for few years with few segfault bugfix here and there. Its not
> very pleasant experience to maintain it, since everything lead to huge
> bikeshedding, since there is no particular standard to follow,
> everyone want different things.

Indeed. I taught an "intro to unix" course at austin community college many
moons ago which had like 20 vi keys on the syllabus (half of which were new to
me, and most of which I've forgotten again). And every time I install a fresh
debian I have to go through my checklist including:

  sudo ln -sf vimrc /etc/vim/vimrc.tiny && echo export EDITOR=vi >> ~/.profile

Because going into "insert" mode and having the cursor keys crap capital letters
all over your text is stupid (this vimrc.tiny mode STILL RUNS THE SAME BIG
EXECUTABLE), and as with dash and upstart and mir and unity I suspect Mark
Shuttleworth was behind it:

  https://mstdn.jp/@landley/112119853431329313

And no I'm not typing "vim" any more than gsed, gawk, or gmake...

> Also from what I understand reading
> your postings, you have never been very satisfied on it. And that is
> understandable.

The thing is, I'm not a vi expert any more than I was a sed expert before I
wrote my own sed (twice). At some point, I have to learn enough awk to write an
awk that can replace gawk in every package build in LFS and BLFS (and hopefully
someday AOSP), and I'm not looking forward to that. I know I _need_ to, but I'm
currently overwhelmed with half-finished stuff and am trying to dig out.

I'm somewhat familiar with the subset busybox chose for its vi, although that
was always missing several things I use, so good point of reference but not a
standard. And I need to read the posix standard for vi. And then I was going to
implement some low-hanging fruit have people tell me what they missed...

>> >> I have been planning one all along, yes. The crunch_str() stuff I did was a
>> >> first pass at general line handling stuff that could be used by less and by
>> >> shell line editing and by vi and so on, but people wrote a vi that does not and
>> >> never will share code with the rest of those so that's off the table
>> >> permanently.
> 
> vi.c uses crunch_str from lib for utf8 handling, there was just few
> corner cases it needs to use vi only crunch_nstr, since it cant spit
> up text until nul all the time. vi.c tried to use some other
> functionality from lib also, but some of it got removed from lib and
> some functionality have probably been added way after vi.c was written
> in 2018-2020.

I tend to do passes over the whole tree from time to time cleaning stuff up and
modernizing it. (I re-review commands I hadn't seen in a while.) I admit my
tree's just been a bit dirty to get full coverage on any of those for quite a
while, but I still browse.

> I think less and vi could share some infrastructure, either by taking
> stuff from vi.c to lib, or rewriting it into lib and taking it out
> from vi. more could be way simpler, since it should not have backward
> movement.

Oh sure, that was always my plan. When I was designing crunch_escape() a lot of
it was based on experimenting to see how vim displayed various things.

Unfortunately, dependency chains turn into the MASH episode where hawkeye gets
tomato juice to colonel potter, doing a series of favors for everyone in camp
along the way. It's faster to just half-ass some stuff for now and replace it
with the RIGHT thing later.

>> Meanwhile, the people who took vi away and ran with it seem to think testing the
>> interactive bits are irrelevant, and that "vi" is just "ex" with a GUI. I have
>> not attempted to argue with them.
> 
> I was not against of doing visual testing, I just didint know how to
> implement it.

I have a partial design but still haven't done it yet. I'm starting to suspect
that I need to implement a debugging mode into the toybox "screen" command and
then have the test depend on toybox screen the way tar --xform depends on toybox
sed (and thus so do the tests for it), or something like that.

Unless I can figure out something VERY clever to do with /dev/ptmx from the
command line? (Kinda hard to call ptsname(), grantpt() and unlockpt() from bash,
let alone the man 2 ioctl_tty constellation...)

It's a hard problem. Even the ANSI escapes to ask how big the current screen is
and where the cursor is require actually updating a terminal and writing output
to it, which is not what other tests do (they just print PASS/SKIP unless they
FAIL verbosely), and assumes the output IS to a tty (it's not), and then it
can't control the screen size and the test may care what it is...

> Also I was never interested on ex that much. Some other
> people were more interested on ex and added some ex commands.

Neither was I.

The thing about vi is it's the only editor in posix.

I went from C64->amiga->dos where I used Turbo C which had "wordstar bindings",
which unix implemented in an editor called "joe" and that's what I used for my
first few years on sun workstations. Except when Rutgers' computing lab migrated
from SunOS to Slowaris joe stopped working right (Half the time ctrl-S would get
eaten as XOFF and freeze the program instead of acting as the wordstar key, and
fairly often the thing would just segfault and exit), which forced me to learn
to at least minimally cope with something else. I'd acutally used a microemacs
on the amiga, but the emacs on slowaris was gnu/emacs which was just HORRIFIC
(it had, as part of the standard install, a mode that would feed "zippy the
pinhead" quotes into Eliza, BOTH OF WHICH WERE IN THE BASE INSTALL), and I
refused to learn lithp in order to use a text editor, so vi it was.

(Nano claimed to be simple, but the cheat sheet at the bottom didn't say how to
cut and paste. It does say "cut" is ctrl-k... which deletes the current line. it
doesn't say what key "undo" is bound to. That's about where I usually give up on
it.)

My own vague plan for text editing was to have shared code that could implement
vi, joe, and nano, as well was the rendering parts top/watch/less/screen needed.
(Display/wrap some subset of a dynamic text buffer onto the screen, seems like a
common problem.) Alas when I described this plan on the list someone went off
and waterfall methoded up a giant all-or-nothing hairball, with a design that
never asked half the questions I'd already pondered, from someone who was very
unhappy I didn't swallow it whole, and who complained at some length. (He'd done
work, and thus I owed him acceptance of his work into toybox.) Kinda put me off
revisiting that space for years...

Here's an excellent explanation of the "waterfall method" by the way:

https://web.archive.org/web/20140122141413/http://www.shirky.com/weblog/2013/11/healthcare-gov-and-the-gulf-between-planning-and-reality/

See also the "Second System Effect" described in Fred Brooks' The Mythical
Man-month (https://wiki.c2.com/?SecondSystemEffect) because you don't really
know what you're doing until you've done it once, at which point the temptation
to start over and do THE ULTIMATE VERSION that will solve every possible use
case you've ever thought of along the way (I.E. "infrastructure in search of a
user" for every possible use case) is very strong. (The Multics project before
Unix was one such system, after
https://en.wikipedia.org/wiki/Compatible_Time-Sharing_System . Unix was a "third
system" where the devs had learned to 80/20 everything, eat their own dog food,
and learn by doing.)

I try to learn by doing, and take a trail of breadcrumbs approach leading
towards what I need, getting stuff that seems generically useful working when I
have a use for it (avoiding "infrastructure in search of a user" that inevitably
bit-rots, or has the "first use is not re-use" problem where stuff awkwardly
tries to incorporate it because it's there). And there's inevitable backtracking
as I learn stuff that forces me to unravel what I've done and take a different
approach. (Right now the toysh variable resolution assumes that curly brackets
are matched up by parse_word(), which involves resolving line continuations, but
HERE documents require deferring line continuation processing, and I need to
cycle back to that and resolve that contradiction, possibly by adding a lot more
checking to my variable resolution code or possibly by figuring out how to
redundantly call parse_word() from multiple contexts. This happens ALL THE TIME...)

I've written a usable "top" and "watch" (both have problems), and there was line
editing in hexedit I need to redo and fix, and I need to do interactive command
line editing and history for toysh which is ACTUALLY very similar to less and vi
because it wordwraps lines with unicode and such, and it would be nice if there
was a function that did this generically. (And then it would be nice if
appending characters to the end of such a line didn't require redrawing the
entire line, because typing into vi, typing at a toysh interactive command line,
and progressive output in less all need very similar behavior...)

Sigh. This stuff is hard for me. If it wasn't I'd be done already. :P

Rob