[Toybox] [PATCH] microcom: move from e[x]it to [q]uit.

[Rob Landley]

On 11/6/23 16:32, enh wrote:
> On Sun, Nov 5, 2023 at 8:16 AM Rob Landley <rob at landley.net> wrote:
>>
>> On 11/3/23 09:44, enh wrote:
>> > On Fri, Nov 3, 2023 at 7:22 AM Rob Landley <rob at landley.net> wrote:
>> >>
>> >> On 11/2/23 16:56, enh via Toybox wrote:
>> >> > This frees up 'x' for xmodem transfers later.
>> >>
>> >> Q to quit makes sense, adding xmodem... do you have an xmodem protocol extension
>> >> in mind that doesn't pad the last packet up to 128 bytes?
>> >
>> > that is annoying, yes, but since my only purpose is transferring
>> > executables, i don't think it matters? (and even in the case where it
>> > does, toybox already has a truncate(1).)
>>
>> A transfer protocol known to modify files is not ideal.
> 
> no, i meant you can truncate(1) the resulting destination file based
> on what ls/stat told you was the exact size of the source file. (if it
> matters, which seems most likely to be a problem for text files
> anyway, and why not just send those as text in 2023?)

Busybox commit dc9495df03f4 made it send CTRL-Z padding, and trim trailing
CTRL-Z, which I didn't know busybox was doing (it wasn't back in 1.2.2) but was
a thing I remembered from my pre-linux days: Ctrl-Z was the CP/M end of file
marker (https://en.wikipedia.org/wiki/End-of-file) and CP/M guys went "if you're
going to pad with anything, might as well be these" and it got picked up. (No
in-band signalling is going to be RIGHT, it's just a question of least-bad...)

Part of my reaction here is probably that trudged through these primordial
swamps back in the day, and did not ENJOY middle school, so design decisions
resurfacing from that era have more "ick" factor associated with them than is
strictly warranted. Commodore 64 to Amiga 1000 to 8086 DOS to a 386 running
Desqview to OS/2 to Linux, but that's just the through line. Several friends had
Commodore 128s, there was a TRS-80 in the 7th grade library and a TI-99-4a
running Logo, and the community college had some old school macs and Rutgers was
Sun workstations as far as the eye could see (which migrated from SunOS to
Slowaris while I was there) and the only course I ever got more than 100% in was
the "assembly language" course I signed up for that turned out not to be i386
assembly but HP mainframe assembly (with binary coded decimal and the Zero and
Add Packed instruction, at the end of the semester they unplugged the mainframe
and THREW IT OUT. Never saw it, we submitted jobs through the sun workstations.)
And when I was at IBM there was a bunch of PowerPC hardware running AIX and I
worked (professionally!) on JavaOS for 6 months, and about three years of "who
cares where it runs, it's java, except here's the long list of bug workarounds
for the 5 environments we've tested it in no two of which behave quite the same
and ALL are buggy and funky and I'm not even counting the native widget toolkits
here", and a migration from Irix to Slowaris-86 at boxx, and as I crept into the
embedded world multiple "glue functions() to u-boot or similar" RTOS variants
(some with a cooperative multitasking sort of thread-ish environment, some not)...

"This is left over from 1983. It does it wrong but we're stuck with it..." sigh.
If we must...

>> >> What xmodem command
>> >> are you going to run at the other end to consume the data and send the replies,
>> >
>> > i was going to write one. the xmodem spec sounds like it's only going
>> > to be tens of lines.
>>
>> There's a spec? It's like 3 status bytes and 128 byte packets... (I was on
>> bulletin boards when I was 12. There's a nonzero chance I implemented it back on
>> the commodore 64, and even then it was inferior to punter.)
> 
> https://pauillac.inria.fr/~doligez/zmodem/ymodem.txt

I read that back when it came out. :)

Which is why I was suggesting implementing ymodem. It has the OPTION to not be
broken, and can presumably clean-ish-ly fall back to the broken stuff if
necessary. (There are a lot of broken things calling themselves ymodem, but
that's true of xmodem as well...)

By the way, busybox 1.2.2's rx.c suggested:

  http://www.textfiles.com/apple/xmodem
  http://www.phys.washington.edu/~belonis/xmodem/docxmodem.txt
  http://www.phys.washington.edu/~belonis/xmodem/docymodem.txt
  http://www.phys.washington.edu/~belonis/xmodem/modmprot.col

The first is still there and the others can be pulled out of wayback.

The zipfile "standard" was some text file (zipnote.txt?) in Phil Katz' source
distributions, which got picked up by IETF long after the fact and reformatted
into one of their standards. The 1980s standards are mostly "some guy said",
written up. Even more recently, the FDPIC standards documents look like
https://gcc.gnu.org/legacy-ml/gcc/2008-02/msg00619.html and if there's better
out there I haven't found it...

>> >> and how will the terminal program know to run it?
>> >
>> > i think for unix implementations of xmodem it's more common for the
>> > user to do that part? the xmodem spec does talk about having sx send
>> > "rx\r" and rx ignoring crap like that at the start, but that seems
>> > gross enough to wait until someone actually _needs_ that.
>>
>> Busybox never bothered to implement the sx command, so I can't check there, but
>> I would expect them to do that because "I am at a command prompt, send file"
>> would just work. :)
>>
>> Except for the part where xmodem doesn't know what the filename is. I'm guessing
>> it would write a.out by default?
> 
> lrzsz takes a [required] filename:
> https://manpages.debian.org/testing/lrzsz/rx.1.en.html

If we're sending rx\n anyway we can supply a filename, modulo control characters...

>> How about ymodem? Ymodem was an attempt to fix the problems with xmodem: 1k
>> packet size since we were no longer on 300 baud modems, block 0 includes the
>> filename and size so it doesn't pad the last block, and 16-bit CRC instead of 8
>> bit checksum. If you're going to do a file transfer protocol from the 1980s BBS
>> world, ymodem is... much less bad.
> 
> see the doc above for this --- it seems like (even back in 1988)
> people were confused about whether those were "really" ymodem or just
> common xmodem extensions. afaik the idea is that you _try_ 1024 byte
> blocks and 16-bit CRCs, but fall back if they don't work.

If ymodem gracefully fails back to xmodem (hard to test without a send side)
then if were gonna implement sx/rx we might as well have it actually do ymodem.

Of course "gracefully fall back" assumes that the newer packets you send that
get NAKed don't contain within their payloads start-of-packet sequences for the
older stuff, but "did not understand packet 0" is probably important signaling,
except "NAK because corruption vs NAK because protocol"...

>> By the way, there are UI questions we're just kinda glossing over here. If you
>> want "send these files to the far end" to be integrated... that may be a call to
>> rsync.
> 
> no, not at all. don't get me wrong, i'm sure there are people who'd
> rather have rsync, but we're in "everything is broken [or was always
> too stupid to cope]" land here...
> 
> i swear i didn't go looking for this, but over the weekend i found
> myself watching a youtube video about
>
> --- even in 2023 (well, okay, 2021), xmodem is still the lowest common
> denominator of serial transfers. that's what i claim rx/sx are useful
> for, and that's also why i'd claim that rz/sz _aren't_ useful. because
> anyone doing anything fancy is probably on scp/rsync, not zmodem :-)

The qwerty keyboard of file transfer protocols...

>> We haven't got a midnight commander variant that lets you select files
>> and get a file list. The traditional unix way of doing this would call the
>> various tools in a shell script, with the output of one being the input to (or
>> command line arguments of) another.
> 
> the users i'm aware of are all just dealing with one file.
> 
>> I believe busybox microcom's expected usage is you can exit microcom without
>> closing the connection, type "sz file.txt" at the command line, and then cursor
>> up twice in your command history and hit enter to get microcom back. I switched
>> the exit character from ctrl-X to ctrl-] because somebody might want to use
>> emacs and ctrl-] is what telnet used back on the sun workstations in 1992.
> 
> (heh, i didn't realize that was you. but, yes, i appreciated the "this
> is like telnet". though i'm reliably informed that the younger
> generation hasn't heard of telnet.)
> 
>> The sharp edge is knowing sz should write to /dev/ttyACM3, which is why people
>> made wrapper shell scripts, and the potential justification for "type child
>> process command line and I'll run it with stdin/stdout going to the serial
>> connection". There was also an environment variable solution at some point, I think?
> 
> (separately i did wonder whether microcom given no argument but
> finding itself on a system with only one /dev/ttyUSB* should just
> default to that?)

If you /dev/tty* and filter out /dev/tty and /dev/tty[0-9] and there's only one
left, sure using it by default makes sense. But there are often ttyS[0-3]
because that's what the chip's got whether or not they're wired up, and I note
that on the emulated sh4 board /dev/console is /dev/ttySC1 because qemu's
/dev/ttySC0 emulation seems to be a type Linux does not seem to have a proper
device driver for. (I've poked people about that repeatedly...)

And I think on the raspberry pi the console is on serial port 4 or some such? I
remember it was annoying, don't remember the details...

(Darn it, I'm sitting in the airport and NOW I remember I forgot my usb to ttl
cable for the Orange PI. Oh well, it's always something...)

>> But that's why they didn't historically build xmodem or similar into the
>> terminal program, because if microcom can exit and let you do stuff and then
>> resume, you're not limited to what they bothered to integrate into the program.
> 
> is it, or is it just that no-one wrote an sx for busybox? (because
> *mini*com definitely has these.)

The fact that nobody wrote an sx for busybox _is_ signaling.

>> You can start up a slip/ppp session or something if you need to. But if you
>> start integrating the functionality _into_ the program, you've never got enough
>> options and it goes fractal fast.
> 
> yeah, to be clear --- i have no particularly strong feelings here. i
> think just having rx/sx is 90% of the value. and i suspect anyone who
> doesn't work out base64 and text paste themselves is going to need to
> copy & paste something from a doc anyway...

Can the doc explain base64 and text paste? :)

>> A typo indicates which style is preferable?
> 
> for me that's always been one of the key "code smells" --- if the
> first time i try to use a new API/UI i get it wrong, i probably need
> to rethink what i'm doing :-)

It says I never ran it and looked at the output.

> even if you just want to argue it was hard to see the mistake you'd
> made, sure --- but that's probably telling you something about the
> immediate grokkability of this prompt.
> 
> which i think is what i like about the "[q]uit" style --- it
> highlights the most interesting bits and forces you to be mnemonic.

In academia, the fighting is so intense because the stakes are so small.

> which, yes, is also a limitation, but if/when that limitation starts
> to bite i'd argue you have way too many options in one menu. (i
> present the *mini*com menu as evidence of this. super fancy
> "curses"-style menu, but a complete nightmare because there's just way
> too much stuff there, some of it seemingly duplicate.)
> 
>> I haven't cleaned up telnet because I use dropbear and netcat. Last time I
>> touched it other than treewide API changes looks like commit 755e040916d9 in
>> 2014 and the only prompt that changed was:
>>
>> -  if(TT.port <= 0 || TT.port > 65535)
>> -    error_exit("PORT can be non-zero upto 65535.");
>> +  if(TT.port <= 0 || TT.port > 65535) error_exit("bad PORT (1-65535)");
> 
> (yeah, you're right that the telnet ui is from the initial commit. but
> _that_ matches the busybox ui.)

http://lists.landley.net/pipermail/toybox-landley.net/2015-April/015293.html

>> >> Then again, grep '"[^"]* .)[^"]*"' toys/*/*.c lib/*.c main.c  isn't finding any
>> >> other non-false-positive hits. If you have a strong opinion about this, I can
>> >> switch, but I just want to be clear that either way this seems to be introducing
>> >> a new rule into the project, and you want that rule to be [a] command rather
>> >> than a) command.
>> >
>> > well, i'm actually suggesting "[c]ommand" rather than "c) command".
>>
>> Which with e[x]it raises the the issue of breaks between multi-word descriptions.
> 
> yeah, tbh i'm not sure why you hated the busybox telnet style of
> multiple lines; it certainly avoids that :-)

Partly I was thinking that if ctrl-] doesn't exit immediately it could
checkpoint the cursor position and jump to a status line at the bottom of the
screen (scrolling once as necessary), do its business there, and then esc[K the
line and jump back to the saved position in the next...

But mostly the wasted space thing.

>> > but, yes, i think if you prefer "c) command" then we should actually
>> > switch to "toybox --help" style where it's _usually_ one per line (and
>> > there's a whole _tab_ for clear and consistent spacing) ...
>> > until/unless you end up with lots of lots of options, at which point
>> > things get a bit ascii-art.
>>
>> I wanted to collate them together on a line because it's an inefficient use of
>> space otherwise, and scrolling stuff off the top of the screen seems impolite.
>>
>> We need _a_ prompt, and we've cleared the next line, so we might as well make
>> use of that space we just reserved for ourselves.
>>
>> We could instead do the "snapshot cursor position and jump to a reserved line"
>> except we don't reserve a line, and jumping to the top might not be noticed and
>> jumping to the bottom means scrolling the screen which we wouldn't want to do
>> multiple types if they typo (since you don't have an ESC key but instead treat
>> any unknown key as ESC with another line of message.)
>>
>> At which point I start going "I started writing not-curses code in hexedit.c and
>> then a second user in less.c and I was going to flesh it out properly in vi.c
>> and have watch.c share it... and if microcom ever needed anything more
>> complicated that was also a potential user... except I didn't get to write vi. I
>> was too slow. And the people who did write the one we are now committed to use
>> didn't know about my infrastructure plans and where else I wanted to reuse
>> common code. Which is part of the reason I haven't started the shell command
>> line editing stuff yet, because that was ALSO tied into this thing that's
>> missing a limb now.
>>
>> Oh well...
>>
>> > (_that_ "toybox --help" style was actually my first thought for toybox
>> > telnet too, if i'm honest, but either choice seemed like "innovation"
>> > and i fell on the side of "same ui as busybox" being "less
>> > innovative". which then naturally led to me doing the same for
>> > microcom.)
>>
>> I haven't looked much at pending/telnet.c. I haven't _used_ telnet it forever, I
>> use ssh or netcat. Does it also have user interface parts like this?>
> 
> _busybox_ telnet does, and toybox telnet looks like it's trying to match that.
> 
> "real" telnet does not. it has a bare "telnet> " prompt and you have
> to ask for help. (plus no-one but us remembers telnet, so i'm not
> convinced even the busybox one really serves as precedent; i only used
> it as such because i literally couldn't think of anything else even
> remotely similar. and *mini*com is the specific nightmare i'm running
> from!)

I applied your patch as-is. It was really the description I objected to, which
was setting policy (it's not "more common"), but you care about this issue a lot
more than I do and a clear policy is not emerging from this discussion. I
suppose we can bring this up again when we've got a second (or third) user to
actually try to be consistent between. :)

That said, I still have issues with the SECOND half of the loop because "read
back echoed characters" vs "not doing that" is, once again, POLICY that's
getting set here by one specific use case. Although when I looked at changing
"[p]aste file" to "send [f]ile" I immediately went "they're going to ask send it
using what protocol and how do I phrase it to be clear..." and I noped RIGHT
back out of there. Trying to get a release out.

AND I want to reuse the line editing code from read_password() but as soon as I
add an echo option the tab wrap nonsense comes back:

$ echo -e '\t\b\b\bx'
     x	

Backspace does not undo the tab! I had a fix for that, but what I NEED to do is
write PROPER interactive line editing code and that's NOT THIS RELEASE.

>> I tend to have unix sensibilities, where "I'm gonna use this command in a shell
>> script" is the default assumption.
> 
> sure, but is microcom useful in that context? are you using
> stty/cat/sx there instead?

If I have to go interact with a remote board to do some manual setup and then
perform automation, yes.

>> A big chunk of your userbase is coming over
>> from windows where everything is gui (no two commands ever connect together,
>> that would be a license violation) and they expect something more like:
>>
>>   https://clip.cafe/who-framed-roger-rabbit-1988/laughing-pull-the-lever/t/1/
>>
>>   https://pinterest.com/pin/324822191849854340
>>
>> One community dials phone numbers and the other has a contact list. One
>> navigates to a destination and the other has elevator buttons for every known
>> destination. One has ingredients, the other has microwaveable dinners.
>>
>> Reality is usually somewhere in between. I have leanings, which I am willing to
>> be talked out of, but I worry about an ever-growing pile of special cases, vs
>> I've had the same hammer for 30 years and learned lots of different ways to use it.
>>
>> I readily admit your first experience being "here's some flour and water" vs
>> "microwaveable ramen" is a big difference. As Yoda said, "Quicker, easier, more
>> seductive..."
> 
> yeah, and i'm mostly here because a bunch of folks who are otherwise
> pretty advanced users struggled to "make the ramyeon". (not helped, i
> think by the whole "wtf? is anyone still using this stuff in 2023?"
> problem that means the web is probably less helpful than usual.)

You have people waiting, the perfect is the enemy of the good, patch applied...

>> > (though i'd still probably have had the ctrl-] ? help
>> > output be one per line, but perhaps you wouldn't have been as offended
>> > by the verbosity if the user had _asked_ for help?)
>>
>> If ctrl-] does NOT exit into an enclosing shell script with a case statement
>> prompting you to send a file or similar, then microcom has a modal interface
>> like vi. Unless microcom has hotkeys for each action (which is what function
>> keys were for but alas desktops started intercepting those so they're useless
>> now, and most of the ALT-X and so on that don't produce an ascii code have been
>> similarly hijacked by greedy GUIs), then you need an obvious visual indicator
>> that you're now in "command mode".
>>
>> As long as you've got that visual indicator anyway, it might as well convey
>> "what can you DO in this command mode?" Having four lines of output do what one
>> line of output could do seemed weird to me.
> 
> yeah, just our usual disagreement of whether "shorter" and "more
> readable" are the same or different :-)

They're... not quite orthogonal, but not inherently the same thing.

>> >> > (and, if nothing else, fix the previously
>> >> > inconsistent spacing between options
>> >>
>> >> Yup, my bad. :)
>> >>
>> >> > and make it clear what new options
>> >> > should look like).
>> >>
>> >> I'd rather not apply the patch as is because of the above commit comment
>> >> grumbles, but don't object to code change itself. Should I switch the -64 to &31
>> >> while I'm there?
>> >
>> > (i still think we should do that properly as you described --- let's
>> > have the macro, but have it do the _right_ thing and `^ 0x20` first.)
>>
>> If replacing three character -64 with 6 characters of macro wrapper seems like
>> the right thing to you... ok?
> 
> i think, like the above arguments, a lot of it comes down to my usual
> "don't make me think". (or, in the less snappy but more folksy version
> favored by a guy i used to work with, "the one thing i can guarantee
> is that i won't be smarter tomorrow than i am today, so i should try
> to make things easy on future me".)

If I could #define my own CTRL() macro without worrying about conflicts with
glibc's headers I'd be happy. Including a header that defines a CTIME that has
nothing to do with stat's ctime makes me go "this is gonna cause a debugging
session in future, isn't it?"

> that header's in macOS too, but like i keep saying --- i'm actually
> suggesting (based on your "what about case?") that we have a _better_
> one where it doesn't matter because the macro does the right thing
> regardless.

But we're already taking that namespace pollution hit apparently, so...

It's not in posix, it's not in LSB. A THIRD non-posix header for toys.h. But
glibc's git says the file's been there since 1997 and uClibc had it attributed
to UC Berkeley circa 1982...

> i suspect https://github.com/tio/tio#32-key-commands is probably
> making new traditions right now. well, that and PuTTY! tbh, i only
> ever see people on youtube using gui stuff; there's even a gtkterm!
> 
> (interestingly, i see tio also only has xmodem and ymodem. i think
> zmodem is just too complex for what it does to have any value any
> more.)

If ymodem gracefully fails back to xmodem (still seems sus) then those two are
the same implementation. I agree zmodem seems to have fallen off the map.

(Maybe check if your first packet has a start of frame header in it and only do
1024 byte packets when it doesn't? Can you "upshift" to bigger packets in the
middle of a stream? I guess you can safely NAK anyway. I should reread the
ymodem doc, they might already cover this. Mostly I remember the extensive "no
two versions do quite the same thing" admonitions...)

I guess first packet should go "send the start of the header type you'd like to
use, WAIT 1/4 SECOND BEFORE SENDING ANY PAYLOAD TO SEE IF THEY NAK IT, then send
payload"...

>> But seriously... xmodem?
> 
> well, when we get that time machine, we can add "include a file size,
> because cp/m isn't going to last long" to our to-do list... :-)

https://landley.net/mp3/S-100.mp3

Rob