[Toybox] [PATCH] microcom: move from e[x]it to [q]uit.

[Rob Landley]

On 11/8/23 19:04, enh wrote:
> On Wed, Nov 8, 2023 at 8:32 AM Rob Landley <rob at landley.net> wrote:
>> "This is left over from 1983. It does it wrong but we're stuck with it..." sigh.
>> If we must...
> 
> there was a good old joel spolsky post about rewriting the netscape
> ftp client (try explaining that phrase to a young person!) pointing
> out that when it comes to compatibility "the cruft is the value". i
> don't think there's anything good to say about xmodem other than its
> ubiquity, but its ubiquity is the point.

Yes and I wrote a rebuttal to it many moons ago about how open source and
standards (or at least the IETF two compatible implementations requirement) in
_theory_ can make it worth doing anyway because you frack the corner cases out
of the stone and _document_ them. Where was that...

Not that, which cited it in support of a different point:
https://landley.net/notes-2011.html#08-05-2011

Not that, which was about mozilla's perpetual implosion:
http://landley.net/notes-2020.html#30-08-2020

Was it the prototype and the fan club talk? My outline doesn't have URL links
(https://landley.net/talks/flourish-2010.txt) and the recording absolutely
botched the video (missed the start, loud buzzing)...

Sigh. Somewhere. Lost in the compost heap. But "Joel makes a good point for a
specific context but rules differ for other development contexts and it's also a
matter of degree". I had like 20 minutes of material on it, but I remember
_editing_ it (I.E. not a new observation) when I was giving a presentation to
management at Timesys, which I was at from 2006-2007... I was comparing mozilla
to what was going on in Staroffice/OpenOffice (had Libre Office happened yet)
where reverse engineering "this one magic implementation is the standard"
microsoft word had produced a _second_ magic codebase and the efforts to
actually document a standard got politically hijacked because they thought
forming committees and letting microsoft participate would somehow help, plus
the whole "document not legislate" thing got seriously violated...

Anyway, yes I'm familiar with that article. When I first read Joel's "Amazon vs
Ben and Jerry's" I wrote such a glowing review of it that my editor licensed
Joel's article to run alongside my review:

https://web.archive.org/web/20000815055446/http://www.fool.com/portfolios/rulemaker/2000/rulemaker000804.htm

(And then did his usual heavy-handed edit to put words in my mouth, remember
when I wrote my review I didn't KNOW they'd be licensing it and running it on
the site...)

>> >> >> What xmodem command
>> >> >> are you going to run at the other end to consume the data and send the replies,
>> >> >
>> >> > i was going to write one. the xmodem spec sounds like it's only going
>> >> > to be tens of lines.
>> >>
>> >> There's a spec? It's like 3 status bytes and 128 byte packets... (I was on
>> >> bulletin boards when I was 12. There's a nonzero chance I implemented it back on
>> >> the commodore 64, and even then it was inferior to punter.)
>> >
>> > https://pauillac.inria.fr/~doligez/zmodem/ymodem.txt
>>
>> I read that back when it came out. :)
>>
>> Which is why I was suggesting implementing ymodem. It has the OPTION to not be
>> broken, and can presumably clean-ish-ly fall back to the broken stuff if
>> necessary. (There are a lot of broken things calling themselves ymodem, but
>> that's true of xmodem as well...)
> 
> yeah, that makes sense, and seems to match what most people actually
> implement as "xmodem" anyway. case in point: look closely at the url i
> gave for xmodem documentation :-)
> 
> it's _z_modem (which is what i thought you were suggesting as an
> alternative) that seems unconvincing to me.

Back in the BBS world xmodem->zmodem left ymodem in the lurch. It's the part
where xmodem outlived the BBS world that confuses me. (I jumped ship from
fidonet to internet when I got my first dialup home ISP account in... 1997? Not
much exposure to xmodem since.)

>> >> Busybox never bothered to implement the sx command, so I can't check there, but
>> >> I would expect them to do that because "I am at a command prompt, send file"
>> >> would just work. :)
>> >>
>> >> Except for the part where xmodem doesn't know what the filename is. I'm guessing
>> >> it would write a.out by default?
>> >
>> > lrzsz takes a [required] filename:
>> > https://manpages.debian.org/testing/lrzsz/rx.1.en.html
>>
>> If we're sending rx\n anyway we can supply a filename, modulo control characters...
> 
> to be clear: i _wasn't_ suggesting sending "rx\n". i think that's a
> bad idea myself!

If it gets ignored by the remote end when rx is already running and is common
behavior, why is it a bad idea? It's _ugly_ but this is xmodem we're talking about.

>> If ymodem gracefully fails back to xmodem (hard to test without a send side)
>> then if were gonna implement sx/rx we might as well have it actually do ymodem.
> 
> from https://cambium.inria.fr/~doligez/zmodem/ymodem.txt it seems like
> that depends on the receiver. the sender (which goes _second_) can't
> tell the difference between an xmodem receiver and a ymodem receiver,

If the far end asks for stupid we give it stupid.

> so you're reliant on an xmodem receiver gracefully rejecting a ymodem
> start (and if you are worried about actual packet problems [which i
> don't think we are], being fine with a problem with first packet
> causing the entire transfer to fall back to xmodem).

If we send just the header and wait 1/2 second for a NAK, it should be fine?

A problem with initial negotiation causing the transfer to fall back to stupid
is an inherent problem with xmodem, yes. Although we could try to "upshift"
again before the end and be ready for a nak. (The end padding is the annoying
part, the rest is just performance, and base64 is a performance hit too,
although probably still less than the latency spikes between each packet of xmodem.)

> "seems fine until/unless proven otherwise?" and if you've implemented
> that, adding flags to let you _start_ with xmodem or _require_ ymodem
> would be trivial anyway.
> 
> i think the real question is "who benefits?". does anyone _actually_
> want more than xmodem, or is that "good enough"?

It corrupts the files transmitted through it. That's the part I have a problem
with. Beyond that a single byte ping pong protocol is "eh, your funeral". People
shiped Asynchronous Transfer Mode into the wild (for some reason). Under the
covers USB is a ping ping protocol! (Each packet waits for acknowledgement from
the host! When were implementing a USB 2.0 device in FPGA that was just... ouch.
Microsoft was clearly on the committee.)

>> Of course "gracefully fall back" assumes that the newer packets you send that
>> get NAKed don't contain within their payloads start-of-packet sequences for the
>> older stuff, but "did not understand packet 0" is probably important signaling,
>> except "NAK because corruption vs NAK because protocol"...
> 
> the first byte tells you what you're dealing with. afacit there's no
> problem with bytes within the packet (unlike telnet, which is probably
> what you're thinking of?).

If you send a 1k packet to a recipient expecting a 128 byte packet, and it barfs
out of the header and sends a NAK, the "larger than it expected" remaining
packet data potentially becomes in-band signaling possibly misinterpreted as a
header.

Hence the "send header and pause a bit waiting for a NAK". Or maybe send header
+ 128 byte payload and then pause and check for a NAK before continuing.
Probably pause in both places on the first packet, to be honest, I dunno how
stupid the other end is...

>> > --- even in 2023 (well, okay, 2021), xmodem is still the lowest common
>> > denominator of serial transfers. that's what i claim rx/sx are useful
>> > for, and that's also why i'd claim that rz/sz _aren't_ useful. because
>> > anyone doing anything fancy is probably on scp/rsync, not zmodem :-)
>>
>> The qwerty keyboard of file transfer protocols...
> 
> exactly! i could have used that example above if i'd thought of it :-)

And now you have thought of it.

>> If you /dev/tty* and filter out /dev/tty and /dev/tty[0-9] and there's only one
>> left, sure using it by default makes sense. But there are often ttyS[0-3]
>> because that's what the chip's got whether or not they're wired up, and I note
>> that on the emulated sh4 board /dev/console is /dev/ttySC1 because qemu's
>> /dev/ttySC0 emulation seems to be a type Linux does not seem to have a proper
>> device driver for. (I've poked people about that repeatedly...)
> 
> (sorry, i really should mark "idle thoughts" clearly so they're not
> confused with "actual suggestions". this was just an idle thought,
> that i wasn't convinced by. at least not until/unless i see someone
> struggle with the argument. i do notice that tio has a -l flag to list
> the available options. _that_ might be more helpful without being
> error prone/failing badly when you have _two_ things plugged in?)

There's probably a way to probe for serial. On devuan, the serial /dev devices
belong to group "dialout", although that's probably udev rules rather than a
probe... Ah, setserial is using ioctl(TIOCGSERIAL); which is in man ioctl_list.
It requires root access to open the devices, but so does microcom...

*shrug* Seems easy enough. Iterate over /dev and call the ioctl on the char
devices to see which ones don't error.

>> The fact that nobody wrote an sx for busybox _is_ signaling.
> 
> it may well be, but i'd argue it's signaling the opposite --- that
> no-one cares whether busybox has an sx because they're all using the
> one that's built in to their terminal. (which _isn't_ busybox microcom
> :-) )

Busybox felt the need for a microcom but not the need for an sx. If nobody uses
busybox microcom, why do they have it?


>> > yeah, tbh i'm not sure why you hated the busybox telnet style of
>> > multiple lines; it certainly avoids that :-)
>>
>> Partly I was thinking that if ctrl-] doesn't exit immediately it could
>> checkpoint the cursor position and jump to a status line at the bottom of the
>> screen (scrolling once as necessary), do its business there, and then esc[K the
>> line and jump back to the saved position in the next...
> 
> at that point i think you have to decide where on the spectrum from
> "minicom" to "busybox microcom" toybox's microcom should be. that kind
> of thing is _very_ minicom :-)
> 
> (which is fine, and seems to be pretty popular, but it's a very
> different part of the design space.)

It's half-assing it that bothers me. I'm happy with _not_ doing it, and I'm
happy with doing it right. (Which is also why C++ bothers me in a way neither C
nor Python do. Either have opaque abstraction that save me from having to know
all the implementation details, or avoid unnecessary abstraction and stay out of
my way. PICK ONE. And yes, I was a C++ programmer for years before going _back_
to C. Still do it at contracts that require it. It's annoying.)

(And no, the ascetic approach of staying within strict arbitrary bounds and
never looking at the cthulian madness bubbling underneath isn't a good answer
for me either. I'm gonna break it in a way the guy who created it will head
scratch at.)



>> > "real" telnet does not. it has a bare "telnet> " prompt and you have
>> > to ask for help. (plus no-one but us remembers telnet, so i'm not
>> > convinced even the busybox one really serves as precedent; i only used
>> > it as such because i literally couldn't think of anything else even
>> > remotely similar. and *mini*com is the specific nightmare i'm running
>> > from!)
>>
>> I applied your patch as-is. It was really the description I objected to, which
>> was setting policy (it's not "more common"), but you care about this issue a lot
>> more than I do and a clear policy is not emerging from this discussion. I
>> suppose we can bring this up again when we've got a second (or third) user to
>> actually try to be consistent between. :)
> 
> yeah, sgtm. and i promise i won't just go straight to unzip to try and
> stuff the ballot :-)

It's on the todo list but I need to circle back and finish the deflate
compression side first. (And the question there is dictionary selection and
reset, mostly because I want to be able to match the hash of generated tarballs...)

> i'd be happier with clear precedent too. _personally_ i'm quite
> persuaded by your "the unix way would be to just exit and come back"
> but (a) i hadn't thought of that myself and (b) we wouldn't be having
> this conversation in the first place if we weren't trying to help
> "less traditionally unixy" users.
> 
> one crazy idea i had was using ^Z instead... it's familiar from
> suspend, and feels like it should "just work",

The problem is it should "just work" at the far end too.

In-band signaling. Ctrl-] was the least bad character I could find. (Well,
there's ONE other: on my system ctrl-backtick produces ascii NUL. That's
unlikely to be part of a normal input stream. But CTRL-] has precedence as being
avoided because telnet. Speaking of which, the menu needs an option to send
through an ascii 29. Maybe ctrl-] ctrl-] twice?)

> and certainly lets you
> `sx`. doesn't let you send breaks, though, and if you've actually
> _exited_ `fg` isn't going to work.

Having Ctrl-Z sometimes suspend and sometimes exit is not a good idea.

>> AND I want to reuse the line editing code from read_password() but as soon as I
>> add an echo option the tab wrap nonsense comes back:
>>
>> $ echo -e '\t\b\b\bx'
>>      x
>>
>> Backspace does not undo the tab! I had a fix for that, but what I NEED to do is
>> write PROPER interactive line editing code and that's NOT THIS RELEASE.
> 
> (yeah, that was roughly my excuse for just copy & pasting out of
> hexedit :-( not least because i'd want fs tab completion here!)

And wildcards? Because that's shell plumbing right there...

>> >> I tend to have unix sensibilities, where "I'm gonna use this command in a shell
>> >> script" is the default assumption.
>> >
>> > sure, but is microcom useful in that context? are you using
>> > stty/cat/sx there instead?
>>
>> If I have to go interact with a remote board to do some manual setup and then
>> perform automation, yes.
> 
> actually: do you use *mini*com at all?

Nope. Last used it... at Polycom I think? 2012. It had weird menu-based setup
persistently saved in a config file that was a huge pain, being able to specify
everything on the command line was SO MUCH EASIER when juggling 3 boards...

> personally i've only used it
> when toybox minicom has been insufficient for some reason, and then
> i've done "just enough" to toybox to not have to continue doing that
> because it's just so ... i don't even have a word for it. is there a
> word for "like some DOS nightmare from the 80s?".

Indeed.

>> > i think, like the above arguments, a lot of it comes down to my usual
>> > "don't make me think". (or, in the less snappy but more folksy version
>> > favored by a guy i used to work with, "the one thing i can guarantee
>> > is that i won't be smarter tomorrow than i am today, so i should try
>> > to make things easy on future me".)
>>
>> If I could #define my own CTRL() macro without worrying about conflicts with
>> glibc's headers I'd be happy. Including a header that defines a CTIME that has
>> nothing to do with stat's ctime makes me go "this is gonna cause a debugging
>> session in future, isn't it?"
> 
> (yes, i'm sad they're camping on the best name with a bad
> implementation. to be fair to glibc, though, i think "the glibc
> header" is just an exact copy of the BSD header. bionic's [and macOS']
> certainly is.)

Commit 8aad4f426718.

The header is copyright 1982 by Berkeley, and the CVS graffiti says
 *      @(#)ttydefaults.h       8.4 (Berkeley) 1/21/94

Meaning the specific _version_ predates Linux 1.0. (Modulo gnu damage.)

The fact it's not in posix despite that is yet more "posix doesn't specify the
init or mount commands" nonsense. A standard with holes big enough to drive
Microsoft Windows NT and IBM OS/360 through so they could get that FIPS-151-2 money.

>> If ymodem gracefully fails back to xmodem (still seems sus) then those two are
>> the same implementation. I agree zmodem seems to have fallen off the map.
>>
>> (Maybe check if your first packet has a start of frame header in it and only do
>> 1024 byte packets when it doesn't? Can you "upshift" to bigger packets in the
>> middle of a stream? I guess you can safely NAK anyway. I should reread the
>> ymodem doc, they might already cover this. Mostly I remember the extensive "no
>> two versions do quite the same thing" admonitions...)
> 
> (yes, they talk about this. there's something about not changing
> packet size in response to a NAK, but other than that you can change
> packet size whenever you like.)

Except if we're doing the downshift because the far end doesn't support 1k
packet sizes, then we ARE changing packet size in response to a NAK.

Rob