[Toybox] [PATCH] microcom: simple menu, new "paste file" functionality.

[enh]

On Fri, Nov 3, 2023 at 6:58 AM Rob Landley <rob at landley.net> wrote:
>
> Blah, forgot to click "send" and let the window get buried...
>
> On 10/28/23 12:18, enh wrote:
> > On Sat, Oct 28, 2023 at 12:29 AM Rob Landley <rob at landley.net> wrote:
> >>
> >> On 10/25/23 19:48, enh via Toybox wrote:
> >> > Rather than take up yet another key for the new functionality, I've moved
> >> > everything to a menu, similar to the one in telnet. It's not amazing,
> >> > but it's the least code I could get away with, and it's good enough for
> >> > now, and it leaves us in a better place if we need to add more stuff
> >> > (such as xmodem, say).
> >> >
> >> > But this is enough to let me stop using minicom and get back to toybox
> >> > microcom for today!
> >>
> >> What I always did was "toybox uuencode target_filename < local_file | xclip
> >> -selection clipboard" and then run uudecode at the far end and right
> >> click->paste it into the relevant xterm window. I hadn't noticed a size limit...
>
> Sorry, applied this yesterday but didn't push. (My maybe-covid has turned into
> conventional cold with sore throat and stuffed up nose and everything, and I'm a
> little out of it right now...)
>
> > yeah, so recently i had one group ask for xmodem and another group who
> > just wanted to use base64. the latter was the thing i could do in half
> > an hour :-)
>
> xmodem? <obiwan>Not that's a name I haven't heard in a long time.</obiwan>

yeah, every week has been retro week lately...

> Sheesh, If you're going to do any of that use zmodem. We were mad at xmodem back
> on the C64: latency spike waiting to ack each 128 byte packet, rounding up the
> file size to 128 bytes, single byte simple sum error detection, no standard way
> to abort a partial transfer without leaving the recipient hanging...

"all i know i learned from the internet over the last week or two" but
it seems like the argument for xmodem is the same as it was in the
1980s --- it's the lowest common denominator (and, hey, if you're
messing about transferring binaries over serial lines at all, that's
definitely the territory you're in).

> (And if you're going to have any nontrivial transfer protocol, why not do what
> all the term programs did and call a child executable you hand the serial
> connection to so it can do the transfer, then exit back into the terminal
> program. That way you're not hardwiring many protocols into your term program.)

(oh, yeah, as you can see on the other thread, i might still argue
that "plain text" is a special case, but anything else should
definitely be a fairly generic "call this program".)

> The only times I've had a corrupt file across serial in the past 20 years were
> because of an insufficiently buffered UART dropping bytes, which was an OS
> latency spike problem not a serial hardware problem. And I say that with a
> UART->ttl cable right next to me because I need to plug it into the orange pi 3b
> I've been gradually setting up as a server. Unfortunately the shortest one I
> have is 6 feet and I'd like maybe a 3 inch one because the kitchen counter space
> next to the router is a finite resource...

i have no reason to believe anyone asking for this cares about
corruption. afaict they care about the size increase of using
base64/uuencode and a text transfer.

_personally_ i'm on the fence. i've been pretty happy with what i've
already sent you (especially after adding the progress indicator), but
then i can trim my binaries so i'm only transferring the specific
routine(s) i'm testing. at least one of the "i wish i had lrzsz" folks
is transferring linux kernels. they would definitely benefit more from
less overhead.

...though at that point, i start to think _all_ of the historical
protocols are shit, and if you only need to care about "toybox on both
ends" then let's take advantage of the fact that cpu time is very
cheap these days and compress/decompress. (though (a) that rules out
all the potential use cases that xmodem would cover and (b) the
specific case of the linux kernel tends to be compressed anyway,
right, so they specifically wouldn't benefit? plus now you have a
_design_ problem to add to the pile.)

> > (others had no idea how to transfer a file over serial at all. no-one
> > came up with your solution _that i know of_ but maybe this is what the
> > people who complained that "doing the paste myself is flaky"?)
>
> My solution evolved from "cut and paste from the terminal" to "cut and paste
> from mousepad" to "actually learn to use the command line clipboard tool" over a
> multi-year period of fiddling with boards that have serial interfaces. (Or qemu
> system emulation with /dev/console connected to stdin/stdout as an emulated
> serial device, which is how mkroot sets everything up and aboriginal linux
> before it.)

yeah, i didn't know your trick. (and even if i did, did i mention yet
how nice it is to have a progress indicator? 115200 is _slow_ by 2023
standards.)

others didn't even work out the base64/uuencode trick. are those
people likely to know wtf xmodem was^wis? probably not, but at least
it gives them something concrete to search the web for!

> In theory I could sha1sum the file afterwards and check the first couple and
> last couple bytes not being nonsense, but in practice I haven't had a corrupted
> file in forever. It's a bit like not running on rad-hardened hardware: yes we
> COULD SECDED all the memory and checksum/resend the memory bus transactions, but
> these days it's hard to even buy parity memory because the voltage of the signal
> and capacitance of the wire are way above what a cosmic ray of alpha particle is
> gonna do. (And if you somehow get one of those giga-giga-whatever jumbo cosmic
> rays deciding to pop inside your electronics, which should never happen, with
> modern microtrace sizes you now have a cut wire.)
>
> >> There was discussion of a CTRL() macro in context of vi, but I don't see where
> >> it was applied to toys.h or lib/lib.h? Ah, it's in glibc's sys/ttydefaults.h
> >> which is included from termios.h... but not in the NDK build, or in the musl
> >> build. It _only_ leaks in the glibc build.
> >
> > ah, oops. i did check that everyone (including macOS) _has_
> > <sys/ttydefaults.h>, but i didn't check that it was getting
> > [transitively] included everywhere.
> >
> >> The macro's actually a bit non-obvious to me because CTRL(A) and CTRL(a) seem
> >> like they should produce different results? Ah, and they want CTRL('A').
> >
> > yeah, if we did our own macro i'd suggest `#define whatever(ch) (((ch)
> > ^ 0x20) - '@')` to avoid that. there doesn't seem to be a strong
> > consensus about whether it's more natural to say A or a, so may as
> > well make both work.
>
> Or just &31 which is what I probably should have done. Assuming CTRL-SHIFT-X
> isn't a thing we care about. (I tried ctrl-shift-h at the command line and it
> did a backspace, so apparently not...)

(the `^ 0x20` is the cheapest way i know to force case; you'd want
that _and_ the subtraction/and to solve the ctrl('a') vs ctrl('A')
problem you were talking about.)

> >> What was wrong with 'A'-64 again?
> >
> > that's a variant i _haven't_ seen yet. 'A'-'@' seems more common, but
> > it's not as readable to n00bs.
>
> I went with the first 3 character fix off the top of my head. &31 is also 3 chars.
>
> >> You want "p) paste file" instead of "f) send file"? (Paste-but-not-clipboard has
> >> more mnemonic weight than File?)
> >
> > having used this for a week, i intend to send you another patch that
> > changes 'e' for exit to 'q' for quit because that's been annoying,
>
> *shrug* Sure.
>
> > but
> > i don't have a strong opinion on 'p'. i went with that because it was
> > the "more direct" of minicom's _two_ ways to send a file as "ascii"
> > (terminology i disliked for obvious reasons).
>
> I haven't used minicom in ages. Busybox grew a microcom back in 2007, before
> that I was using netcat -f.
>
> > it sounds like the cool kids use https://github.com/tio/tio these
> > days, and https://github.com/tio/tio#32-key-commands shows that it
> > doesn't have an equivalent of this functionality anyway.
> >
> >> The read(1) loop is similar to the loop in lib/passwd.c:read_password() and I
> >> should probably factor it out.
> >
> > yeah, and similar to vi and hexedit. this was the minimum i needed
> > right now, though, and i thought you'd have strong opinions on what
> > exactly to factor out anyway, given that the shell is going to need
> > something readline-like.
> >
> >> You only accept DELETE not BACKSPACE to shorten
> >> the input line?
> >
> > yeah, and i didn't do arrow keys or home/end or anything else;
>
> In xfce my backspace key produces ctrl-h, my delete key produces ascii 127. Sun
> microsystems had it the other way around in 1992. I vaguely recall there's a way
> to change the mapping in the tcsetattr)() nonsense, but I've seen backspace be
> "remove character to the left of the cursor" and delete be "remove character to
> the right of the cursor" for a longish time.
>
> I don't remember what KDE or Gnome did, it's been years since I used either
> outside of VMs and guest logins on other people's machines...

we'll have to worry about this when it's time for the shell's
readline(), but i'm hopeful we'll just reuse that for all these other
places. (but, yes, +1 to factoring this stuff out sooner rather than
later, but that sounded like the kind of thing you'd want to do
yourself so i left it :-) )

> > nor did
> > i deal with "what happens when you have more input than fits on one
> > line?" ... in the week i've used this, _tab_ (for file completion) was
> > the thing i would have actually liked, but not enough to warrant
> > implementing it, and especially not before the inevitable [but hard to
> > predict timing of] factoring out!
>
> Somewhere I said I have a big todo item for shell line editing with all the
> bells and whistles. I should bump that up...
>
> >> You have a progress indicator instead of letting the other end do that. (How big
> >> are the files you're sending?)
> >
> > well, i'm using this for `base64 -d -i > file` so the other side isn't
> > going to help me here. (and a quick glance at the xmodem docs suggests
> > that one of the limitations of that protocol is that it doesn't
> > include file size, so even there i think the _sender_ is going to have
> > to do the work.)
> >
> > given that the only use i see for this is "your board can't even do
> > adb yet", the likelihood [for me] is that i'm having to send static
> > binaries. after seeing how long a 20MiB [well, ~10MiB, but base64
> > bloats it up] binary took (two meetings + cooking and eating a meal),
> > i did make an effort to trim my binary down to just what i actually
> > needed to iterate on,
>
> I provide static toybox binaries linked against musl for a _reason_. :)
>
> What do you have your serial rate set at? Anything modern should be able to do a
> megabit no problem even with a ten foot cable, the old boards that maxed out at
> 115,200 were running at 16mhz. Modern uarts have all that bit edge detection
> clock resyncing stuff which is why there's still a 1 in the 8n1 to force a
> periodic transition even when you're sending all zeroes or ones. (Even within
> chips, clock domain crossings looked like tiny half-assed UARTs to me.)

the folks i've been working with are 115200. but let me +Peter
Collingbourne to represent the "i want to transfer linux kernels" case
better than i can (because i don't even know what speed he's running
at/how long his transfers take/whether he sees errors/etc).

> > but even so it was a couple of MiB each time.
> > "long enough that i favored one of minicom's two alternatives for this
> > functionality based on the fact that it showed me progress, but not
> > long enough that i bothered to implement a countdown/ETA".
>
> I've been meaning to factor out the new count -l logic into lib somewhere. I
> just wanted a bit more thought on the hardwired logic ala 4/sec update
> granularity and 8/16 (I forget) seconds trailing history in the rate. (Rate of
> the whole thing is a trivial size/ticks calculation at the end if we want that...)

(i'm not sure i trust it yet, if you (a) claim that your debian yes(1)
gets GiB/s but (b) toybox yes(1) also relying on stdio buffering
doesn't, because strace makes it look like my debian yes(1) is just
using stdio buffering for simpler code _and_ better performance!)

> A second user for "progress indicator" is interesting. And I suppose wget would
> be a third. Both those know how long the total should be (which COULD be a
> command line argument for count, or I could have -l do an fdlength() on stdin to
> see if we're reading from something it can detect length for...)

(yeah, and factoring out that "fdlength() _can't_ tell you the length"
special case and handling it correctly definitely sounds like a good
idea.)

> Almost every command has more work I COULD do on it, I just try to reach a good
> stopping point and prioritize...

(yeah, i've always argued that _two_ copies is often the least worst
option [because you probably don't even understand the wider problem
well enough yet], but _three_ copies is where you want to start
thinking about factoring out.)

> >> Rob
>
> Rob