[Toybox] Weird stdout buffering effects

[enh]

On Tue, Oct 24, 2023 at 1:55 AM Rob Landley <rob at landley.net> wrote:
>
> On 10/23/23 18:13, Ray Gardner wrote:
> > Rob, thanks for the info, I learned some stuff about pipes from it.
> >
> > I also looked up setvbuf(), and the standard says it may not be used after
> > a previous successful call to setvbuf(). That makes my tests below kinda
> > moot.
> >
> > But still, using _IOLBF and then _IOFBF seems to work with glibc and musl
> > to get fast output.
> >
> > So, to get fast output from any toy, I think there should be an option to
> > do no call to setvbuf() at all. What is the reason for defaulting to no
> > buffering? If it's a must, maybe need TOYFLAG_DEFAULTBUF or something.
>
> Didn't we just address these same questions in the other thread?
>
> http://lists.landley.net/pipermail/toybox-landley.net/2023-October/029843.html
>
> My personal history left me distrusting stdio buffering, especially in glibc.
> I'm trying not to let that affect my judgement here, but it's hard.
>
> Here's a presentation Denys Vlasenko (the busybox maintainer i handed off to)
> gave at the Embedded Linux Conference in 2010 where he talks about a bug that
> had been hounding busybox for years, where if you statically linked with
> --gc-sections then stdio wouldn't flush output at exit (and just discard
> anything buffered), because the atexit() blob the library was shoehorning in
> without an actually call to atexit() wasn't labeled in a way that stopped it
> from getting discarded as "unused", and the binutils people said glibc was doing
> it wrong and the glibc people said binutils was doing it wrong and they spent
> more than 5 years pointing fingers at each other rather than fixing it:
>
> slides: https://elinux.org/images/2/2d/ELC2010-gc-sections_Denys_Vlasenko.pdf
> video: https://bootlin.com/pub/video/2010/elc/elc2010-vlasenko-dead-code.ogv
>
> (A chronic pet peeve of mine is "these people know this, those people know that,
> why is it MY JOB to get them to talk to each other, and then when I do they
> fight!" It's exhausting.)
>
> If you're wondering why xexit() does an explicit flush and everything is piped
> through xexit() instead of other methods of exiting... that's only part of it.
> Another part is that atexit() doesn't have a way of removing things from the
> list, nor calling the things on the list but NOT exiting and instead
> longjmp()ing back to a recovery thing, so I had to write my own so sh.c could
> call MAYFORK stuff...
>
> Anyway, needing to remember to manually fflush(stdout) when doing things like a
> "password:" prompt is why I had xprintf() and xputc() do an fflush after every
> call, which Elliott took out because it was slowing stuff down making both of
> those ferror(stdio) but not do the flush that would see if there WAS an error.
> (And you have to check for error otherwise things like "yes" won't exit in
> pipelines, but just sit there spinning and eating CPU, and the whole should
> pipeline processes be killed by signals thing is a can of worms.)
>
> I very much _want_ to not care about stdio buffering and just let libc do its
> thing. Unfortunately the default is inconsistently wrong,

i know what you mean, but i think this is the crux of the problem ---
your alternative is "inconsistently wrong" too. i think it's the
_other_ main problem in computer science: leaky abstractions.

_if_ we could implement buffering that did what you _want_ then, yes,
obviously that would be better. but the only two ways i can think of
to do that are (a) radically restructure _all_ the code to have some
kind of event loop with a heartbeat that flushes the buffer or (b)
spin up a thread to flush buffers. neither of those seem
likely/acceptable, so we're left with the current situation.

and -- though there is much not to like about the _default_ stdio
buffering, mostly the specific case you always complain about of "but
what if i'm a human waiting for output?" -- i think there's much to be
said for being no more/less wrong than the next guy. is the
traditional unix behavior sometimes annoying? yes, you're right, it
is, and i don't think anyone's denying that. but it's fairly well
known, and it's predictable, and it's not that bad. (it's also really
obvious when you're staring at it!)

but "yes(1) is now far more complicated than it should be to work
around a self-inflicted would" doesn't seem like a great place to be,
and nor does "most commands throw away performance 'just in case', and
then we either manually undo our buffering hack or we make the code
more complicated without making either side particularly happy".

> and anywhere I try to
> shove setting a known one (so it's at least CONSISTENTLY wrong) people complain,
> and I'm wisting after just ripping out all use of FILE * and just always using
> filehandles because then I'd know what the behavior _is_.

(that would be an even worse disease: then you'd either (a) have
terrible performance _all_ the time (b) need to reinvent stdio
buffering anyway or (c) have to overcomplicate every command like
yes(1) now is.)

> (Probably some kind of
> lib/lib.c function that takes a null terminated char * array like xexec() uses
> and prints it atomically using writev(), falling back to malloc-ing and
> memcpy-ing a contiguous copy as necessary. With optional separator string. And
> maybe I'd want to move the add_arg() function from sh.c into lib.c so I could
> push stuff to such arrays as a stack with the realloc() handled for me...)
>
> I'm not there yet. But I'm starting to think this discussion won't end short of
> that.
>
> The last time I talked to the C committee I was asking for a standard way to get
> the buffered data out of an input FILE * (specifically asking the number of
> bytes I can read from the FILE * without triggering another underlying read()
> system call on the file descriptor) so that when I call a child process I can
> pass that data on to its file descriptor, the way I'm doing in tar.c line 1091,
> and the posix guys said "FILE * is handled by the C standard" and the C guys
> went "nothing in our standard ever mentions file descriptors" and once again two
> maintainers pointing fingers at each other and refusing to fix stuff:
>
> https://landley.net/notes-2022.html#19-04-2022
>
> (Once again getting people to talk to each other, and they point fingers.)
>
> Luckily, I figured out a different way to mostly avoid triggering the problem
> for my shell implementation:
>
> https://landley.net/notes-2023.html#07-01-2023
> https://landley.net/notes-2023.html#05-02-2023
>
> But this is ALSO why I wrote my own byte-at-a-time get_line() function back at
> the start of toybox development, which was slow but _correct_. Ok, also because
> getline() was still a glibc extension until posix-2008:
>
> 6769f8eb580a
>
> And then the android guys went "nope, too slow" and I sighed and agreed to use
> getline() out of libc rather than writing my own buffered one because the buffer
> reading too far ahead is THE fundamental design problem here that I couldn't do
> a LESS bad job of. So Elliott started on a get_line() removal quest:
>
> b30674681b9d
> 4885e8fea8f7
> 3ead70e503b2
>
> git log 2a5dc105a323a20a and you get like 7 commits removing get_line() from
> stuff, continuing at:
>
> 2243f6f2ad08
> 15cbb92dffc8
> c23b3ff44948
>
> But there's a fundmental problem when you implement something like wget and read
> some lines of input followed by binary data logically handled via sendfile()
> which takes a file descriptor. The handoff between the two sucks conceptually.
> You cannot pass a FILE * to a child process. You cannot ungetc() a pipe. You can
> MSG_PEEK on recv() but writing data into a read source is a security violation
> waiting to happen.
>
> Getting this right is _hard_. I preferred avoiding it, but the performance
> sucked. "Trust the libc" sadly didn't _work_, and when I try to micromanage it
> people ask why I did that.

(well, no, if you'd done _that_ --- rewritten your stdio with the
appropriate back doors to be able to do things like this, that would
be different, and have a much more convincing rationale. but what we
_actually_ have right now is that we disable stdio buffering right at
the start of main, don't really get the behavior _you_ want
[uniformly, anyway], and also don't get the behavior everyone else is
expecting.)

> > I ran some more tests, outside of toybox, with setting stdout mode once,
> > or twice in sequence, before writing to stdout. These tests are all with
> > redirection to a file.
>
> On glibc, bionic, musl, mac, freebsd and qnx?
>
> > BTW, a few days ago you wrote:
> >> Also... adding LINEBUF when the calls are xprintf() and xputc()... those
> >> are explicitly supposed to flush. Flushing and checking errors are what
> >> those DO.
> >
> > But xprintf() does not flush.
>
> It used to! But back during
> http://lists.landley.net/pipermail/toybox-landley.net/2019-April/018382.html I
> caved and did github.com/landley/toybox/commit/2a1f89e5d941 which of course
> broke github.com/landley/toybox/commit/07a896862ddf and
> github.com/landley/toybox/commit/0ab021951b40 and so on...
>
> When something is wrong with a toybox command, I want to fix the problem. That
> means collecting data about what the problem IS and figuring out a solution.
> Alas people keep prescribing a specific tool (stdio) before knowing what the
> problem IS, let alone collecting data about it. The tool is the SOURCE of as
> many problems as it solves.
>
> This discussion is about the micromanagement said tool requires. The discussion
> I would LIKE to have is about when output needs to be atomic (ala the xargs
> test), at what granularity (ala the endless less wait example I posted in the
> other thread), and what the performance issues are (grep and yes). But everybody
> is approaching it from the standpoint of "how does stdio do this for you", with
> the obvious reply that it _doesn't_.

(no, i think the argument isn't "your stdio replacement is no good";
it's "you're still using stdio, but hobbling it, _and_ adding
special-case complexity to stuff to work around it". that seems like
the worst of all worlds? not that i want to encourage you down that
path, but ... a toybox stdio replacement would make a lot more sense.)

> > About the terminal: yes, it's gnome. But I'm not sending anything to it.
> > These things are all command-line piping and redirection, which I thought
> > was all in the shell, so how does the terminal figure into it?
>
> Writes to gnome terminal blocking until the display finishes updating are a big
> (and usually unrecognized) source of program slowdowns. Whatever else you're
> writing to can also block until it handles the transaction. Inserting a pipe
> buffer so the producer doesn't have latency spikes inserted by those blocking
> writes (if nothing else pipe buffers are optimized to grab the data and return
> fast) is an easy optimization. Then feeding data into the blocking consumer is
> done by another kernel thread, often on another processor...
>
> > BTW, I enjoyed the maddog anecdote. And IndyCar switched from methanol to
> > an ethanol blend in 2007 for safety.
>
> And yet last week youtube wouldn't stop recommending
> https://www.youtube.com/watch?v=m6jtCnjl5Vw to me until I explicitly told it not
> to recommend that video again, and in an incognito window it was the second hit
> searching for "methanol" just now.
>
> I mean, I see the appeal:
>
> https://www.youtube.com/watch?v=fRmElElYtnQ
> https://www.youtube.com/watch?v=BLAi6mggIA4
>
> But "Ah. This again." It is NOT my job to get the greentech people and the
> racecar drivers to talk to each other about whether you can be poisoned by skin
> contact or fumes rather than just drinking it (due to the delayed organ damage,
> are their chronic effects...) and if I did I'd expect pointing fingers instead
> of cooperation...
>
> Rob
>
> P.S. I miss early busybox development where I was surrounded by a community of
> people smarter than me and there was always somebody who knew the answer and I
> seldom had to explain anything other people didn't know. You'd think outgrowing
> that context would mean I'd be able to reliably answer my own questions, but
> mostly it means knowing 37 ways it previously DIDN'T work, and wondering if
> things have changed behind my back so some of those didn'ts no longer apply, or
> if I'm just missing something obvious...

(careful, or you'll have the ML folks coming out of the woodwork when
they hear you want "do what i mean" buffering :-) )

> _______________________________________________
> Toybox mailing list
> Toybox at lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net