[Toybox] Weird stdout buffering effects

[Rob Landley]

On 10/23/23 18:13, Ray Gardner wrote:
> Rob, thanks for the info, I learned some stuff about pipes from it.
> 
> I also looked up setvbuf(), and the standard says it may not be used after
> a previous successful call to setvbuf(). That makes my tests below kinda
> moot.
> 
> But still, using _IOLBF and then _IOFBF seems to work with glibc and musl
> to get fast output.
> 
> So, to get fast output from any toy, I think there should be an option to
> do no call to setvbuf() at all. What is the reason for defaulting to no
> buffering? If it's a must, maybe need TOYFLAG_DEFAULTBUF or something.

Didn't we just address these same questions in the other thread?

http://lists.landley.net/pipermail/toybox-landley.net/2023-October/029843.html

My personal history left me distrusting stdio buffering, especially in glibc.
I'm trying not to let that affect my judgement here, but it's hard.

Here's a presentation Denys Vlasenko (the busybox maintainer i handed off to)
gave at the Embedded Linux Conference in 2010 where he talks about a bug that
had been hounding busybox for years, where if you statically linked with
--gc-sections then stdio wouldn't flush output at exit (and just discard
anything buffered), because the atexit() blob the library was shoehorning in
without an actually call to atexit() wasn't labeled in a way that stopped it
from getting discarded as "unused", and the binutils people said glibc was doing
it wrong and the glibc people said binutils was doing it wrong and they spent
more than 5 years pointing fingers at each other rather than fixing it:

slides: https://elinux.org/images/2/2d/ELC2010-gc-sections_Denys_Vlasenko.pdf
video: https://bootlin.com/pub/video/2010/elc/elc2010-vlasenko-dead-code.ogv

(A chronic pet peeve of mine is "these people know this, those people know that,
why is it MY JOB to get them to talk to each other, and then when I do they
fight!" It's exhausting.)

If you're wondering why xexit() does an explicit flush and everything is piped
through xexit() instead of other methods of exiting... that's only part of it.
Another part is that atexit() doesn't have a way of removing things from the
list, nor calling the things on the list but NOT exiting and instead
longjmp()ing back to a recovery thing, so I had to write my own so sh.c could
call MAYFORK stuff...

Anyway, needing to remember to manually fflush(stdout) when doing things like a
"password:" prompt is why I had xprintf() and xputc() do an fflush after every
call, which Elliott took out because it was slowing stuff down making both of
those ferror(stdio) but not do the flush that would see if there WAS an error.
(And you have to check for error otherwise things like "yes" won't exit in
pipelines, but just sit there spinning and eating CPU, and the whole should
pipeline processes be killed by signals thing is a can of worms.)

I very much _want_ to not care about stdio buffering and just let libc do its
thing. Unfortunately the default is inconsistently wrong, and anywhere I try to
shove setting a known one (so it's at least CONSISTENTLY wrong) people complain,
and I'm wisting after just ripping out all use of FILE * and just always using
filehandles because then I'd know what the behavior _is_. (Probably some kind of
lib/lib.c function that takes a null terminated char * array like xexec() uses
and prints it atomically using writev(), falling back to malloc-ing and
memcpy-ing a contiguous copy as necessary. With optional separator string. And
maybe I'd want to move the add_arg() function from sh.c into lib.c so I could
push stuff to such arrays as a stack with the realloc() handled for me...)

I'm not there yet. But I'm starting to think this discussion won't end short of
that.

The last time I talked to the C committee I was asking for a standard way to get
the buffered data out of an input FILE * (specifically asking the number of
bytes I can read from the FILE * without triggering another underlying read()
system call on the file descriptor) so that when I call a child process I can
pass that data on to its file descriptor, the way I'm doing in tar.c line 1091,
and the posix guys said "FILE * is handled by the C standard" and the C guys
went "nothing in our standard ever mentions file descriptors" and once again two
maintainers pointing fingers at each other and refusing to fix stuff:

https://landley.net/notes-2022.html#19-04-2022

(Once again getting people to talk to each other, and they point fingers.)

Luckily, I figured out a different way to mostly avoid triggering the problem
for my shell implementation:

https://landley.net/notes-2023.html#07-01-2023
https://landley.net/notes-2023.html#05-02-2023

But this is ALSO why I wrote my own byte-at-a-time get_line() function back at
the start of toybox development, which was slow but _correct_. Ok, also because
getline() was still a glibc extension until posix-2008:

6769f8eb580a

And then the android guys went "nope, too slow" and I sighed and agreed to use
getline() out of libc rather than writing my own buffered one because the buffer
reading too far ahead is THE fundamental design problem here that I couldn't do
a LESS bad job of. So Elliott started on a get_line() removal quest:

b30674681b9d
4885e8fea8f7
3ead70e503b2

git log 2a5dc105a323a20a and you get like 7 commits removing get_line() from
stuff, continuing at:

2243f6f2ad08
15cbb92dffc8
c23b3ff44948

But there's a fundmental problem when you implement something like wget and read
some lines of input followed by binary data logically handled via sendfile()
which takes a file descriptor. The handoff between the two sucks conceptually.
You cannot pass a FILE * to a child process. You cannot ungetc() a pipe. You can
MSG_PEEK on recv() but writing data into a read source is a security violation
waiting to happen.

Getting this right is _hard_. I preferred avoiding it, but the performance
sucked. "Trust the libc" sadly didn't _work_, and when I try to micromanage it
people ask why I did that.

> I ran some more tests, outside of toybox, with setting stdout mode once,
> or twice in sequence, before writing to stdout. These tests are all with
> redirection to a file.

On glibc, bionic, musl, mac, freebsd and qnx?

> BTW, a few days ago you wrote:
>> Also... adding LINEBUF when the calls are xprintf() and xputc()... those
>> are explicitly supposed to flush. Flushing and checking errors are what
>> those DO.
> 
> But xprintf() does not flush.

It used to! But back during
http://lists.landley.net/pipermail/toybox-landley.net/2019-April/018382.html I
caved and did github.com/landley/toybox/commit/2a1f89e5d941 which of course
broke github.com/landley/toybox/commit/07a896862ddf and
github.com/landley/toybox/commit/0ab021951b40 and so on...

When something is wrong with a toybox command, I want to fix the problem. That
means collecting data about what the problem IS and figuring out a solution.
Alas people keep prescribing a specific tool (stdio) before knowing what the
problem IS, let alone collecting data about it. The tool is the SOURCE of as
many problems as it solves.

This discussion is about the micromanagement said tool requires. The discussion
I would LIKE to have is about when output needs to be atomic (ala the xargs
test), at what granularity (ala the endless less wait example I posted in the
other thread), and what the performance issues are (grep and yes). But everybody
is approaching it from the standpoint of "how does stdio do this for you", with
the obvious reply that it _doesn't_.

> About the terminal: yes, it's gnome. But I'm not sending anything to it.
> These things are all command-line piping and redirection, which I thought
> was all in the shell, so how does the terminal figure into it?

Writes to gnome terminal blocking until the display finishes updating are a big
(and usually unrecognized) source of program slowdowns. Whatever else you're
writing to can also block until it handles the transaction. Inserting a pipe
buffer so the producer doesn't have latency spikes inserted by those blocking
writes (if nothing else pipe buffers are optimized to grab the data and return
fast) is an easy optimization. Then feeding data into the blocking consumer is
done by another kernel thread, often on another processor...

> BTW, I enjoyed the maddog anecdote. And IndyCar switched from methanol to
> an ethanol blend in 2007 for safety.

And yet last week youtube wouldn't stop recommending
https://www.youtube.com/watch?v=m6jtCnjl5Vw to me until I explicitly told it not
to recommend that video again, and in an incognito window it was the second hit
searching for "methanol" just now.

I mean, I see the appeal:

https://www.youtube.com/watch?v=fRmElElYtnQ
https://www.youtube.com/watch?v=BLAi6mggIA4

But "Ah. This again." It is NOT my job to get the greentech people and the
racecar drivers to talk to each other about whether you can be poisoned by skin
contact or fumes rather than just drinking it (due to the delayed organ damage,
are their chronic effects...) and if I did I'd expect pointing fingers instead
of cooperation...

Rob

P.S. I miss early busybox development where I was surrounded by a community of
people smarter than me and there was always somebody who knew the answer and I
seldom had to explain anything other people didn't know. You'd think outgrowing
that context would mean I'd be able to reliably answer my own questions, but
mostly it means knowing 37 ways it previously DIDN'T work, and wondering if
things have changed behind my back so some of those didn'ts no longer apply, or
if I'm just missing something obvious...