[Toybox] PM: code style, was: Re: New Subscriber

[Rob Landley]

On 02/06/2012 09:31 AM, Frank Bergmann wrote:
> Hello,
> 
> On Mon, Feb 06, 2012 at 06:29:02AM -0600, Rob Landley wrote:
>> That the real world has complex output formats, or that id's output
>> looks like that?
> 
> ... that id's output should look like that (regarding POSIX). I know the
> differences between POSIX and i.e. GNU only as described in man-page
> sections 2 and 3. :-)

I'm happy to not implement gnu extensions unless somebody's actually
using them, but if it's in susv4 I need to document deviances and
explain why.  I.E. that needs a good _excuse_ not to implement stuff.

(I've been reading the bionic source a bit, I sympathize with not
implementing internationalization, and blocking C++ makes me smile, but
I'm still going to need to supplement it a bit.)

>> I fail to see how that's an improvement.
> 
> - small code size
> - faster execution due to reduced overhead
> - often reduced stack size
> - sometimes reduce amount of syscalls
> 
> We should not start a "printf-flame-war".

I totally sympathize with your goals, and I'm all for pushing the limits
and asking "what can we get away with not doing".  I do that all the
time, I regularly have people point out things I missed, and I
constantly need to reevaluate stuff.

But in this case, I plan to continue using the printf() family because
doing without it wouldn't actually simplify the code, I'd just wind up
writing something just as bad.

C and string handling are not a good mix.  String handling is the thing
C is weakest at, because string handling actually turns out to be a hard
problem to get right at the hardware level.

> I just wanted to talk about some
> experiences I've made (and Fefe and some others, too).
> IMHO printf() is actually a wonder of existence because it breaks with
> Doug McIlroy's great recommendation. ;-)

Yeah, but after 40 years of being grandfathered in, it's still useful
enough to stick around.

> [...]
> 
> Guessing the number of syscalls a libcall produces at runtime should not
> be done in a "released package" but while development.

I run things under strace rather a lot, but even if it boiled down to
doing a for() loop around write(1, &char, 1) I'd tell you to fix your
libc rather than  change what I was doing.

> Remember your links
> and the article about the tumb programmer? :-) I don't know what printf()
> is actually doing on c-lib abc or c-lib xyz. To get a little bit more
> control there are libcalls like setvbuf() which I rarely found in code
> using stdio out in the wild. Funny thing, isn't it? ;-)

I'm not that interested in micromanaging something that most likely
stays in L1 cache either way.

>> And that's a "performance over simplicity" mis-optimization.  I only
>> care how many system calls it is when it's a hot path, and the ascii
>> FILE structure has a built-in buffer to batch that stuff up a bit.
> 
> Of course, taking printf as example. But you should always keep it in mind.
> The worst thing I've seen was a trace of rrd-update on a Debian system.
> More than 300 Syscalls due to dynamic loading and searching paths.

That's the worst you've seen?  Never run strace on gcc.  Certainly not
right after eating:

  http://landley.net/notes-2006.html#03-11-2006

(This is why I wrote ccwrap.c in aboriginal linux.  The pathological
path logic in gcc is not salvageable, and piling more crap on top
ever-higher in hopes the bad foundations will go away is how we got
Windows and microkernels.)

> After
> that one or two fadvise/madvise-calls and then the write() itself. I
> wonder if this binary was ever measured. :-)
> 
>> Well, for one thing the above functions aren't in the standard C
>> library, and the ones I'm using are.
> [...]
>> the return value for failure".  My largest divergence there is ditching
>> getopt(), but that's because my new infrastructure does all the work for
>> each command before it even runs, and getopt() required the command to
>> have a switch statement.
> 
> This is the thing I was talking / asking about: Which functions are
> provided by the internal lib?

See lib/lib.h

I brought up get_optflags() in response to printf() because they're
conceptually similar: both bundle a lot of commonly used functionality
together into one place that does it right.  Open-coding printf strikes
me as no different than having a for(;;) loop in every command to parse
command line stuff.  It's only a win if you never have to do it more
than once.

> Do you have any plans to use a special
> and license-compatible library like libowfat (this is one I remember) or
> do you write your own basic functions (like I do for some years)?

I write my own basic functions.  I'd only use an external library if I
was willing to fork it, merge a copy in, and maintain it myself.  Which
is almost guaranteed to be infrastructure in search of a user.

Minimal system bootstrapping is theoretically four things:

A) kernel
B) libc
C) POSIX command line
D) compiler

None of those should depend on anything outside that group.  Way back
when I was hoping for "linux, uClibc, busybox, tinycc" for the four.
That didn't pan out, but I'm still working on keeping it simple.

Environmental dependencies are a form of complexity we're bad at
tracking, and which most people forget about, but that doesn't mean
they're any less important.  (I've got code in sort that avoids pulling
in libm, which is supplied by libc, because it wasn't necessary in that
case.)

>> With your calls above, presumably you have a reason for not using
>> "strcat()" instead.  Your buffer has a length but I don't know what it
>> is.  Does it automatically flush itself or is there an implicit "now
>> write all this out" that you didn't include, and if so is filling the
>> buffer an error or an auto-flush that's going to do an implicit system
>> call anyway?
> 
> er... actually I don't know what fnord's buffer routines do. IMHO you
> specify the size of the buffer (comparing setvbuf) and you also got a
> buffer_flush (comparing fflush).
> Writing something like cat you should use a buffer and libcalls managing
> it.

Or you could do toys/cat.c using the global toybuf[4096] which is part
of the bss and only gets its page faulted in if it's actually dirtied.
(Modulo alignment considerations I haven't bothered about.)

> Writing something like id you may use a small linebuffer within your
> main function. It should be measured.

I put toybuf[] in so every command has access to a global 4k buffer they
don't have to malloc (on top of the DEFINE_GLOBALS()), so I avoid some
heap walks that way.

I note I'm not happy with the lib/dirtree.c stuff (which is the only
current lib/* thing that actualy depends on toybuf, everything else
leaves it for the command to use) because it's got a 4k PATH_MAX
limitation, which the linux kernel doesn't currently have. But at the
same time, it made the code noticeably simpler, and the correct fix is
to use openat() and similar anyway.  I think I blogged about this...

>> What's handling write failures, both from the other end prematurely
>> hanging up and from "sigstop" causing short writes, possibly zero
>> length, which you have to check errno to distinguish the sort write from
>> EOF?  And if you don't get the short writes correct then suddenly "tar c
>> blah | gzip | ssh" suddenly corrupts the tarball if you ctrl-z and then
>> fg the pipeline...
> 
> I know how to handle these cases if it should be necessary but I don't
> know how printf will react - even in glibc! Nice question, I should
> investigate now. :-)

I already did.  See lib/lib.c function xprintf().

> (BTW - that's one reason why I re-implmented some basic functions for
> myself.)

Or you could read the spec and see what behavior is _required_ from a
conformant implementation. Back when the c99 committee was doing its
thing they posted drafts of the spec for public review and then demanded
money for the final version, so everybody kept the last draft version
and called it good. I have a copy on my website, and it says:

  http://landley.net/c99-draft.html#7.19.6

> The fprintf function returns the number of characters transmitted, or
> a negative value if an output or encoding error occurred.
...
> The printf function is equivalent to fprintf with the argument stdout
> interposed before the arguments to printf.

(The man 3 printf page says it too, but that documents what glibc did
rather than what the standard requires.)

If there is a standard requiring behavior, I choose the right to require
conformance with that standard in the environments I build in.  I'm
doing doing one of those insane ./configure stages with 8000 tests to
try to work around every possible kind of broken out there (which can't
be done).  I'm letting the build break and telling you to fix your
environment.

I may tolerate specific environments that diverge from that standard
(I.E. bionic, and maybe macosx), but in that case I'm humoring the bugs
of a specific implementation while requiring everything else to conform
to the darn standard.

I do not allow UNKNOWN environments to violate the standard and still
expect to work.  So if your argument is "this printf() is horrible" then
don't use it, and if your argument is "some unknown printf() might be
horrible someday" I can honestly say it's not my problem.

As I think I said in design.html, I'm replying on c99, posix-2008, and
LP64.  (If I wasn't clear enough there tell me and I'll go fix it.)

>> have tried to document most of it.  I'm not sure what this "I don't like
>> printf, let me open-code every occurrence of it" gets you.  Why not
>> write a better printf?
> 
> That was one of the questions you may find between my lines. ;-)

Because the c99 standard supplies one for me and I use rather a lot of
it, including the printf("% *ld", len, val) stuff in df which was one of
the first commands I wrote for toybox.  (Actually I started it for
busybox but left that project before it was finished, so never submitted
it.)

> Do you have one or a license-compatible "foreign" implementation?
> Is it worth to be done?

I'm not here to reimplement c99 unless I'm writing a C library or a
compiler.  Toybox is neither.

The environments that are in front of me are uClibc (which I've patched
before and can push patches for upstream), glibc (which I'm building
against in the first pass), and bionic (which I expect to supplement
heavily and we'll see if Google wants patches or if I need a
libtrionic.c or something).

>> You mean like the xblah() wrappers so I avoid testing return codes, or
>> the way xsmprintf() is doing mostly the same thing asprintf() is but
>> that's one of the gnu/dammit extensions that I didn't want to rely on?
> 
> Yes, some kind of. Another extensions which I often use is stpcpy (shame
> on me).

Never heard of it.  I've got a strlcpy() but everybody does since
strncpy() isn't guaranteed to null terminate the output.)

Huh, apparently it's not an extension, it's in SUSv4:

  http://pubs.opengroup.org/onlinepubs/9699919799/functions/stpcpy.html

That's really cool.  Thanks.  I wonder where that's needed in lib/* or
toys/* (or main.c), and I wonder if uClibc or bionic have caught up yet?

> This leads to the question which interface standard should be
> used? POSIX? 1996 or 2001? What about functions which are "deprecated"
> (i.e. gethostbyname) or "rejected" (i.e. clearenv)?

Did you read http://landley.net/toybox/design.html yet?  Do I need to
fluff that out a bit?

(At a quick glance I see it says SUSv3 because when I wrote it, SUSv4
wasn't out yet.  Yeah, could use some updating.  And explaining how
POSIX-2008 and SUSv4 and the Open Group Base Specifications Issue 7 are
all the same darn thing just like ANSI/ISO C is the same standard
approved by two standards bodies...)

>> But really, "minimal number of syscalls" is speed to me, not simplicity.
>> Hello world is simple code, if the libc it's linked against chooses to
>> do a separate syscall for every character of output that's libc's
>> problem. If performance sucks I'll look into it, but I expect some
>> variation from libc to libc and if it sucks _fix_your_libc_ or use a
>> better one.
> 
> OK. Some foreign lib bashing in source code comments is always nice to
> read. ;-)

I note that I pay no attention to dietlibc, and am unlikely to start.

>> I have a vague academic interest in MacOS X but I've already got mdev
> 
> It's the only OS I know where you can switch off ptrace! ;-)
> Its BSD-API is sometimes frustrating.

Interesting,there doesn't seem to be a kernel CONFIG symbol for ptrace...

There should be.  Possibly only with CONFIG_EMBEDDED, but there really
should be a way to configure that out...

>> upshifted from int 80 to SYSENTER to save a couple clock cycles, and
> 
> Don't talk about Slowlaris. They NEED threads because only threads are
> actually lightweight as processes in Linux are. But even with sysenter
> which is IMHO used on every linux system today

It's an x86 instruction and we're targeting ARM android devices, so "not
so much".  Although I believe android has something similar.

> you should still keep in
> mind that syscalls are the most expensive calls.

Over on my other major hobby project, Aboriginal Linux, I analyzed the
performance of ./configure and determined that statically linking
busybox would speed up ./configure by around 20% due to the lower page
translation overhead from eliminating the self-modifying-code aspect of
dynamic linking that forced executable pages to be retranslated.

Trust me: I know how to profile stuff, and how to understand the
performance of things, and I am not worried about the overhead of a
dozen or so nonblocking syscalls where we're faulting in maybe three
cache lines from L2 to L1.

Implement, _then_ optimize.  I'd like to worry about where the
bottlenecks actually _are_ rather than borrowing trouble.

>> http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html
>>
>> That has gettid() at 100 nanoseconds or less...
> 
> Wow! This is really low. But out in the wild you found crappy software
> (IMHO more than 90%) which does hundreds of time()-calls a second.

Which is why they changed it so gettimeofday() can just read an atomic
variable out of the vsyscall page:

  http://www.win.tue.nl/~aeb/linux/lk/lk-4.html

(They were discussing it, anyway.  I don't remember if that change went
through...)

Yeah, fixing the code not to do that is a good thign too, but for things
like video playback software you need short waits and to be woken up
often because it's what the program _does_...

> And
> sometimes when Nagios warns about raising contextswitches on a specific
> host you detect software with an strace you would never dream of - even if
> you dream of Freddy Kruger all night long. ;-)

I'm totally aware that most existing userspace software is crap:

  http://lwn.net/Articles/192214/

However, you can go too far in over-optimizing (which is actually part
of the _reason_ so much of it's crap).

Software needs to remain flexible.  If you don't understand why it's
doing things the way it's doing, you can't change it later and it gets
brittle and you do workarounds instead of fixes and it all falls apart.

Sometimes, doing a nice tight implementation winds up with knots that
are hard to undo.  (Both of the current patch and sort implementations
are in that category, for example.)  Sometimes, that's the local peak.
Making it simpler or easier to understand results in significantly more
code or signficantly more runtime.  (I still try anyway, and at least
comment the house-of-cards bit.)

But complexity is a cost, I want to make sure we get the best bang for
the bug, and figuring out what simple _is_ can be really hard.  (Simple
is elegant.  I can't always come up with an elegant solution.  Doesn't
mean I don't go back and polish the the dorodango until it gleams, and
sometimes I _do_ figure out how to make an actual gemstone out of it.)

> Rob, you're making me hungry. I should co/clone the current toybox stuff.
> :-)

The more eyeballs the merrier.  I'm stretched a bit thin at present, but
trying to keep up... :)

> Frank

Rob