[Toybox] [PATCH] toysh: fix -Wuse-after-free

[Rob Landley]

On 3/21/24 23:59, Oliver Webb wrote:
> On Thursday, March 21st, 2024 at 22:45, Rob Landley <rob at landley.net> wrote:
>> On 3/17/24 14:52, Oliver Webb wrote:
>> > Same here, I can remember the posix commands.
>> 
>> Can you? I still have to check some from time to time, and the definition of
>> whether "tar" is a posix command or not is outright eldrich bordering on quantum.
> 
> I can certainly remember them better then the LSB commands. Most of the time I can
> remember if a command is in posix, which is what matters when trying to find it usually.

Congratulations?

>> Collapsing the directories together when the last command is
>> promoted (or deleted) out of pending might make sense,
> 
> What would happen when a new command shows up and we need to evaluate it then?

Presumably once caught up there wouldn't usually be a dozen of them submitted
the same month, so I wouldn't fall far enough behind to need a dedicated waiting
room.

> Or glibc does a new release and yet another thing breaks we need to demote and
> re-promote eventually?

I don't de-promote commands because glibc does something stupid each new
release. That's just normal gnu/braindamage:

https://github.com/landley/toybox/issues/450
https://github.com/landley/toybox/pull/364
https://github.com/landley/toybox/issues/362

I de-promoted a command since last release because I rewrite lib/password.c in a
way that broke stuff and didn't want people poking me about it, which was me
being lazy/whelmed. Not having the option to do that is fine too, and would have
made that stay higher on the todo list. (I could also have "default n" it
without moving it, I do that locally all the time when in-progress changes break
stuff. The difference this time was I'd checked IN the stuff that broke a
command, and didn't want to revert it.)

>> I also note I think I've figured out how to replace kconfig: I can just make a
>> list that scrolls up and down with a highlighted entry you hit space on, handle
>> help text, search, exit/save, resolve selects and depends and have "menus" be a
>> label line with its contents nested two spaces further to the right.
> 
> [Some paragraphs bikeshedding about kconfig use to be here, may they rest in a text file
> until we get around to doing the kconfig rewrite]

Technically a project's maintainer explaining upcoming design issues he actually
plans to implement isn't "bikeshedding".

Bikeshedding is vaguely related to the Dunning-Kruger effect, in which the
question "how hard can it be?" requiring some expertise to actually answer gets
people in trouble.

Cyril Parkinson is mostly known for Parkinson's Law (work expands to fill
available time) but he also came up with the bike shed example, where a
committee approving plans for a nuclear reactor defers to the experts enough
that at least its budget approval gets discussed quickly, but a committee
approving plans for a bike shed will argue far longer about every detail because
they think they could do it themselves and have strongly held opinions.

Everybody has an opinion on building the bike shed, and thinks their opinion is
equally valid as everyone else's with no deference to authority, experience, or
expertise. But the thing about a committee approving plans is they STARTED with
a viable plan for the thing, which they then ignore because they know better.

If you feel like I'm "bikeshedding" about a kconfig replacement when I was
involved in https://lkml.indiana.edu/hypermail/linux/kernel/0202.1/2037.html and
argued at length with Roman Zippel about https://lwn.net/Articles/160497/ and
dug rather a lot through busybox's fork of it back around
https://git.busybox.net/busybox/log/scripts/config?id=7a43bd07e64e and already
implemented scroll up/down/left right list logic like I'm describing in the
"top" command... I think we have a different definition of the term.

>> > A possible solution is to...
>> 
>> ...
>> 
>> > Then again...
>> 
>> I need to stop checking email every time I sit down at my laptop, because
>> bikeshedding can eat an endless amount of time and I've got other stuff to do.
>> 
>> For one thing, I promised to look at
>> https://github.com/landley/toybox/issues/486 tonight.
> 
> Sorry for getting in the way of that, the technical discussion about it was
> interesting enough to me to respond to. Recently found something to run off to
> and do while still benefiting toybox, so I'll stop bikeshedding about stuff like this.

I'm complaining about my own insufficient time management skills, I'm not trying
to discourage people from taking an interest in the project.

I do find "why is it like this" easier to deal with than "let me tell you what
you did wrong, and/or provide the obvious solution I'm sure you never thought
about over the past 10 years, without needing to know the backstory".

You can't get up to speed without learning stuff, and I should do MORE support
for that. (Talks, youtube videos, design.html and faq.html and so on...)

To be honest, I still haven't come back up to speed since the pandemic. (I found
the last half-decade somewhat stressful.) I'm not sure how much of it's me and
how much is "this project has more users now so there are more incoming
interruptions and less sustained focus on backlog".)

*shrug* Working on it. I need to go faster...

>> P.S. I never mind people asking "what's the status of" or "why didn't you", and
>> when people say they don't personally like stuff they can never actually be
>> WRONG about that. But suggesting what I "should" do is... tiring.
> 
> I never thought about it in a "what should I make the maintainer do?" way, more in a
> "what benefits the project and makes it more usable?" way, considering possible solutions
> to issues.

I do a lot of design work I don't publish, partly the classic "publishing
negative results" problem:

  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6945059/

And partly that the effort put into writing up plans for external consumption is
NOT effort put into implementing those plans. (Ala "here's how I could simplify
the kconfig design so a rewrite is less work, although that still doesn't
entirely address depends/selects resolution with "SYMBOL && (SYMBOL||SYMBOL)". I
blog, and go into quick stream of consciousness asides on the mailing list, but
that's not remotely complete coverage.)

So when you go over design space I've already gone over, there's a bit of
bringing an outside contributor up to speed with what I've already done to have
context for the discussion, so we agree on where the cutting edge of this
problem IS.

It's not that I haven't thought about this, it's that I haven't FINISHED it.
Which I already feel bad about...

Sometimes I do solicit advice here. I'll hit an uphill bit on an issue and write
it up here and SAY that I don't immediately know what to do. Last week's
http://lists.landley.net/pipermail/toybox-landley.net/2024-March/030163.html for
example.

But when Elliott (who has been here since 2015) pokes me about a design decision
I made in 2012, replying with "you could have done X" instead of "why didn't you
do X" is... a lack of nuance given the context.

I'm likely to respond with infodumps either way, which is not an attempt to
dissuade you from taking an interest. It's basically those writeups I didn't do
before, which brings us back around to my old "the prototype and the fan club"
talk, which I _really_ should redo at some point because flourish had recording
issues both times I spoke there.

Sigh, lemme hit the old damaged .flv file with ffmpeg to convert it to mp4,
upload that, and here's the outline I gave the talk from:

https://landley.net/talks/prototypeandfanclub.mp4
https://landley.net/talks/flourish-2010.txt

If nothing else, the conversion seems to have made the buzzing less bad. (They
had the microphone plugged in wrong.)

The talk was a sequel to the Cathedral and the Bazaar, which compared
distributed collaboration like linux to centralized development models like the
gnu project, but didn't really go into how to DO distributed collaboration. My
talk was about how open source development actually worked (circa 2010).

The part at the beginning they didn't record (flourish didn't start recording
for half a minute after I started talking) was basically the thesis statement:
Somebody creates and publishes a prototype that demonstrates an idea, and a fan
club forms around that prototype. The fan club generates feedback to the
original author, whether ideas and modifications or simply encouragement, and
the maintainer integrates some subset of that into the prototype producing the
next release.

The video can probably take it from there. Or a quick summary of the talk...

The prototype MUST come first. A fan club without a prototype will fail: the
prototype is a condensation nuclei for more code to grow its intricate snowflake
around. without one you get endless discussion and abandoned forks. All
prototypes start out crappy, and talking about what you're GOING to do someday
means any given attempt to reify the discussion in a prototype is not "the code
we're all discussing" (just somebody's pet project) and doesn't match everyone's
(conflicting) expectations. After weeks or months of building ivory towers, a
community can lose the ability to ever make anything real.

The fan club needs to form when the codebase is small and simple, both so older
members can bring newer members up to speed (both technology and culture) and so
the code does what the users want (both because the design was shaped by its
users, and because the users changed what they want to what they're familiar
with). Dumping 20 year old abandonware on the internet provides a vertical cliff
face for newbies to scale, with nobody at the top, that is only coincidentally a
good fit for any given new user's needs, and somebody deciding to rip a
subsystem out and rewrite it is dishonoring the traditions (that's not a real
$THINGY).

A prototype without a fan club will usually advance slowly, but that's not
unusual. Building a fan club can take time, people wander by and don't stick.
Linux inherited the comp.os.minix community (I blogged about that recently:
https://landley.net/notes-2023.html#07-12-2023 ) which is why it took off
running, quickly grew beyond what one person could do, and had such successful
word of mouth. (The fans also promote the project, and teach new people to use
it.) Sometimes a small project will surge forward when it gets a "killer app"
style user, such as busybox spending its first 5 years in obscurity before being
widely promoted both by the original linksys routers (millions and millions
served) and the tomsrtbt single floppy linux rescue disk (a skeleton key to open
a dead PC).

Open source software was not the first to do this, publishers were doing it a
century ago, and we can learn about how this works from them. The maintainer of
an open source project is like the editor of a magazine, accepting unsolicited
submissions into a "slush pile", about which Sturgeon's Law (90% of everything
is crap) was coined. Reading through the slush pile means rejecting 90% of it,
polishing up a few good ones, stitching them together into the next edition of
the magazine, and then publishing. Rinse repeat. tiny fanzines can grow into
publishing empires, or stay small with a loyal fanbase for decades.

The editor can write their own material, but it's only a small fraction of what
the community can produce. A novelist self-publishing their own newsletter can't
produce a tenth the output of Baen or Tor or Random Penguin. Even if the
community is just suggesting topics for a columnist to cover ("ask a manager",
etc), the feedback is useful and also keeps the fanbase enaged. A successful
magazine will quickly grow to the point where the editor hasn't got TIME to do a
lot of writing. (The really big ones will have an Executive Editor with
domain-specific sub-editors underneath them, the way Linus formalized his
Lieutenants in 2002.)

An editor mostly has veto power, I.E. the ability to say "no". Putting out a
call for stories about a given topic doesn't mean they get anything useful. But
if fans WANT their story published in the Official Cannon, then there's
motivation to overcome rejection by resubmitting an improved version, and this
bounce back negotiation lets the editor shape incoming submissions without
having to personally rewrite each one.

I'm sure I covered a bunch of other stuff, but it was a while ago and I haven't
rewatched my old talk in years...

> Sorry if it seems like I'm trying to order you around. I also stated in the
> email that reorganizing commands would be unfeasible, I talked about a
> possible way it could've been done "better", but I didn't say you should do anything.

"Here's how you could have done this better, no don't get up" is not criticism.
Got it.

Rob