[Toybox] sed -e '$a\'

[Rob Landley]

On 03/26/2016 01:36 AM, Andy Chu wrote:
>> The problem to solve here is that Debian is currently broken when using
>> toybox. That's a real problem that needs to be fixed, but there's more
>> than one way to fix it. Debian introduced this bug within the past few
>> years and maybe we can convince them to fix it on their end, that's one
>> possible fix.
> 
> What's the goal of toybox with respect to Debian?  I looked through
> the design and roadmap pages and didn't see much about Debian
> specifically.

I take it you haven't read http://landley.net/aboriginal/about.html
(specficially http://landley.net/aboriginal/about.html#hairball ).

As for the backstory, way back when Red Hat had 50% market share in
Linux workstations, until the new people they brought onboard to run
their IPO showed them how to eat Sun's lunch. I wrote up the whole thing
here:

http://landley.net/notes-2012.html#14-03-2012

The upshot is Ubuntu replaced Sun as the primary workstation vendor more
Linux devs use than anything else, and Ubuntu is based on Debian. So
it's more that I can't really _ignore_ debian. It is, currently, still
quite relevant.

> Do you mean that when you replace all of Debian's builtin commands
> (coreutils, etc.) with toybox, things should work?

Eventually, yes.

> What is the test
> for working -- all packages, or required Debian packages, etc.?  Or
> building Debian packages?

Ideally, building an entire Debian distro from source under aboriginal
linux, as described at http://landley.net/aboriginal/about.html#hairball

I'm sad that http://www.emdebian.org/ died but "because we haven't got
the storage" is not the ONLY reason for a simple system. And not just
security auditing either. If you can't reproduce your results under
laboratory conditions, you're not doing science. We have an awful lot of
programming alchemy we have to outgrow at some point. How does the next
generation learn how anything works if it's built on a layer of magic
created by dead people?

The fact that the truly excellent article on institutional memory at
https://web.archive.org/web/20120105150051/http://wrttn.in/04af1a is
only available on archive.org now is a damning indictment of our entire
industry, if you ask me. (And officially "ironic", given what the topic
was.)

I'm working on it...

> Aboriginal Linux is its own distro,

No, it's explicitly NOT a distro. I covered that in my old 2008
presentation, slides 78-90:

https://speakerdeck.com/landley/developing-for-non-x86-targets-using-qemu?slide=78

It's basically 7 mandatory packages and a couple optional ones, and I'm
NOT adding more. I actually have plans to get those 7 packages down to
4, although doing so involves qcc which I haven't got time to START yet.

> so I thought that was the primary use case of toybox,

It's the primary smoke testing environment of toybox. Android and
chromeos are already using it in production environments. I'm not sure
about Tizen.

> along with Android and building distros like the
> ones that are based on busybox (which don't look much like Debian
> AFAIK.)

My original goal when I started all this nonsense circa 2002 or so was
Knoppix. Linux From Scratch was over 100 megs but tomsrtbt was 1.7 megs,
and I thought if I could save Knoppix 100 megs of space on its CD they
could do WONDERS with it. The goalposts hae shifted a bit over the
years, but oddly enough knoppix was debian based too. :)

>> Also, even if I patch stuff when _I_ build it, but I want _other_ people
>> to be able to build the vanilla versions, without needing my patches. If
>> buildroot or gentoo or something has toybox commands in the $PATH during
>> its build, I want them to _work_.
> 
> OK so I guess you're aiming for a much higher level of compatibility
> than I thought.  Is this true for busybox?

It was when I worked on it. That's how I got involved with busybox in
the first place. I did a writeup of the history once upon a time:

http://landley.net/aboriginal/history.html

> I'm pretty sure that if
> you replace the vanilla Debian or Gentoo tools with busybox, TONS of
> things will break, like building packages and running daemons and all
> that... but I guess you have a better intuition here than me.

It wasn't intuition, I was _testing_ it.

http://landley.net/notes-2008.html#02-07-2008
http://landley.net/notes-2008.html#04-07-2008

But Busybox wasn't done when I left, and since it's wandered off in
different directions. Toybox has that as a goal, but toybox was
monthballed for years until I figured out a new purpose for it circa
2012, and currently I'm busy dealing with missing commands in the roadmap.

Some things, debian and gentoo and such could fix. Other things are
reasonable for me to fix.

> Are you only talking about build time, and excluding runtime?  i.e.
> using toybox to bootstrap the builds of other distros?

They're separate use cases. I want one to work and I want the other to
be a reasonable option.

>> Sometimes I respond with "ew, no" even when evaluating the standard.
>> (I'm not implementing EBCDIC in dd. It's just not happening.) But again,
>> case by case basis, judgement calls, and I can be argued out of my
>> position. (Elliott's done that several times. He represents the final
>> word for Android, and has a billion seats behind him. I did NOT want to
>> open the selinux can of worms, but... it's in there now.)
> 
> The sed thing made me go "ew" but I guess that ship has sailed.

Even the posix sed is full of ew.

And "making me go ew" is not the same as "convincing me not to do it". I
try to minimize the ew, but you've still gotta get the job done in the
best way you can figure out how to do. The ends don't justify the means,
but the means don't justify the ends either.

The one I'm really not looking forward to is awk. I have to learn THAT
language to implement it, and there are HORRORS written in awk in the
binutils and gcc autoconf stuff, that I have to make work.

Back in busybox somebody else was maintaining awk and I could make puppy
eyes at him to fix stuff once I'd isolated a good test case. For toybox,
apparently I've got to do it...

> Though I would judge special cases by the size of the diff, and it
> doesn't fair all that badly in that respect.

I judge them by a bunch of criteria. Size of the diff is one (and part
of that diff was "while I was there" cleanup of stuff. Someday I should
go back and rename all my variable so the entire file isn't a giant pile
of references to Roger Zelanzy's Amber series. But YOU try repeating
"pattern" 40 times before throwing in a "logrus", and then it just
spirals from there.)

The amount of in jokes in my code is actually reasonably correlated to
the amount of sleep deprivation I was under at the time. (Like now, it's
coming up on 3am here.) The "struct {forever[];} strawberry fields[];"
nonsense in ps is a milder example. I edited most of that back _out_
before checking it in. Also that song actually has very little in the
way of lyrics so I COULDN'T pull a lot of variable names from it if I
wanted to.

In the case of cp, there _are_ a half-dozen potential replacement
behaviors and not all of them are currently covered. (The default
behavior of "install" is not something cp can do without that flag.)
That said, I haven't implemented the new flag yet either. :)

(But yeah, I've got your -n fix open in a window. Breaking down and
updating the netbook to 14.04 tonight though.)

>> But anyway, that's part of my judgement on whether or not to implement
>> extensions and corner cases. Is this documentable cleanly?
> 
> I think that is a good litmus test.  Thanks for the detailed explanation.
> 
>> Yes. But somebody is using it to do something specific. It would be nice
>> if they stopped, and if people want to send patches to debian to that
>> effect I'm all for it. But are they the ONLY user of this "trick"?
>>
>> At the moment, what I can fix right now is toybox. If I can find
>> something that qualifies as a "fix".
> 
> Yeah I guess I am wondering where the line is drawn...

Alas: in crayon. Eyeballed.

> do you care
> more about Debian than other distros, or is anything in the Linux
> world up for grabs?  (I already figure that you don't care about BSD
> compatibility)

A) Somebody brought be a bug report. (Waiting for people to complain
means listening when they do.)

B) Ubuntu is a lot of developers, although Ubuntu is also really stupid
and could easily go the way of Red Hat if they keep it up. (pointing
/bin/sh to dash, mir, unity, upstart, basing launchpad on bazaar...
Really NOT following ubuntu's technical lead is pretty justified these
days...)

C) It wasn't that hard to fix. It was a lot of thinking, but a smallish
resulting patch.

>> (P.S. If debian didn't build with bsd's sed, would either debian or bsd
>> care? If not, bsd sed's behavior does not give a good indication of what
>> a Linux sed's behavior should be.)
> 
> Yes, an interesting point.  And I agree about the monolithic design of
> BSDs and adoption.  Linux is ... "postmodern", but that appears to be
> the only way the job gets done.  Git to me is very much the same
> aesthetic -- it doesn't make sense globally, but it works for diverse
> groups of people with their own idiosyncratic use cases, and thus has
> high adoption.

The move from centralized to distributed source control is a compelling
technological shift. Too bad we wound with git as the standard in the
new area, but given that the alternative was mercurial it's not too
surprising.

git has high adoption because kernel developers everywhere had to
install it to work with linux, and that gave it a critical mass of users
who taught it to other people.

Also, while mercurial was superficially easier to use:

A) it sucked in python as an environmental dependency, and the move from
python 2 to python 3 is a bit like the move from gpl 2 to gpl 3,
fragmenting the community into incompatible warring camps and making a
lot of old-timers walk. (It doesn't seem to be turning off newbies
nearly as much, though, so maybe python will survive this if then NEVER
do a python 4.0.)

Building git from source under aboriginal linux wasn't hugely fun, but I
made it work in 2011 and got a static binary that ran ok without too
many prerequisite packages:

  http://landley.net/notes-2011.html#26-10-2011

B) Mercurial's development community had strong opinions about how its
tool should be used and woe unto you if you tried to do something with
it they didn't approve of.

I was a longtime mercurial supporter, but bowed to the inevitable
eventually.

> Andy

Rob

 1458979255.0