[Aboriginal] Fwd: Re: Aboriginal. Wow! and Thanks!

[Rob Landley]

-------- Original Message --------
Subject: Re: Aboriginal. Wow! and Thanks!
Date: Sat, 18 Jun 2011 19:43:12 -0500
From: Rob Landley <rob at landley.net>
To: Paul Kramer <kramerica at me.com>

On 06/15/2011 05:06 PM, Paul Kramer wrote:
>> You might also want to look at the the Linux From Scratch entry in 
>> sources/control-images (the prebuilt binary of which is in 
>> downloads/binaries/control-images).  That's my build automation 
>> infrastructure, with Linux From Scratch 6.7 used as an example.
> 
> I'll definitely look at this.

I'm poking at this now trying to get the packages to build on more
platforms, and LFS had a new release I need to upgrade to, plus I need
to do a BLFS that does at least X11...

>>> While at Sun in the 90's and at 3 startups, I'd tear out entire 
>>> build infrastructure, restructure trees and put in
>>> implementations that sort-a look like this:
>>> 
>>> http://www.qbalsoftware.com/Site/Blog/Entries/2011/5/9_Makefiles_Generic_Flexible_and_Simple_Part-2.html

I
>>> 
went to you archive and read part 1, which is a big long makefile
that's very explicit about what it does but is still "make, only more so".

Reading the second now, I note the pull quote at the top:

> "I just cannot see how that can be true.  Being both generic, 
> flexible and being simple are usually at odds,

No they're not.  The more simple something is the more flexible and
generic it is.  A knife is flexible and generic, a cookie cutter won't
spread butter on your bread or slice meat for sandwiches.  A hammer is
flexible and generic, a nail gun won't pair with a chisel or seal a
paint can lid.

Software is the same way, that's the whole POINT of the Unix philosophy.
 Simple commands speaking a common language (ascii text) connected by pipes.

I'm sure I've written down my make rant before, but I'm not sure
_where_.  (Possibly in my blog, but it goes back years and "make" is a
bad word to search for...)

I know some of it's in the compiler BOF video from OLS:

  http://landley.net/talks

But that's just one topic in an hour-long talk...

Ok, Make stuff, lemme see...

First, make has grown a lot of cruft ala automake, and pointless crap
like libtool which should do NOTHING on Linux.  The point of libtool is
to make non-ELF systems behave passably like ELF systems.  Linux has
been ELF since 1995, meaning on Linux Libtool should be a NOP.  Yet it
REGULARLY breaks builds, especially in cross compile environments.

(Note: attempting to do nothing and FAILING is a trademark move of the
Free Software Foundation.)

Alright, let's look at Make.  The superficial objection is that Python
gets bashed all the time for significant whitespace, but make which
REQUIRES TABS (not spaces) gets grandfathered in.  A less superficial
objection mixing declarative and imperative code in the same file is
mind-bending and evil.

But the FUNDAMENTAL objection of make is that its initial reason for
existing no longer applies.  The point of makefiles is to work out what
needs to be rebuilt, I.E. dependency tracking is its reason for
existing.  That's the whole point of these weird inter-related targets
that aren't in any obvious order.  That was the central design idea.

This stopped working DECADES ago, which is why we have all these ".dep"
files.  We tell the compiler to work this out for us, we do NOT
hand-encode them in the makefiles.

More to the point, Moore's Law has advanced 40 years since then, and
"make all" is the vast majority of all make invocations now.  When you
download a source package, you "configure; make; make install".  And
half the time when DEVELOPERS build stuff we do a "make clean" or "git
clean -fdx && git checkout -f" or similar because I for one don't trust
the makefile dependencies not to introduce weird subtle bugs.

Basically any project that's big enough that make takes forever to bulid
should be broken into directories, and each directory should have its
own "make all" producing its .o or .a file or whatever it makes, and all
this dependency tracking should GO AWAY.  Rebuild with directory
granularity at best.  For small projects, just "make all".

The OTHER point is that if you can feed all the files to gcc at once,
the optimizer can do a better job.  So make is fighting modern
_compilers_, which I talked about in the above mentioned video...

>>> This is how easy I think build systems can be. Of course, what
>>> I'd like to see is 'Make' revisited for the multi-variant
>>> problems we face
>> 
>> My approach to make, autoconf, and automake is a quote from Aliens:
>> "I say we take off and nuke the entire site from orbit: only way to
>> be sure."
> 
> I'm really thinking seriously about solving this for a minimal
> system, like the ones you have created. I just cannot think of
> spending another 10 years with the existing stuff.

In my toybox project, my makefile was just a simple wrapper to give
people the command line API they expected, and then it called shell
scripts in the "make" subdirectory:

  http://landley.net/hg/toybox
  http://landley.net/code/toybox

Alas, I stopped having time for that project years ago.  Busybox is
"good enough", has a ten year headstart and about 4 years of my own work
in there, and is "good enough" that producing an incrementally better
wheel won't get people to leave what they've got.  *shrug*

Also, my build system was a quick and dirty first attempt, and didn't do
parallelism because I had a UP laptop when I started.  (Although really
I should be able to go "cc -o walrus source/*.c -j 4" and let the
COMPILER work out how to parallelize it.  Otherwise I'm ditching the
-fwhole-tree optimizations I ranted about in that BOF video...)

> I'll write something up and send it to you, see what you think

Let me know when you do this, a quick glance at part 2 of your blog
stuff was talking about writing improved makefiles, and to me this is
like writing clearer COBOL programs.  I don't want to mitigate the
problem, I want to eliminate it.

> I can make the current situation much simpler using currently
> technologies, but I think some new technologies should be created to
> take it one big giant step forward

Eric Raymond really likes http://www.scons.org/ but being written in
Python is likely to prevent it from being universally adopted _ever_.

I'm all for high-level languages, but as you see from my system
bootstrapping exercise (Aboriginal Linux) building a low-level system
should not rely on them.  Perl dependencies snuck into the kernel build
(fairly recently, 2.6.25) and I removed them.  I use uClibc rather than
glibc primarily because glibc requires perl to build.  (I'd at least
offer glibc as an option otherwise, I keep meaning to go take another
look at that...)

Rob