[Aboriginal] My dreams about Aboriginal

[Rob Landley]

On 04/08/2011 03:52 AM, Alessio Igor Bogani wrote:
> Rob,
> 
> 2011/4/8 Rob Landley <rlandley at parallels.com>:
> [...]
>>> 1) Switch to klibc (already hosted on kernel.org)
>>
>> I looked at klibc, but Peter Anvin went crazy a few years ago and
>> inserted perl into the build of every single project he had anything to
> [...]
> 
> Is Perl the only obstacle?

Perl is the big one, but the fact it's not a real C library is another.

It's not intended to be a fully functional libc capable of having things
like busybox built against it, it's basically a stub C library with just
enough functionality to build a couple small initramfs utilities and
then switch-root to a real filesystem.  (It wraps the system calls and
provides printf.)

Maybe its' design goals migrated, but last time I tested it you couldn't
build anything real against it.

>>> 2) Migrate toybox to kernel.org and use it for replace busybox
>>
>> I spent years working on that, but haven't had time for it in a while.
>> You're welcome to grab it and run with it if you'd like.  Personally I
>> much prefer the design of toybox to busybox, but the practical effect is
>> I can't stand to work on busybox anymore.
> 
> Did you mean "I can't stand to work on toybox anymore", didn't you?

No, I mean I can't stand to work on busybox anymore.

Working on toybox is "fun but pointless" since I'm up against a team of
a dozen programmers with a 10 year headstart.  I like the toybox
infrastructure, but I'm reinventing the wheel.  This has already been
done, and the existing one is "good enough" for the vast majority of the
audience.

Working on busybox is a slog through ugly code.  Back under Erik it was
reasonably tidy, and I spent a lot of time just cleaning stuff up.
Denys cares more about getting stuff to work today than he does about
making it simple or easy to read.  It's a nest of tangled #ifdefs, bits
of it have sprouted windows support, the main() function for the
multiplexer is no longer at the top level but buried down in libbb
somewhere for no apparent reason, it's got abominations like the
applet_main() prototype right before the applet definition (why the HELL
would anyone think repeating the same information twice in a row would
be an improvement, it annotates things with magic macros like
MAIN_EXTERNALLY_VISIBLE and UNUSED_PARAM that are simultaneously subtle
and pointless (I.E. most developers don't understand what they do and
what would happen if they weren't there, and what they do isn't worth
doing anyway), it not only still has multiple shell implementations but
it's developed forked implementations of all sorts of OTHER applets
(editors/patch.c, editors/patch_bbox.c, editors/patch_toybox.c,
networking/nc.c, networking/nc_bloaty.c, coreutils/od.c,
coreutils/od_bloaty.c...)

I know Denys is attempting to satisfy the needs of a large diverse
userbase, but Alan Cox once told me "a maintainer's job is to say no",
and he's not doing nearly enough of that.  To me the point of busybox
was always that it was simple and clean.  Being small was a _part_ of
being simple and clean.  Being simple made it small, and it was small
because it was simple.  (Being fast or full-featured wasn't THERE when I
started.  When I first picked busybox up it didn't WORK for most of what
I tried, and the performance of what it did have utterly sucked, yet I
was attracted to it anyway.)  Now it has to be all things to all people,
and I am apparently not one of those people anymore.

I use it because it's way better than the gnu bloatware, and because I
spent many years MAKING IT WORK and wanted to follow through on a dev
environment built around it.  (Which was a crazy goal when I started,
and required me becoming project maintainer for a while to make happen,
but now works fine.  Modulo occasionally having to give 'em a kick, ala
aboriginal hg 1318, 1180, 996...)

Note that I have infrastructure in aboriginal linux
(more/record-commands.sh) which figures out the list of commands that
are actually _used_ by the build.  I've even documented it, see:

  http://landley.net/aboriginal/FAQ.html#debug_logging

And look for "using the command logging wrapper".

And yes, this can be used natively in the resulting filesystem to
determine which commands a given package build uses, although I haven't
automated installing it yet.  Possibly I should.

I was using this during toybox development to give me a todo list.

  http://landley.net/notes-2008.html#23-01-2008

And later:

  http://landley.net/notes-2008.html#04-07-2008

>>> 3) Integrate linux-tiny patchset to provide the best and light Linux
>>> embedded system (easy to study, fast to build)
>>
>> Um, which linux-tiny patchset?  Matt Mackall stopped doing it a while
>> back, and I thought the CELF guys were feeding theirs upstream?
> 
> AFAIK The CELF (or should I say LF?) would want to revamp effort on that.

CELF was great when it was Tim Bird backed by Sony.  (Everything
interesting actually gets done by somebody _specific_.  Organizations
are abstractions, they can't implement.)

Now it's a faceless corporate entity that was founded to supply Linus
Torvalds a steady vendor-neutral paycheck but wasn't satisfied to do
JUST that, and suffered years of feature creep until it merged with a
standards body in an explosion of bureaucracy.

The Linux Foundation is to Linux what AOL was to the Internet in the
1990's.  The truly clueless aren't comfortable unless they think
somebody owns it and is responsible for it, and they want to pay that
entity money to give them the illusion of control.

>>> I'm not interested in Ubuntu bootstrap at all. In fact I'm not
>>> interested in any big distro bootstrap. What I meant:
>>>
>>> sudo apt-get install aboriginal-linux-{arm,mips,powerpc,x86}
>>
>> I ship prebuilt binary tarballs under GPLv2.  You can download and
> [...]
>> Basically you extract a system image tarball and a cross compiler
>> tarball, add the cross compiler's "bin" directory to your $PATH, cd into
>> the system image and run dev-environment.sh.  Then play.
> 
> The important point here is:
> Do you will ever provide binary tarball for every combinations of
> arch, compiler, bootstrap and so on?

I tried to document this in:

  http://landley.net/aboriginal/downloads/binaries/README

I provide one instance of each type of tarball per $ARCH.  I'm only
using one compiler (binutils, gcc, kernel headers, uclibc), although it
comes in cross and native versions.  (Cross is from i686 which runs on
x86_64.  You can build other types using the build scripts, although I
doubt the gcc guys have really tested cross compiling sparc from an arm
host.)

I don't do bootloaders, installing it on real hardware is Not My
Problem.  (Outside the scope of the project.)

I have the hw- overlay mechanism if you want to create your own
derivative targets (see sources/targets/hw-tct-hammer for an example,
it's mostly good for creating a second system image from the base root
filesystem using a different kernel).

Also, you can replace my bootscript using SIMPLE_ROOT_OVERLAY (it's a
configuration environment variable, which means the "config" file at the
top level of the build scripts documents its usage.)

> My major concerns are about Aboriginal project is the total absence of
> a clear vision:

Sigh.  I have a clear vision (since 2008-ish, anyway), and I've tried to
articulate it.  Apparently I haven't managed.  One such attempt was:

  http://landley.net/aboriginal/about.html

> What is Aboriginal made for? What is goot at? What are
> the key features? What is the future direction? What are the best
> things that project care to provide (speed vs size, simplicity vs
> flexibility)? What are the differences with Yocto, OE, buildroot,
> crosstool, PTXdist, ELDK, Scratchbox, emdebian and so on?

The point of Aboriginal Linux is to get a minimal native Linux
development environment up and running on a new platform, and then get
out of your way.  The motto "we cross compile so you don't have to"
means once you have that native development environment you should be
able to build anything else you need natively on your target hardware
(or under an emulator for that target).

You can still use a cross compiler via distcc to speed up native builds
by leveraging fast cheap x86 hardware, but this is purely optional and
your package builds still act like fully native package builds when you
do this; you never have to mess with cross compiler prefixes or the many
flavors of host/target confusion just because you're using distcc.

The simplest Linux system I've ever managed to come up with that was
capable of building itself under itself is seven packages (linux,
uClibc, binutils, gcc, busybox, bash, and make), and we cross compile
those seven packages, then package them together with a kernel and a
boot script into a bootable system image configured for use with QEMU.

And then we STOP.  We are not a distro, we don't care what final
packages you build to actually deploy on your target.  We just give you
a build environment, what you build with it is none of our business.
(Making sure you CAN build any package you try to is important,
otherwise we have a bug we need to fix.  But package selection and
package management and such?  Beyond the scope of hte project.  We
provide build automation infrastructure in the control-image.hdc stuff,
and we're working to provide example images that bootstrap popular
distros.  But those are just examples, and that collection of examples
may spin off into a separate project.)

> I'm not asking you to reply on all these answers. I would want know
> only a _clear_ vision about Aboriginal project so I can decide _alone_
> if Aboriginal is good for me.
> 
> I really damned like how you have made Aboriginal but, frankly, a
> project like this without a clear vision drive me to looking for
> something else.

As far as I'm concerned, a design has to know what it DOESN'T do as much
as what it does.  "Where are the edges of this project?"

This project is divided into a series of orthogonal layers, each of
which tries hard not to leak into or depend unnecessarily on the others.
 (As described in presentation.pdf in the downloads directory.)  These
layers are implemented in separate scripts, which build.sh calls in
order, and are described here:

  http://landley.net/aboriginal/downloads/README

(Notice: the README in downloads describes the build scripts, I.E. the
source code.  The README in downloads/binaries describes the binary
tarballs, I.E. the output of the build.)

Here are the stages bulid.sh calls:

  download.sh - download the source packages for the rest of the build

  host-tools.sh - build host binaries used by the rest of the build

  simple-cross-compiler.sh - build a "stage 1" compiler

  cross-compiler.sh - build a "stage 2" compiler

  native-compiler.sh - build a ("stage 2") native compiler

  simple-root-filesystem.sh - just enough to boot to a shell prompt

  root-filesystem.sh - combine native-compiler + simple-root-filesystem

  system-image.sh - package root filesystem into something qemu can boot
                    (also builds kernel configured for use with qemu).

Those layers are orthogonal and most of them are optional.  If you
already have a "packages" directory with source tarballs in it, all
download.sh does is confirm the checksums.  You don't have to use THIS
way to download the packages.

You can skip host-tools.sh if you've set your host up sufficiently
carefully.  You don't have to use THIS way to set up the host environment.

If you already have a cross compiler (ala crosstool-ng) you can use that
instead of building one with this script.  You don't have to use THIS
cross compiler.

You never need cross-compiler.sh (it's there to create a portable one I
can upload to the website, the rest of the build doesn't use it and by
default build.sh won't call it unless you set a config variable).

If you just want a shell prompt with no dev environment skip
native-compiler.sh (setting NO_NATIVE_COMPILER tells build.sh to do
this).  Or if you have a different native compiler (perhaps from
crosstool-ng) you can use that one instead of building one with this script.

If you just want to build toolchains and not a root filesystem skip
simple-root-filesystem.sh.  Or if you have a different root filesystem
you want to use, put it in "build/simple-root-filesystem-$ARCH" and the
build will try to use it.

The root-filesystem.sh script just combines the previous two output
directories, it's just two instances of "cp" with an rm -rf and a call
to strip.

And if you want to chroot into your root filesystem or install it on
target yourself, system-image.sh is irrelevant.  Or you can use it to
package up some other root filesystem (just copy it to
"build/root-filesystem-$ARCH", this script doesn't particularly care
what's in there) and stick a kernel on it.

That's the path from point A to point B.  "I have a small collection of
bash scripts" to "I have a system image qemu can boot".  That's pretty
much what Aboriginal Linux was designed to do.  That's the edge of the
project, at which point we start answering a lot of questions with "not
our problem".

Now beyond that, we do have the control images.  The script

  more/build-control-images.sh

Builds all of those.  Basically when a system image boots up, its init
script checks to see if /mnt/init exists, and if so it transfers control
to that at the end instead of running a shell prompt.  (Actualy it waits
3 seconds for you to press a key first, in which case you get your shell
prompt anyway but now with the control image mounted.)

In each system image there are three shell script that can boot qemu in
different ways:

  run-emulator.sh - root filesystem on /dev/hda, mounted on /
  dev-environment.sh - 2 gig ext2 image on /dev/hdb, mounted on /home
  native-build.sh - control image on /dev/hdc, mounted on /mnt

Note that native-build.sh is a wrapper around dev-environment.sh, which
is a wrapper around run-emulator.sh.  Each one adds some configuration
then calls the next simpler one.

This is generic build automation infrastructure: aboriginal linux
provides a way to run builds without human intervention (from a cron job
or some such), but doesn't care what you're building.  You can supply
any control image you like, it gets mounted on /dev/hdc and its "init"
program gets run.  It can contain anything, and do anything, you like.

On the one hand, maintaining lots of example control images is none of
Aboriginal Linux's business.  On the other hand, building things like
Linux From Scratch under the resulting system is squeezing a lot of bugs
out of the build system (especially uClibc).  So for the moment it's
part of the smae project, it may be broken into a separate project layer
on.  It IS definitely an orthogonal layer you can ignore, though.

So providing a way to run control images: part of aboriginal linux.
Providing specific control images: not really part of aboriginal linux
(in the long term, anyway).  Worth doing in some context, being used as
debugging test cases right now, and a certain number of EXAMPLE control
images are excusable to show you how to make 'em.

But once we have a critical mass of control images, I'll probably split
that part out into a separate project and move them out of Aboriginal
Linux.  I try to make them not depend on aboriginal linux (you should be
able to run 'em under any arbitrary distro's dev environment, not that
I've tested this), and I definitely don't make aboriginal linux ever
depend on anything in a control image.

> Ciao,
> Alessio

Did that help explain?

Rob

 1302276497.0