[Aboriginal] ulfs-bootstrap, shrinking LFS.

[Rob Landley]

On 04/01/2011 09:04 AM, David Seikel wrote:
> On Thu, 31 Mar 2011 07:44:12 -0500 Rob Landley <rob at landley.net> wrote:
> 
>> On 03/31/2011 03:08 AM, David Seikel wrote:
>>> ulfs is my effort to shrink LFS down, mostly to suit my own
>>> purposes, by using the stuff that already exists in Aboriginal
>>> linux.
>>
>> I need better documentation, I'm just not sure where to put it.
> 
> I saw your tags thingy, but it did not match up with what I ended up
> with.  Or saw how your tags should be used.
>  
>> However, that's the sort of thing that could easily open "the distro
>> can of worms" which I'm trying very hard not to do with this
>> project.  Of course if somebody ELSE wanted to take that over... :)
> 
> I'm not volunteering, I got enough work to do.  Only dabbling in the
> stuff I need to, and contributing back what I can.

This is why I want to do distro bootstrapping.  Maintaining package
repositories is something _other_ people do, and do it well, and I want
to leverage that work, not duplicate it.

Even LFS (and BLFS) are pretty much "repositories somebody else
maintains".  They're as simple as you can get doing that (fixed package
selection), but I don't have to work out what depends on what and how to
configure invoke the builds.  (I just have to debug it. :)

>>> no need to replace the tool chain we already got,
>>
>> In the LFS bootstrap script I never bothered to build the new
>> toolchain. I built its prerequisites, but skipped gcc and binutils.
>> (I _do_ rebuilt make, for no readily apparent reason.)
> 
> Yes, I built make to, then removed it after the first draft of my email,
> which is why it was still on my list in the email.  I forgot to remove
> it from the list.
> 
>> Bash I'm installing a newer version of.  The old 2.04b is easy to
>> build and less than 1/3 of the size of the current monstrosity, and I
>> really want to replace it all with busybox hush.  But you need
>> current bash in order to run gentoo's portage, it uses stuff like ~=
>> that old bash hasn't got, and depends on the subtly changed quoting
>> rules...
> 
> Gentoo I'm not interested in.  LFS and BLFS I am interested in, but
> only for those packages I need for the current project.  A later 
> project that will grow out of this will need more from BLFS.  I still
> wont be interested in gentoo.

Debian/Ubuntu and Fedora/Pointy Hair Linux are the two biggest
repositories.  Gentoo is probably third, although Arch Linux is creeping
up on it.

In theory Gentoo's emphasis on building from source made it a good
candidate for bootstrapping systems on arbitrary targets.  In practice
the way they set up their repository individual annotates every single
package with every single platform it's been tested to run on.  (To add
a new platform, you have to touch every package ebuild file in the whole
tree.)

That's insanely stupid for native builds on arbitrary targets you've
already bootstrapped on.  In order to go forward, I need to REDESIGN THE
PORTAGE TREE and push the changes upstream.

>> Readline there's a bsd-licensed replacement I'd prefer to use out of
>> sheer spite for the FSF, and idealy busybox should export that
>> functionality anyway.
> 
> I'd like to look at that readline replacement, though I'd be just as
> happy to not need it for my current project.

There's a couple of them, but this one looks reasonable at first glance:

  https://github.com/antirez/linenoise

>> Don't get me started on autoconf/automake.
> 
> Preaching to the converted, I was glad to get rid of it.
> 
>> pkg-config is its own horror (you do NOT improve on the rpm vs dpkg vs
>> ebuild situation by ADDING YET ANOTHER... grumble) and either not
>> having it installed or setting PKG_CONFIG=/bin/true is my preferred
>> choice, butI got tired of fighting with some package builds.
> 
> The UI library I want to use makes heavy use of pkg-config, so for me
> it stays.

You can't solve having too many layers of indirection by adding another
layer of indirection.

>>> First problem, iana-etc needs the real gawk, it uses fancy options
>>> that the busybox awk does not support.  So put gawk back in.
>>
>> Poke the busybox gawk maintainer about that.
> 
> I may just cut iana-etc down to the few ports I actually need for this
> project.  The next project will need it for sure.  I've not actually
> looked at what iana-etc uses gawk for, but it does smell of overkill to
> just create two text files.  Why not just ship the two text files?

I know.  (Haven't looked at it since october, I think there was some
kind of hammer I could hit it with to make it stop...)

>>> Second problem, autoconf stops part way through asking the user a
>>> question when using busybox instead of coreutils.  A quick
>>> investigation seems to point to the busybox mv command needing to
>>> know if it's OK to overwrite a file, when the coreutils one just
>>> overwrites it without asking.  There may in fact be some deeper
>>> root cause for this problem, I did not look deeper.  See next step
>>> for why.
>>
>> I'd poke busybox about that.
> 
> Well, it became moot the moment I removed autoconf from the mix.  Also
> no point poking busybox unless I bother to dig deeper, which I have no
> intention of doing for a package I'm happy I don't have to include
> anyway.

See yesterday's grumble about restarting toybox.  Not much point in
doing it unless I can _replace_ busybox, but if I change my dev strategy
to porting cleaned-up versions of busybox applets to toybox commands...

Don't have time at the moment, though.

>> Not that I really have time to do so, but eh.  If I copied hush and
>> started cleaning that up and extending it, I might get a reasonable
>> bash replacement THAT way...
> 
> That would be good.

That by _itself_ is probably as big a project as the rest of toybox.

>>> Grub was left off for the same reason it was left out of
>>> lfs-bootstrap.
>>
>> I've never really addressed bootloaders, or installing stuff on real
>> hardware.  It's too system-specific, and QEMU's -kernel option allows
>> me to handwave past the issue entirely for my use cases. :)
> 
> Though in the end I will use grub legacy as it's now known in this
> project.  The real hardware will have to be booted somehow.

Yup, but maintaining a hardware list (how to install and boot on the
zillions of boards out there) is beyond the scope of the Aboriginal
Linux project the same way maintaining a package repositor is.

It would be great if somebody did this, but it's an orthogonal layer.

>>> Bash and readline where removed.  We already have an older bash as
>>> part of aboriginal linux.
>>
>> Which doesn't work with portage, so I needed newer bash as part of the
>> gentoo bootstrap anyway.
> 
> Already mentioned I don't need to care for gentoo.  Not like anyone is
> using LFS to bootstrap gentoo, and this is the LFS bootstrap we are
> talking about.

Actually I was pondering using LFS to bootstrap gentoo (since I'd
already done the work, might as well leverage it), but yeah.  Not in
your prerequisite list.

I designed the LFS bootstrap to be as modular as I could.  You can run
each package build individually:

  /mnt/build-one-package.sh packagename

Fun for debugging.  You can also:

  source /mnt/functions.sh; do_in_chroot /home/chrootdir /bin/ash

Which is clunky and I need to make a separate /usr/src/do-in-chroot.sh
as part of the base script.  (One that autodetects whether the target
directory exists, and just redoes the bind mounts if so.)

(Note my general rule of thumb for underscores vs dashes: do_in_chroot
is a shell function, do-in-chroot.sh is a shell script.  Another thing I
try to be consistent about, but haven't really documented anywhere.  I'm
not the only one doing that, though...)

>>> Readline is only in LFS as a dependency of bash.
>>> Time will tell if this is a bad move.
>>
>> I was erring on the side of building more packages rather than less on
>> the theory that if I could get it to build, anything that actually did
>> need it could then use it, and driving more cybermen across the board
>> would find more digits of pi, or something like that.  ("Used to drive
>> sheep across minefields.  The principle's the same.")
> 
> True.  On the other hand, one of the goals of this project of mine is
> to provide a government testing lab with something for them to audit,
> including the development system that we use to build it all, so that
> the lab can build it to.  The less they have to deal with, the better
> we will be.  So at a later stage I'll very carefully pull out as much
> stuff as I can to pare it down to the bare minimum.  Simple
> minimisations are good enough for this early in the project.  Get it to
> work first, then optimise away more crap later.

I hope the orthogonal layers of the build system are reasonable to audit
separately. :)

> It's a personal goal of mine that the development system we hand to the
> audit lab is just the device itself, and a hard drive they plug into
> the device that has the development environment on it.  This is why
> Aboriginal Linux's ability to compile itself is important to me.

Important to me too.  You haven't got a real build environment if it
doesn't self-bootstrap.

>> yacc/bison is something mathematicians like and which I personally
>> find conceptually unpleasant.
> 
> There might be something using bison that is important to me later.
> I'll know for sure later.

You can always add more packages later.

>> On the Hexagon I built the x11 client libraries and managed to build
>> xeyes and xboard and rxvt and a couple other things.  There was some
>> hacking around uClibc but they were using 0.9.30 rather than 0.9.31 so
>> lots of it was backporting stuff...
> 
> Trying to avoid X this time around, that's for the next project.  For
> this project I think the kernels framebuffer is good enough, and helps
> a lot with the "minimal code for the audit lab to worry about" goal
> nicely.  No need to worry about adding 50 X packages when all I need is
> a framebuffer that already comes with the kernel.

It was more like 35.

(And they all pretty much built the same way, too.  For some obscure
reason the x.org guys decided to break the one big xfree86 packages into
three dozen way to small packages.  I'm guessing they got pestered about
modularity once too often, and were trying to prove a point by making it
virtually unusable in the other direction.)

Rob

 1301724917.0