[Aboriginal] ulfs-bootstrap, shrinking LFS.

[Rob Landley]

On 03/31/2011 03:08 AM, David Seikel wrote:
> ulfs is my effort to shrink LFS down, mostly to suit my own purposes,
> by using the stuff that already exists in Aboriginal linux.

I need better documentation, I'm just not sure where to put it.

If you look in sources/control-images/lfs-bootstrap/mnt/package-list
you'll see I've annotated them with various tags I was thinking of using
to let you switch off various package builds.  (Pretty much with "grep -v".)

I've got tags for busybox (this is already provided by busybox),
development (only needed if you want a development system),
internationaliation (only needed if you want to support
internationalization, and I can link you to several of my old blog
entries on my struggles to get that to work right under uClibc), and
pointless (never needed by anybody for any reason, only exists because
the FSF is insane).

However, that's the sort of thing that could easily open "the distro can
of worms" which I'm trying very hard not to do with this project.  Of
course if somebody ELSE wanted to take that over... :)

I admit to being sort of tempted to separate the control-images stuff
into its own project.  (This is why I split
sources/download_functions.sh out into its own file, because the
control-image generator needs a copy of that.)

Really, they should be able to bootstrap on top of any arbitrary native
development environment.

> So, no
> need for most of the usual gnu tools when we have busybox,

Agreed.  (If the busybox tools are deficient, fix busybox.)

> no need to replace the tool chain we already got,

In the LFS bootstrap script I never bothered to build the new toolchain.
 I built its prerequisites, but skipped gcc and binutils.  (I _do_
rebuilt make, for no readily apparent reason.)

My annotations aren't quite complete: gmp, mpfr, and mpc are parts of
gcc that the FSF broke off into separate packages in some misguided
attempt to make gcc look smaller or something.  In 4.2 they aren't
separate packages.

Bash I'm installing a newer version of.  The old 2.04b is easy to build
and less than 1/3 of the size of the current monstrosity, and I really
want to replace it all with busybox hush.  But you need current bash in
order to run gentoo's portage, it uses stuff like ~= that old bash
hasn't got, and depends on the subtly changed quoting rules...

m4, bison, readline, autoconf, automake, flex, and pkg-config aren't
provided by the existing dev environment.  m4, bison, and flex are
probably necessary.  Readline there's a bsd-licensed replacement I'd
prefer to use out of sheer spite for the FSF, and idealy busybox should
export that functionality anyway.  Don't get me started on
autoconf/automake.

pkg-config is its own horror (you do NOT improve on the rpm vs dpkg vs
ebuild situation by ADDING YET ANOTHER... grumble) and either not having
it installed or setting PKG_CONFIG=/bin/true is my preferred choice,
butI got tired of fighting with some package builds.

> and see how far we can get using
> uClibc.  The result should be able to compile stuff from BLFS, though
> I'll only be testing the stuff from BLFS that I actually need when I
> need it.

I built rather large chunks of BLFS when I ported aboriginal linux to
the Qualcomm Hexagon for my day job last year.  (Alas, it was based on a
toolchain and ginormous kernel patch that we hadn't shipped, so I could
never quite merge it.  They claim that SOMEDAY it'll make it past the
lawyers and out into the world...)

But the point is, lots of BLFS work just fine under uClibc.  And I'm
hoping to get more under uClibc-1.0.  (In fact, once we've got NTPL and
TLS I'd like to treat aborigininal Linux as the /tools system and build
a new glibc based system natively, just so I don't ever have to worry
about cross compiling glibc.  It requires perl to build.  You may barf now.)

> Mostly this is by removing packages, if the removed package turns out
> to be something you need, just put it back again.
> 
> First thing I did was remove gmp, mpfr, and mpc.  They are complex
> maths packages that are only in LFS coz the latest gcc needs them.  We
> are not using a gcc that needs them, we already have an older one that
> works.  The original lfs-bootstrap had already removed the building of
> the toolchain coz aboriginal linux already supplies one.  For the same
> reason make was left out, we already got that.

You have to remove make from the package-list (it's currently in there),
but yes, you shouldn't need it.

> Next thing I did was to leave out packages that busybox could be used
> for.  This step removed sed, util-linux-ng, e2fsprogs, coreutils, gawk,
> procps, grep, inetutils, bzip2, diffutils, findutils, gzip, iproute2,
> kbd, less, module-init-tools, patch, psmisc, shadow, sysklogd,
> sysvinit, tar, udev, and vim.  Util-linux-ng includes a lot of stuff
> that busybox does not, we shall see if it's really needed.

Generally if it's really needed, it's in busybox.

> First problem, iana-etc needs the real gawk, it uses fancy options that
> the busybox awk does not support.  So put gawk back in.

Poke the busybox gawk maintainer about that.

> Second problem, autoconf stops part way through asking the user a
> question when using busybox instead of coreutils.  A quick
> investigation seems to point to the busybox mv command needing to know
> if it's OK to overwrite a file, when the coreutils one just overwrites
> it without asking.  There may in fact be some deeper root cause for
> this problem, I did not look deeper.  See next step for why.

I'd poke busybox about that.

I also just got a fan letter about toybox (from the guy doing
http://www.makelinux.com/emb/fastboot/omap ) that makes me vaguely
wonder if I should start poking at that again.

Something I could do is port busybox applets one at a time to toybox
commands.  I tried pushing toybox infrastructure upstream into busybox
and that sort of worked, but not really all that well.  I still can't
stomach the current state of toybox, working on that bloated pile of
#ifdefs is no fun and trying to clean it up is like trying to bail out
the middle of the ocean: any space I clear immediately fills up with
cruft again.

Not that I really have time to do so, but eh.  If I copied hush and
started cleaning that up and extending it, I might get a reasonable bash
replacement THAT way...

> Next step is to remove autofoo and friends.  The general purpose of
> autofoo is to let developers create source tarballs that users can
> compile on any old unix system.  The target systems don't need autofoo
> installed to allow such tarballs to compile.  So autofoo itself is a
> development tool, and LFS is not supposed to be a development system,
> but a way to get a linux system up and running which can be used to
> install more packages.  So, autofoo is not really needed in an LFS
> system.  Removing it is good, and lets me get rid of perl, which is
> even better.  B-)

autoconf/make is horrible.  I have an entire rant about the current
state of makefiles, which I'll spare you of just now.

> Grub was left off for the same reason it was left out of lfs-bootstrap.

I've never really addressed bootloaders, or installing stuff on real
hardware.  It's too system-specific, and QEMU's -kernel option allows me
to handwave past the issue entirely for my use cases. :)

> Bash and readline where removed.  We already have an older bash as part
> of aboriginal linux.

Which doesn't work with portage, so I needed newer bash as part of the
gentoo bootstrap anyway.

> Readline is only in LFS as a dependency of bash.
> Time will tell if this is a bad move.

I was erring on the side of building more packages rather than less on
the theory that if I could get it to build, anything that actually did
need it could then use it, and driving more cybermen across the board
would find more digits of pi, or something like that.  ("Used to drive
sheep across minefields.  The principle's the same.")

> Libtool and texinfo where left out as being pointless development
> tools.

Pointless in general, really.  Libtool exists to make non-elf systems
behave like elf, meaning on Linux (which has been ELF since 1995) it
serves no purpose and does nothing.  The fact that it fails to do
nothing correctly (and thus regularly breaks builds) is a trademark move
of the FSF.

> I had considered leaving out bison, flex, and m4, but they
> ended up being needed to build those bits of LFS that remain, so left
> them in.

Yup.  m4 is vaguely cool, flex is a sort of sideways grep (instead of
"find all occurrences of this one regex" it's "find the next occurrence
of any of the regexes in this list, and tell me which one and where").

yacc/bison is something mathematicians like and which I personally find
conceptually unpleasant.

> This left the ulfs-bootstrap package-list looking like this -
> 
> setup
> zlib
> m4              #development
> pkg-config      #development
> ncurses
> gettext-stub    #development internationalization
> gawk            #busybox
> iana-etc
> bison           #development
> gdbm
> file
> flex            #development
> groff
> make            #development
> man-db
> 
> A much smaller system that should still allow compiling of stuff from
> BLFS.

Good luck. :)

> As a quick sanity check, I installed the one thing from BLFS that I
> can't live without - midnight commander.  It has glib as a dependency,
> so that's the first thing I tried to install.  It failed due to a lack
> of iconv.  The base config for uClibc has the iconv parts commented
> out, uncommenting them and rebuilding fixed up glib.  Midnight
> commander installed fine after that.  And it actually worked.  B-)

I turned on iconv when I was building BLFS for Qualcomm, but that was on
my work machine and I didn't take any of the qualcomm code home with me,
so I have to reproduce all that stuff when I do a blfs-bootstrap.  I may
have just installed the gnu libiconv package, I don't remember.  (I
wasn't trying to do it RIGHT, I was trying to get the code ready to
demo.  Unfortunately, the demo was delayed repeatedly by toolchain
problems, and although we held it right before I left and it was
successful, my contract expired about a week after that and they didn't
have the budget to renew it, and wouldn't get the budget renewal for at
least two more months, so I found another job rather than waiting around.)

I note I haven't even enabled devtmpfs in the builds yet.  My todo list
runneth over. :)

> I guess iconv was left out as it's just bloat that is not needed if you
> are not doing any internationalization, for the same reason
> gettext-stub is used instead of gettext in lfs-bootstrap.  I think
> gentoo have a stub version of iconv they are working on, but the above
> was easy to do and worked for me.  I'm not doing internationalization
> either.

My 2010 blog (http://landley.net/notes-2010.html) has a lot of hits for
iconv.

> My next step will be to install the dependencies of the GUI library I
> want to use for this contract.  A lot of those dependencies are in BLFS,
> so those will go in first using the BLFS recipes.  This should help to
> shake out any more issues with ulfs.  If people are interested, I'll
> ship the results to Rob later.  Though the package-list above is the
> important bit, you can just copy the rest from lfs-bootstrap.

I lean towards making the lfs-bootstrap build more configurable by using
the annotations with grep -v.  It may need more/better annotations.  How
to specify this configuration is a UI issue I haven't come up with a
good answer for yet.  Environment variable I guess,
KERNEL_EXTRA="LFS_SKIP=busybox,development" maybe?  That could be done
at build time without respinning the control image, which is kind of
important considering how long that sucker takes to build...

> Feel free to let me know if there are other issues with building BLFS
> with this ulfs-bootstrap.

On the Hexagon I built the x11 client libraries and managed to build
xeyes and xboard and rxvt and a couple other things.  There was some
hacking around uClibc but they were using 0.9.30 rather than 0.9.31 so
lots of it was backporting stuff...

So all this was made to work once, I just didn't get permission to open
source it.

Rob

 1301575452.0