[Aboriginal] extending aboriginal -concept idea

[Rob Landley]

On 06/25/2013 07:53:19 PM, scsijon wrote:
> Firstly, may I say I love it and are playing with a couple of basic
> builds at present to get the feeling of what it can do. I am
> considering using it for the basis of a wayland build or even a
> small multi-server system.

Yay!

Sorry I'm behind on email, I normally catch up on weekends but I was at  
a convention last weekend where the power went out for 24 hours, so  
instead I fell further behind. Catching up now, I've apparently made it  
to Tuesday.

> However, I was wondering if there was the ability of extending what
> packages were in a build by using an additional script. Such as, if
> (1)it exists it's added; or (2) by also adding an extra build switch
> to (1) to include it and without the switch it's not.

In the "design goals" section of  
http://landley.net/aboriginal/about.html it does talk about eliminating  
the need for cross compiling. The general idea is building stuff  
natively within a system image.

That said, if you want to add a new build script you can just drop it  
in the top level directory and add it to build.sh. The plumbing works  
like this:

# Suck in common functions and variable definitions
source sources/include.sh || exit 1

# Suck in target-specific data from sources/targets
load_target "$1"

# Find a package "thingy" out of sources/sections.
build_section thingy
build_section otherpackage
build_section blah

# tar up the result into $STAGE_NAME-$ARCH.tar.bz2
create_stage_tarball

Regarding sources/sections: scripts ending in .sh do their own setup  
and teardown (see gcc.sh for example), the "setupfor" function  
snapshots the package source code into sources/temp-$ARCH/packagename  
and changes directory into there, the "cleanup" function does an rm -rf  
on that directory. If you need to build out of tree blank_workdir  
creates a named temporary directory.

The scripts ending in .build have setupfor and cleanup called for them,  
so you can just do the configure make and install. (See make.build for  
example.)

There's a lot more complexity like the "package cache", explained at  
some length in the FAQ, but I try to make it so you can ignore as much  
of the infrastructure as possible.

> concept is something like:
> ---
> for 1- if a builder added to the aboriginal default (location where
> ./build.sh exists) a script called additional.sh its content would be
> added into the standard build.

Ok. I note there's already SIMPLE_ROOT_OVERLAY in config that can point  
to a directory that gets copied into the resulting root filesystem  
verbatim.

> I'm not at this point considering what it should or should not  
> contain and
> what it's structure should be, but something like this for each  
> additional
> package may be sufficient.

I recommend looking at the Linux From Scratch build control image. I  
build additional stuff natively on each target, because cross compiling  
really does suck.

The script more/chroot-splice.sh does native builds in a chroot instead  
of in qemu, which is a bit faster if you're building for i686 or  
x86-64. (That's generally a good first pass test.) As with all of them,  
run it without any arguments to see usage.
What I haven't explicitly documented (probably should) is that you can  
point it at the build directory instead of the sqf file, and it'll  
--bind mount that so live updates you do to the source show up in the  
target.

Something like:

cd ~
wget http://landley.net/hg/control-images
cd control-images
./build.sh lfs-bootstrap
cd aboriginal
more/chroot-splice.sh i686 ../control-images/build/lfs-bootstrap

(and then at that shell prompt)

/mnt/init

The scripts in there are:

1) init - first thing that runs. If the root filesystem is read only  
(which the squashfs under qemu is), copy it into a chroot under /home,  
and chroot into there. Either way, launches run-build-stages.sh next.

2) run-build-stages.sh - iterate through package-list and call  
build-one-package.sh on each entry. There are #tags in there that can  
be used to filter out packages so you don't build them, so if you  
export "FILTER=busybox" it won't build packages that busybox provides  
functionality for.

3) build/* - the build scripts for each individual package.

4) packages/* - the extracted source code for each package.

> name
> source url (complete)
> hash
> error checking for missing pre-requisite packages
> compile scripts (unpack; ./configure;make;make install)
> 
> for 2- including 1- but also adding to your build script something  
> like
> 
>  ./build.sh i486 additional
> 
> to include both the default set AND the additional set

The third argument from build.sh is "restart build from this stage". So  
when I change toybox and want to rebuild that in the target but _not_  
rebuild the cross or native compilers, I can go "./build.sh x86_64  
simple-root-filesystem" and that will do that, root-filesystem,  
root-image, it'll redo linux-kernel if SYSIMAGE_TYPE=initramfs, and  
then system-image.

Probably what you want is a new chunk between cross-compiler and  
simple-root-filesystem that does:

if [ -e additional.sh] && not_already additional
then
   zap simple-root-filesystem

   time ./additional.sh "$ARCH" || exit 1
   export SIMPLE_ROOT_OVERLAY=build/additional-"$ARCH"
fi

Or something like that.

> if you only use ./build.sh i486
> 
> then it would build without the content of the additional.sh packages
> 
> ---
> In all cases it would be up to the builder to populate this file and  
> handle any problems associated with it.
> 
> I'm not sure how much work or changes would be needed to the existing  
> scripts, but I see it as a way of extending aboriginals functionality  
> and thus the community using it.

I've encouraged adding more packages via native building. Cross  
compiling more packages has... issues:

https://speakerdeck.com/mirell/developing-for-non-x86-targets-using-qemu?slide=79

(The next dozen or so slides from there are on that topic.)

If there seriously is interest in this, I can document the required  
infrastructure better. (At one point I had, but it bit-rootted as  
everything changed over the years.)

Rob
 1372567000.0