[Aboriginal] Build from version control system repos instead of from tarballs

[Rob Landley]

On 10/30/2012 03:49:56 AM, Alessio Igor Bogani wrote:
> Hi Rob,
> 
> On 30/10/2012 03:36, Rob Landley wrote:
>> On 09/04/2012 03:49:07 AM, Alessio Igor Bogani wrote:
> [...]
>>> Can Aboriginal grabs sources from git(Linux, musl, uclibc, busybox)
>>> and hg(toybox) repos instead of from tarballs?
> [...]
>> populating the package cache from the git repo, and
> [...]
>> Really what I want to do is say "URL=git://busybox.net/busybox" in
>> download.sh, have that clone the relevant repo under packages (so
>> packages/busybox would be a directory), and then just use that as the
>> package cache version.
>> 
>> The question is, when do I re-pull?  (Every time you run  
>> download.sh?)
> 
> Never anymore.
> 
> I would want use a specific tag or commit (i.e. git reset --hard  
> vX.Y) moreover I would want decide when update and switch to a  
> different (updated) tag or commit.
> 
> Personally I don't use tarballs at all anymore: I'll download the  
> project's repo and build from it. I'm very happy if Aboriginal let me  
> work in this way.

It currently does, using a two step process:

1) Edit "configure" and add the packages you want to do git versions of  
to the USE_ALT variable.  (USE_ALT=busybox,uClibc,linux)

2) Then check out your repos into build/packages/alt-busybox and  
build/packages/alt-uClibc and so on.

What I was talking about was coming up with a more obvious way to do  
it, which doesn't involve quite so much knowledge of the guts of  
aboriginal's build to understand what's going on. (Basically the above  
manually populates the package cache. Using the alt- slot for the  
package disables the normal download and packaging behavior.)

> For example (sorry in git-ish):
> 
> git clone aboriginal

I'm vaguely interested in having a git mirror of the mercurial repo,  
but I have no interest in hosting any of my projects in git. The git  
user interface is appalling, and their solution to all their problems  
is to expose implementation details to the user.

> cd aboriginal/ && ls
> build.sh config cross-compiler.sh .. (these should works with any  
> version and doesn't
> care if gcc is 4.2.2 or 4.7.1)

The build script required for 4.2 and 4.7 are _totally_different_.  For  
one thing, newer gcc versions have something like 4 prerequisite  
packages you have to build first, requirements 4.2 didn't have.

Unfortunately, your desire doesn't match reality there.

> ls package/linux (<- this is a submodule)
> git submodule init
> git submodule update
> cd package/linux

I can add the ability to put extracted source in "packages/linux" and  
build from that, and I can probably teach download.sh to leave it alone  
if it's there.

But the aboriginal build would be orthogonal to where that source came  
from. You can create your own git repo full of submodules for all the  
packages if you like, I'm not doing it.

Right now, the only thing that cares where the contents of the  
"packages" directory came from is download.sh.  The rest of the build  
doesn't. I intend to keep it orthogonal: git as an alternative to  
download.sh.

> git reset --hard origin (reset to the Linus' HEAD)
> (don't change anything other aka let's use Rob's default versions)
> ./build i686-virtio

How about this:

I can modify the scripts so that if there's a directory under packages  
with the package name, we use that source verbatim. (No patches, no  
nothing.) Otherwise, we fall back to the current behavior. (Basically  
it's just implementing a search path for the package cache with the  
first entry searched before attempting to _populate_ the package cache.)

That would mean if you "git checkout git://linux packages/linux" the  
build uses that git repo, and patching/updating it is your problem.

Does that sound reasonable? I can provide an example script that would  
do a checkout of all the package versions we're using, and then the  
aboriginal build would stay out of your way.

> Why I bother with this crazy request? Because In this way I can  
> handle multi-level
> changes in a very simple way. So if I want add a feature which needs  
> to change/patch
> kernel, libc and application I could treat this like a single atomic  
> commit.

Which would never go upstream.

My build uses clean vanilla release versions of upstream packages, and  
separates out all the patches (and I try to send those patches upstream  
so I don't have to carry them, with varying success).

This is a design decision, so you can see what I'm using and what I did  
to it.

Your suggestion makes things easier for you at the expense of making it  
harder for anybody else to see what you did or reproduce it outside  
your build system. That's fine for your purposes, but that's not what  
I'm after.

I'm not _trying_ to scale the aboriginal build to compile more packages  
in a more complciated way. The point of the thing is to bootstrap the  
simplest possible native environment on target, and then let you build  
natively under that. I want to add more targets (hexagon, microblaze,  
get m68k working), and I want to expand the control-images repository  
to include more example packages you can build (especially automated  
distro bootstraps). But if you want a giant hairball that  
cross-compiles 300 packages without verifying that any of them actually  
runs, you want buildroot.

I'm happy to make the build more flexible, and I _want_ my code to to  
stay out of your way so you can do what you want to do. I want to make  
it trivially easy to overlay "git submodule" on top of the build, but  
my build is not becoming dependent on git submodule.

Rob

 1351635745.0