[Aboriginal] Build from version control system repos instead of from tarballs (Rob Landley)

[Rob Landley]

On 10/31/2012 05:36:30 PM, Isaac Dunham wrote:
> 
> > On 10/30/2012 03:49:56 AM, Alessio Igor Bogani wrote:
> > > Hi Rob,
> > >
> > > On 30/10/2012 03:36, Rob Landley wrote:
> > >> On 09/04/2012 03:49:07 AM, Alessio Igor Bogani wrote:
> > >>> Can Aboriginal grabs sources from git(Linux, musl, uclibc,  
> busybox)
> > >>> and hg(toybox) repos instead of from tarballs?
> > >> populating the package cache from the git repo, and
> > >> Really what I want to do is say "URL=git://busybox.net/busybox"  
> in
> > >> download.sh, have that clone the relevant repo under packages (so
> > >> packages/busybox would be a directory), and then just use that  
> as the
> > >> package cache version.
> > >>
> ..
> > > Personally I don't use tarballs at all anymore: I'll download the
> > > project's repo and build from it. I'm very happy if Aboriginal  
> let me
> > > work in this way.
> >
> > It currently does, using a two step process:
> >
> > 1) Edit "configure" and add the packages you want to do git  
> versions of
> > to the USE_ALT variable.  (USE_ALT=busybox,uClibc,linux)
> >
> > 2) Then check out your repos into build/packages/alt-busybox and
> > build/packages/alt-uClibc and so on.
> 
> For what it's worth, it would be quite possible to just use
> "git clone $URL alt-$PACKAGE"

I've done that.

Ok, let me back up and explain my problem:

There are three categories of files in the aboriginal linux directory:

1) Files that come from the aboriginal tarball or repository (*.sh,  
sources/*, more/*, and so on.) These are tracked by aboriginal's source  
control.

2) Downloaded files that come from third parties (packages/*): if you  
delete this download.sh can re-download it, but only if you have net  
access.

3) Generated files (build/*). If you delete this, the build can  
recreate it even without net access. In fact, going "rm -rf build" is  
more or less "make distclean".

Right now the package cache is in build/packages and is derived from  
the downloaded files. (It's the extracted tarball with patches  
applied.) Replacing the package cache with the upstream source control  
repository functionally does what you want, but at a design level it  
means that's the wrong place to put it in the file hierarchy.

I think I've worked out how I want to fix this and I'll document it  
properly once I get it done, I just haven't finished coding it yet. (I  
cleanly ripped out the old alt- mechanism, which should no longer be  
required, and then set it running a test build, which took long enough  
I got distracted.)

> > > For example (sorry in git-ish):
> > >
> > > git clone aboriginal
> >
> > I'm vaguely interested in having a git mirror of the mercurial repo,
> That would be enough for this purpose, AFAICT.
> > but I have no interest in hosting any of my projects in git. The git
> > user interface is appalling, and their solution to all their  
> problems
> To each his own (I actually enjoy it)
> > is to expose implementation details to the user.
> 
> > > cd aboriginal/ && ls
> > > build.sh config cross-compiler.sh .. (these should works with any
> > > version and doesn't
> > > care if gcc is 4.2.2 or 4.7.1)
> >
> > The build script required for 4.2 and 4.7 are _totally_different_.
> > For one thing, newer gcc versions have something like 4 prerequisite
> > packages you have to build first, requirements 4.2 didn't have.
> >
> > Unfortunately, your desire doesn't match reality there.
> >
> > > ls package/linux (<- this is a submodule)
> > > git submodule init
> > > git submodule update
> > > cd package/linux
> >
> <snip long discussion of why it isn't a good idea>
> 
> > > git reset --hard origin (reset to the Linus' HEAD)
> > > (don't change anything other aka let's use Rob's default versions)
> > > ./build i686-virtio
> >
> > How about this:
> >
> > I can modify the scripts so that if there's a directory under  
> packages
> > with the package name, we use that source verbatim. (No patches, no
> > nothing.) Otherwise, we fall back to the current behavior.  
> (Basically
> > it's just implementing a search path for the package cache with the
> > first entry searched before attempting to _populate_ the package  
> cache.)
> 
> Be sure to make {,dist,real}clean...
> Otherwise, sounds reasonable to me.

Not qutie sure what you mean?

Aboriginal hasn't got "make". We do have more/clean.sh but that's  
actually there to avoid deleting/rebuilding the target-independent  
parts (the package cache and the host tools in build/host).

> > > Why I bother with this crazy request? Because In this way I can
> > > handle multi-level
> > > changes in a very simple way. So if I want add a feature which
> > > needs to change/patch
> > > kernel, libc and application I could treat this like a single
> > > atomic commit.
> Doesn't this make tracking down what caused breakage harder?
> ie, if you git bisect to an atomic commit in all three, you can't
> tell if the kernel or the libc patch is responsible.

I try to remain agnostic about what users want to do. That said, the  
project is adapted to my working style for obvious reasons. :)

> > Which would never go upstream.
> >
> > My build uses clean vanilla release versions of upstream packages,  
> and
> > separates out all the patches (and I try to send those patches  
> upstream
> > so I don't have to carry them, with varying success).
> >
> > This is a design decision, so you can see what I'm using and what I  
> did
> > to it.
> >
> > Your suggestion makes things easier for you at the expense of  
> making it
> > harder for anybody else to see what you did or reproduce it outside
> > your build system. That's fine for your purposes, but that's not  
> what
> > I'm after.
> Note that this could make GPL conformance (where it matters) a little  
> harder-otherwise, patches plus tarballs are enough, but this may mean  
> you need to make the repo public (I'm not sure how git archive would  
> work with submodules, though)

Or easier if you go "I don't need to mirror the clean vanilla tarball,  
just the source url from download.sh and my patches against it from  
sources/patches". Having the paches cleanly separated out works better  
with that approach, which you need if the patches will ever go  
upstream. Developers _need_ broken up deltas against a published  
version they did in order to parse the changes, if you don't provide  
that they have to reverse engineer it from your giant hairball, which  
is incredibly painful.

Clueless legal entities like the FSF and Software Conservancy which  
prefer hairballs are blind to the actual engineering requirements.

Still, If somebody wants to use a patched tarball they can. If somebody  
wants to use a repo, they can. I've tried to be agnostic about the data  
sources, I see now I wasn't agnostic enough and am working to fix it.

Rob