[Aboriginal] Refactoring/recycling

[Rob Landley]

On 09/12/2016 06:41 AM, Tristan Van Berkom wrote:
> On Fri, 2016-09-09 at 19:28 -0500, Rob Landley wrote:
>> I've been breaking aboriginal linux down for parts. It's just too
>> much
>> of a giant hairball, despite my attempts to separate it into layers.
>>
> 
> Hi Rob,
> 
> So just to understand, since I have some interesting work based on this
> by now, what is going to be the end result of using the musl-cross-make 
> approach in Aboriginal ?

The project  has been mothballed for several months while I tried to
decide how to move forward, and now I've got an approach I want to try.
So what this means is development is resuming, albeit in a different
direction.

The fundamental constraints are:

1) I will NEVER host GPLv3 binaries anywhere on my own dime

2) my old toolchain is now far enough behind the kernel build is
breaking badly every release

3) LLVM (especialy lld, without which you're stuck using GPLv3 binutils)
isn't ready yet

When LLVM becomes ready, I may reassemble something like the original
aboriginal linux. But for now, I _have_ to break it up into packages I
can develop and packages I passively consume (I'll send back bug reports
but not code). And if I'm going to break it up anyway, I might as well
reexamine the design and see what belongs where.

Note: if my aversion to GPLv3 was the kind of thing I "got over", I
would be a Windows developer. This is a "hell no, death first" kind of
thing. It's 2016 and I don't have a Facebook account because I don't
_want_ one. I'll read a facebook page somebody links me to if it lets me
without a login, I'll sit down at somebody else's windows machine to web
surf or play a game. But I won't create an account or write code that
only runs on windows the platform (it can be ported _to_ it). Similarly
I won't host GPLv3 binaries on a site I control or write GPLv3 code in a
hobbyist context (but an employer can pay me to).

Given a choice between shipping GPLv3 code and shutting down Aboriginal
Linux... I haven't put out a release since the 4.3 kernel for a reason.

> How much will things change from the
> perspective of the Aboriginal end user ?

I should still provide a turnkey solution for building minimal native
development environment system images, but it's going to involve
prerequisite packages. One of which is Rich's musl-cross-make.

You should still be able to download prebuilt binaries of all the bits,
but some of them may be on different sites and require assembly. (I can
post a shell script to wget and collate if it comes to that.)

> If I understand correctly, this will mean:
> 
>   o GPLv3 toolchain

I won't host binaries of GPLv3 code, but Rich has agreed to host
binaries of both static cross compilers and native compilers, once I've
pointed him at the configurations I've made work for the target list
I've gotten running.

If you remove toolchain building from what Aboriginal Linux does, what's
left of the build stages doesn't make design sense, so I'm redesigning
it, and moving bits of it into other projects. (Toybox should probably
have a "make cross_airlock PREFIX=/path" build target, for example,
replacing host-tools.sh. For the moment symlinking host binaries out of
your $PATH for what it used to build busybox for, but it does mean that
the list of remaining commands toybox needs to replace would be
maintained within toybox, which is a step in the right direction.)

In a way, toybox becomes a prerequisite on the level of musl-cross-make,
but toybox was already a prerequisite. The real difference is
musl-cross-make downloads its own source packages, so it's a separate
build. Similarly, my new much simpler root filesystem build script,
mkroot.sh, also downloads its own packages.

  https://github.com/j-core/mkroot/blob/master/mkroot.sh

I'm also thinking of breaking out the download.sh infrastructure into
its own package, so you can use the package cache stuff from other builds:

  http://landley.net/aboriginal/FAQ.html#debug_package_cache

Right now the build control image infrastructure has a hand-copied
version that falls out of sync; that should be cleaned up. I can have
mkroot.sh try to source an include file (and possibly offer it to Rich
as well, and maybe see if buildroot wants to use it), and fall back to
the much simpler built-in download function (lines 19-33 of the current
script) if it's not found.

So musl-cross-make provides the toolchain, mkroot.sh provides the
initramfs, and the third part I need is something to provide kernel
config and qemu wrapper script for a given target (the sources/targets
info in the current stuff) and package everything appropriately.

Speaking of system images, there's a glue layer where I get a compiler
tarball from Rich and need to repackage it as a squashfs file, which
involves having (or building) mksquashfs on the host as a depencency.
Possibly I can make puppy eyes at Rich to host system image tarballs too. :)

>   o Less deviation from upstream gcc (as I understand there had been
>     some patches to make gcc behave differently for aboriginal, while
>     the musl-cross-make patches are mostly upstreamed)

My compiler was GPLv2. I will never ship GPLv3 binaries. (This is part
of the reason I'm slowly switching my interest to Android from
conventional Linux: GPLv3 is a proprietary license and I'd only work on
code under that license if paid to do so by a large corporation whose
legal department stood between me and the FSF loonies. So far, the
corporations all feel the same way about it I do.)

Your GPLv3 fork is just that: a fork. It's part of the reason I
mothballed Aboriginal Linux, I assumed that the userbase would fragment
and half the people wouldn't be using my code so why bother fixing up
the old toolchains?

But if Rich is maintaining a toolchain that can provide working native
compilers, then I don't have to deal with that area at all and can wash
my hands of it entirely and be a passive consumer of that output,
downloading it the same way I download the flash plugin or decss; via an
automated script that happens behind the scenes so I don't get any of it
on me personally.

> Are you planning to continue with ccwrap.c, allowing for the
> relocatable cross compiler ? (I think this was a pretty cool feature,
> although I understand it's painful to keep up with gcc here).

I hadn't planned on it, but I can ask Rich what his opinions are?

I _think_ if you download his toolchain binaries they'll automatically
sysroot themselves in whatever directory they're running from, but I
haven't tried it?

> In the end, will we have essentially the same bootstrapping procedure ?

In flux at the moment. There should be a turnkey solution ala build.sh,
but it may take a while to get back there.

> I think there is a high value in depending on a minimal amount of host
> tools, building toybox with host tools and isolating the build is great
> for repeatability and reliability of the build across multiple host
> OSes.

Which is why I'm migrating that functionality into toybox itself. :)

> On a side note, I wonder if there is knowledge we can borrow from the
> openembedded project - particularly where it comes to targeting the
> compiler for a specific arch, configuring the kernel for that arch and
> putting the qemu invocation together.
> 
> See:
>   http://git.openembedded.org/openembedded-core/tree/meta/conf/machine/

I've looked at buildroot, at debian's supported targets, and like 5
other places. Never looked at openembedded (outside of drilling down
from yocto/tizen) but only because they never had anything particularly
interesting target-wise. (It's mostly arm forks, and all that should go
device tree and stop being silly.)

QEMU really needs to grow a mode where you go "-dtb blah.dtb" and it
parses that and emulates the devices described by it, and also passes
that same dtb on to the kernel the way bootloaders do so the kernel can
bind to devices from the same data file qemu is using to add those
devices. And THAT is your entire board emulation, "-M generic"

> And the include directory for fun.
> 
> Honestly there are a variety of reasons why I prefer using an LFS type
> of build (but automated) based on aboriginal over using the yocto
> approach of cross compiling all the way up the stack, but I think
> openembedded have come a long way in terms of configuring the base
> kernel/compiler/qemu and it would be great to leverage that. It also
> has a similar approach to aboriginal here (inasmuch as it uses a
> machine conf, like aboriginals target files).

If you prefer openembedded, feel free to use it?

> Best Regards,
>     -Tristan

Rob