<div dir="ltr"><div>It'd be great to move off GCC. I'm managing (just) to get Perl, Python, Go, Docker and QEMU compiling under Abo 1.4.5 with the GPLv2 toolchain.</div><div><br></div><div>The switch to musl is nice but has caused me some more headaches with Go (unsuprisingly - why is it so popular?).</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 11 February 2016 at 19:14, Rob Landley <span dir="ltr"><<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 02/08/2016 11:42 PM, Tristan Van Berkom wrote:<br>
> Hi,<br>
><br>
> Sorry for the repeat of this message, it is important to my employer<br>
> (and to me as well) to have this message linkable in the mailing list<br>
> archives and I'm just reposting this message (with a couple of minor<br>
> edits) for posterity. To clarify, the large part of 2015's lost<br>
> archives have been restored (yay!) and in restoring the backup, this<br>
> message and a few others were lost.<br>
><br>
> [Original message]<br>
><br>
> Hello Aboriginal Mailing List !<br>
><br>
> As some who have been present on irc are aware, I have been working<br>
> on a patch set to allow aboriginal linux to build with the a more<br>
> recent (GPLv3) toolchain.<br>
<br>
I note that I will never distribute prebuilt GPLv3 binaries, so either<br>
we retain support for the GPLv2 binaries or we stop distributing<br>
prebuilt binaries entirely.<br>
<br>
This is why, long-term, I've wanted to switch to an LLVM based<br>
toolchain. <a href="http://ellcc.org" rel="noreferrer" target="_blank">http://ellcc.org</a> is doing that with musl, AOSP builds one as<br>
part of its build (it doesn't build android with the host toolchain, it<br>
builds everything with llvm), and there's instructions on how to build<br>
it in the current "Beyond Linux From Scratch".<br>
<br>
I just haven't gotten around to it yet...<br>
<br>
> The initial patch set is working now, I have built and tested the<br>
> compiler and distcc on the resulting images for invocations with<br>
> CROSS_COMPILER_HOST=i686 for the armv5l and mips images.<br>
<br>
It's a pity the FSF changed licenses. Apple used to us gcc in xcode<br>
until the license change, then froze on the last GPLv2 release for 5<br>
years while they sponsored work on a replacement project.<br>
<br>
I.E. the llvm project exists in its current form because I'm not the<br>
ONLY person who was fine with GPLv2 but won't get GPLv3 on them, and is<br>
not going to "get over it" any more than I'd "get over it" about using<br>
Windows.<br>
<br>
> This passes nicely and the resulting image works with your regular<br>
> dev-environment.sh invocation, the compiler builds hello world in C and<br>
> C++ and also successfully distributes compilation over distcc to the<br>
> cross compiler.<br>
><br>
> In this mail I will outline the proposed changes implemented by this<br>
> patch set, along with any caveats and/or observations I can think of at<br>
> the time of writing it. The patch set however documents itself also.<br>
><br>
> The patchset is available at the following github branch and a pull<br>
> request was created for it today:<br>
><br>
> <a href="https://github.com/gtristan/aboriginal/tree/dual-toolchain" rel="noreferrer" target="_blank">https://github.com/gtristan/aboriginal/tree/dual-toolchain</a><br>
><br>
> Enjoy !<br>
><br>
> Best Regards,<br>
> -Tristan<br>
><br>
><br>
> Overview<br>
> ~~~~~~~~<br>
> This patchset does not replace the existing GPLv2 toolchain but instead<br>
> adds the newer toolchain in such a way that you may choose which<br>
> toolchain to build.<br>
<br>
I'm adding toybox to replace busybox, at which point I can remove<br>
busybox. I sometimes give status updates about this in the release notes.<br>
<br>
I'm adding musl to replace uClibc, at which point I can remove uClibc.<br>
The main reason for the delay has been lack of target support in musl; I<br>
didn't start adding it until I could at least see a path to removing<br>
uCLibc. (We need still sparc and m68k and so on.)<br>
<br>
I planned to add llvm/lld to replace gcc/binutils, at which point I<br>
could remove gcc/binutils. THe main reason for the delay has been the<br>
lack of target support in llvm/lld: you can't build sh4 with llvm.<br>
<br>
This adds a second set of infrastructure in parallel, which only some<br>
people will test, only some of the time, in only certain configurations.<br>
<br>
That said, people have real world needs and old toolchain is old, so my<br>
concern for this transitional infrastructure (until llvm is ready) is<br>
whether it can be made sufficiently unobtrusive.<br>
<br>
> Currently, only musl flavored builds are available with the modern<br>
> toolchain, it is possible to try and fix that so as to build uClibc<br>
> with GCC 5.3 if that is desirable to some, but it's just currently not<br>
> supported.<br>
><br>
> By default, the build will continue to build the GPLv2 toolchain, in<br>
> order to build the new toolchain you need only specify:<br>
><br>
> ENABLE_GPLV3=1<br>
><br>
> In order to allow multiple builds of the same package (i.e. for gcc and<br>
> binutils) I have made some changes to functions.sh and<br>
> download_function.sh.<br>
><br>
><br>
> Changes to the payload<br>
> ~~~~~~~~~~~~~~~~~~~~~~<br>
> This patch set introduces the following packages and patches to the<br>
> build.<br>
><br>
><br>
> Binutils 2.25.1<br>
> ~~~~~~~~~~~~~~~<br>
> Latest release tarball of binutils.<br>
><br>
> I have also included the musl related patches, imported from the<br>
> musl-cross-make project[0].<br>
><br>
><br>
> GCC 5.3.0<br>
> ~~~~~~~~~<br>
> Latest release tarball of GCC<br>
><br>
> I have also included Gregor Richards' patch set[1] to build<br>
> GCC 5.3 against musl. These are more up to date than the<br>
> patches in the musl-cross-make project which target GCC 5.2.0.<br>
><br>
><br>
> GMP 4.3.2<br>
> ~~~~~~~~~<br>
> The version of GMP used on the gcc infrastructure page[2], we<br>
> would use the latest version, which is 6.1.0, except that we<br>
> encounter errors when cross building the native compiler for<br>
> the target. These exact errors are discussed on the gmp-bugs<br>
> list in this thread[3].<br>
<br>
You know, in email you _can_ just put in the link here, rather than<br>
footnotes and a bibliography...<br>
<br>
> For this older package, we required an update of config.sub<br>
> and config.guess in order to recognize the -linux-musl* triples,<br>
> this is introduced in the form of a patch in the sources/patches<br>
> directory.<br>
<br>
The FSF isn't dogfooding current versions of its own packages. Splendid.<br>
<br>
> MPC 1.0.3<br>
> ~~~~~~~~~<br>
> Latest release tarball of MPC.<br>
><br>
><br>
> MPFR 3.1.3<br>
> ~~~~~~~~~~<br>
> Latest release tarball of MPFR.<br>
><br>
><br>
> Patches updating config.sub & config.guess<br>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
> The config.sub & config.guess needed to be updated for make, bash and<br>
> distcc in order to compile with the -linux-musl* host triples.<br>
<br>
I didn't use the -linux-musl triplets, I had ccwrap override all of that.<br>
<br>
For native builds, my build-one-package.sh script does:<br>
<br>
# Lobotomize config.guess so it won't complain about unknown target types.<br>
# 99% of packages do not care, but autoconf throws a temper tantrum if<br>
# the version of autoconf that created this back when the package shipped<br>
# didn't know what a microblaze or hexagon was. Repeat after me:<br>
# "Autoconf is useless"<br>
<br>
for guess in $(find . -name config.guess)<br>
do<br>
rm -f "$guess" &&<br>
echo -e "#!/bin/sh\ngcc -dumpmachine" > "$guess" || exit 1<br>
done<br>
<br>
<br>
> This should not effect builds using the older toolchain using GCC<br>
> 4.2.1, it merely updates these packages to recognize the new triple<br>
> at build time.<br>
><br>
> Note that config.sub & config.guess are under GPLv3 but include an<br>
> exception in the license that:<br>
><br>
> "you may include it under the same distribution<br>
> terms that you use for the rest of that program"<br>
><br>
> As an additional permission under section 7 of GPLv3.<br>
<br>
So the license is... public domain?<br>
<br>
If I _source_ it from one of those other packages, I get it under that<br>
other package's license...<br>
<br>
If this is the FSF trying to be lawyers, I really don't want to use any<br>
of the code where they tried to be security experts.<br>
<br>
> Host Tool Changes<br>
> ~~~~~~~~~~~~~~~~~<br>
> To build gcc 5.3, we now require:<br>
><br>
> o System installed c++ compiler<br>
><br>
> GCC now is partly written in C++<br>
><br>
> o System installed ranlib<br>
><br>
> Without this, we encounter problems building gcc, particularly when<br>
> linking GCC libiberty.a in the final native compiler.<br>
<br>
I'm tempted to make a v3build.sh that sets HOST_EXTRA=ranlib. I wonder<br>
how much of this I could do as a wrapper?<br>
<br>
<br>
> Overview how we build GCC 5.3 compared to 4.2.1<br>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
><br>
> o Binutils takes the new flag --enable-install-libiberty, which was<br>
> implied in older versions.<br>
><br>
> This is currently insubstantial and we could do without installing<br>
> libiberty here. We only install libiberty for the sake of building<br>
> elf2flt for the sh2eb target, which is currently only supported<br>
> with uClibc, which we do not build with GCC 5.3.<br>
><br>
> This change was kept to ease transition to GCC 5.3 once we can<br>
> supported it, although it may be useless.<br>
<br>
Aboriginal Linux is never "transitioning to" GCC 5.3. It's transitioning<br>
to LLVM.<br>
<br>
That said, I expect the jcore people are interested in this. :)<br>
<br>
> o We ignore NO_CPLUSPLUS when building the new toolchain, GCC 5.3<br>
> requires C++ to build itself and even pass it's own configure<br>
> scripts, so there is no point to try building without C++.<br>
<br>
Understood. LLVM has similar deficiencies, which is why I haven't<br>
entirely abandoned the idea of doing qcc someday (with a cfront variant<br>
to build llvm under it).<br>
<br>
> o The first stage compiler (or 'simple' cross compiler) is built<br>
> in a different order:<br>
><br>
> - Build GCC with only support for the C language<br>
> - Build libc<br>
> - Build GCC again, this time with C++ support<br>
<br>
I went to great lengths to avoid that in mine. :)<br>
<br>
> This is because you need libc in order to build the libstdc++<br>
> runtime. And you cannot get away with not building C++ at this<br>
> stage of course, because you also need C++ to compile gcc in the<br>
> full canadian cross compiler and native compiler.<br>
<br>
Sad how gcc's deteriorated, isn't it?<br>
<br>
> o GCC's new dependencies, GMP, MPC and MPFR are build directly in the<br>
> gcc build directory, this makes the whole build script a little<br>
> simpler because we dont have to care about configuring and staging<br>
> these libraries by hand as they are build as GCC modules.<br>
<br>
No point in treating them as separate packages: they aren't really.<br>
<br>
> Caveats<br>
> ~~~~~~~<br>
> There remain some dirty hacks and oddities, I will try to specify them<br>
> all here.<br>
><br>
> o We install musl twice<br>
><br>
> When building with GCC 5.3, we install musl twice because our<br>
> ccwrap program expects musl in one location while the gcc build<br>
> itself expects to find it in another location.<br>
><br>
> This does not break anything but is redundant and dirty and<br>
> is relatively easy to fix.<br>
<br>
symlink?<br>
<br>
> o Redundant builds of gmp, mpfr and mpc<br>
><br>
> I did not want to write individual build recipies and try to get<br>
> all the configure flags right everywhere, and thought it prudent to<br>
> allow GCC's toplevel configure script to configure those in it's<br>
> subdirectory as GCC should know how to do that better than us.<br>
><br>
> The downside is that since we necessarily build GCC twice in the<br>
> simple-cross-compiler stage, we end up building these libraries<br>
> twice as well.<br>
><br>
> I have tried issuing a make -C ${subdir} install in the first pass<br>
> and reusing them in the second pass by passing --with-gmp etc<br>
> during that second pass, and while this satisfies the configure<br>
> script it also breaks the build for some reason.<br>
><br>
> It could be the only sane fix is to build them completely<br>
> separately.<br>
<br>
I'm really not interested in optimizing the gplv3 build. If it works for<br>
you, great. In a year or two I want to throw out gcc entirely and switch<br>
to llvm, as soon as it supports a reasonable number of targets.<br>
<br>
> o The target triplets may have bugs right now.<br>
><br>
> In order to build GCC 5.3 against musl for any arch, it is<br>
> necessary to specify the target triplet ending in -musl*<br>
<br>
Why? I never bothered with that...<br>
<br>
> The approach I've used to solve this is a little hack in<br>
> functions.sh which specifies -linux-gnu as the default and<br>
> substitutes 's/gnu/musl' in the specific case that we are building<br>
> GCC 5.3 against musl.<br>
<br>
Why should gcc care what it's libc is? It's a library implementating an api.<br>
<br>
> The thinking is that in the targets, if a specific CROSS_TARGET is<br>
> specified, it should absolutely specify the trailing '-gnu' and let<br>
> the build scripts decide if it is indeed -gnu or -musl.<br>
<br>
In arm eabi I had to specify -gnueabi because it didn't work otherwise,<br>
and I never got around to patching it the rest of the way out. There's<br>
no other mention of "gnu" in any of the tuples.<br>
<br>
> A cleaner fix might be to mandate that if the targets specify a<br>
> CROSS_TARGET, it use a special suffix, so for instance in<br>
> sources/targets/armv5l we could specify:<br>
><br>
> armv5l-unknown-linux-LIBC-SUFFIXeabi<br>
<br>
Way too fancy, it will break.<br>
<br>
> And allow functions.sh to substitute LIBC-SUFFIX depending on which<br>
> libc happens to be chosen for the given build.<br>
><br>
> There could be various approaches to address this in a cleaner way.<br>
<br>
Why does gcc need to know which libc it's building against? The point of<br>
ccwrap is to overrides its attempts to find the headers and dynamic<br>
linker and such, it shouldn't _care_.<br>
<br>
If it does care, my "fix" would be surgery to force it to use names like<br>
"libc.so" and provide symlinks to them during the build.<br>
<br>
> [0]:<a href="https://github.com/richfelker/musl-cross-make" rel="noreferrer" target="_blank">https://github.com/richfelker/musl-cross-make</a><br>
> [1]:<a href="https://github.com/GregorR/musl-gcc-patches" rel="noreferrer" target="_blank">https://github.com/GregorR/musl-gcc-patches</a><br>
> [2]:<a href="ftp://gcc.gnu.org/pub/gcc/infrastructure/" rel="noreferrer" target="_blank">ftp://gcc.gnu.org/pub/gcc/infrastructure/</a><br>
> [3]:<a href="https://gmplib.org/list-archives/gmp-bugs/2015-December/003848.html" rel="noreferrer" target="_blank">https://gmplib.org/list-archives/gmp-bugs/2015-December/003848.html</a><br>
<br>
Devoid of context I dunno why you linked to any of that.<br>
<br>
Ok, looking at your patches...<br>
<br>
First patch modifies download.sh. The way ./download.sh works is that<br>
when you source download_functions.sh START_TIME=`date +%s` snapshots<br>
the current time, and then each "download" function updates the<br>
datestamp on existing tarballs that check out, and at the end<br>
cleanup_oldfiles deletes everything older than START_TIME.<br>
<br>
Your if/else test that avoids calling download for certain tarballs will<br>
therefore delete the tarballs for whichever version it's not currently<br>
downloading, forcing them to be re-downloaded later.<br>
<br>
(That's as far as I made it looking into this the first time. I have<br>
infrastructure in the build control images to have different download<br>
subdirectories for subprojects, I need to make sure I've got that<br>
working in the top level one and then have a packages/biohazard<br>
subdirectory to download the gplv3 stuff into.)<br>
<br>
This _also_ probably works best as a wrapper around download.sh<br>
downloading extra packages... :)<br>
<br>
Alright, I've transferred your patch list onto my netbook, I'll see what<br>
I can do...<br>
<br>
Rob<br>
_______________________________________________<br>
Aboriginal mailing list<br>
<a href="mailto:Aboriginal@lists.landley.net">Aboriginal@lists.landley.net</a><br>
<a href="http://lists.landley.net/listinfo.cgi/aboriginal-landley.net" rel="noreferrer" target="_blank">http://lists.landley.net/listinfo.cgi/aboriginal-landley.net</a><br>
</blockquote></div><br></div>