[Aboriginal] Proposed patches to support modern toolchain

[Tristan Van Berkom]

Hi,

Sorry for the repeat of this message, it is important to my employer
(and to me as well) to have this message linkable in the mailing list
archives and I'm just reposting this message (with a couple of minor
edits) for posterity. To clarify, the large part of 2015's lost
archives have been restored (yay!) and in restoring the backup, this
message and a few others were lost.

[Original message]

Hello Aboriginal Mailing List !

    As some who have been present on irc are aware, I have been working
on a patch set to allow aboriginal linux to build with the a more
recent (GPLv3) toolchain.

The initial patch set is working now, I have built and tested the
compiler and distcc on the resulting images for invocations with
CROSS_COMPILER_HOST=i686 for the armv5l and mips images.

This passes nicely and the resulting image works with your regular 
dev-environment.sh invocation, the compiler builds hello world in C and
C++ and also successfully distributes compilation over distcc to the
cross compiler.

In this mail I will outline the proposed changes implemented by this
patch set, along with any caveats and/or observations I can think of at
the time of writing it. The patch set however documents itself also.

The patchset is available at the following github branch and a pull
request was created for it today:

    https://github.com/gtristan/aboriginal/tree/dual-toolchain

Enjoy !

Best Regards,
    -Tristan


Overview
~~~~~~~~
This patchset does not replace the existing GPLv2 toolchain but instead
adds the newer toolchain in such a way that you may choose which
toolchain to build.

Currently, only musl flavored builds are available with the modern
toolchain, it is possible to try and fix that so as to build uClibc
with GCC 5.3 if that is desirable to some, but it's just currently not
supported.

By default, the build will continue to build the GPLv2 toolchain, in
order to build the new toolchain you need only specify:

  ENABLE_GPLV3=1

In order to allow multiple builds of the same package (i.e. for gcc and
binutils) I have made some changes to functions.sh and
download_function.sh.


Changes to the payload
~~~~~~~~~~~~~~~~~~~~~~
This patch set introduces the following packages and patches to the
build.


  Binutils 2.25.1
  ~~~~~~~~~~~~~~~
  Latest release tarball of binutils.

  I have also included the musl related patches, imported from the
  musl-cross-make project[0].


  GCC 5.3.0
  ~~~~~~~~~
  Latest release tarball of GCC

  I have also included Gregor Richards' patch set[1] to build
  GCC 5.3 against musl. These are more up to date than the
  patches in the musl-cross-make project which target GCC 5.2.0.


  GMP 4.3.2
  ~~~~~~~~~
  The version of GMP used on the gcc infrastructure page[2], we
  would use the latest version, which is 6.1.0, except that we
  encounter errors when cross building the native compiler for
  the target. These exact errors are discussed on the gmp-bugs
  list in this thread[3].

  For this older package, we required an update of config.sub
  and config.guess in order to recognize the -linux-musl* triples,
  this is introduced in the form of a patch in the sources/patches
  directory.


  MPC 1.0.3
  ~~~~~~~~~
  Latest release tarball of MPC.


  MPFR 3.1.3
  ~~~~~~~~~~
  Latest release tarball of MPFR.


  Patches updating config.sub & config.guess
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  The config.sub & config.guess needed to be updated for make, bash and
  distcc in order to compile with the -linux-musl* host triples.

  This should not effect builds using the older toolchain using GCC
  4.2.1, it merely updates these packages to recognize the new triple
  at build time.

  Note that config.sub & config.guess are under GPLv3 but include an
  exception in the license that:

    "you may include it under the same distribution
     terms that you use for the rest of that program"

  As an additional permission under section 7 of GPLv3.


Host Tool Changes
~~~~~~~~~~~~~~~~~
To build gcc 5.3, we now require:

  o System installed c++ compiler

    GCC now is partly written in C++

  o System installed ranlib

    Without this, we encounter problems building gcc, particularly when
    linking GCC libiberty.a in the final native compiler.


Overview how we build GCC 5.3 compared to 4.2.1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  o Binutils takes the new flag --enable-install-libiberty, which was
    implied in older versions.

    This is currently insubstantial and we could do without installing
    libiberty here. We only install libiberty for the sake of building
    elf2flt for the sh2eb target, which is currently only supported
    with uClibc, which we do not build with GCC 5.3.

    This change was kept to ease transition to GCC 5.3 once we can
    supported it, although it may be useless.

  o We ignore NO_CPLUSPLUS when building the new toolchain, GCC 5.3
    requires C++ to build itself and even pass it's own configure
    scripts, so there is no point to try building without C++.

  o The first stage compiler (or 'simple' cross compiler) is built
    in a different order:

    - Build GCC with only support for the C language
    - Build libc
    - Build GCC again, this time with C++ support

    This is because you need libc in order to build the libstdc++
    runtime. And you cannot get away with not building C++ at this
    stage of course, because you also need C++ to compile gcc in the
    full canadian cross compiler and native compiler.

  o GCC's new dependencies, GMP, MPC and MPFR are build directly in the
    gcc build directory, this makes the whole build script a little
    simpler because we dont have to care about configuring and staging
    these libraries by hand as they are build as GCC modules.


Caveats
~~~~~~~
There remain some dirty hacks and oddities, I will try to specify them
all here.

  o We install musl twice

    When building with GCC 5.3, we install musl twice because our
    ccwrap program expects musl in one location while the gcc build
    itself expects to find it in another location.

    This does not break anything but is redundant and dirty and
    is relatively easy to fix.

  o Redundant builds of gmp, mpfr and mpc

    I did not want to write individual build recipies and try to get
    all the configure flags right everywhere, and thought it prudent to
    allow GCC's toplevel configure script to configure those in it's
    subdirectory as GCC should know how to do that better than us.

    The downside is that since we necessarily build GCC twice in the
    simple-cross-compiler stage, we end up building these libraries
    twice as well.

    I have tried issuing a make -C ${subdir} install in the first pass
    and reusing them in the second pass by passing --with-gmp etc
    during that second pass, and while this satisfies the configure
    script it also breaks the build for some reason.

    It could be the only sane fix is to build them completely
    separately.

  o The target triplets may have bugs right now.

    In order to build GCC 5.3 against musl for any arch, it is
    necessary to specify the target triplet ending in -musl*

    The approach I've used to solve this is a little hack in
    functions.sh which specifies -linux-gnu as the default and
    substitutes 's/gnu/musl' in the specific case that we are building
    GCC 5.3 against musl.

    The thinking is that in the targets, if a specific CROSS_TARGET is
    specified, it should absolutely specify the trailing '-gnu' and let
    the build scripts decide if it is indeed -gnu or -musl.

    A cleaner fix might be to mandate that if the targets specify a
    CROSS_TARGET, it use a special suffix, so for instance in
    sources/targets/armv5l we could specify:

             armv5l-unknown-linux-LIBC-SUFFIXeabi

    And allow functions.sh to substitute LIBC-SUFFIX depending on which
    libc happens to be chosen for the given build.

    There could be various approaches to address this in a cleaner way.


[0]:https://github.com/richfelker/musl-cross-make
[1]:https://github.com/GregorR/musl-gcc-patches
[2]:ftp://gcc.gnu.org/pub/gcc/infrastructure/
[3]:https://gmplib.org/list-archives/gmp-bugs/2015-December/003848.html