From rlandley at parallels.com Wed Mar 2 21:44:49 2011 From: rlandley at parallels.com (Rob Landley) Date: Wed, 2 Mar 2011 15:44:49 -0600 Subject: [Aboriginal] Fail to patch uClibc In-Reply-To: References: Message-ID: <4D6EBA51.6040400@parallels.com> Sorry for the delay, last Wednesday I got suddenly informed I was going to Scale in LA over the weekend, where I got Con Crud from which I'm still recovering. On 02/27/2011 05:42 PM, Ken Burk wrote: > Rob, > > I know your busy with your day job so please no rush on this but the > latest as of today fails to patch uClibc when building the cross > compiler.My host os is Knoppix 6.4.3 dvd(usb boot) which built > fd341b09cc27.tar.bz2 without problems as well as your lfs-bootstrap for > i686 and armv5l (thanks for the lfs it is awesome).I am posting the > output below from when it started to error. Current -hg is in flux at the moment because I'm trying to switch to the NPTL version of uClibc. It doesn't work. For that matter, the pthreads stuff doesn't work for half the platforms in the current -rc, but I've decided that debugging _that_ is a waste of time because the point of upgrading is to switch to NPTL. Alas, I haven't gotten much time to work on it recently (see "Scale", above). > Ken > > > > Applying > /media/sda2/root/tech2k/aboriginal-29e1df07e99a/sources/patches/uClibc-backport-humor-udev.patch > patching file libc/sysdeps/linux/common/bits/socket.h That patch is backporting stuff, and now that I've upgraded the uClibc tarball to -rc2 the change is already in there. Probably I deleted it locally out of my patches directory but forgot to commit that. (I'm being a bit tentative about ripping up my config for -rc2 in a way that makes it hard to revert because I would like to do a release this month and I'm not sure -rc2 is going to be in it.) You can just delete that patch, but the _bigger_ problem is that uClibc-rc2 is flaky. So being able to patch it doesn't guarantee it'll do anything useful at present. I should probably just revert the download.sh snippet that's using -rc2 and move back to the last release version, but I've been trying to force myself to test the current stuff and get it to WORK. (It's fiddly, in a bunch of different ways. Currently I have to switch from sjlj exceptions to the dwarf stuff because they only implemented support for one codepath. Sigh.) Rob 1299102289.0 From alanpost at sunflowerriver.org Mon Mar 14 13:36:21 2011 From: alanpost at sunflowerriver.org (Alan Post) Date: Mon, 14 Mar 2011 07:36:21 -0600 Subject: [Aboriginal] /bin/sh error compiling distcc In-Reply-To: <20110314130503.GA40390@alice.local> References: <20110314130503.GA40390@alice.local> Message-ID: <20110314133621.GA40624@alice.local> On Mon, Mar 14, 2011 at 07:05:03AM -0600, Alan Post wrote: > I'm trying to build aboriginal 1.0.1 inside the i486 system image. > The packages download and I get to the next stage, building host > tools. > > busybox appears to compile fine. I'm getting the following error > message tryitg to build distcc: > > === Building host-tools > === distcc (host host-tools) > Snapshot 'distcc'... > configure: WARNING: you should use --build, --host, --target > configure: WARNING: invalid host type: "--with-included-popt--disable-Werror" > checking build system type... Invalid configuration `"--with-included-popt--disable-Werror"': machine `"--with-included-popt--disable' not recognized > configure: error: /bin/bash ./config.sub "--with-included-popt--disable-Werror" failed > > Exiting due to errors (host host-tools distcc) > > > Somewhere in the configure stage for distcc, the two configure options: > '--with-included-popt' and '--disable-Werror' are being joined together. > > Going into the host-temp/distcc build directory, I try running the command > by hand. I get the same error message. This suggests to me that I > have a bug in /bin/sh in busybox. > > If I modify the build script to call "/bin/bash ./configure ..." this problem > goes away. I can certainly modify the configure line for each > invocation to configure. Given that the system image is a squashfs, > I can't see any way to change either the /bin/sh symlink or a way to > tell busybox that in should execute /bin/bash when it is invoked as > sh. > > Have I found the best workaround? I'd also like to find and fix the > underlying issue; that will require more testing on my part; I want > to be able to compile the whole system first. > > -Alan After adding /bin/bash to all of the ./configure lines in host-tools.sh, I get a little further. This time I error out building e2fsprogs: ... On linux-gnu systems, root_prefix defaults to '' On linux-gnu systems, prefix defaults to /usr checking whether linker accepts -static... yes configure: creating ./config.status sh: syntax error at 'S_is_set[key]' Exiting due to errors (host host-tools e2fsprogs) real 2m38.058s user 1m50.739s sys 0m37.634s I'm not sure what I'm seeing here. I believe the 'S_is_set[key]' is awk code, and I surmise from that that the awk from busybox is not compiling some expression correctly. But the line begins with 'sh: ', so maybe what is going on is that some awk code is accidentally being interpreted by the shell. Now that I think about it, that could be what is going on. It could be that the config.status program is generated and executed not by bash, but by the busybox sh, and it has encountered another expression it can't parse. Alas, I'm not sure now how best to work around it. Suggestions? -Alan -- .i ma'a lo bradi ku penmi gi'e du 1300109781.0 From alanpost at sunflowerriver.org Mon Mar 14 13:05:03 2011 From: alanpost at sunflowerriver.org (Alan Post) Date: Mon, 14 Mar 2011 07:05:03 -0600 Subject: [Aboriginal] /bin/sh error compiling distcc Message-ID: <20110314130503.GA40390@alice.local> I'm trying to build aboriginal 1.0.1 inside the i486 system image. The packages download and I get to the next stage, building host tools. busybox appears to compile fine. I'm getting the following error message tryitg to build distcc: === Building host-tools === distcc (host host-tools) Snapshot 'distcc'... configure: WARNING: you should use --build, --host, --target configure: WARNING: invalid host type: "--with-included-popt--disable-Werror" checking build system type... Invalid configuration `"--with-included-popt--disable-Werror"': machine `"--with-included-popt--disable' not recognized configure: error: /bin/bash ./config.sub "--with-included-popt--disable-Werror" failed Exiting due to errors (host host-tools distcc) Somewhere in the configure stage for distcc, the two configure options: '--with-included-popt' and '--disable-Werror' are being joined together. Going into the host-temp/distcc build directory, I try running the command by hand. I get the same error message. This suggests to me that I have a bug in /bin/sh in busybox. If I modify the build script to call "/bin/bash ./configure ..." this problem goes away. I can certainly modify the configure line for each invocation to configure. Given that the system image is a squashfs, I can't see any way to change either the /bin/sh symlink or a way to tell busybox that in should execute /bin/bash when it is invoked as sh. Have I found the best workaround? I'd also like to find and fix the underlying issue; that will require more testing on my part; I want to be able to compile the whole system first. -Alan -- .i ma'a lo bradi ku penmi gi'e du 1300107903.0 From rlandley at parallels.com Mon Mar 14 23:36:00 2011 From: rlandley at parallels.com (Rob Landley) Date: Mon, 14 Mar 2011 18:36:00 -0500 Subject: [Aboriginal] /bin/sh error compiling distcc In-Reply-To: <20110314130503.GA40390@alice.local> References: <20110314130503.GA40390@alice.local> Message-ID: <4D7EA660.8000100@parallels.com> On 03/14/2011 08:05 AM, Alan Post wrote: > I'm trying to build aboriginal 1.0.1 inside the i486 system image. > The packages download and I get to the next stage, building host > tools. Sigh. It used to self-host, I've been a bit distracted by the pending uClibc-NPTL release and the growing list of platform regressions (sparc, sh4, and mips64 have all been really close to working, but then they float away again as new package releases come out...) > busybox appears to compile fine. I beat the busybox guys up when they break stuff. :) > I'm getting the following error > message tryitg to build distcc: > > === Building host-tools > === distcc (host host-tools) > Snapshot 'distcc'... > configure: WARNING: you should use --build, --host, --target > configure: WARNING: invalid host type: "--with-included-popt--disable-Werror" > checking build system type... Invalid configuration `"--with-included-popt--disable-Werror"': machine `"--with-included-popt--disable' not recognized > configure: error: /bin/bash ./config.sub "--with-included-popt--disable-Werror" failed > > Exiting due to errors (host host-tools distcc) > > > Somewhere in the configure stage for distcc, the two configure options: > '--with-included-popt' and '--disable-Werror' are being joined together. It's trying to use the wrong shell. (None of the busybox shells actually work quite right yet. Making hush into a decent bash replacement is a major todo item of mine if I ever get a spare 6 months.) Sigh, it looks like I screwed up the sequencing and /bin/sh isn't pointing to bash. It should be. I'll fix it. > Going into the host-temp/distcc build directory, I try running the command > by hand. I get the same error message. This suggests to me that I > have a bug in /bin/sh in busybox. The bug is that /bin/sh points to busybox. There's a reason I install bash. I'd love to get rid of it, but busybox needs a major upgrade for that to happen. (Denys and I agree that "hush" is the shell to work on, we just need to find the time...) > Have I found the best workaround? I'd also like to find and fix the > underlying issue; that will require more testing on my part; I want > to be able to compile the whole system first. Fix the symlink in root-filesystem and re-run system-image.sh. > -Alan Rob 1300145760.0 From rlandley at parallels.com Tue Mar 15 00:26:48 2011 From: rlandley at parallels.com (Rob Landley) Date: Mon, 14 Mar 2011 19:26:48 -0500 Subject: [Aboriginal] /bin/sh error compiling distcc In-Reply-To: <4D7EA660.8000100@parallels.com> References: <20110314130503.GA40390@alice.local> <4D7EA660.8000100@parallels.com> Message-ID: <4D7EB248.1090309@parallels.com> On 03/14/2011 06:36 PM, Rob Landley wrote: >> Somewhere in the configure stage for distcc, the two configure options: >> '--with-included-popt' and '--disable-Werror' are being joined together. > > It's trying to use the wrong shell. (None of the busybox shells > actually work quite right yet. Making hush into a decent bash > replacement is a major todo item of mine if I ever get a spare 6 months.) > > Sigh, it looks like I screwed up the sequencing and /bin/sh isn't > pointing to bash. It should be. I'll fix it. http://landley.net/hg/aboriginal/rev/1332 I've got a dozen todo items for the next release, but half of 'em are "fix XXX in pending uClibc-NPTL stuff"... I may still just revert that for the moment and do a bugfix-only release, since it's overdue... I head back to Russia for a week on the 26th and I need to get the release out before then. Rob 1300148808.0 From alanpost at sunflowerriver.org Tue Mar 15 02:01:39 2011 From: alanpost at sunflowerriver.org (Alan Post) Date: Mon, 14 Mar 2011 20:01:39 -0600 Subject: [Aboriginal] /bin/sh error compiling distcc In-Reply-To: <4D7EA660.8000100@parallels.com> References: <20110314130503.GA40390@alice.local> <4D7EA660.8000100@parallels.com> Message-ID: <20110315020139.GA47940@alice.local> On Mon, Mar 14, 2011 at 06:36:00PM -0500, Rob Landley wrote: > On 03/14/2011 08:05 AM, Alan Post wrote: > > I'm trying to build aboriginal 1.0.1 inside the i486 system image. > > The packages download and I get to the next stage, building host > > tools. > > Sigh. It used to self-host, I've been a bit distracted by the pending > uClibc-NPTL release and the growing list of platform regressions (sparc, > sh4, and mips64 have all been really close to working, but then they > float away again as new package releases come out...) > > > busybox appears to compile fine. > > I beat the busybox guys up when they break stuff. :) > > > I'm getting the following error > > message tryitg to build distcc: > > > > === Building host-tools > > === distcc (host host-tools) > > Snapshot 'distcc'... > > configure: WARNING: you should use --build, --host, --target > > configure: WARNING: invalid host type: "--with-included-popt--disable-Werror" > > checking build system type... Invalid configuration `"--with-included-popt--disable-Werror"': machine `"--with-included-popt--disable' not recognized > > configure: error: /bin/bash ./config.sub "--with-included-popt--disable-Werror" failed > > > > Exiting due to errors (host host-tools distcc) > > > > > > Somewhere in the configure stage for distcc, the two configure options: > > '--with-included-popt' and '--disable-Werror' are being joined together. > > It's trying to use the wrong shell. (None of the busybox shells > actually work quite right yet. Making hush into a decent bash > replacement is a major todo item of mine if I ever get a spare 6 months.) > > Sigh, it looks like I screwed up the sequencing and /bin/sh isn't > pointing to bash. It should be. I'll fix it. > > > > Going into the host-temp/distcc build directory, I try running the command > > by hand. I get the same error message. This suggests to me that I > > have a bug in /bin/sh in busybox. > > The bug is that /bin/sh points to busybox. There's a reason I install > bash. I'd love to get rid of it, but busybox needs a major upgrade for > that to happen. (Denys and I agree that "hush" is the shell to work on, > we just need to find the time...) > > > Have I found the best workaround? I'd also like to find and fix the > > underlying issue; that will require more testing on my part; I want > > to be able to compile the whole system first. > > Fix the symlink in root-filesystem and re-run system-image.sh. > I noticed that both busybox and bash try to install /bin/sh, and it does look like busybox wins. I used a debian system to build my own system image, modifying the configuration to use ext2. This allowed me to update the /bin/sh symlink. The system is compiling now; it has been all day. I haven't yet set up distcc, but I will be soon; it's painful waiting for the system to compile inside qemu! I wondered at your adding distcc; I'm 100% on board with this idea now. I was going to wait to see if everything compiled fine. I did get past the issue I reported this morning and things seem to be running fine. I'd like to add support for a custom busybox .config, rather than building always with defconfig. Is this the sort of thing that would be welcome? -Alan -- .i ma'a lo bradi ku penmi gi'e du 1300154499.0 From rlandley at parallels.com Tue Mar 15 04:29:25 2011 From: rlandley at parallels.com (Rob Landley) Date: Mon, 14 Mar 2011 23:29:25 -0500 Subject: [Aboriginal] /bin/sh error compiling distcc In-Reply-To: <20110315020139.GA47940@alice.local> References: <20110314130503.GA40390@alice.local> <4D7EA660.8000100@parallels.com> <20110315020139.GA47940@alice.local> Message-ID: <4D7EEB25.3040101@parallels.com> On 03/14/2011 09:01 PM, Alan Post wrote: > I'd like to add support for a custom busybox .config, rather than > building always with defconfig. Is this the sort of thing that > would be welcome? I used to use a custom busybox .config, but I poked enough config changes upstream that I could go with defconfig. The general design idea of Aboriginal Linux is that we build a native system, and then you can build anything you like natively. In busybox, defconfig is generally "allyesconfig minus the broken stuff like selinux or debug options that prevent it from building/working on normal systems", so if that doesn't have everything you need it's a bug in busybox. (And yes, hush being crap is a bug in busybox, but a rather large one that's not easy to fix.) Note that system-image.sh can produce writeable ext2 filesystems (SYSIMAGE_TYPE=ext2 in config), but I don't ship prebuilt versoins those because there are various cans of worms like "when to fsck" and diagnosing bugs due to people modifying their root filesystem that I don't want to get into. (Plus you can mount multiple simultaneous instances of a read only root filesystem, with different hdb images in different qemu/kvm invocations.) But the real deciding factor for not shipping system-image-$ARCH.ext3 tarballs is I made a script to copy the root filesystem into a writeable chroot under your /home directory (in the /dev/hdb partition). It's part of the control-image skeleton code, I suspect I should just copy it into the root filesystem and have it available as part of th standard toolkit. Then if you want to tweak your dev environment in-situ, it's easy. :) Sound like a thing? Then you can build your own busybox on target and swap it in. (That's the design idea: get you a generic development environment on target, then natively fiddle with it all you like.) I need to do an x.org vnc server setup howto. (Or a script. I did it on the Qualcomm Hexagon a few months ago, before my day job was hunting the great white NFS containers fix...) Rob 1300163365.0 From alanpost at sunflowerriver.org Wed Mar 16 12:48:18 2011 From: alanpost at sunflowerriver.org (Alan Post) Date: Wed, 16 Mar 2011 06:48:18 -0600 Subject: [Aboriginal] [PATCH] documentation typos Message-ID: <20110316124818.GA52995@alice.local> I noticed a few typos while reading the documentation. I went through the docs with a spell checker and fixed what I found. -Alan -- .i ma'a lo bradi ku penmi gi'e du -------------- next part -------------- diff -r 29e1df07e99a -r a1af0f19d248 config --- a/config Wed Feb 09 21:06:18 2011 -0600 +++ b/config Thu Mar 10 09:07:26 2011 -0700 @@ -1,6 +1,6 @@ # Setup -# This file is sourced, not run. It provides a place for you to persisently +# This file is sourced, not run. It provides a place for you to persistently # set configuration variables. # Feel free to replace this file with your own version, or to set these @@ -9,7 +9,7 @@ ############################################################################# -# Seting this tells build.sh not to build a native toolchain. (This yields +# Setting this tells build.sh not to build a native toolchain. (This yields # a much smaller root-filesystem tarball.) # export NO_NATIVE_COMPILER=1 @@ -134,7 +134,7 @@ # If this is set, it's points to a directory containing additional files to # copy into the simple-root-filesystem. The path is either absolute or -# relavitve to the aboriginal topdir. +# relative to the aboriginal topdir. # export SIMPLE_ROOT_OVERLAY="overlay" diff -r 29e1df07e99a -r a1af0f19d248 www/README --- a/www/README Wed Feb 09 21:06:18 2011 -0600 +++ b/www/README Thu Mar 10 09:07:26 2011 -0700 @@ -15,7 +15,7 @@ can boot under an emulator (or on real hardware if you have it) to compile code natively. This includes a full native compiler (for both C and C++), and the (optional) ability to run the cross compiler on the host (via distcc) - to speed up the build without reqiring the packages you're building to + to speed up the build without requiring the packages you're building to know anything about cross compiling. --- How do I try it out? (Quick start.) @@ -135,7 +135,7 @@ The file "configure" contains several environment variables you can set to control the behavior of Aboriginal Linux, and has comments documenting what - the all doo. (You can persistently set them by altering this file, or + they all do. (You can persistently set them by altering this file, or set them temporarily in your environment before running a build.) For more information, see the README file in the download directory. diff -r 29e1df07e99a -r a1af0f19d248 www/about.html --- a/www/about.html Wed Feb 09 21:06:18 2011 -0600 +++ b/www/about.html Thu Mar 10 09:07:26 2011 -0700 @@ -20,7 +20,7 @@ -

Also, stock up on asprin and clear a space to beat your head against; you'll +

Also, stock up on aspirin and clear a space to beat your head against; you'll need both. See why cross compiling sucks for more details.

@@ -253,7 +253,7 @@
  • package-mini-native.sh $TARGET
    • -

    In theory, the stages are othogonal. If you have an existing cross +

    In theory, the stages are orthogonal. If you have an existing cross compiler, you can add it to the $PATH and skip cross-compiler.sh. Or you can use _just_ cross-compiler.sh to create a cross compiler, and then go build something else with it. The host-tools.sh stage can often be skipped @@ -307,7 +307,7 @@ installed on the host.

  • Portability - Using a known set of command line utilities -insulates the build from variations in the host's Linux distribtion (such as +insulates the build from variations in the host's Linux distribution (such as Ubuntu's /bin/echo lacking suport for the -e option).

  • Testing - It ensures the resulting system can rebuild itself @@ -317,7 +317,7 @@ restricting $PATH ensures that no necessary commands are missing. (Variation can still show up between x86/arm/powerpc versions, of course.)

    -

    It also moves most failures to the beginning. If anything's going to +

    It also moves most failures to the beginning. If anything is going to break, it's usually the host-tools build. After that runs, we're mostly in a known and tested state.

    • @@ -380,7 +380,7 @@

  • mini-native.sh - Use the cross compiler to create -a minimal native build environment for the target platfrom.

    +a minimal native build environment for the target platform.

    This script takes one argument: the architecture to build for.

    @@ -653,7 +653,7 @@

  • cleanup - delete temporary copy of source code after build. Treat as a fancy "rm -rf" (except that it remembers the working directory -from the last setupfor call, rather than reqiring it to be specified on +from the last setupfor call, rather than requiring it to be specified on the command line).

    If the exit code of the last command was nonzero, @@ -800,7 +800,7 @@ things like "uClibc" and "uClibc++". Packages are allowed to contain dashes (such as gcc-core), but cannot have a digit immediately after the dash.

    -

    FWL implements source cacheing. The first call to setupfor extracts the +

    FWL implements source caching. The first call to setupfor extracts the package into build/sources, and then creates a directory of hard links in the current target's build/temp-$TARGET directory with cp -lfR. Later setupfor calls just create the directory of hard links from the existing source tree. @@ -902,7 +902,7 @@

    The way $RECORD_COMMANDS works is by building a logging wrapper (sources/toys/wrappy.c) and populating a directory (build/wrapper) with symlinks to that logging wrapper for each command name in $PATH. When later -build stages run commands, the wraper appends the command line to the log file +build stages run commands, the wrapper appends the command line to the log file (specified in the environment variable $WRAPPY_LOGPATH, host-tools.sh sets this to "$BUILD/cmdlines.$STAGE_NAME.$PACKAGE_NAME"), recording each command run. The logging wrapper then searches $WRAPPY_REALPATH to find @@ -1102,7 +1102,7 @@

  • busybox - Build busybox command line utilities.

    -

    This provides the bulk of the command line utlities for the new +

    This provides the bulk of the command line utilities for the new system.

    Once upon a time, "make defconfig" provided the largest sane configuration @@ -1222,7 +1222,7 @@

  • distcc - command to distribute compiles across a network cluster.

    • -

    We install this for the distc accelerator +

    We install this for the distcc accelerator trick. It's entirely optional.

    We create a $TOOLS/distcc directory full of symlinks to distcc with the @@ -1523,7 +1523,7 @@ hard to do.

    I wrote an introduction -to cross-compiling which explains the terminology, plusses and minuses, +to cross-compiling which explains the terminology, pluses and minuses, and why you might want to do it. Keep in mind that I wrote that for a company that specializes in cross-compiling. Personally, I consider cross-compiling a necessary evil to be minimized, and that's how Firmware Linux is designed. @@ -1601,14 +1601,14 @@ the user home directories lived. When they got more disk space, they created a new direct (/home) and moved all the user home directories there.

    -

    The real reason we kept it is tradition. The execuse is that the root +

    The real reason we kept it is tradition. The excuse is that the root partition contains early boot stuff and /usr may get mounted later, but these days we use initial ramdisks (initrd and initramfs) to handle that sort of thing. The version skew issues of actually trying to mix and match different versions of /lib/libc.so.* living on a local hard drive with a /usr/bin/* from the network mount are not pretty.

    -

    I.E. The seperation is just a historical relic, and I've consolidated it in +

    I.E. The separation is just a historical relic, and I've consolidated it in the name of simplicity.

    On a related note, there's no reason for "/opt". After the original Unix @@ -1652,7 +1652,7 @@ just symlink them together.

    (I case it hasn't become apparent yet, there's 30 years of accumulated cruft -in the standards, convering a lot of cases that don't apply outside of +in the standards, covering a lot of cases that don't apply outside of supercomputing centers where 500 people share accounts on a mainframe that has a dedicated support staff. They serve no purpose on a laptop, let alone an embedded system.)

    diff -r 29e1df07e99a -r a1af0f19d248 www/history.html --- a/www/history.html Wed Feb 09 21:06:18 2011 -0600 +++ b/www/history.html Thu Mar 10 09:07:26 2011 -0700 @@ -19,7 +19,7 @@

    Back in the late 90's, before linksys routers came out, I installed several masquerading gateways by putting Red Hat on old leftover 386 machines. This -involved removing as many packages as possible from the bas install, both to +involved removing as many packages as possible from the base install, both to get the size down (to fit it on old hard drives) and to reduce the security exposure of all the daemons Red Hat ran by default (including a print server and an NFS server exposed to the world, for no readily apparent @@ -180,7 +180,7 @@ entire new OS images using the internet speeds of 2001, and then update the new OS image without losing data or turning it into a brick.

    -

    WebOffice's original prototype device ran a stock Red Hat 7 intall (the one +

    WebOffice's original prototype device ran a stock Red Hat 7 install (the one that had the Code Red virus when they first demoed it to me after a LUG meeting). The whole OS image took up almost a gigabyte, and that's before they'd implemented any applications or web UI. I rebased the system on Linux @@ -258,7 +258,7 @@

    At a technical level, there was a chicken and egg problem here: the root filesystem was a loopback mount, but the file to loopback mount has to live -somewhere. So the system needed a writeable parition for logging and +somewhere. So the system needed a writeable partition for logging and such anyway, so I made /dev/hda1 be ext3 and mounted it on /var, and put the firmware in that. So during the boot process the initrd needed to mount /dev/hda1 onto a /temp directory, loopback mount the /temp/image file onto a @@ -286,7 +286,7 @@ we persevered.

    Mounting a partition twice and leaking one of the mount points (the old -/temp was inaccessable after the pivot_root) was kind of unclean anyway, the +/temp was inaccessible after the pivot_root) was kind of unclean anyway, the clean thing for the boot to do was actually move the /tmp mount to /sub/var after mounting /sub but before the pivot_root into /sub. But when I asked on linux-kernel how to do that, I was told that "mount --move" didn't exist @@ -316,7 +316,7 @@ except the python CGI) once it shipped never came true. Partly the founders were ambivalent about this whole "open source" thing, wanting every competitive advantage they could get. (They kept trying to patent obvious things I did. -Their patent lawyer was a realy cool dude when he flew in from California.)

    +Their patent lawyer was a really cool dude when he flew in from California.)

    Another contributing factor was that the founders were from Taiwan and had no idea how to address the US market. Their marketer employee #3 hadn't @@ -367,7 +367,7 @@ necessity would make them focus on shipping units and bringing in revenue, but it didn't happen. I left in November and spent the last couple months of that year in Florida watching my mother die of cancer, then driving around the -country distributing her posessions to various relatives, and finally crashing +country distributing her possessions to various relatives, and finally crashing on Eric Raymond's couch for a few months doing an "editing pass" on The Art of Unix Programming that expanded the book from 9 chapters to 20.

    @@ -493,7 +493,7 @@ ncurses, net-tools, patch, perl, procinfo, procps, psmisc, sed, shadow, sysklogd, sysvinit, tar, tcl, texinfo, util-linux, vim, and zlib. There were also two LFS-specific packages, providing boot scripts, config -files, and miscelaneous utilities.

    +files, and miscellaneous utilities.

    I started by removing packages I didn't actually need. Tcl, expect, and dejagnu hadn't been in LFS 4, so obviously it was possible to do without them. @@ -622,13 +622,13 @@ e2fsprogs, file, findutils, gawk, grep, inetutils, less, modutils, net-tools, patch, procps, sed, shadow, sysklogd, sysvinit, tar, util-linux, and vim.

    -

    To do that, I wound up extensively upggrading (or rewriting from scratch) +

    To do that, I wound up extensively upgrading (or rewriting from scratch) dozens of different busybox commands and adding several new ones from scratch.

    One new command was switch_root, for initramfs support. I wrote an initramfs file for the kernel's Documentation directory because I investigated -it for Firmware Linux. (And later gave my first OLS presenation on the topic, +it for Firmware Linux. (And later gave my first OLS presentation on the topic, and write an article series about it when I worked at TimeSys.)

    Another new command was mdev, which was a rewrite of @@ -699,8 +699,7 @@ root access (enough for my needs), but without extensive setup thanks to the "hostfs", and without requiring a new source package (I already had the Linux kernel sources, this was just another way of building them). -I first got it working in the -first got it working in +I first got it working in a patched 2.6.9 kernel, and later integrated it into the build when I got unpatched 2.6.11 to build a usable User Mode Linux image (although I had to @@ -730,7 +729,7 @@ initially understand), made a number of design suggestions, -and even oftered patches.

    +and even offered patches.

    But I didn't really like the design of buildroot (nested makefiles aren't the most straightforward approach to anything, @@ -753,7 +752,7 @@ and wouldn't load without it, even if it was otherwise linked against uClibc.

    -

    This meant the gcc wraper was no longer sufficient, but since libgcc_s.so was +

    This meant the gcc wrapper was no longer sufficient, but since libgcc_s.so was part of gcc, the only way to get a new version of libgcc_s.so that linked against uClibc (instead of glibc) was to download the gcc source code and build gcc itself against uClibc.

    @@ -769,7 +768,7 @@ a new uClibc root filesystem directory containing the uClibc shared libraries, and a second instance of busybox dynamically linked against those, which you could chroot and test out the uClibc version of busybox. -Since he already had the kernel source lying aorund, Eric even taught it to +Since he already had the kernel source lying around, Eric even taught it to build a User Mode Linux binary that could do the chroot for you.

    Problems with buildroot

    @@ -780,7 +779,7 @@ initially understand), made a number of design suggestions, -and even oftered patches.

    +and even offered patches.

    But I didn't really like the design of buildroot (nested makefiles aren't the most straightforward approach to anything, @@ -797,7 +796,7 @@ several years old by that point). Buildroot was an instructive counterexample on many fronts: my project should not require root access to build, shell scripts were more readable than nested makefiles, releases were -important, and it's vital to have boundaries so youc an clearly state +important, and it's vital to have boundaries so you can clearly state what your project DOESN'T do.

    I also checked in with buildroot from time to time to see what it was doing @@ -937,7 +936,7 @@

    I started playing around -with QEMU in novemeber 2005, about the same time I started keeping +with QEMU in november 2005, about the same time I started keeping a more detailed technical blog on my own website, and shortly before I went to work for diff -r 29e1df07e99a -r a1af0f19d248 www/news.html --- a/www/news.html Wed Feb 09 21:06:18 2011 -0600 +++ b/www/news.html Thu Mar 10 09:07:26 2011 -0700 @@ -271,7 +271,7 @@

  • build-control-images.sh - build all control images from sources/native-builds.
    • -
  • for-each-target.sh - iterage through every defined target architecture +
  • for-each-target.sh - iterate through every defined target architecture and run a command.
    • @@ -296,7 +296,7 @@

    Environment variable sanitization: the sanitize_environment function unsets any environment variable it doesn't recognize, either from a whitelist or from config. Clearing all unrecognized environment variables insulates -the build from some of the strage environments out there. To disable this +the build from some of the strange environments out there. To disable this behavior, export NO_SANITIZE_ENVIRONMENT=1.

    New config variables:

    @@ -553,13 +553,13 @@ of one of the unstable packages (linux, uClibc, or busybox). It can test that the build completes, or that the system image boots and successfully builds the static native dropbear binary. (In theory this will allow the -nightly snapshots to automaticaly isolate and report the first commit that +nightly snapshots to automatically isolate and report the first commit that broke them.)

    In other general infrastructure news: some extensive upgrades to the compiler wrapper (ccwrap), such as no longer needing GIMME_AN_S at compile time and instead autodetecting the presence of libgcc_s.so vs libgcc.a -and behaving appropriately. A largeish cleanup/refactoring of the compiler +and behaving appropriately. A largish cleanup/refactoring of the compiler build (described in commit 944) added a new "native-compiler.sh" script, which handles building the @@ -650,7 +650,7 @@ use the right one, rather than moving them around during the build. Updated smoketest-all.sh to show "NONE" for system images that didn't build. Added more support for SKIP_STRIP and CFLAGS=-g to the rest of the -packages. Cleaned out the old USE_COLOR stuff which was superceded by +packages. Cleaned out the old USE_COLOR stuff which was superseded by set_titlebar.

    @@ -681,7 +681,7 @@ tracker has a roadmap showing what we plan to implement for the next release or two.

    -

    It's also a place to file bugs, athough sending them to the mailing list +

    It's also a place to file bugs, although sending them to the mailing list is still the fastest way to get a fix.

    @@ -976,7 +976,7 @@ the last GPLv2 release of Red Hat's binutils fork, 2.17.5.0.17 is under consideration for a future release. But mostly the GNU compiler toolchain -has been upgraded as far as possible for the forseeable future, either because +has been upgraded as far as possible for the foreseeable future, either because new versions provide no significant benefits or because they're not available under a license compatible with the Linux kernel's.

    @@ -1115,7 +1115,7 @@ stripping of the cross compilers and system images, so everything's a bit smaller. The mirror list now lives in download.sh. New buildall.sh is a wrapper around build.sh. New build-static-toolchains.sh builds them under -qemu; set FORK=1 to build them in paralell just like buildall.sh. The +qemu; set FORK=1 to build them in parallel just like buildall.sh. The ext2 system images are now built at a small size with genext2fs and then expanded with resize2fs (because genext2fs slows down exponentially as size increases; this means e2fsprogs is built in host-tools.sh now). diff -r 29e1df07e99a -r a1af0f19d248 www/presentation.html --- a/www/presentation.html Wed Feb 09 21:06:18 2011 -0600 +++ b/www/presentation.html Thu Mar 10 09:07:26 2011 -0700 @@ -245,7 +245,7 @@
  • Building on NFS is painful -
    • Timestamp granularity and skew due to cacheing
      • +
      • Timestamp granularity and skew due to caching
    • From rob at landley.net Tue Mar 22 11:56:42 2011 From: rob at landley.net (Rob Landley) Date: Tue, 22 Mar 2011 06:56:42 -0500 Subject: [Aboriginal] [PATCH] documentation typos In-Reply-To: <20110316124818.GA52995@alice.local> References: <20110316124818.GA52995@alice.local> Message-ID: <4D888E7A.9070404@landley.net> On 03/16/2011 07:48 AM, Alan Post wrote: > I noticed a few typos while reading the documentation. I went > through the docs with a spell checker and fixed what I found. > > -Alan Committed, sorry for the delay. If you noticed Friday's flurry of commits, I've been testing uClibc-rc3. It's _just_ good enough that I don't want to revert to ship the release, but not quite working well enough to ship. Sigh... The new dropbear release is also horked (doesn't like static linking even though he shipped a .1 that _supposedly_ fixed this, looks like I've got to track it down and patch it myself. And there's a new squashfs release out I should test too...) But the merge window is open and my day job is containerizing NFS, trying to get the next patch at least posted before -rc1 ships. I fly to Parallels head office in Moscow on the 26th for a week, be embarassing not to have that done before I report to The Mothership... Rob 1300795002.0 From alanpost at sunflowerriver.org Wed Mar 23 00:31:22 2011 From: alanpost at sunflowerriver.org (Alan Post) Date: Tue, 22 Mar 2011 18:31:22 -0600 Subject: [Aboriginal] [PATCH] documentation typos In-Reply-To: <4D888E7A.9070404@landley.net> References: <20110316124818.GA52995@alice.local> <4D888E7A.9070404@landley.net> Message-ID: <20110323003122.GA74409@alice.local> On Tue, Mar 22, 2011 at 06:56:42AM -0500, Rob Landley wrote: > On 03/16/2011 07:48 AM, Alan Post wrote: > > I noticed a few typos while reading the documentation. I went > > through the docs with a spell checker and fixed what I found. > > > > -Alan > > Committed, sorry for the delay. > > If you noticed Friday's flurry of commits, I've been testing uClibc-rc3. > It's _just_ good enough that I don't want to revert to ship the > release, but not quite working well enough to ship. Sigh... > > The new dropbear release is also horked (doesn't like static linking > even though he shipped a .1 that _supposedly_ fixed this, looks like > I've got to track it down and patch it myself. And there's a new > squashfs release out I should test too...) > > But the merge window is open and my day job is containerizing NFS, > trying to get the next patch at least posted before -rc1 ships. I fly > to Parallels head office in Moscow on the 26th for a week, be > embarassing not to have that done before I report to The Mothership... > > Rob I've been avoiding anything past your latest tarball release because I suspected there was a fair bit of instability in tip. I've been playing around with and getting a feel for Aboriginal Linux. I haven't build my first testing image yet; I have a few things I want to play with first. -Alan -- .i ma'a lo bradi ku penmi gi'e du 1300840282.0 From rlandley at parallels.com Wed Mar 23 07:02:55 2011 From: rlandley at parallels.com (Rob Landley) Date: Wed, 23 Mar 2011 02:02:55 -0500 Subject: [Aboriginal] [PATCH] documentation typos In-Reply-To: <20110323003122.GA74409@alice.local> References: <20110316124818.GA52995@alice.local> <4D888E7A.9070404@landley.net> <20110323003122.GA74409@alice.local> Message-ID: <4D899B1F.5040503@parallels.com> On 03/22/2011 07:31 PM, Alan Post wrote: > I've been avoiding anything past your latest tarball release because > I suspected there was a fair bit of instability in tip. I've been > playing around with and getting a feel for Aboriginal Linux. I > haven't build my first testing image yet; I have a few things I want > to play with first. > > -Alan There's a ginormous presentation.pdf in the downloads directory, the raw text of which is in presentation.html one level up, which tries to explain the design I was going for. I can give you the design rant if you like, but presumably reading that would give it as well. (And for all I remember it's in the FAQ. :) Rob 1300863775.0 From onefang at gmail.com Wed Mar 23 14:01:28 2011 From: onefang at gmail.com (David Seikel) Date: Thu, 24 Mar 2011 00:01:28 +1000 Subject: [Aboriginal] Back to work. Message-ID: <20110324000128.3d2a1e52@U64-desktop> A) I finally found out this list got moved. 2) The client finally pulled his finger out and is now serious about the project I want to use aboriginal linux for. III) The actual aboriginal living downstairs has so far decided he does not object to the name, though more likely he does not know about it. In the end the client seems to have decided to go for generic x86 systems, not the tiny 486 board I was looking at initially. Performance is not a big issue though, anything better than a 486 is way more than we actually need. Plus, I still have copies of that 486 board that I want to use for other things. So I'll go ahead and keep working on that 486 target I had previously contributed. I'll also make a variant that works on the 486 AND generic x86s. Like I said, performance is not critical. I know the client might have trouble coming up with the money, that's why I make him pay monthly in advance. B-) At least the initial work I'll be able to use for other projects, so it wont be until he has invested heavily in the project and it is near the end that we get into the area of "not useful for others". So some of the work I do I can contribute back to the aboriginal linux project. It's good to be back. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From rlandley at parallels.com Wed Mar 23 23:17:08 2011 From: rlandley at parallels.com (Rob Landley) Date: Wed, 23 Mar 2011 18:17:08 -0500 Subject: [Aboriginal] Back to work. In-Reply-To: <20110324000128.3d2a1e52@U64-desktop> References: <20110324000128.3d2a1e52@U64-desktop> Message-ID: <4D8A7F74.6080104@parallels.com> On 03/23/2011 09:01 AM, David Seikel wrote: > A) I finally found out this list got moved. 2) The client finally > pulled his finger out and is now serious about the project I want to > use aboriginal linux for. Yay! > III) The actual aboriginal living downstairs > has so far decided he does not object to the name, though more likely > he does not know about it. The word was coined in 1660, New South Wales colony wasn't established until 1788, and it's still not funny. > In the end the client seems to have decided to go for generic x86 > systems, not the tiny 486 board I was looking at initially. > Performance is not a big issue though, anything better than a 486 is > way more than we actually need. Plus, I still have copies of that 486 > board that I want to use for other things. So I'll go ahead and keep > working on that 486 target I had previously contributed. I'll also > make a variant that works on the 486 AND generic x86s. Like I said, > performance is not critical. As far as I know the 486 version does work on generic x86s? (Or do you mean there's still an actual 386 system in use somewhere?) > I know the client might have trouble coming up with the money, that's > why I make him pay monthly in advance. B-) > > At least the initial work I'll be able to use for other projects, so it > wont be until he has invested heavily in the project and it is near the > end that we get into the area of "not useful for others". So some of > the work I do I can contribute back to the aboriginal linux project. > > It's good to be back. It's good to have you back. The current repository version still has numerous regressions from last release because uClibc-rc3 is iffy. Let's see: armv4l is stuck on sjlj but uClibc-NPTL only supports dwarf stack unwinding, I think I may have to keep that arch using old pthreads. For some reason x86-64 isn't building some of its code with -fpic and the link is failing creating the library. Powerpc builds but segfaults immediately. I need to track down the runtime problems with sparc32 and sh4. And then on the targets that _do_ build, building lfs natively makes it as far as this bit of fun (longish thread): http://lists.uclibc.org/pipermail/uclibc/2011-March/045041.html (There are times I really miss Erik Anderson's participation in Embedded Linux.) Rob 1300922228.0 From onefang at gmail.com Mon Mar 28 22:39:51 2011 From: onefang at gmail.com (David Seikel) Date: Tue, 29 Mar 2011 08:39:51 +1000 Subject: [Aboriginal] Back to work. In-Reply-To: <4D8A7F74.6080104@parallels.com> References: <20110324000128.3d2a1e52@U64-desktop> <4D8A7F74.6080104@parallels.com> Message-ID: <20110329083951.32b845a6@U64-desktop> On Wed, 23 Mar 2011 18:17:08 -0500 Rob Landley wrote: > On 03/23/2011 09:01 AM, David Seikel wrote: > > In the end the client seems to have decided to go for generic x86 > > systems, not the tiny 486 board I was looking at initially. > > Performance is not a big issue though, anything better than a 486 is > > way more than we actually need. Plus, I still have copies of that > > 486 board that I want to use for other things. So I'll go ahead and > > keep working on that 486 target I had previously contributed. I'll > > also make a variant that works on the 486 AND generic x86s. Like I > > said, performance is not critical. > > As far as I know the 486 version does work on generic x86s? (Or do > you mean there's still an actual 386 system in use somewhere?) Nope, not 386. I'm seeing kernel panics on the old one I made last year when running it on things more powerful than that 486. I'll see if it happens with the latest version. Something for me to investigate and send patches if needed. After playing with it a bit, but only running things under QEMU, looks like I'll start with the lfs bootstrap stuff and modify that a bit. Suits my project perfectly. Been working through the night, I'm dead tired, but I can't seem to see if there is a way to actually run the results of building the lfs bootstrap, under QEMU easily. At least without hacking something up. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From rlandley at parallels.com Tue Mar 29 10:07:52 2011 From: rlandley at parallels.com (Rob Landley) Date: Tue, 29 Mar 2011 05:07:52 -0500 Subject: [Aboriginal] Back to work. In-Reply-To: <20110329083951.32b845a6@U64-desktop> References: <20110324000128.3d2a1e52@U64-desktop> <4D8A7F74.6080104@parallels.com> <20110329083951.32b845a6@U64-desktop> Message-ID: <4D91AF78.6090000@parallels.com> On 03/28/2011 05:39 PM, David Seikel wrote: > On Wed, 23 Mar 2011 18:17:08 -0500 Rob Landley > wrote: > >> On 03/23/2011 09:01 AM, David Seikel wrote: >>> In the end the client seems to have decided to go for generic x86 >>> systems, not the tiny 486 board I was looking at initially. >>> Performance is not a big issue though, anything better than a 486 is >>> way more than we actually need. Plus, I still have copies of that >>> 486 board that I want to use for other things. So I'll go ahead and >>> keep working on that 486 target I had previously contributed. I'll >>> also make a variant that works on the 486 AND generic x86s. Like I >>> said, performance is not critical. >> >> As far as I know the 486 version does work on generic x86s? (Or do >> you mean there's still an actual 386 system in use somewhere?) > > Nope, not 386. I'm seeing kernel panics on the old one I made last > year when running it on things more powerful than that 486. I'll see > if it happens with the latest version. Something for me to investigate > and send patches if needed. Odd. I haven't seen these panics, but I haven't really been banging on the 486 target very heavily (or running it outside of qemu). > After playing with it a bit, but only running things under QEMU, looks > like I'll start with the lfs bootstrap stuff and modify that a bit. > Suits my project perfectly. Woot. On my todo list are making a gentoo bootstrap and a debian bootstrap, but I need to finish the uClibc transition first. > Been working through the night, I'm dead tired, but I can't seem to see > if there is a way to actually run the results of building the lfs > bootstrap, under QEMU easily. At least without hacking something up. Um... It makes a tarball, and uploads it to the host. (All the native builds do that.) Afterwards, you can: 1) Run dev-environment.sh reusing the hdb.img file, find the appropriate directory under /home and chroot into it. (I think your chroot can run sbin/init.sh and it'll detect it's not PID 1 and do chroot setup instead of fresh boot setup, but I haven't tested that since last year.) 2) wget the tarball into a fresh /home directory, extract it, and chroot into it there. 3) Play around with genext2fs or loopback on the host to make your own bootable ext2 image. (I should make a script to convert a tarball to an ext2 or squashfs image, but things like /dev nodes and suid bits are nontrivial to handle as non-root. It can be done, but parsing them out of a tarball... Hmmm, actually I could do tar txv and have the script parse the output to generate a device table for genext2fs... But then I have to extract the tarball in such a way that errors from being unable to create dev nodes don't kill the script but errors from the tarball being corrupt are...) Might be a fun todo item, but not a particularly small one. And if I got it to work of course I'd want to redo system-image.sh to take advantage of it... Rob 1301393272.0 From onefang at gmail.com Tue Mar 29 10:56:14 2011 From: onefang at gmail.com (David Seikel) Date: Tue, 29 Mar 2011 20:56:14 +1000 Subject: [Aboriginal] Back to work. In-Reply-To: <4D91AF78.6090000@parallels.com> References: <20110324000128.3d2a1e52@U64-desktop> <4D8A7F74.6080104@parallels.com> <20110329083951.32b845a6@U64-desktop> <4D91AF78.6090000@parallels.com> Message-ID: <20110329205614.526f524f@U64-desktop> On Tue, 29 Mar 2011 05:07:52 -0500 Rob Landley wrote: > On 03/28/2011 05:39 PM, David Seikel wrote: > > After playing with it a bit, but only running things under QEMU, > > looks like I'll start with the lfs bootstrap stuff and modify that > > a bit. Suits my project perfectly. > > Woot. > > On my todo list are making a gentoo bootstrap and a debian bootstrap, > but I need to finish the uClibc transition first. Think I'll make a uLFS bootstrap. Like the LFS one you made, but keeping busybox versions of as much as possible. There's a few things I need from BLFS, adding those can act as a sanity test. > > Been working through the night, I'm dead tired, but I can't seem to > > see if there is a way to actually run the results of building the > > lfs bootstrap, under QEMU easily. At least without hacking > > something up. > > Um... > > It makes a tarball, and uploads it to the host. (All the native > builds do that.) Afterwards, you can: > > 1) Run dev-environment.sh reusing the hdb.img file, find the > appropriate directory under /home and chroot into it. Yep, that worked. Said I was tired. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From onefang at gmail.com Thu Mar 31 08:08:47 2011 From: onefang at gmail.com (David Seikel) Date: Thu, 31 Mar 2011 18:08:47 +1000 Subject: [Aboriginal] ulfs-bootstrap, shrinking LFS. Message-ID: <20110331180847.1e70e680@U64-desktop> ulfs is my effort to shrink LFS down, mostly to suit my own purposes, by using the stuff that already exists in Aboriginal linux. So, no need for most of the usual gnu tools when we have busybox, no need to replace the tool chain we already got, and see how far we can get using uClibc. The result should be able to compile stuff from BLFS, though I'll only be testing the stuff from BLFS that I actually need when I need it. Mostly this is by removing packages, if the removed package turns out to be something you need, just put it back again. First thing I did was remove gmp, mpfr, and mpc. They are complex maths packages that are only in LFS coz the latest gcc needs them. We are not using a gcc that needs them, we already have an older one that works. The original lfs-bootstrap had already removed the building of the toolchain coz aboriginal linux already supplies one. For the same reason make was left out, we already got that. Next thing I did was to leave out packages that busybox could be used for. This step removed sed, util-linux-ng, e2fsprogs, coreutils, gawk, procps, grep, inetutils, bzip2, diffutils, findutils, gzip, iproute2, kbd, less, module-init-tools, patch, psmisc, shadow, sysklogd, sysvinit, tar, udev, and vim. Util-linux-ng includes a lot of stuff that busybox does not, we shall see if it's really needed. First problem, iana-etc needs the real gawk, it uses fancy options that the busybox awk does not support. So put gawk back in. Second problem, autoconf stops part way through asking the user a question when using busybox instead of coreutils. A quick investigation seems to point to the busybox mv command needing to know if it's OK to overwrite a file, when the coreutils one just overwrites it without asking. There may in fact be some deeper root cause for this problem, I did not look deeper. See next step for why. Next step is to remove autofoo and friends. The general purpose of autofoo is to let developers create source tarballs that users can compile on any old unix system. The target systems don't need autofoo installed to allow such tarballs to compile. So autofoo itself is a development tool, and LFS is not supposed to be a development system, but a way to get a linux system up and running which can be used to install more packages. So, autofoo is not really needed in an LFS system. Removing it is good, and lets me get rid of perl, which is even better. B-) Grub was left off for the same reason it was left out of lfs-bootstrap. Bash and readline where removed. We already have an older bash as part of aboriginal linux. Readline is only in LFS as a dependency of bash. Time will tell if this is a bad move. Libtool and texinfo where left out as being pointless development tools. I had considered leaving out bison, flex, and m4, but they ended up being needed to build those bits of LFS that remain, so left them in. This left the ulfs-bootstrap package-list looking like this - setup zlib m4 #development pkg-config #development ncurses gettext-stub #development internationalization gawk #busybox iana-etc bison #development gdbm file flex #development groff make #development man-db A much smaller system that should still allow compiling of stuff from BLFS. As a quick sanity check, I installed the one thing from BLFS that I can't live without - midnight commander. It has glib as a dependency, so that's the first thing I tried to install. It failed due to a lack of iconv. The base config for uClibc has the iconv parts commented out, uncommenting them and rebuilding fixed up glib. Midnight commander installed fine after that. And it actually worked. B-) I guess iconv was left out as it's just bloat that is not needed if you are not doing any internationalization, for the same reason gettext-stub is used instead of gettext in lfs-bootstrap. I think gentoo have a stub version of iconv they are working on, but the above was easy to do and worked for me. I'm not doing internationalization either. My next step will be to install the dependencies of the GUI library I want to use for this contract. A lot of those dependencies are in BLFS, so those will go in first using the BLFS recipes. This should help to shake out any more issues with ulfs. If people are interested, I'll ship the results to Rob later. Though the package-list above is the important bit, you can just copy the rest from lfs-bootstrap. Feel free to let me know if there are other issues with building BLFS with this ulfs-bootstrap. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From rob at landley.net Thu Mar 31 12:44:12 2011 From: rob at landley.net (Rob Landley) Date: Thu, 31 Mar 2011 07:44:12 -0500 Subject: [Aboriginal] ulfs-bootstrap, shrinking LFS. In-Reply-To: <20110331180847.1e70e680@U64-desktop> References: <20110331180847.1e70e680@U64-desktop> Message-ID: <4D94771C.7070708@landley.net> On 03/31/2011 03:08 AM, David Seikel wrote: > ulfs is my effort to shrink LFS down, mostly to suit my own purposes, > by using the stuff that already exists in Aboriginal linux. I need better documentation, I'm just not sure where to put it. If you look in sources/control-images/lfs-bootstrap/mnt/package-list you'll see I've annotated them with various tags I was thinking of using to let you switch off various package builds. (Pretty much with "grep -v".) I've got tags for busybox (this is already provided by busybox), development (only needed if you want a development system), internationaliation (only needed if you want to support internationalization, and I can link you to several of my old blog entries on my struggles to get that to work right under uClibc), and pointless (never needed by anybody for any reason, only exists because the FSF is insane). However, that's the sort of thing that could easily open "the distro can of worms" which I'm trying very hard not to do with this project. Of course if somebody ELSE wanted to take that over... :) I admit to being sort of tempted to separate the control-images stuff into its own project. (This is why I split sources/download_functions.sh out into its own file, because the control-image generator needs a copy of that.) Really, they should be able to bootstrap on top of any arbitrary native development environment. > So, no > need for most of the usual gnu tools when we have busybox, Agreed. (If the busybox tools are deficient, fix busybox.) > no need to replace the tool chain we already got, In the LFS bootstrap script I never bothered to build the new toolchain. I built its prerequisites, but skipped gcc and binutils. (I _do_ rebuilt make, for no readily apparent reason.) My annotations aren't quite complete: gmp, mpfr, and mpc are parts of gcc that the FSF broke off into separate packages in some misguided attempt to make gcc look smaller or something. In 4.2 they aren't separate packages. Bash I'm installing a newer version of. The old 2.04b is easy to build and less than 1/3 of the size of the current monstrosity, and I really want to replace it all with busybox hush. But you need current bash in order to run gentoo's portage, it uses stuff like ~= that old bash hasn't got, and depends on the subtly changed quoting rules... m4, bison, readline, autoconf, automake, flex, and pkg-config aren't provided by the existing dev environment. m4, bison, and flex are probably necessary. Readline there's a bsd-licensed replacement I'd prefer to use out of sheer spite for the FSF, and idealy busybox should export that functionality anyway. Don't get me started on autoconf/automake. pkg-config is its own horror (you do NOT improve on the rpm vs dpkg vs ebuild situation by ADDING YET ANOTHER... grumble) and either not having it installed or setting PKG_CONFIG=/bin/true is my preferred choice, butI got tired of fighting with some package builds. > and see how far we can get using > uClibc. The result should be able to compile stuff from BLFS, though > I'll only be testing the stuff from BLFS that I actually need when I > need it. I built rather large chunks of BLFS when I ported aboriginal linux to the Qualcomm Hexagon for my day job last year. (Alas, it was based on a toolchain and ginormous kernel patch that we hadn't shipped, so I could never quite merge it. They claim that SOMEDAY it'll make it past the lawyers and out into the world...) But the point is, lots of BLFS work just fine under uClibc. And I'm hoping to get more under uClibc-1.0. (In fact, once we've got NTPL and TLS I'd like to treat aborigininal Linux as the /tools system and build a new glibc based system natively, just so I don't ever have to worry about cross compiling glibc. It requires perl to build. You may barf now.) > Mostly this is by removing packages, if the removed package turns out > to be something you need, just put it back again. > > First thing I did was remove gmp, mpfr, and mpc. They are complex > maths packages that are only in LFS coz the latest gcc needs them. We > are not using a gcc that needs them, we already have an older one that > works. The original lfs-bootstrap had already removed the building of > the toolchain coz aboriginal linux already supplies one. For the same > reason make was left out, we already got that. You have to remove make from the package-list (it's currently in there), but yes, you shouldn't need it. > Next thing I did was to leave out packages that busybox could be used > for. This step removed sed, util-linux-ng, e2fsprogs, coreutils, gawk, > procps, grep, inetutils, bzip2, diffutils, findutils, gzip, iproute2, > kbd, less, module-init-tools, patch, psmisc, shadow, sysklogd, > sysvinit, tar, udev, and vim. Util-linux-ng includes a lot of stuff > that busybox does not, we shall see if it's really needed. Generally if it's really needed, it's in busybox. > First problem, iana-etc needs the real gawk, it uses fancy options that > the busybox awk does not support. So put gawk back in. Poke the busybox gawk maintainer about that. > Second problem, autoconf stops part way through asking the user a > question when using busybox instead of coreutils. A quick > investigation seems to point to the busybox mv command needing to know > if it's OK to overwrite a file, when the coreutils one just overwrites > it without asking. There may in fact be some deeper root cause for > this problem, I did not look deeper. See next step for why. I'd poke busybox about that. I also just got a fan letter about toybox (from the guy doing http://www.makelinux.com/emb/fastboot/omap ) that makes me vaguely wonder if I should start poking at that again. Something I could do is port busybox applets one at a time to toybox commands. I tried pushing toybox infrastructure upstream into busybox and that sort of worked, but not really all that well. I still can't stomach the current state of toybox, working on that bloated pile of #ifdefs is no fun and trying to clean it up is like trying to bail out the middle of the ocean: any space I clear immediately fills up with cruft again. Not that I really have time to do so, but eh. If I copied hush and started cleaning that up and extending it, I might get a reasonable bash replacement THAT way... > Next step is to remove autofoo and friends. The general purpose of > autofoo is to let developers create source tarballs that users can > compile on any old unix system. The target systems don't need autofoo > installed to allow such tarballs to compile. So autofoo itself is a > development tool, and LFS is not supposed to be a development system, > but a way to get a linux system up and running which can be used to > install more packages. So, autofoo is not really needed in an LFS > system. Removing it is good, and lets me get rid of perl, which is > even better. B-) autoconf/make is horrible. I have an entire rant about the current state of makefiles, which I'll spare you of just now. > Grub was left off for the same reason it was left out of lfs-bootstrap. I've never really addressed bootloaders, or installing stuff on real hardware. It's too system-specific, and QEMU's -kernel option allows me to handwave past the issue entirely for my use cases. :) > Bash and readline where removed. We already have an older bash as part > of aboriginal linux. Which doesn't work with portage, so I needed newer bash as part of the gentoo bootstrap anyway. > Readline is only in LFS as a dependency of bash. > Time will tell if this is a bad move. I was erring on the side of building more packages rather than less on the theory that if I could get it to build, anything that actually did need it could then use it, and driving more cybermen across the board would find more digits of pi, or something like that. ("Used to drive sheep across minefields. The principle's the same.") > Libtool and texinfo where left out as being pointless development > tools. Pointless in general, really. Libtool exists to make non-elf systems behave like elf, meaning on Linux (which has been ELF since 1995) it serves no purpose and does nothing. The fact that it fails to do nothing correctly (and thus regularly breaks builds) is a trademark move of the FSF. > I had considered leaving out bison, flex, and m4, but they > ended up being needed to build those bits of LFS that remain, so left > them in. Yup. m4 is vaguely cool, flex is a sort of sideways grep (instead of "find all occurrences of this one regex" it's "find the next occurrence of any of the regexes in this list, and tell me which one and where"). yacc/bison is something mathematicians like and which I personally find conceptually unpleasant. > This left the ulfs-bootstrap package-list looking like this - > > setup > zlib > m4 #development > pkg-config #development > ncurses > gettext-stub #development internationalization > gawk #busybox > iana-etc > bison #development > gdbm > file > flex #development > groff > make #development > man-db > > A much smaller system that should still allow compiling of stuff from > BLFS. Good luck. :) > As a quick sanity check, I installed the one thing from BLFS that I > can't live without - midnight commander. It has glib as a dependency, > so that's the first thing I tried to install. It failed due to a lack > of iconv. The base config for uClibc has the iconv parts commented > out, uncommenting them and rebuilding fixed up glib. Midnight > commander installed fine after that. And it actually worked. B-) I turned on iconv when I was building BLFS for Qualcomm, but that was on my work machine and I didn't take any of the qualcomm code home with me, so I have to reproduce all that stuff when I do a blfs-bootstrap. I may have just installed the gnu libiconv package, I don't remember. (I wasn't trying to do it RIGHT, I was trying to get the code ready to demo. Unfortunately, the demo was delayed repeatedly by toolchain problems, and although we held it right before I left and it was successful, my contract expired about a week after that and they didn't have the budget to renew it, and wouldn't get the budget renewal for at least two more months, so I found another job rather than waiting around.) I note I haven't even enabled devtmpfs in the builds yet. My todo list runneth over. :) > I guess iconv was left out as it's just bloat that is not needed if you > are not doing any internationalization, for the same reason > gettext-stub is used instead of gettext in lfs-bootstrap. I think > gentoo have a stub version of iconv they are working on, but the above > was easy to do and worked for me. I'm not doing internationalization > either. My 2010 blog (http://landley.net/notes-2010.html) has a lot of hits for iconv. > My next step will be to install the dependencies of the GUI library I > want to use for this contract. A lot of those dependencies are in BLFS, > so those will go in first using the BLFS recipes. This should help to > shake out any more issues with ulfs. If people are interested, I'll > ship the results to Rob later. Though the package-list above is the > important bit, you can just copy the rest from lfs-bootstrap. I lean towards making the lfs-bootstrap build more configurable by using the annotations with grep -v. It may need more/better annotations. How to specify this configuration is a UI issue I haven't come up with a good answer for yet. Environment variable I guess, KERNEL_EXTRA="LFS_SKIP=busybox,development" maybe? That could be done at build time without respinning the control image, which is kind of important considering how long that sucker takes to build... > Feel free to let me know if there are other issues with building BLFS > with this ulfs-bootstrap. On the Hexagon I built the x11 client libraries and managed to build xeyes and xboard and rxvt and a couple other things. There was some hacking around uClibc but they were using 0.9.30 rather than 0.9.31 so lots of it was backporting stuff... So all this was made to work once, I just didn't get permission to open source it. Rob 1301575452.0