From rob at landley.net Thu Nov 1 22:43:47 2012 From: rob at landley.net (Rob Landley) Date: Thu, 01 Nov 2012 17:43:47 -0500 Subject: [Aboriginal] Breakage while building LFS7.2 In-Reply-To: <20121031174622.GA7061@desktop> (from prazzb@gmail.com on Wed Oct 31 12:46:22 2012) Message-ID: <1351809827.12319.3@driftwood> On 10/31/2012 12:46:22 PM, Prasanna Balan wrote: > Hello > I am trying to build LFS 7.2 using control images.I am using the > patch attached below. > The build breaks while extracting tzdata.I've attached the > ( relevant )build log below the patch. > I necessarily added a ls command above mv in the extract function. > Notice how the packages above show ls properly and for tzdata it > shows up like readlink -f > > I want to confirm that this is for all and my system is not broken. It's saying that the tzdata you're copying _to_ already exists, and is not a directory. (I.E. it's complaining about the target file, not the source.) Let's see, clone a clean directory, apply your patch... === lfs-packages ( ) Extracting 'lfs-packages'... lfs-packages usage: ls [-ACFHLRSacdfiklmnpqrstux1] [directory...] [snip] ls: Unknown option h mv: invalid option -- 'v' BusyBox v1.20.2 (2012-10-30 18:30:59 CDT) multi-call binary. ... My $PATH is to the aboriginal build/host directory (my host system hasn't got mksquashfs), so you've added options that aren't in the busybox binaries. Try "hg revert common/download_functions.sh" and... Ah. the problem is that the extract software assumes that each tarball extracts into a single subdirectory. It then wildcard renames that directory to the new destination. (cd tempdir; tar xvjf blah.tar.*; mv * ~/destination/newdirname) But the tzdata archive in 7.2 is extracting: $ tar tvzf build/lfs-bootstrap/packages/lfs-packages/tzdata-2012.5.tar.gz -rw-r--r-- 0/0 44941 2012-07-18 19:45 africa -rw-r--r-- 0/0 15827 2012-07-18 19:30 antarctica -rw-r--r-- 0/0 112760 2012-07-25 09:13 asia -rw-r--r-- 0/0 68423 2012-07-25 09:40 australasia -rw-r--r-- 0/0 121390 2012-07-25 09:13 europe -rw-r--r-- 0/0 135756 2012-07-25 09:13 northamerica -rw-r--r-- 0/0 73854 2012-07-25 09:13 southamerica -rw-r--r-- 0/0 1190 2012-07-18 19:30 pacificnew -rw-r--r-- 0/0 2955 2012-07-18 19:30 etcetera -rw-r--r-- 0/0 4083 2012-07-18 19:30 backward -rw-r--r-- 0/0 1546 2012-07-18 19:30 systemv -rw-r--r-- 0/0 393 2012-07-18 19:30 factory -rw-r--r-- 0/0 19306 2012-07-18 19:30 solar87 -rw-r--r-- 0/0 19324 2012-07-18 19:30 solar88 -rw-r--r-- 0/0 19600 2012-07-18 19:30 solar89 -rw-r--r-- 0/0 4326 2012-07-18 19:30 iso3166.tab -rw-r--r-- 0/0 19913 2012-07-18 19:30 zone.tab -rw-r--r-- 0/0 3186 2012-07-25 09:13 leapseconds -rw-r--r-- 0/0 680 2012-07-18 19:30 yearistype.sh So the rename is going "that destination is not an existing directory, I can't move multiple files to something that isn't a directory!" Looks like we'll have to special case it. Ah, I see you've already got some special casing: for i in $(cd "$SRCDIR"; ls *.tar.*) do - extract_package $(noversion $i) + a=$(noversion $i) + if [ $a != "binutils" -a $a != "bash" -a $a != "linux" -a $a != "glibc" -a $a != "gcc" ] + then + extract_package $a + fi done I do actually want to build the new toolchain, glibc, and I think I _am_ building the new bash? (Thought I was...) The problem with replacing the existing toolchain is that if you're using the distcc acceleration trick it suddenly stops working, so we might want some kind of config option about it. This is why I've got the "FILTER" functionality in run-build-stages, which lets you exclude any of the categories in images/lfs-bootstrap/mnt/package-list (and exclude multiple ones via FILTER="busybox|internationalization|pointless".) To use it you should be able to do something like: KERNEL_EXTRA="FILTER=busybox|internationalization|pointless" ./native-build.sh \ /path/to/control-image.hdc (KERNEL_EXTRA appends extra stuff to the kernel command line, and any keyword=value thing the kernel doesn't recognize gets turned into an environment variable for PID 1.) And yes, for 7.2 you'll probably have to update package-list. That file can't be automatically generated from the list of existing packages because it controls the build order. (LFS is sensitive to which packages are installed in which order, and you can't work that out from the contents of the tarball-of-tarballs.) Rob 1351809827.0 From rob at landley.net Thu Nov 1 22:21:03 2012 From: rob at landley.net (Rob Landley) Date: Thu, 01 Nov 2012 17:21:03 -0500 Subject: [Aboriginal] Build from version control system repos instead of from tarballs (Rob Landley) In-Reply-To: <20121031153630.eb791f58.idunham@lavabit.com> (from idunham@lavabit.com on Wed Oct 31 17:36:30 2012) Message-ID: <1351808463.12319.2@driftwood> On 10/31/2012 05:36:30 PM, Isaac Dunham wrote: > > > On 10/30/2012 03:49:56 AM, Alessio Igor Bogani wrote: > > > Hi Rob, > > > > > > On 30/10/2012 03:36, Rob Landley wrote: > > >> On 09/04/2012 03:49:07 AM, Alessio Igor Bogani wrote: > > >>> Can Aboriginal grabs sources from git(Linux, musl, uclibc, > busybox) > > >>> and hg(toybox) repos instead of from tarballs? > > >> populating the package cache from the git repo, and > > >> Really what I want to do is say "URL=git://busybox.net/busybox" > in > > >> download.sh, have that clone the relevant repo under packages (so > > >> packages/busybox would be a directory), and then just use that > as the > > >> package cache version. > > >> > .. > > > Personally I don't use tarballs at all anymore: I'll download the > > > project's repo and build from it. I'm very happy if Aboriginal > let me > > > work in this way. > > > > It currently does, using a two step process: > > > > 1) Edit "configure" and add the packages you want to do git > versions of > > to the USE_ALT variable. (USE_ALT=busybox,uClibc,linux) > > > > 2) Then check out your repos into build/packages/alt-busybox and > > build/packages/alt-uClibc and so on. > > For what it's worth, it would be quite possible to just use > "git clone $URL alt-$PACKAGE" I've done that. Ok, let me back up and explain my problem: There are three categories of files in the aboriginal linux directory: 1) Files that come from the aboriginal tarball or repository (*.sh, sources/*, more/*, and so on.) These are tracked by aboriginal's source control. 2) Downloaded files that come from third parties (packages/*): if you delete this download.sh can re-download it, but only if you have net access. 3) Generated files (build/*). If you delete this, the build can recreate it even without net access. In fact, going "rm -rf build" is more or less "make distclean". Right now the package cache is in build/packages and is derived from the downloaded files. (It's the extracted tarball with patches applied.) Replacing the package cache with the upstream source control repository functionally does what you want, but at a design level it means that's the wrong place to put it in the file hierarchy. I think I've worked out how I want to fix this and I'll document it properly once I get it done, I just haven't finished coding it yet. (I cleanly ripped out the old alt- mechanism, which should no longer be required, and then set it running a test build, which took long enough I got distracted.) > > > For example (sorry in git-ish): > > > > > > git clone aboriginal > > > > I'm vaguely interested in having a git mirror of the mercurial repo, > That would be enough for this purpose, AFAICT. > > but I have no interest in hosting any of my projects in git. The git > > user interface is appalling, and their solution to all their > problems > To each his own (I actually enjoy it) > > is to expose implementation details to the user. > > > > cd aboriginal/ && ls > > > build.sh config cross-compiler.sh .. (these should works with any > > > version and doesn't > > > care if gcc is 4.2.2 or 4.7.1) > > > > The build script required for 4.2 and 4.7 are _totally_different_. > > For one thing, newer gcc versions have something like 4 prerequisite > > packages you have to build first, requirements 4.2 didn't have. > > > > Unfortunately, your desire doesn't match reality there. > > > > > ls package/linux (<- this is a submodule) > > > git submodule init > > > git submodule update > > > cd package/linux > > > > > > > git reset --hard origin (reset to the Linus' HEAD) > > > (don't change anything other aka let's use Rob's default versions) > > > ./build i686-virtio > > > > How about this: > > > > I can modify the scripts so that if there's a directory under > packages > > with the package name, we use that source verbatim. (No patches, no > > nothing.) Otherwise, we fall back to the current behavior. > (Basically > > it's just implementing a search path for the package cache with the > > first entry searched before attempting to _populate_ the package > cache.) > > Be sure to make {,dist,real}clean... > Otherwise, sounds reasonable to me. Not qutie sure what you mean? Aboriginal hasn't got "make". We do have more/clean.sh but that's actually there to avoid deleting/rebuilding the target-independent parts (the package cache and the host tools in build/host). > > > Why I bother with this crazy request? Because In this way I can > > > handle multi-level > > > changes in a very simple way. So if I want add a feature which > > > needs to change/patch > > > kernel, libc and application I could treat this like a single > > > atomic commit. > Doesn't this make tracking down what caused breakage harder? > ie, if you git bisect to an atomic commit in all three, you can't > tell if the kernel or the libc patch is responsible. I try to remain agnostic about what users want to do. That said, the project is adapted to my working style for obvious reasons. :) > > Which would never go upstream. > > > > My build uses clean vanilla release versions of upstream packages, > and > > separates out all the patches (and I try to send those patches > upstream > > so I don't have to carry them, with varying success). > > > > This is a design decision, so you can see what I'm using and what I > did > > to it. > > > > Your suggestion makes things easier for you at the expense of > making it > > harder for anybody else to see what you did or reproduce it outside > > your build system. That's fine for your purposes, but that's not > what > > I'm after. > Note that this could make GPL conformance (where it matters) a little > harder-otherwise, patches plus tarballs are enough, but this may mean > you need to make the repo public (I'm not sure how git archive would > work with submodules, though) Or easier if you go "I don't need to mirror the clean vanilla tarball, just the source url from download.sh and my patches against it from sources/patches". Having the paches cleanly separated out works better with that approach, which you need if the patches will ever go upstream. Developers _need_ broken up deltas against a published version they did in order to parse the changes, if you don't provide that they have to reverse engineer it from your giant hairball, which is incredibly painful. Clueless legal entities like the FSF and Software Conservancy which prefer hairballs are blind to the actual engineering requirements. Still, If somebody wants to use a patched tarball they can. If somebody wants to use a repo, they can. I've tried to be agnostic about the data sources, I see now I wasn't agnostic enough and am working to fix it. Rob 1351808463.0 From rob at landley.net Thu Nov 1 23:37:29 2012 From: rob at landley.net (Rob Landley) Date: Thu, 01 Nov 2012 18:37:29 -0500 Subject: [Aboriginal] Breakage while building LFS7.2 In-Reply-To: <20121031174622.GA7061@desktop> (from prazzb@gmail.com on Wed Oct 31 12:46:22 2012) References: <20121031174622.GA7061@desktop> Message-ID: <1351813049.12319.4@driftwood> On 10/31/2012 12:46:22 PM, Prasanna Balan wrote: > Hello > I am trying to build LFS 7.2 using control images.I am using the > patch attached below. > The build breaks while extracting tzdata.I've attached the > ( relevant )build log below the patch. > I necessarily added a ls command above mv in the extract function. > Notice how the packages above show ls properly and for tzdata it > shows up like readlink -f > > I want to confirm that this is for all and my system is not broken. Try the attached patch? It builds an image. The image actually _working_ probably requires a lot of comparing the LFS 7.2 build instructions against the images/lfs-bootstrap/mnt/build/* files and adjusting who knows what. I note that more/chroot-splice.sh might come in handy to debug an i686 build natively before trying the other architectures with emulation overhead... Rob 1351813049.0 From prazzb at gmail.com Sat Nov 3 08:06:39 2012 From: prazzb at gmail.com (Prasanna Balan) Date: Sat, 3 Nov 2012 13:36:39 +0530 Subject: [Aboriginal] Breakage while building LFS7.2 In-Reply-To: <1351813049.12319.4@driftwood> References: <20121031174622.GA7061@desktop> <1351813049.12319.4@driftwood> Message-ID: <20121103080639.GA1688@desktop> On Thu, Nov 01, 2012 at 06:37:29PM -0500, Rob Landley wrote: > On 10/31/2012 12:46:22 PM, Prasanna Balan wrote: > >Hello > > I am trying to build LFS 7.2 using control images.I am using the > > patch attached below. > > The build breaks while extracting tzdata.I've attached the > > ( relevant )build log below the patch. > > I necessarily added a ls command above mv in the extract function. > > Notice how the packages above show ls properly and for tzdata it > > shows up like readlink -f > > > > I want to confirm that this is for all and my system is not broken. > > Try the attached patch? > > It builds an image. The image actually _working_ probably requires a > lot of comparing the LFS 7.2 build instructions against the > images/lfs-bootstrap/mnt/build/* files and adjusting who knows what. > > I note that more/chroot-splice.sh might come in handy to debug an > i686 build natively before trying the other architectures with > emulation overhead... > > Rob It was not building the image and I was confused why. The patch I gave was a patch on top of the current tree that I was using (trying) to build the image. Turns out that tzdata didn't get extracted properly.Unlike all other packages, it doesn't has a subfolder inside the tarball. Hence mv command was breaking. So, I added lines to the extract function to make a directory and now it's done. Can build a proper image. Now is the time to change pkg-list and build-scripts used for building packages PS: I now repent as to why in the first place I cared about tzdata. It's needed to set timezone when using glibc. Not needed here. -- -prazzb GPG key 0xDB63309D -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available URL: From rob at landley.net Mon Nov 5 16:58:50 2012 From: rob at landley.net (Rob Landley) Date: Mon, 05 Nov 2012 10:58:50 -0600 Subject: [Aboriginal] Breakage while building LFS7.2 In-Reply-To: <20121103080639.GA1688@desktop> (from prazzb@gmail.com on Sat Nov 3 03:06:39 2012) References: <20121031174622.GA7061@desktop> <1351813049.12319.4@driftwood> <20121103080639.GA1688@desktop> Message-ID: <1352134730.28283.2@driftwood> On 11/03/2012 03:06:39 AM, Prasanna Balan wrote: > It was not building the image and I was confused why. The patch I > gave was a patch on top of the current tree that I was using (trying) > to build the image. > Turns out that tzdata didn't get extracted properly.Unlike all other > packages, it doesn't has a subfolder inside the tarball. Hence mv > command was breaking. > So, I added lines to the extract function to make a directory and now > it's done. Can build a proper image. > > Now is the time to change pkg-list and build-scripts used for building > packages > > PS: I now repent as to why in the first place I cared about tzdata. > It's > needed to set timezone when using glibc. Not needed here. It's nice to be able to use the lfs bootstrap to set up a glibc chroot on the target. If people _want_ glibc, we should be able to get them a glibc system without ever actually cross compiling glibc. :) Remember, one of the goals of the project is to eliminate the need for cross compiling. Thanks. Sorry I've been distracted recently, still catching up... Rob 1352134730.0 From onefang at gmail.com Tue Nov 13 04:23:51 2012 From: onefang at gmail.com (David Seikel) Date: Tue, 13 Nov 2012 14:23:51 +1000 Subject: [Aboriginal] missing applets in current 1.20 system images In-Reply-To: <1351214828.16660.8@driftwood> References: <5074BB77.20303@barfooze.de> <1351214828.16660.8@driftwood> Message-ID: <20121113142351.012188a4.onefang@gmail.com> On Thu, 25 Oct 2012 20:27:08 -0500 Rob Landley wrote: > On 10/09/2012 07:04:07 PM, John Spencer wrote: > > (mipsel:3) /home # free -m > > /bin/ash: free: not found > > (mipsel:3) /home # /bin/busybox mkswap > > mkswap: applet not found > > (mipsel:3) /home # swapon > > /bin/ash: swapon: not found > > Yup. There's a reason for that. > > I'm migrating the code from busybox to toybox. (Also from uClibc to > musl, but I don't have to write that one.) > > In preparation for this, I stripped down busybox from "defconfig" to > sources/baseconfig-busybox with just the list of commands the build > actually uses (as determiend by more/record-commands.sh). This > includes the set of commands used to boot the target system, and to > build Linux From Scratch under the result. I'm then gradually > replacing those busybox commands with toybox versions. > > Since then, I've been adding back commands like "top" that the build > doesn't use, but which I really miss if they're not there. You've > just spotted three more (which are already in the repository version > of toybox, but not in the older release version aboriginal's using). > > If you want the old "kitchen sink" configuration of busybox, it's: > > BUSYBOX=1 ./build.sh i686 > > That'll use a defconfig version of busybox instead of the stripped > down version plus toybox. SYSIMAGE_TYPE=ext2 tries to use the du command, which is missing from this stripped down busybox. Kitchen sink busybox supplies du though, so I'm using that for now. I've not tried the variation that uses toybox yet, but I plan to eventually. Is it my imagination, or does the latest release of Aboriginal Linux not supply mksquashfs when an earlier version did? I upgraded to Ubuntu 12.04, so now I'm upgrading to the latest Aboriginal Linux, which depends on a later qemu that Ubuntu 10.04 did not have. Now I had to install mksquashfs to get Aboriginal Linux to compile, I don't recall having to do that on Ubuntu 10.04. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From rob at landley.net Fri Nov 16 02:11:35 2012 From: rob at landley.net (Rob Landley) Date: Thu, 15 Nov 2012 20:11:35 -0600 Subject: [Aboriginal] Aboriginal Linux 1.2.1 is up. Message-ID: <1353031895.21193.2@driftwood> Release notes on the website, as usual. This version is actually using toybox, and has the upgraded binutils that can build musl. (Still GPLv2, fished out of the git repository.) It's not using musl yet, but that's the next goal. (The big pushes moving forward are toybox and musl.) Powerpc is broken, I need to go find out why. Possibly cut a 1.2.2 even before linux 3.7 comes out, but I note that linux 3.7 is pretty close and I'll probably be cutting a release much sooner after that comes out anyway. I just fixed the screenshots directory after who knows how long. I may take a look at upgrading Linux From Scratch too. I'm pondering putting http://ph7.symisc.net in static-tools. Rob 1353031895.0 From onefang at gmail.com Fri Nov 16 02:19:34 2012 From: onefang at gmail.com (David Seikel) Date: Fri, 16 Nov 2012 12:19:34 +1000 Subject: [Aboriginal] Aboriginal Linux 1.2.1 is up. In-Reply-To: <1353031895.21193.2@driftwood> References: <1353031895.21193.2@driftwood> Message-ID: <20121116121934.59356fbc.onefang@gmail.com> On Thu, 15 Nov 2012 20:11:35 -0600 Rob Landley wrote: > Release notes on the website, as usual. /me rushes to try this. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From onefang at gmail.com Fri Nov 16 02:33:08 2012 From: onefang at gmail.com (David Seikel) Date: Fri, 16 Nov 2012 12:33:08 +1000 Subject: [Aboriginal] Aboriginal Linux 1.2.1 is up. In-Reply-To: <1353031895.21193.2@driftwood> References: <1353031895.21193.2@driftwood> Message-ID: <20121116123308.5d29cd00.onefang@gmail.com> Might be a good idea to update the date on the release notes. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From rob at landley.net Fri Nov 16 02:42:16 2012 From: rob at landley.net (Rob Landley) Date: Thu, 15 Nov 2012 20:42:16 -0600 Subject: [Aboriginal] missing applets in current 1.20 system images In-Reply-To: <20121113142351.012188a4.onefang@gmail.com> (from onefang@gmail.com on Mon Nov 12 22:23:51 2012) References: <5074BB77.20303@barfooze.de> <1351214828.16660.8@driftwood> <20121113142351.012188a4.onefang@gmail.com> Message-ID: <1353033736.2187.0@driftwood> On 11/12/2012 10:23:51 PM, David Seikel wrote: > On Thu, 25 Oct 2012 20:27:08 -0500 Rob Landley > wrote: > > > On 10/09/2012 07:04:07 PM, John Spencer wrote: > > > (mipsel:3) /home # free -m > > > /bin/ash: free: not found > > > (mipsel:3) /home # /bin/busybox mkswap > > > mkswap: applet not found > > > (mipsel:3) /home # swapon > > > /bin/ash: swapon: not found > > > > Yup. There's a reason for that. > > > > I'm migrating the code from busybox to toybox. (Also from uClibc to > > musl, but I don't have to write that one.) > > > > In preparation for this, I stripped down busybox from "defconfig" to > > sources/baseconfig-busybox with just the list of commands the build > > actually uses (as determiend by more/record-commands.sh). This > > includes the set of commands used to boot the target system, and to > > build Linux From Scratch under the result. I'm then gradually > > replacing those busybox commands with toybox versions. > > > > Since then, I've been adding back commands like "top" that the build > > doesn't use, but which I really miss if they're not there. You've > > just spotted three more (which are already in the repository version > > of toybox, but not in the older release version aboriginal's using). > > > > If you want the old "kitchen sink" configuration of busybox, it's: > > > > BUSYBOX=1 ./build.sh i686 > > > > That'll use a defconfig version of busybox instead of the stripped > > down version plus toybox. > > SYSIMAGE_TYPE=ext2 tries to use the du command, which is missing from > this stripped down busybox. Kitchen sink busybox supplies du though, > so I'm using that for now. I've not tried the variation that uses > toybox yet, but I plan to eventually. There's a du in toybox in the version that just went up. It's a third party contribution I haven't done a cleanup pass on yet, but it seems to basically work. > Is it my imagination, or does the latest release of Aboriginal Linux > not supply mksquashfs when an earlier version did? The host-tools.sh script builds mksquashfs. It isn't in the target filesystem, but if you rebuild aboriginal under itself it should build it again in the new host tools. > I upgraded to Ubuntu > 12.04, so now I'm upgrading to the latest Aboriginal Linux, which > depends on a later qemu that Ubuntu 10.04 did not have. Now I had to > install mksquashfs to get Aboriginal Linux to compile, I don't recall > having to do that on Ubuntu 10.04. You shouldn't have to do so to make it compile. (It's not in my host $PATH.) You'll have to install it on the host if you skip the host-tools.sh step, but you'll have to do a dozen other things in that case to get it to work. (Which you may have already done, dunno...) Rob 1353033736.0 From rob at landley.net Tue Nov 20 22:42:20 2012 From: rob at landley.net (Rob Landley) Date: Tue, 20 Nov 2012 16:42:20 -0600 Subject: [Aboriginal] Aboriginal Linux 1.2.1 is up. In-Reply-To: (from james_mcmechan@hotmail.com on Tue Nov 20 10:42:48 2012) References: <1353031895.21193.2@driftwood> Message-ID: <1353451340.31598.0@driftwood> On 11/20/2012 10:42:48 AM, James McMechan wrote: > > Hi, http://landley.net/aboriginal/ does list 1.2.1 > it however still lists the release date as August 22, 2012 > you might want to fix that... > > Glad to see the new release :) > > Jim Oops. Fixed, Rob 1353451340.0