From niklas.brunback at gmail.com Sun Jun 5 11:58:33 2011 From: niklas.brunback at gmail.com (=?ISO-8859-15?Q?Niklas_Brunb=E4ck?=) Date: Sun, 5 Jun 2011 13:58:33 +0200 (CEST) Subject: [Aboriginal] Building X Message-ID: Hello all, Has anyone successfully built Kdrive/TinyX from X{org,Free86} with aboriginal on x86? Niklas 1307275113.0 From rob at landley.net Sun Jun 5 15:41:52 2011 From: rob at landley.net (Rob Landley) Date: Sun, 05 Jun 2011 10:41:52 -0500 Subject: [Aboriginal] Building X In-Reply-To: References: Message-ID: <4DEBA3C0.4080203@landley.net> On 06/05/2011 06:58 AM, Niklas Brunb?ck wrote: > Hello all, > > Has anyone successfully built Kdrive/TinyX from X{org,Free86} with > aboriginal on x86? I've built x11.org under aboriginal on Hexagon, so x86 should be pretty straightforward. Go read Beyond Linux From Scratch and follow their section on installing X11. You can find the most current one here, see chapter 23: http://www.linuxfromscratch.org/blfs/view/svn/ I note that the question is a bit off. Nobody's built XFree86 under anything since about 2003, it's a dead project because its' maintainer went insane. Keith Packard is one of the original developers who invented X11 at MIT back in 1987, and he's the guy who tried to clean up the horrible mess of XFree86, thus creating tinyx (which was later renamed kdrive, they're the same thing). Keith then got kicked out of the XFree86 project by said insane maintainer who felt extremely threatened by him, since Keith was such a better programmer that he made the other guy look incompetent. Keith then opened a new repository to host his work on x.org, and continued his rewrite of xfree86 (which was all based on his original X11 work at MIT anyway), and that's what everybody uses these days. Keith's x.org project is based on his tinyx/kdrive code, the old bloated/brittle xfree86 core engine is gone. You don't have to build it in a special mode to make massive bloat go away: it's just not bloated to begin with. So asking "has anybody built xfree86 recently" is a bit like asking "has anybody built SCO UnixWare recently". The answer is no, because they went crazy years ago. I followed the BLFS x.org instructions, and got 'em to work. You have to build something like three dozen packages (keith broke up the xfree86 hairball and made it very modular), but you can work out a list and put 'em in a for loop: they all build with the same command line. BLFS actually has said list, although I think it's an attached text file or something... I mean to add an X build stage to the lfs-bootstrap but the uClibc NPTL stuff comes first... > Niklas Rob 1307288512.0 From rob at landley.net Wed Jun 8 02:08:33 2011 From: rob at landley.net (Rob Landley) Date: Tue, 07 Jun 2011 21:08:33 -0500 Subject: [Aboriginal] armv6l works again. Message-ID: <4DEED9A1.9090908@landley.net> The saga is described here: http://lists.nongnu.org/archive/html/qemu-devel/2011-06/msg00829.html I'm going to see what I can get done this weekend, and then probably cut a release. It's waaaaay overdue... Rob 1307498913.0 From rob at landley.net Sat Jun 11 18:24:40 2011 From: rob at landley.net (Rob Landley) Date: Sat, 11 Jun 2011 13:24:40 -0500 Subject: [Aboriginal] Prepping for a release. Message-ID: <4DF3B2E8.10301@landley.net> Yeah, uClibc-0.9.32 shipped. I'm still using 0.9.31 because my testing on -rc3 had lots of problems, and I'm not swapping that this close to a release, but next time 'round I'll have to use it. I'm poking at OpenEmbedded because Khem Raj said he had sh4 working with Linux 2.6.38 under QEMU. Alas, so far it's just a big pile of packages for bitbake (yet ANOTHER package repository to go with rpm, dpkg, portage, and so on). The "getting started" web page is incomplete. There's no obvious "build a base sh4 system" switch. I made a debian chroot, installed a dozen extra packages into it, downloaded two different git repositories, made a wrapper script to set environment variables, and now I'm reading through a config file that's full of little land mines to ensure the default configuration doesn't work to require you to read the whole thing... Might punt fixing sh4 to the next release... Anyway, the other thing I'd like to get done this time is cleaning up the sources/targets/$ARCH things so that instead of being a directory, they're a file. 95% of what they do now is the "settings" file, and the description file could be a settings entry. (I need to redo the snapshot generation logic anyway, although I can do that after the release.) So I might as well combine those and promote them up one level, so the "targets" directory contains one file per target. Nice cleanup if I can pull it off. The real problem though is that right now, you can drop a miniconfig-linux or miniconfig-uClibc (or miniconfig-alt-linux or miniconfig-alt-uClibc) file in the arch directory and it'll override the baseconfig. Some architectures still haven't been converted to use baseconfig, and I need to do that, but some things like the hw-targets aren't _ever_ going to use baseconfig at all, they need to continue to override this. And it's a good debugging technique anyway. So I need to come up with a new place to PUT these other config/miniconfig files. I mean really I can just do LINUX_CONFIG="$(cat filename)" in a settings file, and since it's appended to the baseconfig if it's a full config it overrides everything in the baseconfig, although I could also do a LINUX_FULLCONFIG=filename thing too... But the question is where to _put_ any of those files that I do actually want to check in. I suppose step 1 is to deal with the cleanups that are easy to do and see what's left. Merge the description files in with the settings files, convert the remaining arch standalone configs to baseconfigs... Working on it. :) Rob 1307816680.0 From onefang at gmail.com Sat Jun 11 22:43:25 2011 From: onefang at gmail.com (David Seikel) Date: Sun, 12 Jun 2011 08:43:25 +1000 Subject: [Aboriginal] Prepping for a release. In-Reply-To: <4DF3B2E8.10301@landley.net> References: <4DF3B2E8.10301@landley.net> Message-ID: <20110612084325.7bb64041@U64-desktop> On Sat, 11 Jun 2011 13:24:40 -0500 Rob Landley wrote: > I'm poking at OpenEmbedded because Khem Raj said he had sh4 working > with Linux 2.6.38 under QEMU. Alas, so far it's just a big pile of > packages for bitbake (yet ANOTHER package repository to go with rpm, > dpkg, portage, and so on). The "getting started" web page is > incomplete. There's no obvious "build a base sh4 system" switch. I > made a debian chroot, installed a dozen extra packages into it, > downloaded two different git repositories, made a wrapper script to > set environment variables, and now I'm reading through a config file > that's full of little land mines to ensure the default configuration > doesn't work to require you to read the whole thing... I looked at OpenEmbedded, coz the head developer of the EFL stuff I'm using is enamoured of it, and has suitable stuff for building EFL on top of that in the EFL repo. Looked like too much to me. OE was too big, AL is just right. I might add "bare minimum linux to run EFL" stuff to the EFL repo later. > The real problem though is that right now, you can drop a > miniconfig-linux or miniconfig-uClibc (or miniconfig-alt-linux or > miniconfig-alt-uClibc) file in the arch directory and it'll override > the baseconfig. Some architectures still haven't been converted to > use baseconfig, and I need to do that, but some things like the > hw-targets aren't _ever_ going to use baseconfig at all, they need to > continue to override this Yep, I'm using that for my project. That i486 arch I added before is being used as the base for a new i486-MyLinux target where I add things to the kernel for my project that are not needed just for the i486 target. This should grow later to be the next release of my SourceForge MyLinux project. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From rob at landley.net Mon Jun 13 00:29:40 2011 From: rob at landley.net (Rob Landley) Date: Sun, 12 Jun 2011 19:29:40 -0500 Subject: [Aboriginal] Prepping for a release. In-Reply-To: <20110612084325.7bb64041@U64-desktop> References: <4DF3B2E8.10301@landley.net> <20110612084325.7bb64041@U64-desktop> Message-ID: <4DF559F4.8040702@landley.net> On 06/11/2011 05:43 PM, David Seikel wrote: > On Sat, 11 Jun 2011 13:24:40 -0500 Rob Landley wrote: > >> I'm poking at OpenEmbedded because Khem Raj said he had sh4 working >> with Linux 2.6.38 under QEMU. Alas, so far it's just a big pile of >> packages for bitbake (yet ANOTHER package repository to go with rpm, >> dpkg, portage, and so on). The "getting started" web page is >> incomplete. There's no obvious "build a base sh4 system" switch. I >> made a debian chroot, installed a dozen extra packages into it, >> downloaded two different git repositories, made a wrapper script to >> set environment variables, and now I'm reading through a config file >> that's full of little land mines to ensure the default configuration >> doesn't work to require you to read the whole thing... > > I looked at OpenEmbedded, coz the head developer of the EFL stuff I'm > using is enamoured of it, and has suitable stuff for building EFL on > top of that in the EFL repo. Looked like too much to me. OE was too > big, AL is just right. I might add "bare minimum linux to run EFL" > stuff to the EFL repo later. It's not a bad project, it's just not the same kind of thing I'm trying to do. Aboriginal is not a distro, it just creates a native development environment for the target and then _stops_. What you build on the target _with_ that native development environment is none of our business (unless it demonstrates a bug in that development environment). Openembedded, like buildroot, is a distro. If you have a package repository, you are a distro, and this thing recipes/xmame which it'll cross compile for the target. Which unfortunately means that if you're trying to get it to do what aboriginal linux does and get it to build a basic sh4 target system to boot under qemu, it's not immediately obvious how to go about it. I'm current reading classes/siteinfo.bbclass which at least mentions sh4... >> The real problem though is that right now, you can drop a >> miniconfig-linux or miniconfig-uClibc (or miniconfig-alt-linux or >> miniconfig-alt-uClibc) file in the arch directory and it'll override >> the baseconfig. Some architectures still haven't been converted to >> use baseconfig, and I need to do that, but some things like the >> hw-targets aren't _ever_ going to use baseconfig at all, they need to >> continue to override this > > Yep, I'm using that for my project. That i486 arch I added before is > being used as the base for a new i486-MyLinux target where I add things > to the kernel for my project that are not needed just for the i486 > target. This should grow later to be the next release of my > SourceForge MyLinux project. It's definitely something users need to do, I'm just trying to figure how where this capability fits into the design from a UI perspective. I want to make it easy for you to do this, and clean for me to track it in the repository if it gets checked in... I do note that the horrible "hw-" overlay stuff never fit cleanly into the current design, and I'm tempted to make that distinction be between targets that are directories and targets that are files, instead of a magic prefix. Possibly a directory could have a "base" symlink to the architecture file (../i486 in your case), and then it could have settings (stuff appended to the base config allowing arbitrary overrides), and config-linux and config-uclibc files? Hmmm... OR what I could do is ignore directories entirely, then if a given config wanted to create a subdirectory to suck files out of, it could in the current directory... Aesthetic decisions are always the hard ones... 1307924980.0 From alessioigorbogani at gmail.com Tue Jun 14 09:21:02 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Tue, 14 Jun 2011 11:21:02 +0200 Subject: [Aboriginal] Prepping for a release. In-Reply-To: <4DF559F4.8040702@landley.net> References: <4DF3B2E8.10301@landley.net> <20110612084325.7bb64041@U64-desktop> <4DF559F4.8040702@landley.net> Message-ID: 2011/6/13 Rob Landley : > On 06/11/2011 05:43 PM, David Seikel wrote: [...] > Aboriginal is not a distro, it just creates a native development > environment for the target and then _stops_. ?What you build on the In my opinion that is a defect. You could provide a minimal user-space without bothering with repository and you are still a distro. Indeed it is exactly what I would expected by this project. Ciao, Alessio 1308043262.0 From onefang at gmail.com Tue Jun 14 11:05:39 2011 From: onefang at gmail.com (David Seikel) Date: Tue, 14 Jun 2011 21:05:39 +1000 Subject: [Aboriginal] Prepping for a release. In-Reply-To: References: <4DF3B2E8.10301@landley.net> <20110612084325.7bb64041@U64-desktop> <4DF559F4.8040702@landley.net> Message-ID: <20110614210539.6cd1b134@U64-desktop> On Tue, 14 Jun 2011 11:21:02 +0200 Alessio Igor Bogani wrote: > 2011/6/13 Rob Landley : > > On 06/11/2011 05:43 PM, David Seikel wrote: > [...] > > Aboriginal is not a distro, it just creates a native development > > environment for the target and then _stops_. ?What you build on the > > In my opinion that is a defect. You could provide a minimal user-space > without bothering with repository and you are still a distro. Indeed > it is exactly what I would expected by this project. The busybox part IS a minimal userspace. Rob gets to decide the scope and boundaries of his project. I'm happy with his decisions, they match my needs. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From rob at landley.net Wed Jun 15 00:07:54 2011 From: rob at landley.net (Rob Landley) Date: Tue, 14 Jun 2011 19:07:54 -0500 Subject: [Aboriginal] Prepping for a release. In-Reply-To: References: <4DF3B2E8.10301@landley.net> <20110612084325.7bb64041@U64-desktop> <4DF559F4.8040702@landley.net> Message-ID: <4DF7F7DA.9010003@landley.net> On 06/14/2011 04:21 AM, Alessio Igor Bogani wrote: > 2011/6/13 Rob Landley : >> On 06/11/2011 05:43 PM, David Seikel wrote: > [...] >> Aboriginal is not a distro, it just creates a native development >> environment for the target and then _stops_. What you build on the > > In my opinion that is a defect. You could provide a minimal user-space > without bothering with repository and you are still a distro. Indeed > it is exactly what I would expected by this project. I do provide a minimal userspace (without a repository), I even package up the binaries to be bootable under qemu, and ship tarballs thereof. (That would be the system-image files in downloads/binaries.) That userspace is the smallest development environment capable of natively rebuilding itself under itself, meaning from there you can natively compile and install anything you want. That's sort of the point of the project. However, it's not my job to define the "anything you want" part. I'm not understanding the objection? Rob 1308096474.0 From rob at landley.net Wed Jun 15 00:13:50 2011 From: rob at landley.net (Rob Landley) Date: Tue, 14 Jun 2011 19:13:50 -0500 Subject: [Aboriginal] Prepping for a release. In-Reply-To: <20110614210539.6cd1b134@U64-desktop> References: <4DF3B2E8.10301@landley.net> <20110612084325.7bb64041@U64-desktop> <4DF559F4.8040702@landley.net> <20110614210539.6cd1b134@U64-desktop> Message-ID: <4DF7F93E.2010503@landley.net> On 06/14/2011 06:05 AM, David Seikel wrote: > On Tue, 14 Jun 2011 11:21:02 +0200 Alessio Igor Bogani > wrote: > >> 2011/6/13 Rob Landley : >>> On 06/11/2011 05:43 PM, David Seikel wrote: >> [...] >>> Aboriginal is not a distro, it just creates a native development >>> environment for the target and then _stops_. What you build on the >> >> In my opinion that is a defect. You could provide a minimal user-space >> without bothering with repository and you are still a distro. Indeed >> it is exactly what I would expected by this project. > > The busybox part IS a minimal userspace. Rob gets to decide the scope > and boundaries of his project. I'm happy with his decisions, they > match my needs. There are actually four options if you don't want to run the build scripts: 1) downloads/binaries/extras/simple-root-filesystem-*.tar.bz2 A root filesystem with just busybox. 2) downloads/binaries/extras/native-compiler-*.tar.bz2 A native compiler you should be able to use on just about any target (although installing the shared libraries in the right place is your problem; easiest just to build static). This is built from uClibc, gcc, binutils, make, bash, and the linux kernel headers. (And optionally distcc.) 3) downloads/binaries/extras/root-filesystem-*.tar.bz2 A root filesystem combining the above two, this is the minimal native development environment. 4) downloads/binaries/system-image-*.tar.bz2 This is just #3 packaged up as a squashfs with some shell scripts that run qemu in various ways. run-emulator.sh is the basic one, dev-environment.sh is a wrapper around that which provides a 2 gigabyte /dev/hdb and hooks up distcc to the cross compiler if you have both in your $PATH on the host, and native-build.sh is a wrapper around dev-environment.sh that can drive an automated build using one of the control images in the control-images directory. (I did one for Linux >From Scratch which should provide a fairly elaborate model to copy from, the scripts that built it are in sources/control-images.) Rob 1308096830.0 From rob at landley.net Wed Jun 15 01:56:17 2011 From: rob at landley.net (Rob Landley) Date: Tue, 14 Jun 2011 20:56:17 -0500 Subject: [Aboriginal] Release 1.0.2 is uploading. Message-ID: <4DF81141.8000703@landley.net> Ok, I got just about all the targets working. The only ones that don't pass the hello world smoketest (it natively compiles and runs a threaded hello world program) are armv4eb (which qemu doesn't emulate), m68k (ditto), mips64 (which doesn't even ls -l without showing bugs) and sparc (which is static only and building -lpthreads static is a dependency nightmare between libc and libpthreads, I should poke the qemu guys about that but not today). Everything else that _was_ broken is fixed, including armv6l, sh4, and mips. Even powerpc-440fp worked, which it shouldn't. (I migrated powerpc to use baseconfig, which yanked the old ppc440 symlink dependency, meaning the ppc440 config is baseconfig-linux with no arch-specific additions, yet it booted. Huh? with zero platform specific lines, the DEFAULTS worked? Not questioning it. I need to do a proper -M bamboo board next time now that QEMU's grown more support for ppc440...) And so, I cut a release. Tagged commit 1375, uploading the 8 gazillion binary tarballs and the source tarball as we speak. I'll have to do proper release notes and update the website tomorrow. Rob 1308102977.0 From onefang at gmail.com Wed Jun 15 02:35:54 2011 From: onefang at gmail.com (David Seikel) Date: Wed, 15 Jun 2011 12:35:54 +1000 Subject: [Aboriginal] Release 1.0.2 is uploading. In-Reply-To: <4DF81141.8000703@landley.net> References: <4DF81141.8000703@landley.net> Message-ID: <20110615123554.7c4b7f9d@U64-desktop> Yay! -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From geoffroy.weisenhorn at gmail.com Wed Jun 15 16:39:31 2011 From: geoffroy.weisenhorn at gmail.com (geoffroy weisenhorn) Date: Wed, 15 Jun 2011 18:39:31 +0200 Subject: [Aboriginal] Tiny patch for linux-kernel build issue Message-ID: Hello, I have found a issue when launching linux-kernel.sh on i686 host. In short when compiling Linux kernel, build script is blocked by mktimeconst who seems to be failed on an infinite loop... After investigation I found a bug in kernel/mktimeconst.c This code come from patch linux-noperl-timeconst.patch this test would never be true: if (mul32 >= (1L<<31)) (line 125 in mktimeconst.c) I propose a one character patch : 25c125 < + if (mul32 >= (1UL<<31)) --- > + if (mul32 >= (1L<<31)) With this change linux-kernel build successfully. Geoffroy. -------------- next part -------------- A non-text attachment was scrubbed... Name: 290d89e8689fcefb2344fab9aa1aaeb5279eb5bb Type: application/octet-stream Size: 651 bytes Desc: not available URL: From alessioigorbogani at gmail.com Wed Jun 15 19:26:20 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Wed, 15 Jun 2011 21:26:20 +0200 Subject: [Aboriginal] It is cool, isn't it? Message-ID: Hi, What do you think about this? http://marc.info/?l=kvm&m=130815323831521&w=2 Ciao, Alessio 1308165980.0 From alessioigorbogani at gmail.com Wed Jun 15 21:25:50 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Wed, 15 Jun 2011 23:25:50 +0200 Subject: [Aboriginal] [PATCH 1 of 3] Fix a little typo In-Reply-To: References: Message-ID: <13515dc218b6c03d3dc5.1308173150@picasso> # HG changeset patch # User Alessio Igor Bogani # Date 1308172872 -7200 # Node ID 13515dc218b6c03d3dc52a9263424c9d3039ace7 # Parent a6d18adf198d3b3e9f5c3d2ed1cd9d77782b1f64 Fix a little typo diff -r a6d18adf198d -r 13515dc218b6 www/README --- a/www/README Tue Jun 14 20:48:07 2011 -0500 +++ b/www/README Wed Jun 15 23:21:12 2011 +0200 @@ -32,7 +32,7 @@ - Run "cat /proc/cpuinfo" to confirm the emulator is running. - Run "gcc -lpthread /usr/src/thread-hello2.c" (and run the resulting - "./aout" file) to confirm the native toolchain works. + "./a.out" file) to confirm the native toolchain works. - Type "exit" when done. 1308173150.0 From alessioigorbogani at gmail.com Wed Jun 15 21:25:49 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Wed, 15 Jun 2011 23:25:49 +0200 Subject: [Aboriginal] [PATCH 0 of 3] Little fixes Message-ID: This my first time with Mercurial so please be patient with me! ;-) 1308173149.0 From alessioigorbogani at gmail.com Wed Jun 15 21:25:52 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Wed, 15 Jun 2011 23:25:52 +0200 Subject: [Aboriginal] [PATCH 3 of 3] Update links In-Reply-To: References: Message-ID: <0e6e4971b56372deb13a.1308173152@picasso> # HG changeset patch # User Alessio Igor Bogani # Date 1308172959 -7200 # Node ID 0e6e4971b56372deb13a58dab9c8468bf1e81938 # Parent c406b41df81436cba4c5a31573f97fb0ff0e08f9 Update links diff -r c406b41df814 -r 0e6e4971b563 www/README --- a/www/README Wed Jun 15 23:22:07 2011 +0200 +++ b/www/README Wed Jun 15 23:22:39 2011 +0200 @@ -71,7 +71,7 @@ To download prebuilt binaries for each target: - http://aboriginal.impactlinux.com/downloads/binaries + http://landley.net/aboriginal/downloads/binaries The "system-image" tarballs are what you need to boot a virtual target under QEMU. They contain a squashfs root filesystem (including a native @@ -94,7 +94,7 @@ Download the most recent set of build scripts from: - http://aboriginal.impactlinux.com/downloads + http://landley.net/hg/aboriginal/archive/tip.tar.bz2 To list available targets, do: @@ -130,7 +130,7 @@ For more information, see the README file in the download directory. - http://aboriginal.impactlinux.com/downloads/README + http://landley.net/aboriginal/README To boot a system image under QEMU: 1308173152.0 From alessioigorbogani at gmail.com Wed Jun 15 21:25:51 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Wed, 15 Jun 2011 23:25:51 +0200 Subject: [Aboriginal] [PATCH 2 of 3] Remove dead links In-Reply-To: References: Message-ID: # HG changeset patch # User Alessio Igor Bogani # Date 1308172927 -7200 # Node ID c406b41df81436cba4c5a31573f97fb0ff0e08f9 # Parent 13515dc218b6c03d3dc52a9263424c9d3039ace7 Remove dead links diff -r 13515dc218b6 -r c406b41df814 www/README --- a/www/README Wed Jun 15 23:21:12 2011 +0200 +++ b/www/README Wed Jun 15 23:22:07 2011 +0200 @@ -66,12 +66,6 @@ If you don't know how something works, read the script. If something's unclear, ask about it on the mailing list. ---- How does it work? - - For information on implementation details: - - http://aboriginal.impactlinux.com/implementation.html - =============================================================================== --- How do I use Aboriginal Linux? @@ -109,10 +103,6 @@ We'll use "mipsel" (mips little endian) in the following examples. Replace that with the target you're interested in from this list. - For more information about each target, see: - - http://aboriginal.impactlinux.com/snapshots - To compile a system image (and all prerequisite stages) for a target: ./build.sh mipsel 1308173151.0 From rob at landley.net Thu Jun 16 12:45:05 2011 From: rob at landley.net (Rob Landley) Date: Thu, 16 Jun 2011 07:45:05 -0500 Subject: [Aboriginal] Tiny patch for linux-kernel build issue In-Reply-To: References: Message-ID: <4DF9FAD1.5060805@landley.net> On 06/15/2011 11:39 AM, geoffroy weisenhorn wrote: > Hello, > > I have found a issue when launching linux-kernel.sh on i686 host. In > short when compiling > Linux kernel, build script is blocked by mktimeconst who seems to be > failed on an infinite loop... > After investigation I found a bug in kernel/mktimeconst.c > > This code come from patch linux-noperl-timeconst.patch > this test would never be true: if (mul32 >= (1L<<31)) (line 125 in > mktimeconst.c) > > I propose a one character patch : > 25c125 > < + if (mul32 >= (1UL<<31)) > --- >> + if (mul32 >= (1L<<31)) > > With this change linux-kernel build successfully. > > Geoffroy. Applied. Thanks, Rob 1308228305.0 From rob at landley.net Thu Jun 16 13:18:46 2011 From: rob at landley.net (Rob Landley) Date: Thu, 16 Jun 2011 08:18:46 -0500 Subject: [Aboriginal] It is cool, isn't it? In-Reply-To: References: Message-ID: <4DFA02B6.4080007@landley.net> On 06/15/2011 02:26 PM, Alessio Igor Bogani wrote: > Hi, > > What do you think about this? > > http://marc.info/?l=kvm&m=130815323831521&w=2 > > Ciao, > Alessio I heard they were doing this, I'm still not entirely sure why. They want KVM to implement its' own I/O devices rather than using QEMU's device emulations (as pretty much every virtualization solution including xen and virtualbox does right now). *shrug* Ok. I use KVM as a better kqemu, although my new netbook has an atom processor which apparently hasn't got the processor extension to support it. (No idea _why_, Intel didn't yank MMX but they yanked VT? Ok...) If KVM wants to not be part of qemu, they can join the ranks of user mode linux, lguest, and xen: virtualization solutions built into the Linux kernel that I don't really spend time playing with. Mostly, I use qemu to emulate arm and mips and powerpc and such on my x86 laptop. x86 on x86 I can do via chroot or lxc. (Linux containers are exciting, it can scale to thousands of instances per box. Making virtualization scale beyond a couple dozen instances is a waste of time: congratulations, you've reinvented the microkernel, people spent 20 years trying to make that scale and still insist it'll happen any day now.) Rob 1308230326.0 From rob at landley.net Fri Jun 17 12:50:37 2011 From: rob at landley.net (Rob Landley) Date: Fri, 17 Jun 2011 07:50:37 -0500 Subject: [Aboriginal] [PATCH 3 of 3] Update links In-Reply-To: <0e6e4971b56372deb13a.1308173152@picasso> References: <0e6e4971b56372deb13a.1308173152@picasso> Message-ID: <4DFB4D9D.6050109@landley.net> On 06/15/2011 04:25 PM, Alessio Igor Bogani wrote: > # HG changeset patch > # User Alessio Igor Bogani > # Date 1308172959 -7200 > # Node ID 0e6e4971b56372deb13a58dab9c8468bf1e81938 > # Parent c406b41df81436cba4c5a31573f97fb0ff0e08f9 > Update links Applied all three. Thanks, Rob 1308315037.0 From rob at landley.net Fri Jun 17 13:48:59 2011 From: rob at landley.net (Rob Landley) Date: Fri, 17 Jun 2011 08:48:59 -0500 Subject: [Aboriginal] [PATCH] Remove useless symlink In-Reply-To: References: Message-ID: <4DFB5B4B.3050503@landley.net> On 06/17/2011 05:07 AM, Alessio Igor Bogani wrote: > # HG changeset patch > # User Alessio Igor Bogani > # Date 1308302693 -7200 > # Node ID a041d8bd1b2f60d0d68e969ab2946e7c51d27476 > # Parent 092302c33462716a5a0b5612f7676d76ccd8fe94 > Remove useless symlink > > diff -r 092302c33462 -r a041d8bd1b2f www/screenshots/index.html > --- a/www/screenshots/index.html Wed Jun 15 23:22:39 2011 +0200 > +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 > @@ -1,1 +0,0 @@ > -oldindex.html > \ No newline at end of file > I need to redo the entire screenshots directory. It's gone very stale, and should have a script automatically generating it each release. The question is the automation: a control image won't show a shell prompt, and I haven't got expect set up on the host (and don't want to require that as an environmental dependency, and have no interest in building tcl from source), and my shell script to sort of _fake_ expect in bash is unfinished. (Turns out to be a hard problem.) The quick-and-dirty trick I've been doing (in more/smoketest.sh) is just cat the series of commands I want into qemu, with a line of spaces at the top to overcome the way serial chip initialization in the device driver eats a semi-random amount of data (generally the FIFO buffer size plus a couple "just for luck"). Unfortunately, this is not reliable. It locks powerpc solid (which is a driver bug having to do with input and output interrupts crossing, but the PMAC_ZILOG driver is ancient and only semi-maintained, the mac99 platform is 12 years old, it's just the most convenient target for qemu to emulate on 32 bit PPC). And some commands just plain eat pending input, even when they ignore it. The point of expect is to wait for a prompt (hence the "Type exit when done" string) before producing the output, so you have a bunch of trigger/response pairs. It's reliable, but requires being able ot look at the input and output of the same command, and making a pipeline into a circle is actually kind of hard to do in shell programming. It's on my todo list. :) I actually worked out how to DO expect in shell (see attached, although I wouldn't say I'm _happy_ with that solution), but haven't managed to follow up and rewrite the screenshot generator or smoketest.sh yet. (Hence the reason releases tend to be late: I have more things I want to do. I need to get back to time-based release discipline, no matter how little time I PERSONALLY have. Speaking of which, I'm late for work...) Rob -------------- next part -------------- A non-text attachment was scrubbed... Name: xp2.sh Type: application/x-sh Size: 888 bytes Desc: not available URL: From alessiogirobogani at gmail.com Fri Jun 17 10:07:06 2011 From: alessiogirobogani at gmail.com (Alessio Igor Bogani) Date: Fri, 17 Jun 2011 12:07:06 +0200 Subject: [Aboriginal] [PATCH] Remove useless symlink Message-ID: # HG changeset patch # User Alessio Igor Bogani # Date 1308302693 -7200 # Node ID a041d8bd1b2f60d0d68e969ab2946e7c51d27476 # Parent 092302c33462716a5a0b5612f7676d76ccd8fe94 Remove useless symlink diff -r 092302c33462 -r a041d8bd1b2f www/screenshots/index.html --- a/www/screenshots/index.html Wed Jun 15 23:22:39 2011 +0200 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,1 +0,0 @@ -oldindex.html \ No newline at end of file 1308305226.0 From alessiogirobogani at gmail.com Fri Jun 17 10:07:23 2011 From: alessiogirobogani at gmail.com (Alessio Igor Bogani) Date: Fri, 17 Jun 2011 12:07:23 +0200 Subject: [Aboriginal] [PATCH] Avoid sanitizing useful *_proxy environment variables Message-ID: # HG changeset patch # User Alessio Igor Bogani # Date 1308302791 -7200 # Node ID e26390768b3c371440af7e9a6619705c5651a6d6 # Parent a041d8bd1b2f60d0d68e969ab2946e7c51d27476 Avoid sanitizing useful *_proxy environment variables diff -r a041d8bd1b2f -r e26390768b3c sources/utility_functions.sh --- a/sources/utility_functions.sh Fri Jun 17 11:24:53 2011 +0200 +++ b/sources/utility_functions.sh Fri Jun 17 11:26:31 2011 +0200 @@ -17,6 +17,7 @@ TEMP="$TEMP,LANG,PATH,SHELL,TERM,USER,USERNAME,LOGNAME,PWD,EDITOR,HOME,DISPLAY,_" TEMP="$TEMP,TOPSHELL,START_TIME,STAGE_NAME,TOOLCHAIN_PREFIX,HOST_ARCH" + TEMP="$TEMP,http_proxy,ftp_proxy,https_proxy,no_proxy" # Unset any variable we don't recognize. It can screw up the build. 1308305243.0 From alessiogirobogani at gmail.com Fri Jun 17 10:17:29 2011 From: alessiogirobogani at gmail.com (Alessio Igor Bogani) Date: Fri, 17 Jun 2011 12:17:29 +0200 Subject: [Aboriginal] [PATCH] Update MANIFEST file Message-ID: <02ada84fcbc518e206e5.1308305849@granny> # HG changeset patch # User Alessio Igor Bogani # Date 1308305734 -7200 # Node ID 02ada84fcbc518e206e576bea386a55630f02aac # Parent e26390768b3c371440af7e9a6619705c5651a6d6 Update MANIFEST file diff -r e26390768b3c -r 02ada84fcbc5 sources/functions.sh --- a/sources/functions.sh Fri Jun 17 11:26:31 2011 +0200 +++ b/sources/functions.sh Fri Jun 17 12:15:34 2011 +0200 @@ -290,7 +290,7 @@ Built on $(date +%F) from: Build script: - Firmware Linux (http://landley.net/code/firmware) $FWL_VERS + Aboriginal Linux (http://landley.net/aboriginal) $FWL_VERS Base packages: uClibc (http://uclibc.org) $(identify_release uClibc) 1308305849.0 From alessiogirobogani at gmail.com Fri Jun 17 10:24:31 2011 From: alessiogirobogani at gmail.com (Alessio Igor Bogani) Date: Fri, 17 Jun 2011 12:24:31 +0200 Subject: [Aboriginal] [PATCH] Replace FWL with ABL in the scripts Message-ID: # HG changeset patch # User Alessio Igor Bogani # Date 1308306260 -7200 # Node ID e5cd1b9a5b4052f27204b39c168401675b778198 # Parent 02ada84fcbc518e206e576bea386a55630f02aac Replace FWL with ABL in the scripts diff -r 02ada84fcbc5 -r e5cd1b9a5b40 download.sh --- a/download.sh Fri Jun 17 12:15:34 2011 +0200 +++ b/download.sh Fri Jun 17 12:24:20 2011 +0200 @@ -115,7 +115,7 @@ # Create a MANIFEST file listing package versions. # This can optionally call source control systems (hg and svn) to get version -# information for the FWL build scripts and any USE_UNSTABLE packages. These +# information for the ABL build scripts and any USE_UNSTABLE packages. These # are intentionally excluded from the new path setup by host-tools.sh, so # just in case we've already run that use $OLDPATH for this. diff -r 02ada84fcbc5 -r e5cd1b9a5b40 sources/functions.sh --- a/sources/functions.sh Fri Jun 17 12:15:34 2011 +0200 +++ b/sources/functions.sh Fri Jun 17 12:24:20 2011 +0200 @@ -110,7 +110,7 @@ { PREVIOUS= (cd "$STAGE_DIR" || dienow - find . -depth -newer "$CURSRC/FWL-TIMESTAMP" \ + find . -depth -newer "$CURSRC/ABL-TIMESTAMP" \ | sed -e 's/^.//' -e 's/^.//' -e '/^$/d' ) | while read i do @@ -208,7 +208,7 @@ if [ ! -z "$BINARY_PACKAGE_TARBALLS" ] then - touch "$CURSRC/FWL-TIMESTAMP" || dienow + touch "$CURSRC/ABL-TIMESTAMP" || dienow TIME=$(date +%s) while true do @@ -281,16 +281,16 @@ do_readme() { - # Grab FWL version number + # Grab ABL version number - [ -z "$FWL_VERS" ] && - FWL_VERS="mercurial rev $(cd "$TOP"; hg tip 2>/dev/null | sed -n 's/changeset: *\([0-9]*\).*/\1/p')" + [ -z "$ABL_VERS" ] && + ABL_VERS="mercurial rev $(cd "$TOP"; hg tip 2>/dev/null | sed -n 's/changeset: *\([0-9]*\).*/\1/p')" cat << EOF Built on $(date +%F) from: Build script: - Aboriginal Linux (http://landley.net/aboriginal) $FWL_VERS + Aboriginal Linux (http://landley.net/aboriginal) $ABL_VERS Base packages: uClibc (http://uclibc.org) $(identify_release uClibc) 1308306271.0 From rob at landley.net Fri Jun 17 17:21:27 2011 From: rob at landley.net (Rob Landley) Date: Fri, 17 Jun 2011 12:21:27 -0500 Subject: [Aboriginal] [PATCH] Replace FWL with ABL in the scripts In-Reply-To: References: Message-ID: <4DFB8D17.3060805@landley.net> Yay updates! I applied the first three already. I got a bounce message from gmail for your address earlier today (although that could mean anything). Should it be "giro" or "igor" in the middle there? The fourth patch: On 06/17/2011 05:24 AM, Alessio Igor Bogani wrote: > # HG changeset patch > # User Alessio Igor Bogani > # Date 1308306260 -7200 > # Node ID e5cd1b9a5b4052f27204b39c168401675b778198 > # Parent 02ada84fcbc518e206e576bea386a55630f02aac > Replace FWL with ABL in the scripts I don't use the "ABL" abbreviation for Aboriginal. (FWL wasn't a very good one either, but it wasn't tempting to try to pronounce. ABL looks like "able" and people could try to look up "Able Linux" and get confused.) I'll clean up the FWL references here myself (unlikely to have time on my lunch break, but I can do it this evening). Hmmm... I wonder when the last time I tested BINARY_PACKAGE_TARBALLS was? Oh well, no time like the present... Rob 1308331287.0 From rob at landley.net Sun Jun 19 18:20:25 2011 From: rob at landley.net (Rob Landley) Date: Sun, 19 Jun 2011 13:20:25 -0500 Subject: [Aboriginal] Why more/chroot-splice.sh doesn't work anymore. Message-ID: <4DFE3DE9.4070905@landley.net> So I'm poking at lfs-bootstrap this weekend, and my design assumptions seem to have crossed paths and cancelled each other out at some point. Problem: building big things under qemu is slow enough that for test purposes, it's nice to smoketest stuff on i686 under chroot and _then_ worry about building it for the target. I can set up such a chroot by hand, but it's tedious and there should be an easy way to _automate_ it given a root filesystem and a control image filesystem. After the build, extracted versions of the root filesystem and control image are available in the build/root-filesystem-$ARCH directory and build/control-images/lfs-bootstrap directories. Many moons ago I made a more/chroot-splice.sh script that copies them together and prints out the sudo command line you have to run to chroot into it, and the command line you run inside it to kick off the build. (It doesn't do these steps itself because none of my build scripts ever require root access. Possibly this script should call sudo anyway, since it's really an optional development/debugging thing and the whole POINT is to set up a chroot? Anyway...) Once upon a time, /mnt/init would check if the root filesystem was writeable, and if so it would skip the setup-chroot step: instead it would just build and install packages into the host root filesystem. I suspect that behavior was in the gentoo/LFS bootstrapping stuff I did for Qualcomm's Hexagon port last year, and never actually made it into the reimplementation I did for public consumption. However, the chroot-splice script was written on the assumption that /mnt/init did that. If it _doesn't_, then it copies the gigabyte of lfs-bootstrap files in /mnt into the new chroot (because the copy is using find -xdev). And then it bind mounts /mnt over that. This may actually work (never let it run long enough find out), but it's HORRIBLY inefficient. So yeah, just fixed that. Then I need to track down why one of m4's configure tests ("does some string function work in linear time, apparently not") is hanging forever. I remember looking at it before and it was setting an alarm() to kill it after a few seconds, and the alarm was never triggering due to uClibc's signal handling with -lpthreads being crap. I believe I fixed it with a background process that did "while true; do sleep 5; killall nameofthetestbinary; done". Which means I should look at the current uClibc release again to see if signal handling sucks less under NPTL. But lemme re-establish an LFS native build baseline first. And possibly update it to the current LFS release... Rob P.S. I used to post this kind of stuff to my blog, but I'm trying to post it here now. Lemme know if you have a preference where it should go? If I do post it here, the list traffic is mostly me blathering on about technobabble details, but if I blog it the list traffic tends to get really sparse and the comments I get about it are in private email instead of here... 1308507625.0 From bjorn.forsman at gmail.com Sun Jun 19 20:29:20 2011 From: bjorn.forsman at gmail.com (=?UTF-8?Q?Bj=C3=B8rn_Forsman?=) Date: Sun, 19 Jun 2011 22:29:20 +0200 Subject: [Aboriginal] Why more/chroot-splice.sh doesn't work anymore. In-Reply-To: <4DFE3DE9.4070905@landley.net> References: <4DFE3DE9.4070905@landley.net> Message-ID: On 19 June 2011 20:20, Rob Landley wrote: [...] > P.S. ?I used to post this kind of stuff to my blog, but I'm trying to > post it here now. ?Lemme know if you have a preference where it should > go? ?If I do post it here, the list traffic is mostly me blathering on > about technobabble details, but if I blog it the list traffic tends to > get really sparse and the comments I get about it are in private email > instead of here... I like to have this on the list. Do you have RSS on your blog? 1308515360.0 From rob at landley.net Mon Jun 20 02:53:48 2011 From: rob at landley.net (Rob Landley) Date: Sun, 19 Jun 2011 21:53:48 -0500 Subject: [Aboriginal] Why more/chroot-splice.sh doesn't work anymore. In-Reply-To: References: <4DFE3DE9.4070905@landley.net> Message-ID: <4DFEB63C.5000807@landley.net> On 06/19/2011 03:29 PM, Bj?rn Forsman wrote: > On 19 June 2011 20:20, Rob Landley wrote: > [...] >> P.S. I used to post this kind of stuff to my blog, but I'm trying to >> post it here now. Lemme know if you have a preference where it should >> go? If I do post it here, the list traffic is mostly me blathering on >> about technobabble details, but if I blog it the list traffic tends to >> get really sparse and the comments I get about it are in private email >> instead of here... > > I like to have this on the list. Do you have RSS on your blog? Yup, link's at the top of the page. Rob 1308538428.0 From onefang at gmail.com Mon Jun 20 09:36:09 2011 From: onefang at gmail.com (David Seikel) Date: Mon, 20 Jun 2011 19:36:09 +1000 Subject: [Aboriginal] Why more/chroot-splice.sh doesn't work anymore. In-Reply-To: <4DFE3DE9.4070905@landley.net> References: <4DFE3DE9.4070905@landley.net> Message-ID: <20110620193609.3b360490@U64-desktop> I've got to the stage where I'm actually writing the app that is the device I'm being paid to develop, now that I have enough OS to actually support running it on real hardware. One of the reasons I chose EFL as the graphic framework is that it's happy running on framebuffer or X. So now I can develop the app in my comfortable X environment, and test it every now and then on the real hardware and it's frame buffer. Which means I now have two development streams, and I can switch between them as needed, more or less independently. Polish the OS, and write the app. For the polish the OS part, I'm thinking about how to split things into "stuff needed at run time to support the app", and "stuff needed at compile time to compile everything". Which brings me on topic. On Sun, 19 Jun 2011 13:20:25 -0500 Rob Landley wrote: > So I'm poking at lfs-bootstrap this weekend, and my design assumptions > seem to have crossed paths and cancelled each other out at some point. Cool. I'm yet to update to the latest Aboriginal version, but I should this week. I have been thinking about those tags you have in the package list, and what to do about them. The "# busybox" tags seem to be more useful as a comment about why some LFS package was not built, it's functionality being provided by busybox. The "# development" tag is much more interesting. These are things that could be built into some mountable directory somewhere, such that it can be unplugged, leaving just the built system and an empty mount point. Problem number one is just basic - where should this mount point be? /usr/devel strikes me as a good choice for reasons I wont go into unless someone thinks otherwise. Next is that while the includes from all the other packages should likely go into /usr/devel, libs may or may not. Compile time libs and libs used statically yes, dynamic libs no. Will the user compile everything only statically? For my project, parts of EFL don't like being compiled static, so I'm compiling the app dynamically. I think Rob mentioned something about how to actually drive it all. > Which means I should look at the current uClibc release again to see > if signal handling sucks less under NPTL. But lemme re-establish an > LFS native build baseline first. And possibly update it to the > current LFS release... I'm in two minds about that. I'm using GRUB, and had to back up to the last LFS that still used "legacy" GRUB. On the other hand, I have stripped out most of LFS, either using busybox, or just dumping stuff that is not relevant. Those small parts of LFS I have left are the version from Aboriginal, and the GRUB I'm using seems happy about that. Meh, lets see what happens with what you produce. > P.S. I used to post this kind of stuff to my blog, but I'm trying to > post it here now. Lemme know if you have a preference where it should > go? If I do post it here, the list traffic is mostly me blathering on > about technobabble details, but if I blog it the list traffic tends to > get really sparse and the comments I get about it are in private email > instead of here... Saves me having to deal with your blog stuff that is not about Aboriginal development. I'm a compulsive reader, and avoiding temptation is always good. B-) -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From onefang at gmail.com Mon Jun 20 09:43:37 2011 From: onefang at gmail.com (David Seikel) Date: Mon, 20 Jun 2011 19:43:37 +1000 Subject: [Aboriginal] Why more/chroot-splice.sh doesn't work anymore. In-Reply-To: <20110620193609.3b360490@U64-desktop> References: <4DFE3DE9.4070905@landley.net> <20110620193609.3b360490@U64-desktop> Message-ID: <20110620194337.5ec6f3e7@U64-desktop> On Mon, 20 Jun 2011 19:36:09 +1000 David Seikel wrote: > The "# development" tag is much more interesting. These are things > that could be built into some mountable directory somewhere, such that > it can be unplugged, leaving just the built system and an empty mount > point. Oops, forget to mention - separating the development part of the result from the run time part is part of the scope of my contract. So I'll likely be poking at that soonish. The reason why lfs-bootstrap is still relevant to my project is that it's an example, starting point, and useful tools. My variation of it for this project tries to stick with working the same way. So developing this for one, works for the other. B-) -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From bjorn.forsman at gmail.com Mon Jun 20 10:32:02 2011 From: bjorn.forsman at gmail.com (=?UTF-8?Q?Bj=C3=B8rn_Forsman?=) Date: Mon, 20 Jun 2011 12:32:02 +0200 Subject: [Aboriginal] Why more/chroot-splice.sh doesn't work anymore. In-Reply-To: <4DFEB63C.5000807@landley.net> References: <4DFE3DE9.4070905@landley.net> <4DFEB63C.5000807@landley.net> Message-ID: 2011/6/20 Rob Landley : > On 06/19/2011 03:29 PM, Bj?rn Forsman wrote: >> [...] Do you have RSS on your blog? > > Yup, link's at the top of the page. Thanks! 1308565922.0 From alanpost at sunflowerriver.org Mon Jun 20 14:14:00 2011 From: alanpost at sunflowerriver.org (Alan Post) Date: Mon, 20 Jun 2011 08:14:00 -0600 Subject: [Aboriginal] Why more/chroot-splice.sh doesn't work anymore. In-Reply-To: <4DFE3DE9.4070905@landley.net> References: <4DFE3DE9.4070905@landley.net> Message-ID: <20110620141400.GA37292@alice.local> On Sun, Jun 19, 2011 at 01:20:25PM -0500, Rob Landley wrote: > P.S. I used to post this kind of stuff to my blog, but I'm trying to > post it here now. Lemme know if you have a preference where it should > go? If I do post it here, the list traffic is mostly me blathering on > about technobabble details, but if I blog it the list traffic tends to > get really sparse and the comments I get about it are in private email > instead of here... I like reading them here. It was the first thing I read from my weekend e-mail load. -Alan -- .i ma'a lo bradi ku penmi gi'e du 1308579240.0 From rob at landley.net Tue Jun 21 00:08:16 2011 From: rob at landley.net (Rob Landley) Date: Mon, 20 Jun 2011 19:08:16 -0500 Subject: [Aboriginal] Why more/chroot-splice.sh doesn't work anymore. In-Reply-To: <20110620193609.3b360490@U64-desktop> References: <4DFE3DE9.4070905@landley.net> <20110620193609.3b360490@U64-desktop> Message-ID: <4DFFE0F0.3050204@landley.net> On 06/20/2011 04:36 AM, David Seikel wrote: > I've got to the stage where I'm actually writing the app that is the > device I'm being paid to develop, now that I have enough OS to actually > support running it on real hardware. One of the reasons I chose EFL as > the graphic framework is that it's happy running on framebuffer or X. > So now I can develop the app in my comfortable X environment, and test > it every now and then on the real hardware and it's frame buffer. > > Which means I now have two development streams, and I can switch > between them as needed, more or less independently. Polish the OS, > and write the app. > > For the polish the OS part, I'm thinking about how to split things into > "stuff needed at run time to support the app", and "stuff needed at > compile time to compile everything". Which brings me on topic. I _really_ need to do a blfs-bootstrap with the beyond linux from scratch bits about X11, but it might have to run on top of the output of the linux from scratch build. Dunno what the prerequisites are... I'm playing with the uClibc 0.9.31.1 release to see if fixing the stdio locking made the various non-x86 targets less strangely flaky. (Fingers crossed, being able to reliably build LFS on several different hardware targets would be WONDERFUL)... > On Sun, 19 Jun 2011 13:20:25 -0500 Rob Landley wrote: > >> So I'm poking at lfs-bootstrap this weekend, and my design assumptions >> seem to have crossed paths and cancelled each other out at some point. > > Cool. I'm yet to update to the latest Aboriginal version, but I should > this week. > > I have been thinking about those tags you have in the package list, and > what to do about them. The "# busybox" tags seem to be more useful as > a comment about why some LFS package was not built, it's functionality > being provided by busybox. They're there so you can filter those out from your build, yes. :) Haven't tested it to see what other package builds break yet. I know a couple INSIST on gawk being there but you can make a gawk wrapper script that calls busybox awk and then they're fine. (Lazy build systems...) > The "# development" tag is much more interesting. These are things > that could be built into some mountable directory somewhere, such that > it can be unplugged, leaving just the built system and an empty mount > point. Worth a try, certainly... > Problem number one is just basic - where should this mount point > be? /usr/devel strikes me as a good choice for reasons I wont go into > unless someone thinks otherwise. No strong opinion. That works. > Next is that while the includes from all the other packages should > likely go into /usr/devel, libs may or may not. Compile time libs and > libs used statically yes, dynamic libs no. Will the user compile > everything only statically? For my project, parts of EFL don't like > being compiled static, so I'm compiling the app dynamically. parts of _everything_ don't like being compiled statically. I need to write a FAQ entry on that, and possibly break it out into more generic documentation. I note that I wrote half of busybox.net's FAQ and a lot of stuff (espcially near the end) is just general development documentation for embedded stuff. "So _that's_ what vfork() does" and all that... > I think Rob mentioned something about how to actually drive it all. Feedback on requirements is great. Implementation I tend to mull over at length and discard LOTS of ideas before I find something I'm happy with, so I'm open to suggestions but since I discard 95% of my _own_ ideas here... >> Which means I should look at the current uClibc release again to see >> if signal handling sucks less under NPTL. But lemme re-establish an >> LFS native build baseline first. And possibly update it to the >> current LFS release... > > I'm in two minds about that. I'm using GRUB, and had to back up to the > last LFS that still used "legacy" GRUB. On the other hand, I have > stripped out most of LFS, either using busybox, or just dumping stuff > that is not relevant. Those small parts of LFS I have left are the > version from Aboriginal, and the GRUB I'm using seems happy about that. Bootloaders (other than QEMU's -kernel option) are outside the scope of this project. :) Afraid you're on your own there. I plead ignorance. > Meh, lets see what happens with what you produce. > >> P.S. I used to post this kind of stuff to my blog, but I'm trying to >> post it here now. Lemme know if you have a preference where it should >> go? If I do post it here, the list traffic is mostly me blathering on >> about technobabble details, but if I blog it the list traffic tends to >> get really sparse and the comments I get about it are in private email >> instead of here... > > Saves me having to deal with your blog stuff that is not about > Aboriginal development. I'm a compulsive reader, and avoiding > temptation is always good. B-) Seems vaguely unanimous... Rob 1308614896.0 From rob at landley.net Tue Jun 21 00:14:26 2011 From: rob at landley.net (Rob Landley) Date: Mon, 20 Jun 2011 19:14:26 -0500 Subject: [Aboriginal] Why more/chroot-splice.sh doesn't work anymore. In-Reply-To: <20110620194337.5ec6f3e7@U64-desktop> References: <4DFE3DE9.4070905@landley.net> <20110620193609.3b360490@U64-desktop> <20110620194337.5ec6f3e7@U64-desktop> Message-ID: <4DFFE262.7000507@landley.net> On 06/20/2011 04:43 AM, David Seikel wrote: > On Mon, 20 Jun 2011 19:36:09 +1000 David Seikel > wrote: > >> The "# development" tag is much more interesting. These are things >> that could be built into some mountable directory somewhere, such that >> it can be unplugged, leaving just the built system and an empty mount >> point. > > Oops, forget to mention - separating the development part of the result > from the run time part is part of the scope of my contract. So I'll > likely be poking at that soonish. > > The reason why lfs-bootstrap is still relevant to my project is that > it's an example, starting point, and useful tools. My variation of it > for this project tries to stick with working the same way. So > developing this for one, works for the other. B-) I made it modular intentionally. Any package builds you feel like pushing upstream, I'm happy to take a look at. That said, I don't want to fall into the distro trap. If the bootstrap examples get too out of control, I'll kick 'em into their own repository and try to convince somebody else to maintain them. :) And _that_ said, Linux From Scratch itself is very well maintained, and "bootstrap-gentoo", "bootstrap-debian", "bootstrap-fedora" are all well defined sub-projects with a clear stopping point. Part of the reason I HAVEN'T done a blfs-bootstrap yet is doing the whole thing is too big and fuzzily-defined subsets don't have a bright line constraining their growth... Adding configuration knobs to lfs-bootstrap so it can put the dev packages in a subdir and not replace the busybox stuff is acceptable. Adding packages that aren't IN Linux From Scratch is a much harder sell. (I see the utility, but that way lies OpenEmbedded. speaking of which, I want to do an openembedded-bootstrap control image. Bitbake is like portage and rpm and dpkg, give it an environment and let it build its repo natively...) Rob 1308615266.0 From alessioigorbogani at gmail.com Tue Jun 21 07:35:24 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Tue, 21 Jun 2011 09:35:24 +0200 Subject: [Aboriginal] [PATCH] Check the user input about architecture Message-ID: <67accb5d252602b3b2ad.1308641724@granny> # HG changeset patch # User Alessio Igor Bogani # Date 1308641684 -7200 # Node ID 67accb5d252602b3b2ad2d0b8aa6fd9a62c74030 # Parent d16057658651a27f2c9b99764367475f36285e2e Check the user input about architecture diff -r d16057658651 -r 67accb5d2526 build.sh --- a/build.sh Sun Jun 19 21:51:42 2011 -0500 +++ b/build.sh Tue Jun 21 09:34:44 2011 +0200 @@ -32,15 +32,20 @@ # Start with some housekeeping stuff. If this script was run with no # arguments, list available architectures out of sources/targets. +source sources/utility_functions.sh || exit 1 + +SUPPORTED_ARCHS=$(ls sources/targets) + if [ $# -ne 1 ] then echo "Usage: $0 TARGET" echo "Supported architectures:" - cd sources/targets - ls + echo $SUPPORTED_ARCHS exit 1 +else + is_in_list $1 `echo $SUPPORTED_ARCHS | tr " " ","` || dienow fi ARCH="$1" 1308641724.0 From alessioigorbogani at gmail.com Wed Jun 22 09:46:44 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Wed, 22 Jun 2011 11:46:44 +0200 Subject: [Aboriginal] [PATCH] Move i586 to virtio Message-ID: <9cb1d126a6102d47f326.1308736004@granny> # HG changeset patch # User Alessio Igor Bogani # Date 1308735970 -7200 # Node ID 9cb1d126a6102d47f3266a148247c97758a01bfd # Parent 67accb5d252602b3b2ad2d0b8aa6fd9a62c74030 Move i586 to virtio diff -r 67accb5d2526 -r 9cb1d126a610 sources/targets/i586/settings --- a/sources/targets/i586/settings Tue Jun 21 09:34:44 2011 +0200 +++ b/sources/targets/i586/settings Wed Jun 22 11:46:10 2011 +0200 @@ -13,7 +13,7 @@ GCC_FLAGS= QEMU_TEST=$KARCH -ROOT=hda +ROOT=vda CONSOLE=ttyS0 # Gentoo from Scratch @@ -29,14 +29,22 @@ LINUX_CONFIG=" CONFIG_M586=y CONFIG_ACPI=y -CONFIG_BLK_DEV_PIIX=y -CONFIG_NETDEV_1000=y -CONFIG_E1000=y CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y +CONFIG_HW_RANDOM_VIRTIO=y +CONFIG_PARAVIRT_GUEST=y +CONFIG_VIRTIO_BLK=y +CONFIG_VIRTIO_CONSOLE=y +CONFIG_KVM_CLOCK=y +CONFIG_KVM_GUEST=y " emulator_command() { - echo qemu -cpu pentium $(qemu_defaults "$@") +# echo qemu -cpu pentium $(qemu_defaults "$@") + echo "if [ \"\$WITH_HDC\"X\"\" != \"X\" ]; then WITH_HDC=\"-drive file=\$HDC,if=virtio\"; fi" + echo "if [ \"\$WITH_HDB\"X\"\" != \"X\" ]; then WITH_HDB=\"-drive file=\$HDB,if=virtio\"; fi" + echo -n "qemu -cpu pentium -nographic -no-reboot -kernel $KERNEL \$WITH_HDC \$WITH_HDB" + [ "$SYSIMAGE_TYPE" != "initramfs" ] && echo -n " -drive file=$IMAGE,if=virtio" + echo -n " -append \"$(kernel_cmdline)\" \$QEMU_EXTRA -net nic,model=virtio -net user" } 1308736004.0 From rob at landley.net Wed Jun 22 14:42:44 2011 From: rob at landley.net (Rob Landley) Date: Wed, 22 Jun 2011 09:42:44 -0500 Subject: [Aboriginal] [PATCH] Move i586 to virtio In-Reply-To: <9cb1d126a6102d47f326.1308736004@granny> References: <9cb1d126a6102d47f326.1308736004@granny> Message-ID: <4E01FF64.302@landley.net> On 06/22/2011 04:46 AM, Alessio Igor Bogani wrote: > # HG changeset patch > # User Alessio Igor Bogani > # Date 1308735970 -7200 > # Node ID 9cb1d126a6102d47f3266a148247c97758a01bfd > # Parent 67accb5d252602b3b2ad2d0b8aa6fd9a62c74030 > Move i586 to virtio Hmmm... Good work, very interesting approach. I want to encourage this, but I'm not yet ready to merge it. I've poked at virtio before (and discussed it in email too, was that with you?), but couldn't make it work to my satisfaction. (It's on my todo list, but I'd be thrilled if you could make it work so I don't have to. :) Problems I encountered: 1) The documentation is horrible. (I expect you've noticed.) 2) Not all deployed versions of qemu/kvm support this yet. (Actually, last I checked you still had to build qemu from the -git tree to get this, _and_ you had to install some obscure ACL support package without which configure would silently disable it without explanation. If the i586 target is unconditionally switched to virtio without some kind of config knob, this effectively _disables_ it for people using qemu/kvm from ubuntu 10.04 LTS. They ship a new LTS in October which might support it, but it's not there yet...) 3) It doesn't work on anything _except_ an x86 target; there's no virtio for arm or mips that I've found. That means turning on autoconf is an optimization for x86, the one platform on which you can just use chroot if you really need performance. > diff -r 67accb5d2526 -r 9cb1d126a610 sources/targets/i586/settings > --- a/sources/targets/i586/settings Tue Jun 21 09:34:44 2011 +0200 > +++ b/sources/targets/i586/settings Wed Jun 22 11:46:10 2011 +0200 > @@ -13,7 +13,7 @@ > GCC_FLAGS= > QEMU_TEST=$KARCH > > -ROOT=hda > +ROOT=vda > CONSOLE=ttyS0 > > # Gentoo from Scratch > @@ -29,14 +29,22 @@ > LINUX_CONFIG=" > CONFIG_M586=y > CONFIG_ACPI=y > -CONFIG_BLK_DEV_PIIX=y > -CONFIG_NETDEV_1000=y > -CONFIG_E1000=y > CONFIG_SERIAL_8250=y > CONFIG_SERIAL_8250_CONSOLE=y > +CONFIG_HW_RANDOM_VIRTIO=y > +CONFIG_PARAVIRT_GUEST=y > +CONFIG_VIRTIO_BLK=y > +CONFIG_VIRTIO_CONSOLE=y > +CONFIG_KVM_CLOCK=y > +CONFIG_KVM_GUEST=y > " You switch on VIRTIO_CONSOLE but don't use it? > emulator_command() > { > - echo qemu -cpu pentium $(qemu_defaults "$@") > +# echo qemu -cpu pentium $(qemu_defaults "$@") > + echo "if [ \"\$WITH_HDC\"X\"\" != \"X\" ]; then WITH_HDC=\"-drive file=\$HDC,if=virtio\"; fi" > + echo "if [ \"\$WITH_HDB\"X\"\" != \"X\" ]; then WITH_HDB=\"-drive file=\$HDB,if=virtio\"; fi" > + echo -n "qemu -cpu pentium -nographic -no-reboot -kernel $KERNEL \$WITH_HDC \$WITH_HDB" > + [ "$SYSIMAGE_TYPE" != "initramfs" ] && echo -n " -drive file=$IMAGE,if=virtio" > + echo -n " -append \"$(kernel_cmdline)\" \$QEMU_EXTRA -net nic,model=virtio -net user" > } > Ironically this is exactly the sort of thing the hw-target support was for. "Run the same userspace on a different board", which mostly just means a target building a different kernel but using the same root filesystem (and thus the same compilers) as another filesystem. I have no problem with experimentally adding new features, it's accidentally breaking the base i586 target for people who can't use those new features I'm worried about. Lemme work on fixing the hw-target support this weekend (migrating it to a new less intrusive mechanism, actually) and then we can add this as a new i586-virtio target. Then if enough stuff grows virtio options we can collapse it back into the base in a future release. Sound reasonable? Rob 1308753764.0 From rob at landley.net Thu Jun 23 02:53:08 2011 From: rob at landley.net (Rob Landley) Date: Wed, 22 Jun 2011 21:53:08 -0500 Subject: [Aboriginal] [PATCH] Check the user input about architecture In-Reply-To: <67accb5d252602b3b2ad.1308641724@granny> References: <67accb5d252602b3b2ad.1308641724@granny> Message-ID: <4E02AA94.4020200@landley.net> On 06/21/2011 02:35 AM, Alessio Igor Bogani wrote: > # HG changeset patch > # User Alessio Igor Bogani > # Date 1308641684 -7200 > # Node ID 67accb5d252602b3b2ad2d0b8aa6fd9a62c74030 > # Parent d16057658651a27f2c9b99764367475f36285e2e > Check the user input about architecture > > diff -r d16057658651 -r 67accb5d2526 build.sh > --- a/build.sh Sun Jun 19 21:51:42 2011 -0500 > +++ b/build.sh Tue Jun 21 09:34:44 2011 +0200 > @@ -32,15 +32,20 @@ > # Start with some housekeeping stuff. If this script was run with no > # arguments, list available architectures out of sources/targets. > > +source sources/utility_functions.sh || exit 1 > + > +SUPPORTED_ARCHS=$(ls sources/targets) > + Hmmm... I've avoided sourcing any of the config or functions files because I wanted this file to stay simple and self-contained. It _started_ as a very simple wrapper that just called the other stages in sequence, and conceptually I'd like it to stay that way. Unfortunately, it's gotten more complex over the years, with dependency tracking and config symbols. (I haven't allowed dependency tracking _within_ build stages, system-image.sh was doing some for a while but I ripped it back out again to simplify things. But build.sh checks tarballs to see if it needs to rebuild stages.) That said, we don't really need is_in_list here, and that's all the include is for. You can just go: [ -e "sources/targets/$1" ] instead. Could you redo the patch with that? (Or I can if you prefer...) Thanks, Rob 1308797588.0 From alessioigorbogani at gmail.com Thu Jun 23 06:50:08 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Thu, 23 Jun 2011 08:50:08 +0200 Subject: [Aboriginal] [PATCH] Check the user input about architecture In-Reply-To: <4E02AA94.4020200@landley.net> References: <67accb5d252602b3b2ad.1308641724@granny> <4E02AA94.4020200@landley.net> Message-ID: Hi, 2011/6/23 Rob Landley : [...] > That said, we don't really need is_in_list here, and that's all the > include is for. ?You can just go: > > ?[ -e "sources/targets/$1" ] > > instead. ?Could you redo the patch with that? ?(Or I can if you prefer...) Please do! Thanks. Ciao, Alessio 1308811808.0 From alessioigorbogani at gmail.com Thu Jun 23 08:50:11 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Thu, 23 Jun 2011 10:50:11 +0200 Subject: [Aboriginal] [PATCH] Move i586 to virtio In-Reply-To: <4E01FF64.302@landley.net> References: <9cb1d126a6102d47f326.1308736004@granny> <4E01FF64.302@landley.net> Message-ID: Hi, 2011/6/22 Rob Landley : [...] > 2) Not all deployed versions of qemu/kvm support this yet. (Actually, > last I checked you still had to build qemu from the -git tree to get > this, _and_ you had to install some obscure ACL support package without > which configure would silently disable it without explanation. ?If the > i586 target is unconditionally switched to virtio without some kind of > config knob, this effectively _disables_ it for people using qemu/kvm > from ubuntu 10.04 LTS. ?They ship a new LTS in October which might > support it, but it's not there yet...) I don't understand: The i586 modified version works on my 10.04.02 system What are the problems are you talking about? > 3) It doesn't work on anything _except_ an x86 target; there's no virtio > for arm or mips that I've found. ?That means turning on autoconf is an > optimization for x86, the one platform on which you can just use chroot > if you really need performance. I need performance when I test kernels which differs from the one used by my host. So I can't use chroot nor (unfortunately) LXC. AFAIK ia64, mips, powerpc, s390, sh, tile and x86 should support VIRTIO devices. Obviously KVM kernel config options works on x86 only. [...] > You switch on VIRTIO_CONSOLE but don't use it? Yeah because at the moment I don't know how make it works! [...] > Lemme work on fixing the hw-target support this weekend (migrating it to > a new less intrusive mechanism, actually) and then we can add this as a > new i586-virtio target. ?Then if enough stuff grows virtio options we > can collapse it back into the base in a future release. ?Sound reasonable? Yes definitely. I tried creating the i586-virtio target but binutils refused to build so I made the changes in place. Thanks! Ciao, Alessio 1308819011.0 From alessioigorbogani at gmail.com Fri Jun 24 08:36:22 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Fri, 24 Jun 2011 10:36:22 +0200 Subject: [Aboriginal] Would Aboriginal fit my needs? Message-ID: Hi all, I would want try to adapt Aboriginal to my needs which are: 1) Avoid the build of any host tools (squashfs, genext2fs, make, distcc, e2fsprogs and all others): I would want use the ones provided by my distribution. 2) Avoid the build of cross-compiler: I would want use a pre-built one (Aboriginal, CodeSourcery or others). 3) Build target's things from DVCS (in my case I have a big local git tree with Linux, uClibc and busybox). How and where should I change Aboriginal to achieve those? Thanks in advance. Ciao, Alessio 1308904582.0 From rob at landley.net Sat Jun 25 00:28:10 2011 From: rob at landley.net (Rob Landley) Date: Fri, 24 Jun 2011 19:28:10 -0500 Subject: [Aboriginal] Would Aboriginal fit my needs? In-Reply-To: References: Message-ID: <4E052B9A.7070707@landley.net> On 06/24/2011 03:36 AM, Alessio Igor Bogani wrote: > Hi all, > > I would want try to adapt Aboriginal to my needs which are: > > 1) Avoid the build of any host tools (squashfs, genext2fs, make, > distcc, e2fsprogs and all others): I would want use the ones provided > by my distribution. Skip the host-tools.sh step and it'll use the ones out of your existing $PATH. You can comment that line out of build.sh, or write your own build.sh wrapper. It's designed to be orthogonal, each layer can be removed and either done without or replaced with something else. > 2) Avoid the build of cross-compiler: I would want use a pre-built one > (Aboriginal, CodeSourcery or others). Skip the simple-cross-compiler.sh stage and add your cross tools to the $PATH yourself instead. In theory if you're not using host-tools.sh then just add it to the $PATH and it'll pick it up out of there. (It'll need the appropriate prefix- names, might need to create a directory of symlinks to give it the names it expects.) > 3) Build target's things from DVCS (in my case I have a big local git > tree with Linux, uClibc and busybox). Natively build, you mean? What do you mean by "DVCS"? http://en.wikipedia.org/wiki/DVCS > How and where should I change Aboriginal to achieve those? For the first two, you shouldn't have to do any architectural work (although it's nto extensively tested and you could easily hit bugs), you just have to comment stages out of build.sh. For the third, I need more details about what you mean. Rob 1308961690.0 From onefang at gmail.com Sat Jun 25 01:13:21 2011 From: onefang at gmail.com (David Seikel) Date: Sat, 25 Jun 2011 11:13:21 +1000 Subject: [Aboriginal] Would Aboriginal fit my needs? In-Reply-To: <4E052B9A.7070707@landley.net> References: <4E052B9A.7070707@landley.net> Message-ID: <20110625111321.2410cd1a@U64-desktop> On Fri, 24 Jun 2011 19:28:10 -0500 Rob Landley wrote: > > 3) Build target's things from DVCS (in my case I have a big local > > git tree with Linux, uClibc and busybox). > > Natively build, you mean? > > What do you mean by "DVCS"? > > http://en.wikipedia.org/wiki/DVCS I suspect he means building out of tree from within his git repos. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From rob at landley.net Sat Jun 25 14:17:45 2011 From: rob at landley.net (Rob Landley) Date: Sat, 25 Jun 2011 09:17:45 -0500 Subject: [Aboriginal] Would Aboriginal fit my needs? In-Reply-To: <20110625111321.2410cd1a@U64-desktop> References: <4E052B9A.7070707@landley.net> <20110625111321.2410cd1a@U64-desktop> Message-ID: <4E05EE09.4070305@landley.net> On 06/24/2011 08:13 PM, David Seikel wrote: > On Fri, 24 Jun 2011 19:28:10 -0500 Rob Landley wrote: > >>> 3) Build target's things from DVCS (in my case I have a big local >>> git tree with Linux, uClibc and busybox). >> >> Natively build, you mean? >> >> What do you mean by "DVCS"? >> >> http://en.wikipedia.org/wiki/DVCS > > I suspect he means building out of tree from within his git repos. He has a git repo with the binary output of the build? (The root-filesystem-$ARCH files?) Personally I'd probably try to build git natively within the repo, then use it to check out source and build that source. But there are a zillion potential usage patterns and in theory they all work. My project makes a native root filesystem for target that you can build stuff in. How you go about doing that is really your affair, although I'm trying to work up some decent automated examples with the control-image bootstrap stuff... Rob 1309011465.0 From rob at landley.net Sat Jun 25 22:47:23 2011 From: rob at landley.net (Rob Landley) Date: Sat, 25 Jun 2011 17:47:23 -0500 Subject: [Aboriginal] Fwd: Re: Aboriginal. Wow! and Thanks! Message-ID: <4E06657B.2050709@landley.net> I got permission to move this thread onto the list, so I'm about to forward some of the backlog. FYI. -------- Original Message -------- Subject: Re: Aboriginal. Wow! and Thanks! Date: Tue, 14 Jun 2011 20:18:41 -0500 From: Rob Landley To: Paul Kramer On 06/14/2011 07:04 PM, Paul Kramer wrote: > Hello Rob, > > I just discovered your work today. THANKS! Glad you liked it. Oddly enough, I'm publishing 1.0.2 later today (based on commit 1375). > Just a bit about me. I was gatekeeper, build, release, test for > HP-UX 1.0, 1.1, 1.2, SunOS SparcStation 2 4.1.3, Solaris Kernel > SDK.... then I went off and started going into vertical product teams > setting their dev environments up, refactoring their source trees and > putting in new build system and automation, then off to the next > team. It's a gigantic pain in the ass, isn't it? > Well, I'm coming full circle. I needed to relearn so much I've > forgotten, went to LSF and CLSF, but was feeling something is > missing, it's a bit dated. So off to google search I go and I found > a discussion you were having with a LSF member, and you mentioned > Aboriginal. And so here I am. > > Here is my blog http://www.qbalsoftware.com ... I really like > helping teams get a killer dev, pre-integration environment in place. > I've thought many times over the years of created mini-distro, just > enough to get a board/system up. But at the same time solving a bunch > of fundamental problems... like autotools, and poor makefile > implementations, packaging, etc... You might want to look at the big presentation Mark and I did a few years ago. It's enormous and a bit stale, but it described the design and reasons for it more thoroughly than we've found time to do since: http://landley.net/aboriginal/downloads/presentation.pdf You might also want to look at the the Linux From Scratch entry in sources/control-images (the prebuilt binary of which is in downloads/binaries/control-images). That's my build automation infrastructure, with Linux From Scratch 6.7 used as an example. > While at Sun in the 90's and at 3 startups, I'd tear out entire > build infrastructure, restructure trees and put in implementations > that sort-a look like this: > > http://www.qbalsoftware.com/Site/Blog/Entries/2011/5/9_Makefiles_Generic_Flexible_and_Simple_Part-2.html > > > and a driver to crank out multi-variants builds like this > > http://www.qbalsoftware.com/Site/Blog/Entries/2011/5/9_Makefiles_Generic_Flexible_and_Simple_Part-3.html I'll look at this after I get the 1.0.2 release posted. > I'll show people that... in > http://www.qbalsoftware.com/Site/Blog/Entries/2011/5/9_Makefiles_Generic_Flexible_and_Simple_Part-2.html > > > this design... the makefile.targets is the build system... > makefile, makefile.package and makefile.resolve is another system... > its the data model that we use to drive the build. I could write that > stuff in any language and using the driver script I show above, pass > in the resolved properties. > > This is how easy I think build systems can be. Of course, what I'd > like to see is 'Make' revisited for the multi-variant problems we > face My approach to make, autoconf, and automake is a quote from Aliens: "I say we take off and nuke the entire site from orbit: only way to be sure." Make combines imperative and declarative code in the same file, and then people try to beat some semblance of order out of it with recursive make (the definitive takedown of which is http://aegis.sourceforge.net/auug97.pdf ) and then people write extensive systems like kbuild on top of it... The status quo is horrible. > Java people might call makefile.targets an Inversion of Control > container... the way I define all the tasks as properties, and can > override everything from the command line I look forward to reading your blog posts, and can't follow this part of the discussion until then... > BTW -- I think if more folks considered the pattern I'm using with > this makefile implementation, we'd see a lot of automation in > engineering teams get very simple. > > Thanks Again -- I cannot wait to get to work this evening with > Aborginal The new release is maybe 2 hours away from being posted... Rob 1309042043.0 From rob at landley.net Sat Jun 25 22:48:29 2011 From: rob at landley.net (Rob Landley) Date: Sat, 25 Jun 2011 17:48:29 -0500 Subject: [Aboriginal] Fwd: Re: Aboriginal. Wow! and Thanks! Message-ID: <4E0665BD.6070201@landley.net> -------- Original Message -------- Subject: Re: Aboriginal. Wow! and Thanks! Date: Sat, 18 Jun 2011 19:43:12 -0500 From: Rob Landley To: Paul Kramer On 06/15/2011 05:06 PM, Paul Kramer wrote: >> You might also want to look at the the Linux From Scratch entry in >> sources/control-images (the prebuilt binary of which is in >> downloads/binaries/control-images). That's my build automation >> infrastructure, with Linux From Scratch 6.7 used as an example. > > I'll definitely look at this. I'm poking at this now trying to get the packages to build on more platforms, and LFS had a new release I need to upgrade to, plus I need to do a BLFS that does at least X11... >>> While at Sun in the 90's and at 3 startups, I'd tear out entire >>> build infrastructure, restructure trees and put in >>> implementations that sort-a look like this: >>> >>> http://www.qbalsoftware.com/Site/Blog/Entries/2011/5/9_Makefiles_Generic_Flexible_and_Simple_Part-2.html I >>> went to you archive and read part 1, which is a big long makefile that's very explicit about what it does but is still "make, only more so". Reading the second now, I note the pull quote at the top: > "I just cannot see how that can be true. Being both generic, > flexible and being simple are usually at odds, No they're not. The more simple something is the more flexible and generic it is. A knife is flexible and generic, a cookie cutter won't spread butter on your bread or slice meat for sandwiches. A hammer is flexible and generic, a nail gun won't pair with a chisel or seal a paint can lid. Software is the same way, that's the whole POINT of the Unix philosophy. Simple commands speaking a common language (ascii text) connected by pipes. I'm sure I've written down my make rant before, but I'm not sure _where_. (Possibly in my blog, but it goes back years and "make" is a bad word to search for...) I know some of it's in the compiler BOF video from OLS: http://landley.net/talks But that's just one topic in an hour-long talk... Ok, Make stuff, lemme see... First, make has grown a lot of cruft ala automake, and pointless crap like libtool which should do NOTHING on Linux. The point of libtool is to make non-ELF systems behave passably like ELF systems. Linux has been ELF since 1995, meaning on Linux Libtool should be a NOP. Yet it REGULARLY breaks builds, especially in cross compile environments. (Note: attempting to do nothing and FAILING is a trademark move of the Free Software Foundation.) Alright, let's look at Make. The superficial objection is that Python gets bashed all the time for significant whitespace, but make which REQUIRES TABS (not spaces) gets grandfathered in. A less superficial objection mixing declarative and imperative code in the same file is mind-bending and evil. But the FUNDAMENTAL objection of make is that its initial reason for existing no longer applies. The point of makefiles is to work out what needs to be rebuilt, I.E. dependency tracking is its reason for existing. That's the whole point of these weird inter-related targets that aren't in any obvious order. That was the central design idea. This stopped working DECADES ago, which is why we have all these ".dep" files. We tell the compiler to work this out for us, we do NOT hand-encode them in the makefiles. More to the point, Moore's Law has advanced 40 years since then, and "make all" is the vast majority of all make invocations now. When you download a source package, you "configure; make; make install". And half the time when DEVELOPERS build stuff we do a "make clean" or "git clean -fdx && git checkout -f" or similar because I for one don't trust the makefile dependencies not to introduce weird subtle bugs. Basically any project that's big enough that make takes forever to bulid should be broken into directories, and each directory should have its own "make all" producing its .o or .a file or whatever it makes, and all this dependency tracking should GO AWAY. Rebuild with directory granularity at best. For small projects, just "make all". The OTHER point is that if you can feed all the files to gcc at once, the optimizer can do a better job. So make is fighting modern _compilers_, which I talked about in the above mentioned video... >>> This is how easy I think build systems can be. Of course, what >>> I'd like to see is 'Make' revisited for the multi-variant >>> problems we face >> >> My approach to make, autoconf, and automake is a quote from Aliens: >> "I say we take off and nuke the entire site from orbit: only way to >> be sure." > > I'm really thinking seriously about solving this for a minimal > system, like the ones you have created. I just cannot think of > spending another 10 years with the existing stuff. In my toybox project, my makefile was just a simple wrapper to give people the command line API they expected, and then it called shell scripts in the "make" subdirectory: http://landley.net/hg/toybox http://landley.net/code/toybox Alas, I stopped having time for that project years ago. Busybox is "good enough", has a ten year headstart and about 4 years of my own work in there, and is "good enough" that producing an incrementally better wheel won't get people to leave what they've got. *shrug* Also, my build system was a quick and dirty first attempt, and didn't do parallelism because I had a UP laptop when I started. (Although really I should be able to go "cc -o walrus source/*.c -j 4" and let the COMPILER work out how to parallelize it. Otherwise I'm ditching the -fwhole-tree optimizations I ranted about in that BOF video...) > I'll write something up and send it to you, see what you think Let me know when you do this, a quick glance at part 2 of your blog stuff was talking about writing improved makefiles, and to me this is like writing clearer COBOL programs. I don't want to mitigate the problem, I want to eliminate it. > I can make the current situation much simpler using currently > technologies, but I think some new technologies should be created to > take it one big giant step forward Eric Raymond really likes http://www.scons.org/ but being written in Python is likely to prevent it from being universally adopted _ever_. I'm all for high-level languages, but as you see from my system bootstrapping exercise (Aboriginal Linux) building a low-level system should not rely on them. Perl dependencies snuck into the kernel build (fairly recently, 2.6.25) and I removed them. I use uClibc rather than glibc primarily because glibc requires perl to build. (I'd at least offer glibc as an option otherwise, I keep meaning to go take another look at that...) Rob 1309042109.0 From rob at landley.net Sat Jun 25 22:49:51 2011 From: rob at landley.net (Rob Landley) Date: Sat, 25 Jun 2011 17:49:51 -0500 Subject: [Aboriginal] Fwd: Re: Aboriginal. Wow! and Thanks! Message-ID: <4E06660F.5050300@landley.net> -------- Original Message -------- Subject: Re: Aboriginal. Wow! and Thanks! Date: Sat, 18 Jun 2011 20:59:41 -0500 From: Rob Landley To: Paul Kramer On 06/15/2011 06:51 PM, Paul Kramer wrote: > > On Jun 14, 2011, at 6:18 PM, Rob Landley wrote: > > >> >> Make combines imperative and declarative code in the same file, and >> then people try to beat some semblance of order out of it with >> recursive make (the definitive takedown of which is >> http://aegis.sourceforge.net/auug97.pdf ) and then people write >> extensive systems like kbuild on top of it... > > I read this probably.. in the early 00's. Very well explained, I > learned some things about 'make' I did not know... Other fun things about the current ./configure ; make; make install: Configuration systems (make menuconfig) specify _what_ to build. Make is unaware that these suckers even _exist_, so its' incremental build thing has to be worked around even more. Makefiles got extended into a programming language because the original design of make did not do what people needed, but it's a REALLY BAD programming language designed around assumptions that no longer remotely match reality. 95% of what the ./configure stage does is useless. Things like the LP64 standard (see the 32/64 bit section of http://landley.net/code/toybox/design.html for a writeup on that) mean that a lot of things configure probes for ARE STANDARDIZED NOW, or can be determined at runtime from any of the 8 gazillion compiler built-in #defines you can see a dump of if you go: cc -dM -E - < /dev/null Things like __linux__ and __x86_64__ you do NOT need to probe for, nor do you need to specify on the command line when compiling natively. the fact the FSF can't wrap its head around this is because THEY SUCK AT WRITING SOFTWARE. (As evidenced by the fact that any time that political organization gets its hands on a software package, the package bloats to several times its original size. This is like a car manufacturer buying volkswagen's "bug" line and turning into a rolling living room with fins: using more metal to build a car is not an improvement, using more bytes to build a program is not an improvement. The FSF is to software what 1970's detroit was to cars.) > But if I do another system, I'll use his style of a top level > Makefile and each leaf node having a build.mk ... I'd leverage from > the approach I use to a node build.mk include file might look > like... especially for firmware teams, there stuff is easy to setup. If you use another system, please don't use "make". > In the end though... we need a new build tool still... or a build > modeling tool that spits out build system... i think the targets are > easy... but the variants of properties is where we need > non-declarative language to describe.... > > you know... there is a part of me that even wonders why even do the > build.mks, just one makefile... or why do the build.mks need to live > at the leaf node? they can have a namespace just by foo.mk... I do shell scripts. The shell scripts build the project just fine. The limit to this approach is I need more parallelism because 32-way SMP in laptops is inevitable. (Moore's Law is driven by photolithography die size shrinks, meaning manufacturing process improvements double your transistor density every 18 months or so. This naturally speeds up the chip because shorter trace lengths take less time for a signal to travel down, and because physically smaller components hold fewer electrons so it takes less signal to fill 'em up. However, chips need to be a certain minimum size to reliably connect up all the I/O wires you need, and you have to spread out your high-performance bits with less-used padding (like cache memory) or else cooling the chip becomes impossible when your die size shrinks to 1/4 the area but your power consumption only goes down 50%... So that means your transistor budget keeps going up. Intel's design team's job is to convert those increased transistor budgets into more performance. And thus we had the CISC/RISC fight of the 1980's and early 90's which let us cram multiple execution units in the same processor and have teh second (and even third) look over the shouder of the first and see if it could execute the next instruction in the same clock cycle. We had pipeline reordering, branch prediction and speculative execution requiring multiple register profiles with register renaming (with hyper-threading falling out of that eventually as it reached its logical conclusion). And of course L2 cache sizes expanding until we hit a couple MEGABYTES. But there came a point of diminishing returns (See Itanic and Pentium 4) where adding more transistors to the same chip just didn't work anymore. (Part of it was Intel being stupid, luckily the Israel design team found a back door through which they kicked the India design team's ass, and thus modern chips are descended from Pentium M and NOT from Itanic or P4.) What do you do when you can't add more transistors to one chip, but photolithography density improvements keeps doubling your transitor budgets every few years? (And you see a big fall-off in performance gains between 1 and 4 megs of L2 cache, because most programs' working sets just aren't that big.) You make multiple CPUs out of 'em, which lets you put even more L2 cache on the chip to soak up those transistor budgets... Of course most software isn't taking advantage of this either, but you can do one processor for the CPU and another for your application, and then one for your flash plugin, one for your web browser, one for the OS, gets you an excuse for 4-way... The FUN part is that while software catches up, you can add 3D accelerators to the die. Make those extra cores GPU instead of CPU, which lets you use your L2 cache as texture memory (and thus happily use 128 megs or more), that'll soak up transistor budgets for a while. And thus AMD bought ATI (doo dah, doo dah) to keep up with Intel which had 40% of the graphics market share (all the low-end) BEFORE it started integrating the GPU on die, and this left Via in the cold but they responded by tying up the game console market and then partnering with TI to becoming the default graphics chip of ARM... Fun fun fun. What was the original topic again? Oh yeah, make sucks but its replacement must do SMP. > Even after I look at what I wrote... on 2nd thought I think I'd still > do this Look at the busybox and linux attempts at building the entire tree in one go. (I can try googling for these if you can't find 'em easily, they were back around... 2006 maybe?) Oh, here's another random reason make breaks stuff: http://www.busybox.net/FAQ.html#touch_config Rob 1309042191.0 From rob at landley.net Sat Jun 25 22:54:45 2011 From: rob at landley.net (Rob Landley) Date: Sat, 25 Jun 2011 17:54:45 -0500 Subject: [Aboriginal] Fwd: Re: Fwd: Aboriginal. Wow! and Thanks! Message-ID: <4E066735.8020303@landley.net> -------- Original Message -------- Subject: Re: Fwd: Aboriginal. Wow! and Thanks! Date: Sat, 25 Jun 2011 11:38:07 -0500 From: Rob Landley To: Paul Kramer On 06/24/2011 11:28 PM, Paul Kramer wrote: > Rob -- beside solving the autotools, make problem... here are the > other things I'm interested in > > * packaging (RPM, APT... mediocre at best, I want to do something > that makes sense in 2010's) I'm sure I've ranted about this before, but a quick check of my blog isn't finding it. If you read that aboriginal project history file, you'll notice that I have a buildroot rant. Became an accidental distro. Then OpenEmbedded happened, when some buildroot developers decided to start over because they could do better, and they invented yet another packaging tool called "bitbake" which eventually forked off and became its own project, with openembedded being a bitbake repository. Literally, openembedded is a bunch of bitbake build description files, hundreds of them, for various packages. The "ipkg" format started like this. Portage started like this. Slackware has been using "tarballs with metadata" for decades. When you can name a half-dozen packaging tools off the top of your head, each with one or more associated repositories of the 43,000 packages Debian had in its repo last time I bothered to check (and that was a while ago, I'm sure it's more now)... It's not a simple problem. > * for every web application they we put up on the web (defect > tracking, wiki, build server... anything)... I want everything to be > accessible via the command line and the web gui. once it's available > thru the CLI, then folks can hook into their editors, script, > whatever they want Cool. How does security work into this? (Personally I just tunnel everything through ssh. Once upon a time I created an ssh-based vpn, some of which might still be up at http://dvpn.sf.net/old or something.) > * defect tracking system... instead of a bugzilla for example... I > want a tool kit that can be used to construct a defect tracking > system, and provide from reference implementations.... I plead incompetence here. I've never been good with bug tracking systems. It was one of the big sources of conflict in the busybox development thing, they had mantis and then the site got hacked through mantis and they switched to a bugzilla I've never used because the OSDL guys would admin it security-wise. Mark set up redmine for impactlinux and I spent a weekend filling it with info and then never looked at it again... > * continuous integration... as practiced and implemented by the > majority is wrong... again this should be a tool kit with reference > implementations Ok, "continuous integration" sounds great but I want to hurt people when it comes up. First you're doing git bisect up front which is great if your test suite means anything, but any _real_ project will hit more bugs in the field than during development, so you still have to cope and all this ivory tower BS about your process somehow helping matters is just silly. What you REALLY need are "release early, release often" as described in this MARVELOUS talk from an ex-debian maintainer: http://video.google.com/videoplay?docid=-5503858974016723264 Honestly, go watch it. THAT is what continuous integration is trying to achieve, but instead its used as an excuse NOT to have releases by projects like eglibc, which renders those projects of zero interest to me. > * wikis ... wrong... i want people be able to work in VIM or Emacs > when working not only with the wiki but any engineering web app Wikis came up at my 2008 OLS talk on "where documentation hides". They're a great place to archive a slush heap of ideas, but they don't address editing at all. Wikipedia has no index. It has no plot. There IS no coherent order to read it in, and there's a lot of duplication between articles, and this is inherent in it being a wiki. > * whatever web technology I use, i want to be as close to metal as > possible... the problem with the applications developers is that they > are framework crazy.... next you know... they go overboard like the > Java community and turn hello world into 5,000 lines of code... That's actually a flaw in the Java language: http://steve-yegge.blogspot.com/2007/12/codes-worst-enemy.html It's part of it being caught between fully static (C) and fully dynamic (python, ruby, lua, and other scripting languages). They combine static typing with dynamic memory management, and try to work around it with interfaces, and it fails into massive code generation and bloat. At least it's not C++. > sound > crazy... There is a popular maven build tool... and if we wrote a > hello world program in Java, and then ran maven... it would > download... what seems like hundreds of packages off the net to get > all the dependencies and plugins for the maven build tool and > whatever is need by hello world... it's insanity.... Yup. Environmental dependencies have a cost. I covered that in "The Art of Unix Programming". (In theory I am not a co-author, I just edited that book. In practice it was 9 chapters when I started arguing with Eric, and something like 20 when we stopped arguing.) Rob > Begin forwarded message: > >> From: Paul Kramer Date: June 24, 2011 9:17:04 PM >> PDT To: Rob Landley Subject: Re: Aboriginal. Wow! >> and Thanks! >> >> >> On Jun 19, 2011, at 1:28 PM, Rob Landley wrote: >>> >>> I'm a longstanding "jack of all trades, master of none". >>> Currently I'm watching a youtube video of Jeri Ellsworth >>> interviewing Bill Herd about Phase Locked loops. (I'm not a >>> hardware guy, so I haven't got the background to properly follow >>> this, but it's fascinating...) >> >> me too. people ask me what are your strengths, expecting me to say >> something technical and i tell them, problem solving SDLC issues >> and workflow issues, making stuff that should be easy, easy. >> >>>> >>>> >>>> * redis * mongodb (perfect for engineering disparate data) * >>>> sinatra (very thin web framework) * node.js (this will freak >>>> you out... server side javascript) --- seriously this is >>>> something to pay attention to... it's super liteweight evented >>>> i/o * but wait... coffee-script... a blend of python and ruby >>>> that produces javascript) * and of course ruby (and btw the >>>> way... the ruby folks are all into redis, mongodb, sinatra >>>> node.js) * ruby DSLs... because of the flexible syntax and >>>> language features its great for this... >>> >>> I've heard of some of this but never used any of it. >> >> you know... for folks that are coding down at the bare metal, they >> usually don't have much use for the web stuff... at least my >> friends that are firmware engineers. >> >> the reason I'm into this stuff, is that when setting up >> environments for teams there is going to be a bunch data generated >> from builds, tests, and all the other stuff going on related to >> priorities, decisions, how-to stuff... >> >> and some of this technology can help pull that data together and >> present in multiple formats. >> >> for the longest time, most of the engineers tools have been >> disparate command line utils. then stuff start moving to the web in >> the mid-90's. >> >> know the landscape is littered with fat monolithic java web apps... >> for static analysis, defect tracking, wikis, build and test >> automation >> >> it's all SUPER overkill >> >> what I've wanted to do for a long time is write the utils so we >> can get the data from the web UI or the CLI... >> >> until recently that was painful to implement, now it's getting >> much thinner... folks are finally starting to understand, not >> everything has to be a web app backed by mysql >>> >>>> although there is a bit of misuse... * github * resque (screw >>>> hudson and all those java heavyweight narrow minded continous >>>> integration bullshit... anyone that has a clue has a killer >>>> pre-integration setup) >>> >>> Continuous integration means never having to cut a release. I >>> have a whole rant about why that's a horrible idea. Releases are >>> GOOD things. >> >> Ahhh... my pet peeve too. Slightly different angle. When I get an >> environment setup and we've got a best-practices down... I setup so >> that every build is a release-candidate. Now it my not meet entry >> criteria, but mechanically it's a release-candidate. >> >> So alot about continuous integration is how people use it... For >> example when I setup it up for a compiler team... I said all these >> builds/tests we do during the day are just to help us understand >> where we are before the nitely runs... >> >> just indicators. i >> >> But... On the better teams I get to work with... the sweet spot is >> pre-integration. So I'll setup environments so that folks can farm >> off their builds to run locally on their machine or on a build >> farm. I leave it up to the developer to do as much or as little as >> they want, but I have the environment in place that will ensure >> goodness when they check in. >> >> I've been lucky in that I've worked on teams that have used >> distributed source code control tools all thru the 90's... Suns >> Teamware/Codemgr. Larry McVoy wrote the original in perl during the >> early Solaris development 1991/92 >> >>> >>>> * and finally RESTful APIs --- no folks have been doing this >>>> awhile... but really sloppy... >>> >>> Sounds like a bad marketing term. >> >> HAHAHA. >> >> basically it is this... the URL has semantic meaning, and the >> response you get back from the server has the business logic in >> additional URLs the client can use or discard... thats an >> oversimplification.... >> >> another kind-a-sort-a way to think about it... when unix/linux >> people work, they basically have the file system in their head, and >> the paths have a meaning and purpose to what is contained in the >> directory >> >> so a url like http://fubar.com/users/joeblow/edit means we'll be >> editing the joeblow record in the users collection... >> >> >>> >>>> there is an aspect called hypermedia that folks are not using >>>> and it's very powerfull... >>> >>> Also sounds like a bad marketing term. >> >> HAHAHA yeah... So all that means if ... just like the url above >> that the client (could be command line or some gui) provides, the >> server will return data in say... XML or JSON format, and along >> with that it will return other URLs the client can use to further >> operate on the users collections. So the client does not have to >> know the URLs... the server ... web service provides those to the >> client. This is goodness as it is loosely coupled... the server can >> change implementation, and the client is unaffected, because it is >> providing the api desitination >>> >>>> simply your web request not only provides you with your request >>>> data... it also provides you with what routes/actions to take >>>> next... this is a thing of beauty... why... create little web >>>> services without a UI... and then anyone can create CLI or >>>> javascript widgest to interact with the web services... >>> >>> Because every time you run a command in a command line it tries >>> to guess what you want to do next? >> >> Basically the logic is all predetermine and served up by the web >> service. >> >>> >>>> huge... imagine you are working on a project that build 6 >>>> variants of your codebase... you commit in git... and trigger >>>> than launches a build on 6 remote serves to crank out all the >>>> varaints..... think resque, RESTful API... huge potention.. >>>> with git we can share code easily... what is we could share >>>> everything easily? we need a service to do that... RESTful over >>>> http >>> >>> So it assumes you have a half-dozen servers lying idle and you'd >>> like to speculatively tie them up every time you breathe? >> >> I'm over simplifying... but lets say you have a cronjob that wakes >> up every so often, and if something has changed, it performs some >> action... >> >> these web servers are super thin.... you could launch one and it >> might observe your workspace and when you commit a change it >> clones your workspace and does an opt and debug build... or farms >> that off to another machine >> >> for an individual this may or may not be useful, bit in a team of >> folks it may... especially if the change was bug that had to be >> propogated into many lines of development, you could have web >> service that grab the bug fix and spin builds, runs test >>> >>> This is not an environment I normally work in. >> >> if you have time, it would be cool if you could explain your >> preferred work environment, tools, workflows and stuff that drive >> you nuts >> >>> >>> Rob >> > > 1309042485.0 From rob at landley.net Sat Jun 25 22:55:18 2011 From: rob at landley.net (Rob Landley) Date: Sat, 25 Jun 2011 17:55:18 -0500 Subject: [Aboriginal] i cannot believe how much you've done. In-Reply-To: <43A327F8-2049-4EEE-A76D-A18B35976724@me.com> References: <43A327F8-2049-4EEE-A76D-A18B35976724@me.com> Message-ID: <4E066756.4040004@landley.net> On 06/24/2011 11:48 PM, Paul Kramer wrote: > is was reading your presentation.html... well i'm drinking some vino > and just skimming over it and look forward to really diving in > > I was thinking about his step... what if the .o is left wherever it > was compiled at... I'm wondering what the reads across the network > would be during a link? Since we are copying and then linking... if > we don't copy, and just link... You mean SNAPSHOT_SYMLINK=1 from config at the top of aboriginal linux? You fault in the dentry info when you do that (filesystem metadata), but not the file contents. Those get paged in on demand as the build uses them. Meanwhile, build/packages can be a symlink to a remote filesystem (NFS even) without this introducing huge overhead/instability into the build, because the .o files go to the local filesystem (whatever the rest of "build" is on). There's an explanatory comment in front of each config entry. I really should link "config" from the website when I'm pulling out all the README files and turning them into web pages. It explains all the knobs and levers you can use to control the build. > ? When told to compile .c files to .o files ? calls the preprocessor > on each .c file (gcc -E) to resolve the includes ? Sends the > preprocessed .c file through the network to a server ? Server > compiles preprocessed .c code into .o object code ? Sends the .o file > back when done ? Copies .o file back through network to local > filesystem ? When told to produce an executable ? calls the local > compiler to link the .o files together That's how distcc works in its default mode, yes. I keep meaning to write my own distcc tool to have more control over the process, it won't always distribute builds in circumstances where it _can_... > ok... i'm just talking here.... but WindRiver and MontaVista > environments are nothing to write home about... speed is so valuable > today... Speed is valuable but so is engineering time, and Moore's Law only helps with one of those. A quarter-century ago on 80386 processors we needed every ounce of speed we could get out of the build. Today on quad xeons we need every ounce of speed we can get out of the build, except that our resources have increased by A DOZEN ORDERS OF MAGNITUDE. Look at the system Linux started on circa 1992. 16 mhz 386, 4 megs of ram, 40 megabyte hard drive that didn't even do DMA. The 386 was a single execution unit, multiple clocks per instruction, not even pipelined, and no L1 or L2 cache. Now we have processors clocked 100 times as fast, with tens of kilobytes of L1 and megabytes of L2 cache, pipelined with hardware prefetch and instruction reordering and register renaming and branch prediction/speculative execution and so on, three execution units per core, up to four cores per chip in _laptops_. A system with only 1 gigabyte of ram is _dinky_, and a terabyte of disk is $100. Screaming about this being NOT FAST ENOUGH says to me "hang on, somethign is wrong with your development process". Trust me, it's not the hardware. > I interviewed at XXX... there environment is brutal > and has always been brutal... so brutal == no speed. seems like there > is room for you to compete at some level against these companies I'm doing it because it's the right thing engineering-wise, not because I get paid to. I've had 3 or 4 jobs where I got to use Aboriginal at work (it was always my suggestion to do so, bringing in the toolkit I was familiar with). Nobody ever approached me to sponsor it, although sometimes they appreciated it enough to give me some paid hours to expand it to do Thing Du Jour that they found useful. > just providing a service to bootstrap and provide a build environment > could be interesting.. just rambling here... That's pretty much what it does. Bootstrap a native build environment for new hardware, either under an emulator or on the actual board. > here is what i've always thought... one way to compete is to out > manufacture the competitors, and in software what that means is get > their 50% faster than competitors... i've worked in enviornments > where a VP had to okay... $32.00 of disk space... and our offices was > right across the street from fry's... At Qualcomm I bought a USB to serial adapter on my lunch break. I just didn't TELL anybody. My current gig is at polycom, and one of the managers there has a company credit card which he's happy to use to order stuff from amazon and such (and even pays for overnight shipping). > i just finished a gig at XXX a few months back... IT would not approve or allocate more > than 8GB RAM and 200 GB disk on their servers... no shit... after 5 > months of dicking around... VP of Eng finally okayed 3 * $1K dells... > so I could spin up static analysis... on so many levels silicon > valley has really lost it's core technology talent and engineering > discipline.... most of the hiring is to all these social networking, > gaming companies.... As the presentation says: cheap commodity x86 hardware, plus QEMU to turn it into a native development environment for your board. If you spill coffee on it, replacement time is about half a day to buy new hardware, set it up, install xubuntu LTS on it, and copy your files back over. (I actually keep current xubuntu LTS on a USB key for easy installing. :) Rob 1309042518.0 From rob at landley.net Sat Jun 25 23:15:41 2011 From: rob at landley.net (Rob Landley) Date: Sat, 25 Jun 2011 18:15:41 -0500 Subject: [Aboriginal] Aboriginal. Wow! and Thanks! In-Reply-To: References: <3729B7D1-7F11-49CE-BDAB-ACF1578577CC@me.com> <4DF80871.5010506@landley.net> <7D34D8CD-1BE7-469E-B06A-63C37ECF92B9@me.com> <4DFD580D.2030800@landley.net> <4DFD7435.70909@landley.net> <4DFE5BF8.4010103@landley.net> Message-ID: <4E066C1D.4050602@landley.net> On 06/24/2011 11:17 PM, Paul Kramer wrote: >>> * redis * mongodb (perfect for engineering disparate data) * sinatra >>> (very thin web framework) * node.js (this will freak you out... >>> server side javascript) --- seriously this is something to pay >>> attention to... it's super liteweight evented i/o * but wait... >>> coffee-script... a blend of python and ruby that produces >>> javascript) * and of course ruby (and btw the way... the ruby folks >>> are all into redis, mongodb, sinatra node.js) * ruby DSLs... because >>> of the flexible syntax and language features its great for this... >> >> I've heard of some of this but never used any of it. > > you know... for folks that are coding down at the bare metal, they > usually don't have much use for the web stuff... at least my friends > that are firmware engineers. Last week I got gigabit ethernet working in a MAC-MAC configuration (with no PHY on either end), which involved going down to the lab and having the nice man with the oscilloscope hook the thing up and figure out that the Netra was talking gigabit but the switch was talking 100baseT, and being able to fix that. I tend to like having my hand held when it gets down to the hardware level, because if I screw up badly enough you can't restore from backup. (And when I'm funding my own experiments, even blowing out a $20 part often puts an end to that line of equiry.) > the reason I'm into this stuff, is that when setting up environments for teams > there is going to be a bunch data generated from builds, tests, and > all the other stuff going on related to priorities, decisions, how-to stuff... > > and some of this technology can help pull that data together and > present in multiple formats. > > for the longest time, most of the engineers tools have been disparate > command line utils. then stuff start moving to the web in the mid-90's. > > know the landscape is littered with fat monolithic java web apps... for > static analysis, defect tracking, wikis, build and test automation > > it's all SUPER overkill > > what I've wanted to do for a long time is write the utils so we can > get the data from the web UI or the CLI... > > until recently that was painful to implement, now it's getting much > thinner... folks are finally starting to understand, not everything > has to be a web app backed by mysql My first programming environment was commodore 64 basic, later compiled with "Blitz!". (Apparently the exclamation point was part of the name.) That had a simple line-number-based text editing API built into the ROM, with this strange line continuation thing (if you typed off the end of a line it inserted a new line, moving the screen contents below it down if necessary, and then _remembered_ that new line was part of the previous one. It would only do this once per line, and if you did it at the bottom of the screen and then hit backspace you'd trigger a bug that would print out an error message and lock up the keyboard.) My first C programming environment was Turbo C for DOS. (Later upgraded to Borland C++. Both copies a friend gave me. And a loaned "how to program in Turbo C" book by Herbert Schildt that I still have, the definition of loan there is a bit shaky. I was a teenager with no money, of course, went over to a friend's house and used his computers.) The sad part is I've never found an IDE on Linux that's as good as Turbo C for DOS was. Everybody keeps going "just use Emacs, if you learn to program in Lithp you can turn it into your perfect IDE with less than three years of full-time work!" I used microemacs on the amiga, I got it out of my system, I don't _like_ vi but it's _there_. When I introduce new people to this stuff I just give 'em mousepad (xfce's notepad equivalent) but that doesn't work in a terminal window or without using the mouse to copy and paste. I used qedit for years on DOS because it ahd the same key bindings as turbo C (all based on wordstar, apparently) but joe on the sun workstations at Rutgers was a buggy piece of crap (when it wasn't dumping core it was failing to prevent ctrl-k-ctrl-s from suspending all console output), so I forced myself to learn enough VI to get along. Doesn't mean I like it... >>> although there is a bit of misuse... * github * resque (screw hudson >>> and all those java heavyweight narrow minded continous integration >>> bullshit... anyone that has a clue has a killer pre-integration >>> setup) >> >> Continuous integration means never having to cut a release. I have a >> whole rant about why that's a horrible idea. Releases are GOOD things. > > Ahhh... my pet peeve too. Slightly different angle. When I get an environment > setup and we've got a best-practices down... I setup so that every build > is a release-candidate. Now it my not meet entry criteria, but mechanically > it's a release-candidate. Every build should compile and run so as not to screw up "git bisect", but calling halfway through a re-engineering of how somethign works a release candidate is disingenous. Unless you force yourself to check in a month's work as One Big Lump, which is bad for different reasons. > So alot about continuous integration is how people use it... For example > when I setup it up for a compiler team... I said all these builds/tests we do > during the day are just to help us understand where we are before the nitely runs... Checking in something you haven't tested is silly. Setting up a bureaucratic policy where we think running the test suite against every checkin actually MEANS something is not necessarily less silly. (Memo: the _interesting_ failures are the ones that force you to add something new to your test suite.) > just indicators. i > > But... On the better teams I get to work with... the sweet spot is pre-integration. > So I'll setup environments so that folks can farm off their builds to run locally > on their machine or on a build farm. I leave it up to the developer to do as > much or as little as they want, but I have the environment in place that will > ensure goodness when they check in. > > I've been lucky in that I've worked on teams that have used distributed > source code control tools all thru the 90's... Suns Teamware/Codemgr. > Larry McVoy wrote the original in perl during the early Solaris development 1991/92 Sun being used as a _good_ example of something. I need a moment... I have two todo items: 1) Make a cron job that builds all targets for every commit and runs the smoketest scripts on all of 'em. 2) Come up with an expanded test suite that tests every config knob and knows what to expect. I note that #2 is a lot more valuable than #1. >> >>> * and finally RESTful APIs --- no folks have been doing this >>> awhile... but really sloppy... >> >> Sounds like a bad marketing term. > > HAHAHA. > > basically it is this... the URL has semantic meaning, and the response you > get back from the server has the business logic in additional URLs the > client can use or discard... thats an oversimplification.... > > another kind-a-sort-a way to think about it... when unix/linux people > work, they basically have the file system in their head, and the paths have a meaning > and purpose to what is contained in the directory > > so a url like http://fubar.com/users/joeblow/edit means we'll be editing > the joeblow record in the users collection... > > >> >>> there is an aspect called hypermedia >>> that folks are not using and it's very powerfull... >> >> Also sounds like a bad marketing term. > > HAHAHA yeah... So all that means if ... just like the url above > that the client (could be command line or some gui) provides, > the server will return data in say... XML or JSON format, and > along with that it will return other URLs the client can use > to further operate on the users collections. So the client > does not have to know the URLs... the server ... web service > provides those to the client. This is goodness as it is loosely > coupled... the server can change implementation, and the client > is unaffected, because it is providing the api desitination >> >>> simply your web >>> request not only provides you with your request data... it also >>> provides you with what routes/actions to take next... this is a thing >>> of beauty... why... create little web services without a UI... and >>> then anyone can create CLI or javascript widgest to interact with the >>> web services... >> >> Because every time you run a command in a command line it tries to guess >> what you want to do next? > > Basically the logic is all predetermine and served up by the web service. > >> >>> huge... imagine you are working on a project that >>> build 6 variants of your codebase... you commit in git... and trigger >>> than launches a build on 6 remote serves to crank out all the >>> varaints..... think resque, RESTful API... huge potention.. with git >>> we can share code easily... what is we could share everything easily? >>> we need a service to do that... RESTful over http >> >> So it assumes you have a half-dozen servers lying idle and you'd like to >> speculatively tie them up every time you breathe? > > I'm over simplifying... but lets say you have a cronjob that wakes up > every so often, and if something has changed, it performs some action... I do. It pulls the busybox, linux, uClibc, and qemu repositories and tries each one against the current build scripts. (bits of it are in more/cronjob.sh). It breaks too often to be useful. > these web servers are super thin.... you could launch one and it might > observe your workspace and when you commit a change it clones > your workspace and does an opt and debug build... or farms that off > to another machine It runs on the machine upstairs, it's a quad processor machine with 8 gigs of ram and it takes half a day to complete the full set of builds on all architectures. (And I haven't added config knob tests yet.) I note that when I get going, I can do a half-dozen checkins an hour... > for an individual this may or may not be useful, bit in a team of > folks it may... especially if the change was bug that had to > be propogated into many lines of development, you could > have web service that grab the bug fix and spin builds, runs test I'm all for improved testing and optimization, but I've seen too many _attempts_ to get too excited about any of them. They have an uphill battle to prove their worth to me. >> This is not an environment I normally work in. > > if you have time, it would be cool if you could explain your preferred work > environment, tools, workflows and stuff that drive you nuts I wouldn't say "preferred", just where I stopped fighting it. I just try to get the work environment out of my way. I take a laptop everywhere, to coffee shops and restaurants and stuff. (If I try to work at home I get mobbed by cats.) I switched to laptops as my primary dev environment about 10 years ago, and never looked back. Currently using my shiny new Acer Aspire One D255E-1802, with the memory upgraded to 2 gigs. Internet through my phone (nexus one hooked to t-mobile, USB tethering). I have wireless at home but have never managed to get networkmangler to work with it. I can connect to the wireless if I kill networkmangler and do iwconfig and ifconfig by hand, but it's too much work unless I have a really big file to transfer. Last time I did it was to cut the 1.0.2 release.) I edit code in vim because it's there. Yes, I'm aware that to fix it you have to "ln -sf vimrc /etc/vim/vimrc.tiny" as part of your system setup. And I wrote a toy to take the darn periodic sync() out as an LD_PRELOAD. But any machine I set down at, it's there. Any distro I switch to: it's there. It sucks, but it sucks ubiquitously. I compile stuff from the command line. I tend to have lots of open terminal tabs, and multiple terminal windows, on multiple desktops. I check stuff into mercurial given a chance, because git's user interface is designed to piss off everybody who didn't implement it. Git has a horrible UI. Yes I have to cope with it on a regular basis, but DUDE. I used to use gnome, because I'd never tried KDE. Then Kubuntu 4.0 was so UTTERLY HORRIBLE that I went back to gnome for six months (it was about as bad, bloated overcomplciated crap with way too many layers and I spent all my time fighting the automation, plus the actual responsiveness was like ging from chrom to firefox: latency, lag, and bloat, oh my...). Then I found xfce/xubuntu. I like it. It's simple, minimal, and it stays out of my way. I'm not recommending it to anybody, I'm using the third most popular desktop on the third most popular operating system, I am a rounding error. i don't care. I'm using thunderbird for email. I hate it, the sucker is a single threaded program that blocks on network access (which it starts asynchronously all the time) and I can't TYPE for up to 30 seconds while it goes off and diddles its' files. I was previously using kmail, but as with Konqueror it was tied to KDE and went down with the ship. (Luckily Apple peeled KHTML engine out of konqueror and turned it into Webkit, which is behind safari and chrome. Konqueror itself is still bundled with KDE, and so is dead to me.) I am not fond of most tools. I do not generally recommend them. When I do get happy with one, future releases tend to screw it up. But you work with what you've got... Rob 1309043741.0 From rob at landley.net Sun Jun 26 00:08:32 2011 From: rob at landley.net (Rob Landley) Date: Sat, 25 Jun 2011 19:08:32 -0500 Subject: [Aboriginal] [PATCH] Move i586 to virtio In-Reply-To: References: <9cb1d126a6102d47f326.1308736004@granny> <4E01FF64.302@landley.net> Message-ID: <4E067880.30903@landley.net> On 06/23/2011 03:50 AM, Alessio Igor Bogani wrote: > Hi, > > 2011/6/22 Rob Landley : > [...] >> 2) Not all deployed versions of qemu/kvm support this yet. (Actually, >> last I checked you still had to build qemu from the -git tree to get >> this, _and_ you had to install some obscure ACL support package without >> which configure would silently disable it without explanation. If the >> i586 target is unconditionally switched to virtio without some kind of >> config knob, this effectively _disables_ it for people using qemu/kvm >> from ubuntu 10.04 LTS. They ship a new LTS in October which might >> support it, but it's not there yet...) > > I don't understand: The i586 modified version works on my 10.04.02 system > What are the problems are you talking about? At a guess, this is one of the things that changed between 10.04 and 10.04.02. >> 3) It doesn't work on anything _except_ an x86 target; there's no virtio >> for arm or mips that I've found. That means turning on autoconf is an >> optimization for x86, the one platform on which you can just use chroot >> if you really need performance. > > I need performance when I test kernels which differs from the one used > by my host. So I can't use chroot nor (unfortunately) LXC. Gotcha. I've applied your patch to my repo, but I'm going to semi-revert it after I get the base architecture vs derived architecture stuff fixed up a bit, and then re-introduce it as a new i586-virtio or similar. > AFAIK ia64, mips, powerpc, s390, sh, tile and x86 should support VIRTIO devices. Should, yes. Do, no. And where's arm in that list? > Obviously KVM kernel config options works on x86 only. > > [...] >> You switch on VIRTIO_CONSOLE but don't use it? > > Yeah because at the moment I don't know how make it works! Neither do I. I asked on the qemu list and nobody replied. I think it creates a FIFO on the host, you can't redirect it to stdin/stdout like you can with other consoles... (It's massively overcomplicated, whatever it is.) > [...] >> Lemme work on fixing the hw-target support this weekend (migrating it to >> a new less intrusive mechanism, actually) and then we can add this as a >> new i586-virtio target. Then if enough stuff grows virtio options we >> can collapse it back into the base in a future release. Sound reasonable? > > Yes definitely. > > I tried creating the i586-virtio target but binutils refused to build > so I made the changes in place. Sigh. There's a bit of subtle black magic there that I'm uncomfortable with. If you don't set CROSS_TARGET, it's set to ${ARCH}-unknown-linux meaning the name of the architecture is fed to the FSF crap which wants a three-stage-tuple. Don't get me started on the FSF's horrible architecture naming scheme. It's deeply stupid. Deeply, deeply stupid... Anyway, lots of things need to set this to some magic value to humor binutils and gcc: $ grep CROSS_TARGET sources/targets/*/settings targets/armv4tl/settings:CROSS_TARGET=armv4tl-unknown-linux-gnueabi targets/armv4tl/settings:GFS_CHOST="$CROSS_TARGET" targets/armv5l/settings:CROSS_TARGET=armv5l-unknown-linux-gnueabi targets/armv6l/settings:CROSS_TARGET="armv6l-unknown-linux-gnueabi" targets/powerpc-440fp/settings:CROSS_TARGET=powerpc-unknown-linux targets/sh4/settings:CROSS_TARGET=sh-superh-linux Rob 1309046912.0 From rob at landley.net Sun Jun 26 17:55:01 2011 From: rob at landley.net (Rob Landley) Date: Sun, 26 Jun 2011 12:55:01 -0500 Subject: [Aboriginal] Aboriginal. Wow! and Thanks! In-Reply-To: <555B6DF6-81C8-4EA4-B158-885F248E47AA@me.com> References: <3729B7D1-7F11-49CE-BDAB-ACF1578577CC@me.com> <4DF80871.5010506@landley.net> <7D34D8CD-1BE7-469E-B06A-63C37ECF92B9@me.com> <4DFD580D.2030800@landley.net> <4DFD7435.70909@landley.net> <4DFE5BF8.4010103@landley.net> <4E066C1D.4050602@landley.net> <555B6DF6-81C8-4EA4-B158-885F248E47AA@me.com> Message-ID: <4E077275.6070605@landley.net> On 06/25/2011 07:12 PM, Paul Kramer wrote: >> Every build should compile and run so as not to screw up "git >> bisect", but calling halfway through a re-engineering of how >> somethign works a release candidate is disingenous. Unless you >> force yourself to check in a month's work as One Big Lump, which is >> bad for different reasons. > > Rob -- what do you mean by build should compile and run as not to > screw up 'git bisect'... i've still not fully spun up on git I wrote up a thing about this once: http://landley.net/writing/git-bisect-howto.html Git (and mercurial) let you binary search between a known good commit and a known bad commit to find where the bug was introduced. (The same thing can be used to track down a fix if you need to backport it.) Basically, you go: git bisect bad BADCOMMIT git bisect good GOODCOMMIT It then checks out a revision, you test it, and then go either "git bisect good" or "git bisect bad" depending on the test results. Did that version have the bug or not? Eventually, instead of doing a checkout, it'll spit out a commit description. The problem comes in when you get to a commit that is broken for a reason OTHER than the bug you're looking for. If it doesn't compile, you can't test it. Bisecting to find a bug is easiest in an otherwise clean tree. During development when you merge lots of stuff, you can get into an area where dozens of commits in a row don't work for an unrelated reason. You can of course stop what you're doing (git bisect log > tempfile), track down the fix for that bug (start saying "bad" for commits that exhibit that bug and "good" for ones that don't and it should converge on the commit that fixed it), save that commit as a patch (git show > file.patch), back up to where you were bisecting (git bisect replay < tempfile), and then But it's a flaming pain and very unreliable. You wind up doign a lot of hand hacking. Or you just say "git bisect skip" meaning "I can't test this commit, gimme another one" but A) the git logic to find the next commit is deeply stupid and will give you an ADJACENT commit rather than trying to chop off 1/3 from a random end of the range and then hopefully the midpoint of that will be somewhere sane, but no that's not what they do, you loop for DOZENS OF COMMITS testing all the adjacent commits in the same broken range, which is a huge waste of time...) Bisecting is easy when it works, and really painful when it doesn't, and when it doesn't it's because you have overlapping bugs preventing you from testing whether or not a given commit has the one you're interested in. I've stacked FOUR BUGS DEEP in kernel commits sometimes because non-x86 platforms don't get tested the way x86 does. Did I mention that 2.6.39 shipped with mips not _booting_ in my config? The mips maintainer was very nice about tracking it down, but you hit stuff like "somebody broke the stat structure by swapping a 16 bit value for a 32 bit value and screwing up the offsets of all the fields after that, meaning ls -l doesn't work on 32 bit sparc and hasn't for 4 releases and NOBODY NOTICED. Why did nobody notice? Because they're all still using the last Debian sparc stable release, which has a kernel that's THREE YEARS OLD... That was fun. One of the points of aboriginal linux is to smoketest the basic package functionality across multiple architectures, because the people doing real hardware all wait for everybody ELSE to debug it before upgrading, and they cluster anywhere from 1 year to _5 years_ back depending on platform. At work we're using 2.6.37 on arm, which is pretty darn cutting edge for most production environment's I've seen. (Only 2 releases back! Yes, our USB3 bug was fixed by trying a more current kernel, and next week I gotta find and backport the fix, or something.) >>> I've been lucky in that I've worked on teams that have used >>> distributed source code control tools all thru the 90's... Suns >>> Teamware/Codemgr. Larry McVoy wrote the original in perl during >>> the early Solaris development 1991/92 >> >> Sun being used as a _good_ example of something. > > > LOL... I worked on 2 outstanging releases at Sun... 1990 Sparcstation > 2 on Sun OS 4.1.3 and 1995 OpenGL team... the rest were nothing to > write home about... I graduated from college in 1995. That was the last time I physically used a Sun workstation. The server in the back room was Solaris, the workstations in the lab were SunOS, and they weren't _entirely_ compatible... > As with any large company, there was some great talent, and some > not-so-great. I basically think the beginning of the end was in > 1989... when the got rid of all field engineers and out-sourced, and > went from being a pretty good integrator of a number of vendors > technology and adding their stuff in, to being Sun-on-Sun ... during > the 1991-1993 years... Lots of folks don't realized that if Cray does > not go bankrupt sell the SparcServer 64 processors to SGI, and SGI > does not sell to Sun... Sun does not have a million dollar machine > to sell to a bunch of people all thru the late 1990's.... I wrote half a document on my understanding of "the sun civil war" once. Being forced out of the workstation market into a server-only niche cost sun its' identity and old sun went into huge denial about what it was. (Without workstations, what dothe developers use? The whole solaris-x86 debacle: what do you THINK your developer workstations are running, a $30k production machine or a $2k desktop? You tried to cancel the product line 3 times. Brilliant.) Along the way java happened, and that tail wagged the dog for quite a while, with Java development sheltering the growth of some VERY different software development philosophies with goals that conflcited directly with the old sun. The "We give Java away for free, open source is great" crowd (openoffice, opensolaris, eventually gpl java itself), the "Java runs on desktops, we need a position on the desktop" crowd (buy staroffice, produce looking glass, and a HUGE amount of nostalgia from their history as a workstation provider), the "most Java deployments are windows, we must kiss the black widow" crowd (brought in $2 billion payment from Microsoft shilling for SCO, that gave them a lot of political power)... And of course when Bill Joy went nuts and they swapped him out for a spare co-founder (Andy Bechtolsheim or whatever his name was), and HE brought in Opterons, the monolithic "Solaris is sparc, sparc is solaris" block cracked right down the middle, and EVERYTHING became a poltical football. Looking glass is Linux based? No, switch it to Solaris. Open source solaris (waaaaaah!) under a license nobody else will ever use so it can't share code: eh, GPLv3 counts there. GPLv2 Java at the copyright leve, but sue over patents! Open source openoffice but require copyright assignment so nobody outside of Sun contributes any code to it! Honestly, Sun was doomed when it lost the high end workstation market (to Windows NT of all things, but really to commodity PC hardware). The Pentium showed x86 wasn't going to be taken down by CISC/RISC considerations, Sparc was dead and just didn't know it at that point. And Linux was _obviously_ going to eat proprietary Unix circa 1993 (just a matter of time). Sparc and solaris stopped being _interesting_ in the early 90's. Yeah, the dot-com boom fed enough sugar and amphetamines into their system to keep them upright despite several missing internal organs, and they COULD have invented a whole new business given that time, but their old one was doomed. (They tried inventing a new business with java, but as with so many dot-com things costs going to zero, revenues going near zero is great if your company is a team of six people working out of a strip-mall, but if you have an existing 10k employees to support you're screwed.) Sorry, tangent. :) > I think sun made real good hardware/software systems... Real good 1992 hardware is junk by 1995. Cheap plastic white-box PCs running Linux ate their use cases, doesn't matter whether you're talking sparc, alpha, itanic... Being a high-end niche product in the face of Moore's Law is not a stable position. (Apple is high-end due to industrial design, not due to performance. Their price premium comes from the hardware and software being _pretty_ and fun to use, not due to outperforming anything on any empirical metric.) > but when > mcnealy and crew thought they could become a pure software play... it > was one failure after another. they lost focus. in the 80's... You've read http://www.blinkenlights.com/classiccmp/javaorigin.html right? Sun was doomed by cheap commodity PC hardware. When the pentium showed CISC can translate to RISC internally, the _smart_ risc players went down to the embedded space to compete on power consumption to performance ratio rather than price to performance or absolute best performance. They got out of x86's _way_, and let it continue upmarket so they could attack its soft underbelly. If AMD hadn't forced Intel to do the celeron, and if Clayton Christensen hadn't explained the situation to Andy Grove and gotten him to SMACK SOME SENSE INTO HIS ENTIRE COMPANY, as documented here: http://www.forbes.com/forbes/1999/0125/6302088a.html Then Intel would have gone off a cliff and would probably be in Sun's shoes right now, instead of repeatedly ratcheting downmarket (Pentium 4 was as dead end as Itanic, but Pentium M led to Core 2 duo. They bought strongarm and then sold it again to marvell because it wasn't x86, but they're doing Atom instead which is what my current netbook has). For a company like Intel to jump downmarket is like salmon swimming upstream in big leaps, but Christensen drove home the necessity of doing so RIGHT before it was too late for them to do it. Sun gleefully ran upmarket and off a cliff, classive failure of sustaining technology against disruptive atack. The lateral thrashing was a dying thing nailed to a wall trying to find an escape, it just took them a LONG TIME to die. > they > were at that time the fastest company at going from $0 to 1 > billion... and completely lost focus after that. Get a copy of The Innovator's Dilemma by Clayton Christensen. Marvelous book. (New York Times #1 bestseller in 1997.) > But ... teamware/codemgr was much much better than using rcs, sccs, > cvs, perforce (although perforce is reasonable to chip development > teams) Never heard of teamware/codemgr, but the above comparisons are called "damning with faint praise". Linux kernel development kept an archive of tarballs with patch files between 'em rather than deal with any of that, because doing it by hand with tarballs was an honestly superior source control system to any of that crap. > we actually had distributed branches on hp-ux in 1988... but we'd > still all share the tree old-school. when i arrived at Sun in 1990, > they had had smerge.... clone and merge sccs trunks, but we still > shared the tree old-school... it was not until Larry wrote smosh... > wrapped it in perl scripts and called it NSE-Lite and then they > re-wrote in C and shipped it as a product... teamware/codemgr... Sun > tried to create a clearcase like cm system... called NSE... it failed > miserably > >> >> I need a moment... >> >> I have two todo items: >> >> 1) Make a cron job that builds all targets for every commit and >> runs the smoketest scripts on all of 'em. > > yep. i often question myself, and the engineering community on how > when the web came along we as a community lost our sense of > simplicity... it's like tools folks see every problems as a web > application... The people with a sense of simplicity wound up doing embedded development. "Gimme a small system and I'll make it sing. My commodore 64 had 38911 basic bytes free and we made it work. You keep wanting to write a novel instead of a short story. I grew up doing haiku, and this calls for a drabble: luxury." > basically... my motto is focus on the individual contributor and how > to scale... but our industry does not go at it like that... they go > at it from the project and release level... You probably want to read those "three waves" articles I keep pointing you at. Everything I have to say on the topic of individual vs group contributors is prefixed on the material in those... Rob 1309110901.0 From rob at landley.net Sun Jun 26 22:18:13 2011 From: rob at landley.net (Rob Landley) Date: Sun, 26 Jun 2011 17:18:13 -0500 Subject: [Aboriginal] Aboriginal. Wow! and Thanks! In-Reply-To: References: <3729B7D1-7F11-49CE-BDAB-ACF1578577CC@me.com> <4DF80871.5010506@landley.net> <7D34D8CD-1BE7-469E-B06A-63C37ECF92B9@me.com> <4DFD580D.2030800@landley.net> <4DFD7435.70909@landley.net> <4DFE5BF8.4010103@landley.net> <4E066C1D.4050602@landley.net> Message-ID: <4E07B025.3080600@landley.net> On 06/25/2011 07:43 PM, Paul Kramer wrote: > > On Jun 25, 2011, at 4:15 PM, Rob Landley wrote: > >> >> I used to use gnome, because I'd never tried KDE. Then Kubuntu 4.0 >> was so UTTERLY HORRIBLE that I went back to gnome for six months >> (it was about as bad, bloated overcomplciated crap with way too >> many layers and I spent all my time fighting the automation, plus >> the actual responsiveness was like ging from chrom to firefox: >> latency, lag, and bloat, oh my...). >> >> Then I found xfce/xubuntu. I like it. It's simple, minimal, and >> it stays out of my way. I'm not recommending it to anybody, I'm >> using the third most popular desktop on the third most popular >> operating system, I am a rounding error. i don't care. >> >> I'm using thunderbird for email. I hate it, the sucker is a >> single threaded program that blocks on network access (which it >> starts asynchronously all the time) and I can't TYPE for up to 30 >> seconds while it goes off and diddles its' files. I was previously >> using kmail, but as with Konqueror it was tied to KDE and went down >> with the ship. (Luckily Apple peeled KHTML engine out of konqueror >> and turned it into Webkit, which is behind safari and chrome. >> Konqueror itself is still bundled with KDE, and so is dead to me.) > > from 1984-1998 i only worked on systems running BSD 4.2, HP-UX, > SunOS, Solaris, Zenix.... and bought my first pc a dell inspiron 9000 > laptop... thru redhat linux on it... and dual booted into window 98 > running cygwin... My first encounter with Linux was when the SLS floppies came across fidonet in 1993. A friend of mine installed it on one of his machines, and we went "Why would anyone want to clone a sun workstation? This can't run dos programs, so no migration path. It can't handle a diamond stealth multimedia (video) card in higher than MCGA mode, so X11 doesn't work. The developers think that threading is a bad idea and SMP is not a priority. Right, let's try OS/2..." I kept track of Linux because I'd used free downloads for years (connecting to fidonet involved hooking up binkleyterm, the fossil drivers, and zmodem, all free downloads) and I'd done open soruce development on WWIV (I learned C in the first place so I could apply "wwiv mod" files like http://landley.net/history/wwiv/CALVIN01.422 and http://landley.net/history/wwiv/EMTITLE.422 to the WWIV bulletin board source code you got if you registered the shareware: yes, we did open source without the "patch" program, and we liked it!) So I know Linux wasn't going AWAY, but also that it wasn't of any obvious use to me. So I did OS/2 for about five years, went to work for IBM straight out of college doing OS/2 for PowerPC and then OS/2 4.0. I believe I described the reason OS/2 died here: http://www.catb.org/~esr/writings/world-domination/world-domination-201.html And when I did (1998) I switched to Linux. It was still around, had made progress, and was the one I wanted to get behind and push. > ... so between 1998-2005 had dual boot laptop windows/redhat ... i > did much around with some of the other distros, but stuck with redhat > as the local companies were mostly using redhat in the 00's... bsd in > the late 90's Red Hat was the distro that was seriously trying to grow the open source developer pool. They seriously made a push for the desktop (which Debian wasn't doing, they were tech-snobs spending their time infighting), but weren't trying to seal the guts of the system _away_ from people (as Xandros and such later did). BSD was useless: it was part of the proprietary Unix tradition that had fragmented itself into irrelevance. Linux was modular and had interchangeable parts available from multiple sources, but "distributions" were like white box PCs. Somebody assembled a bunch of parts into something that worked out of the box, but Dell Compaq Gateway and so on were a coat of paint over basically the same thing. BSD built the whole distro from one big tree, and if you wanted to change a userspace package you wound up arguing with the kernel guys, you were either in or out, you couldn't find a niche and ignore the rest. Also, the licensing of BSD mean every time they got momentum a propreitary company harvested their talent pool to work on a proprietary fork. Sun got Bill Joy in 1982, BSDI got Bill Jolitz around 1989, Apple got Jordan Hubbard in 1997... Each of them hired away from the open project to get paid to work on the exact same code they'd been working on for fun, with the promise they can open source it "someday". and of course they can still keep working on the open version in there spare time, and then worked 90 hours a week until "their" version and the open version were so far apart that upstream no longer appealed... Linux wasn't vulnerable to this. Red Hat never stopped Alan Cox from submitting code upstream because the license said they _couldn't_ do a proprietary fork. Not even when they had their IPO, not even the Android-style stupidity where "we develop it closed then give you a Big Useless Hairball drop of the past 6 months of work" (although Red Hat's started doing that now, but they stopped being relevant after they figured out what was keeping Sun in business and how to walk away with Sun's market share). You know how large "enterprise" purchasing contracts cap allowed profit at a percentage of the cost of materials? Vendors thus spec needlessly expensive parts because 10% of a $5000 Solaris seat is $500 profit and 10% of a $29.95 Red Hat 7 retail box is $3. Red Hat figured this out and tried offering a $5000 version to Enterprise customers, and everybody switched over to the technically superior product. This meant that Sun starved to death, but it also meant that Red Hat got dragged out of the desktop market because its $100 million/year business vs a $10 million/year business sucked all the engineering time away. They renamed Red Hat Enterprise Rawhide to "Fedora" and tried to convince the open source community to do it for them, which started out like this: http://lwn.net/Articles/83360/ And ended the farce here: http://lwn.net/Articles/178518/ This left Knoppix holding the Linux on the desktop bag until Ubuntu came about. (Debian got so bad that its developers left for Gentoo, which then developed all of Debian's politics and almost died. "Gee, before we got here this place was nice and civilized, now it's the same pissing contest we left behind, why is that?" Sigh. Slackware's maintainer had some kind of chronic illness, SuSE went bankrupt in the dot-com bust and then got bought by the Brown Thumb of Novell which had previously killed proprietary Unix, Desqview, DR-DOS, wordperfect... it was where software went to die. Yeah, I expected that to collapse _before_ they kissed the black widow.) Anyway... > 2005 i bought a macpro laptop... OS X i wanted a simple user > interface to use as a non-technical system, and also have unix > environment... and this is still my setup with vmware and ubuntu and > redhat... My wife has an apple box. It'll continue to be shiny until about 5 years after Steve Jobs dies (apparently his pancreatic/liver cancer has relapsed). We've seen apple without Steve Jobs. It wasn't pretty. > i always ssh into the vmware instance... so In affect I > never deal with any of the linux GUI stuff... i went and spoiled > myself and bought a 30" cinema display... for combined i have 45" of > display to play with. Xubuntu's not so bad. Binary only flash plugin works, VLC plays multimedia, chrome's a decent browser. I need a new email client. Rob 1309126693.0 From rob at landley.net Sun Jun 26 22:25:00 2011 From: rob at landley.net (Rob Landley) Date: Sun, 26 Jun 2011 17:25:00 -0500 Subject: [Aboriginal] i cannot believe how much you've done. In-Reply-To: <30F4EF6B-BCE4-400F-9C5F-5F7CAB9B2CCD@me.com> References: <43A327F8-2049-4EEE-A76D-A18B35976724@me.com> <4E066756.4040004@landley.net> <30F4EF6B-BCE4-400F-9C5F-5F7CAB9B2CCD@me.com> Message-ID: <4E07B1BC.70501@landley.net> On 06/25/2011 06:29 PM, Paul Kramer wrote: > > On Jun 25, 2011, at 3:55 PM, Rob Landley wrote: > >> >> >>> ok... i'm just talking here.... but WindRiver and MontaVista >>> environments are nothing to write home about... speed is so valuable >>> today... >> >> Speed is valuable but so is engineering time, and Moore's Law only helps >> with one of those. > > Yeah I should be more clear. Sadly... I see little effort on helping the individual > contributor be more productive... and I think that is the sweet spot. Generally > speaking: an engineering organization... should do everything possible to > empower the developer and put the money on their desk, not in a server room I do all my development on a laptop, currently a ~$300 netbook I'm pretty happy with. I'd be a bit more productive with a fire breathing 4x laptop with 8 gigs of ram, but I carry it everywhere with me and weight to keyboard comfort ratio is an important consideration. Plus: 4 hours of batter life under heavy use, vs maybe an hour if you tax the CPU on those "desktop replacement" laptops. I have a machine I can ssh to if I need cpu-intensive stuff. It was about $800 when I bought it last year. > For example... back in 95, when I was on Sun's OpenGL team.. we had a lab > with a ton of systems... sitting idle. Why? most teams dedicate specific machines > to specific projects/releases' and tasks... So I'd see a bunch of developers queuing > up to use a few machines, while 90% of them sat idle... And thus people are doing "cloud" stuff, and linux containers, and... > So... what I did is said screw the lab, and we put 2 to 3 machines in each developers > office. sparc, x86, ppc and put just enough machines the in lap to crank builds > and tests At home I have an arm system, a coldfire system, a blackfin, and a mips box. But I don't use 'em: I have qemu and it emulates everything I need. I need the hardware for driver development, benchmarking, or occasionally to test qemu's behavior vs what real hardware does. > The way I got more use of the lab machines... I had each machine disc reserve > different disk partitions for different builds (operating systems), and test.. and > could recover or load from jumpstart... > > In the last few years... companies are giving engineers $700.00 laptops... this is bullshit. > Ever since I've left Sun in 1998... I bring my own machine to work... You can get a lot for $700 if you do it right. But yeah, given how much you pay the people, how much you pay for the office... Maybe they're reluctant to put money into assets that depreciate on an expoential curve? Dunno... > Conceptually that is what I'm into. > >> >> A quarter-century ago on 80386 processors we needed every ounce of speed >> we could get out of the build. Today on quad xeons we need every ounce >> of speed we can get out of the build, except that our resources have >> increased by A DOZEN ORDERS OF MAGNITUDE. >> >> Look at the system Linux started on circa 1992. 16 mhz 386, 4 megs of >> ram, 40 megabyte hard drive that didn't even do DMA. The 386 was a >> single execution unit, multiple clocks per instruction, not even >> pipelined, and no L1 or L2 cache. Now we have processors clocked 100 >> times as fast, with tens of kilobytes of L1 and megabytes of L2 cache, >> pipelined with hardware prefetch and instruction reordering and register >> renaming and branch prediction/speculative execution and so on, three >> execution units per core, up to four cores per chip in _laptops_. A >> system with only 1 gigabyte of ram is _dinky_, and a terabyte of disk is >> $100. >> >> Screaming about this being NOT FAST ENOUGH says to me "hang on, >> somethign is wrong with your development process". Trust me, it's not >> the hardware. > > YES! 100% with you!!! > > Since 2002, every gig I've had... the VP or Director would tell me one > thing in the interview, and then when I started... automate this, automate that... > and I'd say, hold on here... you are asking me to automate bad process, bad > infrastructure... you want me to bring in new engineering tools without fixing > the underlying issues... we are just putting nice paint on a pig They always want to do that. Open source is very good at "fixing stuff that already works". For-profit corporations consider cleanup a waste of money, but the open source guys know that just because you made a prototype do a trick DOES NOT MEAN YOU ARE DONE. >>> I interviewed at XXX... there environment is brutal >>> and has always been brutal... so brutal == no speed. seems like there >>> is room for you to compete at some level against these companies >> >> I'm doing it because it's the right thing engineering-wise, not because >> I get paid to. I've had 3 or 4 jobs where I got to use Aboriginal at >> work (it was always my suggestion to do so, bringing in the toolkit I >> was familiar with). Nobody ever approached me to sponsor it, although >> sometimes they appreciated it enough to give me some paid hours to >> expand it to do Thing Du Jour that they found useful. > > I am of the same mindset. I believe many of the issues that exist in > engineering can be eliminated, many can be simplified. Technology matures. Cars used to be hand cranked with a clutch pedal, now they're not. Once upon a time you needed an operator to patch your telephone call though. Then you dialed a number yourself. Now you speed dial. Computers have gone through 4 "Model T" reinventions: mainframe, minicomputer, microcomputer, smart phone. The current laptops descend from the Mits Altair in 1975, and current smart phones descend from either the iPhone or the Apple Newton depending on how you want to look at it. I wanna go to the Wikipedia[citation needed] page on the newton, find a random word in there, and add [egg freckles] after it, in square brackets, just like that. But I'm not a fan of Wikipedia[citation needed], and don't edit it. (You will never find a more wretched hive of politics, bikeshedding, and delete wars about "notability" due to senior editor biases.) > Just what little time I've had to spend on your site I've learned > some things, and I'm really looking forward to learning more. > > I'd like to make a contribution to the engineering community such that > our day jobs can be a much more productive environment. I have a strange view of all this stuff, I mentioned the 7 articles I wrote on the three waves linked from landley.net/writing What counts as a "productive environment" is a bit more complicated than it appears. :) >> At Qualcomm I bought a USB to serial adapter on my lunch break. I just >> didn't TELL anybody. >> >> My current gig is at polycom, and one of the managers there has a >> company credit card which he's happy to use to order stuff from amazon >> and such (and even pays for overnight shipping). > > I saw a post for a build engineer in Austin at Polycom... If I were younger, > I'd move out there... but I have folks I've known for 30+ years living > in Northern Calif... hard to leave I've been in austin, on and off, for 15. They're about a half-hour commute each way down a reasonably major road, not too bad. Enjoying it so far... :) Rob 1309127100.0 From rob at landley.net Mon Jun 27 12:56:04 2011 From: rob at landley.net (Rob Landley) Date: Mon, 27 Jun 2011 07:56:04 -0500 Subject: [Aboriginal] [PATCH] Move i586 to virtio In-Reply-To: <9cb1d126a6102d47f326.1308736004@granny> References: <9cb1d126a6102d47f326.1308736004@granny> Message-ID: <4E087DE4.9070008@landley.net> On 06/22/2011 04:46 AM, Alessio Igor Bogani wrote: > # HG changeset patch > # User Alessio Igor Bogani > # Date 1308735970 -7200 > # Node ID 9cb1d126a6102d47f3266a148247c97758a01bfd > # Parent 67accb5d252602b3b2ad2d0b8aa6fd9a62c74030 > Move i586 to virtio I moved this to i686-kvm target. Could you test if it works for you? (It doesn't work for me, but my netbook hasn't got the VT extensions kvm needs. Quite possible I screwed something up in all the shuffling...) Rob 1309179364.0 From alessioigorbogani at gmail.com Mon Jun 27 15:17:11 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Mon, 27 Jun 2011 17:17:11 +0200 Subject: [Aboriginal] [PATCH] Move i586 to virtio In-Reply-To: <4E087DE4.9070008@landley.net> References: <9cb1d126a6102d47f326.1308736004@granny> <4E087DE4.9070008@landley.net> Message-ID: Rob, 2011/6/27 Rob Landley : > On 06/22/2011 04:46 AM, Alessio Igor Bogani wrote: >> # HG changeset patch >> # User Alessio Igor Bogani >> # Date 1308735970 -7200 >> # Node ID 9cb1d126a6102d47f3266a148247c97758a01bfd >> # Parent ?67accb5d252602b3b2ad2d0b8aa6fd9a62c74030 >> Move i586 to virtio > > I moved this to i686-kvm target. ?Could you test if it works for you? Unfortunately it doesn't work on my system: [...] Root device is (9, 0) Setup is 12940 bytes (padded to 13312 bytes). System is 1378 kB CRC e329e245 Kernel: arch/x86/boot/bzImage is ready (#1) Exiting due to errors (i686-kvm linux-kernel linux) [...] The ls build/ reports: linux-kernel-i686-kvm root-image-i686-kvm.tar.bz2 simple-cross-compiler-i686-kvm.tar.bz2 simple-root-filesystem-i686-kvm.tar.bz2 packages root-image-i686.tar.bz2 simple-cross-compiler-i686.tar.bz2 simple-root-filesystem-i686.tar.bz2 root-image-i686 simple-cross-compiler-i686 simple-root-filesystem-i686 temp-i686-kvm root-image-i686-kvm simple-cross-compiler-i686-kvm simple-root-filesystem-i686-kvm Strange thing seem happens building kernel (please note that "linux-kernel-i686-kvm" is a directory which I suspect should be named "linux-kernel-i686" instead). In any case I would called that i686-virt or i686-virtio instead of i686-kvm. The most important part is the use of virtio devices where the kvm is only a secondary optimization handled by the system so I would used qemu (instead of kvm) in run_emulator() . Ciao, Alessio 1309187831.0 From rob at landley.net Tue Jun 28 10:39:03 2011 From: rob at landley.net (Rob Landley) Date: Tue, 28 Jun 2011 05:39:03 -0500 Subject: [Aboriginal] [PATCH] Move i586 to virtio In-Reply-To: References: <9cb1d126a6102d47f326.1308736004@granny> <4E087DE4.9070008@landley.net> Message-ID: <4E09AF47.2070705@landley.net> Alas, the work week's started again so my free time is in much smaller chunks... On 06/27/2011 10:17 AM, Alessio Igor Bogani wrote: > Rob, > > 2011/6/27 Rob Landley : >> On 06/22/2011 04:46 AM, Alessio Igor Bogani wrote: >>> # HG changeset patch >>> # User Alessio Igor Bogani >>> # Date 1308735970 -7200 >>> # Node ID 9cb1d126a6102d47f3266a148247c97758a01bfd >>> # Parent 67accb5d252602b3b2ad2d0b8aa6fd9a62c74030 >>> Move i586 to virtio >> >> I moved this to i686-kvm target. Could you test if it works for you? > > Unfortunately it doesn't work on my system: > [...] > Root device is (9, 0) > Setup is 12940 bytes (padded to 13312 bytes). > System is 1378 kB > CRC e329e245 > Kernel: arch/x86/boot/bzImage is ready (#1) > > Exiting due to errors (i686-kvm linux-kernel linux) > [...] The kernel built successfully and then exited due to errors? That's odd... > The ls build/ reports: > linux-kernel-i686-kvm > root-image-i686-kvm.tar.bz2 > simple-cross-compiler-i686-kvm.tar.bz2 > simple-root-filesystem-i686-kvm.tar.bz2 All of these should be symlinks to the base architecture (the equivalent -i686 versions). A base architecure is the target we have the same compiler as, meaning the arch tuple, compiler flags, and uClibc config are all the same. In that case there's no point in rebuilding that stuff, we should just use the existing version. You'll want to do an ls -l on build to see what you actually have. Ah, I didn't test the dependencies if the base isn't built yet. In that case it should build the base version and create a symlink to it. Hmmm... yeah, that could break stuff. I'll have to try that when I get a moment... > packages > root-image-i686.tar.bz2 > simple-cross-compiler-i686.tar.bz2 > simple-root-filesystem-i686.tar.bz2 > root-image-i686 simple-cross-compiler-i686 > simple-root-filesystem-i686 temp-i686-kvm > root-image-i686-kvm simple-cross-compiler-i686-kvm > simple-root-filesystem-i686-kvm > > Strange thing seem happens building kernel (please note that > "linux-kernel-i686-kvm" is a directory which I suspect should be named > "linux-kernel-i686" instead). There is a linux-kernel-i686 already. That's the base architecture, and its kernel/qemu config emulates actual hardware (IDE drive and such). In the real world there are often several different boards that use the same compiler and root filesystem but need different kernel .configs and bootloaders and such. I've had some support for that for a while through the "hardware target" (hw- prefix) stuff, with the Tin Can Tools Hammer board being an example (derived from the armv4tl base architecture). But this support was always a hack, and I was never happy with the design, and it bit-rotted for lack of testing. I'm trying to fix it now. It seems to me that i686 with virtio support is the same sort of variant of the i686 base architecture: the compiler and root filesystem are the same as a "real hardware" version, but the kernel config and the environment you run it in differ. So I'm trying to get it to work again, but this time the distinguishing factor isn't a magic prefix on the name, but for base architectures the target is a file, and for variants of those architectures the target is a directory. (That way buildall.sh knows to build all the base architectures and can skip the variants, but build.sh can build whichever one you tell it to and lists all of them.) Part of the reason for doing this is variants need their own kernel .config, and you have an existing .config and just want to use that without messing with miniconfig or baseconfig. So this gives a directory to drop that in. (I'll probably change the name it looks for to "linux-config" or something similarly obvious.) There are still rough edges to this design. I like most target configs being a single file now that it's cleaned up enough for that to work, but supplying an external kernel .config breaks that. I had an existing hw- target distinction which was always a sore spot, using the file/directory distinction to indicate that makes more sense. It's still a bit "black magic" though, I need to at least document it a lot better once I get the kinks worked out. As always, if you have design suggestions I'm happy to hear 'em. Making this stuff simple is _hard_, Making it do 8000 conflicting things is easy, making the result SIMPLE rather than a giant pile of special cases that collapses under its' own weight: that's hard. > In any case I would called that i686-virt or i686-virtio instead of > i686-kvm. The most important part is the use of virtio devices where > the kvm is only a secondary optimization handled by the system so I > would used qemu (instead of kvm) in run_emulator() . *shrug* I can call it i686-virtio if you prefer, sure. But more importantly, I need to figure out why it didn't build for you, and fix it... > Ciao, > Alessio Rob 1309257543.0 From rob at landley.net Wed Jun 29 12:52:51 2011 From: rob at landley.net (Rob Landley) Date: Wed, 29 Jun 2011 07:52:51 -0500 Subject: [Aboriginal] [PATCH] Move i586 to virtio In-Reply-To: References: <9cb1d126a6102d47f326.1308736004@granny> <4E087DE4.9070008@landley.net> Message-ID: <4E0B2023.6000303@landley.net> On 06/27/2011 10:17 AM, Alessio Igor Bogani wrote: > Rob, > > 2011/6/27 Rob Landley : >> On 06/22/2011 04:46 AM, Alessio Igor Bogani wrote: >>> # HG changeset patch >>> # User Alessio Igor Bogani >>> # Date 1308735970 -7200 >>> # Node ID 9cb1d126a6102d47f3266a148247c97758a01bfd >>> # Parent 67accb5d252602b3b2ad2d0b8aa6fd9a62c74030 >>> Move i586 to virtio >> >> I moved this to i686-kvm target. Could you test if it works for you? > > Unfortunately it doesn't work on my system: > [...] > Root device is (9, 0) > Setup is 12940 bytes (padded to 13312 bytes). > System is 1378 kB > CRC e329e245 > Kernel: arch/x86/boot/bzImage is ready (#1) > > Exiting due to errors (i686-kvm linux-kernel linux) > [...] The actual error line was cut out of your bug report, it was the attempt at tarball creation that failed. I fixed it, and now i686-kvm is building for me when I haven't built the i686 base architecture target yet. Still no idea if the resulting system image works for you... Rob 1309351971.0 From alessioigorbogani at gmail.com Wed Jun 29 13:49:21 2011 From: alessioigorbogani at gmail.com (Alessio Igor Bogani) Date: Wed, 29 Jun 2011 15:49:21 +0200 Subject: [Aboriginal] [PATCH] Move i586 to virtio In-Reply-To: <4E0B2023.6000303@landley.net> References: <9cb1d126a6102d47f326.1308736004@granny> <4E087DE4.9070008@landley.net> <4E0B2023.6000303@landley.net> Message-ID: Rob, [...] > The actual error line was cut out of your bug report, Sorry for that. > it was the attempt at tarball creation that failed. ?I fixed it, and now i686-kvm is > building for me when I haven't built the i686 base architecture target yet. Great thanks! > Still no idea if the resulting system image works for you... I can build i686-kvm target on Ubuntu Natty 11.04 and Ubuntu Lucid 10.04.2 and execute it with the qemu shipped with these without any problem. Unfortunately my KVM enabled system broke three days ago so my test is made only on two systems which don't have that feature. I'll test the new target on a KVM capable system as soon as possible. In any case the i686-kvm image is faster than of the i686 one. Thanks again! Ciao, Alessio 1309355361.0