From james_mcmechan at hotmail.com Thu Dec 1 20:43:10 2011 From: james_mcmechan at hotmail.com (James McMechan) Date: Thu, 1 Dec 2011 12:43:10 -0800 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: <4ED61C00.1060304@landley.net> References: <20111123151627.7e79c691.onefang@gmail.com>, <20111123161610.13b6f007.onefang@gmail.com>, <20111123164529.6ec76a89.onefang@gmail.com>, <20111123174854.73b5e14b.onefang@gmail.com>, <20111124021351.77614762.onefang@gmail.com> , <4ED61C00.1060304@landley.net> Message-ID: > > I saw David Seikel's comments about still being interested in trim-config > > Here is how I use mini-config with busybox, it is not hard > > though I have not yet gotten the auto build stuff to include the aboriginal > > tarball and the package directory in my initramfs > > There was an initramfs bug recent-ish-ly, are you using a version after > http://landley.net/hg/aboriginal/rev/1366 ? I was using Aboriginal-1.1.0 but am having a few problems I seem to remember it working fine with 1.0.1, 1.0.0 and 0.9.11 now for some reason the patch command is failing, I will need to double check the config file for busybox it may have a new knob in config somewhere > > --- busybox.orig 2011-11-27 05:41:26.000000000 -0800 > > +++ busybox.build 2011-11-27 16:19:20.000000000 -0800 > > @@ -5,7 +5,8 @@ > > # Build busybox > > > > #make allyesconfig KCONFIG_ALLCONFIG="${SOURCES}/trimconfig-busybox" && > > -make defconfig && > > +#make defconfig && > > +make allnoconfig KCONFIG_ALLCONFIG="${SOURCES}/mini-busybox" && > > I need to provide a way to select this, do a defconfig if no busybox > config is provided, otherwise use that. Probably some kind of config > setting. well there is the classic method to try all three, there are prettier layouts though done=0 [ $done -eq 0 -a -e "${SOURCES}/mini-busybox" ] && make allnoconfig KCONFIG_ALLCONFIG="${SOURCES}/mini-busybox" && done = 1 [ $done -eq 0 -a -e "${SOURCES}/trim-busybox" ] && make allyesconfig KCONFIG_ALLCONFIG="${SOURCES}/trim-busybox" && done = 1 [ $done -eq 0 ] && make defconfig && done = 1 [ $done -ne 0 ] && \ > I'm also back to working on toybox, which means testing toybox, which > means getting toybox and busybox to install alongside each other and > gradually reduce the number of busybox commands I'm using. Personally, I like building up from the bottom, rather than trying to trim the fat from the top... On a side note, I noticed busybox defconfig even has a webserver so you could add your own private package mirror to dev-environment.sh > I'm not inflicting toybox on aboriginal's default configuration any time > soon, but I personally need the option. > > > cp .config "$WORK"/config-busybox && > > LDFLAGS="$LDFLAGS $STATIC_FLAGS" make -j $CPUS $VERBOSITY $DO_CROSS && > > make busybox.links || dienow > > > > I was adding very little to config > > > > export NO_CPLUSPLUS=1 > > export SYSIMAGE_TYPE=initramfs > > > > My preference is for mini-config rather than trim-config so I can turn on just what I want > > I admit the trimconfig was a bit of a hack; trying to use defconfig but > it didn't work. And now busybox defconfig is back to not working > because the UBI stuff doesn't build on ubuntu 11.10. I like miniconfig primarily because of the lack of changes, if some later version needs a extra config option the option is ignored by previous versions, the base stuff in busybox mostly just works.? It is the advanced stuff and new options that usually give me grief. > > I am using a ~98 line mini-config which needs few changes between busybox versions > > It is possible that a few more options can be turned off I would like to convert the > > initramfs to bz2 and drop gzip. > > Without genext2fs it appears it doesn't need TAR_OLDGNU_COMPATIBILITY > > apparently that program's build environment was very very old. > > I need to write my own genext2fs. Now that I'm back working on toybox, > I may manage that eventually... I usually just use initramfs I only need ~2 device files to boot for testing so the setup for cpio is quite easy even by hand. mdev from busybox takes care of the rest of the device files later. The new DEVTMPFS in the kernel might make even that unneeded I have not tested without /dev/console yet though... > > #needed for od -t option in busybox build > > CONFIG_DESKTOP=y > > I've argued with Denys about that. CONFIG_DESKTOP is not well defined, > there's no clear rule for what it does and doesn't do. > > This config is enough to build aboriginal linux itself, with this in > hostconfig? It was working earlier, last tested about 1.0.2, I need to check with the new version. > A) Does that include running the ./download.sh stage? download.sh does not use the host tools path and breaks at the moment I did not set wget or ifconfig in busybox nor test download.sh > B) Have you tried lfs-bootstrap.hdc with this? I have not been checking LFS I was looking more like tinycore linux and most of my testing has been under user-mode-linux since I had that ready to run already When testing I repeatedly run the same kernel with different initramfs or recompile the kernel repeatedly with the same initramfs they are conceptually separate in my mind so I almost always work on one or the other alternately. I noted that the command I have been testing with does not seem to work for host tools like might be hoped. STAGE_NAME=host-tools more/test.sh i686 build_section busybox it gets all unhappy about pod2text pod2man & pod2html, but leaves a statically linked busybox binary in host-tools-i686 It would be somewhat easier if host acted like a real target. The current special case logic seem to get in my way as much as help when trying repeated host-tools setups. It was also quite awkward to use the default simple-cross-compiler.sh, when I was having trouble with uClibc or linux-headers as it kept rebuilding binutils & gcc. I quickly commented them out, but it would seem that having each package saved in the build directory not just the assembled meta-packages would be nice. For example busybox-host.tbz busybox-i686.tbz uClibc-host.tbz My opinion is when a system is working well simple things are simple, and hard things are straightforward. I almost seems like too much has been spent here make the hard parts easy and the simpler parts have gotten more complex as a result. 1322772190.0 From rob at landley.net Fri Dec 2 13:34:42 2011 From: rob at landley.net (Rob Landley) Date: Fri, 02 Dec 2011 07:34:42 -0600 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: References: <20111123151627.7e79c691.onefang@gmail.com>, <20111123161610.13b6f007.onefang@gmail.com>, <20111123164529.6ec76a89.onefang@gmail.com>, <20111123174854.73b5e14b.onefang@gmail.com>, <20111124021351.77614762.onefang@gmail.com> , <4ED61C00.1060304@landley.net> Message-ID: <4ED8D3F2.9020907@landley.net> On 12/01/2011 02:43 PM, James McMechan wrote: > >>> I saw David Seikel's comments about still being interested in trim-config >>> Here is how I use mini-config with busybox, it is not hard >>> though I have not yet gotten the auto build stuff to include the aboriginal >>> tarball and the package directory in my initramfs >> >> There was an initramfs bug recent-ish-ly, are you using a version after >> http://landley.net/hg/aboriginal/rev/1366 ? > > I was using Aboriginal-1.1.0 but am having a few problems > I seem to remember it working fine with 1.0.1, 1.0.0 and 0.9.11 > now for some reason the patch command is failing, I will need to double > check the config file for busybox it may have a new knob in config somewhere I fixed the toybox/busybox patch command in August and again in October, possibly you're hitting one of those bugs. I need to get the next release out, but I keep hoping I can beat some sense out of sparc first. (It's almost building LFS! Progress!) >>> --- busybox.orig 2011-11-27 05:41:26.000000000 -0800 >>> +++ busybox.build 2011-11-27 16:19:20.000000000 -0800 >>> @@ -5,7 +5,8 @@ >>> # Build busybox >>> >>> #make allyesconfig KCONFIG_ALLCONFIG="${SOURCES}/trimconfig-busybox" && >>> -make defconfig && >>> +#make defconfig && >>> +make allnoconfig KCONFIG_ALLCONFIG="${SOURCES}/mini-busybox" && >> >> I need to provide a way to select this, do a defconfig if no busybox >> config is provided, otherwise use that. Probably some kind of config >> setting. > > well there is the classic method to try all three, Yeah, but should it expect to find them in $SOURCES (where all the other files would be checked into source control, but these wouldn't) or at a location specified in config? The overlay mechanism for extra root filesystem files is config based for the same reason, mixing tracked and untracked files in the same directory is untidy. Long-term I want to move to toybox and just not HAVE buckets of extraneous crap needing to be disabled... > there are prettier layouts though > done=0 > [ $done -eq 0 -a -e "${SOURCES}/mini-busybox" ] && make allnoconfig KCONFIG_ALLCONFIG="${SOURCES}/mini-busybox" && done = 1 > [ $done -eq 0 -a -e "${SOURCES}/trim-busybox" ] && make allyesconfig KCONFIG_ALLCONFIG="${SOURCES}/trim-busybox" && done = 1 > [ $done -eq 0 ] && make defconfig && done = 1 > [ $done -ne 0 ] && \ Probably an if/else staircase. That's what they're for. >> I'm also back to working on toybox, which means testing toybox, which >> means getting toybox and busybox to install alongside each other and >> gradually reduce the number of busybox commands I'm using. > > Personally, I like building up from the bottom, rather than trying to trim > the fat from the top... When you're migrating a system that currently works by replacing components, you have to build the new thing while dismantling the old, so you wind up doing both... > On a side note, I noticed busybox defconfig even has a webserver so > you could add your own private package mirror to dev-environment.sh I could. But I prefer to leave that sort of thing to other people: I'm intentionally agnostic about what you _do_ with the resulting system. The hdc infrastructure lets you make self contained build images, but my examples include one that just runs the busybox test suite. >> I'm not inflicting toybox on aboriginal's default configuration any time >> soon, but I personally need the option. >> >>> cp .config "$WORK"/config-busybox && >>> LDFLAGS="$LDFLAGS $STATIC_FLAGS" make -j $CPUS $VERBOSITY $DO_CROSS && >>> make busybox.links || dienow >>> >>> I was adding very little to config >>> >>> export NO_CPLUSPLUS=1 >>> export SYSIMAGE_TYPE=initramfs >>> >>> My preference is for mini-config rather than trim-config so I can turn on just what I want >> >> I admit the trimconfig was a bit of a hack; trying to use defconfig but >> it didn't work. And now busybox defconfig is back to not working >> because the UBI stuff doesn't build on ubuntu 11.10. > > I like miniconfig primarily because of the lack of changes, if some later version needs a extra > config option the option is ignored by previous versions, the base stuff in busybox mostly > just works. It is the advanced stuff and new options that usually give me grief. Busybox had a 1.0 years ago and was presumably feature complete at that time. Several of its commands weren't feature-complete enough to work in a build environment, but very few were actually _missing_. Since then it's grown about 100 commands. >>> I am using a ~98 line mini-config which needs few changes between busybox versions >>> It is possible that a few more options can be turned off I would like to convert the >>> initramfs to bz2 and drop gzip. >>> Without genext2fs it appears it doesn't need TAR_OLDGNU_COMPATIBILITY >>> apparently that program's build environment was very very old. >> >> I need to write my own genext2fs. Now that I'm back working on toybox, >> I may manage that eventually... > > I usually just use initramfs I only need ~2 device files to boot > for testing so the setup for cpio is quite easy even by hand. For /dev I'm just mounting -t devtmpfs these days, as of commit 1414 in July. If you only have a root user in the emulated system, that's plenty. > mdev from busybox takes care of the rest of the device files later. > The new DEVTMPFS in the kernel might make even that unneeded > I have not tested without /dev/console yet though... Doesn't devtmpfs give you /dev/console? >>> #needed for od -t option in busybox build >>> CONFIG_DESKTOP=y >> >> I've argued with Denys about that. CONFIG_DESKTOP is not well defined, >> there's no clear rule for what it does and doesn't do. >> >> This config is enough to build aboriginal linux itself, with this in >> hostconfig? > > It was working earlier, last tested about 1.0.2, I need to check with the new > version. > >> A) Does that include running the ./download.sh stage? > > download.sh does not use the host tools path and breaks at the moment It will if you run host-tools.sh, zap packages, and then re-run ./download.sh. But yeah, that's not the normal order of operations, which is why I've learned to test that separately. I only occasionally run the aboriginal build itself under a system image (to catch that sort of thing), and it's usually download.sh that breaks if something's going to. > I did not set wget or ifconfig in busybox nor test download.sh Note that sources/root-filesystem/sbin/init.sh has two paths twoards the end, one for PID 1 and one for being called in a chroot. >> B) Have you tried lfs-bootstrap.hdc with this? > > I have not been checking LFS I was looking more like tinycore linux > and most of my testing has been under user-mode-linux since I had > that ready to run already I mention it because lfs-bootstrap.hdc automates building LFS 6.8, so that's my biggest test case right now. http://landley.net/aboriginal/control-images > When testing I repeatedly run the same kernel with different initramfs > or recompile the kernel repeatedly with the same initramfs they are > conceptually separate in my mind so I almost always work on one > or the other alternately. > > I noted that the command I have been testing with does not seem to > work for host tools like might be hoped. > STAGE_NAME=host-tools more/test.sh i686 build_section busybox > it gets all unhappy about pod2text pod2man & pod2html, but Busybox always gets unhappy about those when perl isn't installed, but it's not a fatal error. > leaves a statically linked busybox binary in host-tools-i686 host-tools-i686? Um, host-tools.sh is target agnostic, it imports sources/include.sh but never calls load_target. I'm confused? > It would be somewhat easier if host acted like a real target. It's sort of the point of the host stuff is to be target agnostic. You should be able to build all the targets under the same host environment. (The fact that different releases of Ubuntu and Fedora and such don't _provide_ the same host environment is why host-tools.sh exists, but the point stands.) > The current special case logic seem to get in my way as much as help > when trying repeated host-tools setups. Which special case logic? I broke the initialization in half so include.sh sets the basic variables and then load_target does all the target-specific stuff. So the "special case is" that host-tools.sh and download.sh are _not_ doing the target initialization. Now there's a bunch of special case crap for "on some versions of ubuntu the 'gcc' in the path is a perl script, we need to unwrap that", and "you have distcc installed so we have to care about more than one instance of 'gcc' in the path", and so on. But that's because various host environments we run on are funky. > It was also quite awkward to use the default simple-cross-compiler.sh, > when I was having trouble with uClibc or linux-headers as it kept > rebuilding binutils & gcc. This is why I broke the package builds into sources/sections and made the build_section function to call them, and also why I added more/test.sh to run snippets in build context (with sources/include.sh and load_target called): http://landley.net/aboriginal/FAQ.html#debug_test Its "usage:" example suggests setting STAGE_NAME so in this case you probably want: STAGE_NAME=simple-cross-compiler more/test.sh sparc build_section uClibc I should probably put that example in the FAQ... Done. > I quickly commented them out, but it would > seem that having each package saved in the build directory not just > the assembled meta-packages would be nice. Um, do you mean like the config option BINARY_PACKAGE_TARBALLS? > For example busybox-host.tbz busybox-i686.tbz uClibc-host.tbz > > My opinion is when a system is working well simple things are > simple, and hard things are straightforward. I almost seems like > too much has been spent here make the hard parts easy and the > simpler parts have gotten more complex as a result. That's always a danger. I agonized about putting in the package cache, and wound up writing EXTENSIVE DOCUEMNTATION on it (it's about half the FAQ). But the fact was: A) it sped up the build _tremendously_, especially when building multiple architectures on a machine with a finite amount of memory. (And even for single builds you could wind up extracting the kernel tarball _four_times_ during the build: kernel headers for simple cross compiler, cross compiler, and native compiler, and then actually building the kernel binary. Even busybox is in host-tools _and_ simple-root-filesystem...) B) You can conceptually completely ignore it, and pretend "setupfor blah" is just a wrapper for "mkdir build/blah && tar xvjfC packages/blah.tbz build/blah && cd build/blah" and that cleanup is "cd .. && rm -rf blah". I wouldn't have done A without B. I _didn't_ do A until I could figure out how to make B true. Rob 1322832882.0 From james_mcmechan at hotmail.com Sat Dec 3 02:24:59 2011 From: james_mcmechan at hotmail.com (James McMechan) Date: Fri, 2 Dec 2011 18:24:59 -0800 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: <4ED8D3F2.9020907@landley.net> References: <20111123151627.7e79c691.onefang@gmail.com>, <20111123161610.13b6f007.onefang@gmail.com>, <20111123164529.6ec76a89.onefang@gmail.com>, <20111123174854.73b5e14b.onefang@gmail.com>, <20111124021351.77614762.onefang@gmail.com> , <4ED61C00.1060304@landley.net> , <4ED8D3F2.9020907@landley.net> Message-ID: > >>> I saw David Seikel's comments about still being interested in trim-config > >>> Here is how I use mini-config with busybox, it is not hard > >>> though I have not yet gotten the auto build stuff to include the aboriginal > >>> tarball and the package directory in my initramfs > >> > >> There was an initramfs bug recent-ish-ly, are you using a version after > >> http://landley.net/hg/aboriginal/rev/1366 ? > > > > I was using Aboriginal-1.1.0 but am having a few problems > > I seem to remember it working fine with 1.0.1, 1.0.0 and 0.9.11 > > now for some reason the patch command is failing, I will need to double > > check the config file for busybox it may have a new knob in config somewhere > > I fixed the toybox/busybox patch command in August and again in October, > possibly you're hitting one of those bugs. > > I need to get the next release out, but I keep hoping I can beat some > sense out of sparc first. (It's almost building LFS! Progress!) > > >>> --- busybox.orig 2011-11-27 05:41:26.000000000 -0800 > >>> +++ busybox.build 2011-11-27 16:19:20.000000000 -0800 > >>> @@ -5,7 +5,8 @@ > >>> # Build busybox > >>> > >>> #make allyesconfig KCONFIG_ALLCONFIG="${SOURCES}/trimconfig-busybox" && > >>> -make defconfig && > >>> +#make defconfig && > >>> +make allnoconfig KCONFIG_ALLCONFIG="${SOURCES}/mini-busybox" && > >> > >> I need to provide a way to select this, do a defconfig if no busybox > >> config is provided, otherwise use that. Probably some kind of config > >> setting. > > > > well there is the classic method to try all three, > > Yeah, but should it expect to find them in $SOURCES (where all the other > files would be checked into source control, but these wouldn't) or at a > location specified in config? The overlay mechanism for extra root > filesystem files is config based for the same reason, mixing tracked and > untracked files in the same directory is untidy. if you had a config directory I would have aimed for there, SOURCES is where the trimconfig used to reside.? Also I would expect that mini-config & company would be checked in. > Long-term I want to move to toybox and just not HAVE buckets of > extraneous crap needing to be disabled... It would be very nice. > > there are prettier layouts though > > done=0 > > [ $done -eq 0 -a -e "${SOURCES}/mini-busybox" ] && make allnoconfig KCONFIG_ALLCONFIG="${SOURCES}/mini-busybox" && done = 1 > > [ $done -eq 0 -a -e "${SOURCES}/trim-busybox" ] && make allyesconfig KCONFIG_ALLCONFIG="${SOURCES}/trim-busybox" && done = 1 > > [ $done -eq 0 ] && make defconfig && done = 1 > > [ $done -ne 0 ] && \ > > Probably an if/else staircase. That's what they're for. I don't remember if classic shell had else, I would still like it able to run under ash or other shells from the days of chipped flint systems. > >> I'm also back to working on toybox, which means testing toybox, which > >> means getting toybox and busybox to install alongside each other and > >> gradually reduce the number of busybox commands I'm using. > > > > Personally, I like building up from the bottom, rather than trying to trim > > the fat from the top... > > When you're migrating a system that currently works by replacing > components, you have to build the new thing while dismantling the old, > so you wind up doing both... I was thinking that if you don't have the extra 100 commands from busybox the failures might be shorter. e.g. turn off od in busybox turn on in toybox test and repeat > > On a side note, I noticed busybox defconfig even has a webserver so > > you could add your own private package mirror to dev-environment.sh > > I could. But I prefer to leave that sort of thing to other people: I'm > intentionally agnostic about what you _do_ with the resulting system. > The hdc infrastructure lets you make self contained build images, but my > examples include one that just runs the busybox test suite. Err I was thinking of having a local package mirror that could be setup automatically when running dev-enviornment.sh after all when you have just built this version you know where a valid set of packages are. also it is somewhat humorous that the minimal busybox includes a webserver... > >> I'm not inflicting toybox on aboriginal's default configuration any time > >> soon, but I personally need the option. > >> > >>> cp .config "$WORK"/config-busybox && > >>> LDFLAGS="$LDFLAGS $STATIC_FLAGS" make -j $CPUS $VERBOSITY $DO_CROSS && > >>> make busybox.links || dienow > >>> > >>> I was adding very little to config > >>> > >>> export NO_CPLUSPLUS=1 > >>> export SYSIMAGE_TYPE=initramfs > >>> > >>> My preference is for mini-config rather than trim-config so I can turn on just what I want > >> > >> I admit the trimconfig was a bit of a hack; trying to use defconfig but > >> it didn't work. And now busybox defconfig is back to not working > >> because the UBI stuff doesn't build on ubuntu 11.10. > > > > I like miniconfig primarily because of the lack of changes, if some later version needs a extra > > config option the option is ignored by previous versions, the base stuff in busybox mostly > > just works. It is the advanced stuff and new options that usually give me grief. > > Busybox had a 1.0 years ago and was presumably feature complete at that > time. Several of its commands weren't feature-complete enough to work > in a build environment, but very few were actually _missing_. > > Since then it's grown about 100 commands. > > >>> I am using a ~98 line mini-config which needs few changes between busybox versions > >>> It is possible that a few more options can be turned off I would like to convert the > >>> initramfs to bz2 and drop gzip. > >>> Without genext2fs it appears it doesn't need TAR_OLDGNU_COMPATIBILITY > >>> apparently that program's build environment was very very old. > >> > >> I need to write my own genext2fs. Now that I'm back working on toybox, > >> I may manage that eventually... > > > > I usually just use initramfs I only need ~2 device files to boot > > for testing so the setup for cpio is quite easy even by hand. > > For /dev I'm just mounting -t devtmpfs these days, as of commit 1414 in > July. If you only have a root user in the emulated system, that's plenty. > > > mdev from busybox takes care of the rest of the device files later. > > The new DEVTMPFS in the kernel might make even that unneeded > > I have not tested without /dev/console yet though... > > Doesn't devtmpfs give you /dev/console? I have not yet tried a devtmpfs only dev it should work but does not automount in initramfs if I am reading the documentation correctly some early parts of userspace used to break without /dev/console > >>> #needed for od -t option in busybox build > >>> CONFIG_DESKTOP=y > >> > >> I've argued with Denys about that. CONFIG_DESKTOP is not well defined, > >> there's no clear rule for what it does and doesn't do. > >> > >> This config is enough to build aboriginal linux itself, with this in > >> hostconfig? > > > > It was working earlier, last tested about 1.0.2, I need to check with the new > > version. > > > >> A) Does that include running the ./download.sh stage? > > > > download.sh does not use the host tools path and breaks at the moment > > It will if you run host-tools.sh, zap packages, and then re-run > ./download.sh. you sure it looked like it overrode the path back to OLDPATH? > But yeah, that's not the normal order of operations, which is why I've > learned to test that separately. I only occasionally run the aboriginal > build itself under a system image (to catch that sort of thing), and > it's usually download.sh that breaks if something's going to. > > > I did not set wget or ifconfig in busybox nor test download.sh > > Note that sources/root-filesystem/sbin/init.sh has two paths twoards the > end, one for PID 1 and one for being called in a chroot. > > >> B) Have you tried lfs-bootstrap.hdc with this? > > > > I have not been checking LFS I was looking more like tinycore linux > > and most of my testing has been under user-mode-linux since I had > > that ready to run already > > I mention it because lfs-bootstrap.hdc automates building LFS 6.8, so > that's my biggest test case right now. > > http://landley.net/aboriginal/control-images > > > When testing I repeatedly run the same kernel with different initramfs > > or recompile the kernel repeatedly with the same initramfs they are > > conceptually separate in my mind so I almost always work on one > > or the other alternately. > > > > I noted that the command I have been testing with does not seem to > > work for host tools like might be hoped. > > STAGE_NAME=host-tools more/test.sh i686 build_section busybox > > it gets all unhappy about pod2text pod2man & pod2html, but > > Busybox always gets unhappy about those when perl isn't installed, but > it's not a fatal error. > > > leaves a statically linked busybox binary in host-tools-i686 > > host-tools-i686? Yep, imore/test.sh will not accept a arch of host so I was using i686 and it puts stuff in $STAGE_NAME-$ARCH > Um, host-tools.sh is target agnostic, it imports sources/include.sh but > never calls load_target. I'm confused? I was trying out different mini-configs for busybox in a manual fashion Um-- I think of host as the first target the one built with the host toolchain kernel headers and host C library. > > It would be somewhat easier if host acted like a real target. > > It's sort of the point of the host stuff is to be target agnostic. You > should be able to build all the targets under the same host environment. > (The fact that different releases of Ubuntu and Fedora and such don't > _provide_ the same host environment is why host-tools.sh exists, but the > point stands.) Err, building the aboriginal host tool set was the target I was referring to. > > The current special case logic seem to get in my way as much as help > > when trying repeated host-tools setups. > > Which special case logic? whether busybox is linked with the host libc or uClibc from aboriginal for example. > I broke the initialization in half so include.sh sets the basic > variables and then load_target does all the target-specific stuff. So > the "special case is" that host-tools.sh and download.sh are _not_ doing > the target initialization. > > Now there's a bunch of special case crap for "on some versions of ubuntu > the 'gcc' in the path is a perl script, we need to unwrap that", and > "you have distcc installed so we have to care about more than one > instance of 'gcc' in the path", and so on. But that's because various > host environments we run on are funky. > > > It was also quite awkward to use the default simple-cross-compiler.sh, > > when I was having trouble with uClibc or linux-headers as it kept > > rebuilding binutils & gcc. > > This is why I broke the package builds into sources/sections and made > the build_section function to call them, and also why I added > more/test.sh to run snippets in build context (with sources/include.sh > and load_target called): Yes, more/test.sh is the one I was using above for busybox > http://landley.net/aboriginal/FAQ.html#debug_test > > Its "usage:" example suggests setting STAGE_NAME so in this case you > probably want: > > STAGE_NAME=simple-cross-compiler more/test.sh sparc build_section uClibc > > I should probably put that example in the FAQ... Done. > > > I quickly commented them out, but it would > > seem that having each package saved in the build directory not just > > the assembled meta-packages would be nice. > > Um, do you mean like the config option BINARY_PACKAGE_TARBALLS? Looks like it, does it remember the tarballs and not build them again when present? I will have to check this option. > > For example busybox-host.tbz busybox-i686.tbz uClibc-host.tbz > > > > My opinion is when a system is working well simple things are > > simple, and hard things are straightforward. I almost seems like > > too much has been spent here make the hard parts easy and the > > simpler parts have gotten more complex as a result. > > That's always a danger. > > I agonized about putting in the package cache, and wound up writing > EXTENSIVE DOCUEMNTATION on it (it's about half the FAQ). But the fact was: > > A) it sped up the build _tremendously_, especially when building > multiple architectures on a machine with a finite amount of memory. > > (And even for single builds you could wind up extracting the kernel > tarball _four_times_ during the build: kernel headers for simple cross > compiler, cross compiler, and native compiler, and then actually > building the kernel binary. Even busybox is in host-tools _and_ > simple-root-filesystem...) > > B) You can conceptually completely ignore it, and pretend "setupfor > blah" is just a wrapper for "mkdir build/blah && tar xvjfC > packages/blah.tbz build/blah && cd build/blah" and that cleanup is "cd > .. && rm -rf blah". > > I wouldn't have done A without B. I _didn't_ do A until I could figure > out how to make B true. > > Rob 1322879099.0 From rob at landley.net Mon Dec 5 18:26:52 2011 From: rob at landley.net (Rob Landley) Date: Mon, 05 Dec 2011 12:26:52 -0600 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: References: <20111123151627.7e79c691.onefang@gmail.com>, <20111123161610.13b6f007.onefang@gmail.com>, <20111123164529.6ec76a89.onefang@gmail.com>, <20111123174854.73b5e14b.onefang@gmail.com>, <20111124021351.77614762.onefang@gmail.com> , <4ED61C00.1060304@landley.net> , <4ED8D3F2.9020907@landley.net> Message-ID: <4EDD0CEC.3070100@landley.net> On 12/02/2011 08:24 PM, James McMechan wrote: > >>>>> I saw David Seikel's comments about still being interested in trim-config >>>>> Here is how I use mini-config with busybox, it is not hard >>>>> though I have not yet gotten the auto build stuff to include the aboriginal >>>>> tarball and the package directory in my initramfs >>>> >>>> There was an initramfs bug recent-ish-ly, are you using a version after >>>> http://landley.net/hg/aboriginal/rev/1366 ? >>> >>> I was using Aboriginal-1.1.0 but am having a few problems >>> I seem to remember it working fine with 1.0.1, 1.0.0 and 0.9.11 >>> now for some reason the patch command is failing, I will need to double >>> check the config file for busybox it may have a new knob in config somewhere >> >> I fixed the toybox/busybox patch command in August and again in October, >> possibly you're hitting one of those bugs. >> >> I need to get the next release out, but I keep hoping I can beat some >> sense out of sparc first. (It's almost building LFS! Progress!) >> >>>>> --- busybox.orig 2011-11-27 05:41:26.000000000 -0800 >>>>> +++ busybox.build 2011-11-27 16:19:20.000000000 -0800 >>>>> @@ -5,7 +5,8 @@ >>>>> # Build busybox >>>>> >>>>> #make allyesconfig KCONFIG_ALLCONFIG="${SOURCES}/trimconfig-busybox" && >>>>> -make defconfig && >>>>> +#make defconfig && >>>>> +make allnoconfig KCONFIG_ALLCONFIG="${SOURCES}/mini-busybox" && >>>> >>>> I need to provide a way to select this, do a defconfig if no busybox >>>> config is provided, otherwise use that. Probably some kind of config >>>> setting. >>> >>> well there is the classic method to try all three, >> >> Yeah, but should it expect to find them in $SOURCES (where all the other >> files would be checked into source control, but these wouldn't) or at a >> location specified in config? The overlay mechanism for extra root >> filesystem files is config based for the same reason, mixing tracked and >> untracked files in the same directory is untidy. > > if you had a config directory I would have aimed for there, Hmmm... I don't want to have too _much_ configuration. A hammer isn't a better hammer if it's got 37 switches you have to flip before pounding nails. I had the target directories (which I eventually cleaned up into individual target files for most of them), because each target is somewhat different and I wanted a single place to collect all those differences. The stuff in "sources" is the stuff that's the same for all targets. I added an "overlay" capability so people didn't have to extensively fiddle with sources/root-filesystem. > SOURCES is where the trimconfig used to reside. > Also I would expect that mini-config & company > would be checked in. Back then trimconfig was "what all targets needed". It belonged in sources for the same reason the kernel and uClibc baseconfigs do: it was generic. Then I got enough changes pushed upstream into busybox that defconfig could work for all targets, and from the point of view of _this_ project using defconfig was simpler than specifying a config that could get out of sync with what upstream was doing. (Which is inherent in trimconfig: a delta against upstream.) That said, a busybox baseconfig makes sense from a design standpoint. The minimal native development environment capable of rebuilding itself under itself: stuff like linux from scratch has to bootstrap its way up (possibly by building busybox defconfig first)... >> Long-term I want to move to toybox and just not HAVE buckets of >> extraneous crap needing to be disabled... > > It would be very nice. I'm working on it. Just did a blog entry about what record-commands says I need for the build. (Might not get to publishing it until tomorrow, needs some cleanup.) It boils down to me needing to add this to toybox in order for it to be able to replace busybox in aboriginal linux (at least for the i586 build, I need to check other targets, and I'm not quite sure what LFS needs...): awk basename bzip2 chmod chown cmp cut date dd diff dirname egrep env expr find grep gzip head hostname id install ld ln ls mkdir mksquashfs mktemp mv od patch readlink rm sed sh tail tar touch tr uniq wget whoami xargs About half of those are low-hanging fruit anyway... >>> there are prettier layouts though >>> done=0 >>> [ $done -eq 0 -a -e "${SOURCES}/mini-busybox" ] && make allnoconfig KCONFIG_ALLCONFIG="${SOURCES}/mini-busybox" && done = 1 >>> [ $done -eq 0 -a -e "${SOURCES}/trim-busybox" ] && make allyesconfig KCONFIG_ALLCONFIG="${SOURCES}/trim-busybox" && done = 1 >>> [ $done -eq 0 ] && make defconfig && done = 1 >>> [ $done -ne 0 ] && \ >> >> Probably an if/else staircase. That's what they're for. > > I don't remember if classic shell had else, Yup, SUSv3 and SUSv4. > I would still like it able to run under > ash or other shells from the days of chipped flint systems. The current aboriginal linux scripts don't. They use blah/{curly,bracket} wildcards and <(this kind) of pipe where the command gets a filename it can read from to get the output of a command (and there's a >(version) that's a file that outputs _to_ that command), so you can do: diff -u <(blah | sort -u) < (blah2 | sort -u) And so on... Probably other stuff. I intend to write a shell that can cope with all this, not strip down my script to be cripped for the Defective Annoying SHell in bash. Remember: I didn't remove the -i option from uses of sed, I extended busybox sed to cope with -i. I haven't seriously banged on busybox in years, but now I'm back throwing scraps of spare time at toybox. >>>> I'm also back to working on toybox, which means testing toybox, which >>>> means getting toybox and busybox to install alongside each other and >>>> gradually reduce the number of busybox commands I'm using. >>> >>> Personally, I like building up from the bottom, rather than trying to trim >>> the fat from the top... >> >> When you're migrating a system that currently works by replacing >> components, you have to build the new thing while dismantling the old, >> so you wind up doing both... > > I was thinking that if you don't have the extra 100 commands from busybox > the failures might be shorter. e.g. turn off od in busybox turn on in toybox > test and repeat I was pretty much planning on doing that anyway. :) Although there's a cheap and easy way to do this: install toybox first, then have the busybox installer simply fail to create symlinks when they already exist. This is a long email, I'll have to finish replying to it later... Rob 1323109612.0 From onefang at gmail.com Mon Dec 5 23:08:14 2011 From: onefang at gmail.com (David Seikel) Date: Tue, 6 Dec 2011 09:08:14 +1000 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: <4EDD0CEC.3070100@landley.net> References: <20111123151627.7e79c691.onefang@gmail.com> <20111123161610.13b6f007.onefang@gmail.com> <20111123164529.6ec76a89.onefang@gmail.com> <20111123174854.73b5e14b.onefang@gmail.com> <20111124021351.77614762.onefang@gmail.com> <4ED61C00.1060304@landley.net> <4ED8D3F2.9020907@landley.net> <4EDD0CEC.3070100@landley.net> Message-ID: <20111206090814.7b9c0b99.onefang@gmail.com> On Mon, 05 Dec 2011 12:26:52 -0600 Rob Landley wrote: > > I'm working on it. Just did a blog entry about what record-commands > says I need for the build. (Might not get to publishing it until > tomorrow, needs some cleanup.) > > It boils down to me needing to add this to toybox in order for it to > be able to replace busybox in aboriginal linux (at least for the i586 > build, I need to check other targets, and I'm not quite sure what LFS > needs...): > > awk basename bzip2 chmod chown cmp cut date dd diff dirname > egrep env expr find grep gzip head hostname id install ld ln ls > mkdir mksquashfs mktemp mv od patch readlink rm sed sh > tail tar touch tr uniq wget whoami xargs As the author of the i486 target, I'll certainly be trying it there. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From rob at landley.net Tue Dec 6 12:31:50 2011 From: rob at landley.net (Rob Landley) Date: Tue, 06 Dec 2011 06:31:50 -0600 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: <20111206090814.7b9c0b99.onefang@gmail.com> References: <20111123151627.7e79c691.onefang@gmail.com> <20111123161610.13b6f007.onefang@gmail.com> <20111123164529.6ec76a89.onefang@gmail.com> <20111123174854.73b5e14b.onefang@gmail.com> <20111124021351.77614762.onefang@gmail.com> <4ED61C00.1060304@landley.net> <4ED8D3F2.9020907@landley.net> <4EDD0CEC.3070100@landley.net> <20111206090814.7b9c0b99.onefang@gmail.com> Message-ID: <4EDE0B36.7040505@landley.net> On 12/05/2011 05:08 PM, David Seikel wrote: > On Mon, 05 Dec 2011 12:26:52 -0600 Rob Landley wrote: >> >> I'm working on it. Just did a blog entry about what record-commands >> says I need for the build. (Might not get to publishing it until >> tomorrow, needs some cleanup.) >> >> It boils down to me needing to add this to toybox in order for it to >> be able to replace busybox in aboriginal linux (at least for the i586 >> build, I need to check other targets, and I'm not quite sure what LFS >> needs...): >> >> awk basename bzip2 chmod chown cmp cut date dd diff dirname >> egrep env expr find grep gzip head hostname id install ld ln ls >> mkdir mksquashfs mktemp mv od patch readlink rm sed sh >> tail tar touch tr uniq wget whoami xargs > > As the author of the i486 target, I'll certainly be trying it there. I'm running the Linux From Scratch build under busybox defconfig with the record-commands.sh wrapping. (My first attempt died because the $PATH _only_ had the wrapper directory without falling back to /bin:/sbin meaning the new commands like m4 it built and added weren't in the $PATH, so the build died when it needed m4. Oops.) Basically I need to figure out what the full build _needs_, and then do a baseconfig-busybox with that, and then target it for replacement with toybox. (I'm working on a new "TOYBOX=" config entry you can set to busybox_defconfig, busybox, or toybox.) I'm vaguely aiming to get a release out around the 15th with sparc working, unless qemu doesn't cooperate. (The sparc command line has a length limit, it's not enforced by anything it's just openbios dies if you go over it. Rebuilding openbios is fiddly and not something end users can do, and pushing a fix upstream would go _where_ exactly...?) Rob 1323174710.0 From rob at landley.net Tue Dec 6 14:25:11 2011 From: rob at landley.net (Rob Landley) Date: Tue, 06 Dec 2011 08:25:11 -0600 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: References: <20111123151627.7e79c691.onefang@gmail.com>, <20111123161610.13b6f007.onefang@gmail.com>, <20111123164529.6ec76a89.onefang@gmail.com>, <20111123174854.73b5e14b.onefang@gmail.com>, <20111124021351.77614762.onefang@gmail.com> , <4ED61C00.1060304@landley.net> , <4ED8D3F2.9020907@landley.net> Message-ID: <4EDE25C7.7090800@landley.net> On 12/02/2011 08:24 PM, James McMechan wrote: >>> On a side note, I noticed busybox defconfig even has a webserver so >>> you could add your own private package mirror to dev-environment.sh >> >> I could. But I prefer to leave that sort of thing to other people: I'm >> intentionally agnostic about what you _do_ with the resulting system. >> The hdc infrastructure lets you make self contained build images, but my >> examples include one that just runs the busybox test suite. > > Err I was thinking of having a local package mirror that could be setup > automatically when running dev-enviornment.sh > after all when you have just built this version you know where a valid > set of packages are. also it is somewhat humorous that the minimal busybox > includes a webserver... The "packages" directory is just enough to re-run the aboriginal linux build, which isn't actually a very interesting thing to do under the target. Yeah, it's great that busybox has a webserver, but what do you want to export? That's the problem: dev-environment.sh has no idea what you want to DO on the target, it's intentionally agnostic. And if we're going to set up a server, it might as well have upload _and_ download capability, hence the ftpd with ftpsend and such that the existing hdc images are using. >>> mdev from busybox takes care of the rest of the device files later. >>> The new DEVTMPFS in the kernel might make even that unneeded >>> I have not tested without /dev/console yet though... >> >> Doesn't devtmpfs give you /dev/console? > > I have not yet tried a devtmpfs only dev it should work but does not > automount in initramfs if I am reading the documentation correctly > some early parts of userspace used to break without /dev/console The existing stuff has been using it for a while, and it's working for me. I believe I mount it from the init script and provide a /dev/console in the existing root filesystem, though. >>>>> #needed for od -t option in busybox build >>>>> CONFIG_DESKTOP=y >>>> >>>> I've argued with Denys about that. CONFIG_DESKTOP is not well defined, >>>> there's no clear rule for what it does and doesn't do. >>>> >>>> This config is enough to build aboriginal linux itself, with this in >>>> hostconfig? >>> >>> It was working earlier, last tested about 1.0.2, I need to check with the new >>> version. >>> >>>> A) Does that include running the ./download.sh stage? >>> >>> download.sh does not use the host tools path and breaks at the moment >> >> It will if you run host-tools.sh, zap packages, and then re-run >> ./download.sh. > > you sure it looked like it overrode the path back to OLDPATH? Only for do_manifest, which is an optional step. (Which can call "git" and "hg" to get version numbers.) Let's see, by manual inspection do_manifest can call: hg, sed, cat, echo, zcat, bzcat, git, svn. But fundamentally it isn't interesting, it's a documentation step that can happily fail if you haven't got the appropriate source control plumbing installed to query version numbers of development packages. (Why/how you'd be using development snapshots without the appropriate stuff installed, I have no idea, but it shouldn't barf on it.) >> Busybox always gets unhappy about those when perl isn't installed, but >> it's not a fatal error. >> >>> leaves a statically linked busybox binary in host-tools-i686 >> >> host-tools-i686? > > Yep, imore/test.sh will not accept a arch of host so I was using i686 > and it puts stuff in $STAGE_NAME-$ARCH Ah. Ok. Fixed: say "host" and it'll skip the load_target. >> Um, host-tools.sh is target agnostic, it imports sources/include.sh but >> never calls load_target. I'm confused? > > I was trying out different mini-configs for busybox in a manual fashion > Um-- I think of host as the first target the one built with the host toolchain > kernel headers and host C library. Host isn't really a target. It doesn't work like the other targets, and the design assumptions are different. >>> It would be somewhat easier if host acted like a real target. >> >> It's sort of the point of the host stuff is to be target agnostic. You >> should be able to build all the targets under the same host environment. >> (The fact that different releases of Ubuntu and Fedora and such don't >> _provide_ the same host environment is why host-tools.sh exists, but the >> point stands.) > > Err, building the aboriginal host tool set was the target I was referring to. So you want me to build a toolchain for the host before building packages for the host? How? With what? (The cross-compiler.sh step has to cope with this anyway to get a properly portable toolchain, which is why you have to specify the CROSS_COMPILER_HOST for that. But that simple-cross-compiler is built with the actual host tool chain. Can't escape it at some level, and adding more layers of indirection makes things _less_ flexible if you do it wrong.) >>> The current special case logic seem to get in my way as much as help >>> when trying repeated host-tools setups. >> >> Which special case logic? > > whether busybox is linked with the host libc or uClibc from aboriginal > for example. export BUILD_STATIC=none # Set this to a comma separated list of packages to build statically, # or "none" to build all packages dynamically. Set to "all" to build # all packages statically (and not install static libraries on the # target). # By default, busybox and the native compiler are built statically. # (Using a static busybox on the target provides a 20% performance # boost to autoconf under qemu, and building the native compiler static # makes it much more portable to other target root filesystems.) # export BUILD_STATIC=busybox,binutils,gcc-core,gcc-g++,make The fact that you CAN'T build it statically for the host is due to a bug in glibc, as described at length here: http://elinux.org/images/2/2d/ELC2010-gc-sections_Denys_Vlasenko.pdf http://free-electrons.com/pub/video/2010/elc/elc2010-vlasenko-dead-code.ogv The reason the glibc bug never got fixed (last I checked) is that glibc maintainer Ulrich Drepper went crazy and deprecated static linking entirely: http://www.akkadia.org/drepper/no_static_linking.html >> http://landley.net/aboriginal/FAQ.html#debug_test >> >> Its "usage:" example suggests setting STAGE_NAME so in this case you >> probably want: >> >> STAGE_NAME=simple-cross-compiler more/test.sh sparc build_section uClibc >> >> I should probably put that example in the FAQ... Done. >> >>> I quickly commented them out, but it would >>> seem that having each package saved in the build directory not just >>> the assembled meta-packages would be nice. >> >> Um, do you mean like the config option BINARY_PACKAGE_TARBALLS? > > Looks like it, does it remember the tarballs and not build them again when present? > I will have to check this option. No, that's handled at another level. The "build.sh" script checks for the stage tarballs, and won't rebuild them if they already exist. (Except that if has to rebuild a prerequisite stage, it deletes tarballs that would depend on that, so it rebuilds those dependent stages when it gets to them.) You can force it to rebuild even if the tarball is there by naming the stage on your command line. So you can go: ./build.sh i686 native-compiler And it should rebuild native-compiler, and go on to rebuild root-filesystem.sh, root-image.sh, and system-image.sh. (But it shouldn't have to rebuild simple-root-filesystem.sh because that doesn't depend on native-compiler.sh.) It'll redo the binary package tarballs whenever it calls build_section: it rebuilt the binaries, it rebuilds the tarball. Rob 1323181511.0 From rob at landley.net Wed Dec 7 04:32:05 2011 From: rob at landley.net (Rob Landley) Date: Tue, 06 Dec 2011 22:32:05 -0600 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: <4EDD0CEC.3070100@landley.net> References: <20111123151627.7e79c691.onefang@gmail.com>, <20111123161610.13b6f007.onefang@gmail.com>, <20111123164529.6ec76a89.onefang@gmail.com>, <20111123174854.73b5e14b.onefang@gmail.com>, <20111124021351.77614762.onefang@gmail.com> , <4ED61C00.1060304@landley.net> , <4ED8D3F2.9020907@landley.net> <4EDD0CEC.3070100@landley.net> Message-ID: <4EDEEC45.5080108@landley.net> On 12/05/2011 12:26 PM, Rob Landley wrote: > I'm working on it. Just did a blog entry about what record-commands > says I need for the build. (Might not get to publishing it until > tomorrow, needs some cleanup.) > > It boils down to me needing to add this to toybox in order for it to be > able to replace busybox in aboriginal linux (at least for the i586 > build, I need to check other targets, and I'm not quite sure what LFS > needs...): > > awk basename bzip2 chmod chown cmp cut date dd diff dirname > egrep env expr find grep gzip head hostname id install ld ln ls > mkdir mksquashfs mktemp mv od patch readlink rm sed sh > tail tar touch tr uniq wget whoami xargs > > About half of those are low-hanging fruit anyway... I added the record-commands plumbing to the target, and here's what LFS needed: ar awk basename bash cat cc chgrp chmod cmp comm cp cpp date diff dirname echo egrep expr find g++ gcc grep gunzip gzip hostname id install ld ldconfig less ln logname ls make man mkdir mktemp mt mv nm objdump od ranlib readlink rm rmdir sed sh size sleep sort split strip sync tail tar tee test time touch tr true uname uniq wc which Subtracting the build tools (ar, cc, cpp, g++, gcc, ld, make, nm, objdump, ranlib, and strip) and the stuff in the first list, we get: bash chgrp comm gunzip ldconfig less logname man mt size split sync tee test time (Don't ask me what the "mt" command is doing in there. There's no earthly way the build NEEDS it, this is presumably more random "./configure runs crap". The "man" command isn't a priority either. I'm surprised to see bash called out of $PATH instead of via #!/blah. And test is a shell builtin, more or less. The rest need doing, though.) Still, I believe that gives me the list of things I need to make sure your baseconfig-busybox has, and then I can re-run the build and LFS builds and see if anything complains. Rob 1323232325.0 From james_mcmechan at hotmail.com Fri Dec 9 02:34:07 2011 From: james_mcmechan at hotmail.com (James McMechan) Date: Thu, 8 Dec 2011 18:34:07 -0800 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: <4EDD0CEC.3070100@landley.net> References: <20111123151627.7e79c691.onefang@gmail.com>, <20111123161610.13b6f007.onefang@gmail.com>, <20111123164529.6ec76a89.onefang@gmail.com>, <20111123174854.73b5e14b.onefang@gmail.com>, <20111124021351.77614762.onefang@gmail.com> , <4ED61C00.1060304@landley.net> , <4ED8D3F2.9020907@landley.net> , <4EDD0CEC.3070100@landley.net> Message-ID: > Date: Mon, 5 Dec 2011 12:26:52 -0600 > On 12/02/2011 08:24 PM, James McMechan wrote: > >> Yeah, but should it expect to find them in $SOURCES (where all the other > >> files would be checked into source control, but these wouldn't) or at a > >> location specified in config? The overlay mechanism for extra root > >> filesystem files is config based for the same reason, mixing tracked and > >> untracked files in the same directory is untidy. > > > > if you had a config directory I would have aimed for there, > > Hmmm... I don't want to have too _much_ configuration. A hammer isn't > a better hammer if it's got 37 switches you have to flip before pounding > nails. Well yes, but I also have half a dozen different hammers in the tool box next to the computer so since they can't be configured I need multiples > I had the target directories (which I eventually cleaned up into > individual target files for most of them), because each target is > somewhat different and I wanted a single place to collect all those > differences. > > The stuff in "sources" is the stuff that's the same for all targets. > I added an "overlay" capability so people didn't have to extensively > fiddle with sources/root-filesystem. > > > SOURCES is where the trimconfig used to reside. > > Also I would expect that mini-config & company > > would be checked in. > > Back then trimconfig was "what all targets needed". It belonged in > sources for the same reason the kernel and uClibc baseconfigs do: it was > generic. I thought of it as this stuff breaks on at least one target > Then I got enough changes pushed upstream into busybox that defconfig > could work for all targets, and from the point of view of _this_ project > using defconfig was simpler than specifying a config that could get out > of sync with what upstream was doing. (Which is inherent in trimconfig: > a delta against upstream.) > > That said, a busybox baseconfig makes sense from a design standpoint. > The minimal native development environment capable of rebuilding itself > under itself: stuff like linux from scratch has to bootstrap its way up > (possibly by building busybox defconfig first)... In general I dislike negative options such as NO_CPLUSPLUS and would much rather have a CPLUSPLUS=1 in the config file by default which would then be commented out to turn off C++ This leads to my strong preference for the additive mini-config, it is much easier for me to think about sections added together to build everything desired, e.g. first part has the options needed to build aboriginal the second has what is needed to run under qemu, say a third for LFS build requirements it could be a single text file with commented sections or a series of separate files catted together to provide all the desired features. That way if I want to build gentoo instead of LFS I would just change that section and avoid cluttering the environment with low usage commands e.g. why exactly when building a minimal system do I need ifenslave if I need that I would turn it on later. > >> Long-term I want to move to toybox and just not HAVE buckets of > >> extraneous crap needing to be disabled... > > > > It would be very nice. > > I'm working on it. Just did a blog entry about what record-commands > says I need for the build. (Might not get to publishing it until > tomorrow, needs some cleanup.) > > It boils down to me needing to add this to toybox in order for it to be > able to replace busybox in aboriginal linux (at least for the i586 > build, I need to check other targets, and I'm not quite sure what LFS > needs...): > > awk basename bzip2 chmod chown cmp cut date dd diff dirname > egrep env expr find grep gzip head hostname id install ld ln ls > mkdir mksquashfs mktemp mv od patch readlink rm sed sh > tail tar touch tr uniq wget whoami xargs > > About half of those are low-hanging fruit anyway... > > >>> there are prettier layouts though > >>> done=0 > >>> [ $done -eq 0 -a -e "${SOURCES}/mini-busybox" ] && make allnoconfig KCONFIG_ALLCONFIG="${SOURCES}/mini-busybox" && done = 1 > >>> [ $done -eq 0 -a -e "${SOURCES}/trim-busybox" ] && make allyesconfig KCONFIG_ALLCONFIG="${SOURCES}/trim-busybox" && done = 1 > >>> [ $done -eq 0 ] && make defconfig && done = 1 > >>> [ $done -ne 0 ] && \ > >> > >> Probably an if/else staircase. That's what they're for. > > > > I don't remember if classic shell had else, > > Yup, SUSv3 and SUSv4. I appear to be a old fogey, I still remember when we upgraded to csh and later ksh and the upgrade guide for X10 to X11, so my standards of what is a "old" shell are a bit out of date. > > I would still like it able to run under > > ash or other shells from the days of chipped flint systems. > > The current aboriginal linux scripts don't. They use > blah/{curly,bracket} wildcards and <(this kind) of pipe where the > command gets a filename it can read from to get the output of a command > (and there's a >(version) that's a file that outputs _to_ that command), > so you can do: > > diff -u <(blah | sort -u) < (blah2 | sort -u) > > And so on... > > Probably other stuff. I intend to write a shell that can cope with all > this, not strip down my script to be cripped for the Defective Annoying > SHell in bash. bourne shell was small and simple, bash is um... less small and less simple dash is a odd duck, I never quite understood its purpose. looking at "man sh" it appears feature creep has been busy shell history, vi line editing mode and others have wandered into the shell > Remember: I didn't remove the -i option from uses of sed, I extended > busybox sed to cope with -i. I haven't seriously banged on busybox in > years, but now I'm back throwing scraps of spare time at toybox. > > >>>> I'm also back to working on toybox, which means testing toybox, which > >>>> means getting toybox and busybox to install alongside each other and > >>>> gradually reduce the number of busybox commands I'm using. > >>> > >>> Personally, I like building up from the bottom, rather than trying to trim > >>> the fat from the top... > >> > >> When you're migrating a system that currently works by replacing > >> components, you have to build the new thing while dismantling the old, > >> so you wind up doing both... > > > > I was thinking that if you don't have the extra 100 commands from busybox > > the failures might be shorter. e.g. turn off od in busybox turn on in toybox > > test and repeat > > I was pretty much planning on doing that anyway. :) > > Although there's a cheap and easy way to do this: install toybox first, > then have the busybox installer simply fail to create symlinks when they > already exist. or just have the toybox installer not overwrite the file behind a symlink when it exists already and create a new symlink instead > This is a long email, I'll have to finish replying to it later... > > Rob 1323398047.0 From rob at landley.net Mon Dec 12 18:17:54 2011 From: rob at landley.net (Rob Landley) Date: Mon, 12 Dec 2011 12:17:54 -0600 Subject: [Aboriginal] miniconfig-busybox In-Reply-To: References: <20111123151627.7e79c691.onefang@gmail.com>, <20111123161610.13b6f007.onefang@gmail.com>, <20111123164529.6ec76a89.onefang@gmail.com>, <20111123174854.73b5e14b.onefang@gmail.com>, <20111124021351.77614762.onefang@gmail.com> , <4ED61C00.1060304@landley.net> , <4ED8D3F2.9020907@landley.net> , <4EDD0CEC.3070100@landley.net> Message-ID: <4EE64552.60005@landley.net> Sorry for the delay in answering this, I'm trying to extend your baseconfig into something that's got everything _I_ need switched on (including for download.sh and the LFS build), and then I'm checking it in as the new default. I can then do a "build busybox defconfig" as a native-build hdc image. So far I have: 1) Your experimentally determined list 2) the record-commands lists for aboriginal and LFS builds. 3) The list of commands in the SUSv4 utilities page. I'm thinking some combination of these three still removes around 100 commands from busybox. On 12/08/2011 08:34 PM, James McMechan wrote: >> Hmmm... I don't want to have too _much_ configuration. A hammer isn't >> a better hammer if it's got 37 switches you have to flip before pounding >> nails. > > Well yes, but I also have half a dozen different hammers in the tool box next > to the computer so since they can't be configured I need multiples "Do one thing and do it well." I'm trying to remember if that quote is from Ken Thompson or Brian Kernighan. It's one of the original unix guys, anyway... >> I had the target directories (which I eventually cleaned up into >> individual target files for most of them), because each target is >> somewhat different and I wanted a single place to collect all those >> differences. >> >> The stuff in "sources" is the stuff that's the same for all targets. >> I added an "overlay" capability so people didn't have to extensively >> fiddle with sources/root-filesystem. >> >>> SOURCES is where the trimconfig used to reside. >>> Also I would expect that mini-config & company >>> would be checked in. >> >> Back then trimconfig was "what all targets needed". It belonged in >> sources for the same reason the kernel and uClibc baseconfigs do: it was >> generic. > > I thought of it as this stuff breaks on at least one target Trimconfig itself contained stuff that broke on at least one target and thus needed to be removed for busybox to build the same way on all targets. What the trimconfig _produced_ was a defconfig that worked on all targets. I then got defconfig itself to work on all targets, but it keeps bloating and breaking... >> Then I got enough changes pushed upstream into busybox that defconfig >> could work for all targets, and from the point of view of _this_ project >> using defconfig was simpler than specifying a config that could get out >> of sync with what upstream was doing. (Which is inherent in trimconfig: >> a delta against upstream.) >> >> That said, a busybox baseconfig makes sense from a design standpoint. >> The minimal native development environment capable of rebuilding itself >> under itself: stuff like linux from scratch has to bootstrap its way up >> (possibly by building busybox defconfig first)... > > In general I dislike negative options such as NO_CPLUSPLUS and would > much rather have a CPLUSPLUS=1 in the config file by default > which would then be commented out to turn off C++ I see your point, but each config option is at its default value when it's blank, thus to set config back to its default values comment everything out. This means that anything that defaults to "on" but which can be removed needs a negative option. (I'm not saying that's a great way to do it, I'm saying that's the reasonably consistent rationale behind the way it currently is.) > This leads to my strong preference for the additive mini-config, it is much > easier for me to think about sections added together to build everything desired, > e.g. first part has the options needed to build aboriginal the second has > what is needed to run under qemu, say a third for LFS build requirements > it could be a single text file with commented sections or a series of separate files > catted together to provide all the desired features. If you're creating a build environment on target, LFS is a good (but not perfect) high water mark (since if all else fails, you can recreate LFS). The defconfig was "everything busybox can possibly provide that might help", which was a defensible design decision but brittle as far as building it goes and unstable because upstream keeps adding stuff. The problem with "the absolute minimum set of commands we need" is it's a bit brittle the other way (we didn't know we needed X), but that just means we need a lot of testing, and that's what "build each target through to the end of LFS" is for... If you're creating some other kind of target system, I intentionally have no opinion on _what_ you'll need in it, it may just be three commands. > That way if I want to build gentoo instead of LFS I would just change > that section and avoid cluttering the environment with low usage commands > e.g. why exactly when building a minimal system do I need ifenslave > if I need that I would turn it on later. LFS builds m4 before building things that need m4. If you can do that natively, you don't need more stuff from busybox. If you CAN'T do that natively, the build environment we're producing is defective. In theory, you can build a defconfig busybox on target as part of your bootstrapping. I'm not personally interested in having a bunch of collated config chunks for special-purpose stuff (although it's trivial to add on top of the infrastructure we're providing). >>>> Probably an if/else staircase. That's what they're for. >>> >>> I don't remember if classic shell had else, >> >> Yup, SUSv3 and SUSv4. > > I appear to be a old fogey, I still remember when we upgraded to csh and later ksh > and the upgrade guide for X10 to X11, so my standards of what is a "old" shell > are a bit out of date. I'm providing a C99 compiler and a 3.x kernel. I'm not providing a 2.0 kernel, a c89 compiler, and libc5. SUSv4 isn't perfect (_why_ does it still include "ed" and "sccs"?) but it's reasonably straightforward to support, and there are no obvious alternatives: the same standard got approved as IEEE posix, open group base specifications, and single unix standard, so I can shut 'em all up with one pass. >> Probably other stuff. I intend to write a shell that can cope with all >> this, not strip down my script to be cripped for the Defective Annoying >> SHell in bash. > > > bourne shell was small and simple, bash is um... It's from the gnu/dammit project. I intend to replace it, just not this week. In the meantime, sticking with an old version reverses a bunch of gnu "upgrades" that make it bigger and slower for no obvious reason. > less small and less simple > dash is a odd duck, I never quite understood its purpose. Defective, Annoying SHell. > looking at "man sh" it appears feature creep has been busy Shell command line options: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html Shell scripts: http://pubs.opengroup.org/onlinepubs/9699919799/idx/shell.html Plus I did a lot of random research about what shells do and need, searching my blog for toysh finds lots of things like: http://landley.net/notes-2007.html#19-10-2007 But tackling the shell comes after filling in the toybox command list rather a lot... > shell history, vi line editing mode and others have wandered into the shell 1) If you build bash 2.05b on a system that hasn't got curses, it won't include history or command line editing. (Which is kinda painful.) 2) DOS had shell history (doskey) so this predates the transition from 16-bit to 32-bit hardware. 3) we have cursor keys now, along with home/end, del and backspace, and page up and down. All standardized for about 30 years. I'm happy to rely on them being there. >>> I was thinking that if you don't have the extra 100 commands from busybox >>> the failures might be shorter. e.g. turn off od in busybox turn on in toybox >>> test and repeat >> >> I was pretty much planning on doing that anyway. :) >> >> Although there's a cheap and easy way to do this: install toybox first, >> then have the busybox installer simply fail to create symlinks when they >> already exist. > > or just have the toybox installer not overwrite the file behind a symlink > when it exists already and create a new symlink instead It's "ln -sf" either way. One of the toybox ideas I managed to shove upstream is getting the darn project to list the links to create, so you can do: for i in $(busybox --list); do ln -sf busybox $i; done The toybox --list gives paths, which I can chop out with sed but I think a non-path version would be good to. It's a todo item... Rob 1323713874.0 From rob at landley.net Wed Dec 14 02:23:13 2011 From: rob at landley.net (Rob Landley) Date: Tue, 13 Dec 2011 20:23:13 -0600 Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <20111211111801.5942ab39@sf.home> References: <4EBEAB5A.5020809@landley.net> <4EBF0A25.20604@landley.net> <20111211111801.5942ab39@sf.home> Message-ID: <4EE80891.70604@landley.net> On 12/11/2011 02:17 AM, Sergei Trofimovich wrote: >>> Boot time fixup v1.6. 4/Mar/98 Jakub Jelinek (jj at ultra.linux.cz). >>> Patching kernel for srmmu[Fujitsu TurboSparc]/iommu >>> Fixup i f029ddfc doesn't refer to a valid instruction at >>> f00de648[95eea000] >>> halt, power off >> >> If anyone would like a reproduction sequence for this, the toolchain I use >> (which compiles commit 9933fc0ac1ac14b^1 just fine) is at: >> >> http://landley.net/aboriginal/downloads/binaries/cross-compiler-sparc.tar.bz2 > > Tried to build both with your crosscompiler and with gcc-4.5.3/binutils-2.21.1 > Both boot but don't get to prompt (minimal config I guess). > (v3.1 works the same as 3.2.0-rc5) Not for me it doesn't. The bug goes away if I even upgrade to 3.1.2, it's very precise and fiddly in a 'stars align' sort of way. It's building vanilla 3.1 with the attached .config, and doesn't happen if I change the .config too much. > Linux version 3.2.0-rc5 (slyfox at sf) (gcc version 4.2.1) #1 Sun Dec 11 11:08:25 FET 2011 > bootconsole [earlyprom0] enabled > ARCH: SUN4M > TYPE: SPARCstation 5 > Ethernet address: 52:54:00:12:34:56 > Boot time fixup v1.6. 4/Mar/98 Jakub Jelinek (jj at ultra.linux.cz). Patching kernel for srmmu[Fujitsu TurboSparc]/iommu > OF stdout device is: /obio/zs at 0,100000:a > PROM: Built device tree with 23073 bytes of memory. > Booting Linux... > Power off control detected. > Built 1 zonelists in Zone order, mobility grouping on. Total pages: 28018 > ... > mousedev: PS/2 mouse device common for all mice > rtc-m48t59 rtc-m48t59.0: rtc core: registered m48t08 as rtc0 > turn off boot console earlyprom0 > > Can you share broken image to drive out qemu bug? (0.14.0 is here) I don't think it is a qemu bug. I marked up the fixup code with buckets of printk() and it didn't change anything, my understanding is it's btfixupprep.c producing bad output. But so far I haven't made any sense out of btfix.S and how it corresponds to the addresses the fixup code is working on. The address it's complaining about does not actually correspond to any of the symbols in the "objdump -x vmlinux" output, although they are between the ext4 symbols listed there... I put the broken image up at http://landley.net/sparc-image for the moment, but if you build 3.1 with the attached .config and the toolchain mentioned last time, it should reproduce for you. It's 100% reliable for me... Thanks, Rob -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: sparc-config URL: From slyich at gmail.com Sun Dec 11 08:17:57 2011 From: slyich at gmail.com (Sergei Trofimovich) Date: Sun, 11 Dec 2011 11:17:57 +0300 Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <4EBF0A25.20604@landley.net> References: <4EBEAB5A.5020809@landley.net> <4EBF0A25.20604@landley.net> Message-ID: <20111211111801.5942ab39@sf.home> > > Boot time fixup v1.6. 4/Mar/98 Jakub Jelinek (jj at ultra.linux.cz). > > Patching kernel for srmmu[Fujitsu TurboSparc]/iommu > > Fixup i f029ddfc doesn't refer to a valid instruction at > > f00de648[95eea000] > > halt, power off > > If anyone would like a reproduction sequence for this, the toolchain I use > (which compiles commit 9933fc0ac1ac14b^1 just fine) is at: > > http://landley.net/aboriginal/downloads/binaries/cross-compiler-sparc.tar.bz2 Tried to build both with your crosscompiler and with gcc-4.5.3/binutils-2.21.1 Both boot but don't get to prompt (minimal config I guess). (v3.1 works the same as 3.2.0-rc5) Linux version 3.2.0-rc5 (slyfox at sf) (gcc version 4.2.1) #1 Sun Dec 11 11:08:25 FET 2011 bootconsole [earlyprom0] enabled ARCH: SUN4M TYPE: SPARCstation 5 Ethernet address: 52:54:00:12:34:56 Boot time fixup v1.6. 4/Mar/98 Jakub Jelinek (jj at ultra.linux.cz). Patching kernel for srmmu[Fujitsu TurboSparc]/iommu OF stdout device is: /obio/zs at 0,100000:a PROM: Built device tree with 23073 bytes of memory. Booting Linux... Power off control detected. Built 1 zonelists in Zone order, mobility grouping on. Total pages: 28018 ... mousedev: PS/2 mouse device common for all mice rtc-m48t59 rtc-m48t59.0: rtc core: registered m48t08 as rtc0 turn off boot console earlyprom0 Can you share broken image to drive out qemu bug? (0.14.0 is here) -- Sergei -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From slyich at gmail.com Wed Dec 14 17:42:25 2011 From: slyich at gmail.com (Sergei Trofimovich) Date: Wed, 14 Dec 2011 20:42:25 +0300 Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <4EE80891.70604@landley.net> References: <4EBEAB5A.5020809@landley.net> <4EBF0A25.20604@landley.net> <20111211111801.5942ab39@sf.home> <4EE80891.70604@landley.net> Message-ID: <20111214204225.7c8ec86c@sf.home> [ CCed Jakub ] >>> Boot time fixup v1.6. 4/Mar/98 Jakub Jelinek (jj at ultra.linux.cz). >>> Patching kernel for srmmu[Fujitsu TurboSparc]/iommu >>> Fixup i f029ddfc doesn't refer to a valid instruction at >>> f00de648[95eea000] >>> halt, power off > I put the broken image up at http://landley.net/sparc-image for the > moment, but if you build 3.1 with the attached .config and the toolchain > mentioned last time, it should reproduce for you. It's 100% reliable > for me... Nice! With this config it breaks for me on your and mine toolchains. The offending function is ext4_kvmalloc (and similar ext4_kvzalloc). The usual relocation in sparc looks like a pair of instructions loading two pats of address in 2 instructions: Like that: > sethi %hi(ext4_fill_super), %o4 !, tmp113 > or %o4, %lo(ext4_fill_super), %o4 ! tmp113,, tmp28 In our case relocatable symbol sits in tail call, so %lo part is in "unusual" RESTORE instruction: > ext4_kvmalloc: ... > sethi %hi(___i_page_kernel), %i2 !, tmp112 > call __vmalloc, 0 ! > restore %i2, %lo(___i_page_kernel), %o2 ! tmp112,, ... David: is this code correct? Or it's a compiler bug? I am sparc32 newbie. (C source and asm sources of function are in [1]) I think this kind of code is generated only in -Os. So to workaround it I tried this hack: > --- a/fs/ext4/super.c > +++ b/fs/ext4/super.c > ret = kmalloc(size, flags); > if (!ret) > ret = __vmalloc(size, flags, PAGE_KERNEL); > + > + asm __volatile__("nop":::"memory"); > + > return ret; > } (for both ext4_kvmalloc / ext4_kvzalloc. Attached workaround as a patch.) It forces compiler to geterate "usual" pattern for relocation. I think of 2 solutions: 1. trying to fix sparc/boot/btfixupprep.c and arch/sparc/mm/btfixup.c to distinct HI22 and LO10 relocations as different ones. Right now they are merged into one 'i' type and rely on instruction heuristics to fix it. 2. Add a hack to arch/sparc/mm/btfixup.c to recognize restore instruction as well Any others? Hope that helps. [1]: void *ext4_kvmalloc(size_t size, gfp_t flags) { void *ret; ret = kmalloc(size, flags); if (!ret) ret = __vmalloc(size, flags, PAGE_KERNEL); return ret; } .global ext4_kvmalloc .type ext4_kvmalloc, #function .proc 0120 ext4_kvmalloc: save %sp, -96, %sp ! mov %i0, %o0 ! size, call __kmalloc, 0 !, mov %i1, %o1 ! flags, cmp %o0, 0 ! ret, bne .LL274 ! sethi %hi(___i_page_kernel), %i2 !, tmp112 call __vmalloc, 0 ! restore %i2, %lo(___i_page_kernel), %o2 ! tmp112,, .LL274: jmp %i7+8 restore %g0, %o0, %o0 ! ret, .size ext4_kvmalloc, .-ext4_kvmalloc -- Sergei -------------- next part -------------- A non-text attachment was scrubbed... Name: pessimizing-hack.patch Type: text/x-patch Size: 554 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From eric.dumazet at gmail.com Wed Dec 14 17:49:55 2011 From: eric.dumazet at gmail.com (Eric Dumazet) Date: Wed, 14 Dec 2011 18:49:55 +0100 Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <20111214204225.7c8ec86c@sf.home> References: <4EBEAB5A.5020809@landley.net> <4EBF0A25.20604@landley.net> <20111211111801.5942ab39@sf.home> <4EE80891.70604@landley.net> <20111214204225.7c8ec86c@sf.home> Message-ID: <1323884995.2334.71.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> Le mercredi 14 d?cembre 2011 ? 20:42 +0300, Sergei Trofimovich a ?crit : > [ CCed Jakub ] > > >>> Boot time fixup v1.6. 4/Mar/98 Jakub Jelinek (jj at ultra.linux.cz). > >>> Patching kernel for srmmu[Fujitsu TurboSparc]/iommu > >>> Fixup i f029ddfc doesn't refer to a valid instruction at > >>> f00de648[95eea000] > >>> halt, power off > > > I put the broken image up at http://landley.net/sparc-image for the > > moment, but if you build 3.1 with the attached .config and the toolchain > > mentioned last time, it should reproduce for you. It's 100% reliable > > for me... > > Nice! With this config it breaks for me on your and mine toolchains. > The offending function is ext4_kvmalloc (and similar ext4_kvzalloc). > > The usual relocation in sparc looks like a pair of instructions loading > two pats of address in 2 instructions: > > Like that: > > sethi %hi(ext4_fill_super), %o4 !, tmp113 > > or %o4, %lo(ext4_fill_super), %o4 ! tmp113,, tmp28 > > In our case relocatable symbol sits in tail call, so %lo part is in "unusual" > RESTORE instruction: > > > ext4_kvmalloc: > ... > > sethi %hi(___i_page_kernel), %i2 !, tmp112 > > call __vmalloc, 0 ! > > restore %i2, %lo(___i_page_kernel), %o2 ! tmp112,, > ... > > David: is this code correct? Or it's a compiler bug? I am sparc32 newbie. > (C source and asm sources of function are in [1]) > > I think this kind of code is generated only in -Os. > So to workaround it I tried this hack: > > > --- a/fs/ext4/super.c > > +++ b/fs/ext4/super.c > > ret = kmalloc(size, flags); > > if (!ret) > > ret = __vmalloc(size, flags, PAGE_KERNEL); > > + > > + asm __volatile__("nop":::"memory"); > > + > > return ret; > > } > (for both ext4_kvmalloc / ext4_kvzalloc. Attached workaround as a patch.) > > It forces compiler to geterate "usual" pattern for relocation. > > I think of 2 solutions: > > 1. trying to fix sparc/boot/btfixupprep.c and arch/sparc/mm/btfixup.c > to distinct HI22 and LO10 relocations as different ones. > Right now they are merged into one 'i' type and rely on instruction heuristics to fix it. > 2. Add a hack to arch/sparc/mm/btfixup.c to recognize restore instruction as well > > Any others? 3) Adding a memset() in ext4_kvmalloc() and ext4_kvzalloc() to prefault pages ? 4) (Unrelated) : add __GFP_HIGHMEM to __vmalloc() flags, so that high memory pages can be used for large allocations. 1323884995.0 From davem at davemloft.net Wed Dec 14 17:53:55 2011 From: davem at davemloft.net (David Miller) Date: Wed, 14 Dec 2011 12:53:55 -0500 (EST) Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <20111214204225.7c8ec86c@sf.home> References: <20111211111801.5942ab39@sf.home> <4EE80891.70604@landley.net> <20111214204225.7c8ec86c@sf.home> Message-ID: <20111214.125355.2114905092680399303.davem@davemloft.net> From: Sergei Trofimovich Date: Wed, 14 Dec 2011 20:42:25 +0300 >> ext4_kvmalloc: > ... >> sethi %hi(___i_page_kernel), %i2 !, tmp112 >> call __vmalloc, 0 ! >> restore %i2, %lo(___i_page_kernel), %o2 ! tmp112,, > ... > > David: is this code correct? Or it's a compiler bug? I am sparc32 newbie. > (C source and asm sources of function are in [1]) It is correct, it's a tail call. 1323885235.0 From davem at davemloft.net Wed Dec 14 17:54:43 2011 From: davem at davemloft.net (David Miller) Date: Wed, 14 Dec 2011 12:54:43 -0500 (EST) Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <20111214204225.7c8ec86c@sf.home> References: <20111211111801.5942ab39@sf.home> <4EE80891.70604@landley.net> <20111214204225.7c8ec86c@sf.home> Message-ID: <20111214.125443.973289484729064945.davem@davemloft.net> From: Sergei Trofimovich Date: Wed, 14 Dec 2011 20:42:25 +0300 > It forces compiler to geterate "usual" pattern for relocation. No, this is wrong. The right fix is to simply teach btfixup to be able to patch the 'restore' just as equally as it would patch an 'or'. 1323885283.0 From davem at davemloft.net Wed Dec 14 18:18:10 2011 From: davem at davemloft.net (David Miller) Date: Wed, 14 Dec 2011 13:18:10 -0500 (EST) Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <20111214.125443.973289484729064945.davem@davemloft.net> References: <4EE80891.70604@landley.net> <20111214204225.7c8ec86c@sf.home> <20111214.125443.973289484729064945.davem@davemloft.net> Message-ID: <20111214.131810.916491563666428853.davem@davemloft.net> From: David Miller Date: Wed, 14 Dec 2011 12:54:43 -0500 (EST) > The right fix is to simply teach btfixup to be able to patch > the 'restore' just as equally as it would patch an 'or'. Try this: -------------------- sparc32: Be less strict in matching %lo part of relocation. The "(insn & 0x01800000) != 0x01800000" test matches 'restore' but that is a legitimate place to see the %lo() part of a 32-bit symbol relocation, particularly in tail calls. Signed-off-by: David S. Miller --- arch/sparc/mm/btfixup.c | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/arch/sparc/mm/btfixup.c b/arch/sparc/mm/btfixup.c index 5175ac2..8a7f817 100644 --- a/arch/sparc/mm/btfixup.c +++ b/arch/sparc/mm/btfixup.c @@ -302,8 +302,7 @@ void __init btfixup(void) case 'i': /* INT */ if ((insn & 0xc1c00000) == 0x01000000) /* %HI */ set_addr(addr, q[1], fmangled, (insn & 0xffc00000) | (p[1] >> 10)); - else if ((insn & 0x80002000) == 0x80002000 && - (insn & 0x01800000) != 0x01800000) /* %LO */ + else if ((insn & 0x80002000) == 0x80002000) /* %LO */ set_addr(addr, q[1], fmangled, (insn & 0xffffe000) | (p[1] & 0x3ff)); else { prom_printf(insn_i, p, addr, insn); -- 1.7.6.401.g6a319 1323886690.0 From slyich at gmail.com Wed Dec 14 18:55:16 2011 From: slyich at gmail.com (Sergei Trofimovich) Date: Wed, 14 Dec 2011 21:55:16 +0300 Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <20111214.131810.916491563666428853.davem@davemloft.net> References: <4EE80891.70604@landley.net> <20111214204225.7c8ec86c@sf.home> <20111214.125443.973289484729064945.davem@davemloft.net> <20111214.131810.916491563666428853.davem@davemloft.net> Message-ID: <20111214215516.26ffe884@sf.home> > sparc32: Be less strict in matching %lo part of relocation. > > The "(insn & 0x01800000) != 0x01800000" test matches 'restore' > but that is a legitimate place to see the %lo() part of a 32-bit > symbol relocation, particularly in tail calls. > > Signed-off-by: David S. Miller Tested-by: Sergei Trofimovich Was about to send exactly the same. Thanks! -- Sergei -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From davem at davemloft.net Wed Dec 14 19:13:16 2011 From: davem at davemloft.net (David Miller) Date: Wed, 14 Dec 2011 14:13:16 -0500 (EST) Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <20111214215516.26ffe884@sf.home> References: <20111214.125443.973289484729064945.davem@davemloft.net> <20111214.131810.916491563666428853.davem@davemloft.net> <20111214215516.26ffe884@sf.home> Message-ID: <20111214.141316.931101347023790867.davem@davemloft.net> From: Sergei Trofimovich Date: Wed, 14 Dec 2011 21:55:16 +0300 >> sparc32: Be less strict in matching %lo part of relocation. >> >> The "(insn & 0x01800000) != 0x01800000" test matches 'restore' >> but that is a legitimate place to see the %lo() part of a 32-bit >> symbol relocation, particularly in tail calls. >> >> Signed-off-by: David S. Miller > > Tested-by: Sergei Trofimovich > > Was about to send exactly the same. Thanks! Thanks for testing Sergei. 1323889996.0 From rob at landley.net Wed Dec 14 19:26:19 2011 From: rob at landley.net (Rob Landley) Date: Wed, 14 Dec 2011 13:26:19 -0600 Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <20111214.131810.916491563666428853.davem@davemloft.net> References: <4EE80891.70604@landley.net> <20111214204225.7c8ec86c@sf.home> <20111214.125443.973289484729064945.davem@davemloft.net> <20111214.131810.916491563666428853.davem@davemloft.net> Message-ID: <4EE8F85B.9060209@landley.net> On 12/14/2011 12:18 PM, David Miller wrote: > From: David Miller > Date: Wed, 14 Dec 2011 12:54:43 -0500 (EST) > >> The right fix is to simply teach btfixup to be able to patch >> the 'restore' just as equally as it would patch an 'or'. > > Try this: > > -------------------- > sparc32: Be less strict in matching %lo part of relocation. Yay! With that patch, it booted to a shell prompt. Thanks, Rob 1323890779.0 From rob at landley.net Wed Dec 14 19:27:46 2011 From: rob at landley.net (Rob Landley) Date: Wed, 14 Dec 2011 13:27:46 -0600 Subject: [Aboriginal] Sparc-32 doesn't work in 3.1. In-Reply-To: <20111214.141316.931101347023790867.davem@davemloft.net> References: <20111214.125443.973289484729064945.davem@davemloft.net> <20111214.131810.916491563666428853.davem@davemloft.net> <20111214215516.26ffe884@sf.home> <20111214.141316.931101347023790867.davem@davemloft.net> Message-ID: <4EE8F8B2.3020203@landley.net> On 12/14/2011 01:13 PM, David Miller wrote: > From: Sergei Trofimovich > Date: Wed, 14 Dec 2011 21:55:16 +0300 > >>> sparc32: Be less strict in matching %lo part of relocation. >>> >>> The "(insn & 0x01800000) != 0x01800000" test matches 'restore' >>> but that is a legitimate place to see the %lo() part of a 32-bit >>> symbol relocation, particularly in tail calls. >>> >>> Signed-off-by: David S. Miller >> >> Tested-by: Sergei Trofimovich >> >> Was about to send exactly the same. Thanks! > > Thanks for testing Sergei. > Is that the line we're using now? I'm out of touch... Tested-by: Rob Landley Rob 1323890866.0 From rob at landley.net Sun Dec 18 20:53:16 2011 From: rob at landley.net (Rob Landley) Date: Sun, 18 Dec 2011 14:53:16 -0600 Subject: [Aboriginal] Weekend work... Message-ID: <4EEE52BC.40402@landley.net> Ok, you saw the sparc thread: that's mostly fixed now. (The command line length is still limited on sparc to an amount smaller than dev-environment.sh needs to pass through, which means the native build layered on top of that isn't working yet. This is either a qemu board emulation limitation or more likely an openbios limitation, but I'm not holding up the release for it.) I'm banging on James' baseconfig-busybox now, adding all sorts of stuff like ifconfig, route, and mountpoint that the boot needs but apparently aren't in there...? Plus he's using ash as the busybox shell and I've been using hush, with lotsa options switched on. And then I need to test the LFS build... Long term, I'd still like to replace busybox with toybox, because I care about toybox development and don't really care much about busybox development anymore. Restricting busybox to the apps we actually use are a logical step along the way to doing that, so I have a better todo list and test environment. Trying to get a release out in the next couple days... Rob 1324241596.0 From rob at landley.net Wed Dec 28 04:00:23 2011 From: rob at landley.net (Rob Landley) Date: Tue, 27 Dec 2011 22:00:23 -0600 Subject: [Aboriginal] Resurfacing... Message-ID: <4EFA9457.8040500@landley.net> Ok, it actually took a lot of doing to get a baseconfig-busybox that would actually build Linux From Scratch and didn't have OBVIOUS missing things (like vi and shell history) when I pulled up a shell prompt. I hit a fun little hitch that my LFS build scripts were depending on CONFIG_FEATURE_NON_POSIX_CP which is frankly disgusting so I fixed the build scripts instead. (It's the symlinks into a read only filesystem thing, breaking symlinks makes that work, but isn't how posix says CP should work so not how I plan to implement toybox... it's a thing.) That said, it's in now, and the commit comment has more details: http://landley.net/hg/aboriginal/rev/431d5b4ee537 Anybody care to give it a whirl? Rob 1325044823.0