From rob at landley.net Wed Jan 4 00:17:03 2012 From: rob at landley.net (Rob Landley) Date: Tue, 03 Jan 2012 18:17:03 -0600 Subject: [Aboriginal] Awaiting the 3.2 kernel release. Message-ID: <4F039A7F.1040908@landley.net> FYI, the 3.2 kernel is close enough that I'm holding off the 1.1.1 release until I can include that. Rob 1325636223.0 From james_mcmechan at hotmail.com Thu Jan 5 22:28:18 2012 From: james_mcmechan at hotmail.com (James McMechan) Date: Thu, 5 Jan 2012 14:28:18 -0800 Subject: [Aboriginal] Thoughts on xargs and aboriginal Message-ID: I was reading your http://landley.net/notes.html and saw the weirdness you were having with xargs' command line options for toybox As you said the most commonly used option -0 is not part of the standard I think it is claimed as a GNU extension, it seems oddly useful to be GNU though. It occurred to me that having a option like TOYBOX_LOG_WEIRD_POSIX_FLAGS might be handy, rather than implementing the full standard on the first pass, make toybox's option parsing know about all the POSIX flags but just log them and their args (or the whole command line) to a file and report failure. This might make "mind-bogglingly strange" a problem for a later date and the log file could be used like record-commands.sh to find what oddball option someone is actually using, who knows -I might not turn up until someone is compiling the ancient CDE window manager on a cell phone... busybox would not be my first choice for an alternate implementation swapping uClibc for musl or glibc or one of the other C libraries might be a better choice, several are available even if only for a architecture or two.? When testing the infrastructure for gcc-isms uClibc-isms ld-isms etc. it would seem to be ok to test with only a architecture or two just to ring out the gotchas and make sure things are mostly portable. Of course the compiler is a problem gcc-isms have crept in especially in the make dependency mess rather than using a correct make file to build the global directed graph, we use a compiler option to partially compile every file to produce a list of its includes to be used by make to determine if the file needs to be compiled, but since the makefiles are called recursively it may need another pass at the end to check if the other directories have made a update after this pass was checked. see aegis.sourceforge.net/auug97.pdf This I seem to remember was the reason that tcc could not compile the kernel last time I checked.? pcc, Clang/llvm, icc and tcc (after the dep phase) have been reported to compile the kernel, and so I would expect could compile aboriginal and could be used to ring out those gotchas. Regarding the mini-busybox config I was using, I just wanted to be able to recompile aboriginal under itself I have no direct need for LFS support, I can after all compile busybox with defconfig under a running aboriginal setup. Hey, vi and shell history are not something used in aboriginal's build process. I note that linux 3.2 is now out I look forward to aboriginal 1.1.1 James McMechan 1325802498.0 From rob at landley.net Fri Jan 6 13:08:20 2012 From: rob at landley.net (Rob Landley) Date: Fri, 06 Jan 2012 07:08:20 -0600 Subject: [Aboriginal] Thoughts on xargs and aboriginal In-Reply-To: References: Message-ID: <4F06F244.80301@landley.net> On 01/05/2012 04:28 PM, James McMechan wrote: > > I was reading your http://landley.net/notes.html and saw the weirdness > you were having with xargs' command line options for toybox > As you said the most commonly used option -0 is not part of the > standard I think it is claimed as a GNU extension, it seems oddly useful > to be GNU though. It's really a "linux extension". The Linux users have been using the gnu tools and push stuff upstream into them. The FSF is really big into taking credit for other people's ideas. > It occurred to me that having a option like > TOYBOX_LOG_WEIRD_POSIX_FLAGS > might be handy, rather than implementing the full standard on the first > pass, make toybox's option parsing know about all the POSIX flags but > just log them and their args (or the whole command line) to a file and > report failure. It already reports failure on unknown optinos, and I have the record-commands.sh wrapper that gives me a log of the full command lines called for each command, so I can already get this. > This might make "mind-bogglingly strange" a problem for a later > date and the log file could be used like record-commands.sh to find > what oddball option someone is actually using, who knows -I might not > turn up until someone is compiling the ancient CDE window manager > on a cell phone... A lot of this happens anyway. What I'm really looking at is "what's involved in claiming posix compliance". I want to do small and simple implementations of the commands, and that basically means figuring out everything it should do up front rather than bolting on features one at a time. (Getting new features to share code with older features usually means rewriting the older features' implementation to provide at least some of the new stuff too.) > busybox would not be my first choice for an alternate implementation > swapping uClibc for musl or glibc or one of the other C libraries might > be a better choice, several are available even if only for a architecture > or two. When testing the infrastructure for gcc-isms uClibc-isms ld-isms > etc. it would seem to be ok to test with only a architecture or two just to > ring out the gotchas and make sure things are mostly portable. I'm redoing toybox anyway, to replace android toolbox. It's not driven by aboriginal, I'm just using aboriginal to test it. (Busybox works fine at the moment for aboriginal linux.) http://landley.net/notes-2011.html#13-11-2011 > Of course the compiler is a problem gcc-isms have crept in especially in > the make dependency mess rather than using a correct make file to build the > global directed graph, we use a compiler option to partially compile every > file to produce a list of its includes to be used by make to determine if > the file needs to be compiled, but since the makefiles are called recursively > it may need another pass at the end to check if the other directories have made > a update after this pass was checked. see aegis.sourceforge.net/auug97.pdf > This I seem to remember was the reason that tcc could not compile the kernel > last time I checked. pcc, Clang/llvm, icc and tcc (after the dep phase) have > been reported to compile the kernel, and so I would expect could compile > aboriginal and could be used to ring out those gotchas. I still need to do http://lists.celinuxforum.org/pipermail/celinux-dev/2009-December/000217.html Remember, I had a tinycc fork, when the original repo had been dead for a year and a half. http://landley.net/code/tinycc When my fork revived interest in the project, Fabrice Bellard handed off maintainership of the original dead branch to a windows developer who said he didn't understand the work I'd done and cherry picked a few commits to the original CVS branch and then continued on from there. Third party contributions (such as x86-64 support) went to the original branch that had the longer history and the tinycc.org domain name (and which lwn.net covered each new release of, which it never did for my version because it wasn't "official".) The end result is that my project was overshadowed to the point I stopped wanting to do it, and the original version is no closer to building an unmodified linux kernel than it was when tccboot came out in 2004, and never will because the windows developer running it is only interested in making a free windows compiler. > Regarding the mini-busybox config I was using, I just wanted to be able to > recompile aboriginal under itself I have no direct need for LFS support, I can > after all compile busybox with defconfig under a running aboriginal setup. > Hey, vi and shell history are not something used in aboriginal's build process. > > I note that linux 3.2 is now out I look forward to aboriginal 1.1.1 Yesterday I kicked off a test build on a 16-way server borrowed from Funtoo, but haven't had a chance to check back in yet (work deadline today). I need to adjust one of the perl removal patches to apply without fuzz, and yank the sparc fix that got merged upstream (yay!), and regression test the LFS build on all the targets. I should have something out this weekend. Rob 1325855300.0 From rob at landley.net Mon Jan 9 15:13:58 2012 From: rob at landley.net (Rob Landley) Date: Mon, 09 Jan 2012 09:13:58 -0600 Subject: [Aboriginal] 3.2 is out, prepping release. Message-ID: <4F0B0436.9070003@landley.net> I spent the portion of the weekend I could claw away from other emergencies triaging the 3.2 kernel release, which mostly works out of the box, but "mostly" is (as always) fiddly. One of the perl removal patches needed to be rediffed to apply without "fuzz", but the patch itself still works fine unmodified. The kernel guys once again did something insane with the config symbols, reorganizing the ethernet drivers menu under a new CONFIG_ETHERNET and then adding _vendor_ guard symbols under that, so I now have to care that the Sun "lance" driver in the sparc board emulation was manufactured by AMD. (I really, really need to write a tool to expand a miniconfig and _set_ guard symbols automatically. The current stuff only does dependencies in one direction, it won't go "I just tried to set a symbol that can ONLY be set if this one, unambiguous other symbol is first set". (This is really visibility rules rather than dependencies per se, but menuconfig has always conflated those two. But this is 90% of the conf changes you need from version to version, I need to _fix_ it so it _goes_away_. Not holding up the release for it, though.) There are some other fiddly bugs I'll mention in another message, since they're not 3.2 specific. Not holding up the release for those either, since they're not regressions. I'll try to do the release notes tonight, Rob 1326122038.0 From rob at landley.net Mon Jan 9 15:17:15 2012 From: rob at landley.net (Rob Landley) Date: Mon, 09 Jan 2012 09:17:15 -0600 Subject: [Aboriginal] Bugs I need to fix after 1.1.1 is out. Message-ID: <4F0B04FB.9040502@landley.net> 1) Powerpc threading segfaults immediately on application launch. (Non-threaded stuff seems to work fine.) 2) Sparc threading hangs, and the g++ "hello world" build fails with an uknown link type. 3) The sh4 board only supports 64 megs of memory, which is totally inadequate for gcc 3.x or later. (Builds hello world from the command line, but a big C file with lots of headers spawned from "make" with the assembler piping output to the compiler, not so much.) 4) If the network card config fails the CPUS count is still set to 3, but without distcc the builds don't ge distributed, and boards with only 256 megs of ram can't do -j 3 builds locally without dying (again, memory exhaustion). Whether or not this is a separate bug from "ethernet isn't workign" is debatable, but distcc is brittle enough that failures should be handled with at least good error messages. Rob 1326122235.0 From rob at landley.net Sat Jan 21 19:03:17 2012 From: rob at landley.net (Rob Landley) Date: Sat, 21 Jan 2012 13:03:17 -0600 Subject: [Aboriginal] Sorry for the radio silence... Message-ID: <4F1B0BF5.3060500@landley.net> My spare development time's been taken up with toybox, when I wasn't staying late at work. Ok, mips is broken, and the REASON mips is broken is QEMU broke between 0.15.1 and 1.0. Specfiically commit 5632ae46: "mips_malta: move i8259 initialization after piix4 initialization" broke the network on mips. It still comes up, but doesn't pass packets. I was bisecting the kernel and trying to track down the regression, but the last release binaries didn't work either, which clued me in it was qemu. (There's a reason I keep those suckers around.) I'm not quite sure how to deal with that one. I'm pinging the qemu list, and I'm going to take a stab at getting e1000 to work on mips. Other than that, the release notes are written up and I have binaries ready and waiting, just... need to actually get stuff _working_ before it ships. :) Rob 1327172597.0 From blink at bojary.koba.pl Sat Jan 21 20:28:02 2012 From: blink at bojary.koba.pl (Paul Onyschuk) Date: Sat, 21 Jan 2012 21:28:02 +0100 Subject: [Aboriginal] GCC - messed up search dirs Message-ID: <20120121212802.39c0350c.blink@bojary.koba.pl> First a bit of the background (apologies for my poor english writing). I'm trying to bootstrap pkgsrc under chrooted Aboriginal Linux. That should be pretty easy, beside chroot image pkgsrc needed also zlib, ncurses and groff (I used mdocml instead). Small fix in bootstrap script was also needed: busybox awk fails at some point, so pkgsrc should build own version (nawk). Problems starts here, C preprocessor fails on conftest.c in one of the packages (bmake): > > cpp: error trying to exec 'cc1': execvp: No such file or directory > As it turns out GCC cannot point to proper cc1 location: > > $ gcc -print-search-dirs > install: /usr/bin/../ > programs: /usr/bin:/usr/bin/../tools/bin:/usr/kerberos/sbin: \ > /usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin > libraries: /usr/bin/../cc/lib/:/usr/bin/../lib/:/usr/bin/../cc/ > > $ gcc -print-prog-name=cc1 > cc1 > This can be fixed by setting: > > $ export GCC_EXEC_PREFIX=/usr/i586-unknown-linux/bin/ > In the end, others problems are showing up (still regarding C preprocessor) like missing limits.h header. I don't like idea of adding another few environment variables just to point GCC to own files. I'm using root-filesystem-i586 image [1] and the questions are: (A) Am I doing something wrong (chroot setup and so on)? (B) Are additional setps needed before using GCC under Aboriginal? (C) Is there something wrong with GCC under Aboriginal? I'm not sure if this is needed, but here is "gcc -v" output just in case: > > Invoked as gcc > Reference path: /usr/bin/.. > arg[ 0] = rawcc > arg[ 1] = -U__nptl__ > arg[ 2] = -v > Using built-in specs. > Target: i586-unknown-linux > Configured > with: /home/landley/work/ab3/build/temp-i586/gcc-core/configure > --target=i586-unknown-linux > --prefix=/home/landley/work/ab3/build/native-compiler-i586 > --disable-multilib --disable-nls --enable-c99 --enable-long-long > --enable-__cxa_atexit --enable-languages=c,c++ --disable-libstdcxx-pch > --program-prefix= --enable-threads=posix --enable-shared > --build=x86_64-walrus-linux --host=i586-unknown-linux Thread model: > posix gcc version 4.2.1 > [1] http://landley.net/aboriginal/downloads/binaries/ \ root-filesystem/root-filesystem-i586.tar.bz2 1327177682.0 From rob at landley.net Sun Jan 22 03:56:18 2012 From: rob at landley.net (Rob Landley) Date: Sat, 21 Jan 2012 21:56:18 -0600 Subject: [Aboriginal] GCC - messed up search dirs In-Reply-To: <20120121212802.39c0350c.blink@bojary.koba.pl> References: <20120121212802.39c0350c.blink@bojary.koba.pl> Message-ID: <4F1B88E2.8020605@landley.net> On 01/21/2012 02:28 PM, Paul Onyschuk wrote: > First a bit of the background (apologies for my poor english writing). > I'm trying to bootstrap pkgsrc under chrooted Aboriginal Linux. What... ah, you mentioned the version and target info at the end of the message, got it. > That > should be pretty easy, beside chroot image pkgsrc needed also zlib, > ncurses and groff (I used mdocml instead). The static-tools.hdc native build does zlib, the bootstrap-gentoo.hdc does ncurses, and the lfs bootstrap does groff. So all of those have built under aboriginal before. > Small fix in bootstrap > script was also needed: busybox awk fails at some point, so pkgsrc > should build own version (nawk). Which bootstrap script? (When you chrooted, did you run /sbin/init.sh to enter the chroot?) Fails how? I've built the whole of linux from scratch under a chrooted aboriginal linux, using the above hdc builds. (All of them are at http://landley.net/aboriginal/control-images). > Problems starts here, C preprocessor fails on conftest.c in one of the > packages (bmake): I presume you're referring to: http://www.linuxfromscratch.org/hints/downloads/files/bmake.txt >> cpp: error trying to exec 'cc1': execvp: No such file or directory Oh right, I forgot to wrap the preprocessor with ccwrap.c (all the Linux stuff just uses "gcc -E" instead of ever calling cpp). The easy thing to do is replace cpp with a shell script: #!/bin/bash gcc -E "$@" (I have no idea why the preprocessor would try to execute cc1, but ok.) > As it turns out GCC cannot point to proper cc1 location: I use a wrapper binary that takes all the path logic decisions out of gcc's hands, adjusting the $PATH and rewriting the gcc command line to start with --nostdinc --nostdlib and then adding -I and -L for everywhere it should look, explicitly. However, cpp != gcc. >> $ gcc -print-search-dirs >> install: /usr/bin/../ >> programs: /usr/bin:/usr/bin/../tools/bin:/usr/kerberos/sbin: \ >> /usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin >> libraries: /usr/bin/../cc/lib/:/usr/bin/../lib/:/usr/bin/../cc/ >> >> $ gcc -print-prog-name=cc1 >> cc1 >> > > > This can be fixed by setting: > >> $ export GCC_EXEC_PREFIX=/usr/i586-unknown-linux/bin/ That's going behind the wrapper's back and telling gcc explicitly where to look at stuff. That works, but there's a dozen things you need to adjust and many of them are conditional on what you're trying to do. Bascially, I got the toolchain to work portably as a cross compiler (from wherever it's currently installed) by using the wrapper, and then I just did the same thing for the native toolchain. Mostly it just works and you don't have to pay attention to it, but I missed wrapping the cpp binary because nothing in Linux ever seems to use it. My bad. > In the end, others problems are showing up (still regarding C > preprocessor) like missing limits.h header. This is honestly the first time in several years I've encountered something using cpp (apparently bsd uses it but linux doesn't), but I'll fix it now it's pointed out. I'll take a stab at this tomorrow and check it into the repo. Either the above wrapper script to use gcc -E for cpp, or by applying ccwrap.c to the cpp binary. > I don't like idea of > adding another few environment variables just to point GCC to own files. Neither did I. I spent a long time trying to fix gcc's path logic. The only way to fix it is to RIP IT OUT ENTIRELY, and that was over a 10,000 line patch. > I'm using root-filesystem-i586 image [1] and the questions are: > > (A) Am I doing something wrong (chroot setup and so on)? > (B) Are additional setps needed before using GCC under Aboriginal? > (C) Is there something wrong with GCC under Aboriginal? The answer is C) something finally used the cpp binary, which I forgot to wrap. My bad. (I acually encountered in this in 2009 and put it on my todo list, but I forgot I hadn't actually done it yet. And I only noticed because I was playing with a compiler other than gcc, not because a package used it...) Rob 1327204578.0 From blink at bojary.koba.pl Sun Jan 22 15:37:00 2012 From: blink at bojary.koba.pl (Paul Onyschuk) Date: Sun, 22 Jan 2012 16:37:00 +0100 Subject: [Aboriginal] GCC - messed up search dirs In-Reply-To: <4F1B88E2.8020605@landley.net> References: <20120121212802.39c0350c.blink@bojary.koba.pl> <4F1B88E2.8020605@landley.net> Message-ID: <20120122163700.16d0bc6a.blink@bojary.koba.pl> This may very well lead to offtopic, but I feel that I need to explain myself better. Sadly my writing in English isn't on the level I would like it to be. This message ended up much longer than I planed, apologies for that. What is Pkgsrc? In short portable ports-like system for building software and handling dependencies (it works on some weird platforms, I will go into that later). Descriptions like that can be found everywhere, although most of the times they don't provide informations about pkgsrc technicalities. Pkgsrc is pretty much a big bag of makefiles and nothing else. Those makefiles aren't GNU make compatible, instead NetBSD bmake is used. Makefile of cp(1) from NetBSD source tree is a good example how bmake is used: > > PROG= cp > SRCS= cp.c utils.c > > .include > As a side note FreeBSD is using currently pmake, which is similar to bmake, but has some different subtleties. OpenBSD is also using bmake, it was inherited when Theo de Raadt forked off from NetBSD. Since then OpenBSD bmake was maintained independently, so right now it isn't the same compared to NetBSD version. Second side note. This isn't related to Aboriginal Linux, but could be interesting for toybox project (or not). Some time ago MINIX project started replacing their userland tools with NetBSD counterparts. It got me thinking if *BSD userland can be ported to Linux (uclibc based to be specific). I started with OpenBSD tools, but it quickly turned out that those were heavily using OpenBSD libc specific features (I will not go into the details). On other hand porting NetBSD tools was in most cases trivial. Some informations about this effort can be found in my github repository [1]. This link leads to wiki page with some details about porting software from NetBSD, some tools can be found in the same repository. I used brute force, especially when it comes to macros. Also makefiles are ad-hoc solution. I was going to start from scratch and make it right, but that never happened. Back to Pkgsrc. Pkgsrc is released every quarter, which makes it easier to use for external projects and not only NetBSD (release timing and so on). Currently pkgsrc is used also by DragonFly BSD, SmartOS from Joyent (Solaris based system) and there are big efforts in MINIX project to use it as official solution for providing third party software. It isn't perfect solution, Pkgsrc has own quirks, but it gets job done and BSD license makes it even more attractive. I introduced some confusion in previous message, when speaking about Pkgsrc bootstrapping, so I will describe the process. It starts with fetching tarball "full of makefiles" [2], typically it is unpacked under /usr (/usr/pkgsrc is main tree in this case). Alternatively It can be unpacked under user home directory and bootstrapped without privileges (nice feature, because you can keep more than one Pkgsrc version e.g. with different settings). I pointed out dependencies in previous message. Then bootstrap script (it can be found under /usr/pkgsrc/bootstrap, it just a shell script) is used to build some common tools used by Pkgsrc, bmake is one of those. Because Pkgsrc is designed to support so many systems, some additional tools can be built e.g. if system only provides old awk or sed isn't compatible enough and so on. The same goes for Busybox awk (one of the tools built by bootstrap is using long awk script - I don't remember which one), but fix for that is trivial: change "need_awk" to yes on line 596 in bootstrap script. Another side node. I "ported" (ported is too big word to describe that in my opinion) some software from one to another system and I can say that awk is one of the biggest offenders there. Some software provides GNU awk specific scripts in their build system (command line flags and other features). This creates problems, when porting software to *BSD, because those are using "One True AWK" by Brian W. Kernighan aka bawk/nawk. In most cases this is just build dependency, so it is easier to provide gawk than fix awk scripts in every 3rd party software. Scripts written for nawk, work just fine with gawk, but not with Busybox awk in some cases. Of course I could fix awk script in Pkgsrc bootstrap case, but I have been there and I know that using Busybox awk with Pkgsrc will hit you later, when building other software. Pkgsrc already distinguishes software that uses GNU awk - it is listed as make dependency in makefiles. As for everything else, it easy to specify if Pkgsrc should use version provided by system or use own version: > > TOOLS_PLATFORM.awk= /usr/pkg/bin/nawk > This is good example of Pkgsrc properties. Where other solutions are integrated heavily with system, Pkgsrc is quite opposite. Pkgsrc coexist with existing system infrastructure, but should never mess up system libraries and other things (in the end /usr/pkg can be removed without worries). This opens also interesting options e.g. you can bootstrap pkgsrc-2011Q4 under /usr/pkg prefix and have another version let say pkgsrc-current (non-stable version, moving target) under /home/user_name/pkg. It is easy to specify what libraries Pkgsrc should use: ncurses provided by system or Pkgsrc? > > PREFER.pkgsrc= ncurses > TOOLS_PLATFORM.nroff= /usr/bin/mandoc > In first case Pkgsrc will build own version of ncurses and use it when compiling packages - this can be also done for other libraries and even compiler. Wrappers are used to "hide" everything else in build process. As for second one, I'm not big fan of Groff (I could provide long list of reasons). Mdocml (aka mandoc) [3] isn't a full blown replacement for troff/nroff formatter. Instead it supports only man and mdoc macros, which makes it very small and fast for this purpose. Other than that, HTML output is high quality. Groff in OpenBSD base system was replaced by Mdocml some time ago. This projects provides also by far the best guide [4] for writing man pages and history of man pages is pure gold, if someone is interested in "computer archeology". Back on track. When it comes to CPP error, configure script used by bmake has some sanity checks for environment. It checks if compiler and preprocessor are usable and rest is the story. I could change configure script, but I don't see a point in fixing something that isn't broken. I appreciate fast response and I'm looking forward for using Aboriginal. After writing all of this, one thing left is explanation for using Pkgsrc under Aboriginal. For anything beside basic software, tracking dependencies can be pain in the ass. There is no point of writing another system just for that, when you can use existing solution like Pkgsrc. Moreover I would like to investigate if there is a easy way to force Pkgsrc to use only static linking. Pkgsrc is already used by MINIX in some degree and this system as of time being doesn't have support for shared libraries at all. btw. I'm regular reader of your blog, keep it up. Thanks for all your work. I'm subscribed to Aboriginal and to toybox mailing lists. [1] https://github.com/blinkkin/netbsd-coreutils/wiki/Porting [2] http://ftp.netbsd.org/pub/pkgsrc/pkgsrc-2011Q4/pkgsrc.tar.bz2 [3] http://mdocml.bsd.lv/ [4] http://manpages.bsd.lv/ 1327246620.0 From rob at landley.net Sun Jan 22 22:24:18 2012 From: rob at landley.net (Rob Landley) Date: Sun, 22 Jan 2012 16:24:18 -0600 Subject: [Aboriginal] GCC - messed up search dirs In-Reply-To: <20120122163700.16d0bc6a.blink@bojary.koba.pl> References: <20120121212802.39c0350c.blink@bojary.koba.pl> <4F1B88E2.8020605@landley.net> <20120122163700.16d0bc6a.blink@bojary.koba.pl> Message-ID: <4F1C8C92.1020301@landley.net> On 01/22/2012 09:37 AM, Paul Onyschuk wrote: > As a side note FreeBSD is using currently pmake, which is similar to > bmake, but has some different subtleties. OpenBSD is also using bmake, > it was inherited when Theo de Raadt forked off from NetBSD. Since then > OpenBSD bmake was maintained independently, so right now it isn't the > same compared to NetBSD version. This would explain why I've never heard of the package before. However, building packages I've never heard of is a valid use case of Aboriginal Linux: I made a build environment, you use it to do whatever you like. The fact it didn't work right is a bug, and I need to fix it. I think I have a fix, which should be in the upcoming release. If you'd like to try it now, do this: cat > /tmp/cpp << 'EOF' #!/bin/bash gcc -E "$@" EOF mount --bind /tmp/cpp /bin/cpp > Second side note. This isn't related to Aboriginal Linux, but could be > interesting for toybox project (or not). Some time ago MINIX project > started replacing their userland tools with NetBSD counterparts. It > got me thinking if *BSD userland can be ported to Linux (uclibc based > to be specific). I started with OpenBSD tools, but it quickly turned > out that those were heavily using OpenBSD libc specific features (I will > not go into the details). I've got to get as much of itas possible to work with bionic. That's going to be interesting. I'll probably have to write my own regex engine... > On other hand porting NetBSD tools was in most cases trivial. Some > informations about this effort can be found in my github repository > [1]. This link leads to wiki page with some details about porting > software from NetBSD, some tools can be found in the same repository. > I used brute force, especially when it comes to macros. Also makefiles > are ad-hoc solution. I was going to start from scratch and make it > right, but that never happened. I recall a bunch of subtle incompatabilities, such as netbsd awk and the gnu/dammit version of awk disagreeing about the index of command line entries... > Back to Pkgsrc. Pkgsrc is released every quarter, which makes it > easier to use for external projects and not only NetBSD (release timing > and so on). Currently pkgsrc is used also by DragonFly BSD, SmartOS > from Joyent (Solaris based system) and there are big efforts in MINIX > project to use it as official solution for providing third party > software. It isn't perfect solution, Pkgsrc has own quirks, but it > gets job done and BSD license makes it even more attractive. Minix has a project? > I introduced some confusion in previous message, when speaking about > Pkgsrc bootstrapping, so I will describe the process. It starts with > fetching tarball "full of makefiles" [2], typically it is unpacked > under /usr (/usr/pkgsrc is main tree in this case). Alternatively It > can be unpacked under user home directory and bootstrapped without > privileges (nice feature, because you can keep more than one Pkgsrc > version e.g. with different settings). I pointed out dependencies in > previous message. Did my above workaround fix your problem, or not? > Then bootstrap script (it can be found under /usr/pkgsrc/bootstrap, it > just a shell script) is used to build some common tools used by Pkgsrc, > bmake is one of those. Because Pkgsrc is designed to support so many > systems, some additional tools can be built e.g. if system only > provides old awk or sed isn't compatible enough and so on. The same > goes for Busybox awk (one of the tools built by bootstrap is using long > awk script - I don't remember which one), but fix for that is trivial: > change "need_awk" to yes on line 596 in bootstrap script. > > Another side node. I "ported" (ported is too big word to describe that > in my opinion) some software from one to another system and I can say > that awk is one of the biggest offenders there. Some software provides > GNU awk specific scripts in their build system (command line flags and > other features). This creates problems, when porting software to *BSD, > because those are using "One True AWK" by Brian W. Kernighan aka > bawk/nawk. In most cases this is just build dependency, so it is > easier to provide gawk than fix awk scripts in every 3rd party > software. Scripts written for nawk, work just fine with gawk, but not > with Busybox awk in some cases. I comissioned a half-dozen fixes to busybox awk over the years, and for toybox I'm going to have to write my own. I'll make it work according to SUSv4 and the Linux From Scratch ./configure scripts. I don't care about BSD but if you want to test it there, I'll take bug reports once I run out of my own bugs to fix in it. But I haven't started that one yet... > Back on track. When it comes to CPP error, configure script used by > bmake has some sanity checks for environment. It checks if compiler > and preprocessor are usable and rest is the story. I could change > configure script, but I don't see a point in fixing something that > isn't broken. I appreciate fast response and I'm looking forward for > using Aboriginal. I was pretty sure that gcc didn't call cpp or else the path issue would have bit us beforenow, but I just confirmed that replacing it with a tiny "gcc -E" wrapper script seems to work. I'll put that in the upcoming release. > After writing all of this, one thing left is explanation for using > Pkgsrc under Aboriginal. For anything beside basic software, tracking > dependencies can be pain in the ass. There is no point of writing > another system just for that, when you can use existing solution like > Pkgsrc. Or portage. Or rpm. Or dpkg. Or pacman. Or bitbake. Or buildroot... > Moreover I would like to investigate if there is a easy way to > force Pkgsrc to use only static linking. find root-filesystem-i586 -name "*.so*" | xargs rm > Pkgsrc is already used by > MINIX in some degree and this system as of time being doesn't have > support for shared libraries at all. > > btw. I'm regular reader of your blog, keep it up. Thanks for all your > work. I'm subscribed to Aboriginal and to toybox mailing lists. Yay! Thanks. I'm sorry if I'm not interested in the new tangent but my plate's full: toybox is probably going to suck up my free time for the next six months, and I still need to get the bootstrap-gentoo.hdc stuff working, and get the Linux From Scratch build to 7.0, and I need to sparc working the rest of the way and fix m68k and hack up an sh4 board with more than 64 megs and write a distcc clone that doesn't suck and figure out what to do about armv8 and mips32 and if I managed all _that_ I'd dig up my old qcc notes... Rob 1327271058.0 From blink at bojary.koba.pl Mon Jan 23 10:23:45 2012 From: blink at bojary.koba.pl (Paul Onyschuk) Date: Mon, 23 Jan 2012 11:23:45 +0100 Subject: [Aboriginal] GCC - messed up search dirs In-Reply-To: <4F1C8C92.1020301@landley.net> References: <20120121212802.39c0350c.blink@bojary.koba.pl> <4F1B88E2.8020605@landley.net> <20120122163700.16d0bc6a.blink@bojary.koba.pl> <4F1C8C92.1020301@landley.net> Message-ID: <20120123112345.3b799822.blink@bojary.koba.pl> On Sun, 22 Jan 2012 16:24:18 -0600 Rob Landley wrote: > > I think I have a fix, which should be in the upcoming release. If > you'd like to try it now, do this: > > cat > /tmp/cpp << 'EOF' > #!/bin/bash > > gcc -E "$@" > EOF > mount --bind /tmp/cpp /bin/cpp > > Did my above workaround fix your problem, or not? > Yes, it works just fine this way. I forgot about this after writing this wall of text, sorry about that. I should have informed you about this earlier. > > I've got to get as much of itas possible to work with bionic. That's > going to be interesting. I'll probably have to write my own regex > engine... > That's great, I'll be definitely looking forward to it. I agree that musl libc [1] was wasted opportunity. Personally I found it amusing that same mistakes were once again made, when it comes to adaptation - dietlibc and its GPL license vs. musl libc and LGPL. > > I recall a bunch of subtle incompatabilities, such as netbsd awk and > the gnu/dammit version of awk disagreeing about the index of command > line entries... > There is no such thing as NetBSD awk per se. nawk stands for new awk and it is maintained by Brian Kernighan [2]. It is used by every *BSD AFAIK. This is standard awk implementation as you could get, in the end it was written by author of language. Side note on name confusion. NetBSD is probably the only system that still maintains own version of curses (it is called just "curses"), they don't use GNU ncruses as everyone else. > > Minix has a project? > They got huge grant from EU. NetBSD libc was ported and they switched to Clang/LLVM as default compiler. [1] http://www.landley.net/notes.html#18-01-2012 [2] http://www.cs.princeton.edu/~bwk/btl.mirror/ 1327314225.0 From rob at landley.net Mon Jan 23 12:24:23 2012 From: rob at landley.net (Rob Landley) Date: Mon, 23 Jan 2012 06:24:23 -0600 Subject: [Aboriginal] GCC - messed up search dirs In-Reply-To: <20120123112345.3b799822.blink@bojary.koba.pl> References: <20120121212802.39c0350c.blink@bojary.koba.pl> <4F1B88E2.8020605@landley.net> <20120122163700.16d0bc6a.blink@bojary.koba.pl> <4F1C8C92.1020301@landley.net> <20120123112345.3b799822.blink@bojary.koba.pl> Message-ID: <4F1D5177.3090003@landley.net> On 01/23/2012 04:23 AM, Paul Onyschuk wrote: > On Sun, 22 Jan 2012 16:24:18 -0600 > Rob Landley wrote: >> Did my above workaround fix your problem, or not? >> > > Yes, it works just fine this way. I forgot about this after writing > this wall of text, sorry about that. I should have informed you about > this earlier. Yay! It's checked into the repo now, and I set binaries building last night. I should get a chance to test them this evening. >> I've got to get as much of itas possible to work with bionic. That's >> going to be interesting. I'll probably have to write my own regex >> engine... >> > > That's great, I'll be definitely looking forward to it. I agree that > musl libc [1] was wasted opportunity. Personally I found it amusing > that same mistakes were once again made, when it comes to adaptation - > dietlibc and its GPL license vs. musl libc and LGPL. It wasn't _exactly_ a mistake. The Free Software Foundation and its spin-off the SFLC managed to poison the GPL so that what was a viable strategy 5 years ago isn't any longer. They just missed the widespread backlash against GPLv3 and the lawsuits. They also missed the opportunity posed by the switch to android, they're still chasing the Linux on the Desktop That Cried Wolf. (Year of the Linux Desktop, 10th anniversary tour.) But it is, in current context, a waste of effort. >> I recall a bunch of subtle incompatabilities, such as netbsd awk and >> the gnu/dammit version of awk disagreeing about the index of command >> line entries... >> > > There is no such thing as NetBSD awk per se. nawk stands for new awk > and it is maintained by Brian Kernighan [2]. It is used by every *BSD > AFAIK. This is standard awk implementation as you could get, in the > end it was written by author of language. So it's a "standard" that isn't nearly as widely used as the nonstandard version. Ok. > Side note on name confusion. NetBSD is probably the only system that > still maintains own version of curses (it is called just "curses"), > they don't use GNU ncruses as everyone else. I don't use ncurses either. I output ansi escape sequences, same as I did under DOS, because they still work. Unix stopped having hardware TTYs by the early 80's. The output of tty devices is displayed by software. Having 250 different output formats so that two pieces of software can talk to each other is _dumb_. >> Minix has a project? >> > They got huge grant from EU. NetBSD libc was ported and they switched > to Clang/LLVM as default compiler. Well, I suppose it's nice to know there _is_ an also-ran to Illumos. Apple's been sponsoring Clang/LLVM to wean MacOS X off of gcc because they stayed at 4.2.1 and binutils 2.17 the same reason I did: last GPLv2 release. Rob 1327321463.0 From venkivoice at gmail.com Wed Jan 25 02:56:51 2012 From: venkivoice at gmail.com (venkat raman) Date: Wed, 25 Jan 2012 08:26:51 +0530 Subject: [Aboriginal] Reg: Exiting due to errors (host host-tools busybox) Message-ID: Hi , I was trying to build for i686 and my host machine is ubuntu 11.10. I got the error as "Exiting due to errors (host host-tools busybox)" . I checked in the mailing list where I found a message related to this but it has been built for i486 , below is the link for the patch issued for this error. Its seems like the build root is up-to-date with the patch. but i still get this error. http://landley.net/hg/aboriginal/rev/1332 Cheers, Venkat http://brainreaders.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob at landley.net Wed Jan 25 13:28:10 2012 From: rob at landley.net (Rob Landley) Date: Wed, 25 Jan 2012 07:28:10 -0600 Subject: [Aboriginal] Reg: Exiting due to errors (host host-tools busybox) In-Reply-To: References: Message-ID: <4F20036A.6010605@landley.net> On 01/24/2012 08:56 PM, venkat raman wrote: > Hi , > > I was trying to build for i686 and my host machine is ubuntu 11.10. I > got the error as "Exiting due to errors (host host-tools busybox)" That tells me three things: 1) What architecture it was building for (host, I.E. the host-tools step which is target independent). 2) what stage it was on (running host-tools.sh) 3) what package it was trying to build (busybox). So I know that building busybox for the host failed, but not _why_ it failed. You didn't include the part of the build output with the actual error message from busybox, which might tell me the reason the busybox build failed. Earlier in the log gcc or ld or make produced an error message, which you didn't include in this bug report. I note that if you're using an SMP machine, you can "export CPUS=1" to force a single-processor build, which tends to give more informative error messages and leave them at the end of the log instead of potentially lost in the noise further up like parallel builds sometimes do. > . I checked in the mailing list Which mailing list, the busybox mailing list? This mailing list? > where I found a message related to this Which you didn't link to either, so I have no idea what it said. > but it has been built for i486, The failure you had was before it got to anything target-specific, it was still building prerequisites. So it shouldn't matter which target you told it to build, you'd get the same error. > below is the link for the patch issued for this error. I still have no idea which specific error you're referring to. > Its seems like the build root What does "the build root" mean in this instance? (Do you mean the build scripts? It didn't get as far as building cross compilers, let alone a root filesystem...) > is up-to-date with the patch. but i still get this error. > > http://landley.net/hg/aboriginal/rev/1332 That's a link to the aboriginal linux source control repository from a year ago. You never specified what version you were trying, but that patch has been in the last couple releases, so unless you're trying to build a really old version it shouldn't matter. Could you please send me the last screenful of log produced by the failing build? Rob 1327498090.0 From rob at landley.net Wed Jan 25 19:24:07 2012 From: rob at landley.net (Rob Landley) Date: Wed, 25 Jan 2012 13:24:07 -0600 Subject: [Aboriginal] Reg: Exiting due to errors (host host-tools busybox) In-Reply-To: References: <4F20036A.6010605@landley.net> Message-ID: <4F2056D7.1000801@landley.net> On 01/25/2012 12:55 PM, venkat raman wrote: > HI, > > Version Using : aboriginal-1.1.0 > > I am really sorry for not sending the complete error log message. Please > find the attached error log file which has produced during build. No problem. The error is this: /usr/include/mtd/ubi-user.h:329:3: error: conflicting types for '__packed' Which is actually a bug in Ubuntu 11.10, which I complained about on the busybox mailing list here: http://lists.busybox.net/pipermail/busybox/2011-August/076430.html And then again here: http://lists.busybox.net/pipermail/busybox/2011-November/076916.html Which is pretty much the reason I did this in current source control: http://lists.busybox.net/pipermail/busybox/2011-November/076916.html Tell you what: try the current source control version (which has that fix in it): http://landley.net/hg/aboriginal/archive/tip.tar.bz2 The only remaining bug I need to track down is a powerpc regression, the rest should all work. I'm mean to ship another release again this weekend, so it's pretty much stable and shouldn't change noticeably before then except to fix the ppc thing. Thanks for your patience, Rob 1327519447.0 From tony at stclaire.com Tue Jan 31 11:58:52 2012 From: tony at stclaire.com (=?utf-8?B?dG9ueUBzdGNsYWlyZS5jb20=?=) Date: Tue, 31 Jan 2012 06:58:52 -0500 Subject: [Aboriginal] =?utf-8?q?cpp_in_the_native_environment?= Message-ID: <201201311307.q0VD7dk04341@ns1.ceenet.com> Hello Rob, Any chance to include the full cpp preprocessor in aboriginal. The problem with cc -E is it will not process includes. Although cpp is not used by much in linux it has proven handy as a non-c code preprocessor. Any chance it will make it in 1.1.1 ? Thanks Rob. -------------- next part -------------- An HTML attachment was scrubbed... URL: