From rob at landley.net Sun Nov 1 11:14:14 2015 From: rob at landley.net (Rob Landley) Date: Sun, 1 Nov 2015 05:14:14 -0600 Subject: [Aboriginal] Using aboriginal as a bare-bones linux install In-Reply-To: <5627E337.1010405@gmail.com> References: <561D5513.3080307@landley.net> <5627E337.1010405@gmail.com> Message-ID: <5635F406.7070705@landley.net> On 10/21/2015 02:10 PM, Avery Payne wrote: > I have an old 32-bit atom-based HP 311 netbook that I want to install > aboriginal on. The goal is to have a "clean slate" Linux install that > has the absolute minimum software; literally, a Linux install with the > "fewest" preconceived notions from a distro maintainer. I have my own > pedantic reasons for doing this, but they are my reasons; the biggest is > not being dependent on a distro maintainer, and by extension, some of > their decisions. Well, there's still some decisions: Does your "absolute minimum" software include the toolchain, or just boot to a shell prompt? If it does should the packages be statically or dynamically linked? Is your idea of minimal "busybox defconfig" (minimum number of configuration decisions) or the combination of busybox+toybox I've worked out as the minimal set of commands needed to build linux from scratch (plus ones like "ps" that you kinda miss if they're not there on the command line)? More pragmatically, packaging-wise do you want root-filesystem running out of initramfs, or do you want it on the sd card filesystem too? If the latter, in theory you can copy the contents of root-filesystem-i686.tar.gz and native-compiler-i686.tar.gz into the same directory and they should work. I.E. cribbing from more/chroot-splice.sh: # Copy root filesystem and native compiler cp -lan "build/root-filesystem-$1/." "$CHROOT" || exit 1 cp -lan "build/native-compiler-$1/." "$CHROOT" || exit 1 > I really, really want to be able to build aboriginal in such a way that > it builds direct to a SSD, filesystem and all, and install a boot > loader, and bamo-presto, my very own bare-bones Linux install. The system-image.sh stage builds the kernel and packages it into a tarball, but there's nothing to say you can't add another script at the end that does the above two cp to a target directory and adds your bootloader. > I've > managed to get nice clean i686 builds, and the SSD mounted inside of > QEMU as a "full drive"; Ok, you're ahead of me. :) > I'm still working out the details needed to get > the files moved from the image to the SSD without horking the target > image (think /dev & friends coupled with handling all the symlinks), Most of the symlinks are relative ones designed to be moved, the "init" symlink in sources/root-filesystem for example. The symlinks setup by sources/toys/hdainit.sh are probably absolute though: cp -rFs /usr/overlay/. / That happens at boot time. The above hdainit gets copied into the native-compiler squashfs as "init" at the top level, and then sources/toys/sbin/init.sh mounts /dev/hda on /usr/overlay and then calls /usr/overlay/init if it exists. For chroots I do those fixups manually, see more/setup-chroot.sh for example. > and > finally, some kind of bootloader installed (which terrifies me because > if I screw that up, I could hork the host bootloader). One of the reasons I do so much with qemu is it's very difficult to persistently break it. :) You'll want a boot USB stick or similar standing by when you play with bootloaders. > That being all said, thank you so much for your hard work on aboriginal. > > (the following was hijacked from another thread on qcc): >> What initially attracted me to tinycc was: >> http://bellard.org/tcc/tccboot.html And the windows clowns following >> grishka's "mob branch" (we don't need a maintainer, we can have >> drive-by checkins with nobody steering!) are _father_ away from making >> that work on a vanilla kernel than what that shipped _eleven_years_ago_. > Since I know bumpkis about C, just what about the current Linux kernel > makes it so it can't be compiled with an older C compiler like tcc? (or > pcc for that matter...) Is there some dependency in the kernel that > requires a specific part of the language that older compilers won't handle? It's not "older compilers" such as "using every part of the buffalo when it comes to gcc features". They're getting better about it now that http://llvm.linuxfoundation.org is providing pushback, but only half the fixes are weaning the kernel off of gcc-isms, the rest are llvm implementing more features the kernel uses. I remember building the kernel died near the start trying to create an asm-offsets with objdump and sed, but it looks like that part of the kernel's been rewritten not to require that anymore? Really, make a $PATH that only has tinycc as the cc and try it and see what breaks. That's what I did. (No binutils, no nothing. Aboriginal's host-tools.sh and more/record-commands.sh and more/report-recorded-commands.sh give you an example.) >> As I mentioned in the 2013 "why is toybox" youtube video mentioned >> above, a self-hosting development environment needs 4 things: >> >> 1) kernel >> 2) libc >> 3) command line >> 4) compiler > > You need to be able to load source into the environment. I know from > your view, you would do this while the image is still hosted, build > inside the image, and deploy later. Actually I create control-image filesystems that have source and attach them to generic system-images that do what the control-images tell them. That way I don't have to rebuild the system image to do an arbitrary build. > But for my needs, it will not > really be self-hosting until I can pull source *after* deployment. Busybox has wget and it's on the toybox roadmap. I also plan to add rsync to toybox, and you can network mount filesystems (virtfs is built into qemu, I'm told the new smb 2.0 is actually a sane protocol I should write a server for, I was very interested in v9fs until its developers convinced me not to...) Heck, one of the things I've done is uuencode stuff, cut and paste it through a serial console to uudecode. Mostly to get a binary onto a recalcitrant system after the fact, but heck. There's a bunch of ways. I'm pretty sure that was a section of my old 2008 presentation... https://speakerdeck.com/landley/developing-for-non-x86-targets-using-qemu I went over a little of it starting on slide 133, and other bits of it elsewhere in the presentation... > Is there a ftp client in toybox? Not yet, but there's one in busybox. ftpget/ftpput. The various control-image files use that to upload results to the host. (The dev-environment.sh build script launches a temporary ftp server to receive the results, if it's available in the $PATH.) > If there isn't a pre-existing client for > my needs, I'm looking at loading curl in the image. Using wget is > another option but I have no clue how many GNUisms will be needed to > build it (and by extention how much the requirements will "pollute" the > install with stuff that I might not want). Busybox has one, toybox is scheduled to add one. I note that 90% of the sites on the net you go to these days need https, which means you'll need openssl built so toybox can call the openssl comand line program to pipe the wget through. (Not implemented yet but there was a message about it from... Isaac Dunham I think? posted to the toybox mailing list a month or two back.) Rob From rob at landley.net Tue Nov 3 07:53:34 2015 From: rob at landley.net (Rob Landley) Date: Tue, 3 Nov 2015 01:53:34 -0600 Subject: [Aboriginal] Using aboriginal as a bare-bones linux install In-Reply-To: References: <561D5513.3080307@landley.net> <5627E337.1010405@gmail.com> <5635F406.7070705@landley.net> Message-ID: <563867FE.8030507@landley.net> On 11/02/2015 09:31 PM, Avery Payne wrote: > > > On Sun, Nov 1, 2015 at 3:14 AM, Rob Landley > wrote: > > Well, there's still some decisions: > > Does your "absolute minimum" > software include the toolchain, or just boot to a shell prompt? > > > I want the toolchain. In this case, aboriginal will be used as a way to > create a blank-slate no-frills install that I can fit to my desires, > minus the "vision" of a distribution maintainer. Also, if something > breaks, I get to keep both pieces. :) Aboriginal Linux has a vision, it's just based on knowing where we _stop_ and hand off to other projects. :) > If it does should the packages be statically or dynamically linked? > > The base install from aboriginal + the minimum rescue tools needed, all > statically linked. This means I can either (a) recover a botched boot > or (b) rebuild something from source. Either way the system can be > fixed and set up-right with a bit of effort. I should explain a design change I made a year or so back. (I think it's in the release notes but should probably be in the FAQ or something.) There used to be two ways to build aboriginal: with and without the native compiler. The simple-root-filesystem.sh build was busybox+toybox (plus it copies uClibc's shared libraries out of the cross compiler if you were building anything dynamically, since those would be needed on the target at runtime). The native-compiler.sh step built gcc+binutils, uClibc, and three other packages (make, bash, and distcc). This step did NOT run if you selected NO_NATIVE_COMPILER=1. Then a third script, root-filesystem.sh, would combine the two into a single squashfs image. This only ran if you _didn't_ set NO_NATIVE_COMPILER, otherwise the later stages would use build/simple-root-filesytem directly. One of the things this script did was skip the simple-root-filesystem shared libraries and replace them with the native-compiler shared libraries. (They didn't _quite_ match because they were built with slightly different options.) Then I switched to using initramfs, put what used to be simple-root-filesystem into initramfs (either built into the kernel binary for SYSIMAGE_TYPE=initramfs or as a seperate cpio.gz file fed in through the external initrd mechanism if you did SYSIMAGE_TYPE=cpio, the second is the default because it's more useful to people who want to build their own kernel with a different config after the fact, they can reuse the cpio.gz image as-is.) This meant that run-emulator.sh just ran the kernel and didn't mount _any_ partitions (it ran from initramfs), and that dev-environment.sh mounted the native compiler on /dev/hda and had to splice it into the root filesystem at runtime (using a new init script at the top of the native compiler squashfs image and a hook from /sbin/init to check for and call it). (It didn't particularly change native-build.sh.) So instead of selecting a "simple" or "full" environment at compile time, this design change moved the selection to runtime. And made it possible to run the minimal system on a board or emulator that had _no_ block devices. (Meaning you could set up a network mount to access the native compiler if you wanted to, although you don't want to build on nfs. Seriously, even samba is better than nfs for building. Virtfs works. I added nbd-client to busybox and toybox because that works fine...) However, if also means I don't ship a "unified" filesystem image anymore. If you want the root-filesystem and native-compiler combined, you have to do that yourself and package up the result. The more/chroot-splice.sh script gives an example of combining them, but not packaging them. And I believe that runs the script to create the various symlinks to /usr/overlay (where the native-compiler filesystem gets mounted), meaning it expects / to be writeable and runs the script at runtime to write to it. If you wanted a combed hda squashfs like we used to have, one _without_ a bunch of symlinks, you'd have to fiddle with it a bit. That's why I'm going "er..." when you way you want to avoid maintainer design decisions. There are still several decisions I've made within my highly circumscribed design goals. I know where the project _stops_ and what I _don't_ do, but within what I do provide there's a bunch of ways of packaging it: 1) Just root-filesystem or root-filesystem + native compiler. 2) Shipped on two partitions (initramfs and squashfs) or one. If using initramfs, built into the kernel or loaded as initrd=cpio.gz? If one partition, a read-only root filesystem or a writeable one? 3) Each package (busybox, toybox, gcc, binutils, make, bash, distcc) statically or dynamically linked. You seem mostly happy with my defaults (and I did choose them for a reason), but when repackaging for use on something other than qemu, you should be aware what the options are. (I'm elaborating on this point because I don't feel I've documented it sufficiently yet on the website. I should do that...) > Beyond those two requirements, anything else fetched and built will > probably be dynamic due to the laptop's limited 2Gb memory. > > Is your idea of minimal "busybox defconfig" (minimum number of > configuration > decisions) or the combination of busybox+toybox I've worked out as the > minimal set of commands needed to build linux from scratch (plus ones > like "ps" that you kinda miss if they're not there on the command line)? > > More the later than the former. The goal is to pull improvements in > as-needed. If the existing toybox/busybox command is adequate, then > there's no need to build a replacement. I probably want a full shell; > possibly ash, but I'm open to others. It's shipping hush an bash. One of the things that /usr/overlay/init does when splicing in the native-compiler.sqf is delete the /bin/sh symlink that points to hush and replace it with one that points to /usr/overlay/bash. Well, in the upcoming release anyway. Last release the initramfs hasn't got a /bin/sh. (My bad. The switch to hush screwed up the config slightly. In dev-environment.sh you didn't notice, but in run-emulator.sh there wasn't a /bin/sh, just /bin/hush.) > If ps is missing, I'll want that too. Last release was using busybox ps, this upcoming release is using toybox ps. Either way there is a ps. :) > But the built-ins for toybox/busybox would have to be sorely > lacking for me to replace them. I take bug reports for the toybox commands, if there's something specific you need it to do, let me know. (Busybox I mostly leave alone these days, although I did just upgrade to the current busybox version for this release.) > More pragmatically, packaging-wise do you want root-filesystem running > out of initramfs, or do you want it on the sd card filesystem too? > > I have not had to deal with initramfs, root pivots, etc. but I have read > briefly about them. I need to think about this aspect a bit more. I'm > not wild about using a ramdisk - hard to debug, and all that - but if I > need to, I will. Initramfs is not a ramdisk. I wrote some documentation about initramfs years ago which explained the difference between it and a ramdisk: https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt Much later I added initmpfs (using tmpfs instead of ramfs) so you never have to pivot off of it, it can continue to _be_ your real root filesystem, as explained in the patch series introductory message: http://lkml.iu.edu/hypermail/linux/kernel/1306.3/04204.html Presumably I should document _this_ on the website too. :) > > I really, really want to be able to build aboriginal in such a way that > > it builds direct to a SSD, filesystem and all, and install a boot > > loader, and bamo-presto, my very own bare-bones Linux install. > > The system-image.sh stage builds the kernel and packages it into a > tarball, but there's nothing to say you can't add another script at the > end that does the above two cp to a target directory and adds your > bootloader. > > > I could probably write the follow-on script you're talking about, minus > the bootloader portion. It shouldn't be too hard. More like > mind-numbing, trying to get everything copied to the right location with > the right settings. Or you could use the existing script as a template. :) Seriously, most of it's cp -a. I made the layout of the overlay and the layout of the root filesystem it goes on top of line up for a reason. :) > Having an add-on script to automate all of this would be my ideal goal, > as it means I could re-run the process anytime, wait about 15 minutes, > and get a freshly-baked install on a physical hard drive. I know that > runs opposite of what you are trying to achieve with > phones/tablets/whatever, but there's lots of used "obsolete" laptops out > there that could use a 2nd chance and a good home... Oh no, it's fine. Keep in mind I've been doing aboriginal linux for about 15 years now, loooong predating the rise of smartphones. > Given how busy I have been in my personal life lately, I'm not going to > promise that this will happen. But if I do manage to both carve out the > time needed, and actually finish the script, I'll forward a copy to you. Yay! I may merge some stuff upstream if it helps. I note that the sh2eb target in the repo is aimed at real hardware (the numato jcore boards) rather than qemu, and previous things (such as the nail board target) were also real hardware. QEMU is the default packaging choice, but by no means the only one. (Mostly because _installing_ on real hardware and dealing with bootloaders and such is a can of worms I didn't want to open in this project, it's out of scope.) > For chroots I do those fixups manually, see more/setup-chroot.sh for > example. > > Will do. > > You'll want a boot USB stick or similar standing by when you play with > bootloaders. > > Good point; and noted. > > Really, make a $PATH that only has tinycc as the cc and try it and see > what breaks. That's what I did. (No binutils, no nothing. Aboriginal's > host-tools.sh and more/record-commands.sh and > more/report-recorded-commands.sh give you an example.) > > If I can find the time, I'll try. It would be interesting to see how > far it will go, and what issues it uncovers. Been there, done that. It ate about 3 years of my life and then I backed out and did other things. I'm vaguely pondering trying to do _just_ a preprocessor, and having aboriginal's sources/toys/ccwrap.c call that instead of gcc -E, although I have to teach ccwrap to be its own distcc first. Backing up: distcc mode preprocesses every file before it sends it across the network to the distccd compile server instance. It runs gcc -E, copies the outputt across the network, copies the resulting .o file back accross the network, and write sit to the local filesystem to be linked locally later. This means that headers and libraries are all resolved locally, and there's only one set of each, and only one type of code being generated both locally and remotely, so this does _not_ reintroduce most of the complexity of cross compiling even though it's using a cross compiler to do part of the work. The failure mode of distcc is to _not_ distribute a lot of stuff that it could easily distribute, for example any compile that doesn't have -c won't get distributed, so compile-and-link happens natively inside the emulator even though it would be faster to break it into two commands. The thing is distcc doesn't have a very deep understanding of the gcc command line. But ccwrap.c DOES have a deep understanding, it has to in order to rewrite all the $PATH logic and take path searching decisions away from gcc's broken insane code (which is its main job). Meanwhile the normal gcc front-end is mostly just a wrapper: it calls "cpp" to do preprocessing, calls "cc1" to do compiling, and calls either collect2 or ld to do linking. In theory, I could expand ccwrap to completely replace this front end and call the other tools directly. (I'm over halfway there already, most of what ccwrap.c does is take decisions AWAY from gcc, making them explicit.) Anyway, I have a vague goal of teaching ccwrap to always call cpp (or gcc -E) to preprocess a file and then feed that file to cc1 (or pipe it directly between processes). This would let me break off the preprocessor and treat it as an entirely seperate process, and then I could use what I learned from tinycc to write a new preprocessor AND compare its output against gcc -E's output for every single file building the whole of linux from scratch. (Kind of like record-commands.sh does now for command lines.) If nothing else, I know this can run FASTER than gcc -E does because tinycc's preprocessor is already way way faster than gcc's, and preprocesing is one of the big bottlenecks of building under qemu with distcc. (CPU time in the emulator is at much more of a premium than cpu time outside the emulator: not only is emulated code maybe 1/5 the speed of native execution on a good day, but QEMU runs on a single host processor (SMP is emulated in software, emulating the semantics of barriers and IPI between multiple threads on a different hardware architecture is something the qemu guys just didn't feel they could do.) Anyway, once I got a cpp I was happy with, then I could tackle the linker. And if I got THAT done, I could poke at an assembler, objdump, readelf, nm... And if I got all of those working, I could revisit the actual compiler part of qcc. (Meanwhile, llvm+my linker and assembler instead of binutils is a reasonable goal...) But as I said, all this is _after_ toybox 1.0. And it competes with the system bootstrapping stuff (natively building distros on an arbitrary target under aboriginal). And toybox 1.0 isn't going to STOP demanding time, it just goes down to a lower level (like aboriginal has: not abandoned, just not demanding to the partial exclusion of all else...) > Busybox has wget and it's on the toybox roadmap. I also plan to add > rsync to toybox, and you can network mount filesystems > > Wget will suffice to fetch a tarball. From there I'm sure I can get > builds going. I was only looking at curl because of libcurl; I'm making > a bad assumption that other software packages might actually use it (and > reduce the install footprint in the process). You might want to try taking apart the existing build control images. The static-tools one is fairly simple, the lfs-bootstrap one has a lot of common native build infrastructure shared between package builds. > > Is there a ftp client in toybox? > > Not yet, but there's one in busybox. ftpget/ftpput. The various > control-image files use that to upload results to the host. > > Perfect! The script to launch the host ftp sever is in native-build.sh by the way. > I note that 90% of the sites on the net you go to these days need https, > which means you'll need openssl built so toybox can call the openssl > comand line program to pipe the wget through. (Not implemented yet but > there was a message about it from... Isaac Dunham I think? posted to the > toybox mailing list a month or two back.) > > Just how insane would I be to attempt to shim libressl in place of > openssl, if only because of openssl's problems? Presumably fairly straightforward? "openssl s_client -quiet -connect" is basically encrypted netcat, that's the part I need installed on the system for toybox wget to do https. Then to implement wget I take the information from http://podium.unsupported.io/#/ and https://en.wikipedia.org/wiki/List_of_HTTP_header_fields and maybe https://en.wikipedia.org/wiki/Server_Name_Indication and/or https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header ... Anyway: todo item. Rob From ska-devel at skarnet.org Tue Nov 3 10:43:50 2015 From: ska-devel at skarnet.org (Laurent Bercot) Date: Tue, 3 Nov 2015 11:43:50 +0100 Subject: [Aboriginal] Using aboriginal as a bare-bones linux install In-Reply-To: <563867FE.8030507@landley.net> References: <561D5513.3080307@landley.net> <5627E337.1010405@gmail.com> <5635F406.7070705@landley.net> <563867FE.8030507@landley.net> Message-ID: <56388FE6.6080808@skarnet.org> On 03/11/2015 08:53, Rob Landley wrote: > Meanwhile the normal gcc front-end is mostly just a wrapper: it calls > "cpp" to do preprocessing, calls "cc1" to do compiling, and calls either > collect2 or ld to do linking. In theory, I could expand ccwrap to > completely replace this front end and call the other tools directly. > (I'm over halfway there already, most of what ccwrap.c does is take > decisions AWAY from gcc, making them explicit.) Whenever you get to it, please add a runtime option / configuration variable / something to ccwrap so that it can use a different musl from the one you're providing (which is currently hardcoded). musl evolves faster than Aboriginal toolchains, and I'd like to be able to use Aboriginal toolchains directly to link stuff against my current version of musl. Right now, in order to do that, I have to ignore ccwrap and make my own wrappers to call rawcc, so I lose all the functionality you can add to ccwrap. If what I want cannot be achieved without some degree of leakage (which is probably the reason why you used such a big hammer in ccwrap in the first place), then I'm okay with some degree of leakage when the option is used, since I'm only using * the precompiled toolchains * on glibc-less systems. Flexibility is more important to me than perfect relocatability here. Thanks, -- Laurent From rob at landley.net Thu Nov 5 10:20:52 2015 From: rob at landley.net (Rob Landley) Date: Thu, 5 Nov 2015 04:20:52 -0600 Subject: [Aboriginal] 1.4.3 is out. Message-ID: <563B2D84.4010608@landley.net> Release notes on the website: http://landley.net/aboriginal Rob From rob at landley.net Thu Nov 5 16:07:59 2015 From: rob at landley.net (Rob Landley) Date: Thu, 5 Nov 2015 10:07:59 -0600 Subject: [Aboriginal] Sigh. Bugfix. Message-ID: <563B7EDF.2000400@landley.net> I built the release with a patch locally applied but not checked in. I need to add that to my release.txt checklist. (Apparently checking for local patches before I delete a working directory is habit, but checking before a _release_ isn't? Sigh.) It's http://landley.net/hg/aboriginal/rev/1809 and it only matters for building cross-compiler-sh2eb (not simple-cross-compiler, the non-simple one is portable and meant to be used outside the build, the simple one is just used to build the root filesystem and kernel and such, and then discarded). The posted binaries work and you can rebuild sh2eb from the current repo, but I should cut another release next week anyway. (If nothing else I'm still two kernel versions behind, so an update to 4.1 is a good catch-up move.) Please test the current stuff if there's anything you want fixed. Rob From brendan.simon at etrix.com.au Sat Nov 7 11:01:08 2015 From: brendan.simon at etrix.com.au (Brendan Simon (eTRIX)) Date: Sat, 7 Nov 2015 22:01:08 +1100 Subject: [Aboriginal] Can Aboriginal Linux be used to create a Debian rootfs In-Reply-To: <5632FB37.4070907@landley.net> References: <5632C2D5.7090109@etrix.com.au> <5632FB37.4070907@landley.net> Message-ID: <563DD9F4.4070103@etrix.com.au> On 30/10/2015 4:08 PM, Rob Landley wrote: > On 10/29/2015 08:07 PM, Brendan Simon (eTRIX) wrote: >> Hi. Just stumbled across Aboriginal Linux and it looks like an great >> project :) >> >> I am trying to create a Debian rootfs for an embedded system that uses >> Dual ARM A9 cores (Xilinx Zynq SoC). >> >> I'm currently trying to use multistrap (from the Emdebian project) and >> am now trying moved to brickstrap (a fork of polystrap which is a fork >> of multistrap). Brickstrap does everything in userspace via proot and >> uses QEMU for the final configuration phase. >> >> I was wondering if it is possible to use Aboriginal Linux to do some or >> all of this? i.e. create a Debian rootfs from debian binary packages. >> Would it be simpler? > In theory, yes. In practice, it's a big todo item as explained here: > > http://landley.net/aboriginal/about.html#hairball > > For Debian I've been vaguely looking at: > > https://wiki.debian.org/HelmutGrohne/rebootstrap I've had success with Brickstrap (well in creating a rootfs, but not tested yet). Brickstrap now uses Linux user namespaces instead of proot. https://blog.mister-muffin.de/category/linux/ The project/tool is called 'user-unshare', but it looks like it is going to be call uroot (as it is intended as a replacement for proot). https://gitlab.mister-muffin.de/josch/user-unshare Brickstrap uses user-unshare, multistrap and libguestfs to create my rootfs from .deb packages via apt, and configure them, creates a disk image, all without requiring superuser rights :) Cheers, Brendan. From james.w.mcmechan at gmail.com Sun Nov 8 20:03:38 2015 From: james.w.mcmechan at gmail.com (James McMechan) Date: Sun, 8 Nov 2015 12:03:38 -0800 Subject: [Aboriginal] I am likely forgetting something obvious Message-ID: Hi everyone, It has been a while since I last tested rebuilding in a chroot, and my build is failing I had done what I remembered as my procedure: # more/chroot-splice.sh x86_64 /tmp then copied it over to my /mnt/build directory and mounted proc,sys,dev and then unpacked aboriginal into it # tar xf ~mcmechan/Downloads/aboriginal-1.4.3.tar.gz -C /mnt/build when I start up my chroot # chroot /mnt/build /bin/bash # cd /aboriginal-1.4.3/ # ./build.sh x86_64 === Download source code. wget: bad address 'uclibc.org'z2 wget: bad address 'landley.net' Connecting to 127.0.0.1 (127.0.0.1:80) wget: server returned error: HTTP/1.0 404 Not Found Could not download uClibc-0.9.33.2.tar.bz2 Exiting due to errors (host download ) real 0m10.098s user 0m0.000s sys 0m0.000s ##### Ok, I have now checked and downloads have not worked for some time at least back to 1.2.9 but if I have copied in the package directory 1.2.9 will build busybox & toybox and 1.4.3 fails with === busybox (host host-tools) Extracting 'busybox'........................................................................... Snapshot 'busybox'... sh: set: -e: invalid option HOSTCC scripts/basic/fixdep sh: set: -e: invalid option HOSTCC scripts/basic/split-include sh: set: -e: invalid option HOSTCC scripts/basic/docproc sh: local: not in a function sed: : No such file or directory sed: : No such file or directory Ok that appears to be hush so with # ln -sf /bin/bash /bin/sh I get a little further, the more/chroot-splice.sh should likely do this and it would be nice if it did not require the second option, I at least am just wanting a plain chroot without a mount inside. so using bash 1.4.3 dies in toybox with: Extracting 'toybox'............... Applying /aboriginal-1.4.3/sources/patches/toybox-grep.patch patching toys/posix/grep.c Snapshot 'toybox'... scripts/genconfig.sh cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \ -DPROJECT_NAME=\"ToyBox\" kconfig/conf -D /dev/null Config.in > /dev/null scripts/make.sh scripts/make.sh: line 7: set: pipefail: invalid option name Generate headers from toys/*/*.c... generated/newtoys.h Library probe....... Make generated/config.h from .config. generated/flags.h generated/globals.h generated/help.h Compile toybox.scripts/make.sh: line 1: jobs: command not found .scripts/make.sh: line 1: jobs: command not found .scripts/make.sh: line 1: jobs: command not found so should I be looking to turn on pipefail & jobs in bash or do I need to fix up the build script. 1.2.9 did not have jobs or pipefail. Anyway I am putting it out for comment as I seem to have forgotten a step somewhere. Jim McMechan -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick at gentoo.org Mon Nov 9 09:10:20 2015 From: patrick at gentoo.org (Patrick Lauer) Date: Mon, 9 Nov 2015 10:10:20 +0100 Subject: [Aboriginal] I am likely forgetting something obvious In-Reply-To: References: Message-ID: <564062FC.3070902@gentoo.org> On 11/08/2015 09:03 PM, James McMechan wrote: > Hi everyone, > It has been a while since I last tested rebuilding in a chroot, and my > build is failing > I had done what I remembered as my procedure: > # more/chroot-splice.sh x86_64 /tmp > then copied it over to my /mnt/build directory and mounted proc,sys,dev > and then unpacked aboriginal into it > # tar xf ~mcmechan/Downloads/aboriginal-1.4.3.tar.gz -C /mnt/build > when I start up my chroot > # chroot /mnt/build /bin/bash > # cd /aboriginal-1.4.3/ > # ./build.sh x86_64 > === Download source code. > wget: bad address 'uclibc.org 'z2 > wget: bad address 'landley.net ' bad or missing /etc/resolv.conf in the chroot? > Connecting to 127.0.0.1 (127.0.0.1:80 ) > wget: server returned error: HTTP/1.0 404 Not Found > Could not download uClibc-0.9.33.2.tar.bz2 > > Exiting due to errors (host download ) > > real 0m10.098s > user 0m0.000s > sys 0m0.000s > > ##### > Ok, I have now checked and downloads have not worked for some time at > least back to 1.2.9 > > but if I have copied in the package directory 1.2.9 will build busybox > & toybox > and 1.4.3 fails with > > === busybox (host host-tools) > Extracting > 'busybox'........................................................................... > Snapshot 'busybox'... > sh: set: -e: invalid option > HOSTCC scripts/basic/fixdep > sh: set: -e: invalid option > HOSTCC scripts/basic/split-include > sh: set: -e: invalid option > HOSTCC scripts/basic/docproc > sh: local: not in a function > sed: : No such file or directory > sed: : No such file or directory > > Ok that appears to be hush so with > # ln -sf /bin/bash /bin/sh > I get a little further, the more/chroot-splice.sh should likely do > this and it would be nice if it did not require the second option, I > at least am just wanting a plain chroot without a mount inside. You shouldn't require bash. Maybe hush is not posixy enough? If you want a small shell I think busybox sh and mksh are currently the easiest options. > > so using bash 1.4.3 dies in toybox with: > > Extracting 'toybox'............... > Applying /aboriginal-1.4.3/sources/patches/toybox-grep.patch > patching toys/posix/grep.c > Snapshot 'toybox'... > scripts/genconfig.sh > cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \ > -DPROJECT_NAME=\"ToyBox\" > kconfig/conf -D /dev/null Config.in > /dev/null > scripts/make.sh > scripts/make.sh: line 7: set: pipefail: invalid option name > Generate headers from toys/*/*.c... > generated/newtoys.h Library probe....... > Make generated/config.h from .config. > generated/flags.h generated/globals.h generated/help.h > Compile toybox.scripts/make.sh: line 1: jobs: command not found > .scripts/make.sh: line 1: jobs: command not found > .scripts/make.sh: line 1: jobs: command not found > > so should I be looking to turn on pipefail & jobs in bash or do I need > to fix up the build script. 1.2.9 did not have jobs or pipefail. Yes, or use another shell. Disabling random features might not be a good idea ;) Have fun, Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.w.mcmechan at gmail.com Mon Nov 9 20:57:01 2015 From: james.w.mcmechan at gmail.com (James McMechan) Date: Mon, 9 Nov 2015 12:57:01 -0800 Subject: [Aboriginal] I am likely forgetting something obvious In-Reply-To: <564062FC.3070902@gentoo.org> References: <564062FC.3070902@gentoo.org> Message-ID: Thank you, Putting in the missing /etc/resolv.conf fixed the download problem. Now I just need to figure out why toybox is failing to compile or how to turn job control back on in bash. On Mon, Nov 9, 2015 at 1:10 AM, Patrick Lauer wrote: > > > On 11/08/2015 09:03 PM, James McMechan wrote: > > Hi everyone, > It has been a while since I last tested rebuilding in a chroot, and my > build is failing > I had done what I remembered as my procedure: > # more/chroot-splice.sh x86_64 /tmp > then copied it over to my /mnt/build directory and mounted proc,sys,dev > and then unpacked aboriginal into it > # tar xf ~mcmechan/Downloads/aboriginal-1.4.3.tar.gz -C /mnt/build > when I start up my chroot > # chroot /mnt/build /bin/bash > # cd /aboriginal-1.4.3/ > # ./build.sh x86_64 > === Download source code. > wget: bad address 'uclibc.org'z2 > wget: bad address 'landley.net' > > > bad or missing /etc/resolv.conf in the chroot? > > Connecting to 127.0.0.1 (127.0.0.1:80) > wget: server returned error: HTTP/1.0 404 Not Found > Could not download uClibc-0.9.33.2.tar.bz2 > > Exiting due to errors (host download ) > > real 0m10.098s > user 0m0.000s > sys 0m0.000s > > ##### > Ok, I have now checked and downloads have not worked for some time at > least back to 1.2.9 > > but if I have copied in the package directory 1.2.9 will build busybox & > toybox > and 1.4.3 fails with > > === busybox (host host-tools) > Extracting > 'busybox'........................................................................... > Snapshot 'busybox'... > sh: set: -e: invalid option > HOSTCC scripts/basic/fixdep > sh: set: -e: invalid option > HOSTCC scripts/basic/split-include > sh: set: -e: invalid option > HOSTCC scripts/basic/docproc > sh: local: not in a function > sed: : No such file or directory > sed: : No such file or directory > > Ok that appears to be hush so with > # ln -sf /bin/bash /bin/sh > I get a little further, the more/chroot-splice.sh should likely do this > and it would be nice if it did not require the second option, I at least am > just wanting a plain chroot without a mount inside. > > You shouldn't require bash. Maybe hush is not posixy enough? If you want a > small shell I think busybox sh and mksh are currently the easiest options. > > > so using bash 1.4.3 dies in toybox with: > > Extracting 'toybox'............... > Applying /aboriginal-1.4.3/sources/patches/toybox-grep.patch > patching toys/posix/grep.c > Snapshot 'toybox'... > scripts/genconfig.sh > cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \ > -DPROJECT_NAME=\"ToyBox\" > kconfig/conf -D /dev/null Config.in > /dev/null > scripts/make.sh > scripts/make.sh: line 7: set: pipefail: invalid option name > Generate headers from toys/*/*.c... > generated/newtoys.h Library probe....... > Make generated/config.h from .config. > generated/flags.h generated/globals.h generated/help.h > Compile toybox.scripts/make.sh: line 1: jobs: command not found > .scripts/make.sh: line 1: jobs: command not found > .scripts/make.sh: line 1: jobs: command not found > > so should I be looking to turn on pipefail & jobs in bash or do I need to > fix up the build script. 1.2.9 did not have jobs or pipefail. > > Yes, or use another shell. Disabling random features might not be a good > idea ;) > > Have fun, > > Patrick > > _______________________________________________ > Aboriginal mailing list > Aboriginal at lists.landley.net > http://lists.landley.net/listinfo.cgi/aboriginal-landley.net > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob at landley.net Sat Nov 14 03:44:22 2015 From: rob at landley.net (Rob Landley) Date: Fri, 13 Nov 2015 21:44:22 -0600 Subject: [Aboriginal] aboriginal changes In-Reply-To: <56469F76.6020001@se-instruments.com> References: <56469F76.6020001@se-instruments.com> Message-ID: <5646AE16.9010702@landley.net> On 11/12/2015 08:23 PM, Geoff Salmon wrote: > Hi Rob > > I've been experimenting with aboriginal some more. The attached patch > adds support for applying patches held in an external directory. The > use-case I had in mind was for things we don't want to open source yet. Sorry for the delay, when I haven't been testing the 4.2 kernel for the next release I'm trying to work out the design implications of your patch. (Plus I'm traveling through california next week and need to pack and such.) The minor issue is a consistent naming scheme for the variables. The "overlay" directory is basically an extra sources/root-filesystem, this is an extra sources/patches, and the CROSS_COMPILER_PATH I added is sort of a third extra path to something. Trying to think of a consistent naming scheme between them that lets people know what behavior to expect after they've learned one. (And really, PREFERRED_MIRROR is another one, but that's a url not a path in the local filesystem, so it's already outside the pattern.) There's enough of these now maybe it should be consistent. Hmmm... LOCAL? EXTRA? MY? "MY_CROSS_COMPILER=/path, MY_PATCHES=/path, MY_ROOT_FILES=/path" sorta works. EXTRA_CROSS_COMPILER doesn't make sense, and LOCAL_ROOT_FILES is slightly weird... Sigh. Naming things, cache invalidation, and off by one errors... The other thing is that your external directory uses a subdirectory for the patch name, and the original one uses a prefix. I'd kinda like that to be consistent. (I'll take a stab at fixing it up myself.) Just to confirm: you're ok with the built-in patches always being applied before the local patches? (PREFERRED_MIRROR gets searched first and the original stuff is a fallback, but as I said it's an oddball. CROSS_COMPILER_PATH is currently a replacement, and providing a directory under packages/ acts as a replacement that prevents automatic patching. OVERLAY is applied after. Consistent semantics to go with consistent naming would be nice, and "searched after the existing path" is the consistent one. The change shouldn't screw up the cross compiler path stuff as long as a toolset with the specified prefix is in the path somewhere.) > We can have patches in the soc_sw/aboriginal_patches/linux/ and when > they graduate to something we want to share we can move them to > aboriginal's jcore patch, or a separate aboriginal patch, and then as > they are upstreamed they can be removed from aboriginal. Does that > workflow make sense? Sounds good to me. > What do you think of the way I implemented it? Comments above. I've applied it locally but haven't pushed it pending probably renaming the variables. (Best I've come up with so far is the MY_ prefixes.) > I'm still not sure what to do about partial builds. Losing the ability > to test small, uncommitted changes to the kernel without full rebuilds > seems like too large a sacrifice. Out-of-tree kernel builds with INITRAMFS_SOURCE="/path/to/build/root-filesystem-sh2eb" are probably your best bet. I need to have system-image.sh work for the sh2eb target so open source developers have an easy way to get started, and it helps you feed something stable to the jenkins builds, but for your local test iterations system-image.sh is overkill. Note: to extract the kernel config I'm using out of the build (combining the source/baseconfig-linux and sources/targets/sh2eb LINUX_CONFIG section, then expanding the miniconfig to full .config) do: cd ~/aboriginal more/test.sh sh2eb getconfig linux > ~/linux/sh2eb.miniconfig cd ~/linux make ARCH=sh allnoconfig KCONFIG_ALLCONFIG=sh2eb.miniconfig (It should also be in build/native-compiler-sh2eb/usr/src/config-linux but that's a REALLY awkward place for it and I need to think of a better one. Problem is system-image.sh builds the kernel after the root filesystem's been packaged up, so there's a sequencing issue. It's fixable, I just haven't done it yet. So many rough edges to polish, even after all these years...) > Here's a couple random things I noticed in aboriginal that you may want > to fix: > > This sed > > echo "Build stages:" > sed -n 's/#.*//;s at .*[.]/\([^.]*\)[.]sh.*@\1 at p' "$0" | uniq | xargs echo > > in the help output of "build.sh" prints the following for me > > Build stages: > $STAGE > > > The README refers to the file "configure" but it's named "config". Added to the todo list. (I've also got the "fix armv6l" todo item which is basically revive the kernel patch so I can tell linux that the versatile board can take a different processor, and a pile of documentation updates...) Thanks, Rob From gsalmon at se-instruments.com Sun Nov 15 23:54:02 2015 From: gsalmon at se-instruments.com (Geoff Salmon) Date: Sun, 15 Nov 2015 18:54:02 -0500 Subject: [Aboriginal] aboriginal changes In-Reply-To: <5646AE16.9010702@landley.net> References: <56469F76.6020001@se-instruments.com> <5646AE16.9010702@landley.net> Message-ID: <56491B1A.7070707@se-instruments.com> On 11/13/2015 10:44 PM, Rob Landley wrote: > There's enough of these now maybe it should be consistent. Hmmm... > LOCAL? EXTRA? MY? "MY_CROSS_COMPILER=/path, MY_PATCHES=/path, > MY_ROOT_FILES=/path" sorta works. EXTRA_CROSS_COMPILER doesn't make > sense, and LOCAL_ROOT_FILES is slightly weird... Sigh. Naming things, > cache invalidation, and off by one errors... Yeah, naming is hard. I like your MY_ suggestion. It communicates the unofficial nature of what you're doing. Anyone setting a MY_ variable should be aware they're introducing something that wasn't tested upstream. > The other thing is that your external directory uses a subdirectory for > the patch name, and the original one uses a prefix. I'd kinda like that > to be consistent. (I'll take a stab at fixing it up myself.) I think I was reasoning it would be easier to use a tool like quilt if I could point it at a directory containing only the relevant patches. I admit I haven't actually tried the workflow with quilt yet though. > Just to confirm: you're ok with the built-in patches always being > applied before the local patches? Yeah, I think that makes the most sense. - Geoff From gsalmon at se-instruments.com Mon Nov 16 15:46:04 2015 From: gsalmon at se-instruments.com (Geoff Salmon) Date: Mon, 16 Nov 2015 10:46:04 -0500 Subject: [Aboriginal] aboriginal changes In-Reply-To: <56491B1A.7070707@se-instruments.com> References: <56469F76.6020001@se-instruments.com> <5646AE16.9010702@landley.net> <56491B1A.7070707@se-instruments.com> Message-ID: <5649FA3C.2080408@se-instruments.com> On 15-11-15 06:54 PM, Geoff Salmon wrote: > On 11/13/2015 10:44 PM, Rob Landley wrote: >> There's enough of these now maybe it should be consistent. Hmmm... >> LOCAL? EXTRA? MY? "MY_CROSS_COMPILER=/path, MY_PATCHES=/path, >> MY_ROOT_FILES=/path" sorta works. EXTRA_CROSS_COMPILER doesn't make >> sense, and LOCAL_ROOT_FILES is slightly weird... Sigh. Naming things, >> cache invalidation, and off by one errors... Do we also need MY_UCLIBC_CONFIG and MY_LINUX_CONFIG? If MY_PATCHES lets us add things to the kernel, we may also need to enable them in the config. - Geoff From gsalmon at se-instruments.com Mon Nov 16 16:57:45 2015 From: gsalmon at se-instruments.com (Geoff Salmon) Date: Mon, 16 Nov 2015 11:57:45 -0500 Subject: [Aboriginal] aboriginal changes In-Reply-To: <5649FA3C.2080408@se-instruments.com> References: <56469F76.6020001@se-instruments.com> <5646AE16.9010702@landley.net> <56491B1A.7070707@se-instruments.com> <5649FA3C.2080408@se-instruments.com> Message-ID: <564A0B09.8080005@se-instruments.com> On 15-11-16 10:46 AM, Geoff Salmon wrote: > On 15-11-15 06:54 PM, Geoff Salmon wrote: >> On 11/13/2015 10:44 PM, Rob Landley wrote: >>> There's enough of these now maybe it should be consistent. Hmmm... >>> LOCAL? EXTRA? MY? "MY_CROSS_COMPILER=/path, MY_PATCHES=/path, >>> MY_ROOT_FILES=/path" sorta works. EXTRA_CROSS_COMPILER doesn't make >>> sense, and LOCAL_ROOT_FILES is slightly weird... Sigh. Naming things, >>> cache invalidation, and off by one errors... > > Do we also need MY_UCLIBC_CONFIG and MY_LINUX_CONFIG? If MY_PATCHES lets > us add things to the kernel, we may also need to enable them in the config. Hmm, alternatively load_target() could optionally source a MY_TARGET file, but that might be step too far... I see some directories are only set if not blank export_if_blank SOURCES="$TOP/sources" export_if_blank SRCDIR="$TOP/packages" export_if_blank PATCHDIR="$SOURCES/patches" maybe CONFIG_DIR should be set with export_if_blank inside load_target() so the config can be replaced entirely if desired instead of adding MY_UCLIBC_CONFIG and MY_LINUX_CONFIG. Anyone replacing the config target script entirely would need to pay close attention to any changes in the upstream script though. -- Geoff Salmon, Software Developer Smart Energy Instruments Inc. gsalmon at se-instruments.com +1 647-476-2112 www.se-instruments.com Measure Once, Use Everywhere From gsalmon at se-instruments.com Mon Nov 16 21:11:22 2015 From: gsalmon at se-instruments.com (Geoff Salmon) Date: Mon, 16 Nov 2015 16:11:22 -0500 Subject: [Aboriginal] aboriginal changes In-Reply-To: <564A0B09.8080005@se-instruments.com> References: <56469F76.6020001@se-instruments.com> <5646AE16.9010702@landley.net> <56491B1A.7070707@se-instruments.com> <5649FA3C.2080408@se-instruments.com> <564A0B09.8080005@se-instruments.com> Message-ID: <564A467A.2000007@se-instruments.com> Hi Rob There's a problem with my original patch that this thread stared with. In sources/download_functions.sh, I made this change - for i in "$SHA1TAR" $(sha1file "$PATCHDIR/$PACKAGE"-*.patch 2>/dev/null) + for i in "$SHA1TAR" $(sha1file $(gather_patches)) but if there are no patches for a package then gather_patches will return nothing and sha1file will hang because sha1sum is waiting for input. This happens when building busybox. The original version relied on sha1sum 'sources/busybox-*.patch' failing immediately and ignored the error with 2>/dev/null. An alternate way to deal with gather_patches returning nothing would be to modify sha1file like this: sha1file() { - sha1sum "$@" | awk '{print $1}' + sha1sum /dev/null "$@" | tail -n +2 | awk '{print $1}' } Except then you can't pipe input into sha1file. I don't see this being done anywhere, but maybe it's something you want to support. - Geoff From rob at landley.net Tue Nov 17 14:24:42 2015 From: rob at landley.net (Rob Landley) Date: Tue, 17 Nov 2015 08:24:42 -0600 Subject: [Aboriginal] aboriginal changes In-Reply-To: <5649FA3C.2080408@se-instruments.com> References: <56469F76.6020001@se-instruments.com> <5646AE16.9010702@landley.net> <56491B1A.7070707@se-instruments.com> <5649FA3C.2080408@se-instruments.com> Message-ID: <564B38AA.7090904@landley.net> On 11/16/2015 09:46 AM, Geoff Salmon wrote: > On 15-11-15 06:54 PM, Geoff Salmon wrote: >> On 11/13/2015 10:44 PM, Rob Landley wrote: >>> There's enough of these now maybe it should be consistent. Hmmm... >>> LOCAL? EXTRA? MY? "MY_CROSS_COMPILER=/path, MY_PATCHES=/path, >>> MY_ROOT_FILES=/path" sorta works. EXTRA_CROSS_COMPILER doesn't make >>> sense, and LOCAL_ROOT_FILES is slightly weird... Sigh. Naming things, >>> cache invalidation, and off by one errors... > > Do we also need MY_UCLIBC_CONFIG and MY_LINUX_CONFIG? If MY_PATCHES lets > us add things to the kernel, we may also need to enable them in the config. Hmmm, maybe. There used to be infrastructure for this (sources/targets/subdirs) but nobody used it and it got simplified away... Would you want it to point to an external full .config, or a miniconfig snippet on top of baseconfig like the others? (And would it replace the sources/targets snippet or supplement it?) Rob From rob at landley.net Tue Nov 17 14:40:07 2015 From: rob at landley.net (Rob Landley) Date: Tue, 17 Nov 2015 08:40:07 -0600 Subject: [Aboriginal] aboriginal changes In-Reply-To: <564A0B09.8080005@se-instruments.com> References: <56469F76.6020001@se-instruments.com> <5646AE16.9010702@landley.net> <56491B1A.7070707@se-instruments.com> <5649FA3C.2080408@se-instruments.com> <564A0B09.8080005@se-instruments.com> Message-ID: <564B3C47.1060209@landley.net> On 11/16/2015 10:57 AM, Geoff Salmon wrote: > On 15-11-16 10:46 AM, Geoff Salmon wrote: >> On 15-11-15 06:54 PM, Geoff Salmon wrote: >>> On 11/13/2015 10:44 PM, Rob Landley wrote: >>>> There's enough of these now maybe it should be consistent. Hmmm... >>>> LOCAL? EXTRA? MY? "MY_CROSS_COMPILER=/path, MY_PATCHES=/path, >>>> MY_ROOT_FILES=/path" sorta works. EXTRA_CROSS_COMPILER doesn't make >>>> sense, and LOCAL_ROOT_FILES is slightly weird... Sigh. Naming things, >>>> cache invalidation, and off by one errors... >> >> Do we also need MY_UCLIBC_CONFIG and MY_LINUX_CONFIG? If MY_PATCHES lets >> us add things to the kernel, we may also need to enable them in the >> config. > > Hmm, alternatively load_target() could optionally source a MY_TARGET > file, but that might be step too far... > > > I see some directories are only set if not blank > > export_if_blank SOURCES="$TOP/sources" > export_if_blank SRCDIR="$TOP/packages" > export_if_blank PATCHDIR="$SOURCES/patches" While this was always part of the design, it was never tested. It looks like the environment variable sanitizing code in sources/variables.sh needs to be taught to look for export_if_blank invocations in sources/variables.sh. Hmmm, and "CC" is tricky, we export_if_blank it there but I vaguely recall that part of the reason for the original sanitization is that when Wolfgang Denk (the u-boot guy) tried aboriginal years ago it blew up for him in part because he had all sorts of environment variables set, one of which was CC=/absolute/path/to/an/inappropriate/compiler/executable and it blew up. Possibly that one should be unconditionally set. You can NO_SANITIZE_ENVIRONMENT and play with that now though. Might work? If not I can take a look wednesday. (I need to test and check in the MY_XXX stuff then anyway.) > maybe CONFIG_DIR should be set with export_if_blank inside load_target() > so the config can be replaced entirely if desired instead of adding > MY_UCLIBC_CONFIG and MY_LINUX_CONFIG. Anyone replacing the config target > script entirely would need to pay close attention to any changes in the > upstream script though. You can also locally have sources/targets/mytarget and just set CROSS_TARGET to the right "thingy-unknown-linux" tuple in there. (It defaults to ${ARCH}-unknown-linux with ARCH being the script name, I believe that's the only use we make of the actual name outside of naming directories and the compiler prefix and such.) Rob From rob at landley.net Tue Nov 17 14:55:23 2015 From: rob at landley.net (Rob Landley) Date: Tue, 17 Nov 2015 08:55:23 -0600 Subject: [Aboriginal] aboriginal changes In-Reply-To: <564A467A.2000007@se-instruments.com> References: <56469F76.6020001@se-instruments.com> <5646AE16.9010702@landley.net> <56491B1A.7070707@se-instruments.com> <5649FA3C.2080408@se-instruments.com> <564A0B09.8080005@se-instruments.com> <564A467A.2000007@se-instruments.com> Message-ID: <564B3FDB.3010307@landley.net> On 11/16/2015 03:11 PM, Geoff Salmon wrote: > Hi Rob > > There's a problem with my original patch that this thread stared with. > > In sources/download_functions.sh, I made this change > > - for i in "$SHA1TAR" $(sha1file "$PATCHDIR/$PACKAGE"-*.patch > 2>/dev/null) > + for i in "$SHA1TAR" $(sha1file $(gather_patches)) > > but if there are no patches for a package then gather_patches will > return nothing and sha1file will hang because sha1sum is waiting for > input. This happens when building busybox. The original version relied on > > sha1sum 'sources/busybox-*.patch' > > failing immediately and ignored the error with 2>/dev/null. > > An alternate way to deal with gather_patches returning nothing would be > to modify sha1file like this: > > sha1file() > { > - sha1sum "$@" | awk '{print $1}' > + sha1sum /dev/null "$@" | tail -n +2 | awk '{print $1}' > } > > Except then you can't pipe input into sha1file. I don't see this being > done anywhere, but maybe it's something you want to support. sha1sum /dev/null gives an sha1sum, probably doesn't hurt anything in this case but it's a behavior change. Sounds like an existence test is needed somewhere. sha1file exists because there's no "just give me the darn hash without appending the filename to it" command line option for sha1sum. (Ok, I added sha1sum -b to toybox, but I shouldn't assume that's there...) Ok, let's say sha1file outputs the sha1sums of its arguments, and with no arguments outputs nothing. That means the existence test needs to go in sha1file, so a for loop over the returned values. (Which would break if a filename has a space in it but that's already the case...) I'll try to check in a fix today. Thanks, Rob From gsalmon at se-instruments.com Tue Nov 17 15:04:45 2015 From: gsalmon at se-instruments.com (Geoff Salmon) Date: Tue, 17 Nov 2015 10:04:45 -0500 Subject: [Aboriginal] aboriginal changes In-Reply-To: <564B3FDB.3010307@landley.net> References: <56469F76.6020001@se-instruments.com> <5646AE16.9010702@landley.net> <56491B1A.7070707@se-instruments.com> <5649FA3C.2080408@se-instruments.com> <564A0B09.8080005@se-instruments.com> <564A467A.2000007@se-instruments.com> <564B3FDB.3010307@landley.net> Message-ID: <564B420D.9090806@se-instruments.com> On 15-11-17 09:55 AM, Rob Landley wrote: >> > sha1file() >> > { >> >- sha1sum "$@" | awk '{print $1}' >> >+ sha1sum /dev/null "$@" | tail -n +2 | awk '{print $1}' >> > } >> > >> >Except then you can't pipe input into sha1file. I don't see this being >> >done anywhere, but maybe it's something you want to support. I was wrong about not supporting piping into sha1file. You just have to be explicit about it and pass a "-" argument. > sha1sum /dev/null gives an sha1sum, probably doesn't hurt anything in > this case but it's a behavior change. Sounds like an existence test is > needed somewhere. sha1sum /dev/null does always return a hash, but the tail -n +2 ignores it. > Ok, let's say sha1file outputs the sha1sums of its arguments, and with > no arguments outputs nothing. That means the existence test needs to go > in sha1file, so a for loop over the returned values. (Which would break > if a filename has a space in it but that's already the case...) Are you saying the existence check is needed to handle the no args case or to handle non-existent args? If it's for the no args case, I think sha1sum /dev/null | tail -n +2 is simpler. If you want to support non-existent args, then 2>/dev/null either inside or outside sha1file as before seems fine. - Geoff From rob at landley.net Wed Nov 18 15:22:41 2015 From: rob at landley.net (Rob Landley) Date: Wed, 18 Nov 2015 09:22:41 -0600 Subject: [Aboriginal] aboriginal changes In-Reply-To: <564B420D.9090806@se-instruments.com> References: <56469F76.6020001@se-instruments.com> <5646AE16.9010702@landley.net> <56491B1A.7070707@se-instruments.com> <5649FA3C.2080408@se-instruments.com> <564A0B09.8080005@se-instruments.com> <564A467A.2000007@se-instruments.com> <564B3FDB.3010307@landley.net> <564B420D.9090806@se-instruments.com> Message-ID: <564C97C1.9060707@landley.net> On 11/17/2015 09:04 AM, Geoff Salmon wrote: > On 15-11-17 09:55 AM, Rob Landley wrote: >>> > sha1file() >>> > { >>> >- sha1sum "$@" | awk '{print $1}' >>> >+ sha1sum /dev/null "$@" | tail -n +2 | awk '{print $1}' >>> > } >>> > >>> >Except then you can't pipe input into sha1file. I don't see this being >>> >done anywhere, but maybe it's something you want to support. > > I was wrong about not supporting piping into sha1file. You just have to > be explicit about it and pass a "-" argument. > >> sha1sum /dev/null gives an sha1sum, probably doesn't hurt anything in >> this case but it's a behavior change. Sounds like an existence test is >> needed somewhere. > > sha1sum /dev/null does always return a hash, but the tail -n +2 ignores it. You're right, that fixes it. Merged and pushed. Rob From gsalmon at se-instruments.com Wed Nov 18 17:30:56 2015 From: gsalmon at se-instruments.com (Geoff Salmon) Date: Wed, 18 Nov 2015 12:30:56 -0500 Subject: [Aboriginal] Use ${CC_PREFIX} in root-filesystem.sh and PS1 change Message-ID: <564CB5D0.9050209@se-instruments.com> Hi Rob There are some uses of ${ARCH}-cc and ${ARCH}-strip remaining in root-filesystem.sh. Below is a patch to replace them with ${CC_PREFIX}. Also I'll first sneak in a small patch to change PS1 in init.sh to expand the HOST and CPUS vars. - Geoff diff --git a/sources/root-filesystem/sbin/init.sh b/sources/root-filesystem/sbin/init.sh --- a/sources/root-filesystem/sbin/init.sh +++ b/sources/root-filesystem/sbin/init.sh @@ -20,7 +20,7 @@ export CPUS=$(echo /sys/devices/system/cpu/cpu[0-9]* | wc -w) [ "$CPUS" -lt 1 ] && CPUS=1 fi -export PS1='($HOST:$CPUS) \w \$ ' +export PS1="($HOST:$CPUS) \w \$ " # When running under qemu, do some more setup if [ $$ -eq 1 ] diff --git a/root-filesystem.sh b/root-filesystem.sh --- a/root-filesystem.sh +++ b/root-filesystem.sh @@ -54,12 +54,12 @@ # Put statically and dynamically linked hello world programs on there for # test purposes. -"${ARCH}-cc" "${SOURCES}/root-filesystem/src/hello.c" -Os $CFLAGS \ +"${CC_PREFIX}cc" "${SOURCES}/root-filesystem/src/hello.c" -Os $CFLAGS \ -o "$STAGE_USR/bin/hello-dynamic" || dienow if [ "$BUILD_STATIC" != none ] then - "${ARCH}-cc" "${SOURCES}/root-filesystem/src/hello.c" -Os $CFLAGS -static \ + "${CC_PREFIX}cc" "${SOURCES}/root-filesystem/src/hello.c" -Os $CFLAGS -static \ -o "$STAGE_USR/bin/hello-static" || dienow STATIC=--static else @@ -68,7 +68,7 @@ # Debug wrapper for use with /usr/src/record-commands.sh -"${ARCH}-cc" "$SOURCES/toys/wrappy.c" -Os $CFLAGS $STATIC \ +"${CC_PREFIX}cc" "$SOURCES/toys/wrappy.c" -Os $CFLAGS $STATIC \ -o "$STAGE_USR/bin/record-commands-wrapper" || dienow # Do we need shared libraries? @@ -78,18 +78,18 @@ echo Copying compiler libraries... mkdir -p "$STAGE_USR/lib" || dienow (path_search \ - "$("$ARCH-cc" --print-search-dirs | sed -n 's/^libraries: =*//p')" \ + "$("${CC_PREFIX}cc" --print-search-dirs | sed -n 's/^libraries: =*//p')" \ "*.so*" 'cp -H "$DIR/$FILE" "$STAGE_USR/lib/$FILE"' \ || dienow) | dotprogress [ -z "$SKIP_STRIP" ] && - "${ARCH}-strip" --strip-unneeded "$STAGE_USR"/lib/*.so + "${CC_PREFIX}strip" --strip-unneeded "$STAGE_USR"/lib/*.so fi # Clean up and package the result [ -z "$SKIP_STRIP" ] && - "${ARCH}-strip" "$STAGE_USR"/{bin/*,sbin/*} + "${CC_PREFIX}strip" "$STAGE_USR"/{bin/*,sbin/*} create_stage_tarball From eiselekd at gmail.com Sat Nov 28 20:59:27 2015 From: eiselekd at gmail.com (Konrad Eisele) Date: Sat, 28 Nov 2015 21:59:27 +0100 Subject: [Aboriginal] patches ubuntu 15.10 Message-ID: Hi, Some patches needed to build on a ubuntu 15.10 machine. First 2 are for gcc inline removal, the third is to skip texinfo build with newer version of texinfo installed. // Konrad -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- --- aboriginal-1.4.3.ori/sources/sections/binutils.build 2015-11-04 08:32:30.000000000 +0100 +++ aboriginal-1.4.3/sources/sections/binutils.build 2015-11-28 21:48:41.951606037 +0100 @@ -12,6 +12,8 @@ --disable-nls --disable-shared --disable-multilib --disable-werror \ --with-lib-path=lib --program-prefix="$TOOLCHAIN_PREFIX" $BINUTILS_FLAGS + echo "MAKEINFO = :" >> Makefile + [ $? -ne 0 ] && dienow } -------------- next part -------------- A non-text attachment was scrubbed... Name: gcc-core-top.patch Type: text/x-patch Size: 805 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gcc-core-top-h.patch Type: text/x-patch Size: 1887 bytes Desc: not available URL: