From rob at landley.net  Tue Jun  4 10:13:09 2013
From: rob at landley.net (Rob Landley)
Date: Tue, 04 Jun 2013 05:13:09 -0500
Subject: [Aboriginal] Replacing bash with mksh
In-Reply-To: <51a858b6.0a04980a.6ba1.ffff9ac6@mx.google.com> (from
	metaniklas@gmail.com on Fri May 31 03:00:09 2013)
References: <51a858b6.0a04980a.6ba1.ffff9ac6@mx.google.com>
Message-ID: <1370340789.2776.58@driftwood>

On 05/31/2013 03:00:09 AM, Niklas Larsson wrote:
> Another nice thing with mksh is that it's the default shell in Android
> since Ice cream sandwich.

It might be nice to try to build linux from scratch with it and see  
what breaks. I've restarted my 7.3 conversion pass last weekend and I'm  
going to try to switch on the compiler and libc builds this time, but  
I'm doing it under bash first.

Ideally I want to get toysh implemented ASAP, and might take some code  
from the public domain version of mksh to help that. In practice,  
neither of my last two day jobs has been particularly amenable to  
having as much free time to program as I'd like, and scheduled stuff  
likekeeping up with kernel releases (3 targets currently broken in  
3.10-rc4) take priority. Plus I should really finish the musl  
switchover...

Working on it. :)

Rob
 1370340789.0

From prazzb at gmail.com  Tue Jun  4 17:16:57 2013
From: prazzb at gmail.com (Prasanna Balan)
Date: Tue, 4 Jun 2013 22:46:57 +0530
Subject: [Aboriginal] Replacing bash with mksh
In-Reply-To: <1370340789.2776.58@driftwood>
References: <51a858b6.0a04980a.6ba1.ffff9ac6@mx.google.com>
	<1370340789.2776.58@driftwood>
Message-ID: <20130604171657.GA6725@desktop>

On 04/06/13, Rob Landley wrote:
> On 05/31/2013 03:00:09 AM, Niklas Larsson wrote:
> > Another nice thing with mksh is that it's the default shell in Android
> > since Ice cream sandwich.
Have been reading about it... I am thinking of replacing my user shell
from bash to mksh..need some more time to read up on it

> It might be nice to try to build linux from scratch with it and see  
> what breaks. I've restarted my 7.3 conversion pass last weekend and I'm  
> going to try to switch on the compiler and libc builds this time, but  
> I'm doing it under bash first.

For the point... bash 4.2 broke on x86-64 when I tried about a month
ago due to some bison issue and kbd broke due to some flex issue...
everything else did build fine... as for the aboriginal patches,
uClibc needed include/sys/mount changes via two patches... the ones
they did to be "up-to-date" with glibc and obviously the chmod -f from
toybox
> 
> Rob

prazzb

 1370366217.0

From idunham at lavabit.com  Wed Jun  5 02:29:47 2013
From: idunham at lavabit.com (idunham at lavabit.com)
Date: Tue, 4 Jun 2013 19:29:47 -0700
Subject: [Aboriginal] Replacing bash with mksh
In-Reply-To: <mailman.1.1370377841.17940.aboriginal-landley.net@lists.landley.net>
References: <mailman.1.1370377841.17940.aboriginal-landley.net@lists.landley.net>
Message-ID: <20130605022947.GA2327@Caracal>

> Message: 1
> Date: Tue, 04 Jun 2013 05:13:09 -0500
> From: Rob Landley <rob at landley.net>
> To: Niklas Larsson <metaniklas at gmail.com>
> Cc: "aboriginal at lists.landley.net" <aboriginal at lists.landley.net>
> Subject: Re: [Aboriginal] Replacing bash with mksh
> Message-ID: <1370340789.2776.58 at driftwood>
> Content-Type: text/plain; charset=us-ascii; DelSp=Yes; Format=Flowed
> 
> On 05/31/2013 03:00:09 AM, Niklas Larsson wrote:
> > Another nice thing with mksh is that it's the default shell in Android
> > since Ice cream sandwich.
> 
> It might be nice to try to build linux from scratch with it and see  
> what breaks. I've restarted my 7.3 conversion pass last weekend and I'm  
> going to try to switch on the compiler and libc builds this time, but  
> I'm doing it under bash first.
> 
> Ideally I want to get toysh implemented ASAP, and might take some code  
> from the public domain version of mksh to help that. In practice,  
> neither of my last two day jobs has been particularly amenable to  
> having as much free time to program as I'd like, and scheduled stuff  
> likekeeping up with kernel releases (3 targets currently broken in  
> 3.10-rc4) take priority. Plus I should really finish the musl  
> switchover...
> Rob

That would be OpenBSD pdksh, for which I'd strongly recomment using one
of these two Linux ports:
https://github.com/bobertlo/openbsd-pdksh
http://slackbuilds.org/repository/14.0/system/ksh-openbsd/
IIRC, I heard that the latter includes fixes for a number of things that 
"are subtly broken in OpenBSD".

I forget exactly how to go about applying the patches from the latter
source, but it didn't take that long.
The libbsd dependency is for one or two functions that are in musl.

That said, I'll warn you: there are well over 20 C files to go through.
And there are a few builtins that we have, like mknod.

PS: be sure to ignore "oksh", which is openbsd pdksh + the old pdksh
configure + a GPL3 notice and a few unsafe hacks to make it build
with glibc.
Also ignore "posh", a pdksh fork under GPL3 which focused on removing
non-POSIX features, but doesn't support all the traps mandated by POSIX
or any interactive line editing.

> Date: Tue, 4 Jun 2013 22:46:57 +0530
> From: Prasanna Balan <prazzb at gmail.com>
> To: aboriginal at lists.landley.net
> Subject: Re: [Aboriginal] Replacing bash with mksh

> On 04/06/13, Rob Landley wrote:
> > On 05/31/2013 03:00:09 AM, Niklas Larsson wrote:
> > > Another nice thing with mksh is that it's the default shell in Android
> > > since Ice cream sandwich.
> Have been reading about it... I am thinking of replacing my user shell
> from bash to mksh..need some more time to read up on it

No real issues for most things; I've used mksh as login shell on my
netbook for a few years, due to the bloat of modern bash with all
extensions loaded. 
Even shell expressions (like a.{c,sh}), work the same.

> > It might be nice to try to build linux from scratch with it and see  
> > what breaks. I've restarted my 7.3 conversion pass last weekend and I'm  
> > going to try to switch on the compiler and libc builds this time, but  
> > I'm doing it under bash first.
> 
> prazzb
> 


 1370399387.0

From rob at landley.net  Sat Jun  8 07:00:49 2013
From: rob at landley.net (Rob Landley)
Date: Sat, 08 Jun 2013 02:00:49 -0500
Subject: [Aboriginal] Replacing bash with mksh
In-Reply-To: <20130605022947.GA2327@Caracal> (from idunham@lavabit.com on
	Tue Jun  4 21:29:47 2013)
Message-ID: <1370674849.2776.78@driftwood>

On 06/04/2013 09:29:47 PM, idunham at lavabit.com wrote:
> > Ideally I want to get toysh implemented ASAP, and might take some  
> code
> > from the public domain version of mksh to help that. In practice,
> > neither of my last two day jobs has been particularly amenable to
> > having as much free time to program as I'd like, and scheduled stuff
> > likekeeping up with kernel releases (3 targets currently broken in
> > 3.10-rc4) take priority. Plus I should really finish the musl
> > switchover...
> > Rob
> 
> That would be OpenBSD pdksh, for which I'd strongly recomment using  
> one
> of these two Linux ports:
> https://github.com/bobertlo/openbsd-pdksh
> http://slackbuilds.org/repository/14.0/system/ksh-openbsd/
> IIRC, I heard that the latter includes fixes for a number of things  
> that
> "are subtly broken in OpenBSD".

I wasn't planning to take large chunks of code verbatim, I just tend  
not to actually read GPL code before writing a new non-gpl thing. I  
read man pages and standards, and run things under strace.

For public domain stuff, I'm happy to have the PD version open in one  
window while I write another one in a second window, and use it as a  
template. But we've got our own library of reusable code that nothing  
else has, and you've seen how much

> I forget exactly how to go about applying the patches from the latter
> source, but it didn't take that long.
> The libbsd dependency is for one or two functions that are in musl.
> 
> That said, I'll warn you: there are well over 20 C files to go  
> through.
> And there are a few builtins that we have, like mknod.

I noticed this. :)

I'm not opening the shell can of worms until I get the current set of  
loose ends tied off a bit more first. I got "stat" out of pending, I'm  
working to get ifconfig out of pending but the limiting factor at the  
moment is I don't understand what command lines it's parsing and  
there's no ifconfig.test. (set_sockaddr() is called by both  
set_address() and set_ipv6_addr(), and I _think_ 3/4 of set_sockaddr()  
is ipv6-only stuff but... I've used "ifconfig eth0 192.168.1.42/24"  
back in 2002 and this ifconfig implementation only has / parsing in the  
ipv6 path, and it seems to be doing something else...? I'm not familiar  
with ipv6 enough to understand what this is doing, so I'm trying to  
learn these funky corner cases I've never used so I can understand,  
untangle, and test...)

I've got a p9d server in progress, and I've dug up my mount  
implementation because I've had _three_ mount implementations submitted  
ot me now so I need to get that in...

Plus I still have a todo list of commands to triage that predate  
"pending" (vmstat, login, du, vconfig, mountpoint, free, chroot, cut,  
touch modinfo, expand) and some of them have shrapnel in lib/lib.c (for  
example "cut" added get_int_value() which is something lib/args.c  
already does...)

> PS: be sure to ignore "oksh", which is openbsd pdksh + the old pdksh
> configure + a GPL3 notice and a few unsafe hacks to make it build
> with glibc.

I grabbed the original massively stale version from 1999, on the theory  
there's probably less of it to read. "wc -l *.c" says the total is  
27180 lines, which is a couple solid days triaging right there.

But this weekend, go to visit my sister and the niecephews again...

> Also ignore "posh", a pdksh fork under GPL3 which focused on removing
> non-POSIX features, but doesn't support all the traps mandated by  
> POSIX
> or any interactive line editing.

I'm sticking with the ancient long-dead version, I think. I'm less  
likely to catch anything from it. :)

And really, I have the design of a new shell that I did a months of  
design work for (as in a couple hours every evening for about three  
months in early 2007). It's just not the kind of thing you want to  
start and stop repeatedly on, I want to do it all in one go and I have  
_not_ had time for it.

> > Date: Tue, 4 Jun 2013 22:46:57 +0530
> > From: Prasanna Balan <prazzb at gmail.com>
> > To: aboriginal at lists.landley.net
> > Subject: Re: [Aboriginal] Replacing bash with mksh
> 
> > On 04/06/13, Rob Landley wrote:
> > > On 05/31/2013 03:00:09 AM, Niklas Larsson wrote:
> > > > Another nice thing with mksh is that it's the default shell in  
> Android
> > > > since Ice cream sandwich.
> > Have been reading about it... I am thinking of replacing my user  
> shell
> > from bash to mksh..need some more time to read up on it
> 
> No real issues for most things; I've used mksh as login shell on my
> netbook for a few years, due to the bloat of modern bash with all
> extensions loaded.
> Even shell expressions (like a.{c,sh}), work the same.

So it's already better than the Defective Annoying SHell, then. :)

Which fork are you using?

Rob
 1370674849.0

From rob at landley.net  Sat Jun  8 07:00:05 2013
From: rob at landley.net (Rob Landley)
Date: Sat, 08 Jun 2013 02:00:05 -0500
Subject: [Aboriginal] Replacing bash with mksh
In-Reply-To: <20130604171657.GA6725@desktop> (from prazzb@gmail.com on Tue
	Jun  4 12:16:57 2013)
Message-ID: <1370674805.2776.77@driftwood>

On 06/04/2013 12:16:57 PM, Prasanna Balan wrote:
> On 04/06/13, Rob Landley wrote:
> > On 05/31/2013 03:00:09 AM, Niklas Larsson wrote:
> > > Another nice thing with mksh is that it's the default shell in  
> Android
> > > since Ice cream sandwich.
> Have been reading about it... I am thinking of replacing my user shell
> from bash to mksh..need some more time to read up on it
> 
> > It might be nice to try to build linux from scratch with it and see
> > what breaks. I've restarted my 7.3 conversion pass last weekend and  
> I'm
> > going to try to switch on the compiler and libc builds this time,  
> but
> > I'm doing it under bash first.
> 
> For the point... bash 4.2 broke on x86-64 when I tried about a month
> ago due to some bison issue and kbd broke due to some flex issue...
> everything else did build fine... as for the aboriginal patches,
> uClibc needed include/sys/mount changes via two patches... the ones
> they did to be "up-to-date" with glibc and obviously the chmod -f from
> toybox

I have the partial 7.3 conversion you sent me many moons ago. (Sorry  
it's taken so long, I was down to about 4 hours of sleep a night for a  
while there. Got through that and now have semi-normal evenings and  
weekends again, and am trying to catch up with multiple backlogs. I'll  
probably jump straight to the 3.10 kernel, but since 3.8 the kernel  
guys broke 3 architectures I still have to bisect and fix...)

Oh, and architecturally I vaguely want to fit toybox into initramfs and  
have the toolchain be the /dev/hda squashfs mounted somewhere and added  
to the $PATH. Right now the SYSIMAGE_TYPE=initramfs stuff doesn't get  
enough testing, or the no native tools version of the root filesystem.  
Having it be a runtime option makes that easier. When I'll get time to  
_do_ this, who knows...

Rob
 1370674805.0

From idunham at lavabit.com  Mon Jun 10 07:33:06 2013
From: idunham at lavabit.com (idunham at lavabit.com)
Date: Mon, 10 Jun 2013 00:33:06 -0700
Subject: [Aboriginal] Replacing bash with mksh
In-Reply-To: <1370674849.2776.78@driftwood>
References: <20130605022947.GA2327@Caracal>
 <1370674849.2776.78@driftwood>
Message-ID: <20130610073306.GB2319@Caracal>

On Sat, Jun 08, 2013 at 02:00:49AM -0500, Rob Landley wrote:
> On 06/04/2013 09:29:47 PM, idunham at lavabit.com wrote:
> >> Ideally I want to get toysh implemented ASAP, and might take
> >some code
> >> from the public domain version of mksh to help that. In practice,
> >> neither of my last two day jobs has been particularly amenable to
> >> having as much free time to program as I'd like, and scheduled stuff
> >> likekeeping up with kernel releases (3 targets currently broken in
> >> 3.10-rc4) take priority. Plus I should really finish the musl
> >> switchover...
> >> Rob
> >
> >That would be OpenBSD pdksh, for which I'd strongly recomment
> >using one
> >of these two Linux ports:
> >https://github.com/bobertlo/openbsd-pdksh
> >http://slackbuilds.org/repository/14.0/system/ksh-openbsd/
> >IIRC, I heard that the latter includes fixes for a number of
> >things that
> >"are subtly broken in OpenBSD".
> 
> For public domain stuff, I'm happy to have the PD version open in
> one window while I write another one in a second window, and use it
> as a template. But we've got our own library of reusable code that
> nothing else has, and you've seen how much
> 
> >I forget exactly how to go about applying the patches from the latter
> >source, but it didn't take that long.
> >The libbsd dependency is for one or two functions that are in musl.
> >
> >That said, I'll warn you: there are well over 20 C files to go
> >through.
> >And there are a few builtins that we have, like mknod.
> 
> I noticed this. :)
> 
> I'm not opening the shell can of worms until I get the current set
> of loose ends tied off a bit more first. I got "stat" out of
> pending, I'm working to get ifconfig out of pending but the limiting
> factor at the moment is I don't understand what command lines it's
> parsing and there's no ifconfig.test. (set_sockaddr() is called by
> both set_address() and set_ipv6_addr(), and I _think_ 3/4 of
> set_sockaddr() is ipv6-only stuff but... I've used "ifconfig eth0
> 192.168.1.42/24" back in 2002 and this ifconfig implementation only
> has / parsing in the ipv6 path, and it seems to be doing something
> else...? I'm not familiar with ipv6 enough to understand what this
> is doing, so I'm trying to learn these funky corner cases I've never
> used so I can understand, untangle, and test...)
> 
> I've got a p9d server in progress, and I've dug up my mount
> implementation because I've had _three_ mount implementations
> submitted ot me now so I need to get that in...

So there are now four versions of mount floating around, none of which
are publicly available?

> Plus I still have a todo list of commands to triage that predate
> "pending" (vmstat, login, du, vconfig, mountpoint, free, chroot,
> cut, touch modinfo, expand) and some of them have shrapnel in
> lib/lib.c (for example "cut" added get_int_value() which is
> something lib/args.c already does...)
> 
modinfo seems to be good for uncompressed modules afaict.  
I'd write a test for it if I could figure out how to do one that works with 
all Linux hosts...but the corner cases it makes sense to check may not be 
present on many systems. (Coincidentally, toybox modinfo does the right
thing for "modinfo oss-usb" where busybox fails; oss_usb.ko is the file
name, it's part of the out-of-tree OSS4 code...)
Examples of test cases:
modinfo a-module -> a_module.ko
modinfo a_module -> a-module.ko
modinfo radeon
(mile long list of firmware that changes with each kernel version
follows)

> I grabbed the original massively stale version from 1999, on the
> theory there's probably less of it to read. "wc -l *.c" says the
> total is 27180 lines, which is a couple solid days triaging right
> there.

23460 for ksh-openbsd, patches applied.
So someone has achieved negative code growth.

> >> Have been reading about it... I am thinking of replacing my user
> >shell
> >> from bash to mksh..need some more time to read up on it
> >
> >No real issues for most things; I've used mksh as login shell on my
> >netbook for a few years, due to the bloat of modern bash with all
> >extensions loaded.
> >Even shell expressions (like a.{c,sh}), work the same.
> 
> So it's already better than the Defective Annoying SHell, then. :)
> 
> Which fork are you using?

Debian/Ubuntu package, maintained by Thorsten Glaser (an upstream
developer.)
But it's also true of at least the Debian Squeeze pdksh package.
Of course, ostensibly "some non-portable" features get disabled when you use the
name "sh"...haven't tested which ones.

(BTW, dash is Debian's fork from 2002 of an ancient port of NetBSD's 
Almquist shell.  AFAIK it is technically conformant to POSIX, and even
supports a few bash-like features...but nowhere near all the ones people
use.)

Isaac


 1370849586.0

From rob at landley.net  Tue Jun 11 03:44:37 2013
From: rob at landley.net (Rob Landley)
Date: Mon, 10 Jun 2013 22:44:37 -0500
Subject: [Aboriginal] Replacing bash with mksh
In-Reply-To: <20130610073306.GB2319@Caracal> (from idunham@lavabit.com on
	Mon Jun 10 02:33:06 2013)
Message-ID: <1370922277.2776.92@driftwood>

On 06/10/2013 02:33:06 AM, idunham at lavabit.com wrote:
> On Sat, Jun 08, 2013 at 02:00:49AM -0500, Rob Landley wrote:
> > On 06/04/2013 09:29:47 PM, idunham at lavabit.com wrote:
> > >> Ideally I want to get toysh implemented ASAP, and might take
> > >some code
> > >> from the public domain version of mksh to help that. In practice,
> > >> neither of my last two day jobs has been particularly amenable to
> > >> having as much free time to program as I'd like, and scheduled  
> stuff
> > >> likekeeping up with kernel releases (3 targets currently broken  
> in
> > >> 3.10-rc4) take priority. Plus I should really finish the musl
> > >> switchover...
> > >> Rob
> > >
> > >That would be OpenBSD pdksh, for which I'd strongly recomment
> > >using one
> > >of these two Linux ports:
> > >https://github.com/bobertlo/openbsd-pdksh
> > >http://slackbuilds.org/repository/14.0/system/ksh-openbsd/
> > >IIRC, I heard that the latter includes fixes for a number of
> > >things that
> > >"are subtly broken in OpenBSD".
> >
> > For public domain stuff, I'm happy to have the PD version open in
> > one window while I write another one in a second window, and use it
> > as a template. But we've got our own library of reusable code that
> > nothing else has, and you've seen how much
> >
> > >I forget exactly how to go about applying the patches from the  
> latter
> > >source, but it didn't take that long.
> > >The libbsd dependency is for one or two functions that are in musl.
> > >
> > >That said, I'll warn you: there are well over 20 C files to go
> > >through.
> > >And there are a few builtins that we have, like mknod.
> >
> > I noticed this. :)
> >
> > I'm not opening the shell can of worms until I get the current set
> > of loose ends tied off a bit more first. I got "stat" out of
> > pending, I'm working to get ifconfig out of pending but the limiting
> > factor at the moment is I don't understand what command lines it's
> > parsing and there's no ifconfig.test. (set_sockaddr() is called by
> > both set_address() and set_ipv6_addr(), and I _think_ 3/4 of
> > set_sockaddr() is ipv6-only stuff but... I've used "ifconfig eth0
> > 192.168.1.42/24" back in 2002 and this ifconfig implementation only
> > has / parsing in the ipv6 path, and it seems to be doing something
> > else...? I'm not familiar with ipv6 enough to understand what this
> > is doing, so I'm trying to learn these funky corner cases I've never
> > used so I can understand, untangle, and test...)
> >
> > I've got a p9d server in progress, and I've dug up my mount
> > implementation because I've had _three_ mount implementations
> > submitted ot me now so I need to get that in...
> 
> So there are now four versions of mount floating around, none of which
> are publicly available?

I think a couple got posted to the mailing list? (People keep emailing  
me off list. Dunno why. The most recent is attached, the previous one  
was... october I think?)

Plus klibc has one, android toolbox has one...

Last time I sat down to write mount, it used loopback stuff first so I  
had to write a proper losetup (which I did, although that had grown a  
lot of new features since last I'd checked). Then I started researching  
all the new mount flags, and I have a stub that can enumerate them and  
print out help text for them (also attached). The "actually does  
something useful" variant trails off in the middle and does not  
currently compile...

Also, keep in mind that I did several months digging into the guts of  
NFS back in 2010 and would like to use the new string interface instead  
of the old binary interface:

   http://landley.livejournal.com/52663.html

Basically all the "smbmount" and "nfsmount" and such front-ends are  
doing three things:

1) prompting you for a password from /dev/tty and adding "pass=" to the  
options string.

2) DNS lookups so you can say \\hostname or similar instead of an IP  
address.

3) converting the command line options a bit so you don't actually have  
to say "-t nfs" or "-t cifs" when you use that front-end.

I.E. if you know what you're doing you can do an NFS or samba mount  
from a mount command that doesn't know anything special about that  
filesystem. (But if the command DOES know about it, it can intercept  
what you actually said and screw it up.)

Having large amounts of extra code for this is silly...

> > Plus I still have a todo list of commands to triage that predate
> > "pending" (vmstat, login, du, vconfig, mountpoint, free, chroot,
> > cut, touch modinfo, expand) and some of them have shrapnel in
> > lib/lib.c (for example "cut" added get_int_value() which is
> > something lib/args.c already does...)
> >
> modinfo seems to be good for uncompressed modules afaict.

Needing to triage doesn't mean they don't work. The ifconfig command  
worked before I started triaging it, it was just twice the size of the  
current one (and I'm not done yet).

I like to do polishing passes. If nothing else, I like to have the  
design of toybox commands clear in my head so I can support them. Some  
of the commands I want to review may not wind up having any actual  
_changes_, but right now I haven't read them closely enough and thought  
through the interactions to say they _don't_ need changes.

Tangent warning: feel free to skip this bit:

Something in my old prototype and fan club talk (on  
http://landley.net/talks but it's not my finest hour/recording) was  
that reading code is harder than writing code. When writing code you  
have a complete model in your head and what's on the screen trails the  
model in your head. When _reading_ code the model in your head trails  
what's on the screen, _and_ the order of execution is neither linear  
nor deterministic with all sorts of potential jumps to other functions  
in whole other files so you need five tabs open to follow the plot.

This is part of the reason there's so much "not invented here" stuff in  
programming: even when it's perfectly good code, if you wrote it you  
know what it does, and the amount of reading and pondering necessary to  
get that level of familiarity with code you _didn't_ write is always a  
lot higher than you expect.

(This is one of the differentiators between ok coders and really good  
ones: being _aware_ that code isn't _bad_ just because you didn't write  
it. It's harder for you personally to understand, but that's true of  
everybody. Is "does it smell like me" actually an improvement, or is  
the rewrite just causing pointless churn?)

In toybox's case I do have the luxury of doing a bit of smells-like-me  
rewriting because I expect to have to support this going forward.  
(Privilege of being a project maintainer.) To be brutally honest,  
commits like these:

   http://landley.net/hg/toybox/rev/765
   http://landley.net/hg/toybox/rev/767

are as much "ok, now I'm comfortable with it" as actually improving the  
code. (And because I can use source control to check if I'm the last  
committer to files as a way of seeing what I've reviewed and what I  
haven't.)

But when I'm done with those sorts of patches I go "is this worth  
checking in" and sometimes I discard it instead. And doing that sort of  
thing to a project you're _not_ the maintainer of is just rude. There  
are a lot of times I open up a second window, transcribe my own version  
of whatever I'm reading, and then _delete_ that new copy when I'm done  
because now I understand the old one and I'd just be peeing on it so it  
smells like me rather than actually improving anything.

(I'm not saying all polishing is cosmetic, although it is an easy trap  
to fall into. But this is why "I need to find time to review this"  
doesn't necessarily mean there's something known to be wrong with it.  
And also why "I've got five minutes, let me read through cut.c" is not  
the same thing as a proper review: a proper review can balloon to 4  
hours even if all I did was something "indent" could have done...)

> I'd write a test for it if I could figure out how to do one that  
> works with
> all Linux hosts...but the corner cases it makes sense to check may  
> not be
> present on many systems.

I don't understand. Care to explain?

> (Coincidentally, toybox modinfo does the right
> thing for "modinfo oss-usb" where busybox fails; oss_usb.ko is the  
> file
> name, it's part of the out-of-tree OSS4 code...)

If I recall, modinfo doesn't parse the elf tables but just does a  
string search of the binary. I'm not sure I'd call that "the right  
thing", it seems extrordinarily brittle to me.)

> Examples of test cases:
> modinfo a-module -> a_module.ko
> modinfo a_module -> a-module.ko
> modinfo radeon
> (mile long list of firmware that changes with each kernel version
> follows)

I'm always happy to fluff out the test suite, if you'd like to send a  
patch. :)

> > I grabbed the original massively stale version from 1999, on the
> > theory there's probably less of it to read. "wc -l *.c" says the
> > total is 27180 lines, which is a couple solid days triaging right
> > there.
> 
> 23460 for ksh-openbsd, patches applied.
> So someone has achieved negative code growth.

Yay?

Is it still public domain? I removed the "and you must copy this  
license text into all modified versions so even though you can  
relicense it GPL you _also_ have to include incompatible license terms  
and then explain why they don't apply" clause from toybox's license,  
which is essentially public domain now but _looks_ like BSD (and I  
describe it as such) so as not to freak out the corporate types who go  
"Jim Butterfield's public domain programs may have been on the  
Commodore Bonus Disk shipped with 1541 disk drives in 1982 but don't  
act like that's a precedent! This is weird untried unproven... it  
doesn't give us any way to get paid to advise you!"

Sorry, tangent. (Bit of a headache. Late for lunch.)

> > >> Have been reading about it... I am thinking of replacing my user
> > >shell
> > >> from bash to mksh..need some more time to read up on it
> > >
> > >No real issues for most things; I've used mksh as login shell on my
> > >netbook for a few years, due to the bloat of modern bash with all
> > >extensions loaded.
> > >Even shell expressions (like a.{c,sh}), work the same.
> >
> > So it's already better than the Defective Annoying SHell, then. :)
> >
> > Which fork are you using?
> 
> Debian/Ubuntu package, maintained by Thorsten Glaser (an upstream
> developer.)
> But it's also true of at least the Debian Squeeze pdksh package.
> Of course, ostensibly "some non-portable" features get disabled when  
> you use the
> name "sh"...haven't tested which ones.

That's true of bash, though. :)

> (BTW, dash is Debian's fork from 2002 of an ancient port of NetBSD's
> Almquist shell.  AFAIK it is technically conformant to POSIX, and even
> supports a few bash-like features...but nowhere near all the ones  
> people
> use.)

I looked up the pedigree of that monster once upon a time.

My objection is that running bash was the reason Linus Torvalds added a  
system call layer to his terminal program. (Uploading/downloading files  
from the vax at his university was the reason he added a filesystem  
layer.) This made bash the default shell of Linux before the 0.0.1  
release, which it continued to be until Ubuntu unilaterlly changed it.  
When ubuntu did so, they broke the kernel build. Their stated reasons  
for doing so were not sufficient justification for doing it, _and_ they  
failed at their stated goal (and had to introduce upstart as another  
way of addressing the problem) without admitting their mistake and  
reverting it.

So it's not dash that's at fault per se. It's ubuntu was ignorant of  
history, caused massive collateral damage, made a bad move by their own  
reasoning, failed at their stated goal, and refused to revert their  
mistake after further developments rendered their original reason moot.

Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: other-mount.c
Type: text/x-csrc
Size: 36296 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20130610/4e810978/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mount.c
Type: text/x-csrc
Size: 3131 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20130610/4e810978/attachment-0001.bin>

From idunham at lavabit.com  Wed Jun 12 06:24:42 2013
From: idunham at lavabit.com (Isaac)
Date: Tue, 11 Jun 2013 23:24:42 -0700
Subject: [Aboriginal] Replacing bash with mksh
In-Reply-To: <mailman.73703.1370922667.15463.aboriginal-landley.net@lists.landley.net>
References: <mailman.73703.1370922667.15463.aboriginal-landley.net@lists.landley.net>
Message-ID: <20130612062442.GA2405@newbook>

On Mon, Jun 10, 2013 at 08:51:07PM -0700, aboriginal-request at lists.landley.net wrote:
> From: Rob Landley <rob at landley.net>
> On 06/10/2013 02:33:06 AM, idunham at lavabit.com wrote:
> > On Sat, Jun 08, 2013 at 02:00:49AM -0500, Rob Landley wrote:
> > So there are now four versions of mount floating around, none of which
> > are publicly available?
> 
> I think a couple got posted to the mailing list? (People keep emailing  
> me off list. Dunno why. The most recent is attached, the previous one  
> was... october I think?)

Ah, it was before I subscribed.
 
> Plus klibc has one, android toolbox has one...

> Last time I sat down to write mount, it used loopback stuff first so I  
> had to write a proper losetup (which I did, although that had grown a  
> lot of new features since last I'd checked). Then I started researching  
> all the new mount flags, and I have a stub that can enumerate them and  
> print out help text for them (also attached). The "actually does  
> something useful" variant trails off in the middle and does not  
> currently compile...

Also there's this one:
http://lists.landley.net/pipermail/toybox-landley.net/2012-November/000691.html

Which is not a full mount, by any stretch of the imagination.

> Also, keep in mind that I did several months digging into the guts of  
> NFS back in 2010 and would like to use the new string interface instead  
> of the old binary interface:
> 
>    http://landley.livejournal.com/52663.html
> 
> Basically all the "smbmount" and "nfsmount" and such front-ends are  
> doing three things:
<sniP>
> I.E. if you know what you're doing you can do an NFS or samba mount  
> from a mount command that doesn't know anything special about that  
> filesystem. (But if the command DOES know about it, it can intercept  
> what you actually said and screw it up.)
> 
> Having large amounts of extra code for this is silly...

Which is ~ 1/2 or more of the large version...
 
> > > Plus I still have a todo list of commands to triage that predate
> > > "pending" (vmstat, login, du, vconfig, mountpoint, free, chroot,
> > > cut, touch modinfo, expand) and some of them have shrapnel in
> > > lib/lib.c (for example "cut" added get_int_value() which is
> > > something lib/args.c already does...)
> > >
> > modinfo seems to be good for uncompressed modules afaict.
> 
> Needing to triage doesn't mean they don't work. The ifconfig command  
> worked before I started triaging it, it was just twice the size of the  
> current one (and I'm not done yet).

> > I'd write a test for it if I could figure out how to do one that  
> > works with
> > all Linux hosts...but the corner cases it makes sense to check may  
> > not be
> > present on many systems.
> 
> I don't understand. Care to explain?

The tests that could be done would depend on having certain uncompressed 
modules (since we don't yet support compressed modules...) present to check 
results against.
*Not every system even has kernel modules present/supports modules
*Anything that can be a module can be disabled
*It's possible to disable anything that uses firmware.

Of course now that I'm writing this I test and see that multiple modules 
should be supported, but aren't; expect a test and a patch sometime soon.

As far as supporting compressed modules goes...
It could probably be done via a hook in lib/*.c (xdecompress?) which will
-use magic numbers (NOT extensions, we may want support for autodetecting
 file format of streams for tar...) to identify the format <A>
 (about 6 bytes should be enough:
BZh bzip2, 
{ 0xFD, '7', 'z', 'X', 'Z', 0x00 } xz,
1F 8B 08 gz,
..)

-fork <A>cat with the file/stream as input and the toy then reads the 
output stream


> > (Coincidentally, toybox modinfo does the right
> > thing for "modinfo oss-usb" where busybox fails; oss_usb.ko is the  
> > file
> > name, it's part of the out-of-tree OSS4 code...)
> 
> If I recall, modinfo doesn't parse the elf tables but just does a  
> string search of the binary. I'm not sure I'd call that "the right  
> thing", it seems extrordinarily brittle to me.)
Ah, I see.

> > Examples of test cases:
> > modinfo a-module -> a_module.ko
> > modinfo a_module -> a-module.ko
> > modinfo radeon
> > (mile long list of firmware that changes with each kernel version
> > follows)
> 
> I'm always happy to fluff out the test suite, if you'd like to send a  
> patch. :)
> 
> > > I grabbed the original massively stale version from 1999, on the
> > > theory there's probably less of it to read. "wc -l *.c" says the
> > > total is 27180 lines, which is a couple solid days triaging right
> > > there.
> > 
> > 23460 for ksh-openbsd, patches applied.
> > So someone has achieved negative code growth.
> 
> Yay?
> 
> Is it still public domain? 
Yes, except for alloc.c, mknod.c, and strl{cpy,cat}.c, which are 3-clause
 (or less) BSD, and posix_signals.h, which apparently is still 4-clause BSD.
Of those, alloc.c is the only one that would be relevant.



 1371018282.0

From rob at landley.net  Thu Jun 13 05:02:18 2013
From: rob at landley.net (Rob Landley)
Date: Thu, 13 Jun 2013 00:02:18 -0500
Subject: [Aboriginal] Replacing bash with mksh
In-Reply-To: <20130612062442.GA2405@newbook> (from idunham@lavabit.com on
	Wed Jun 12 01:24:42 2013)
Message-ID: <1371099738.2776.106@driftwood>

On 06/12/2013 01:24:42 AM, Isaac wrote:
> On Mon, Jun 10, 2013 at 08:51:07PM -0700,  
> aboriginal-request at lists.landley.net wrote:
> > From: Rob Landley <rob at landley.net>
> > On 06/10/2013 02:33:06 AM, idunham at lavabit.com wrote:
> > > On Sat, Jun 08, 2013 at 02:00:49AM -0500, Rob Landley wrote:
> > > So there are now four versions of mount floating around, none of  
> which
> > > are publicly available?
> >
> > I think a couple got posted to the mailing list? (People keep  
> emailing
> > me off list. Dunno why. The most recent is attached, the previous  
> one
> > was... october I think?)
> 
> Ah, it was before I subscribed.

Even though this project got relaunched in 2006 or so, I've been  
trimming this bonsai since about 2001. Random snapshot of the previous  
version's in (http://landley.net/aboriginal/old and yes there was a  
version before _that_...)

These days toybox is taking up most of my time, and the multiple out of  
tree mounts are actually a toybox issue...

> > Plus klibc has one, android toolbox has one...
> 
> > Last time I sat down to write mount, it used loopback stuff first  
> so I
> > had to write a proper losetup (which I did, although that had grown  
> a
> > lot of new features since last I'd checked). Then I started  
> researching
> > all the new mount flags, and I have a stub that can enumerate them  
> and
> > print out help text for them (also attached). The "actually does
> > something useful" variant trails off in the middle and does not
> > currently compile...
> 
> Also there's this one:
> http://lists.landley.net/pipermail/toybox-landley.net/2012-November/000691.html
> 
> Which is not a full mount, by any stretch of the imagination.

A simple mount on the level of what toolbox does is fifteen minutes  
work. Doing it _right_ is a weekend. Probably not _this_ weekend since  
I'm visiting my sister, and probably not _next_ weekend since my wife's  
in town for a convention (4th street). last weekend I visited my sister  
and the niecephews. The weekend before that I think I was in Austin...

Today I've bisected the mips regression:
http://lkml.indiana.edu/hypermail/linux/kernel/1306.1/02284.html

And am poking at sh4. Also poking at p9d (a 9p server I'm adding for  
fun), and catching up on email, and catching up on the daily show. And  
all that's because I haven't go the brainpower right now to handle the  
next round of ifconfig cleanup.

> > Also, keep in mind that I did several months digging into the guts  
> of
> > NFS back in 2010 and would like to use the new string interface  
> instead
> > of the old binary interface:
> >
> >    http://landley.livejournal.com/52663.html
> >
> > Basically all the "smbmount" and "nfsmount" and such front-ends are
> > doing three things:
> <sniP>
> > I.E. if you know what you're doing you can do an NFS or samba mount
> > from a mount command that doesn't know anything special about that
> > filesystem. (But if the command DOES know about it, it can intercept
> > what you actually said and screw it up.)
> >
> > Having large amounts of extra code for this is silly...
> 
> Which is ~ 1/2 or more of the large version...

I feel bad that people are waiting for this, but I'm already  
overwhelmed with todo stuff in pending. All I've really done in the  
past month is cleanup, and I'm nowhere near caught up. (Largely because  
I have maybe 5 hours of work a week to put into this, and it's mostly  
in twenty minute chunks.)

> > > I'd write a test for it if I could figure out how to do one that
> > > works with
> > > all Linux hosts...but the corner cases it makes sense to check may
> > > not be
> > > present on many systems.
> >
> > I don't understand. Care to explain?
> 
> The tests that could be done would depend on having certain  
> uncompressed
> modules (since we don't yet support compressed modules...) present to  
> check
> results against.
> *Not every system even has kernel modules present/supports modules
> *Anything that can be a module can be disabled
> *It's possible to disable anything that uses firmware.
> 
> Of course now that I'm writing this I test and see that multiple  
> modules
> should be supported, but aren't; expect a test and a patch sometime  
> soon.

The kernel's I've been building for aboriginal don't actually have  
module support switched on. It's easy enough to add, I just haven't  
done it because this way there's no boot sequencing issues or strong  
version dependencies between the root filesystem and the kernel.

> As far as supporting compressed modules goes...
> It could probably be done via a hook in lib/*.c (xdecompress?) which  
> will
> -use magic numbers (NOT extensions, we may want support for  
> autodetecting
>  file format of streams for tar...) to identify the format <A>
>  (about 6 bytes should be enough:
> BZh bzip2,
> { 0xFD, '7', 'z', 'X', 'Z', 0x00 } xz,
> 1F 8B 08 gz,
> ..)

The "file" command is one I'd like to implement but the magic is a  
GIANT list of file types in a big datafile, and toybox is a single  
portable binary you can statically link. Possibly I could pick a couple  
dozen of the most common file types (elf, gzip, png/jpeg/gif, ascii)  
and have it check for those? Dunno...

> -fork <A>cat with the file/stream as input and the toy then reads the
> output stream

An insmod that takes a module from stdin shouldn't be too hard. We've  
already got "-" logic for that really...

> > > > I grabbed the original massively stale version from 1999, on the
> > > > theory there's probably less of it to read. "wc -l *.c" says the
> > > > total is 27180 lines, which is a couple solid days triaging  
> right
> > > > there.
> > >
> > > 23460 for ksh-openbsd, patches applied.
> > > So someone has achieved negative code growth.
> >
> > Yay?
> >
> > Is it still public domain?
> Yes, except for alloc.c, mknod.c, and strl{cpy,cat}.c, which are  
> 3-clause
>  (or less) BSD, and posix_signals.h, which apparently is still  
> 4-clause BSD.
> Of those, alloc.c is the only one that would be relevant.

That's just fiddly enough that at midnight on a wednesday with work in  
9 hours, I'm not feeling hugely like adding it to the todo list...

Rob
 1371099738.0

From scsijon at lamiaworks.com.au  Wed Jun 26 00:53:19 2013
From: scsijon at lamiaworks.com.au (scsijon)
Date: Wed, 26 Jun 2013 10:53:19 +1000
Subject: [Aboriginal] extending aboriginal -concept idea
Message-ID: <51CA3B7F.1030908@lamiaworks.com.au>

Firstly, may I say I love it and are playing with a couple of basic 
builds at present to get the feeling of what it can do. I am considering 
using it for the basis of a wayland build or even a small multi-server 
system.

However, I was wondering if there was the ability of extending what 
packages were in a build by using an additional script. Such as, if 
(1)it exists it's added; or (2) by also adding an extra build switch to 
(1) to include it and without the switch it's not.

concept is something like:
---
for 1- if a builder added to the aboriginal default (location where 
./build.sh exists) a script called additional.sh its content would be 
added into the standard build.

I'm not at this point considering what it should or should not contain 
and what it's structure should be, but something like this for each 
additional package may be sufficient.

name
source url (complete)
hash
error checking for missing pre-requisite packages
compile scripts (unpack; ./configure;make;make install)

for 2- including 1- but also adding to your build script something like

  ./build.sh i486 additional

to include both the default set AND the additional set

if you only use ./build.sh i486

then it would build without the content of the additional.sh packages

---
In all cases it would be up to the builder to populate this file and 
handle any problems associated with it.

I'm not sure how much work or changes would be needed to the existing 
scripts, but I see it as a way of extending aboriginals functionality 
and thus the community using it.

regards
scsijon


 1372207999.0

From scsijon at lamiaworks.com.au  Wed Jun 26 01:20:09 2013
From: scsijon at lamiaworks.com.au (scsijon)
Date: Wed, 26 Jun 2013 11:20:09 +1000
Subject: [Aboriginal] addition - log file of build
Message-ID: <51CA41C9.4060507@lamiaworks.com.au>

I was wondering if when ./build.sh was run, a second copy of the 
displayed build process was logged into a text file, maybe with the 
filename in the format build-$date-$time.log, so it can be investigated 
if problems are found then or later, or you want to check something that 
occurred during the build happened in the right way.

regards
scsijon

 1372209609.0

From onefang at gmail.com  Wed Jun 26 01:26:34 2013
From: onefang at gmail.com (David Seikel)
Date: Wed, 26 Jun 2013 11:26:34 +1000
Subject: [Aboriginal] extending aboriginal -concept idea
In-Reply-To: <51CA3B7F.1030908@lamiaworks.com.au>
References: <51CA3B7F.1030908@lamiaworks.com.au>
Message-ID: <20130626112634.137c39ea.onefang@gmail.com>

On Wed, 26 Jun 2013 10:53:19 +1000 scsijon <scsijon at lamiaworks.com.au>
wrote:

> Firstly, may I say I love it and are playing with a couple of basic 
> builds at present to get the feeling of what it can do. I am
> considering using it for the basis of a wayland build or even a small
> multi-server system.
> 
> However, I was wondering if there was the ability of extending what 
> packages were in a build by using an additional script. Such as, if 
> (1)it exists it's added; or (2) by also adding an extra build switch
> to (1) to include it and without the switch it's not.
> 
> concept is something like:
> ---
> for 1- if a builder added to the aboriginal default (location where 
> ./build.sh exists) a script called additional.sh its content would be 
> added into the standard build.
> 
> I'm not at this point considering what it should or should not
> contain and what it's structure should be, but something like this
> for each additional package may be sufficient.
> 
> name
> source url (complete)
> hash
> error checking for missing pre-requisite packages
> compile scripts (unpack; ./configure;make;make install)
> 
> for 2- including 1- but also adding to your build script something
> like
> 
>   ./build.sh i486 additional
> 
> to include both the default set AND the additional set
> 
> if you only use ./build.sh i486
> 
> then it would build without the content of the additional.sh packages
> 
> ---
> In all cases it would be up to the builder to populate this file and 
> handle any problems associated with it.
> 
> I'm not sure how much work or changes would be needed to the existing 
> scripts, but I see it as a way of extending aboriginals functionality 
> and thus the community using it.

This is what control images are for.

http://www.landley.net/code/aboriginal/control-images/

-- 
A big old stinking pile of genius that no one wants
coz there are too many silver coated monkeys in the world.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20130626/5abd0fb8/attachment.sig>

From onefang at gmail.com  Wed Jun 26 01:27:49 2013
From: onefang at gmail.com (David Seikel)
Date: Wed, 26 Jun 2013 11:27:49 +1000
Subject: [Aboriginal] addition - log file of build
In-Reply-To: <51CA41C9.4060507@lamiaworks.com.au>
References: <51CA41C9.4060507@lamiaworks.com.au>
Message-ID: <20130626112749.1b70fe57.onefang@gmail.com>

On Wed, 26 Jun 2013 11:20:09 +1000 scsijon <scsijon at lamiaworks.com.au>
wrote:

> I was wondering if when ./build.sh was run, a second copy of the 
> displayed build process was logged into a text file, maybe with the 
> filename in the format build-$date-$time.log, so it can be
> investigated if problems are found then or later, or you want to
> check something that occurred during the build happened in the right
> way.

You could use the tee shell command.

-- 
A big old stinking pile of genius that no one wants
coz there are too many silver coated monkeys in the world.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20130626/d062e748/attachment.sig>

From scsijon at lamiaworks.com.au  Sat Jun 29 01:18:33 2013
From: scsijon at lamiaworks.com.au (scsijon)
Date: Sat, 29 Jun 2013 11:18:33 +1000
Subject: [Aboriginal] package building problems
Message-ID: <51CE35E9.2040901@lamiaworks.com.au>

with aboriginal-84bd3248fad8:

We have had a problem or two attempting to build a few packages, it was 
found that adding "UCLIBC_HAS_GETPT=y" to sources/baseconfig-uClibc has 
solved some of them such as rxvt.

On the other hand we seem to have a few expected /include '.h' files 
missing that are normally used for building with after building the new 
aboriginal to 'play with'. Such as libc.h, sys/byteorder.h, 
sys/strredir.h, sys/sockio.h, sys/sockio.h, sys/stropts.h. I was 
wondering if this is deliberate or just not used.

thanks
scsijon

 1372468713.0

From idunham at lavabit.com  Sat Jun 29 23:59:30 2013
From: idunham at lavabit.com (Isaac)
Date: Sat, 29 Jun 2013 16:59:30 -0700
Subject: [Aboriginal] missing headers?
In-Reply-To: <mailman.1.1372537885.19049.aboriginal-landley.net@lists.landley.net>
References: <mailman.1.1372537885.19049.aboriginal-landley.net@lists.landley.net>
Message-ID: <20130629235930.GA1368@newbook>

On Sat, Jun 29, 2013 at 01:31:25PM -0700, aboriginal-request at lists.landley.net wrote:
> From: scsijon 
> 
> We have had a problem or two attempting to build a few packages, it was 
> found that adding "UCLIBC_HAS_GETPT=y" to sources/baseconfig-uClibc has 
> solved some of them such as rxvt.
> 
> On the other hand we seem to have a few expected /include '.h' files 
> missing that are normally used for building with after building the new 
> aboriginal to 'play with'. Such as libc.h, sys/byteorder.h, 
> sys/strredir.h, sys/sockio.h, sys/sockio.h, sys/stropts.h. I was 
> wondering if this is deliberate or just not used.

Ubuntu Lucid, all headers installed:
$ dlocate /libc.h
$ dlocate byteorder.h|grep sys/                              
$ dlocate sys/strredir.h                                     
$ dlocate strredir.h                                         
$ dlocate sys/sockio.h                                       
$ dlocate sockio.h                                           
$ dlocate sys/stropts.h                                      
libc6-dev: /usr/include/sys/stropts.h

In other wordds, the vast majority are not present on a standard Linux system.
And <sys/stropts.h> contains only this:

#include <stropts.h>

> thanks
> scsijon



 1372550370.0

From rob at landley.net  Sun Jun 30 04:36:40 2013
From: rob at landley.net (Rob Landley)
Date: Sat, 29 Jun 2013 23:36:40 -0500
Subject: [Aboriginal] extending aboriginal -concept idea
In-Reply-To: <51CA3B7F.1030908@lamiaworks.com.au> (from
	scsijon@lamiaworks.com.au on Tue Jun 25 19:53:19 2013)
Message-ID: <1372567000.2776.168@driftwood>

On 06/25/2013 07:53:19 PM, scsijon wrote:
> Firstly, may I say I love it and are playing with a couple of basic
> builds at present to get the feeling of what it can do. I am
> considering using it for the basis of a wayland build or even a
> small multi-server system.

Yay!

Sorry I'm behind on email, I normally catch up on weekends but I was at  
a convention last weekend where the power went out for 24 hours, so  
instead I fell further behind. Catching up now, I've apparently made it  
to Tuesday.

> However, I was wondering if there was the ability of extending what
> packages were in a build by using an additional script. Such as, if
> (1)it exists it's added; or (2) by also adding an extra build switch
> to (1) to include it and without the switch it's not.

In the "design goals" section of  
http://landley.net/aboriginal/about.html it does talk about eliminating  
the need for cross compiling. The general idea is building stuff  
natively within a system image.

That said, if you want to add a new build script you can just drop it  
in the top level directory and add it to build.sh. The plumbing works  
like this:

# Suck in common functions and variable definitions
source sources/include.sh || exit 1

# Suck in target-specific data from sources/targets
load_target "$1"

# Find a package "thingy" out of sources/sections.
build_section thingy
build_section otherpackage
build_section blah

# tar up the result into $STAGE_NAME-$ARCH.tar.bz2
create_stage_tarball

Regarding sources/sections: scripts ending in .sh do their own setup  
and teardown (see gcc.sh for example), the "setupfor" function  
snapshots the package source code into sources/temp-$ARCH/packagename  
and changes directory into there, the "cleanup" function does an rm -rf  
on that directory. If you need to build out of tree blank_workdir  
creates a named temporary directory.

The scripts ending in .build have setupfor and cleanup called for them,  
so you can just do the configure make and install. (See make.build for  
example.)

There's a lot more complexity like the "package cache", explained at  
some length in the FAQ, but I try to make it so you can ignore as much  
of the infrastructure as possible.

> concept is something like:
> ---
> for 1- if a builder added to the aboriginal default (location where
> ./build.sh exists) a script called additional.sh its content would be
> added into the standard build.

Ok. I note there's already SIMPLE_ROOT_OVERLAY in config that can point  
to a directory that gets copied into the resulting root filesystem  
verbatim.

> I'm not at this point considering what it should or should not  
> contain and
> what it's structure should be, but something like this for each  
> additional
> package may be sufficient.

I recommend looking at the Linux From Scratch build control image. I  
build additional stuff natively on each target, because cross compiling  
really does suck.

The script more/chroot-splice.sh does native builds in a chroot instead  
of in qemu, which is a bit faster if you're building for i686 or  
x86-64. (That's generally a good first pass test.) As with all of them,  
run it without any arguments to see usage.
What I haven't explicitly documented (probably should) is that you can  
point it at the build directory instead of the sqf file, and it'll  
--bind mount that so live updates you do to the source show up in the  
target.

Something like:

cd ~
wget http://landley.net/hg/control-images
cd control-images
./build.sh lfs-bootstrap
cd aboriginal
more/chroot-splice.sh i686 ../control-images/build/lfs-bootstrap

(and then at that shell prompt)

/mnt/init

The scripts in there are:

1) init - first thing that runs. If the root filesystem is read only  
(which the squashfs under qemu is), copy it into a chroot under /home,  
and chroot into there. Either way, launches run-build-stages.sh next.

2) run-build-stages.sh - iterate through package-list and call  
build-one-package.sh on each entry. There are #tags in there that can  
be used to filter out packages so you don't build them, so if you  
export "FILTER=busybox" it won't build packages that busybox provides  
functionality for.

3) build/* - the build scripts for each individual package.

4) packages/* - the extracted source code for each package.

> name
> source url (complete)
> hash
> error checking for missing pre-requisite packages
> compile scripts (unpack; ./configure;make;make install)
> 
> for 2- including 1- but also adding to your build script something  
> like
> 
>  ./build.sh i486 additional
> 
> to include both the default set AND the additional set

The third argument from build.sh is "restart build from this stage". So  
when I change toybox and want to rebuild that in the target but _not_  
rebuild the cross or native compilers, I can go "./build.sh x86_64  
simple-root-filesystem" and that will do that, root-filesystem,  
root-image, it'll redo linux-kernel if SYSIMAGE_TYPE=initramfs, and  
then system-image.

Probably what you want is a new chunk between cross-compiler and  
simple-root-filesystem that does:

if [ -e additional.sh] && not_already additional
then
   zap simple-root-filesystem

   time ./additional.sh "$ARCH" || exit 1
   export SIMPLE_ROOT_OVERLAY=build/additional-"$ARCH"
fi

Or something like that.

> if you only use ./build.sh i486
> 
> then it would build without the content of the additional.sh packages
> 
> ---
> In all cases it would be up to the builder to populate this file and  
> handle any problems associated with it.
> 
> I'm not sure how much work or changes would be needed to the existing  
> scripts, but I see it as a way of extending aboriginals functionality  
> and thus the community using it.

I've encouraged adding more packages via native building. Cross  
compiling more packages has... issues:

https://speakerdeck.com/mirell/developing-for-non-x86-targets-using-qemu?slide=79

(The next dozen or so slides from there are on that topic.)

If there seriously is interest in this, I can document the required  
infrastructure better. (At one point I had, but it bit-rootted as  
everything changed over the years.)

Rob
 1372567000.0

From rob at landley.net  Sun Jun 30 04:39:27 2013
From: rob at landley.net (Rob Landley)
Date: Sat, 29 Jun 2013 23:39:27 -0500
Subject: [Aboriginal] addition - log file of build
In-Reply-To: <51CA41C9.4060507@lamiaworks.com.au> (from
	scsijon@lamiaworks.com.au on Tue Jun 25 20:20:09 2013)
References: <51CA41C9.4060507@lamiaworks.com.au>
Message-ID: <1372567167.2776.169@driftwood>

On 06/25/2013 08:20:09 PM, scsijon wrote:
> I was wondering if when ./build.sh was run, a second copy of the  
> displayed build process was logged into a text file, maybe with the  
> filename in the format build-$date-$time.log, so it can be  
> investigated if problems are found then or later, or you want to  
> check something that occurred during the build happened in the right  
> way.

There's a FAQ on this:

   http://landley.net/aboriginal/FAQ.html#debug_logging

In addition, the more/buildall.sh script (which builds all  
architectures, doing so in parallel if you export FORK=1) logs all the  
output to build/logs.

Rob
 1372567167.0