From rob at landley.net  Tue Sep  9 13:21:00 2014
From: rob at landley.net (Rob Landley)
Date: Tue, 09 Sep 2014 08:21:00 -0500
Subject: [Aboriginal] 1.2.8 release.
Message-ID: <540EFEBC.1000506@landley.net>

Uploading as we speak, release notes on the main/news page.

This has musl support in the build scripts, but I still need to fix
-Bsymbolic-functions in the linker so for the moment the prebuilt
binaries are still all uClibc. (Musl static linking should work, dynamic
linking may sort of work depending on what you're linking and what
target. The failure mode should be build break rather than runtime, so
it's not exactly subtle.)

This is an interim release to get the new infrastructure regression
tested (it built Linux From Scratch for i686, grinding through the other
targets in progress but that means the new ccwrap.c implementation
basically works). This uses the 2.15 kernel and no other package
upgrades, next release should have the 2.16 kernel and a toybox upgrade.

I've stopped bothering to upgrade busybox because when testing against
the upcoming toybox release we're down to using this from busybox:

  [, [[, ash, awk, bunzip2, bzcat, bzip2, dd, diff, expr, fdisk, ftpd,
  ftpget, ftpput, gunzip, gzip, less, man, pgrep, ping, pkill, ps,
  route, sed, sh, sha512sum, tar, test, tr, unxz, vi, wget, xzcat, zcat

Half of that I've got implementations in toys/pending awaiting cleanup.
Three of those commands are aliases for "test", and I've already got
things like gunzip promoted but can't switch over to them until I switch
over tar (in pending)... easiest to just focus on toybox from here on out.

So yeah, toybox and musl. Light at the end of the tunnel. Once I've got
both of those in, I open the can of worms that is "new toolchain".
(Amused by http://marc.info/?l=pcc-list&m=141020111215576&w=2 but
there's no real license on cfront and we still need a linker, which
currently means http://lld.llvm.org which http://ellcc.org hasn't made
work yet. For qcc I want to rip the linker out of tinycc, but by the
time I get any real work put in on that will it still be relevant...?)

If crosstool-ng or buildroot or code sourcery or somebody just put out
_native_ toolchain binaries, life would be so much easier. But no, they
just do cross compilers, no native compilers. Sigh...

Anyway, have fun. Might do this again next week.

Rob

 1410268860.0

From stephen.n.turner at gmail.com  Thu Sep 25 21:49:21 2014
From: stephen.n.turner at gmail.com (stephen Turner)
Date: Thu, 25 Sep 2014 17:49:21 -0400
Subject: [Aboriginal] aboriginal linux
In-Reply-To: <CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>
References: <CAA7aPHhU45fQReVBOLkoBevZQKL5jzYg-NyoEY9MHR5Dvci76g@mail.gmail.com>
	<541B7AC0.7030508@landley.net>
	<CAA7aPHgba016B8-bb5S6kxF3zTF6gQPFW_3FjgYSoSi8Ym+mQg@mail.gmail.com>
	<5422D26E.6020806@landley.net>
	<CAA7aPHgfF1jufyaqAjCdFe-ZH7kZYdX2CE7WskhM35XoCnGL7A@mail.gmail.com>
	<CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>
Message-ID: <CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>

On Wed, Sep 24, 2014 at 11:19 PM, Rob Landley <rob at landley.net> wrote:

> On Wed, Sep 24, 2014 at 4:55 PM, stephen Turner
> <stephen.n.turner at gmail.com> wrote:
> > I had no idea, not sure how i missed it either. I will join that in a
> bit as
> > i'm sure there is more fun posts to follow there.
> >
> > Right now i think i'm having issues with my headers. I rebuilt a
> successful
> > aboriginal clone (just the programs not scripts) and GCC is able to
> compile
> > code but the a.out file created is unreadable to some extent as the file
> > reports not there when you try to run it (don't you love linux saying a
> file
> > cannot be found just because it cant be run when it exists and is
> present?)
>
> The file that can't be found is either the dynamic linker or the
> program interpreter. Lemme guess: you're not using ccwrap.c.
>

Is ccwrap.c something i need to turn on? I didnt disable it and GCC is in
the /bin dir while i still call ./configure and while running ./configure i
receive this output:

"configure error cannot run c compiled programs."

Now checking the config.log i see

./conftest
./configure: line 3900: ./conftest: no such file or directory

For the sake of argument i decided to perform the old hello world test.c
test. I did so and recieved a a.out file which clearly had some data in it.
trying to run that a.out file returned the error "file not found "

>
> > I found a snippet on the web that when using musl libc you may need to
> patch
> > the headers as well as gcc. gotta do more home work to see what needs to
> be
> > done there.
>
> You're building with MUSL=1 ? That's still a bit buggy. I need to dig
> into the toolchain to figure out why -Bsymbolic-functions isn't
> reliably working. (It seems to for some symbols, but not for all.)
>

Im using musl 1.1.3 gcc 4.6.4 musl patched


>
> > I gotta say man, building a mini distro is fun even if it doesn't go
> > mainstream! and what a learning experience!
>
> Long ago I gave a BOF talk about why gcc path logic sucks. It's
> probably on http://landley.net/talks or some such.
>
> > If you run into any problems rebuilding your system for musl let me
> know, i
> > might already know a work around.
>
> You know what's wrong with dynamic linking with the last GPLv2 version
> of binutils fished out of git (between 2.17 and 2.18, just before the
> change to the license file went in) with regard to
> -Bsymbolic-functions? It only affects dynamic linking, not static
> linking. Might have something to do with weak symbol binding, haven't
> spent a weekend to track it down yet...
>
>
I am trying to dynamic link, i also have binutils 2.24 and i couldnt find
any other data on musl patching the headers which doesnt make sense to me.
why would linux headers version 3.16 need to be musl patched? headers dont
even deal with the libc do they?



> Rob
>
> P.S. Nobody ever wants to cc their messages to the aboriginal linux
> mailing list. I has a sad.
>

Fixed that for you :) no more cc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20140925/53380666/attachment.htm>

From stephen.n.turner at gmail.com  Thu Sep 25 23:02:44 2014
From: stephen.n.turner at gmail.com (stephen Turner)
Date: Thu, 25 Sep 2014 19:02:44 -0400
Subject: [Aboriginal] aboriginal linux
In-Reply-To: <CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>
References: <CAA7aPHhU45fQReVBOLkoBevZQKL5jzYg-NyoEY9MHR5Dvci76g@mail.gmail.com>
	<541B7AC0.7030508@landley.net>
	<CAA7aPHgba016B8-bb5S6kxF3zTF6gQPFW_3FjgYSoSi8Ym+mQg@mail.gmail.com>
	<5422D26E.6020806@landley.net>
	<CAA7aPHgfF1jufyaqAjCdFe-ZH7kZYdX2CE7WskhM35XoCnGL7A@mail.gmail.com>
	<CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>
	<CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>
Message-ID: <CAA7aPHg2f218Gvr6x8t_KRd-96S0LQ51qfNabGo_q=t6iSV-_g@mail.gmail.com>

never mind i fixed that error, well sortof, apparently while running
./config its looking for my headers at /include instead of the standard
/usr/include

Now im receiving error grep doesnt handle long lines or -e im using toybox
for my implementation of grep

*sign* another issue to dig into.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20140925/a64ed055/attachment.htm>

From rob at landley.net  Fri Sep 26 15:18:27 2014
From: rob at landley.net (Rob Landley)
Date: Fri, 26 Sep 2014 10:18:27 -0500
Subject: [Aboriginal] aboriginal linux
In-Reply-To: <CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>
References: <CAA7aPHhU45fQReVBOLkoBevZQKL5jzYg-NyoEY9MHR5Dvci76g@mail.gmail.com>	<541B7AC0.7030508@landley.net>	<CAA7aPHgba016B8-bb5S6kxF3zTF6gQPFW_3FjgYSoSi8Ym+mQg@mail.gmail.com>	<5422D26E.6020806@landley.net>	<CAA7aPHgfF1jufyaqAjCdFe-ZH7kZYdX2CE7WskhM35XoCnGL7A@mail.gmail.com>	<CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>
	<CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>
Message-ID: <542583C3.1000207@landley.net>

On 09/25/14 16:49, stephen Turner wrote:
> On Wed, Sep 24, 2014 at 11:19 PM, Rob Landley <rob at landley.net
> <mailto:rob at landley.net>> wrote:
> 
>     On Wed, Sep 24, 2014 at 4:55 PM, stephen Turner
>     <stephen.n.turner at gmail.com <mailto:stephen.n.turner at gmail.com>> wrote:
>     > I had no idea, not sure how i missed it either. I will join that in a bit as
>     > i'm sure there is more fun posts to follow there.
>     >
>     > Right now i think i'm having issues with my headers. I rebuilt a successful
>     > aboriginal clone (just the programs not scripts) and GCC is able to compile
>     > code but the a.out file created is unreadable to some extent as the file
>     > reports not there when you try to run it (don't you love linux saying a file
>     > cannot be found just because it cant be run when it exists and is present?)
> 
>     The file that can't be found is either the dynamic linker or the
>     program interpreter. Lemme guess: you're not using ccwrap.c.
> 
> 
> Is ccwrap.c something i need to turn on? I didnt disable it and GCC is
> in the /bin dir while i still call ./configure and while running
> ./configure i receive this output:
> 
> "configure error cannot run c compiled programs."

It's not part of gcc:

http://landley.net/hg/aboriginal/file/tip/sources/sections/ccwrap.sh

The path logic in gcc is utterly horrible. I tried fixing it properly
once but when my patch removing bad logic passed 10,000 lines I decided
I didn't want to try to maintain that. (This was before the project
switched licenses to GPLv3 and I stayed with the last GPLv2 release.)

http://landley.net/notes-2006.html#15-11-2006

The problem with cross compiling is not just getting it to look in the
right place(s) but not to look in the WRONG places. If it leaks host
headers or libraries into the target build, it can subtly break stuff.
Getting gcc to _not_ hallucinate new paths to check is a serious
challenge. (These days there's sysroot, which does a strncmp() on the
start of the hardwired paths backed into the thing at build time and
replaces the ones that match with what you fed to it in the --sysroot
command line argument. No really, that's how they implemented it.)

> Now checking the config.log i see
> 
> ./conftest
> ./configure: line 3900: ./conftest: no such file or directory
>  
> For the sake of argument i decided to perform the old hello world test.c
> test. I did so and recieved a a.out file which clearly had some data in
> it. trying to run that a.out file returned the error "file not found " 
> 
> 
>     > I found a snippet on the web that when using musl libc you may need to patch
>     > the headers as well as gcc. gotta do more home work to see what needs to be
>     > done there.
> 
>     You're building with MUSL=1 ? That's still a bit buggy. I need to dig
>     into the toolchain to figure out why -Bsymbolic-functions isn't
>     reliably working. (It seems to for some symbols, but not for all.)
> 
> Im using musl 1.1.3 gcc 4.6.4 musl patched

Ah. I meant aboriginal has partial built-in musl support, which I really
need to finish soonish.

>     > I gotta say man, building a mini distro is fun even if it doesn't go
>     > mainstream! and what a learning experience!
> 
>     Long ago I gave a BOF talk about why gcc path logic sucks. It's
>     probably on http://landley.net/talks or some such.
> 
>     > If you run into any problems rebuilding your system for musl let me know, i
>     > might already know a work around.
> 
>     You know what's wrong with dynamic linking with the last GPLv2 version
>     of binutils fished out of git (between 2.17 and 2.18, just before the
>     change to the license file went in) with regard to
>     -Bsymbolic-functions? It only affects dynamic linking, not static
>     linking. Might have something to do with weak symbol binding, haven't
>     spent a weekend to track it down yet...
> 
> 
> I am trying to dynamic link, i also have binutils 2.24 and i couldnt
> find any other data on musl patching the headers which doesnt make sense
> to me. why would linux headers version 3.16 need to be musl patched?
> headers dont even deal with the libc do they?

Headers (/usr/include/*.h files) are compile time, libraries
(/lib/blah.so and /usr/lib/blah.*.a and so on) are link time, and in
some cases runtime. They're separate things.

Musl produces its own heades. You can add kernel headers to the ones it
provides (locally I use the attached script to build new versions for
local testing with their supplied wrapper).

>  
> 
>     Rob
> 
>     P.S. Nobody ever wants to cc their messages to the aboriginal linux
>     mailing list. I has a sad.
> 
> 
> Fixed that for you :) no more cc

Yay!

Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: whack2.sh
Type: application/x-shellscript
Size: 493 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20140926/40599c4b/attachment.bin>

From rob at landley.net  Fri Sep 26 15:21:15 2014
From: rob at landley.net (Rob Landley)
Date: Fri, 26 Sep 2014 10:21:15 -0500
Subject: [Aboriginal] aboriginal linux
In-Reply-To: <CAA7aPHg2f218Gvr6x8t_KRd-96S0LQ51qfNabGo_q=t6iSV-_g@mail.gmail.com>
References: <CAA7aPHhU45fQReVBOLkoBevZQKL5jzYg-NyoEY9MHR5Dvci76g@mail.gmail.com>	<541B7AC0.7030508@landley.net>	<CAA7aPHgba016B8-bb5S6kxF3zTF6gQPFW_3FjgYSoSi8Ym+mQg@mail.gmail.com>	<5422D26E.6020806@landley.net>	<CAA7aPHgfF1jufyaqAjCdFe-ZH7kZYdX2CE7WskhM35XoCnGL7A@mail.gmail.com>	<CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>	<CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>
	<CAA7aPHg2f218Gvr6x8t_KRd-96S0LQ51qfNabGo_q=t6iSV-_g@mail.gmail.com>
Message-ID: <5425846B.3080602@landley.net>

On 09/25/14 18:02, stephen Turner wrote:
> never mind i fixed that error, well sortof, apparently while running
> ./config its looking for my headers at /include instead of the standard
> /usr/include  
> 
> Now im receiving error grep doesnt handle long lines or -e im using
> toybox for my implementation of grep
> 
> *sign* another issue to dig into.

I have a musl patch for the grep -e thing.

The "rm -r dir" failure is currently being discussed on the toybox list.
(Dalias insisted it's not a musl bug, and that the Linux man page is
wrong. I told him to pull the other one, it's got bells on. That was on
irc, discussion has now gone to the list...)

Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: regex.patch
Type: text/x-diff
Size: 1503 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20140926/120eac9b/attachment.bin>

From stephen.n.turner at gmail.com  Fri Sep 26 16:44:22 2014
From: stephen.n.turner at gmail.com (stephen Turner)
Date: Fri, 26 Sep 2014 12:44:22 -0400
Subject: [Aboriginal] aboriginal linux
In-Reply-To: <542583C3.1000207@landley.net>
References: <CAA7aPHhU45fQReVBOLkoBevZQKL5jzYg-NyoEY9MHR5Dvci76g@mail.gmail.com>
	<541B7AC0.7030508@landley.net>
	<CAA7aPHgba016B8-bb5S6kxF3zTF6gQPFW_3FjgYSoSi8Ym+mQg@mail.gmail.com>
	<5422D26E.6020806@landley.net>
	<CAA7aPHgfF1jufyaqAjCdFe-ZH7kZYdX2CE7WskhM35XoCnGL7A@mail.gmail.com>
	<CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>
	<CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>
	<542583C3.1000207@landley.net>
Message-ID: <CAA7aPHjCun8WOzprpcLcgpBAu4_tF4U-qBpms0KQ9Ee3EtA6wg@mail.gmail.com>

>
>
>
> The problem with cross compiling is not just getting it to look in the
> right place(s) but not to look in the WRONG places. If it leaks host
> headers or libraries into the target build, it can subtly break stuff.
> Getting gcc to _not_ hallucinate new paths to check is a serious
> challenge. (These days there's sysroot, which does a strncmp() on the
> start of the hardwired paths backed into the thing at build time and
> replaces the ones that match with what you fed to it in the --sysroot
> command line argument. No really, that's how they implemented it.)
>

Tell me about it! I have been writing down notes and on the first page in
big bold letters DO NOT USE SYSROOT!
I found shortly after getting GCC to mostly work that Sysroot did as you
say and hardcoded links to the full path! I had to recreate the
path/to/build/dir and link it to / as a temp fix. similiarly how i have
done my current system to account for the "missing" headers. That however
should be a easy fix i just need to check the documentation and find the
libdir= and includedir= etc to properly place the directories/files when
running ./configure.


>
>
> > I am trying to dynamic link, i also have binutils 2.24 and i couldn't
> > find any other data on musl patching the headers which doesn't make sense
> > to me. why would linux headers version 3.16 need to be musl patched?
> > headers don't even deal with the libc do they?
>
> Headers (/usr/include/*.h files) are compile time, libraries
> (/lib/blah.so and /usr/lib/blah.*.a and so on) are link time, and in
> some cases runtime. They're separate things.
>
> Musl produces its own headers. You can add kernel headers to the ones it
> provides (locally I use the attached script to build new versions for
> local testing with their supplied wrapper).
>

I install the linux headers first and musl headers over (if any get
replaced) the kernel headers. this wont be an issue for a all musl
environment do you think?  The more i learn about linux the more i realize
how little i know. This is a very good breakdown of the pages i was reading
on the topic several days ago, thanks for helping establish a clear picture
of how that works!

>I have a musl patch for the grep -e thing.
>
>The "rm -r dir" failure is currently being discussed on the toybox list.
>(Dalias insisted it's not a musl bug, and that the Linux man page is
>wrong. I told him to pull the other one, it's got bells on. That was on
>irc, discussion has now gone to the list...)

I'm on the toybox and musl lists as well. I've been seeing a bit of
discussion about that. I'm not familiar with posix standards yet but do
they have anything to say about the behavior of applications?

Since you bring it up, I ran into this issue just yesterday with the rm -r
dir bug. I'm using the static linked busybox and toybox binaries (toybox as
the primary, busybox for everything else) in a chroot environment with musl
libs and standard 3.16 kernel headers (with musl headers) if that helps you
with the issue at all.  if there is anything i can do to narrow it down let
me know.

Is the regex file the patch for grep -e?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20140926/6c49666e/attachment.htm>

From rob at landley.net  Fri Sep 26 20:42:10 2014
From: rob at landley.net (Rob Landley)
Date: Fri, 26 Sep 2014 15:42:10 -0500
Subject: [Aboriginal] aboriginal linux
In-Reply-To: <CAA7aPHjCun8WOzprpcLcgpBAu4_tF4U-qBpms0KQ9Ee3EtA6wg@mail.gmail.com>
References: <CAA7aPHhU45fQReVBOLkoBevZQKL5jzYg-NyoEY9MHR5Dvci76g@mail.gmail.com>
	<541B7AC0.7030508@landley.net>
	<CAA7aPHgba016B8-bb5S6kxF3zTF6gQPFW_3FjgYSoSi8Ym+mQg@mail.gmail.com>
	<5422D26E.6020806@landley.net>
	<CAA7aPHgfF1jufyaqAjCdFe-ZH7kZYdX2CE7WskhM35XoCnGL7A@mail.gmail.com>
	<CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>
	<CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>
	<542583C3.1000207@landley.net>
	<CAA7aPHjCun8WOzprpcLcgpBAu4_tF4U-qBpms0KQ9Ee3EtA6wg@mail.gmail.com>
Message-ID: <CAOS_Y6TskBBrGBiyGT_7_kx0XyKPaauBtgtL+f7oDg6wnzW81w@mail.gmail.com>

On Fri, Sep 26, 2014 at 11:44 AM, stephen Turner
<stephen.n.turner at gmail.com> wrote:
>>
>>
>> The problem with cross compiling is not just getting it to look in the
>> right place(s) but not to look in the WRONG places. If it leaks host
>> headers or libraries into the target build, it can subtly break stuff.
>> Getting gcc to _not_ hallucinate new paths to check is a serious
>> challenge. (These days there's sysroot, which does a strncmp() on the
>> start of the hardwired paths backed into the thing at build time and
>> replaces the ones that match with what you fed to it in the --sysroot
>> command line argument. No really, that's how they implemented it.)
>
>
> Tell me about it! I have been writing down notes and on the first page in
> big bold letters DO NOT USE SYSROOT!
> I found shortly after getting GCC to mostly work that Sysroot did as you say
> and hardcoded links to the full path! I had to recreate the
> path/to/build/dir and link it to / as a temp fix. similiarly how i have done
> my current system to account for the "missing" headers. That however should
> be a easy fix i just need to check the documentation and find the libdir=
> and includedir= etc to properly place the directories/files when running
> ./configure.

My ccwrap is a horrible solution to a horrible problem. Here's how it works:

1) Parse incoming command line. Figure out whether we're linking or
preprocessing or what, and if so whether it's static or dynamic or
what.

2) Create a new command line starting with --nostdinc --nostdlib

3) Explicitly add all the headers and paths you should be looking in.

4) If we're linking, add all the implicit libraries and .o files (like
libcrt1.o).

5) Add everything else that was on the command line back in.

Except there are ordering dependencies so the above is an
oversimplification. But that's the theory.

Even doing all that, we still need some patches so that --nostdlib
really _means_ it, most recently:

http://landley.net/hg/aboriginal/rev/ba28c96cba42

>> > I am trying to dynamic link, i also have binutils 2.24 and i couldn't
>> > find any other data on musl patching the headers which doesn't make
>> > sense
>> > to me. why would linux headers version 3.16 need to be musl patched?
>> > headers don't even deal with the libc do they?
>>
>> Headers (/usr/include/*.h files) are compile time, libraries
>> (/lib/blah.so and /usr/lib/blah.*.a and so on) are link time, and in
>> some cases runtime. They're separate things.
>>
>> Musl produces its own headers. You can add kernel headers to the ones it
>> provides (locally I use the attached script to build new versions for
>> local testing with their supplied wrapper).
>
>
> I install the linux headers first and musl headers over (if any get
> replaced) the kernel headers. this wont be an issue for a all musl
> environment do you think?

Given that musl doesn't use the exported kernel headers (and
vice-versa), it shouldn't matter what order you do it in.

P.S. Don't get me _started_ on fixincludes:

http://ewontfix.com/12/

> The more i learn about linux the more i realize
> how little i know. This is a very good breakdown of the pages i was reading
> on the topic several days ago, thanks for helping establish a clear picture
> of how that works!

I mentioned the compiler BOF I led at OLS 2008. (There is, inexplicably, video.)

(I went there to learn but everybody kept wanting me to talk instead.
I even wandered away from the front of the room and leaned against a
side wall, and the darn camera followed me...)

I tend to rant about things in passing in my blog. Alas, filtering the
wheat from the chaff there is kinda hard. The path logic stuff is
mentioned here though:

http://landley.net/notes-2009.html#21-11-2009

Oh hey, I gave a tutorial on this stuff once. I remember that!

http://landley.net/ols/ols2007/cross-compiling.html

(Google found that. I'd totally forgotten about it...)

I wonder if there was video?

A longer version of the rant I was giving about that time (which I
never cleaned up into a formal presentation or article, with the
possible exception of said tutorial) is at:

http://marc.info/?l=pcc-list&m=119189519922940&w=2

Alas, "how it should work" and "how it does work" are completely
unrelated when it comes to gcc. (Sadly, llvm is overengineered, pcc is
stalled, and I haven't had time to look at http://landley.net/qcc in
years.)

>>I have a musl patch for the grep -e thing.
>>
>>The "rm -r dir" failure is currently being discussed on the toybox list.
>>(Dalias insisted it's not a musl bug, and that the Linux man page is
>>wrong. I told him to pull the other one, it's got bells on. That was on
>>irc, discussion has now gone to the list...)
>
> I'm on the toybox and musl lists as well. I've been seeing a bit of
> discussion about that. I'm not familiar with posix standards yet but do they
> have anything to say about the behavior of applications?

POSIX-2008 is here:

http://pubs.opengroup.org/onlinepubs/9699919799/

(Posix-2013 is a dot release for posix-2008, the tweaks are trivial.
2001->2008 was a big deal, 2008->2013 is not.)

Note that you can (legally!) download a tarball copy for personal use from:

http://pubs.opengroup.org/onlinepubs/9699919799/download/index.html

The "utilities" subdirectory (shell and utilities in top left window,
utilities in lower left) gives you all the command line stuff, about
which I grumbled at:

http://landley.net/toybox/roadmap.html#susv4

The "system interfaces" part (system interfaces->system interfaces the
same way) lists all the libc functions and headers, that's the bit
musl uses.

There's also a bunch of stuff on shell scripting language and the
whole "base definitions" section is full of all sorts of useful random
info about environment variables and process context and so on...

The whole thing is a multi-day read, but not bad.

The other big spec is the Linux Standard Base:

http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-generic/

Which is an _appallingly_ bad standard, but it's the only thing that
even tries to define "mount", so toybox has to care about at least
http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/cmdbehav.html
of it.

(I note it defines a recommended filesystem layout, which I largely
ignore. Mine was derived from an earlier version of Linux From
Scratch, which derived some of what it was doing from LSB so it's
there in the ancestry somewhere.)

But computer history is a hobby of mine, thus I tend to track down WHY
various things got done that way:

landley.net/writing/hackermonthly-issue022-pg33.pdf

> Since you bring it up, I ran into this issue just yesterday with the rm -r
> dir bug. I'm using the static linked busybox and toybox binaries (toybox as
> the primary, busybox for everything else) in a chroot environment with musl
> libs and standard 3.16 kernel headers (with musl headers) if that helps you
> with the issue at all.  if there is anything i can do to narrow it down let
> me know.

It's that faccessat() says (in the man page) that I can feed in
AT_SYMLINK_NOFOLLOW and it works fine (albeit "is ignored") on glibc
and uclibc, but in musl they explicitly check for unknown flags and
return -EINVAL. To clarify: this owrks on _klibc_ but not in musl
because the maintainer went out of his way to break it. And when I
called him on it he said the man page should be changed to match his
opinion.

The fix I coded up that evening boiled down to adding a stanza to
portability.h something like:

#ifdef __MUSL__
// Long explanation of why musl is wrong.
#include <someheader.h>
#define faccessat(A, B, C, D) faccessat(A, B, C, 0)
#endif

And then doing an echo "#define __MUSL__" to the end of features.h
during the musl build in aboriginal. But I didn't check it in, and
since then he's posted to the toybox list. (Which you'd never know
because the web archive only updates like twice a week, I need to poke
dreamhost about that but @dreamhostcares on twitter just says to use
the web interface to open a ticket. It's on the todo list...)

> Is the regex file the patch for grep -e?

Yes. Internally grep is doing "REGEX1\|REGEX2\|REGEX3" to glue 'em
together, and tre only supported | in extended regex mode, not as an
escape in normal regex. So I fixed that.

(I'll think about writing entirely posix compliant toybox code when
somebody can tell me how to implement ps or losetup that way. Until
then, linux has man pages that say we can do things, and I believe
them.)

Rob

 1411764130.0

From stephen.n.turner at gmail.com  Sat Sep 27 00:40:42 2014
From: stephen.n.turner at gmail.com (stephen Turner)
Date: Fri, 26 Sep 2014 20:40:42 -0400
Subject: [Aboriginal] aboriginal linux
In-Reply-To: <CAOS_Y6TskBBrGBiyGT_7_kx0XyKPaauBtgtL+f7oDg6wnzW81w@mail.gmail.com>
References: <CAA7aPHhU45fQReVBOLkoBevZQKL5jzYg-NyoEY9MHR5Dvci76g@mail.gmail.com>
	<541B7AC0.7030508@landley.net>
	<CAA7aPHgba016B8-bb5S6kxF3zTF6gQPFW_3FjgYSoSi8Ym+mQg@mail.gmail.com>
	<5422D26E.6020806@landley.net>
	<CAA7aPHgfF1jufyaqAjCdFe-ZH7kZYdX2CE7WskhM35XoCnGL7A@mail.gmail.com>
	<CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>
	<CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>
	<542583C3.1000207@landley.net>
	<CAA7aPHjCun8WOzprpcLcgpBAu4_tF4U-qBpms0KQ9Ee3EtA6wg@mail.gmail.com>
	<CAOS_Y6TskBBrGBiyGT_7_kx0XyKPaauBtgtL+f7oDg6wnzW81w@mail.gmail.com>
Message-ID: <CAA7aPHjiNjdfU7vVL6R7FhMFT5ME_yrZ+T_LzhbXfUogyFujEw@mail.gmail.com>

>
> > Since you bring it up, I ran into this issue just yesterday with the rm
-r
> > dir bug. I'm using the static linked busybox and toybox binaries
(toybox as
> > the primary, busybox for everything else) in a chroot environment with
musl
> > libs and standard 3.16 kernel headers (with musl headers) if that helps
you
> > with the issue at all.  if there is anything i can do to narrow it down
let
> > me know.
>
> It's that faccessat() says (in the man page) that I can feed in
> AT_SYMLINK_NOFOLLOW and it works fine (albeit "is ignored") on glibc
> and uclibc, but in musl they explicitly check for unknown flags and
> return -EINVAL. To clarify: this owrks on _klibc_ but not in musl
> because the maintainer went out of his way to break it. And when I
> called him on it he said the man page should be changed to match his
> opinion.
>
> The fix I coded up that evening boiled down to adding a stanza to
> portability.h something like:
>
> #ifdef __MUSL__
> // Long explanation of why musl is wrong.
> #include <someheader.h>
> #define faccessat(A, B, C, D) faccessat(A, B, C, 0)
> #endif
>
> And then doing an echo "#define __MUSL__" to the end of features.h
> during the musl build in aboriginal. But I didn't check it in, and

I was using the static compiled binary from your site when I had this
issue. Are you sure its musl lib? Is that what your building toybox against
currently?

> since then he's posted to the toybox list. (Which you'd never know
> because the web archive only updates like twice a week, I need to poke
> dreamhost about that but @dreamhostcares on twitter just says to use
> the web interface to open a ticket. It's on the todo list...)
>
> > Is the regex file the patch for grep -e?
>
> Yes. Internally grep is doing "REGEX1\|REGEX2\|REGEX3" to glue 'em
> together, and tre only supported | in extended regex mode, not as an
> escape in normal regex. So I fixed that.
>
> (I'll think about writing entirely posix compliant toybox code when
> somebody can tell me how to implement ps or losetup that way. Until
> then, linux has man pages that say we can do things, and I believe
> them.)
>
> Rob

Should I be applying this patch to gcc, musl, or toybox?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20140926/604cd988/attachment.htm>

From rob at landley.net  Sat Sep 27 01:22:51 2014
From: rob at landley.net (Rob Landley)
Date: Fri, 26 Sep 2014 20:22:51 -0500
Subject: [Aboriginal] 1.2.9 is out (another catch-up release)
Message-ID: <5426116B.90602@landley.net>

I noticed I forgot to upgrade toybox to 0.4.9 last time (the release
from July), I'm hugely overdue for a new toybox release (trying to
finish up pending todo items!) and the 3.17-rc6 kernel came out last
weekend. So a new kernel with 3.17 and 0.4.10 (0.5.0?) would logically
drop in a week or so anyway...

This fixes a bug from last time that prevented the lfs-bootstrap.hdc
image from working. (It used cpio -s to setup the chroot, and that
wasn't implemented in toybox. It's in the repo now, but not in 0.4.9, so
I switched that back to busybox and built the LFS images. Yeah, still
LFS 6.8, I know...)

Less behind now!

Rob

 1411780971.0

From rob at landley.net  Sat Sep 27 01:26:05 2014
From: rob at landley.net (Rob Landley)
Date: Fri, 26 Sep 2014 20:26:05 -0500
Subject: [Aboriginal] aboriginal linux
In-Reply-To: <CAA7aPHjiNjdfU7vVL6R7FhMFT5ME_yrZ+T_LzhbXfUogyFujEw@mail.gmail.com>
References: <CAA7aPHhU45fQReVBOLkoBevZQKL5jzYg-NyoEY9MHR5Dvci76g@mail.gmail.com>	<541B7AC0.7030508@landley.net>	<CAA7aPHgba016B8-bb5S6kxF3zTF6gQPFW_3FjgYSoSi8Ym+mQg@mail.gmail.com>	<5422D26E.6020806@landley.net>	<CAA7aPHgfF1jufyaqAjCdFe-ZH7kZYdX2CE7WskhM35XoCnGL7A@mail.gmail.com>	<CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>	<CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>	<542583C3.1000207@landley.net>	<CAA7aPHjCun8WOzprpcLcgpBAu4_tF4U-qBpms0KQ9Ee3EtA6wg@mail.gmail.com>	<CAOS_Y6TskBBrGBiyGT_7_kx0XyKPaauBtgtL+f7oDg6wnzW81w@mail.gmail.com>
	<CAA7aPHjiNjdfU7vVL6R7FhMFT5ME_yrZ+T_LzhbXfUogyFujEw@mail.gmail.com>
Message-ID: <5426122D.20507@landley.net>

On 09/26/14 19:40, stephen Turner wrote:
>> The fix I coded up that evening boiled down to adding a stanza to
>> portability.h something like:
>>
>> #ifdef __MUSL__
>> // Long explanation of why musl is wrong.
>> #include <someheader.h>
>> #define faccessat(A, B, C, D) faccessat(A, B, C, 0)
>> #endif
>>
>> And then doing an echo "#define __MUSL__" to the end of features.h
>> during the musl build in aboriginal. But I didn't check it in, and
> 
> I was using the static compiled binary from your site when I had this
> issue. Are you sure its musl lib? Is that what your building toybox
> against currently?

Yup, the 0.4.9 binaries were compiled against musl. (I did test them
against musl, but obviously not thoroughly enough.)

Next release should have it fixed.

>> > Is the regex file the patch for grep -e?
>>
>> Yes. Internally grep is doing "REGEX1\|REGEX2\|REGEX3" to glue 'em
>> together, and tre only supported | in extended regex mode, not as an
>> escape in normal regex. So I fixed that.
>>
>> (I'll think about writing entirely posix compliant toybox code when
>> somebody can tell me how to implement ps or losetup that way. Until
>> then, linux has man pages that say we can do things, and I believe
>> them.)
>>
>> Rob
> 
> Should I be applying this patch to gcc, musl, or toybox?

The regex patch is against musl.

Rob


 1411781165.0

From james_mcmechan at hotmail.com  Sun Sep 28 01:43:42 2014
From: james_mcmechan at hotmail.com (James McMechan)
Date: Sat, 27 Sep 2014 18:43:42 -0700
Subject: [Aboriginal] aboriginal linux
In-Reply-To: <5426122D.20507@landley.net>
References: <CAA7aPHhU45fQReVBOLkoBevZQKL5jzYg-NyoEY9MHR5Dvci76g@mail.gmail.com>
	<541B7AC0.7030508@landley.net>
	<CAA7aPHgba016B8-bb5S6kxF3zTF6gQPFW_3FjgYSoSi8Ym+mQg@mail.gmail.com>
	<5422D26E.6020806@landley.net>
	<CAA7aPHgfF1jufyaqAjCdFe-ZH7kZYdX2CE7WskhM35XoCnGL7A@mail.gmail.com>
	<CAOS_Y6Q_y4_tNQ0fVacDzY-SJpBtUc0dBQvguJ+jOnomii1Dpw@mail.gmail.com>
	<CAA7aPHiFYhQqyjryKP6RySDiNj6sJgNa=W42NvgOc=_BTPyrcg@mail.gmail.com>
	<542583C3.1000207@landley.net>
	<CAA7aPHjCun8WOzprpcLcgpBAu4_tF4U-qBpms0KQ9Ee3EtA6wg@mail.gmail.com>
	<CAOS_Y6TskBBrGBiyGT_7_kx0XyKPaauBtgtL+f7oDg6wnzW81w@mail.gmail.com>,
	<CAA7aPHjiNjdfU7vVL6R7FhMFT5ME_yrZ+T_LzhbXfUogyFujEw@mail.gmail.com>,
	<5426122D.20507@landley.net>
Message-ID: <COL125-W37646F32B39837CECFBF43E2BD0@phx.gbl>

just a couple comments

I would think that?

#define faccessat(A, B, C, D) faccessat(A, B, C, 0)

might better be as?

#define faccessat(A, B, C, D) faccessat(A, B, C, D & ( ~AT_SYMLINK_NOFOLLOW) )

so that when something actually wants to use?AT_EACCESS it will either get passed through or fail noisily.
Long ago I developed a strong dislike for the library emulating features when GNU's glibc emulated aio by helpfully making my program multi-threaded?and hijacking?signals.

I found your workaround of multiple -e clauses?simpler and easier to read, than the alternation "\|" but back in 1986 when I was learning about grep maybe it was more primitive, but even now I like the simple answer better.

Jim McMechan

----------------------------------------
> Date: Fri, 26 Sep 2014 20:26:05 -0500
> From: rob at landley.net
> To: stephen.n.turner at gmail.com
> CC: Aboriginal at lists.landley.net
> Subject: Re: [Aboriginal] aboriginal linux
>
> On 09/26/14 19:40, stephen Turner wrote:
>>> The fix I coded up that evening boiled down to adding a stanza to
>>> portability.h something like:
>>>
>>> #ifdef __MUSL__
>>> // Long explanation of why musl is wrong.
>>> #include <someheader.h>
>>> #define faccessat(A, B, C, D) faccessat(A, B, C, 0)
>>> #endif
>>>
>>> And then doing an echo "#define __MUSL__" to the end of features.h
>>> during the musl build in aboriginal. But I didn't check it in, and
>>
>> I was using the static compiled binary from your site when I had this
>> issue. Are you sure its musl lib? Is that what your building toybox
>> against currently?
>
> Yup, the 0.4.9 binaries were compiled against musl. (I did test them
> against musl, but obviously not thoroughly enough.)
>
> Next release should have it fixed.
>
>>>> Is the regex file the patch for grep -e?
>>>
>>> Yes. Internally grep is doing "REGEX1\|REGEX2\|REGEX3" to glue 'em
>>> together, and tre only supported | in extended regex mode, not as an
>>> escape in normal regex. So I fixed that.
>>>
>>> (I'll think about writing entirely posix compliant toybox code when
>>> somebody can tell me how to implement ps or losetup that way. Until
>>> then, linux has man pages that say we can do things, and I believe
>>> them.)
>>>
>>> Rob
>>
>> Should I be applying this patch to gcc, musl, or toybox?
>
> The regex patch is against musl.
>
> Rob
>
> _______________________________________________
> Aboriginal mailing list
> Aboriginal at lists.landley.net
> http://lists.landley.net/listinfo.cgi/aboriginal-landley.net
 		 	   		  
 1411868622.0

From stephen.n.turner at gmail.com  Tue Sep 30 02:18:49 2014
From: stephen.n.turner at gmail.com (stephen Turner)
Date: Mon, 29 Sep 2014 22:18:49 -0400
Subject: [Aboriginal] Headers
Message-ID: <CAA7aPHgR0SZt1hAy6sQyO8yT6xVeoZUz5FV13MyKna-+atBXng@mail.gmail.com>

Im having problems finding the info I want which probobly means im not
asking the right question the right way. I want to know two things, firstly
can you use newer headers with older kernels or older headers with newer
kernels? Second i see that both the libc and gcc installs libs and headers.
I would like to know more about libs and headers and such in general.

Anyone have good links for this kind of reading material? If so I would
greatly appreciate a point in the right direction.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/aboriginal/attachments/20140929/3965664c/attachment.htm>

From rob at landley.net  Tue Sep 30 15:01:42 2014
From: rob at landley.net (Rob Landley)
Date: Tue, 30 Sep 2014 10:01:42 -0500
Subject: [Aboriginal] Headers
In-Reply-To: <CAA7aPHgR0SZt1hAy6sQyO8yT6xVeoZUz5FV13MyKna-+atBXng@mail.gmail.com>
References: <CAA7aPHgR0SZt1hAy6sQyO8yT6xVeoZUz5FV13MyKna-+atBXng@mail.gmail.com>
Message-ID: <542AC5D6.60605@landley.net>

On 09/29/14 21:18, stephen Turner wrote:
> Im having problems finding the info I want which probobly means im not
> asking the right question the right way. I want to know two things,
> firstly can you use newer headers with older kernels or older headers
> with newer kernels?

You can always use older headers with newer kernels. The other way is
not recommended and can give random segfaults when it tries to use
features that aren't there. (I.E. kernel headers provide backwards but
not forwards compatability.)

> Second i see that both the libc and gcc installs
> libs and headers. I would like to know more about libs and headers and
> such in general.

Here's a link to a bunch of links:

http://marc.info/?l=uclibc&m=135873994251146

> Anyone have good links for this kind of reading material? If so I would
> greatly appreciate a point in the right direction.

I need to do a proper tutorial on this someday, but my todo list runneth
over...

In addition to the OLS 2007 tutorial link I pointed at (in the marc.info
message above), I also had a web page for that tutorial at:

http://landley.net/ols/ols2007/cross-compiling.html

Which has a lot of good info but unfortunately trailis off into unedited
slush and sentence fragments. (As so many things I never had time to
actually _finish_ do, ala http://landley.net/aboriginal/history.html and
http://landley.net/kdocs/local/hotplug-history.html and so on...)

Rob

 1412089302.0