[Toybox] GNU autotools are not working

[Rob Landley]

On 11/3/24 13:24, Brian Mayer wrote:
> This time I sent it correctly to the list!

And my first reply went just to you, because _I_ forgot to manually add 
the list to my reply window. Sigh...

And cutting and pasting the previous reply into a new reply so the 
threading is right (copy outbox copy into drafts folder and double click 
to open editor still works, but I did not find "add reply msgid" in the 
menu pulldowns), thunderbird completely screwed up the wordwrapping. No 
"disable wordwrap" in the pulldown without a plugin, and its "rewrap" 
option does as much harm as good. Oh well, that's software "upgrades" 
for you. I'd learned how/when to work around the old quirks, now they 
force me to use a different UI where they've given the quirks a big stir...

Oh well.

On 10/31/24 20:28, Brian Mayer wrote:
 > Hi Rob, thank you for taking the time to answer this.
> 
> Sorry my previous email was a bit obtuse, let me give you more
> context.
> 
> I am trying to create an ultra simple meta-distro. The idea is to 
> provide the minimum working system that the user can use to build 
> their things on.

The phrase I've been using is "minimal native development environment":

https://landley.net/aboriginal/about.html

https://landley.net/toybox/about.html

http://lists.landley.net/pipermail/toybox-landley.net/2020-July/011898.html

2013:
http://landley.net/talks/celf-2013.txt
http://www.youtube.com/watch?v=SGmtP5Lg_t0

2017:
https://www.youtube.com/watch?v=Sk9TatW9ino

2019:
https://www.youtube.com/watch?v=MkJkyMuBm3g#t=1m18s

> So I want toybox, a libc, a shell and a c compiler in it, make is a 
> good addition because most software depends on it. The url for the
> project is:
> 
> https://terminal.pink/lin0/tree/index.html

Good luck. I've tried to do exactly this before and share the roadblocks
I hit if you'd like. Back in 2008 I made a presentation on it:

https://landley.net/aboriginal/downloads/presentation.pdf

Bit stale in places, but...

> Sorry about the certificate, I'm working on it.
> 
> In order to do this I started with musl. Compiled it, musl ships a 
> musl-gcc wrapper that simply uses a spec file to tell gcc to use the 
> musl libraries and all, which is basically doing what is described
> here:

I've been very tangentially involved in musl-libc development since
before Rich even named it or made it public (he was on the busybox list
and #busybox irc channel back on Freenode back when I was maintaining
that). I'm the one who convinced him to switch it to a BSD license back
in 2012:

https://landley.net/notes-2012.html#18-01-2012

Last I checked it works fine. It's the libc in Alpine Linux which is a
pretty full distro.

>> Modulo glibc's dependencies (perl!) are sad enough that this time
>> around a musl-libc gcc toolchain was my first pass to establish a
>> baseline, but
 >> 
http://lists.landley.net/pipermail/toybox-landley.net/2024-September/030526.html
>> strongly implies that it's working fine. Plus, you know, alpine
 >> existing. :)
> 
> So for the rest of the project I'm using this compiler, next I build 
> tcc, linux, toybox and mksh. All are ok and work fine, I booted to 
> this system after all this.

So what you built _with_ and the resulting environment you provide
aren't the same thing.

I have an "airlock step" where the first thing I did was built host
versions of the compiler and command line tools I intended to put in the
target, and built everything with them.

I've explained this a number of times over the years, it's probably in
https://github.com/landley/aboriginal/blob/master/www/README

>>> I tested the c compiler and it works,
>> 
>> Define "works". It took years for people to build linux-kernel
>> with llvm, there was a whole PROJECT for it, which has mostly gone
>> the way of uclinux.org (declared victory and disbanded). And I
>> specifically mean "the kernel",
>> 

https://www.cnx-software.com/2012/07/16/ellcc-multi-target-cross-compiler-based-on-clang-and-llvm-compiler-infrastructure/

>> was about trying to get userspace to work...
> 
> Works means it is able to generate valid elf programs and shared
> libs that the musl's ld loader can run.

I got tinycc to do that back in the day too. Building hello world and
building a bootable system have a bit of distance between them.

> I didn't test compiling the kernel with it, but now I'm curious, I'm 
> gonna use the compiled tcc to compile the rest of the system in my 
> following versions.

My old fork taught tinycc basic constant propagation (which I believe
Grischka merged into his version), but it didn't have dead code
elimination, was missing c99 vararrays, didn't even try c11 features
(These days toybox uses struct blah){x=y;} style complex inline
constants and __has_include(), the latter of which is really a
preprocessor feature not a language feature.)

One thing I definitely wanted to do that I have yet to hear of tinycc
implementing is a multiplexer so you can symlink cc/ld/as/strip/cpp to
tinycc and it'll behave as the appropriate binary. Heck, just being able
to use tcc's (much faster) preprocessor in distcc would be a big win,
but alas there's a LOT of implicit architecture knowledge baked into
that. Try running: gcc -E -dM - </dev/null

I remember  trying to get just tinycc's linker to work circa
https://landley.net/notes-2008.html#09-04-2008 but it didn't have the
concept of --gc-sections. That day's blog entry also mentions trying to
get it to work on an arm host but it couldn't build a libtcc.a for that
target so I had to use libgcc...

Grishka wasn't _interested_ in fixing any of these issues, because they
didn't affect Windows.

>> Don't ask me why gnu/they don't gnu/emit the gnu/failures to
>> stderr. That would make too much sense, they need gnu/hazing
>> rituals to keep out the unwashed masses I guess.
> 
> And a ton of extensions to make things non compliant...

So? Compliant with what, Turbo C's "near" and "far" keywords? Microsoft
Visual C? Half the posix-2008 features like openat() started as glibc
extensions, the C committee is just a bit slower on the uptake than
posix. (Which is sad because there was a SIXTEEN YEAR GAP between
posix-2008 and posix-2024, and we're arguing about "make -j" on the list
because of COURSE they didn't standardize "nproc". Linux 2.0 added SMP
support in 28 years ago, Intel shipped its first SMP laptop processor 18
years ago, the first SMP smartphonephone came out 13 years ago... And
posix came out with a uniprocessor standard earlier this year. Of course.)

Intel's icc was the second compiler to build Linux in something like
2005, because they implemented all the gcc extensions Linux used.
Fabrice Bellard's tccboot work also involved either implementing or
stripping out stuff like "a ? : b" as a synonym for "a ? a : b" (which
does not recalculate A), which I think got picked up by one of the newer
C standards? (C11 was it? I forget, I only moved from C99 to C11 because
I wanted __has_include() and the ability to typecast up an inline array
instance, I think there's at least one more C standard released since
but I glance at the change list when they come out and generally go "oh
hey, a different syntax for atomics, they're still ignoring LP64 I
see..." and continue with what I was doing.)

>> Doing toybox _and_ tcc at the same time seems a bit extreme, in
>> general changing multiple variables at once makes experimental
>> results WAY >>

harder to parse. Hence the invention of laboratory conditions. :)

> 
> nahh

I got my project to work. Aboriginal Linux rebuilt itself under itself
and built Linux From Scratch under the result. Alpine Linux built on
that work to do a full distro.

I only shut it down in 2017 because I was starting over with mkroot,
which is A) way simpler, C) uses toybox instead of busybox and musl-libc
instead of uClibc, D) outsources the toolchain build(s) to a separate
script, E) isn't finished yet. :)

>> You might also want to coordinate with
 >> 
http://lists.landley.net/pipermail/toybox-landley.net/2024-September/030526.html
>> (which is another reply window I have open but haven't managed to
>> scrape up the spoons for yet).
> Cool, thanks.
> 
>> (I have "writing a new make" on my toybox todo list,
> 
> That would be awesome, I also tried building bmake (BSD Make) but it 
> also needs me to run ./configure, so I ended up in the same place.

https://landley.net/toybox/faq.html#hermetic

https://github.com/landley/toybox/blob/master/scripts/prereq/build.sh

>> but hit burnout this year. Sometime between moving from TX to MN
>> this spring, getting flooded here on the list by a well-meaning
>> developer who just took the fun out of everything, getting covid
>> YET AGAIN (still sucks), having yet to find a proper coffee shop
>> setup here in minneapolis (workspace is important), and the looming
>> return out outright fascism, I've spent the past few months kinda
>> spinning my wheels...)
> 
> I am so sorry about this, hope you get back on track soon.

Me too. :(

>>> so it may be something related to how GNU autotools interact with
>>> toybox or the compiler.
>> 
>> Do you mean you built gmake with tinycc, and the rest of the $PATH
>> being debian or similar?
> 
> I'm trying to build make on the resulting system (toybox, mksh &
> tcc).

You've changed multiple variables at once.

>>> The errors are hard to follow, but it seems to be related with
>>> the confdefs.h and .c files, after a lot of sed and cat
>>> commands. The idea is to bootstrap the build tools, i.e. compile
>>> make without make or other tools that depend on make.
>> 
>> Ok, I'll bite. How do you compile make without make? (Do they
>> provide a shell script, like toybox's
>> https://landley.net/toybox/faq.html#hermetic or what?)
> 
> Fortunately, the make tarball comes with a bootstrap script that, 
> supposedly, builds make without make.

Cool. Like mine, above. :)

> I guess the gnu tools must start somewhere... But it runs ./configure
> as a first step, which is where I'm stuck.

Yeah, that's silly. Mine does NOT run configure, it's a packaged up
single compiler invocation with known behavior and no dependencies
except libc and the compiler. (The point is your compiler should know
how to produce binaries for your system.)

It wants a shell, but it's just doing simple variable assignment and
substitution, even lash could presumably handle that.

>> That "can't create temporary file" looks like the first error.
>> Which is not a string that exists in toybox (where we'd almost
>> certainly say "temp" to save 5 bytes :) but presumably is configure
>> responding to a command failure? Again, check config.log.
> 
> The contents of config.log:
> 
> This file contains any messages produced by compilers while running
> configure, to aid debugging if configure makes a mistake....
 > configure:982: checking for cc
> configure:1065: checking whether the C compiler (cc  ) works 
> configure:1081: cc -o conftest    conftest.c  1>&5 tcc: error:
> undefined symbol 'main'

That could be a tcc bug, but I dunno what was in "conftest.c". (The test
didn't say -c to create a .o file so I assume there SHOULD be a main()...)

> configure: failed program was:
> 
> This tcc error I was able to reproduce by trying to compile an empty
> file.

Yeah but why was it empty? That compiler invocation doesn't seem like it
expected an empty file. What generated that conftest.c?

Rob