From enh at google.com  Tue Jul  1 09:16:57 2025
From: enh at google.com (enh)
Date: Tue, 1 Jul 2025 12:16:57 -0400
Subject: [Toybox] Shell Compatibility Reports from Oils - ~800 tests
 passing
In-Reply-To: <CAJgzZoo_iLpe9JOb=h1AzD_Dby9WmwUQFfJXuN8VjMP_gWGNvQ@mail.gmail.com>
References: <CAOGGbnOASxYxppmBbmUu3bViE8POpS7+0EGvy19Ogb2seL5O_g@mail.gmail.com>
 <a32e8f8d-a9fb-480b-8196-ad7b9fdea999@landley.net>
 <CAOGGbnNHATGsaGn6LnhEg2Uop4M8Zw6jrfNNF63bunwUQDPzgQ@mail.gmail.com>
 <CAOGGbnOPJ7NsrhOECxhAUQysPBiyyQ74H5fLiFJv8BexBUQ52Q@mail.gmail.com>
 <090179ad-a3a0-4b49-a7b8-ad14712522a6@landley.net>
 <CAJgzZoo_iLpe9JOb=h1AzD_Dby9WmwUQFfJXuN8VjMP_gWGNvQ@mail.gmail.com>
Message-ID: <CAJgzZoo74yuuTAaUivZLcA0-7iwBpVqpuPAFoc6tzjCweC+JLg@mail.gmail.com>

+toybox list back in, since i accidentally dropped it...

On Mon, Jun 30, 2025 at 3:20?PM enh <enh at google.com> wrote:
>
> On Sun, Jun 29, 2025 at 6:18?PM Rob Landley <rob at landley.net> wrote:
> >
> > On 6/28/25 23:18, Andy Chu wrote:
> > > Hm I looked at the goals of toybox again:
> > >
> > >> Toybox's main goal is to make Android self-hosting by improving Android's command line utilities so it can build an installable Android Open Source Project image entirely from source under a stock Android system.
> > >
> > >> Toybox aims to provide one quarter of a theoretical "minimal native development environment"
> > >
> > >> In theory, this should only require four packages
> > >
> > > I don't know much about Android -- is this at all realistic for FIVE
> > > packages -- if you add mksh, which I believe is the Android system
> > > shell ?
> >
> > Eh, define realistic. AOSP is built around git (kind of conceptually),
> > and their build infrastructure uses python 3. So you'd build a system to
> > build a system.
>
> AOSP includes its own clang, its own soong/ninja/make, its own
> bison/flex, its own python... bash is one of the few exceptions that
> come from the host:
> https://android.googlesource.com/platform/build/soong/+/master/ui/build/paths/config.go#86
>
> > Aboriginal Linux had 7 packages: linux, busybox, uclibc, gcc, binutils,
> > make and bash. and could built Linux From Scratch under the result in a
> > fully automated target-independent fashion using
> > https://landley.net/aboriginal/control-images/
> >
> > Ok,
> > https://github.com/landley/control-images/tree/master/images/lfs-bootstrap/mnt
> > cheated slightly with one extra package, as
> > https://github.com/landley/control-images/blob/master/images/lfs-bootstrap/download.sh
> > attests. But https://landley.net/aboriginal/mirror/gettext-stub-1.tar.gz
> > was a tiny little thing to stub out some gnu/stupid, stub versions of a
> > dozen internationalization functions that all either returned their
> > first argument, NULL, or "C". The header could have been a here document
> > and then an empty .a file to satisfy gnu builds that insisted on pulling
> > in the library.
> >
> > As for getting mkroot to do what aboriginal linux used to, I have no
> > interest in testing mksh beyond not breaking Android's use of the toybox
> > test suite (which runs it under mksh).
> >
> > The AOSP build is large and has a lot of other dependencies, but
> > Elliott's been doing what he calls "hermetic builds" where AOSP tries to
> > provide a lot of its build prerequisites as shipped binaries, and Toybox
> > provides a lot of those. (Search for the world "hermetic" in toybox's
> > news.html page, it's been mentioned with links a few times.)
> >
> > The https://landley.net/toybox/roadmap.html#dev_env section of the
> > toybox roadmap is my old dependency list that Aboriginal Linux needed to
> > rebuild itself under itself, and then build Linux From Scratch under the
> > result. But it's been a moving target. I regression test kernel builds
> > with mkroot each release. It uses the same "airlock step" that
> > aboriginal had, where the build $PATH is replaced with a single
> > directory with all the binaries the build needs before building the
> > packages:
> >
> > https://github.com/landley/toybox/blob/master/mkroot/mkroot.sh#L54
> >
> > The airlock is mostly set up by toybox's "make install_airlock" target
> > which uses a PENDING and TOOLCHAIN command list, the first being
> > commands that toybox should eventually provide (but doesn't yet) and the
> > second being commands the host needs to provide (mostly the compiler):
> >
> > https://github.com/landley/toybox/blob/master/scripts/install.sh#L105
> >
> > Currently PENDING has: expr git tr bash sh gzip awk bison flex make ar
> >
> > (All but bison, flex, and make have semi-complete "pending" versions in
> > toybox.)
> >
> > And TOOLCHAIN has: as cc ld objdump bc gcc
> >
> > And the last two of those I have patches to remove the need for from the
> > kernel build,
> > https://landley.net/bin/mkroot/0.8.12/linux-patches/0004-Replace-timeconst.bc-with-mktimeconst.c.patch
> > and
> > https://landley.net/bin/mkroot/0.8.12/linux-patches/0001-try-generic-compiler-name-cc-before-falling-back-to-.patch
> > respectively.
> >
> > My tool to instrument a build so I can see every command line called out
> > of the $PATH is currently mkroot/record-commands (which builds
> > toys/example/logpath.c), and descends from the "command logging wrapper"
> > described in https://landley.net/aboriginal/FAQ.html#debug_logging
> >
> > (This doesn't catch the ones called from absolute paths, usually by
> > scripts with #!/usr/blah at the start. Also gmake will call /bin/sh
> > (instead of sh out of the $PATH) unless you set SHELL, see
> > https://www.gnu.org/software/make/manual/make.html#Choosing-the-Shell
> > for the gnu/stupid du jour.)
> >
> > > Can Android even be built on Android at all, with any number of
> > > packages?   e.g. if you download all the dev tools onto an Android
> > > device ...  I imagine it is a ton of tools, and not very fun.
> >
> > That's an Elliott question,
>
> could you? yes. would you want to? not if you value your time.
>
> (and note that prebuilt-wise, we currently support darwin/arm64 but
> not linux/arm64. only linux/x86-64.)
>
> > and they did some sort of container
> > infratructure (which may or may not be related to
> > https://www.youtube.com/watch?v=Eu-rqMHqM6I ) in newer versions of
> > Android than my phone runs, which can presumably install arbitrary linux
> > distros in either containers or VMs, so it's a semi-philosophical
> > question? (But the countering trusting trust stuff still applies.)
> >
> > I've been working _towards_ it since 2011, but... let's just say the
> > past decade has not provided my ideal work environment.
> >
> > > Anyway, if there is something realistic we could do here with OSH,
> > > that may be of interest to our funders http://nlnet.nl
> >
> > https://github.com/landley/toybox/blob/master/toys/pending/sh.c has most
> > of the infrastructure in place already. If I wanted to use bash or mksh
> > in another aboriginal linux style LFS build setup, I could. (And Alpine
> > Linux exists, which benefited from all the busybox work I did back in
> > the day.)
> >
> > > e.g. testing that important packages can actually be built, and
> > > reducing real failures to reproducible test cases.  That is a lot of
> > > real work
> >
> > Which Alpine has presumably done. I'm not trying to patch packages, I'm
> > using them as test cases. Which is how you wind up with stuff like:
> >
> > https://github.com/landley/toybox/commit/32b3587af261
> >
> > Which is CLEARLY THEIR BUG, yet we must cope.
> >
> > >  From some viewpoints it could be theoretical, but proving that you can
> > > build a real system is important!
> >
> > I've done it. The old "lfs-bootstrap" images in
> > https://landley.net/aboriginal/downloads/old/binaries/1.4.1/extras/ were
> > "here's the linux from scratch 6.x root filesystem that built under qemu
> > from the minimal native development environment system image this release".
> >
> > I got FANCY back then. If you're wondering why the (current) airlock
> > scripts detect multiple instances of the same command in the $PATH and
> > symlink them into numbered fallback directories, it's for things like
> > distcc, which the old scripts used to move the heavy lifting of
> > compilation out of the emulated environment to run on the host machine:
> >
> > https://landley.net/notes-2008.html#07-06-2008
> >
> > I probably blathered about that at Ottawa Linux Symposium:
> >
> > https://bootlin.com/pub/video/2008/ols/ols2008-rob-landley-linux-compiler.ogg
> >
> > Still on the TODO list for the new stuff. Back in the day I could get
> > about -j3 usefully going before the emulator became the bottleneck. Well
> > using SMP for the actual compile part, the configure stage was 100% the
> > bottleneck in all the gnu package builds. Still is. More totally
> > unnecessary gnu/stupid: the compiler sets a zillion builtin macros you
> > can see with:
> >
> > $ :|cc -dM -E -
> >
> > And between that, c11's __has_include(), and features.h you can
>
> it's been implemented by clang/gcc for a long time, but if you're
> going to be a standards lawyer about it, __has_include() is c++17 and
> c23.
>
> > eliminate almost all configure time probes because it ALREADY KNOWS.
> > Just set your cross compiler and let your headers pick through the
> > symbols to figure out what to do.
> >
> > One of my many todo items is re-testing whether running ./configure with
> > static linked binaries is still 20% faster under QEMU these days:
> >
> > https://landley.net/notes-2009.html#14-10-2009
> >
> > I _think_ that was back before PLT and GOT were collated into arrays,
> > meaning QEMU the dynamic references were patched in-situ instead of
> > redirecting off an object table, so QEMUJ had to re-translate each
> > executable page every time it was written to (self modifying code REALLY
> > fscks with dynamic translation) meaning the overhead of dynamic linking
> > patching all the jumps in place was just pathological. Then there was
> > that terrible RTLD_LAZY nonsense which SOMEHOW MADE IT WORSE, and of
> > course SOME linking variations would always indirect off the PLT/GOT and
> > others would patch the relocation into the caller as part of the first
> > call... I think -fPIC or not was involved here somehow? (PIE is SORT of
> > nice, but static PIE not using the dynamic linker but STILL DYNAMIC
> > LINKING ITSELF means it has to be STATICALLY LINKING THE RUNTIME DYNAMIC
> > LINKER and that's about where I step away from the keyboard.
> >
> > Don't ask me how using dlopen messes with any of that. Sigh, I keep
> > thinking Rich Felker's dlopen() rant is on https://ewontfix.com/
> > somewhere but no, it's buried in the musl openwall list which Google
> > can't find anymore since
> > https://www.wheresyoured.at/the-men-who-killed-google/
> >
> > Anyway, it's been a while since I last seriously dug into linking,
> > because it's a can of worms.
> > (https://landley.net/bin/mkroot/0.8.11/linux-patches/0002-sh4-fdpic.patch
> > doesn't count because I actually needed it for something.)
> >
> > Sigh, everything has so much backstory. QEMU having to translate pages
> > is a thing I blathered about back when I was trying to do a "qemu weekly
> > news", I explained how/why dyngen worked:
> >
> > https://landley.net/qemu/2008-01-15.html#Jan_17,_2008_-_[PATCH_0_5]_Enable_building_of_op.o_on_gcc4
> >
> > Right before it got ripped out and replaced:
> >
> > https://landley.net/qemu/2008-01-29.html#Feb_1,_2008_-_TCG
> >
> > But the general principles still apply. (SO MUCH of computer science is
> > "we learned how the principles worked from some old obsolete thing
> > that's been replaced, and the new one still works fundamentally the same
> > way but it's a lot more complicated so you can't actually SEE that
> > unless you understand where it came from. It's a pedagogical disaster
> > leading to
> > https://www.landley.net/history/mirror/institutional_memory.html loss
> > and I dunno what to do about that, but what else is new?)
> >
> > > Andy
> >
> > Rob
> > _______________________________________________
> > Toybox mailing list
> > Toybox at lists.landley.net
> > http://lists.landley.net/listinfo.cgi/toybox-landley.net

From rob at landley.net  Tue Jul  1 09:41:29 2025
From: rob at landley.net (Rob Landley)
Date: Tue, 1 Jul 2025 11:41:29 -0500
Subject: [Toybox] clang build still broken
In-Reply-To: <CAJgzZopa372PWJfZMAu4Y0AR_HMWFGEoL7DPWih815bx1KVdFw@mail.gmail.com>
References: <CAJgzZopa372PWJfZMAu4Y0AR_HMWFGEoL7DPWih815bx1KVdFw@mail.gmail.com>
Message-ID: <fc2059cc-d716-47e0-8ee1-4cfec30afc2f@landley.net>

On 6/30/25 14:46, enh wrote:
> sending as a separate thread, so it's obvious that there's new
> information here... in particular, note that clang isn't complaining
> about _attribute_ ordering, it's complaining that the _definition_ it
> already saw doesn't match the redeclaration:

It built for me with the android-ndk-r27. I'm confused.

> external/toybox/lib/hash.c:309:16: error: attribute declaration must
> precede definition [-Werror,-Wignored-attributes]
>    309 | __attribute__((__weak__)) void hash_by_name(int fd, char
> *name, char *result)
>        |                ^

Define "precede". The attribute declaration is now the first thing on 
the line. (That's what commit 17d77a264ab6 changed.)

> external/toybox/lib/hash.c:22:6: note: previous definition is here
>     22 | void hash_by_name(int fd, char *name, char *result)
>        |      ^
> external/toybox/lib/hash.c:309:32: error: redefinition of 'hash_by_name'
>    309 | __attribute__((__weak__)) void hash_by_name(int fd, char
> *name, char *result)
>        |                                ^

That's not "does not match", that's "it was redefined". Which gcc allows 
if only one isn't weak, but clang ignored the weak.

> external/toybox/lib/hash.c:22:6: note: previous definition is here
>     22 | void hash_by_name(int fd, char *name, char *result)
>        |      ^

void blah(int, char *, char *)
void blah(int, char *, char *)

What differs other than one being weak? I'm not spotting it.

Sigh, I can go back to #ifdefs if clang is that buggy, but lib/* built 
with the NDK? In a clean checkout even:

$ git clone toybox clean
Cloning into 'clean'...
done.
$ cd clean
$ readlink -f ~/llvm
/home/landley/android-ndk-r27/toolchains/llvm/prebuilt/linux-x86_64/bin
$ ../llwrap make defconfig
generated/unstripped/kconfig -d > .config
$ ../llwrap make true
scripts/single.sh true
true:generated/{Config.in,newtoys.h,flags.h}
Compile true
....................
$ ./true

My ndk wrapping setup is fairly simple:

$ cat ../llwrap
#!/bin/bash
CROSS_COMPILE=~/llvm/llvm- LDFLAGS+=" --static" CFLAGS+=" 
-Werror=format-security -Werror=unused-result" "$@"
$ readlink ~/llvm
android-ndk-r27/toolchains/llvm/prebuilt/linux-x86_64/bin
$ cat ~/llvm/llvm-cc
#!/bin/bash
"$(dirname "$0")"/clang ${LLTARGET:---target=x86_64-linux-android35} "$@"

Rob

From rob at landley.net  Tue Jul  1 10:52:08 2025
From: rob at landley.net (Rob Landley)
Date: Tue, 1 Jul 2025 12:52:08 -0500
Subject: [Toybox] Shell Compatibility Reports from Oils - ~800 tests
 passing
In-Reply-To: <CAJgzZoo74yuuTAaUivZLcA0-7iwBpVqpuPAFoc6tzjCweC+JLg@mail.gmail.com>
References: <CAOGGbnOASxYxppmBbmUu3bViE8POpS7+0EGvy19Ogb2seL5O_g@mail.gmail.com>
 <a32e8f8d-a9fb-480b-8196-ad7b9fdea999@landley.net>
 <CAOGGbnNHATGsaGn6LnhEg2Uop4M8Zw6jrfNNF63bunwUQDPzgQ@mail.gmail.com>
 <CAOGGbnOPJ7NsrhOECxhAUQysPBiyyQ74H5fLiFJv8BexBUQ52Q@mail.gmail.com>
 <090179ad-a3a0-4b49-a7b8-ad14712522a6@landley.net>
 <CAJgzZoo_iLpe9JOb=h1AzD_Dby9WmwUQFfJXuN8VjMP_gWGNvQ@mail.gmail.com>
 <CAJgzZoo74yuuTAaUivZLcA0-7iwBpVqpuPAFoc6tzjCweC+JLg@mail.gmail.com>
Message-ID: <de43c9bd-4266-4c1c-aeb8-a1fa9f6df79b@landley.net>

On 7/1/25 11:16, enh wrote:
>>>> I don't know much about Android -- is this at all realistic for FIVE
>>>> packages -- if you add mksh, which I believe is the Android system
>>>> shell ?
>>>
>>> Eh, define realistic. AOSP is built around git (kind of conceptually),
>>> and their build infrastructure uses python 3. So you'd build a system to
>>> build a system.
>>
>> AOSP includes its own clang, its own soong/ninja/make, its own
>> bison/flex, its own python... bash is one of the few exceptions that
>> come from the host:
>> https://android.googlesource.com/platform/build/soong/+/master/ui/build/paths/config.go#86

"Building a system to build a system" would initially be creating a 
chroot within which AOSP ran with all its prebuilts (so providing the 
required external dependencies in a way that could theoretically be 
added to AOSP base), but would _then_ involved hermetically rebuilding 
the prebuilts.

Although honestly, half what I did with LFS was untangle what it was 
doing so I could make a less complex but equivalently functional system. 
Even the automated builder was generic build infrastructure:

https://github.com/landley/control-images/tree/master/common/bootstrap

Going over a package list in order (with comments that let grep -v 
filters drop out package categories):

https://github.com/landley/control-images/blob/master/images/lfs-bootstrap/mnt/package-list

To call individual build scripts, many of which were symlinked together 
to a generic build script that could handle a "normal" package:

https://github.com/landley/control-images/tree/master/images/lfs-bootstrap/mnt/build

That's not exactly how AOSP upstream does it, but was provably 
equivalent. :)

AOSP is a large edifice with its own significant domain expertise. Which 
Elliott has (and I do not), but he manages a large team and last I 
checked didn't have a lot of spare bandwidth for cleaning up stuff that 
already works.

>>>> Can Android even be built on Android at all, with any number of
>>>> packages?   e.g. if you download all the dev tools onto an Android
>>>> device ...  I imagine it is a ton of tools, and not very fun.
>>>
>>> That's an Elliott question,
>>
>> could you? yes. would you want to? not if you value your time.

It's feasible. You can buy a new Android phone for $200 with 4 gigs ram 
(plus 8 gigs swap) with 2.4 ghz 8xSMP processor and can use a 1 TB sdcard.

https://www.motorola.com/us/en/p/phones/moto-g/moto-g-2025/pmipmhh39mq?pn=PB5X0002US

Add a USB hub, keyboard, mouse, and video adapter to hook it up to a TV 
(I _used_ to say chromecast but Google discontinued those last year, I 
assume there's some sort of replacement or maybe 
https://en.wikipedia.org/wiki/USB_video_device_class finally means 
something) and you can theoretically use a phone as a workstation.

Current AOSP is resource intensive enough that the last time I built it 
on my laptop it ran overnight and into the next day, but that's a 
separate issue. The above phone (not even one the cheap chinese ones) is 
faster than my laptop and I do real work on it. (Although I did upgrade 
my laptop's ram to 16tb, because I could.)

>> (and note that prebuilt-wise, we currently support darwin/arm64 but
>> not linux/arm64. only linux/x86-64.)

Back under aboriginal I tried to cross compile everything from 
everything once (build a powerpc->mips cross compiler and run it under 
qemu to create a mips chroot) but combinatorial stuff gets uncomfortable 
fast.

But "build and run on the host without having to be told what the host 
is" was generally quite doable. That's why I put it through such a small 
bottleneck though: those base 7 packages building on Linux distros I'd 
never heard of was always 50 times as much trouble "something uniquely 
broke on S390".

I remember when Wolfgang Denk (the u-boot guy) tried Aboriginal and it 
broke spectacularly because he had a whole bunch of weird environment 
variables set, so I created an environment variable sanitizer with a big 
whitelist:

https://github.com/landley/aboriginal/blob/master/sources/variables.sh

And then for mkroot I just used "exec env -i $0 $@" passing through 
$HOME and $PATH (and 4 API variables I was used to providing as 
prefixes), and then had one line in a for loop grab the NAME=VALUE 
arguments and export set them as environment variables, and keep the 
rest of the arguments as additional packages to build out of 
mkroot/packages. Currently mkroot/mkroot.sh starts with:

# Clear environment variables by restarting script w/bare minimum passed 
through
[ -z "$NOCLEAR" ] && exec env -i NOCLEAR=1 HOME="$HOME" PATH="$PATH" \
     LINUX="$LINUX" CROSS="$CROSS" CROSS_COMPILE="$CROSS_COMPILE" "$0" "$@"

# assign command line NAME=VALUE args to env vars, the rest are packages
for i in "$@"; do
   [ "${i/=/}" != "$i" ] && export "$i" || { [ "$i" != -- ] && PKG="$PKG 
$i"; }
done

I spent many years figuring out what I could throw away. That quote from 
the guy who wrote "The Little Prince" was my email sig for a while:

https://www.goodreads.com/quotes/19905-perfection-is-achieved-not-when-there-is-nothing-more-to

>>> Still on the TODO list for the new stuff. Back in the day I could get
>>> about -j3 usefully going before the emulator became the bottleneck. Well
>>> using SMP for the actual compile part, the configure stage was 100% the
>>> bottleneck in all the gnu package builds. Still is. More totally
>>> unnecessary gnu/stupid: the compiler sets a zillion builtin macros you
>>> can see with:
>>>
>>> $ :|cc -dM -E -
>>>
>>> And between that, c11's __has_include(), and features.h you can
>>
>> it's been implemented by clang/gcc for a long time, but if you're
>> going to be a standards lawyer about it, __has_include() is c++17 and
>> c23.

Sigh. Did one of the standards finally include gcc's empty ? : middle 
argument? (Which I use all the time...)

Sadly, standards remain a frame of reference to diverge from...

Rob

From rob at landley.net  Tue Jul  1 17:17:04 2025
From: rob at landley.net (Rob Landley)
Date: Tue, 1 Jul 2025 19:17:04 -0500
Subject: [Toybox] Excluding octal.
Message-ID: <27a270a7-b1cc-4c0d-9b78-787010e72140@landley.net>

I've had this in my tree forever, but haven't checked it in. Anybody 
have an objection to me pulling the trigger?

--- a/lib/lib.c
+++ b/lib/lib.c
@@ -312,7 +312,8 @@ long long atolx(char *numstr)
    char *c = numstr, *suffixes="cwbkmgtpe", *end;
    long long val;

-  val = xstrtol(numstr, &c, 0);
+  // exclude octal to avoid confusion
+  val = xstrtol(numstr, &c, strstr(numstr, "0x") ? 0 : 10);
    if (c != numstr && *c && (end = strchr(suffixes, tolower(*c)))) {
      int shift = end-suffixes-2;
      ++c;

I don't _think_ it'll break anything? I've gotten multiple bug reports 
from people being confused by octal. It was really a thing for 6 bit 
hardware like the PDP-10 back in the 1960s (before ASCII took over), 
which was eliminated by the falling cost of RAM making the ability to 
represent upper AND lowercase letters at the same time more important 
than squeezing 25% more uncompressed text into the same number of bits.

Unix file permissions preserve its PDP-7 heritage in amber, but don't 
use this codepath to parse their input.

Rob

P.S. The strstr() is so "seq 1 -1 -0x10" still works. :)