[Toybox] [mkroot] Cannot Overwrite non-directory "$ROOT/bin/" with directory "[Path to overlay]"

[Rob Landley]

On 5/6/24 23:33, Oliver Webb wrote:
> On Sunday, May 5th, 2024 at 21:21, Rob Landley <rob at landley.net> wrote:
> 
>> Oh, the other todo item here is "multiple overlays". The current overlay package
>> was a quick hack, never did the design work to figure out what what more
>> complication should look like. Partly waiting for people to complain to me that
>> they need more than it does...
> 
> Maybe making the OVERLAY variable a delimiter separated list, looping over
> it each time the overlay package is specified. Then indexing the OVERLAY variable like a
> array with that counter (I don't really know how bash arrays work, I think this is easy
> with them though from my vague knowladge, although I don't know)?

Various things are easy to implement, the question is what user interface works
best. The rest of mkroot uses CSV internally so having CSV in an option isn't
that heavy a lift. (Although it hasn't been presented as external UI before, and
relative vs absolute paths in a comma separated list is a bit tricksy, and we
never DID address "what happens if you define the same variable twice on the
command line? Right now it overwrites...)

The other sharp edge is "when files conflict between overlays, do you overwrite
or leave the old one or what".

And of course the "following symlinks out of tree" problem, which I added a tar
option to address and what I've vaguely thought of doing here was having toybox
tar handle it doing the tar c | tar xv trick with --restrict to just leverage
the existing stuff.

And I have a note about sparse file handling, which is at least 3 todo items
combined into one note:

1) cpio sparse handling is part of the periodic https://lwn.net/Articles/789228/
threads that never resolved last I checked, the last attempt wound up diverging
into https://lkml.iu.edu/hypermail/linux/kernel/2207.3/06939.html which
eventually went upstream (after it got completely rewritten to not smell like me
so Greg KH would tolerate it) but the cpio extension part didn't get brought
back up that I've been cc'd on...

2) tar sparse handling should have both modes (SEEK_HOLE and detect runs of
zeroes), and then the tar.test stuff updated to mostly use the runs of zeroes
because there are some TERRIBLE FILESYSTEM implementations out there and none of
them seem to agree on span granularity. (How big IS the run of zeroes? Where are
the edges? Just seek past 4k aligned blocks isn't good enough, and it doesn't
look like 64k is either. Don't get me started on "ecryptfs"...

3) add sparse support to cp.c. (Grumble grumble --sparse longopt without short
opt, and should --sparse=auto be the default behavior? If the filesystem doesn't
support sparse files then presumably seek-and-write will zero fill anyway and we
don't have to do anything. Or seek would fail, which I guess we should
gracefully handle but sendfile_pad() already has plumbing for that?)

>> It hasn't got "make". Kind of limiting factor not to have a make command on the
>> target.
> 
> gmake has a "./build.sh" that you can use to bootstrap it up on a system
> without make. My first step in this after I hacked together a overlay was
> to get a gmake tarball and try to build it with "./configure && ./build.sh && ./make install",
> which configure (on host bash, not toysh) goes into a infinite loop without
> expr, and putting that in will fail because "host compiler does not produce
> run-able executable" (Which might be true because I have to manually hack together
> a overlay each time and I throw out quick "hello world" tests mostly).

Good to know.

Way back when, I had a script that would splice the toolchain.sqf into the host
filesystem with a bunch of symlinks, ala
https://github.com/landley/aboriginal/blob/master/system-image.sh#L65 splicing
together
https://github.com/landley/aboriginal/blob/master/sources/toys/dev-environment.sh
and https://github.com/landley/aboriginal/blob/master/sources/toys/make-hdb.sh
although the interesting part was probably
https://github.com/landley/aboriginal/blob/master/sources/toys/dev-environment.sh#L72

And THAT was based on the old environment setup I used to do in Firmware Linux
to give User Mode Linux a mostly writeable chroot despite starting with
https://user-mode-linux.sourceforge.net/hostfs.html but that was back before git
was invented so I just have a bunch of tarball snapshots over the years (at
https://landley.net/aboriginal/downloads/old/) rather than

Pretty sure I have old blog entries at https://landley.livejournal.com
explaining what I was doing and why, but ever since the servers moved around I
haven't wanted to fish in them, archive.org is slow and has terrible UI, and my
backup disks from that period are... somewhere. Everything's still packed from
the move, I could even find a working turtle board yesterday. (Found a
non-working turtle board, have to look up how to re-flash the bitstream again
because it had a VPN hat and the kernels I've been building are not compatible
with the VPN hardware setup, but to take the hat _off_ I need a different FPGA
bitstream driving the processor from the turtle board's internal clock crystal
instead of the USB chip's exported clock signal...)

> Thank you for the large amount of useful technical information.

You're welcome. Glad you found it useful.

> P.S. About Tinycc: The <slash>insecure</slash>"mob" branch of tcc can build
> toybox in 5.5x less time than gcc to produce a executable only 25% bigger.

Ooh, did they finally get that to work? I maintained my own tinycc fork for 3
years long ago (https://landley.net/hg/tinycc) but
https://landley.net/code/tinycc and https://landley.net/code/qcc and I basically
just didn't have _time_ for it...

I was trying to get it to A) build an unmodified Linux kernel (tccboot was an
old 2.6 kernel that had been modified to rephrase several constructs the
compiler couldn't handle at the time), B) turn it into a multicall binary so you
could symlink cc, ld, as, strip, objdump and so on to the tinycc binary and it
would behave as the appropriate command, C) add more target support (at the time
using qemu's tcg seemed and easy way to get support for all the boards qemu
could emulate)..

> The only 2 changes are to not link with LDOPTIMIZE flags tcc doesn't have,

Shouldn't be hard to get it to gracefully ignore them?

> and to define the QUIET macro in portability.h that isn't defined on anything
> but clang and gcc. I haven't tried this on anything but the mob branch, and
> the old version of tcc most repos provides chokes on the headers.

The old version was all I'd tested. Grishka only seemed to be interested in
windows, and the mailing list got quiet for a while there. I wonder who took an
interest in fixing it up? (Does it FINALLY have dead code elimination?)

Rob