[Toybox] gcc vs llvm -static-libasan

[enh]

On Tue, Jul 9, 2024 at 2:49 PM Rob Landley <rob at landley.net> wrote:
>
> On 7/9/24 09:02, enh wrote:
> > On Tue, Jul 9, 2024 at 8:47 AM Rob Landley <rob at landley.net> wrote:
> >> Any suggestions?
> >
> > none you're going to like, but...
>
> I don't have to like it, I just have to make it work. :)
>
> > you're basically checking all the "not supported" boxes here:
>
> Welcome to my life.
>
> > x86-64 (as opposed to arm64),
>
> The android-ndk-r26d/toolchains/llvm/prebuild directory only has linux-x86_64
> binaries. (The download page hass Platform "Linux 64-bit (x86)" as the only
> Linux option. It can produce arm output, but doesn't provide a toolchain that
> would run on an arm system, therefore a native compile-and-test cycle...

yeah, but clang's always a cross-compiler. just _target_ arm64.

(yes, you're not the only one who'd like a linux/arm64-hosted NDK, but
supporting another host architecture isn't currently funded. i haven't
yet heard a request for Windows/arm64, but give it time...)

> > asan (as opposed to hwasan),
>
> Which wasn't available on x86-64 last I checked.

not until LAM (intel's Top Byte Ignore) is in actual hardware, no.

> I could whip up a raspberry-pi alike if pushed, but if you don't publish an ndk
> that runs there... (I could try building it from source again?)

i'm not aware that anyone's ever tried building the NDK for
linux/arm64 hosts. easy mode is definitely "cross compile from
linux/x86-64 to android/arm64".

> Adding qemu system emulation into the compile and test cycle is on the todo
> list. Adding qemu APPLICATION emulation is a recipe for weird corner cases in
> system call translation and something I've generally tried to avoid because
> system call translation is a categorically harder problem than emulating
> hardware interfaces. (Containers leverage the existing kernel, based on work
> done by OpenVZ since 1999, running the same host/guest binaries with all the
> endianness and page size and magic constants and structure packing issues
> guaranteed identical, and it STILL took 10 to get it load bearing.)
>
> > static (as opposed to dynamic).
>
> Again, running on a debian host. When they come out with debian-bionic I'll
> happily debootstrap a chroot for testing.

you don't need to do anything [very] fancy to run static bionic
binaries on debian. i do it fairly regularly. (32-bit is more painful,
but it's time to stop poking that corpse.)

there's even a shell script to do the setup for _dynamic_ binaries:
https://android.googlesource.com/platform/bionic/+/main/build/run-on-host.sh

> Getting an android chroot to work as a development environment is the goal,
> hence chicken-and-egg problem.
>
> > i believe that arm64 hwasan static binaries "just work",
> > and if they don't, that's something we'd actually be able to find the
> > time to look at. until there's x86-64 hardware with whatever they're
> > calling "x86-64 top byte ignore", though, you're stuck with asan, and
> > that's been known to be mostly/somewhat broken on Android for years
> > (with no-one to work on it).
> >
> > why can't you run the dynamic version? i thought you'd upgraded your
> > laptop?
>
> I did a fresh OS install on a spare laptop of the same model, but it's still a
> glibc system.

(see the script above, unless you're averse to a /system symlink on
your host. but even if you are, you could probably just hide that in a
chroot.)

> > is the new one also too old to run ~2012-era x86-64 binaries?
>
> It runs the static binaries just fine, but it's a glibc system: "./toybox:
> cannot execute: required file not found". (And then "ldd toybox" said
> "/usr/lib/x86_64-linux-gnu/libm.so: invalid ELF header" which isn't _entirely_
> surprising given that gnu/ldd is a bash script calling glibc but was another
> "wha...?" moment I did not pursue further.)
>
> I vaguely remembered you saying you could create a "/system" symlink into the
> ndk somewhere, so I ran readelf on the binary which is requesting
> /system/bin/linker64 and did a "find android-ndk-r26d -name linker64" and there
> were no hits, so even setting up a dynamic chroot for it remains a todo list item.

yeah, that's the dynamic linker binary off the device. (well,
realistically "emulator" when you're talking about x86-64, but ykwim.)

> > alternatively, build arm64 hwasan binaries and run them on your phone
> > or orangepi?
> >
> > i'll forward this to the llvm asan folks anyway, in particular the
> > "gcc and llvm use different flags" is the kind of papercut that i
> > think both gcc and llvm have been trying harder to avoid lately.
>
> Thanks.
>
> As long as ASAN works for me _somewhere_ I don't really care where (should be
> the same bugs), and I mostly want all the tests to pass with ASAN before cutting
> a release (which is due again: time flies like an arrow, fruit flies like an
> apple, of the two time flies eating all your arrows sounds like the bigger
> problem...).
>
> But if I can't check in the "make it work with gcc on debian" flag without
> breaking the Android build, I has a sad. :(

(eh, arguably the "Android" build is the Android.bp file in AOSP, and
keeping that working is my problem :-) but, yeah, looks like i'm a
week behind and there's an entire awk gone in while i wasn't looking!
i'll sync now...)

> Rob