[Toybox] gcc vs llvm -static-libasan

[enh]

On Tue, Jul 9, 2024 at 10:02 AM enh <enh at google.com> wrote:
>
> On Tue, Jul 9, 2024 at 8:47 AM Rob Landley <rob at landley.net> wrote:
> >
> > On 6/12/24 16:08, Rob Landley wrote:
> > > <p>The new debian toolchain also broke gcc/glibc ASAN, complaining (at runtime)
> > > "ASan runtime does not come first in initial library list; you should
> > > either link runtime to your application or manually preload it with
> > > LD_PRELOAD." which is that <a href=27-04-2024>library ordering</a>
> > > nonsense back to rear its ugly head again and I refuse to humor
> > > these INSANE ASSHOLES. If LLVM/bionic works without this, then it's
> > > NOT REQUIED, they're just really bad at it. Notice how the error message
> > > doesn't don't say which library to LD_PRELOAD if I _did_ want to fix it,
> > > it just refuses to work where the previous version worked. A clear regression.
> > > Which I'm late enough in reporting it's a fait accompli, and I'm in the wrong
> > > for not noticing their fuck-up in a timely manner. Far too late to start making
> > > a fuss about it now...</p>
> > >
> > > <p>(Is a required library not installed? I used "build-essential" instead
> > > of manually installing gcc and make precisely so it would scoop up that kind
> > > of nonsense... And it's complaining about library ORDERING, which is not
> > > supposed to be a thing when dynamic linking.)</p>
> >
> > So the local fix I've had for this is to add -static-libasan to CFLAGS in the
> > ASAN portion of scripts/portability.sh which fixes gcc. Note that I need this to
> > do a NON-STATIC ASAN=1 build, because otherwise I get the "runtime does not come
> > first" error above. And no, this is not a toybox issue:
> >
> > $ gcc hello.c -fsanitize-address -O1 -g -fno-omit-frame-pointer
> > -fno-optimize-sibling-calls
> > $ ./a.out
> > ==5942==ASan runtime does not come first in initial library list; you should
> > either link runtime to your application or manually preload it with LD_PRELOAD.
> >
> > But I wanted to regression test -static-libasan against the NDK before checking
> > it in, and I just installed ndk-r26d on the new laptop to do that.
> >
> > The first reason it doesn't work is that llvm chose -static-libsan instead. I
> > note that gcc does not accept "static-libsan" and llvm does not accept
> > "-static-libasan".
> >
> > The SECOND reason it doesn't work is that a --static linked NDK toybox binary
> > with -static-libsan says "error: undefined sunbol: _DYNAMIC" referenced by
> > sanitizer_linux.cpp called from libclang_rt.asan-x86_64-android.a
> >
> > To reproduce that, I extracted the ndk zip file into my home directory and did:
> >
> > $ ln -s ~/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin ~/llvm
> > $ echo -e '#!/bin/bash\n"$(dirname "$0")"/clang --target=x86_64-linux-android34
> > "$@"' > ~/llvm/llvm-cc
> > $ chmod +x ~/llvm/llvm-cc
> > $ cd ~/toybox/toybox
> > $ CROSS_COMPILE=~/llvm/llvm- LDFLAGS=--static make clean defconfig toybox
> >
> > And then switched off SU and LOGIN in .config because of crypt (it's on the todo
> > list), but the rest built! But then adding ASAN=1, the NDK build broke, although
> > -static-libsan doesn't seem to affect that (it builds dynamic with it but I
> > can't run the result, and the build breaks static without it because the OTHER
> > asan libraries inappropriately assume dynamic linking).
> >
> > I'd like to check in the -static-libasan at the end of the CFLAGS string for
> > ASAN to fix the debian build (on the theory ASAN doesn't work for me in the NDK
> > _anyway_), but I don't want to break the (dynamic) android build (which
> > complains it's an unknown flag with "do you mean libsan").
> >
> > Any suggestions?
>
> none you're going to like, but...
>
> you're basically checking all the "not supported" boxes here: x86-64
> (as opposed to arm64), asan (as opposed to hwasan), static (as opposed
> to dynamic). i believe that arm64 hwasan static binaries "just work",
> and if they don't, that's something we'd actually be able to find the
> time to look at. until there's x86-64 hardware with whatever they're
> calling "x86-64 top byte ignore", though, you're stuck with asan, and
> that's been known to be mostly/somewhat broken on Android for years
> (with no-one to work on it).
>
> why can't you run the dynamic version? i thought you'd upgraded your
> laptop? is the new one also too old to run ~2012-era x86-64 binaries?
> alternatively, build arm64 hwasan binaries and run them on your phone
> or orangepi?
>
> i'll forward this to the llvm asan folks anyway, in particular the
> "gcc and llvm use different flags" is the kind of papercut that i
> think both gcc and llvm have been trying harder to avoid lately.

https://github.com/llvm/llvm-project/pull/98194 for that.

apparently your "static asan" request is even harder than i thought...
the folks who actually know what they're talking about (because they
implemented all of these things), fully static asan isn't a thing ---
"bionic's hwasan support is a special well-integrated combination that
works in static binaries, but everything else relies on symbol
interposition and the presence of the dynamic loader".

so my default knee-jerk "have you tried hwasan?" was closer to the
mark than i realized at the time :-)

> > Rob