[Toybox] tar --null

[enh]

On Mon, Jul 25, 2022 at 2:00 PM Rob Landley <rob at landley.net> wrote:

> On 7/25/22 14:58, enh wrote:
> > On Tue, Jul 19, 2022 at 6:34 AM Rob Landley <rob at landley.net
> > <mailto:rob at landley.net>> wrote:
> >     >     Good suggestion, but I'm never sure what actually _is_ easy. I
> shelved
> >     this
> >     >     after doing about half of mp3 identification, which turns out
> to be a
> >     >     surprisingly large rathole due to funky container formats.
> (And I
> >     don't trust
> >     >     anything microsoft's ever touched not to be turing-complete to
> solve...)
> >     >
> >     > heh, i know exactly what you mean because (a) i have this problem
> all the time
> >     > at work, where people don't finish their "starter project" for
> years and (b)
> >     > your specific jpeg size example was one _i_ punted when i
> originally submitted
> >     > jpeg support because it turned out to be non-obvious.
> >
> >     The problem with "leaving easy stuff for other people to do" is they
> don't do
> >     it. I submitted a series of updated patches to make the kernel's
> >     CONFIG_DEVTMPFS_MOUNT work for initramfs and not just the fallback
> root= mount,
> >     and nobody else ever picked that up and put it in.
> >
> >     https://lkml.org/lkml/2017/9/13/651 <
> https://lkml.org/lkml/2017/9/13/651>
> >
> >     That seems quite easy, no? Here it is again 3 years later...
> >
> >     https://lkml.iu.edu/hypermail/linux/kernel/2005.1/09399.html
> >
> >     There's a type of salesmanship in getting Huck Finn's friends to
> paint his fence
> >     which is a completely different skillset from doing the work
> yourself. It's a
> >     quite useful skill I do not have.
> >
> > yeah, but at the same time we do get people asking "is there something i
> can do
> > to help?", and having a pile of fences to paint eases that.
>
> I am terrible at maintaining such piles. :)
>
> (If it looks simple and I haven't already done it, generally there's some
> subtle
> sharp edge. Doesn't mean it was a SIGNIFICANT problem, just what made me
> put it
> down where a drift of papers blew over it...)
>
> >     > i still think this is the "least worst" option though, and that's
> actually one
> >     > reason why i suggested a separate bug for each: it lets people
> thrash about a
> >     > bit until they find one that _is_ easy (for them).
> >
> >     People have been trying to get me to do more with bug trackers for a
> larger
> >     number of years than I like to think about.
> >
> >     There's a whole lot of years of unmedicated ADHD tangled up in
> there. I use them
> >     when somebody ELSE manages them and regularly reviews what's been
> sitting there
> >     composting.
> >
> > yeah, sorry for not having stayed on top of that after the last time you
> > mentioned it was an issue. i've been through just now and closed out the
> stuff
> > where we think it's fixed (but haven't heard back) or you've said you're
> not
> > going to do the thing (such as move README to markdown).
>
> Thanks. I reopened two of them. :)
>
> >     My objection to ASAN is I'm not yet convinced it ISN'T a false
> positive
> >     generator, although I should give it a closer look. (My first
> encounter with it
> >     being commit 472599b99bec is a contributing factor here.)
> >
> >     > with one of your diff.c changes. i've sent a patch (and a separate
> patch
> >     to add
> >     > that -Werror= to the default toybox configure, since that's one we
> always have
> >     > to fix in the end anyway; may as well catch them fresh?).
> >
> >     I agree I should hit the false positives before you hit the false
> positives.
> >
> >     I need something like a ./testy.sh script that builds with the NDK
> (ASAN
> >     enabled) and runs the test suite... which involves getting the test
> suite to
> >     pass when built with the NDK. Working on it, I'll try to go faster
> and see if I
> >     can reshuffle the priorities a bit. I have been accused of trying to
> boil the
> >     ocean on more than one occasion...
> >
> > yeah, though i think asan by default is a great idea, i think the NDK
> only adds
> > to your problems with no real benefit.
>
> I added ASAN=1 to the llwrap script I did building with NDK, but as I
> mentioned
> in the github comment I need to fix passwd.c now the compile time probe
> isn't
> skipping it. (There's no reason to use the "shadow" functions if I've got
> direct
> /etc/passwd reading logic in lib anyway, I just hadn't finished converting
> stuff
> over. I've got a large patch in a tree here for that, and promoting a half
> dozen
> toys/pending/useradd.c variants as part of it, but that's DEFINITELY
> something
> to test under mkroot and it's queued up behind getting mkroot to run the
> test
> suite...)
>
> > my point with asan is that any C
> > programmer should probably just run with that on all the time.
>
> This promise was made by like 15 different address sanitizer packages over
> the
> years, from "electric fence" through the bounds checker built into tinycc,
> and I
> broke every one I tried. But if this one's different...
>

it catches a lot more than those tools (because it instruments reads and
writes; it's closer to valgrind than to them), and it's fast enough that we
*do* run with it on all the time. we run all the Android tests (CTS and ad
hoc like the toybox tests) under asan (x86-64) and hwasan (arm64, but
hopefully "coming soon" for x86-64 with intel's Linear Address Masking and
whatever amd's equivalent is called).

so even if you do manage to break it ... we'll need to work around that at
some point before we can sync AOSP anyway. (and i know the folks who
invented and maintain asan, so i'm not worried even if you do find
anything.)


> (I still have "set up valgrind" on my todo list,


valgrind never really worked on Android, and upstream were terrible about
taking patches so i just gave up.


> and need a memory leak detector
> for toysh when that's ready to promote.


asan's that too (unless you tell it not to be). like i said: no C/C++
programmer should be without it :-)


> That's a long running thing where I need
> to see if I've missed any allocation lifetimes after running real world
> scripts
> through it. I can police filehandles via /proc/self/fds but not heap
> objects...)
>

(fwiw, bionic has an fdsan/fdtrack system for this kind of thing, but it's
not on for "random executables" by default. well, fd*san* is, but that's
correctness not leaks. fdtrack is only enabled in system_server, aka "the
giant monolith" which suffers most from a tragedy of the commons as far as
leaks are concerned.)


> > it's great at
> > catching memory errors early and it's pretty damn cheap on x86-64 ---
> unless i'm
> > benchmarking i run most stuff under asan most of the time. (and toybox
> doesn't
> > have a lot of Android-only code, so it's unlikely that we'd get
> significantly
> > better coverage from building with the NDK, and the *platform* is always
> ahead
> > of the NDK anyway, so "builds with the NDK" isn't as strong a guarantee
> as you
> > might think, even ignoring the fact that if you _do_ build with the NDK,
> we
> > don't actually build the Android-specific stuff in exactly the same way.)
>
> Oh sure. I've just never managed to build with the AOSP toolchain outside
> of
> AOSP, so the NDK is the closest thing I've got. (The only musl+llvm
> toolchain I
> have is that hexagon cross compiler...)
>

again, don't worry about it --- if you want to do something, build with gcc
and clang. it's not common that we have "our clang is so new that..."
issues, and it's likely we can test and send you patches in less time than
you'd have wasted doing "mostly fine" NDK builds.


> Alas, AOSP hasn't got an obvious mkroot equivalent where I can get JUST
> build a
> quick boot to shell prompt test. It's "set that thing running and go to
> lunch"
> territory, and not something you can usefully do on battery either...
>

yeah, _that's_ a bit more of a loss. but tbh you can probably fake most of
our differences by just being in a chroot with /bin/sh pointing to mksh :-)

if that was where you were going with your question about "what filesystems
are allowed?", i can try harder to get a canonical answer (or at least just
tell you what we're currently using on Pixel and/or cuttlefish!).


> Rob
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20220725/ea8a4789/attachment-0001.htm>