[Toybox] [PATCH] Fix truncate.test for macOS.
enh
enh at google.com
Mon Jul 4 11:08:12 PDT 2022
On Mon, Jul 4, 2022 at 3:14 AM Rob Landley <rob at landley.net> wrote:
> On 6/27/22 14:40, enh wrote:
> > On Sat, Jun 25, 2022 at 3:45 PM Rob Landley <rob at landley.net
> > <mailto:rob at landley.net>> wrote:
> >
> > On 6/24/22 19:35, enh wrote:
> > > On Thu, Jun 23, 2022 at 11:48 PM Rob Landley <rob at landley.net
> > <mailto:rob at landley.net>
> > > <mailto:rob at landley.net <mailto:rob at landley.net>>> wrote:
> > >
> > > On 6/22/22 20:02, enh wrote:
> > > > On Wed, Jun 22, 2022 at 1:52 PM Rob Landley <rob at landley.net
> > <mailto:rob at landley.net>
> > > <mailto:rob at landley.net <mailto:rob at landley.net>>
> > > > <mailto:rob at landley.net <mailto:rob at landley.net>
> > <mailto:rob at landley.net <mailto:rob at landley.net>>>> wrote:
> > >
> > > The problem with the mac tar test is even though it's easy
> enough to
> > find what
> > > /etc/passwd calls UID 0:
> > >
> > > ROOT="$(sed -n '/[^:]*:[^:]*:0:/s/:.*//p' /etc/passwd)"
> > >
> > > That doesn't change the fact it'll be putting a different
> string into the
> > > tarball, with different sha1sums. Um. (I was using "root" as
> the one known
> > > constant account that didn't vary across distros. Possibly I
> need a
> > way to tell
> > > it to use an alternate /etc/passwd file to lookup usernames.
> This is
> > why I've
> > > been poking at mkroot, but making that work on a mac is
> just... ow.)
> >
> > FYI, I committed your patch shortly after sending that message.
> >
> > thanks. interestingly, i realized that i think we also wouldn't get a
> red cross
> > in the github ui if we broke the _linux_ tests?
>
> Hmmm... looks like I broke VERBOSE=all's exit code when I moved the tests
> into a
> subshell (commit e00b4c26553b) and de474ba03950 wasn't a full enough fix...
>
> Try now?
>
looks plausible: github's saying that linux passed (
https://github.com/landley/toybox/runs/7178532773?check_suite_focus=true)
but macos failed (
https://github.com/landley/toybox/runs/7178532509?check_suite_focus=true).
> > it's only a build failure that
> > counts? not obvious to me from the .yaml syntax why that is/what we
> could do
> > about it.
>
> Wasn't the yaml, it was my fault.
>
> (I semi-regularly do "make distclean defconfig toybox tests" but that
> stops at
> first failure with a visible result, so I hadn't noticed the return
> code...)
>
> That said, I'm not sure the MAC tests failing is really something we're
> ready to
> call a failure yet?
>
i look at the red/green status specifically for this purpose. well, not
specifically for the _mac_, but to see if something's going to be broken
the next time i take an update, and get it fixed upstream first.
and the good news is that it's _only_ tar that's failing.
(btw, one thing i did with the Android toybox test runner was collect the
_names_ of the failing tests. that's a handy thing to have at the bottom of
the output, but perhaps more handy for me when i have to answer the
question "is this bug a dupe of the existing one i'm already looking at, or
do i have two problems?".)
> > I can't immediately think of a better short-term fix, with the
> possible
> > exception of tagging tests as "known to fail on macos because macos
> is buggy".
> > (And "we extended a zero length file three times with truncate() and
> along the
> > way it allocated a megabyte of storage to store LITERALLY NO DATA"
> sounds like a
> > bug to me. I am neither interested in fixing nor reporting MacOS
> bugs because
> > they're 100% proprietary with 0% open source input, and they ain't
> paying me to
> > make them richer thanks. For the same reason, I don't want to put a
> lot more
> > cycles into _thinking_ about macos either.)
> >
> > The mkroot stuff is all about "I can mount ext2 or tmpfs to run this
> test on and
> > have exactly known behavior". I understand "somebody ran the test on
> xfs and it
> > behaved differently than any other filesystem so far", but I think
> this is a bug
> > in the VFS layer in a test environment I haven't got. When porting
> tests into
> > mkroot, I'd presumably do some annotation for "this test runs in the
> > known/mkroot environment" anyway, and logically I'd tag the ones
> that have known
> > problems outside that environment, whatever those problems may be...
> >
> > or we could have a more specific fs-specific "skipnot", since "what fs
> is this?"
> > seems to be one of the most common problems.
>
> Is there a portable way to determine filesystem type, though? df . doesn't
> say,
> I have to look in /proc/mounts and I doubt mac has that?
>
> $ grep -w "^$(df . | tail -n 1 | toybox cut -DF 1)" /proc/mounts | toybox
> cut -DF 3
>
> (I THINK that if the device has a space in it df will output the escaped
> version, which should match for grep... But again: Linux.)
>
on the mac it looks like mount is your best bet:
~$ mount
/dev/disk3s1s1 on / (apfs, sealed, local, read-only, journaled)
...
but maybe just "it's a mac, it's probably using apple's default fs" is
sufficient?
> > Another TODO item is packing up debootstrap and alpine root
> filesystems to test
> > under mkroot as more rigorous "TEST_HOST=1" runs. With the kernel
> > config/version, mountpoint selection, and qemu board emulation
> parameters.
> > Presumably running my init script instead of theirs to do the setup
> and start
> > the test, but using their $PATH of binaries (gnu/fsf and busybox,
> respectively).
> > But that's after I get the base mkroot testing well...
> >
> > > note that it's /etc/*group* that's weird, not *passwd*. uid 0 is
> root, but
> > group
> > > 0 is wheel. (i think that's true of all bsds?)
>
> I suspect the easy solution is "skipnot grep -qw root /etc/group".
>
> Because it's not gonna make the same tarball otherwise, and this is
> testing data
> fetched out of /etc/passwd. (We have --owner=NAME:ID tests: this isn't
> that.)
>
wouldn't it make more sense to use one of the low-numbered groups that's
the same on both though, rather than skip?
> > I'm not spotting any negative gids in /etc/group on devuan. And I
> think that
> > violates posix?
> >
> >
> https://pubs.opengroup.org/onlinepubs/9699919799/functions/chown.html
> >
> > chown(-1) means "don't change". So you can't set it to -1 through
> the posix
> > specified API.
> >
> > they're just taking advantage of a scanf("%u") somewhere else. the
> interesting
> > part is that this means those _aren't_ actually the 64Ki and 64Ki-1 i
> was expecting:
>
> Wait, scanf("%u") will accept an input starting with a minus sign?
>
> Yes, posix says so, and the boilerplate says they got that from C99. They
> defer
> to strtoul() which says "If the subject sequence begins with a
> <hyphen-minus>,
> the value resulting from the conversion shall be negated" but nothing
> about what
> strtoul() should do about that...
>
> (I'm interested because the C++ loons who hijacked gcc development declared
> signed integer wrapping to be "undefined behavior", a thing I noticed a few
> years back when the compiler "optimize out" code that did that with a
> constant
> at compile time. I had to typecast it to unsigned and then back again to
> do the
> math...)
>
> > ~$ id nobody
> > uid=4294967294(nobody) gid=4294967294(nobody)
> > groups=4294967294(nobody),12(everyone),61(localaccounts),100(_lpoperator)
> >
> > but the small positive ones look okay?
>
> Linux uses 65534 for nobody, and it's the OTHER magic UID in Linux
> hardwired
> into the kernel. Well, these days they give you a gratuitous sysctl to
> change
> it, but I've never seen it used. And they don't have an equivalent sysctl
> to
> move root off of UID 0.
>
> include/linux/highuid.h:#define DEFAULT_OVERFLOWUID 65534
>
> It's kind of like the new ping API needing you to
> /proc/sys/net/ipv4/ping_group_range before it can be used. Linux kernel
> development collapsed into a pile of bureaucracy some years back. Kind of
> sad.
>
> They just broke my "build with gelf" fix for a third time:
>
>
> /home/landley/toybox/cleanser/root/build/x86_64-tmp/linux/tools/objtool/include/objtool/elf.h:10:10:
> fatal error: gelf.h: No such file or directory
>
> No other architecture requires you to install a magic extra elf package on
> your
> host, but the x86 maintainers are insane:
>
> https://lore.kernel.org/lkml/20211024192742.uo62mbqb6hmhafjs@treble/
>
> (No other target needs this. I'm building kernels to run in a dedicated
> environment. I don't need spectre/meltdown mitigation, and Linux has THREE
> COPIES of elf plumbing in the tree already before this! Pulling in an
> external
> package to do ELF stuff is just SAD here...)
>
> > $ rm -f empty; for i in 1k 1m 1g; do truncate -s $i empty; stat -c
> %b empty;
> > done; ls -l empty
> > 0
> > 0
> > 0
> > -rw-r--r-- 1 landley landley 1073741824 Jun 25 17:15 empty
> >
> > I'm guessing it's not gonna say 0.
> >
> > ~/toybox$ rm -f empty; for i in 1k 1m 1g; do ./toybox truncate -s $i
> empty;
> > ./toybox stat -c %b empty; done; ls -l empty
> > 0
> > 2048
> > 2048
> > -rw-r--r-- 1 enh primarygroup 1073741824 Jun 27 12:38 empty
> > ~/toybox$
>
> So "truncate -s 1k empty; truncate -s 1m empty" allocates a megabyte of
> disk space.
>
> The second truncate makes the file NOT SPARSE.
>
> > the weird part for me was that it wasn't obvious _what_ the non-zero
> number was
> > going to be.
>
> The bug seems to be if you extend a sparse file the result is not sparse.
> I'm
> guessing if you go:
>
> truncate -s 1k empty; truncate -s 2m empty; stat -c %b empty
>
> You'll get 4096. (Because 2048*512=1m and that's what we asked it to
> expand the
> sparse file to.)
>
/tmp$ ~/toybox/toybox truncate -s 1k empty; ~/toybox/toybox truncate -s 2m
empty; ~/toybox/toybox stat -c %b empty
4096
> > > They swapped to zsh three years ago:
> > >
> > >
> >
> https://www.theverge.com/2019/6/4/18651872/apple-macos-catalina-zsh-bash-shell-replacement-features
> > >
> > >
> > > oh, yeah, good point. my reaction to that was similar to your
> reaction to
> > dash.
> > > _i'll_ be using bash on macos until they remove it.
> >
> > All my stuff says #!/bin/bash at the top, but I dunno how github is
> running
> > what...
> >
> > just `VERBOSE=all make tests`
>
> Which might be fixed now, for a definition of "fixed" that means "notices
> it's
> failing".
>
> Lateral progress!
>
> > > I'm tempted to borrow my wife's mac for a bit, but I have no
> idea how
> > to set up
> > > a development environment on a mac. The first google hit is
> > > https://sourabhbajaj.com/mac-setup/Xcode/
> > <https://sourabhbajaj.com/mac-setup/Xcode/> which looks... more
> elaborate than I
> > > want to do on a borrowed machine.
> > >
> > > iirc it's a bit simpler than that (if you don't have some company
> policy that
> > > says you can only install binaries from their servers) --- you
> just run "make"
> > > and it pops up a window saying "you want to install all that
> shit?" and
> > you say
> > > "it's a unix system; i know this", bish bash bosh, job done.
> >
> > Except I can't easily _undo_ it afterwards and don't want to eat I
> dunno how
> > many gigs on my wife's machine with Apple's soldered-in ssd.
> >
> > yeah, it's not small.
>
> It's a pity there isn't a mac dev environment I can ssh into. I can can
> think of
> three Linux ones off the top of my head that are still up which I probably
> have
> credentials for. (Make that five. Probably more if I thought about it.)
> But then
> Fabrice Bellard's jslinux run a Linux vm in a web page, so "lemme try this
> out
> really quick" was never a high bar for Linux.
>
> Mac, not so much...
>
> Rob
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20220704/4dc70442/attachment-0001.htm>
More information about the Toybox
mailing list