[Toybox] [PATCH] Fix truncate.test for macOS.

[Rob Landley]

On 6/24/22 19:35, enh wrote:
> On Thu, Jun 23, 2022 at 11:48 PM Rob Landley <rob at landley.net
> <mailto:rob at landley.net>> wrote:
> 
>     On 6/22/22 20:02, enh wrote:
>     > On Wed, Jun 22, 2022 at 1:52 PM Rob Landley <rob at landley.net
>     <mailto:rob at landley.net>
>     > <mailto:rob at landley.net <mailto:rob at landley.net>>> wrote:
> 
>     The problem with the mac tar test is even though it's easy enough to find what
>     /etc/passwd calls UID 0:
> 
>     ROOT="$(sed -n '/[^:]*:[^:]*:0:/s/:.*//p' /etc/passwd)"
> 
>     That doesn't change the fact it'll be putting a different string into the
>     tarball, with different sha1sums. Um. (I was using "root" as the one known
>     constant account that didn't vary across distros. Possibly I need a way to tell
>     it to use an alternate /etc/passwd file to lookup usernames. This is why I've
>     been poking at mkroot, but making that work on a mac is just... ow.)

FYI, I committed your patch shortly after sending that message.

I can't immediately think of a better short-term fix, with the possible
exception of tagging tests as "known to fail on macos because macos is buggy".
(And "we extended a zero length file three times with truncate() and along the
way it allocated a megabyte of storage to store LITERALLY NO DATA" sounds like a
bug to me. I am neither interested in fixing nor reporting MacOS bugs because
they're 100% proprietary with 0% open source input, and they ain't paying me to
make them richer thanks. For the same reason, I don't want to put a lot more
cycles into _thinking_ about macos either.)

The mkroot stuff is all about "I can mount ext2 or tmpfs to run this test on and
have exactly known behavior". I understand "somebody ran the test on xfs and it
behaved differently than any other filesystem so far", but I think this is a bug
in the VFS layer in a test environment I haven't got. When porting tests into
mkroot, I'd presumably do some annotation for "this test runs in the
known/mkroot environment" anyway, and logically I'd tag the ones that have known
problems outside that environment, whatever those problems may be...

Another TODO item is packing up debootstrap and alpine root filesystems to test
under mkroot as more rigorous "TEST_HOST=1" runs. With the kernel
config/version, mountpoint selection, and qemu board emulation parameters.
Presumably running my init script instead of theirs to do the setup and start
the test, but using their $PATH of binaries (gnu/fsf and busybox, respectively).
But that's after I get the base mkroot testing well...

> note that it's /etc/*group* that's weird, not *passwd*. uid 0 is root, but group
> 0 is wheel. (i think that's true of all bsds?)
> 
> it actually looks like gid 0 is one of the few low gids that differs?

Alas I believe this is one of the holes in posix, because Windows NT and System
360 did not have user and group names but they REALLY wanted that FIPS 151-2 money.

They specify ways to change uid and gid of resources, but don't ever say what
any of those IDs _are_. (Unless I'm missing it? They handwave "appropriate
privileges" and such instead of saying uid 0 or root...)

> here's macos' first few gids:
> 
> nobody:*:-2:
> nogroup:*:-1:
> wheel:*:0:root
> daemon:*:1:root
> kmem:*:2:root
> sys:*:3:root
> tty:*:4:root
> 
> most of those are the same on linux.

I'm not spotting any negative gids in /etc/group on devuan. And I think that
violates posix?

https://pubs.opengroup.org/onlinepubs/9699919799/functions/chown.html

chown(-1) means "don't change". So you can't set it to -1 through the posix
specified API.

>     >     On 6/20/22 21:32, enh via Toybox wrote:
>     >     > Specifically we were ending up with 2048 blocks allocated, and that --
>     >     > not the stat(1) behavior -- was the reason why this test was failing
>     >     > on macOS.
>     >
>     >     Yay, I got thunderbird sending email through gmail again! (The version
>     upgrade
>     >     changed the authentication type in the smtp server settings, but I
>     could still
>     >     set it BACK...)
>     >
>     >     I wanted a partially sparse file there, which truncate was preserving the
>     >     contents of.
>     >
>     > i must be missing something ... why is it _partially_ sparse?
> 
>     Because truncate should be able to extend an existing file sparsely without
>     damaging the existing contents, but I see that the existing test wasn't doing
>     what I thought it was doing...
> 
>     So hang on, if we made a file completely sparse in 3 segments, how many blocks
>     does MacOS think it's allocated? Can I dig this out of the macos test log on
>     github?
> 
>     https://github.com/landley/toybox/runs/7008521598?check_suite_focus=true
>     <https://github.com/landley/toybox/runs/7008521598?check_suite_focus=true>
> 
>     scripts/test.sh
>     scripts/runtest.sh: line 223: syntax error near unexpected token `;'
>     scripts/runtest.sh: line 223: `      R) LEN=0; B=1; ;&'
> 
>     It _starts_ with a syntax error, and THEN proceeds to run the tests. I don't
>     understand what it's doing here?
> 
> (that's similar to what you see on Android with mksh at the moment --- there's
> some error but apparently it's harmless because the tests run anyway.)

Another test case to file under "syntax error abort granularity":

  $ cat one
  #!/bin/bash
  source two
  echo hello
  $ cat two
  syntax error)
  $ bash one
  two: line 1: syntax error near unexpected token `)'
  two: line 1: `syntax error)'
  hello

You lose txpect, mkchroot, and dochroot. None of which those particular tests
are using...

> 
>     Scroll, scroll, scroll through the test results. VERBOSE=allnopass is probably
>     your friend here...
> 
>     No it does not. And I don't have a mac to try this on. Hmmm...
> 
> happy to run commands if you send me them.

$ rm -f empty; for i in 1k 1m 1g; do truncate -s $i empty; stat -c %b empty;
done; ls -l empty
0
0
0
-rw-r--r-- 1 landley landley 1073741824 Jun 25 17:15 empty

I'm guessing it's not gonna say 0.

>     >     The problem is, you're not using bash or toysh for these tests, you're
>     using a
>     >     shell of unknown capabilities. Can the macos and android shells do that?
>     >
>     > macos _is_ bash, it's just 3.2.57 from 2007, for exactly the reason you'd
>     assume.
> 
>     They swapped to zsh three years ago:
> 
>     https://www.theverge.com/2019/6/4/18651872/apple-macos-catalina-zsh-bash-shell-replacement-features
> 
> 
> oh, yeah, good point. my reaction to that was similar to your reaction to dash.
> _i'll_ be using bash on macos until they remove it.

All my stuff says #!/bin/bash at the top, but I dunno how github is running what...

>     That said I THINK all this $((math)) stuff I've been proposing is posix? So it
>     should work...?
> 
> yeah, certainly your 
> 
> x=1+3; echo $((x))
> 
> example works the same on both zsh and old bash (printing 4) on this mac.

And on current toysh. :)

>     > android is mksh, which is available as a debian package; i often do "android"
>     > shell testing that way for convenience :-)
>     >
>     > if you send me a patch i can test it for you ... but feel free to just commit
>     > "the right thing" and we can work from there!
> 
>     I'm no longer convinced I know what the right thing is here. Possibly the
>     "delete and recreate" workaround is best, but I have no idea WHY it works...
> 
>     I'm tempted to borrow my wife's mac for a bit, but I have no idea how to set up
>     a development environment on a mac. The first google hit is
>     https://sourabhbajaj.com/mac-setup/Xcode/ which looks... more elaborate than I
>     want to do on a borrowed machine.
> 
> iirc it's a bit simpler than that (if you don't have some company policy that
> says you can only install binaries from their servers) --- you just run "make"
> and it pops up a window saying "you want to install all that shit?" and you say
> "it's a unix system; i know this", bish bash bosh, job done.

Except I can't easily _undo_ it afterwards and don't want to eat I dunno how
many gigs on my wife's machine with Apple's soldered-in ssd.

*shrug* I can ask...

> or, like i say, commit something that works on linux and i'll be your test monkey.

I committed the patch you sent me. It has a big enough comment I can clean it up
on a future pass through...

>     Rob

Rob