[Toybox] Test suite quandry.

[enh]

On Wed, May 13, 2020 at 4:23 AM Rob Landley <rob at landley.net> wrote:
>
> I'm trying to make a scripts/root/tests that'll package up the toybox test suite
> and run it in mkroot (no toysh isn't nearly ready for that yet, but I'm working
> on it), and it's _mostly just:
>
>   mkdir blah
>   cp -a config scripts/ tests/ blah
>   cd blah
>   TEST_HOST=1 VERBOSE=fail scripts/test.sh
>
> Except... TEST_HOST fails on a lot of the tests. Yeah mkroot has toybox as its
> "host" commands, but still. The tests are SUPPOSED to run on the host too. So I
> went through and started fixing a few, and I hit this:
>
> --- a/tests/cpio.test
> +++ b/tests/cpio.test
> @@ -33,7 +33,7 @@ rm a bb ccc dddd
>
>  # archive dangling symlinks and empty files even if we cannot open them
>  touch a; chmod a-rwx a; ln -s a/cant b
> -testing "archives unreadable empty files" "cpio -o -H newc|cpio -it" "a\nb\n"
> "" "a\nb\n"
> +toyonly testing "archives unreadable empty files" "cpio -o -H newc|cpio -it"
> "a\nb\n" "" "a\nb\n"
>  chmod u+rw a; rm -f a b
>
> And the PROBLEM is it's _important_ that the dangling symlinks be archived
> because if they're an absolute path or something they may not be dangling when
> the initramfs extracts them. tl;dr: yet another gnu/dammit command is buggy,
> film at 11.
>
> But the problem I'm facing NOW is sticking "toyonly" in front of it isn't the
> right thing. This isn't functionality only toybox has, this is functionality the
> other guy SHOULD have, but the other guy (a euphemism for "The Hulk") is
> expected to fail here because screamy and nonverbal. (Stallman hasn't been gone
> long enough for Taika Waititi to fix the FSF.)

add a "hostbug" that's functionally the same as "toyonly" but makes
the intent clear?

(seems better than a generic "xfail" mechanism.)

> Rob
>
> P.S. Simiarly, factor -32 is not "unknown option 3" for toybox, but TEST_HOST...
> DOES NOBODY TEST GNU/DAMMIT SOFTWARE? Honestly, this is vicarously embarassing.
>
> P.P.S. GNU fmt converted a space to a tab? Was it supposed to? The initial
> indentation wasn't a tab...
> FAIL: fmt matched tab/space
> echo -ne '        hello\n\tworld' | fmt
> --- expected    2020-05-13 03:33:24.880134324 -0500
> +++ actual      2020-05-13 03:33:24.880134324 -0500
> @@ -1 +1 @@
> -        hello world
> +       hello world
>
> If I comment that test out the NEXT one passes, which means it's leaning "tab"
> rather than "preserve the starting indentation" like toybox's is? What's the
> logic here? There doesn't seem to be a posix spec for this, the man page says
> nothing, and I refuse to read info pages on principle...
>
> P.P.P.S. Oh hey:
>
>   $ touch file1 file2
>   $ chmod 400 file1 file2
>   $ mv file2 file1
>   mv: replace 'file1', overriding mode 0400 (r--------)? ^C
>   $ echo n | mv file2 file1
>   $ ls file*
>   file1
>
> That's, um, wrong. That's the regression test suite finding an actual bug in
> coreutils or whatever's providing that gnu/dammit version of mv. Seriously, I
> SHOULD NOT BE FINDING THIS. THIS SHOULD NOT BE MY PROBLEM. (The behavior is when
> stdin isn't a tty, it doesn't warn and prompt, it just fries the target. Even
> though mv has a -f, I guess the guy guys don't care? The behavior is "prompt to
> stderr, read from stdin" whether or not it's a tty? Did I miss something?)
>
> P.P.P.P.S. coreutils paste hasn't got utf8 support?
>
> P.P.P.P.P.S. readlink -q doesn't work in debian:
>
>   $ ln -snf link link
>   $ readlink -q link
>   link
>
> Seriously, -q is DOCUMENTED AS QUIET IN YOUR MAN PAGE. I am not trying to have
> toybox have a HIGHER standard than the conventional Linux command line, I'm just
> trying for my version of BASIC COMPETENCE here. Do you not test ANYTHING? I mean
> yeah when Redhat's "which filenotfound" started crapping output in the not found
> case when that's NOT HOW THAT COMMAND EVER WORKED I thought that was weird, but
> that was post-enterprise Red Hat I.E. Pointy Hair Linux trying to shove systemd
> down everyone's throat so they could be the only people in the world who
> understand how it works and thus justify $5k/seat support contracts. It's
> amazing any _part_ of that still works. I expected better from "not that"...
>
> (Good grief, this isn't the THOROUGH test suite I want to do with a close
> reading of the spec and thinking through every corner case. This is just what
> I've slapped together so far by recording tests I ran to answer questions while
> implementing this stuff.)
>
> I have reached "step away from the keyboard" time, it would seem.
>
> Still Rob
> _______________________________________________
> Toybox mailing list
> Toybox at lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net