[Toybox] tar test failures

[enh]

On Fri, Oct 2, 2020 at 9:37 PM Rob Landley <rob at landley.net> wrote:

> On 10/2/20 4:13 PM, enh via Toybox wrote:
> > bonus question: why didn't the toybox CI catch this? looks like the tar
> tests
> > aren't being run there for some reason, despite the `make
> > defconfig`? https://github.com/landley/toybox/runs/1196859091
> >
> > ah, it's the "missing execute permission on the .test file" issue again.
> isn't
> > that a bug for everything not in pending?
>
> Yes and no. I was using +x to indicate the test itself is essentially in
> pending, and a lot of TESTS need work even when the command itself has been
> promoted. For example, tar.tests has a big "if false" block at the end
> with a
> bunch of TODO comments on tests I should add, but haven't yet.
>
> That said, at this point I need to do a full pass over all the tests
> again, it
> looks like I've commited other people's patches that changed permissions
> for
> tests that aren't remotely complete. (Maybe someday somebody will want to
> sponsor me putting time into the test suite, until then it's a todo item
> for
> somewhere post-1.0. Right now I'm trying to get a release out before I fly
> to
> Japan on the 14th, at which point I probably ignore this project entirely
> for a
> few months because I expect to be busy with paying work.)
>
> Anyway, the tests that aren't +x even though their commands aren't in
> pending are:
>
> for i in $(ls -l tests/*.test | grep  '[-] ' | sed 's@
> .*tests/@@;s/[.]test//');
> do echo toys/*/$i.c; done | grep -v pending
>
> And I guess I can just chmod +x all of the ones that pass their tests and
> go
> "test_kill has two tests, and that's it for the foreseeable future I
> guess, oh
> well, I no longer have an obvious way to track what needs work"...
>

you can always add a TODO in the .test files for the ones you're not happy
with? the current setup seems like the cure is worse than the disease:
"these tests aren't perfect ... so let's not run any of them" :-)

this gets caught eventually because i run all the tests on Android (and
have a "if actual tool not a symlink to toybox, report but ignore test
failures" hack). but it does mean that it's easier to miss things when
developing on the host (which is how i usually work for anything that's not
Android-specific), and it means that the github continuous build isn't
running these tests and letting us know after the fact that we've missed
something.

i'm happy to send you a "+x and add TODO" patch, but i'm assuming you have
something more specific to add about _what_ you think is missing from the
tests currently? (otherwise it would just be the default assumption of "of
course we could do more testing", even for something like chromium:
https://analysis.chromium.org/p/chromium/coverage !)


> Rob
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20201005/9f3cc016/attachment-0001.htm>