[Toybox] [PATCH SET] Fix test harness issues and fix tests

[Andy Chu]

This is a pretty big cleanup, still in progress, but I think these
parts are ready to merge.

I think we should change the test "protocol" to be a subprocess rather
sourcing the .test file, as you alluded to.  But that can be done in a
separate step on top of this.

There's also some cruft like '. testing.sh' at the top of most tests,
but it doesn't seem to exist.  As long as I'm going in the right
direction I can fix all this.


Summary after patch 1 but before patch 2:

Tested 66 toybox commands: 90 test failures

Commands with test failures:  bzcat chattr date find groupadd groupdel
hostname ls modinfo mount mv pgrep pkill renice tar touch xzcat zcat

(skipped commands not listed)

After patch 2:

Tested 61 toybox commands: 36 test failures

Commands with test failures:  date find ls modinfo mv pgrep pkill
renice tar zcat

Commands skipped:  chattr chgrp chown groupadd groupdel hostname
ifconfig losetup mount useradd

make: *** [tests] Error 1

-----

 [PATCH 1/2] Refactor the test harness to triage tests, and fix some
 bugs.

genconfig.sh:
- Fix a name clash bug where 'make test' would run all tests and then try to
  build the 'test' binary.  'make test_bin' is special cased for now.
- Make the test targets .PHONY, since we're not creating files called test_sed,
  etc.
- Use here docs to write Makefile fragments, for readability and to be
  consistent with the rest of the script.  Refactor into functions.
- Add comments.

make.sh:
- Add the output binary name as the first param.  This fixes an issue where
  'make; make test_sed' would leave you without a 'toybox' binary because
  single.sh created another toybox binary and renamed it to sed.

runtest.sh:
- Changed the 'testing' function to return 0.  This fixes a bug where the
  return value of the script depends on the return code of the last command --
  NOT whether it passed or failed.  For example, 'make test_test' would fail
  with code 1, even though all tests passed (because 'test' exits 1 validly).
  On the other hand, tests with failures would still return 0 if the last
  command didn't fail.
- Use local variables instead of globals since we are sharing globals with the
  tests themselves.
- Add a common function to skip tests if not running as root, and keep track of
  skipped commands.

test.sh:
- Refactor it into separate functions/actions and add usage.
- Fix a bug where tests would succeed under 'make test_$COMMAND' but fail under
  'make test'.  The problem is if you are running the xzcat command, 'tar'
  could be either the host binary or the toybox binary, depending on what tools
  are in the modified test $PATH.  To fix this, the test harness finds the host
  binary and sets $HOST_BIN_TAR, etc. before invoking the test.
- Keep track of the number of failures per *command*, and exit 1 if any failed
- Use a distinct test directory for each command, instead of repeatedly setting
  up and tearing down the same one.
- When running all tests, print out a summary at the end.

single.sh:
- Call scripts/make.sh directly instead of 'make'.
- Add comments.
---
 Makefile             |   2 +-
 scripts/genconfig.sh | 103 ++++++++++++++++++++++++++---------
 scripts/make.sh      |  16 ++++--
 scripts/runtest.sh   |  40 +++++++++-----
 scripts/single.sh    |  28 +++++++---
 scripts/test.sh      | 149 +++++++++++++++++++++++++++++++++++++++++----------
 6 files changed, 258 insertions(+), 80 deletions(-)



[PATCH 2/2] Various test fixes and cleanup.

bzcat: Fix a test that was succeeding for the wrong reason.  The test
was looking for a 1 exit code in the case of bad data, but there was a
typo in the input path, so it failed 1 for the wrong reason.

bzcat, xzcat, zcat: Use $HOST_BIN_TAR to fix the issue where they fail
under 'make test' but not 'make test_bzcat'.

hostname: Use $HOST_BIN_HOSTNAME, and skip one test if not root.

chgrp, groupadd, groupdel, ifconfig, losetup, mount, useradd: Skip if
not root.

pgrep, pkill: style cleanup and speedup of the tests (change sleep 1 to
sleep 0.1).  No fixes yet.

renice: Fix some quoting problems (revealing further errors, not fixed)

tar: Style cleanup in preparation to fix failures under 'make test' but
not 'make test_tar' (due to host/toybox discrepancy)

touch: Use $HOST_BIN_DATE because toybox doesn't support %N nanoseconds
---
 tests/bzcat.test    |  41 ++++++++++---------
 tests/chattr.test   |   4 +-
 tests/chgrp.test    |   7 +---
 tests/chown.test    |   7 +---
 tests/groupadd.test |   2 +
 tests/groupdel.test |   2 +
 tests/hostname.test |  16 +++++---
 tests/ifconfig.test |   7 +---
 tests/losetup.test  |   7 +---
 tests/mount.test    |   6 ++-
 tests/pgrep.test    |  51 ++++++++++++------------
 tests/pkill.test    |  54 ++++++++++++-------------
 tests/renice.test   |   8 ++--
 tests/tar.test      | 111 ++++++++++++++++++++++++++++++++--------------------
 tests/touch.test    |   6 +--
 tests/useradd.test  |   7 +---
 tests/xzcat.test    |  36 +++++++++--------
 tests/zcat.test     |  39 +++++++++---------
 18 files changed, 216 insertions(+), 195 deletions(-)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Various-test-fixes-and-cleanup.patch
Type: text/x-patch
Size: 29602 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20160318/fb0f672d/attachment-0008.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Refactor-the-test-harness-to-triage-tests-and-fix-so.patch
Type: text/x-patch
Size: 15476 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20160318/fb0f672d/attachment-0009.bin>