[Toybox] Toybox test image / fuzzing

[Andy Chu]

On Sun, Mar 13, 2016 at 11:55 AM, Rob Landley <rob at landley.net> wrote:
> On 03/13/2016 01:06 PM, enh wrote:
>> #include <cwhyyoushouldbedoingunittestinginstead>
>>
>> only having integration tests is why it's so hard to test toybox ps
>> and why it's going to be hard to fuzz the code: we're missing the
>> boundaries that let us test individual pieces. it's one of the major
>> problems with the toybox design/coding style. sure, it's something all
>> the existing competition in this space gets wrong too, but it's the
>> most obvious argument for the creation of the _next_ generation
>> tool...
>
> I started adding test_blah commands to the toys/example directory. I
> plan to move the central ps plumbing to lib/proc.c and untangle the 5
> commands in there into separate files, we can add test_proc commands if
> you can think of good individual pieces to test.
>
> I'm open to this category of test, and have the start of a mechanism.
> I'm just spread a bit thin, and it's possible I don't understand the
> kind of test harness you want?

The toys/example/test_*.c files seem to print to stdout, so I guess
they still need a shell wrapper to test correctness.  That's
technically still an integration test rather than a unit test --
roughly I would say integration tests involve one more than one
process (e.g. for a system of servers) whereas unit tests are run
entirely within the language using a unit test framework in that
language.

Google uses gunit/googletest for testing, and I guess Android does too:

https://github.com/google/googletest

Example: https://android.googlesource.com/platform/system/core.git/+/master/libziparchive/zip_archive_test.cc

You basically write a bunch of functions wrapped in TEST_ macros and
they are linked into a binary with a harness and run.

I guess toybox technically could use it if the tests were in C++ but
the code is in C, though it seems like it clashes with the style of
the project pretty badly.

I think the main issue that Elliot is pointing to is that there are no
internal interfaces to test against or mock out so you don't hose your
system while running tests (i.e. you can "reify" the file system state
and kernel state, and the substitute them with fake values in tests).

I agree it would be nicer if there were such interfaces, but it's
fairly big surgery, and somewhat annoying to do in C.  I think you
would have to get rid of most global vars, and use a strategy like Lua
or sqlite, where they pass around a context struct everywhere, which
can have system functions like open()/read()/write()/malloc()/etc.
sqlite has a virtual file system (VFS) abstraction for extensive tests
and Lua lets you plug in malloc/free at least.  They are libraries and
not programs so I guess that is more natural.

I think this is worth keeping in mind perhaps, but it seems like there
is a lot of other low hanging fruit to address beforehand.

Andy

 1457898776.0