There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.
-- C.A.R. Hoare

Introduction

Ever after the radioactive TDD bug bit me and I'm now infected with that mode of development I've become more and more interested in good unittest frameworks to help me develop bugfree code fast. If you have not yet experienced Test Driven Development, I'd recommend reading up a little bit on it before continuing with this article, see the resources at the end of the page. One thing that still is lacking is a good solid unittest framework for shader development. Shaders have today come a long way from my first introduction to them during shader model 1.0 where we wrote asm code directly. Today the shaders are much longer and with the introduction of Cg and HLSL we can write complex shaders that suddenly have a lot of bugs in them. Well, one of the solutions to bugs are ... unit tests. So enter UnitTestCg, which is a very simple first iteration of a unittest framework for shader code.

How it works

The core of the testing framwork is the support runtime, the one thing that it does is to render a fullscreen fragment and depending upon the color of the resulting framebuffer we can make a conclusion of the result of the test. Writing tests are relativly straight forward. You need to in a file implement a function called runTest that should return a boolean weather or not it succeeded with the test. In this file you can pretty much do anything you want, within reason and rules for the pixle/vertex shader you're writing. UnitTestCg will then take over and for the vertex shader generate bogus vertices to in the end reject the polygon and in the pixel shader it will simply generate a red pixel. The distribution comes with some tests allready, most of them runs in either mode, vertex or pixel shaders. The nant script included in the dist also contains the tests I use to verify the build. 

 
bool runTest()
{
  return g_testMatrix[3].x == 12 &&
         g_testMatrix[3].y == 13 &&
         g_testMatrix[3].z == 14;
}
Listing 1: A sample test written in Cg. This one uses the predefined matrix.

The output from the program looks something like this: 

 
C:\Tests>UnitTestCg.exe -v 0 -n -p vs -I . TestFailedMatrix.fx
TestFailedMatrix.fx: error: Test failed. Color was [0, 0, 0, 1]

C:\Tests>UnitTestCg.exe -v 1 -n -p vs -I . TestMatrixLayout.fx
TestMatrixLayout.fx: ok.

C:\Tests>UnitTestCg.exe -b dx -v 1 -n -p ps -I . TestMatrixLayout.fx
TestMatrixLayout.fx: ok.
Listing 2: Sample output from UnitTestCg. This is code compiled for Cg and using the test on the vertex shader.

Cross platform

So when I say cross platform here I don't mean across operating systems, rather I mean across languages. Cg and HLSL started out from the same roots but today they are a little bit different. Some extensions to HLSL are not supported in Cg and the compilers are definitly different. UnitTestCg supports both Cg 1.5b2 and DX9 HLSL. It's a pretty handy tool to quickly resolve weather or not something works in both dialects and inspect the output assembly.

Download

I've put up the full source to the project at google code if anyone is interested. The code was developed in a rush as a proof of concept, but it became rather usable in the end although the code might not have been. It also started out small enough that I didn't feel I needed unittests, so it's not that I hold back the tests as much as I don't have them. Shame on me, advocating TDD and not using it. But there you go. I think it's important to realize that I don't use TDD for experiments or short snippets, more for larger long production things that need to live for a long time and are part of something bigger. I hope this little experiment doesn't explode into something too big, if it does I'd probably rewrite it with tests.

You can access the source code here: UnitTestCg

Optionally there is a gratification right now option to download precompiled binaries here: UnitTestCg.zip.

In closing

Well. That was one of my projects that have been near the trashcan on my harddrive, I guess it was good it was finally put up on a source repository. Gave me an excuse to write this rambling about it as well. I think the tests for shaders might have more impact finding where they broke rather than helping develop shaders in the end, as on the CPU side where the DD part of TDD is important. On the shader side I think it can be a boon if you have a comprehensive suite of tests that can pinpoint where you went wrong when you muck around in the code to shoehorn a new feature in there. Or to help you prove or disprove a theory you have about the code, all the good things that come from unit tests. Ah, and of course refactoring becomes a breeze now. Until next time...

Resources

Comments