Comments on Test Driven Development

It's always nice to spend a bit of time on nerd rage. I was considering a follow up to my meta-rant instead of this but I'll refrain from doing so. If anyone wishes to have a follow up on something specific feel free to leave a comment. But before doing so, yes I'm aware of the author's weak follow up article. Stating (paraphrasing) "the article was a joke, rebutting it shows you have no humor" is, in my opinion, a cop out. Either back up your statements or admit you made mistakes. On to the actual blog.

Test Driven Development is Cool

For those who are unaware test driven development (TDD for short) is a method for making software where you start by developing automated tests for your program then write your program to pass those tests. It's a process which I'm a big fan of. Since I've been using TDD for a while now I figured I could share a few comments from my experiences using TDD.

Test Driven Development Makes you Think

More specifically, TDD forces you to think about how the code you're writing will be used. This is especially useful when designing an API. A typical process begins with you designing an interface for whatever you're doing, then writing tests that use that interface. Remember my earlier article on web services? For a short summary; I discussed how you should try automate common work within an API to minimize the code required to do useful stuff with it. I used Testlink's XML RPC API as an example of a suboptimal API. It's not awful but using it results in a lot of code bloat and, as a result, poor performance.

In order to develop a good unit test you need to first think of how your API is going to be used. Each potential use for your API becomes a single test. It's very easy to come up with a few dozen tests for a relatively simple API. Doing this you'll see what parts of the API get re-used very quickly, i.e. boilerplate. And if you're developing a lot of tests I think you'll begin to see why I refer to such code as menial bullshit. And when your API makes you angry you'll begin to think of ways of making it easier to use.

Testing is Faster with TDD

Let's imagine you're making a game. Games have a huge variety of data that needs to be stored. Sure, the absolute quantity of data isn't all that big (relatively speaking, financial and analytics software have games beat by lightyears) but there is a lot of very different data. Some things such as sound samples, music, 3D models, 2D images and such have established formats others such as game levels, entities, AI code, saved games, recorded gameplay footage (for games that still do that, I miss old FPS games...) tend to vary from game to game. And all of that data needs code to handle loading it into the game and saving it from creation tools.

Using a conventional dev/test cycle would mean creating the code, integrating it into the game and tools and finally creating content using the tools before being able to do any effective testing of the code. And testing the resulting data means opening up the game and tools and manually using them to see that the data is correct. In-game testing in particular tends to require going through a lot of extra effort (i.e. starting the game, getting to a testing level, and setting up your test.) That's slow.

Using Test Driven methods means deciding what sort of data your file format will initially contain producing some mock sample then writing unit tests that use your game's file loader to load the file then verify the data they produce matches with what you expect. Each unit test works with a small amount of data to verify a specific part of your file loader works correctly. You can also easily create additional tests to check that different combinations of features in your file loader work correctly. Obviously there's the risk of combinatorial explosion but that's easy enough to avoid by limiting unit tests to covering features in isolation and obvious feature combinations. This will give you the most bang for your buck. Other tests can be added if bugs are discovered.

The speed advantage comes from eliminated testing overhead. When you need to test your code you simply run your unit tests. No need to run your asset creation tools, or the game or anything else. The only limit to the speed of your testing is the speed of your code. The best part, if you're skeptical about TDD, is you don't need to fully buy into TDD to do this. Just supplement your manual testing effort with unit tests. Even better, if you automate your product's build process then you can have your tests run automatically too by making your build tool run the tests right after a successful build finishes. Your entire dev team can then be notified about failing tests just like they would be for build breaks.

Test Driven Development has Problems

So far I've painted an extremely rosy picture of TDD. However, like all processes, it's not perfect. Far from it in fact. There are a good number of caveats about using TDD.

You Must Make Testable Code

Remember my comment on testing being made faster by TDD? This is only true if your code can easily be unit tested.

This is both good and bad. On one hand testable code tends to be very easy to work with since it doesn't have a lot of baggage that prevents it from being picked up and used. On the other hand, that baggage may be necessary for your code to perform optimally. Nine times out of ten however you're best off developing testable code. And if you think you think your case might just be that one you're probably wrong. Check first. This applies to all code regardless of whether you're making web applications or embedded software on a high performance real time OS

One tip: modularize. This means not only conventional modularization by keeping objects and procedures from being overly inter-dependent, but also moving parts of your project into libraries to make it easier to build tests without pulling in huge quantities of code. This pushes compile times downward for when you're testing and allows faster testing.

You Still need Conventional Manual Tests

Unit tests and other automated tests that are commonly produced by test driven development are great for verifying software interfaces and behavior. However, unless you're producing automation tools, you're not going to find anywhere near the full set of issues.

For example, you can't test that your software's workflow fits your users' expectations. Automated tests can't verify that a game is fun. Artistic merit is impossible to judge with software alone. The most bizarre interoperability issues typically only crop up during general use of your software and require manual testing. For a completely contrived example on a smartphone, when playing music on a media player, visiting a flash site on one browser tab and visiting an HTML5 site on another and using the GPS causes the phone to crash at random. But only for certain commuters in the big city during rush hour.

Unless your test developers are completely evil and have a lot of time and money you're not going to get these kinds of things happening in your automated test. Particularly if you follow conventional best practices for unit tests which tests stuff in isolation. The only way you're going to find these issues is if you have manual testing reflective of how your real users would use your product.

Test driven development works as an excellent complement to ad hoc manual testing. Without TDD you're unlikely to have the tester time available to do the kinds of heavy exploratory testing that would find issues like our phone crash during a morning commute and without skilled testers doing that sort of testing you're unlikely to get the critical information need to fix the issue.

A Look at Web Service API Design

Lately I've been spending some time working with the open source test tracking tool testlink For tracking manual testing efforts and linking tests to software requirements and other SDLC essentials it is a very nice tool. However its API leaves something to be desired. So today's topic is designing a usable API.

The purpose of the API is to allow automated testing to be tracked and reported in testlink much like manual tests are. In addition it could be used to provide a hook into an automated build system so testers are aware of new work asap. Testlink tracks test sets in two ways. First is test suites. These represent your entire pool of available tests. Second is test plans. Test plans represent some set of tests you need executed by your testers or your automation system and are taken from the test suites. Test plans are executed against builds of your software on one or more platforms (eg windows and Mac)

So with that background we can come up with some ideas for what we need in an API and compare with what testlink actually provides. Let's assume that you already have a working automated build and test system but need better reporting because there is too much data to work with without a tool to help. Since this system is already running and more or less fully automated it's pretty unlikely that anyone would be willing to manually add anything major to the database such as test case information and such.

Given that... Here's what I would expect a reasonable API to provide.

• Create a test case within a suite which may be part of another suite. Also create the suites if they don't exist.
• Create a test plan containing a list of pre-created test cases. Update as appropriate.
• Assign results to tests in a test plan run against a build for a platform.
• Create a build and associate it with a test plan (or several)

Not too much eh? The nice thing is that this hides a great deal of complexity which is handled manually in normal testing. Hiding complexity is always a major goal for a good API. Another advantage is that this concept keeps requests larger and less frequent. When dealing with distributed systems you pay a fixed time cost for every request made. That cost goes into negotiating connections, security, protocol headers and so on. So, fewer requests to a web API is usually faster.

Sadly testlink makes a very easy mistake in its API design. The developer chose to expose every simple step used in creating test plans and recording results except for a few cases where the API is more difficult to use than necessary. The list of actions look something like this

• Create a single test suite optionally belonging to another (bad stuff happens if the suite already exists)
• Create a test case belonging to a previously created test suite (ditto)
• Create a test plan
• Create a build
• Create a platform
• Assign a platform to a test plan
• Assign a test case to a test plan
• Record a result for a test case in a test plan run against a build for a platform
• Yadda Yadda blah blah.

So. A user of this API needs to jump through the same hoops as they would when doing things manually. While this saves mental effort in designing the API you end up paying for it every time you have to write code to check that a suite (and the suites it's contained in) exists and create everything that's missing via a dozen or so function calls. Not fun.

Think of an API as a time investment. The more menial bullshit your API does for you the less you have to do when using it.

One last example. A test case in testlink has a number of properties such as its author, a description and the steps that are taken to carry out the test. When you use the testlink API you provide a key (a bunch of gibbrish) to get access. The key is unique to you. Now when calling an API to create a test case using your key the API ought to be smart enough to assume you are the author unless told otherwise. The version of testlink I'm using fails to make this leap. This means you need to modify the API (I did this, small win for Open Source) or have both a user name and their key to use the API for just this one function. Again if an API can reasonably do something for a user it should.

The Importance of Profiling

Ah, profiling. A tool that I've neglected to use for far too long until forced to do so recently at work. No I don't mean that! No not that either!! Profiling is a tool which is used to examine how long each part of a given program executes to find out why it's slower than molasses in Antarctica.

A major tool that I support is a test automation framework that supports remotely exercising APIs under test via a scripting language. I had just finished the initial work of porting this system to support testing of some new software when I discovered a problem.

Under normal circumstances the program can run over one hundred commands per second depending on the time taken by the command itself. An early test however revealed that I was barely managing one or two commands per second. Something was wrong!

Worse, examining the code provided no obvious answers. There were no poorly made algorithms, no pointless sleeps, no redundant work, nothing. Embarrassingly, this was the first time I've actually used a profiling tool to solve a problem. All other times I've simply muddled around with the code until I found some obvious problem like the ones I mentioned. Whelp, no time like the present, right?

The tool I was using provided two ways to profile a program's internals. Both involve what's called instrumentation. What that means is the tool generates some extra code in the program for the purpose of measuring your code's behavior. The first profiling method, and by far the simplest, is to count the number of times every function is called. This is helpful in finding candidates for improvement or potential bugs that cause more calls to some code than necessary. The second measures the amount of time spent in each function in addition to the number of times it's called. This is better since it can tell you where your program is getting stuck longest.

In my case a function which waited for a command stood out as the long runner. Great! Except all it does is call a function to receive data. Digging through the function I found some informational prints, and a call to recv. Not much to go on. Prints are used in a lot of different places in the program and none of those calls took very long at all, so that's out. Time to put on my google Shinobi Shozoku and find out if there are situations that make recv slow.

The result? Yes! However it's not recv that's the problem. It's a little something called Nagle's Algorithm, which is part of the TCP/IP protocol stack. It's designed to reduce network traffic by attempting to combining multiple small packets and sending them at once. It does this by holding on to a small packet and waiting for a period of time before sending it, in case more data is coming. Since the tool was designed to minimize bandwidth and be very lightweight in design it would send encoded commands in the form of very small packets, typically a few dozen bytes being large for the tool. Disabling Nagle's Algorithm did the trick.

Without profiling I might have found the issue in question, but I would have wasted significantly more time either in trial and error. Or worse, not realizing I've made an error, and wasting hours trying to fix something that isn't the problem with negligible improvement.