Develop with Attention to Details

As a software developer or a tester a critical asset is the ability to pay attention to small details. Regardless of what kind of project you're doing everything boils down to a collection of details that form your product. Most people love to brag about these details when showing off their work. Details like how their game can render the most realistic bloom effects of any game with no impact on framerate. But individual details aren't the whole story.

Attention to detail doesn't just mean being aware of every little nit-picky speck of your brilliant piece of artistry. Attention to detail means understanding how each detail relates to each other detail. Your program isn't a list of discrete bullet points. Your program is all of the interactions that the entire set of points taken as a whole.

You may remember me criticizing the design of testlink's XML RPC API a few blogs ago. For this example I'm going to use testlink again. This time I'm going after the user experience of creating a test plan.

The process of creating a test plan within testlink is meant to be made up of a specific sequence of steps:

• Create any new test cases that are needed
• Create the platforms the tests are to run on
• Create the test plan
• Assign the platforms to be tested to the test plan
• Assign the test cases to be run to the test plan
• You're done!

So, pretty straightforward. Here's what happens behind the curtains (minus a lot of irrelevant details, mainly a ton of housekeeping work with the table nodes_hierarchy):

• Fill the database tables testcases, testsuites to create tree of test suites and cases
• Fill the database table platforms
• Fill the database table testplans
• Fill a join table associating platforms with testplans
• Fill a join table associating pairs of platforms and testcases with testplans

Still straightforward right? Except the UI of testlink doesn't require users to associate any platforms with a test plan before assigning test cases. This is reasonable since a lot of projects aren't multiplatform. Even if a project is multiplatform there's no obvious reason creating test cases first can't be done if mostly the same tests will run regardless of platform. Or at least based on the information presented by the UI.

This is what happens if test cases are associated first:

• Fill the database tables testcases and testsuites
• Fill the database table platforms
• Fill the database table testplans
• Fill a join table associating testcases with testplans
• Fill a join table associating platforms with testplans

The problem here lies in the fact that the 2nd to last step does not associate test cases in the test plan with platforms since that information isn't available. In most cases the final step doesn't correct the error causing testlink to enter a messed up state that a regular user can't easily correct.

Now, in testlink's defence, there is an attempt to detect the problem condition but it requires the user to notice an easily ignored "critic" message and follow instructions provided on it. The user has to first add a single platform to the test plan, save, then add the others. Assuming the user follows the instructions they'll have their existing test cases paired up with one platform. All of the other platforms added to the test plan need to have their test cases flagged manually. For most real-world applications this is highly tedious.

My first change is to automatically associate test cases with any platforms added to the test plan. My assumption is that if you're targeting multiple platforms you want the majority of your features to work the same on most platforms with a few exceptions. Adding those exceptions would be a lot less tedious than adding everything.

Now there's a problem with my approach which isn't a problem in the original. What happens if someone makes a test plan with no platforms, runs it, then decides to port their software to multiple platforms, showing this by adding all the supported platforms to their test plan in one go?

Feel free to comment with what you think the logic error in my fix (there is one) is. Hint: the instructions provided when testlink detects the problem (add one platform which associates with existing test cases, then all others) won't run into the logic error in my approach.

A Look at Web Service API Design

Lately I've been spending some time working with the open source test tracking tool testlink For tracking manual testing efforts and linking tests to software requirements and other SDLC essentials it is a very nice tool. However its API leaves something to be desired. So today's topic is designing a usable API.

The purpose of the API is to allow automated testing to be tracked and reported in testlink much like manual tests are. In addition it could be used to provide a hook into an automated build system so testers are aware of new work asap. Testlink tracks test sets in two ways. First is test suites. These represent your entire pool of available tests. Second is test plans. Test plans represent some set of tests you need executed by your testers or your automation system and are taken from the test suites. Test plans are executed against builds of your software on one or more platforms (eg windows and Mac)

So with that background we can come up with some ideas for what we need in an API and compare with what testlink actually provides. Let's assume that you already have a working automated build and test system but need better reporting because there is too much data to work with without a tool to help. Since this system is already running and more or less fully automated it's pretty unlikely that anyone would be willing to manually add anything major to the database such as test case information and such.

Given that... Here's what I would expect a reasonable API to provide.

• Create a test case within a suite which may be part of another suite. Also create the suites if they don't exist.
• Create a test plan containing a list of pre-created test cases. Update as appropriate.
• Assign results to tests in a test plan run against a build for a platform.
• Create a build and associate it with a test plan (or several)

Not too much eh? The nice thing is that this hides a great deal of complexity which is handled manually in normal testing. Hiding complexity is always a major goal for a good API. Another advantage is that this concept keeps requests larger and less frequent. When dealing with distributed systems you pay a fixed time cost for every request made. That cost goes into negotiating connections, security, protocol headers and so on. So, fewer requests to a web API is usually faster.

Sadly testlink makes a very easy mistake in its API design. The developer chose to expose every simple step used in creating test plans and recording results except for a few cases where the API is more difficult to use than necessary. The list of actions look something like this

• Create a single test suite optionally belonging to another (bad stuff happens if the suite already exists)
• Create a test case belonging to a previously created test suite (ditto)
• Create a test plan
• Create a build
• Create a platform
• Assign a platform to a test plan
• Assign a test case to a test plan
• Record a result for a test case in a test plan run against a build for a platform
• Yadda Yadda blah blah.

So. A user of this API needs to jump through the same hoops as they would when doing things manually. While this saves mental effort in designing the API you end up paying for it every time you have to write code to check that a suite (and the suites it's contained in) exists and create everything that's missing via a dozen or so function calls. Not fun.

Think of an API as a time investment. The more menial bullshit your API does for you the less you have to do when using it.

One last example. A test case in testlink has a number of properties such as its author, a description and the steps that are taken to carry out the test. When you use the testlink API you provide a key (a bunch of gibbrish) to get access. The key is unique to you. Now when calling an API to create a test case using your key the API ought to be smart enough to assume you are the author unless told otherwise. The version of testlink I'm using fails to make this leap. This means you need to modify the API (I did this, small win for Open Source) or have both a user name and their key to use the API for just this one function. Again if an API can reasonably do something for a user it should.