The Importance of Profiling

Ah, profiling. A tool that I've neglected to use for far too long until forced to do so recently at work. No I don't mean that! No not that either!! Profiling is a tool which is used to examine how long each part of a given program executes to find out why it's slower than molasses in Antarctica.

A major tool that I support is a test automation framework that supports remotely exercising APIs under test via a scripting language. I had just finished the initial work of porting this system to support testing of some new software when I discovered a problem.

Under normal circumstances the program can run over one hundred commands per second depending on the time taken by the command itself. An early test however revealed that I was barely managing one or two commands per second. Something was wrong!

Worse, examining the code provided no obvious answers. There were no poorly made algorithms, no pointless sleeps, no redundant work, nothing. Embarrassingly, this was the first time I've actually used a profiling tool to solve a problem. All other times I've simply muddled around with the code until I found some obvious problem like the ones I mentioned. Whelp, no time like the present, right?

The tool I was using provided two ways to profile a program's internals. Both involve what's called instrumentation. What that means is the tool generates some extra code in the program for the purpose of measuring your code's behavior. The first profiling method, and by far the simplest, is to count the number of times every function is called. This is helpful in finding candidates for improvement or potential bugs that cause more calls to some code than necessary. The second measures the amount of time spent in each function in addition to the number of times it's called. This is better since it can tell you where your program is getting stuck longest.

In my case a function which waited for a command stood out as the long runner. Great! Except all it does is call a function to receive data. Digging through the function I found some informational prints, and a call to recv. Not much to go on. Prints are used in a lot of different places in the program and none of those calls took very long at all, so that's out. Time to put on my google Shinobi Shozoku and find out if there are situations that make recv slow.

The result? Yes! However it's not recv that's the problem. It's a little something called Nagle's Algorithm, which is part of the TCP/IP protocol stack. It's designed to reduce network traffic by attempting to combining multiple small packets and sending them at once. It does this by holding on to a small packet and waiting for a period of time before sending it, in case more data is coming. Since the tool was designed to minimize bandwidth and be very lightweight in design it would send encoded commands in the form of very small packets, typically a few dozen bytes being large for the tool. Disabling Nagle's Algorithm did the trick.

Without profiling I might have found the issue in question, but I would have wasted significantly more time either in trial and error. Or worse, not realizing I've made an error, and wasting hours trying to fix something that isn't the problem with negligible improvement.

Three Projects... Four Projects... Five Projects!?!

Yes, well (conservatively) counting what I'm up to at RIM. The project I'm wrapping up is a particularly fun and rewarding one, creating a tool for providing detailed code coverage tracking for the various testing efforts within the OS Group. Sadly, the details aren't all that exciting. Mostly it's just processing gobs of similar data into an easily queried form. The majority of the work is in the logistics of collecting the data in a way that doesn't interfere with testing and manages to collect as much information is possible given the limitations of the testing environment. My other project(s) relate to some internal OS Group initiatives. I could tell you about them, but then I'd have to kill you ;)

So that's two...ish. Where's the other three? Well...

1. Space Combat Sim
   Yup, that thing is still going. Adding bots, improving the graphics, cleaning the code up and generally learning as much about 3D networked games as possible.
2. A shmup game developed from scratch for XNA on X-Box 360 and PC.
   Something simple for building a strong C# codebase that I can port to other projects. Goals here are making a programmable effects system and some decent GUI widgets.
3. Another shmup, using FlatRedBall
   Why? Because screw you! That's why... Actually I figured I'd just start messing around with plugging tools together just since that's normally not the way I make games.

Actually, to go on a boring long monologue...

I have finished only a few game projects, mainly for school. That tends to be because I approach the subject by first building tools and such, then building the game using them. I end up having so much fun making the tools that the project devolves into adding new features and refining code that I lose focus on making the game. Then I get bored and start a new project.

Now, I'm a tools developer professionally I can finally spend all hours of the working day making all sorts of exciting tools for a moderately hardcore development environment. This means that I'm not getting quite the same buzz from making yet more tools at home. So... time to make some fugging games!

Later d00dz! I'll try to keep the interval to next post to less than a month this time.

Some Inspiration

Lately I've been getting most of my coding jollies from my job as a tools developer at RIM. Mostly Python with a nice dose of C, SQL, and other stuff. Mainly fixing some reporting systems that go to external reports seen by management. Plus a dash of code janatorial work since nobody else seemed to be available/willing. After eight months doing solo work it's very nice to be working with a team again.

Anyway, that's not actually what I wanted to talk about. Today's just a shorty (I have a longer rant about myself coming later.) TIGSource recently posted a couple montages of a huge collection of free, indie games. I've played a few of them (Fraxy, Cave Story, Cho Ren Sha 68k, Prototype 2, Warning Forever, Clean Asia and one or two I can't recall... yea I'm a SHMUP fan) Holy hell there's some awesomeness.

Check it out!

So, What Now?

Well, today I spent a TON of time writing up blogs that I've been meaning to finish for a long time now. Three of them in fact! The first on exporting from Blender to XNA, the second on how the demos for Space Combat Sim went, and the third is a postmortem. But, I haven't talked about what I'm doing now, have I?

The short answer is that I have a job. In fact, I started my job before my "good demo" which was on May 4th. I'm working as a testing tools developer for Research in Motion. Unless you've been living under a rock for the past few years you've probably heard of their flagship product. Before I started this blog I worked as an intern for RIM over a period of 16 months. I work for the Operating Systems group which is responsible for all of the hardware drivers etc. that make the BlackBerry phone work. The culture of the OS Group, relaxed, friendly and fairly close knit, is what motivated me to return to RIM after finishing school. I'm quite happy working somewhere that I can spend the day working on hard technical problems with minimal political friction then finish the day by hanging around the pub.

My job now is to maintain and improve tools for automatically testing the Blackberry OS. Obviously I can't go into specific but I will be able to come out with articles on scripting, automation, testing and testing tools in a general sense. On top of the occasional post about work I'll still continue my game development work which means, of coure, more posts on Space Combat Sim and other projects as well.

Cheers.