The Importance of Profiling

Ah, profiling. A tool that I've neglected to use for far too long until forced to do so recently at work. No I don't mean that! No not that either!! Profiling is a tool which is used to examine how long each part of a given program executes to find out why it's slower than molasses in Antarctica.

A major tool that I support is a test automation framework that supports remotely exercising APIs under test via a scripting language. I had just finished the initial work of porting this system to support testing of some new software when I discovered a problem.

Under normal circumstances the program can run over one hundred commands per second depending on the time taken by the command itself. An early test however revealed that I was barely managing one or two commands per second. Something was wrong!

Worse, examining the code provided no obvious answers. There were no poorly made algorithms, no pointless sleeps, no redundant work, nothing. Embarrassingly, this was the first time I've actually used a profiling tool to solve a problem. All other times I've simply muddled around with the code until I found some obvious problem like the ones I mentioned. Whelp, no time like the present, right?

The tool I was using provided two ways to profile a program's internals. Both involve what's called instrumentation. What that means is the tool generates some extra code in the program for the purpose of measuring your code's behavior. The first profiling method, and by far the simplest, is to count the number of times every function is called. This is helpful in finding candidates for improvement or potential bugs that cause more calls to some code than necessary. The second measures the amount of time spent in each function in addition to the number of times it's called. This is better since it can tell you where your program is getting stuck longest.

In my case a function which waited for a command stood out as the long runner. Great! Except all it does is call a function to receive data. Digging through the function I found some informational prints, and a call to recv. Not much to go on. Prints are used in a lot of different places in the program and none of those calls took very long at all, so that's out. Time to put on my google Shinobi Shozoku and find out if there are situations that make recv slow.

The result? Yes! However it's not recv that's the problem. It's a little something called Nagle's Algorithm, which is part of the TCP/IP protocol stack. It's designed to reduce network traffic by attempting to combining multiple small packets and sending them at once. It does this by holding on to a small packet and waiting for a period of time before sending it, in case more data is coming. Since the tool was designed to minimize bandwidth and be very lightweight in design it would send encoded commands in the form of very small packets, typically a few dozen bytes being large for the tool. Disabling Nagle's Algorithm did the trick.

Without profiling I might have found the issue in question, but I would have wasted significantly more time either in trial and error. Or worse, not realizing I've made an error, and wasting hours trying to fix something that isn't the problem with negligible improvement.

Component Based Architecture

This is a relatively old presentation now, but it's a useful one.
http://cmpmedia.vo.llnwd.net/o1/vault/gdccanada09/slides/marcinchadyGDCCanada.ppt

It describes some of the architecture of the game Prototype. It has a very nice description of what goes wrong with simple OO solutions for games when as the scale of a project grows.

Looking Forward to this One

Well it's been... far too long since the last time I've posted here so I'm going to try to get back into the habit again.

Short story: I'm married, and have been since August 2010, work varies between satisfyingly challenging to insane, I have a pet cat named Doom, and I've barely had any time to do real hobby development work.

I'm going to keep things short so that I'm less likely to get overly lazy and stop posing for months at a time. I'm also not going to restrict my posts to anything in particular... beyond avoiding certain particulars of my job due to NDAs.

During work, I'll spend a smallish amount of time browsing the web. This helps to prevent me from going into mental vapor-lock and keeps me working faster than I would otherwise. I have a few sites I tend to go to that don't take much time to read, and relate just enough to what I'm doing that I don't switch out of the appropriate mindset for coding. One of these sites is the TIGsource blog. This is where I discovered a game that I'm very very excited to see. Hawken.

The game is a mech-sim like game with a pace more on the level of a classic multiplayer deathmatch. Either way, it's been far too long since I got a real fun mech game (years since I got Mechwarrior 4) and Hawken has a very nice feel based on the released gameplay footage. I also quite like its futuristic high tech shantytown setting. Hope the game turns out as awesome as it looks.

Three Projects... Four Projects... Five Projects!?!

Yes, well (conservatively) counting what I'm up to at RIM. The project I'm wrapping up is a particularly fun and rewarding one, creating a tool for providing detailed code coverage tracking for the various testing efforts within the OS Group. Sadly, the details aren't all that exciting. Mostly it's just processing gobs of similar data into an easily queried form. The majority of the work is in the logistics of collecting the data in a way that doesn't interfere with testing and manages to collect as much information is possible given the limitations of the testing environment. My other project(s) relate to some internal OS Group initiatives. I could tell you about them, but then I'd have to kill you ;)

So that's two...ish. Where's the other three? Well...

1. Space Combat Sim
   Yup, that thing is still going. Adding bots, improving the graphics, cleaning the code up and generally learning as much about 3D networked games as possible.
2. A shmup game developed from scratch for XNA on X-Box 360 and PC.
   Something simple for building a strong C# codebase that I can port to other projects. Goals here are making a programmable effects system and some decent GUI widgets.
3. Another shmup, using FlatRedBall
   Why? Because screw you! That's why... Actually I figured I'd just start messing around with plugging tools together just since that's normally not the way I make games.

Actually, to go on a boring long monologue...

I have finished only a few game projects, mainly for school. That tends to be because I approach the subject by first building tools and such, then building the game using them. I end up having so much fun making the tools that the project devolves into adding new features and refining code that I lose focus on making the game. Then I get bored and start a new project.

Now, I'm a tools developer professionally I can finally spend all hours of the working day making all sorts of exciting tools for a moderately hardcore development environment. This means that I'm not getting quite the same buzz from making yet more tools at home. So... time to make some fugging games!

Later d00dz! I'll try to keep the interval to next post to less than a month this time.

Some Inspiration

Lately I've been getting most of my coding jollies from my job as a tools developer at RIM. Mostly Python with a nice dose of C, SQL, and other stuff. Mainly fixing some reporting systems that go to external reports seen by management. Plus a dash of code janatorial work since nobody else seemed to be available/willing. After eight months doing solo work it's very nice to be working with a team again.

Anyway, that's not actually what I wanted to talk about. Today's just a shorty (I have a longer rant about myself coming later.) TIGSource recently posted a couple montages of a huge collection of free, indie games. I've played a few of them (Fraxy, Cave Story, Cho Ren Sha 68k, Prototype 2, Warning Forever, Clean Asia and one or two I can't recall... yea I'm a SHMUP fan) Holy hell there's some awesomeness.

Check it out!

So, What Now?

Well, today I spent a TON of time writing up blogs that I've been meaning to finish for a long time now. Three of them in fact! The first on exporting from Blender to XNA, the second on how the demos for Space Combat Sim went, and the third is a postmortem. But, I haven't talked about what I'm doing now, have I?

The short answer is that I have a job. In fact, I started my job before my "good demo" which was on May 4th. I'm working as a testing tools developer for Research in Motion. Unless you've been living under a rock for the past few years you've probably heard of their flagship product. Before I started this blog I worked as an intern for RIM over a period of 16 months. I work for the Operating Systems group which is responsible for all of the hardware drivers etc. that make the BlackBerry phone work. The culture of the OS Group, relaxed, friendly and fairly close knit, is what motivated me to return to RIM after finishing school. I'm quite happy working somewhere that I can spend the day working on hard technical problems with minimal political friction then finish the day by hanging around the pub.

My job now is to maintain and improve tools for automatically testing the Blackberry OS. Obviously I can't go into specific but I will be able to come out with articles on scripting, automation, testing and testing tools in a general sense. On top of the occasional post about work I'll still continue my game development work which means, of coure, more posts on Space Combat Sim and other projects as well.

Cheers.

Space Combat Sim Postmortem

In the spirit of continually learning I'm going to do a postmortem of Space Combat Sim as a school project. Don't worry though, this doesn't mean the project is dead. But the priority now is on having fun with it again, instead of busting my ass getting it demo-ready.

I'm basing my postmortem on those typically found on Gamasutra. If you haven't read one do so, they're quite eye-opening.

What Went Right
1. Tight, focused scope
Game projects can be massively complex. Particularly when they're 3d action-oriented online ones :). Most of my early projects started out relatively simply but grew into complex behemoths due to "wouldn't it be cool if" syndrome. For example a simple game where you killed enemies before they could leave the screen evolved into a vertical shmup with procedurally generated levels.

For Space Combat Sim the scope was set in stone from the beginning. The project would be a multiplayer game combining elements of X-Wing, Quake 3 and Counterstrike. Tools were only to be developed for the game if they were absolutely necessary (ignoring content pipeline extensions only two tools were made, one for creating hit boxes and one for creating effects.)

With the scope in hand I was able to set out a series of broad goals which defined how the game was to be built from the ground up. These broad goals were converted into a 24 page document describing all of the required features of the game (required for the course.) That was then turned into the detailed list of tasks that needed to be done to produce the game. Making the game was then simply a matter of finishing all of the tasks on the list. Since all time was spent exclusively on those broad goals the amount of wasted effort was minimized.

2. Continuous integration
Continuous integration means always having a working version of a project while it's being built. There may be missing features, and if it's a game it may look and play like crap, but it at least exists. One thing that always happens with software projects is scheduling issues. Feature creep, scope creep, unanticipated challenges or whatever, eventually something will cause a project to go off course. Having a working version means that even if the deadling is looming at least something is there for everyone to see.

In the case of Space Combat Sim, a lot could go wrong. I was challenging myself technically since I hadn't produced a 3D game before, nor had I produced a multiplayer game before. Except for a few occasions I succeeded in this goal. This was very helpful in keeping me from being too ambitious in coding features and forced me to break down overly lofty goals into more doable tasks.

3. Scrum-like scheduling
Space Combat Sim used a barebones version of Scrum, albeit with each cycle of development being much longer than specified. By dividing the project into sprints and assigning a set of real deliverables to be completed in each I was able to keep on task. The faculty's imposed limit of about 8 hours per deliverable also helped keep tasks from meandering and getting out of control.

What Went Wrong
1. Not enough time spent getting tools
The first four or five months of development time were spent building up the back-end code needed to support the first network session. A huge amount of that could have been saved if I spent that first month just looking at rendering engines and playing with them. No more shader development, effect tool development, etc. etc. in all about two months of work there. Sure, I'd still have to spend time on managing game objects and networking but that was kinda the point of the game. Instead I gave up after about two weeks of looking at tools, honestly, because it was boring.

2. Working solo
Especially in combination with making my own tools working solo made the game take a painfully long time to get done. It would have been quite nice if I could have focused more on object management, networking and gameplay while someone else took care of UI, loading, and other backend stuff. With only myself I generally had to focus on every task of game development which tended to result in polish getting short shrift.

The game's code is poorly documented in places, hacks abound and not all of the visuals that I wanted are in the game.

3. Continuous integration
Continuous integration has proven to be a double-edged sword. On one hand, having a working copy of Space Combat Sim all the time was motivating and helped reduce the impact of schedule slippage. On the other hand it made me midjudge whether a particular piece of the game's architecture was actually done or not. The area where this resulted in the most wasted time was in the game's startup sequence. The issue was that, while the game did indeed have space for its menus and loading sequence it wasn't really set up with its final usage in mind.

Having something that works, and having something that works correctly are different things. Continuous integration overall has been very useful, but at the same time it can cause one to do something that is expedient and fast rather than the correct solution suitable for the remainder of the project.

Conclusion
Overall I'd consider Space Combat Sim a modest success. I met my basic goal, which was to produce a multi-player combat game in space. My more lofty goals of having mission-oriented combat, style etc. weren't met due to time pressures though. The experience has definately been worth while and I learned a lot about actual project development through it.