The New Game: Staying Agile

Three months ago, I was invited at an Extreme Programming seminar held by a large French company. During four hours, I watched people from various departments show off to their colleagues the successes they were having with XP. A viral marketer’s dream come true!

This is yet another indication that agile methodologies are crossing the chasm.

One must admit that agility comes with many enticing promises. Adapting easily to change, putting the customer first, enjoying high levels of quality, fostering teamwork... Very nice indeed! The next question is: will people who make the switch automatically get those benefits?

I entered the world of software development in the mid-nineties. At that time, the Big Thing was object-oriented design. There is no doubt that OO is now a mainstream technology, yet how many developers really know how to do OO design? How many projects end up with clean, modular code structures? How many teams create components that are reused across projects? From my own experience, not that many.

My concern is that Agile is probably about to encounter the same fate.

For teams that make the switch to agile processes, there’s a honeymoon period - maybe one or two years - before the code starts showing its age and its weight. What do you do when your 2.000 tests take an hour or so to run? What about 10.000 tests? What do you do when you realize that some features would require a refactoring session of more than three weeks because the code is too rigid?

I know, I know… In theory, you should have done something way before your tests were taking an hour to run. Your code should have been refactored mercilessly, making it impossible for any new evolution to take more than a couple of days. But in practice, teams that have been trying to be agile for a few years discover that as the code base expands it becomes increasingly difficult to play by the rules.

The new game

My impression is that the switch to agile methodologies is changing the rules of the software development game.

In the waterfall era, the game was about completing a clear goal fast - give us the specs, and we’ll be the fastest to put up a project and ship the software with acceptable quality, at the lowest cost. Never mind if at the end of that project the code base is rigid and fragile, because what follows is supposed to be a “maintenance” phase where only minor evolutions are expected.

In the agile era, the emphasis is put on responding quickly to customer demand, shaping the software along the way, and without getting tangled into one’s own code. The challenge is to be to able to satisfy new customer requests quickly, even when the application becomes large. In other words, the challenge is not only to be agile, but to stay agile over the years.

I stumbled a few years ago on a good metaphor for this. A guy was comparing the creation of software with a game of Tetris:

From the developers perspective, new features are “falling” on their software, and their job is to integrate these new features as cleanly as possible into the existing code. The more they fail, the less room (and time) they have for new features, and the more difficult the game becomes - which makes the situation even worse. Rings a bell?

No escape

Sometimes we think that the flow of new features will eventually stop, and that we can just live with a less than ideal code structure. The tricky part is knowing whether the flow of user requests stops because all of their needs are genuinely satisfied, or if it stops because they have given up any hope of getting a better tool for a reasonable price.

I have worked on projects for which the functional scope was quite limited. For instance, on a communications server, once you have implemented a given communication protocol, you are more or less done. But in the majority of cases, the problem is completely open-ended - you have people out there doing their jobs, and you try to devise software solutions to help them save time. What I have found is that in the second case, the flow of new features can be nearly infinite.

First, as soon as users start working with the software, they will uncover many cumbersome interactions. You will be asked to improve these interactions again and again.

Second, when we start a project, we usually address only a tiny part of the business domain we are tackling. When new needs appear in an area that is functionally close to that of our software, there will be a strong temptation to integrate them into the existing software, for at least two reasons:

• users do not want to have to switch unnecessarily from one system to the other.
• there may be strong links between the new concepts and those managed by our system.

In these contexts, the limiting factor is the ability of the development team to keep adding features cheaply, and organize the UI so that it remains uncluttered. Easier said than done.

The pitfalls

This new game has its own typical difficulties:

• Declining performance: you have more and more code to compile, plus more and more tests to run. How do you optimize all this so that it still feels like working on a small project?
• Code complexity: the code gets larger and larger, and more and more complex. It becomes increasingly difficult to keep a view of the whole in one’s head. How do you maintain a coherent whole?
• Maturity levels: as you come up with new ways of doing things, you start mixing “legacy” and “new” ways of coding in the same application. Once again, how do you maintain a coherent whole?
• UI complexity: when adding new features, it is more and more difficult to keep the integrity of the whole, and from the users perspective the application is getting too complex. How do you satisfy a growing number of users and needs, without turning users down?
• Team turnover: people get tired of working on this project, and the people who know the most about it are replaced by newcomers. How do you maintain your development speed and the integrity of the whole? How do you keep your best people?

Conclusion

This is a tough game, isn’t it?

I will not claim to have an easy answer for all these problems, and I suspect nobody can. However, my goal in this blog is to explore how this new game can be played successfully. In the next posts, I will start exploring solutions in the realm of Lean management, and very soon Alexander's Theory of Centers.

By the way... How well are you doing on your own project?