Beyond XP: learning from Lean

Ten years already! Wow, time flies.

Ten years ago, I was avidly looking for material about object-oriented design and design patterns when I stumbled upon a little website named "Armaties.com". On this site, a guy named Ron Jeffries was writing about a new software development process where people were coding in pairs, releasing often, and writing lots of automated tests... We were in 1998, and Extreme Programming was being invented.

Three years later, I started using XP on my own projects. I created a French website about XP, I co-authored a book about XP, I gave talks about XP... Seven years and about 150 XP iterations later, I still feel that I have a lot to learn to do it really right, but let's be honest: I am feeling a strong need for fresh air.

The edge

I have had great results with XP, and I have also learned fundamental things - the power of automated tests, growing code through continuous refactoring, releasing often to create value and gather client feedback, and above all the magic of efficient teamwork.

That being said, there are a few points where XP still feels incomplete.

• What about usability? There is more to say about that than just having an on-site customer.
• How do you measure the performance of your team? Velocity is not an answer, since it is too easy to simply raise your estimates, and since you may brag about a high velocity but still produce useless features.
• If you cannot measure performance, how do you manage the improvement process?
• If you cannot measure performance, how do you know whether your process is getting you good results?
• How do you find a good balance between investing in refactoring and adding new features?

I do not see these questions as fundamental flaws in XP. XP is not an end in itself, and I do not expect any methodology to be 100% complete. I rather see these points as opportunities to complete my understanding of the development process, and to improve my ability to deliver better software faster. It is in that context that I am approaching Lean today.

Enter Lean

I discovered the world of Lean two years ago through Mary Poppendieck's book "Lean Toolkit", which encouraged me to explore the domain a little further. I then went through Goldratt's "The Goal", and later on Taichi Ohno's "The Toyota Production System". Six months ago, I had the opportunity to meet people from the French Lean community, and since then I am feeling the same interest in Lean as I felt for XP a decade ago. To make things worse, I just finished reading the excellent novel "The Gold Mine", by Freddy and Michael Balle, which brought many new ideas that now keep my mental cogs turning. As a result, I am seeing Lean processes in restaurants, at the canteen, at the hospital... everywhere! I even wonder sometimes whether all this Lean stuff is having a negative influence on my mental health.

Of course, I also see Lean processes in development projects.

Exploring Lean

According to authors Womack and Jones, the Lean approach evolves around five fundamental principles:

• Specify value in the eyes of the customer with concrete measures and figures.
• Identify the value stream and eliminate waste, with a strong focus on reducing inventory to uncover inefficiencies.
• Make value flow at the pull of the customer instead of pushing products onto heaps of inventory.
• Involve and empower employees, teaching them how to improve their processes on their own.
• Continuously improve in the pursuit of perfection, using a scientific approach in which you state hypotheses and measure results.

When I explore these concepts, I find many similarities with XP principles:

• the willingness to deliver value early and often
• having the customer himself drive the process by pulling features
• striving to have a minimal process, to let people focus on delivering value instead of feeding the process
• the pursuit of perfection through continuous improvement
• standardizing the process with time-boxed iterations
• relying on direct communication and fostering self-organization

However, one should not overlook two important differences between a manufacturing process and a software development process.

First, no two tasks are the same. If they were, the development team would automate them, leaving only what is different. Over time, repetitive tasks would be significantly reduced - or even completely automated. As a consequence, what is left is mostly unpredictible work.

Second, a development process is a system with memory. Each cycle leaves some code that new cycles will have to deal with, and all these modifications accumulate until the complexity brings the process to a halt. As I stated in a previous post, staying agile is the new game. This is a big constraint for developers, but not for manufacturers (at least when seen from the outside).

For these reasons, I cannot see how Lean could be efficiently implemented in a development team without using many XP practices to take into account the specific needs of a software development process. My take here is that Lean is a means for pushing XP further, or going beyond XP if you prefer.

What we can learn from Lean

I think that there are four general areas in which we can draw lessons from the Lean approach.

The first area is the concept of measuring value. As I noted above, the team velocity is not a measure of performance, but only a tool for planning. What could we use to measure user-perceived performance and drive the improvement process?

The second area is that of standardizing work, mapping the process and eliminating waste. Existing efforts have been made at the iteration and task levels. That's a good thing, but I think that we can go much further (and closer to the Lean ideal) by going into much more detail - i.e. what the developers have to deal with minute by minute. This relates to my work on cognitive friction and software usability.

The third area is the idea of using a formal approach for process improvement - analyze problems, draw hypotheses, perform experiments, and validate findings.

The fourth area is the idea of managers being responsible for helping workers optimize the process. In software development, many managers lose contact with the daily work of developers. They only see people in front of keyboards, and are not able to really see what the developers are doing. We need team leaders who can observe the work of developers at a fine level of detail, and help them find ways to alleviate any friction that may be slowing them down.

This leaves lots of work for future posts!