Don’t Get Drowned by Waterfall: Break out of the Delusion

Many development projects are built with an approach traditionally called “Waterfall”, or “Big bang all at once”. This means that the entire system is defined, designed, developed, tested, then released in that precise order. In its purest form, a system built using waterfall techniques is utterly useless until the time when nearly all of its features are claimed to be “feature complete” and “ready for testing”.

To put it very simply: if a project has 50 features, a team will attempt to build all 50 and only then test all 50 at the same time without any formal, professional QA and automated testing done during the construction of the features. This is a recipe for wildly missed delivery dates, at best, and utter disaster, at worst.

Irrational Justifications for Waterfall

Why in the world would a team try to build a system this way? You might hear something like “The whole system must work before we go live, so we’re going to test everything together”. On the surface, this sounds reasonable. Of course the whole system needs functional, end-to-end testing. However, in any system of non-triviality, this is, at best, an ignorant statement that betrays a lack of experience in developing complex systems. At worst, it’s a statement of learned helplessness or laziness born out of a lack of desire or perceived lack of time to break down the system into smaller, independently testable parts.

The Fantasy of Waterfall

The fantasy vision that people developing a waterfall project have is something like the blue line in the chart below from Steve Bohlen’s Autumn of Agile screencast series. In this vision, late in the project all the components that have not yet worked together in completion during the first 80% of the project get tested all at once and suddenly start to work and the project is released on time on an budget.

Yeah. Right.

Here’s the fantasy vision, compared with an Agile approach in terms of business value accrued over time, especially if the project suddenly is called to halt:

Continuing with the Waterfall delusion, the idea is that all aspects of the system can be developed in isolation from each other, never needing feedback or rework:

 

The Reality of Waterfall

The reality is shown below. The sharp decrease in the component stability represents the big “Oops, wish we would have thought of that sooner” moments that we all know and love during waterfall projects.

I can think of two instances off the top of my head during recent real-world projects where I’ve seen this:

1. While working on a large electronic commerce application rewrite, our team warned our senior management that the legacy COM architecture would not scale well under .NET. Despite our warnings and evidence, our management wanted us to “just forge ahead” as if it was more brave or honorable to continue doing something completely stupid rather than sit down with all of us and think through the difficulties necessary to properly solve the problem and devise a plan to reach the market and achieve positive ROI. Psychologists call this cognitive dissonance. I call it lack of planning, ignorant, or maybe just cognitionless dumbness. I eventually left that position and have kept in contact with my old colleagues who eventually had to rewrite the entire COM layer in pure .NET to attain the desired business value (release). 
2. On another recent project I’ve worked on, I was assigned the responsibility of designing and implementing a complex security model on top of the existing custom entity model the new application was already using. Hundreds of entities had already been created in this system, and a large, wide horizontal stretch of pages and controls had been created already. But, no features had yet been designed vertically deep enough to use security. As a result, the system was thus far built entirely with static stored-procedures, but the complex security model required that all SQL statements be appended with additional where clauses and custom filters. Fixing this took weeks to refactor the data-access layer and rework static stored procedures to use more of a Query Object pattern.

Special Note: Microsoft’s Advice for Systems Re-architecture and Migration

Microsoft has a document entitled “Microsoft .NET/COM Migration and Interoperability'”, located here: http://msdn.microsoft.com/en-us/library/ee817653.aspx. It will serve all .NET or COM / C++ developers to read this document.

Microsoft recommends that when you are re-architecting a system using .NET from an older technology, such as COM / C++ that you do not attempt a big-bang, horizontal migration. Instead, they recommend that you create a completely functional, vertical slice of the application first, before expanding horiztonally.

Here is an excerpt:

“You might choose to adopt a vertical migration strategy for a number of reasons:

• Planning to re-architect
  If you plan to re-architect your application, vertically migrating part of your application to the new architecture provides a good test bed for the new design. The .NET Framework also makes it easier to provide the functionality that newer architectures are built on. For example, you can use HttpHandlers to perform many of the tasks for which you would previously use ISAPI extensions, but with a much simpler programming model.”

Here is a diagram from the same document depicting a vertical migration:

How to Stop The Horizontal Waterfall Madness

If you or your project is on a path of waterfall, horizontal development, then you have your work cut out for you, but it’s not too late. It takes discipline, honesty, and courage to set things upright and vertical.

Here are a few key practical steps:

1. Stop adding code to the system that is not scheduled for testing in the current or next month. If you do not have a product road-map and product back-log prioritized by business-value and thus don’t know when a feature is going to need that code, then stop adding it, immediately.
2. Focus instead on the fact that you need a road-map and a prioritized product back-log and define this in terms of users’ needs. If your product owner cannot or won’t prioritize the backlog or features, then simply list them in the order that your users encounter the features or the order that your help-desk team tells you needs most improvement.
3. Now, identify a vertical slice of the application that you can focus everyone on the team on implementing from top-to-bottom. Determine how to get this slice as close 100% functional as possible.
4. Work daily with your users, test team, developers, and other stakeholders to create, together, a strategy for standing up and ruthlessly testing that vertical slice.
5. Once you’ve gotten critical user feedback for usability and functionality, apply those lessons learned to the next vertical slice, and so forth!

Whose Responsibility is it to Stop Waterfall Thinking?

It’s far too easy for us as developers or architects to say that we’re just doing what we’re told by our managers or our business owners. After all, they “Just want the product done”, right?

Certainly, we must listen to what our managers and owners ask of us. It’s very important that we do so. However, what’s more important, to them and to the success of projects, is that we as professionals bring our expertise and our knowledge to the problem-at-hand and that we act with resolve and courage to do the job correctly. Often times this means sitting down with our managers and owners and explaining to them, however uncomfortable it makes them or you, what it really means to do iterative development with high quality. Often we must educate them about the history of waterfall and its miserable rate of success.

Thus, the responsibility is yours. The responsibility is mine. The responsibility is the team’s. 

Craig Larman on the History of Waterfall and Iterative & Incremental Development

Horizontal waterfall approaches, despite being popular and widespread today, came after early practitioners used iterative & incremental development techniques to greater success. The sad but true history of waterfall is that the U.S. Department of Defense (DOD) misinterpreted Winston W. Royce's paper on systems development and went on to enforce it as a government standard to which large contractors then adhered. Commercial industry followed suit from the government contractors. The DOD went on the revise their standards, and within the last twenty years they have recommended iterative & incremental practices.

Object-oriented development guru Craig Larman has written several books about iterative & increment project management over the years. He also wrote an extensive article about the history of iterative & increment development (IID) here: http://www.highproductivity.org/r6047.pdf

Here’s a crucial excerpt history from that article:

In the 1970s and 1980s, some IID projects still incorporated a preliminary major specification stage, although their teams developed them in iterations with minor feedback. In the 1990s, in contrast, methods tended to avoid this model, preferring less early specification work and a stronger evolutionary analysis approach.

The DoD was still experiencing many failures with “waterfall-mentality” projects. To correct this and to reemphasize the need to replace the waterfall model with IID, the Defense Science Board Task Force on Acquiring Defense Software Commercially, chaired by Paul Kaminski, issued a report in June 1994 that stated simply, “DoD must manage programs using iterative development. Apply evolutionary development with rapid deployment of initial functional capability.”

Consequently, in December 1994, Mil-Std-498 replaced 2167A. An article by Maj. George Newberry summarizing the changes included a section titled “Removing the Waterfall Bias,” in which he described the goal of encouraging evolutionary acquisition and IID:

Mil-Std-498 describes software development in one or more incremental builds. Each build implements a specified subset of the planned capabilities. The process steps are repeated for each build, and within each build, steps may be overlapping and iterative.

Mil-Std-498 itself clearly states the core IID practices of evolving requirements and design incrementally with implementation:

If a system is developed in multiple builds, its requirements may not be fully defined until the final build…. If a system is designed in multiple builds, its design may not be fully defined until the final build.

Tom Gilb’s Principles of Software Engineering Management was the first book with substantial chapters dedicated to IID discussion and promotion. Computer Meanwhile, in the commercial realm, Jeff Sutherland and Ken Schwaber at Easel Corp. had started to apply what would become known as the Scrum method, which employed time-boxed 30-day iterations. The method took inspiration from a Japanese IID approach used for non-software products at Honda, Canon, and Fujitsu in the 1980s; from Shashimi (“slices” or iterations); and from a version of Scrum described in 1986. A 1999 article described their later refinements to Scrum.

Stop laying down flat. Stand up straight. Have more fun, and kick waterfall to the curb!