Agile Software Construction is Like Neighborhood Construction

In my last article, I commented on Dr. Jan Chong's dissertation presentation "Software Development Practices and Knowledge Sharing: A Comparison of XP and Waterfall Team Behaviors". I raised some questions about the utility of long-term design documents and provided links to Scott Ambler and Mary Poppendieck's writings about unit tests and automated tests as a form of long-term executable specifications.

In this article, I'm going to give a better version of an often heard analogy that "Software construction is like building construction". Usually you hear this when product owners, project managers, or architects say things like, "We can't do that until we've got the foundation in place. Think of a building. You have to have a solid foundation first."

OK. Maybe so, but you only need it for the very first independent feature, and you should be building features as independently as you can. Here's why:

The Simple Building to Software Analogy

Here is a simplified one-to-one comparison that maps the "layers" of a building to the layers of a software application:

Building Component	Software Component
Furnishings	User Interface Layer
Internal Structure	Business Logic Layer
External Structure	Data Access Layer
Foundation	Database
Blueprints	Requirements Specifications

Traditional Versus Agile Software Development

To prepare my final point, I present the following two slides come from the Autumn of Agile screen-cast series mentioned in prior posts.

This first slide depicts the traditional sequential waterfall approach to building an application. The main theme is that waterfall projects typically result in teams developing horizontal layers that end up delivering business value only after all layers are complete.

 

This next slide depicts how agile (or simply iterative & incremental) projects develop systems. The main idea here is that features are developed vertically. This means that the features are independently valuable, and can represent a portion of the total desired value at any time should the project schedule get cut short.

 

 

Neighborhoods Get Built Using Agile, Not Waterfall

Let's assume we have 5 total features in the software system depicted in slides one and two. In the first slide, all 5 features are "finished" at the same time when the user-interface is built on top of the lower layers. In the second slide, a single feature is built from "top to bottom" during each iteration.

Can You Say Fffired?

Imagine a team of developers building a neighborhood like the traditional sequential waterfall approach to building software. Let's assume each iteration is 5 weeks for simplicity's sake. After 5 weeks, the developers would have built 5 separate sets of blueprints for the 5 houses to be built. Imagine that these are 5 or more separate parts of an overall software requirements specification in a software development project. They might describe database table schemas, relationships, multiplicities, etc.

	Week 1	Week 2	Week 3	Week 4	Week 5	Elapsed Weeks
Iteration 5						25
Iteration 4						20
Iteration 3						15
Iteration 2						10
Iteration 1	Blueprints	Blueprints	Blueprints	Blueprints	Blueprints	5

I'm sure you already see how asinine this approach to neighborhood development is.

Nevertheless, let's continue with the next 5 weeks and see how far our team has gotten toward delivering the first "potentially habitable housing increment":

 

	Week 1	Week 2	Week 3	Week 4	Week 5	Elapsed Weeks
Iteration 5						25
Iteration 4						20
Iteration 3						15
Iteration 2	Foundation	Foundation	Foundation	Foundation	Foundation	10
Iteration 1	Blueprints	Blueprints	Blueprints	Blueprints	Blueprints	5

After 10 weeks of work, our construction company has failed to complete a single inhabitable house. What are we paying these people for? Why are they building a foundation, then moving to the next lot to build the next one? I guess they're just going to get back to it sometime later.

Let's just speed ahead and see what we have after 25 weeks on the project:

	Week 1	Week 2	Week 3	Week 4	Week 5	Elapsed Weeks
Iteration 5	Furnishings	Furnishings	Furnishings	Furnishings	Furnishings	25
Iteration 4	Internal Structure	Internal Structure	Internal Structure	Internal Structure	Internal Structure	20
Iteration 3	External Structure	External Structure	External Structure	External Structure	External Structure	15
Iteration 2	Foundation	Foundation	Foundation	Foundation	Foundation	10
Iteration 1	Blueprints	Blueprints	Blueprints	Blueprints	Blueprints	5

This is great, finally. After 25 weeks we finally have inhabitable houses! Actually we had one inhabitable house after week 20, then one more per week until 25. That almost feels like a privilege at this point to get so many houses finished so quickly in succession.

Not in My Backyard!

Do construction companies ever really build houses like this? Not that I've ever seen. They try to complete a house as quickly as they can after the foundation concrete sets. Of course, that means they may indeed start on a second foundation before they build the remainder of the first house, but you get my point.

Suppose it were 30 houses instead of 5. Could you imagine if this were a new neighborhood being built just behind your backyard. What would you think if you saw the crews building 30 separate foundations, then coming back to the first to frame it, then moving to the next to frame it, etc, etc. You'd think they had last their minds or have absolutely no respect or understanding of the Time-Value-of-Money or concern for a Return-On-Investment.

Neighborhood Construction is More Vertical, Like Agile

As you can see below, our construction crew progresses through each house from "the ground up", which means that they finish one house completely every five weeks. This, in agile terminology, is their velocity per iteration. The learn more about agile teams and velocity, read my article "D = V * T : The formula in software DeVelopmenT to get features DONE".

	Iteration 1	Iteration 2	Iteration 3	Iteration 4	Iteration 5
Weeks Elapsed / Houses Completed	5 : 1	10 : 2	15 : 3	20 : 4	25 : 5
Week  5	Furnishings	Furnishings	Furnishings	Furnishings	Furnishings
Week 4	Internal Structure	Internal Structure	Internal Structure	Internal Structure	Internal Structure
Week 3	External Structure	External Structure	External Structure	External Structure	External Structure
Week 2	Foundation	Foundation	Foundation	Foundation	Foundation
Week 1	Blueprints	Blueprints	Blueprints	Blueprints	Blueprints

Comparing Velocity between Waterfall and Agile Approaches

In the following table, we're looking at how many houses we complete per iteration.

So, the calculation is number of iterations passed / number of houses completed.

Thus, the agile velocity is constant in terms of 5 weeks. We can always count on 1 house becoming complete every 5 weeks.

	Total Completed Under Waterfall	Waterfall Velocity per Iteration	Total Completed Under Agile	Agile Velocity per Iteration
Week 5	0	0	1	1
Week 10	0	0	2	1
Week 15	0	0	3	1
Week 20	0	0	4	1
Week 25	5	1	5	1

Combining Return on Investment with Velocity

Finally, since we know that nobody can live in a house until it's completed, we can calculate the potential accrued Return-on-Investment (ROI) after each iteration for actually having sold houses.

Let's assume we can sell each house for $100,000.

	Potential Waterfall ROI	Potential Agile ROI
Week 5	0	$100,000
Week 10	0	$200,000
Week 15	0	$300,000
Week 20	0	$400,000
Week 25	$500,000	$500,000

Taking into account the Time-Value-of-Money (TVM) we understand that it's certainly more valuable to realize returns via the agile approach to building neighborhoods as opposed to the waterfall method!

Similarly, when we talk about ROI and TVM regarding software projects developed using agile methods, certainly we sometimes mean that we can gain monetary returns earlier by producing "potentially shippable product increments" faster and getting them to the market. However, what we also mean is that by focusing on building features top-to-bottom, we can get the complete experience and feedback necessary to then apply toward the next feature and we prevent the pains of having to refactor code later when it's no longer fresh and when requirements have been changing all round.

Here is a good video on YouTube which depicts this relationship between time and value accrual very well:

Until next time, stay agile, not fragile!

Review: Software Development Practices and Knowledge Sharing: A Comparison of XP and Waterfall Team Behaviors

Jan Chong, Ph.D., wrote her dissertation on "Software Development Practices and Knowledge Sharing: A Comparison of XP and Waterfall Team Behaviors". I came across a recorded presentation of her discussing her dissertation at The Research Channel here:

http://www.researchchannel.org/prog/displayevent.aspx?rID=16075&fID=345

Here is the description of the recorded presentation:

My dissertation research explores knowledge sharing behaviors among two teams of software developers, looking at how knowledge sharing may be effected by a team's choice of software development methodology. I conducted ethnographic observations with two product teams, one which adopted eXtreme Programming (XP) and one which used waterfall methods. Through analysis of 808 knowledge sharing events witnessed over 9 months in the field, I demonstrate differences across the two teams in what knowledge is formally captured (say, in tools or written documents) and what knowledge is communicated explicitly between team members. I then discuss how the practices employed by the programmers and the configuration of their work setting influenced these knowledge sharing behaviors. I then suggest implications of these differences, for both software development practice and for systems that might support software development work.

Jan's full biography at the time of her dissertation:

Jan is a doctoral candidate in the Department of Management Science and Engineering at Stanford University.  She is affiliated with the Center for Work, Technology and Organization.  Her research interests include collaborative software engineering, agile methods, knowledge management and computer supported collaborative work.  Jan holds a B.S. and an M.S. in Computer Science from Stanford University.

Her complete dissertation is available here: http://www.amazon.com/Knowledge-Sharing-Software-Development-Comparing/dp/3639100840

Highlighted Slides from Recorded Presentation

It's nice to see a formal study that compares these different styles of work. Jan spent time observing both teams for about 9 months. Here is a bit more about her methodology.

Study Methodology

Team Communication Styles

First, for the XP team, communication is more open, faciltated by information radiators and an open workspace:

The waterfall team has team members who work alone, in their own cubicles, and who communicate primarily through an online chat program. Thus, they sometimes "broadcast" information to others inside of the chatroom:

Observed Events and Data Coding

Through analyzing her recordings and notes, Jan classified all the different knowledge-transfer events into the following categories:

Knowledge Seeking Behaviors

This slide compares the actions taken by teammates in explicitly asking for knowledge transfer from others:

Knowledge Offering and Relevance

Below Jan has categorized types of communication that are offered and their respective relevancies.

Recorded Knowledge

It's very interesting to note that the waterfall team recorded knowledge far more for the "long-term", percentage-wise than on the XP team. What is not clear, however, is whether this means that the XP teams simply recorded more personal items in addition to the same type of long-term items or whether they leave out certain long-term items that waterfall teams recorded properly.

Summary Slides

The following three slides are her concluding slides:

 

 

Review and Analysis

I'm about to join a great team for an agile project that will be built using Scrum & XP practices. This opportunity is very exciting for me. That excitement is born out of my own "in the trenches" experience and observations on both Agile/Scrum/XP and waterfall. If you're reading this and unfamiliar with the differences between agile and waterfall, I recommend you take both of them for a "test drive" by reading or listening to my article entitled "From Waterfall to Agile Development in 10 Minutes: An Introduction for Everyone". Also read "Don't Get Drowned by Waterfall: Break out the Delusion" and  "D = V * T : The formula in software DeVelopmenT to get features DONE"

If you cannot tell by now, my preference is for agile development and not waterfall. Waterfall, at least in its pure form, as you can read in the first article, is a mistake and always has been a mistake for software systems development. While Jan stops short of claiming a preference for one style of development or the other in her analysis, it's important to note that this is because that was not her intention. She is working on improving software methodology as a whole, and seeks to synthesize best practices from actual empirically observed behavior and data.

How Useful is the Long-Term Design Documentation by the Waterfall Team?

Jan observed that the waterfall teams had members who worked alone and went "back to the code" when they needed to understand something or had to work on a new module they had not worked on before. What I'm not clear on is whether she meant "automated test case code" or "implementation code". She also noted that the XP team members consulted one another more often about how things work prior to looking at code. She also noted that the waterfall teams created a higher percentage of recorded knowledge about the long-term aspects of the project. I am assuming this meant written documentation judging by her comments on video.

Question: How often the waterfall developers actually refer back to those long-term design documents, and which of those documents did they originally anticipate being useful to other developers?

The reason I would ask this is that she already noted that the waterfall team members spent a lot of time reading code, and also reading CVS check in messages when others checked in changes, but she didn't address whether they (or anyone) reads the long-term design documents for any useful purpose.

Experience Highest Communication Bandwidth via Face-to-Face at Whiteboard Collaboration

It has been my personal experience as a developer and architect that when working more closely, in a collaborative, open workspace, with other members of a team, I do not need to refer to implementation code or documentation as often anyway. Instead, we rely more a constantly evolved shared language, basic metaphors, automated test cases, and whiteboards to communicate the "gist" of how something works, then we refer to detailed implementation code as soon as we need the details or pinpoint a trouble area. Scott Ambler has written extensively on the subject on agile documentation and communication. Here is a chart he produced based on a survey about the most effective forms of communication:

 

Source: http://www.agilemodeling.com/essays/agileDocumentation.htm 

As you can see, paper documentation is by far the absolute worst form of communication. Face-to-face whiteboard is the most effective. Kevin Skibbe, a friend that I used to work with, is the most effective whiteboard communicator I know. What he explained to me was that when two, or more, people try to communicate via the whiteboard they must focus on developing a shared mental model. Even when it's just face-to-face communication, without a whiteboard, both people are still maintaining independent, non-shared mental models of what the other person is thinking.

 

Building Long-Term Executable Knowledge / Documentation via Automated Tests

One measurement I don't explicitly noted is the notion of "Executable Knowledge", or more frequently called "Executable Specifications". Scott Ambler writes about this here: http://www.agilemodeling.com/essays/executableSpecifications.htm, and Mary Poppendieck writes about it in her books and presentations: http://www.poppendieck.com/

I suggest that teams, whether waterfall or agile, incorporate this practice into their development in order to produce fewer defects, increase explicit knowledge in the code-base, and reduce the need to continually read implementation code.

We should first think about software systems in light of their intrinsic nature and in terms of the change process required to modify or enhance them with maximum ability to produce the desired result. That result might be faster ROI in the market, increased product quality, or what have you.

What we never want to do however, is result in broken functionality after we release. This is why we build regression test suites that can be executed at will to confirm as much as computationally possible of the encoded knowledge about the business domain and user requirements.

This is how Test-Driven-Development works. Here is how Ambler depicts this view:

 

That is a nice technical view of things, but what about the view from the business side? Ambler presents a written form of the requirement that would be provided to developers from a business analyst in the article linked above.

 

Waterfall is Always Wrong Compared to Agile for Building a New System

I'm going to assume here a definition of waterfall that is primarily the standard sequential approach where requirements come first, followed by, detailed design, development, (test, drop scope, rework) (repeat prior three as necessary until completion), and deployment and maintenance.

If a project is about developing a completely new product from the ground up, then adopting the sequential model for a system of any degree of complexity beyond perhaps an estimated month of duration is simply calling for pain. I'm not going to explain why in this article, but I refer you to the three articles I wrote above for complete explanation. But, to summarize:

When you begin a moderate to large custom development project, there are many requirements you simply cannot know until you have started to develop a subset of the entire envisioned project and that subset makes its way into the hands of the critical stakeholders like the project owner and the target users.

For more evidence, refer to Craig Larman's research and explanation about the history of waterfall in this article: http://www.highproductivity.org/r6047.pdf

Waterfall is Still Wrong for Enhancing an Existing System

However, suppose a software system is already released and "in production", should a team then use waterfall techniques to build additional features?

I believe the answer is no. The reason is reflected above in the TDD diagram. You want to reduce the impact of changes and build up a suite of regression tests as you develop the solution. And, you want to seek feedback as early as possible to reduce time spent working on incorrect or undesired functionality.

 

Waterfall is Especially Wrong for Rebuilding an Existing System

In my experience, and I've been through it a couple of times now, waterfall is notoriously wrong when you are asked to rebuild an existing system in a new technology. The reason is that often the project sponsor will state little more than the fact that they want the existing system functionality essentially duplicated. Unfortunately, much time will be wasted by the team if it tries to clone piece-by-piece the existing product.

A much better approach is to use the existing system as a very high fidelity model. It should then use iterative and incremental agile practices to deliver features that are focused on making and keeping the project potentially shippable as soon as possible. This ensures a complete vertical "slice" of functionality through all the application's layers gets built as soon as possible. To learn more about these practices, see the screen-cast series Autumn of Agile at http://www.autumnofagile.net. The first episode gives a comprehensive and very compelling explanation of why agile presents better business value than waterfall. My article "Don't Get Drowned by Waterfall: Break out the Delusion" references a few key slides from that series.

 

The Embarrassment of Waterfall's Persistence

Waterfall continues to exist in the technology world because it sounds easy to understand on the surface, but everyone also realizes the intrinsic contradiction in software development: requirements constantly change. I don't blame product owners or business people for waterfall's persistence. I blame ourselves, the developers and project managers. It is our fault for not being more responsive to changing environmental conditions and changing requirements.

But, balancing the ability to change on demand with the requirement to remain stable at all other times is what agility is all about. That is why agile focuses on rigorous empirical testing, visual monitoring, and continuous feedback. That is why automated tests as executable specifications speed up the ability to change while simultaneously increasing quality and confidence.

Stay tuned for my next article which will update the "age old" building construction metaphor that says building is software is like building a building. In many ways it is, but I will make key distinctions that enable agility!

Until then, stay agile, not fragile.

Don’t Get Drowned by Waterfall: Break out of the Delusion

Many development projects are built with an approach traditionally called “Waterfall”, or “Big bang all at once”. This means that the entire system is defined, designed, developed, tested, then released in that precise order. In its purest form, a system built using waterfall techniques is utterly useless until the time when nearly all of its features are claimed to be “feature complete” and “ready for testing”.

To put it very simply: if a project has 50 features, a team will attempt to build all 50 and only then test all 50 at the same time without any formal, professional QA and automated testing done during the construction of the features. This is a recipe for wildly missed delivery dates, at best, and utter disaster, at worst.

Irrational Justifications for Waterfall

Why in the world would a team try to build a system this way? You might hear something like “The whole system must work before we go live, so we’re going to test everything together”. On the surface, this sounds reasonable. Of course the whole system needs functional, end-to-end testing. However, in any system of non-triviality, this is, at best, an ignorant statement that betrays a lack of experience in developing complex systems. At worst, it’s a statement of learned helplessness or laziness born out of a lack of desire or perceived lack of time to break down the system into smaller, independently testable parts.

The Fantasy of Waterfall

The fantasy vision that people developing a waterfall project have is something like the blue line in the chart below from Steve Bohlen’s Autumn of Agile screencast series. In this vision, late in the project all the components that have not yet worked together in completion during the first 80% of the project get tested all at once and suddenly start to work and the project is released on time on an budget.

Yeah. Right.

Here’s the fantasy vision, compared with an Agile approach in terms of business value accrued over time, especially if the project suddenly is called to halt:

Continuing with the Waterfall delusion, the idea is that all aspects of the system can be developed in isolation from each other, never needing feedback or rework:

 

The Reality of Waterfall

The reality is shown below. The sharp decrease in the component stability represents the big “Oops, wish we would have thought of that sooner” moments that we all know and love during waterfall projects.

I can think of two instances off the top of my head during recent real-world projects where I’ve seen this:

1. While working on a large electronic commerce application rewrite, our team warned our senior management that the legacy COM architecture would not scale well under .NET. Despite our warnings and evidence, our management wanted us to “just forge ahead” as if it was more brave or honorable to continue doing something completely stupid rather than sit down with all of us and think through the difficulties necessary to properly solve the problem and devise a plan to reach the market and achieve positive ROI. Psychologists call this cognitive dissonance. I call it lack of planning, ignorant, or maybe just cognitionless dumbness. I eventually left that position and have kept in contact with my old colleagues who eventually had to rewrite the entire COM layer in pure .NET to attain the desired business value (release). 
2. On another recent project I’ve worked on, I was assigned the responsibility of designing and implementing a complex security model on top of the existing custom entity model the new application was already using. Hundreds of entities had already been created in this system, and a large, wide horizontal stretch of pages and controls had been created already. But, no features had yet been designed vertically deep enough to use security. As a result, the system was thus far built entirely with static stored-procedures, but the complex security model required that all SQL statements be appended with additional where clauses and custom filters. Fixing this took weeks to refactor the data-access layer and rework static stored procedures to use more of a Query Object pattern.

Special Note: Microsoft’s Advice for Systems Re-architecture and Migration

Microsoft has a document entitled “Microsoft .NET/COM Migration and Interoperability'”, located here: http://msdn.microsoft.com/en-us/library/ee817653.aspx. It will serve all .NET or COM / C++ developers to read this document.

Microsoft recommends that when you are re-architecting a system using .NET from an older technology, such as COM / C++ that you do not attempt a big-bang, horizontal migration. Instead, they recommend that you create a completely functional, vertical slice of the application first, before expanding horiztonally.

Here is an excerpt:

“You might choose to adopt a vertical migration strategy for a number of reasons:

• Planning to re-architect
  If you plan to re-architect your application, vertically migrating part of your application to the new architecture provides a good test bed for the new design. The .NET Framework also makes it easier to provide the functionality that newer architectures are built on. For example, you can use HttpHandlers to perform many of the tasks for which you would previously use ISAPI extensions, but with a much simpler programming model.”

Here is a diagram from the same document depicting a vertical migration:

How to Stop The Horizontal Waterfall Madness

If you or your project is on a path of waterfall, horizontal development, then you have your work cut out for you, but it’s not too late. It takes discipline, honesty, and courage to set things upright and vertical.

Here are a few key practical steps:

1. Stop adding code to the system that is not scheduled for testing in the current or next month. If you do not have a product road-map and product back-log prioritized by business-value and thus don’t know when a feature is going to need that code, then stop adding it, immediately.
2. Focus instead on the fact that you need a road-map and a prioritized product back-log and define this in terms of users’ needs. If your product owner cannot or won’t prioritize the backlog or features, then simply list them in the order that your users encounter the features or the order that your help-desk team tells you needs most improvement.
3. Now, identify a vertical slice of the application that you can focus everyone on the team on implementing from top-to-bottom. Determine how to get this slice as close 100% functional as possible.
4. Work daily with your users, test team, developers, and other stakeholders to create, together, a strategy for standing up and ruthlessly testing that vertical slice.
5. Once you’ve gotten critical user feedback for usability and functionality, apply those lessons learned to the next vertical slice, and so forth!

Whose Responsibility is it to Stop Waterfall Thinking?

It’s far too easy for us as developers or architects to say that we’re just doing what we’re told by our managers or our business owners. After all, they “Just want the product done”, right?

Certainly, we must listen to what our managers and owners ask of us. It’s very important that we do so. However, what’s more important, to them and to the success of projects, is that we as professionals bring our expertise and our knowledge to the problem-at-hand and that we act with resolve and courage to do the job correctly. Often times this means sitting down with our managers and owners and explaining to them, however uncomfortable it makes them or you, what it really means to do iterative development with high quality. Often we must educate them about the history of waterfall and its miserable rate of success.

Thus, the responsibility is yours. The responsibility is mine. The responsibility is the team’s. 

Craig Larman on the History of Waterfall and Iterative & Incremental Development

Horizontal waterfall approaches, despite being popular and widespread today, came after early practitioners used iterative & incremental development techniques to greater success. The sad but true history of waterfall is that the U.S. Department of Defense (DOD) misinterpreted Winston W. Royce's paper on systems development and went on to enforce it as a government standard to which large contractors then adhered. Commercial industry followed suit from the government contractors. The DOD went on the revise their standards, and within the last twenty years they have recommended iterative & incremental practices.

Object-oriented development guru Craig Larman has written several books about iterative & increment project management over the years. He also wrote an extensive article about the history of iterative & increment development (IID) here: http://www.highproductivity.org/r6047.pdf

Here’s a crucial excerpt history from that article:

In the 1970s and 1980s, some IID projects still incorporated a preliminary major specification stage, although their teams developed them in iterations with minor feedback. In the 1990s, in contrast, methods tended to avoid this model, preferring less early specification work and a stronger evolutionary analysis approach.

The DoD was still experiencing many failures with “waterfall-mentality” projects. To correct this and to reemphasize the need to replace the waterfall model with IID, the Defense Science Board Task Force on Acquiring Defense Software Commercially, chaired by Paul Kaminski, issued a report in June 1994 that stated simply, “DoD must manage programs using iterative development. Apply evolutionary development with rapid deployment of initial functional capability.”

Consequently, in December 1994, Mil-Std-498 replaced 2167A. An article by Maj. George Newberry summarizing the changes included a section titled “Removing the Waterfall Bias,” in which he described the goal of encouraging evolutionary acquisition and IID:

Mil-Std-498 describes software development in one or more incremental builds. Each build implements a specified subset of the planned capabilities. The process steps are repeated for each build, and within each build, steps may be overlapping and iterative.

Mil-Std-498 itself clearly states the core IID practices of evolving requirements and design incrementally with implementation:

If a system is developed in multiple builds, its requirements may not be fully defined until the final build…. If a system is designed in multiple builds, its design may not be fully defined until the final build.

Tom Gilb’s Principles of Software Engineering Management was the first book with substantial chapters dedicated to IID discussion and promotion. Computer Meanwhile, in the commercial realm, Jeff Sutherland and Ken Schwaber at Easel Corp. had started to apply what would become known as the Scrum method, which employed time-boxed 30-day iterations. The method took inspiration from a Japanese IID approach used for non-software products at Honda, Canon, and Fujitsu in the 1980s; from Shashimi (“slices” or iterations); and from a version of Scrum described in 1986. A 1999 article described their later refinements to Scrum.

Stop laying down flat. Stand up straight. Have more fun, and kick waterfall to the curb!

Book Review: Effective REST Services via .NET For .NET Framework 3.5

 
I’m currently reading Kenn Scribner and Scott Seely’s book Effective REST Services via .NET For .NET Framework 3.5. This is a great book which gives a thorough background on the history of REST, quoting often from Roy Fielding’s Ph.D. dissertation on the subject. The authors do not treat the subject with “kid gloves” or “framework gloves”. Instead, they exlain the HTTP protocol and its purpose, benefits, and its drawbacks.

Warning: I’m not really that good at giving book reviews for the general audience, and I feel like there’s already plenty of information on the web about all these subjects, but for own personal benefit it helps me to summarize subjects in my own words to clarify them. For me, it’s best for me to review a table-of-contents and then just give my best shot at summarizing what I’ve read in brief paragraphs using examples. This misses lots of details, of course, and some of what I remember won’t even come from the book.

For me, this is crucial research material for two projects I am working on. One of these is currently public and is a simple RESTful gaming platform. You can see the progress so far on that at http://apps.ultravioletconsulting.com/Skookzee.

So, don’t mistake this as a book review for anyone other than myself, but I hope you enjoy it anyway.

Chapter Summaries

1: RESTful Systems: Back to the Future

 

This chapter discusses how early on web “services” consisted simply of web servers and URIs that delivered chunks of data. Sometimes this was done through CGI processing that responded to the invocation of the incoming request, but eventually most web sites became simply file servers. Later, SOAP-based services became popular, but the problem here was that SOAP services are intrinsically Remote-Procedure-Call centric (RPC). The problem with this is that SOAP services typically encapsulate both NOUNS and VERBS, or Resources and Actions, Objects and Methods –- pick your metaphor – behind a single URI.

You might ask why this is a problem. One reason is that the HTTP and URI protocols were designed, as Roy Fielding describes in his dissertation, to work well in a distributed network and to enable distributed caching. Because SOAP specifically, and RPC in general, deal in coupling an invocation-time command to a remote resource, there is no architectural support to cache the results. Indeed, the idea of caching a call to a SOAP web service is something that each service would have to completely encapsulate.

Conversely, the way the web was actually designed was with a few key principles in mind to enable the distributed, cacheable structure of resources. First of all, the URI stands for Universal Resource Identifier, and a resource is a noun, it’s not an action or a method call. SOAP services are about exposing a single noun, the service entry point, and then encapsulating any number of nouns and verbs behind that one single noun.

So, in a RESTful system, each URI refers to a single, unique representation of a remote resource. For example, http://agilefromthegroundup.blogspot.com/2009/09/done-features-equals-velocity.html refers specifically to the HTML representation of my post about development velocity and tracking. There is, however, another URI which refers to this resource in another way: http://agilefromthegroundup.blogspot.com/feeds/posts/default/4352538909387142862. If you click this link you will get the content of that same blog post, but as XML inside of an “<entry>” element.

A good depiction of this distinction is found in the document Architecture of The World Wide Web on the W3C web site:

This becomes important because of REST’s concept of a single, generic interface, or a limited number of verbs/actions/methods. In terms of HTTP this turns out to be the standard set of HTTP methods: GET, PUT, POST, DELETE, HEAD, OPTIONS. Each of these methods must behave in a consistent, expected way when combined with a requested URI.

Another good reference, for quicker reading, is the summary of the web architecture document. This gives a list of prescriptive good practices for design.

For example, assuming I were authenticated and that Blogger supported this ability (I’m not sure whether it does) then issuing a request with each of the common HTTP methods above against the single URI http://agilefromthegroundup.blogspot.com/feeds/posts/default/4352538909387142862 would produce the following results:

Method	Entity Body Contents	Expected Result
GET	Blank	Returns the HTTP headers for and the current representation of the resource, in this case an XML document. But, does not MODIFY the resource in anyway specified from the request.
HEAD	Blank	Returns only the HTTP headers associated with this resource, but not the XML document itself
OPTIONS	Blank	Returns the list of HTTP methods this resource supports, such as GET, PUT, POST, DELETE
PUT	A complete replacement XML <entry> document	A response code indicating whether the replacement was accepted.
POST	Arbitrary content, perhaps a string of content intended to be added as a comment to the blog post	A response code indicating the result, and possibly the URI of a newly created, subordinate resource, such as a direct URI to the comment added to the entry.
DELETE	Blank	A response code indicating whether the resource was deleted.

There is much more to be said about this chapter, but I leave it to you to read it for yourself and enjoy what you learn.

2: The Hypertext Transfer Protocol and the Universal Resource Identifier

This chapter goes into more detail about the HTTP verbs and the standard response codes and header values, then discusses much more about specific HTTP and REST anti-patterns. These are, broadly:

Anti-Pattern Name	Description	Real-World Examples
GET Tunneling	Specifying method information to the server via GET, which is supposed to NOT modify resources from client-specified commands (idempotency)	Flickr used to have GET methods with commands like “delete” embedded in them. Google’s look-ahead pre-fetch tool caused HAVOC on many user’s accounts as a result!
POST Tunneling	Specifying method information inside of the POST entity body.	SOAP based RPC services are a wholesale violation of this. They embed arbitrary command names inside the envelope.
Misused Content-Types
Miusing Status Codes
Misusing Cache
Cookies
Lack of Hypermedia Support
Lack of Self-Description

 

The concept of URI and Addressability is covered thoroughly. This is so important for so many reasons to get into, but suffice it to say for Search-Engine-Optimization (googleiciousness / bingability) a URI should identify a single resource unambiguously so that search engine indexes can link to it.

Aside from this there is a wide range of computer science concepts wrapped up inside the old trusty URI.

For example, when you look at the URI: http://agilefromthegroundup.blogspot.com/feeds/posts/default/4352538909387142862, you actually have a specification that tells a user agent how to attempt fetching the resource (HTTP protocol), where to start looking for it (agilefromthegroundup.blogspot.com), where within that space to look further (/feeds/post/ etc). What you also have is a contract that is late-bound by definition. The intent of the URI is to provide, of course, a Universal Resource Identifier, which specifies NOTHING about what actually resides behind that identifier. What resides behind it is completely up to the control of the server. It can be absolutely anything. It just so happens to usually be HTML, but of course can be XML, PDF, TXT, etc.

3: Desktop Client Operations

4: Web Client Operations

5: IIS and ASP.NET Internals and Instrumentation

Recommended Resources: Becoming an Expert, Extending Agile, and Individual Improvement

There are three recent presentations posted from Agile 2009 to InfoQ.com that I highly recommend you listen to and learn from. Here are the links with the descriptions:

Mary Poppendieck: Deliberate Practices in Software Development

http://www.infoq.com/presentations/poppendieck-deliberate-practice-in-software-development

Description:

Summary
In the nature vs. nurture debate, researchers have declared nurture the winner. People who excel are the ones who work the hardest; it takes ten+ years of deliberate practice to become an expert. Deliberate practice is not about putting in hours, it’s about working to improve performance. It does not mean doing what you are good at; it means challenging yourself under the guidance of a teacher.
Bio
Mary Poppendieck started her career as a process control programmer, moved on to manage the IT department of a manufacturing plant, and then ended up in product development, where she was both a product champion and a department manager. Mary tried to retire in 1998, but instead found herself managing a government software project where she first encountered the word "waterfall".

Josh Comments:

I’ve listened to Mary’s talks at Google and read most of her Implementing Lean Software Development book recently. This talk is excellent. She discusses not just software, but also music performance and artistic talent development, citing studies that have shown it typically takes about 10,000 focused hours for musicians to truly reach the level of expert, and that many of them who begin early in life reach this number of hours by age 20! Regarding software development, 10 years of working professionally is about 20,000 working hours, but of course not all of those hours are spent crafting software. I’ve been working professionally about 10 years, and I think I’m near that level of expertise in a broadened sense, but have much, much, much more to learn in the depth direction.

 

Allistair Cockburn: I Come to Bury Agile, Not to Praise It

http://www.infoq.com/presentations/cockburn-bury-not-praise-agile

Description:

Summary
Agile came from small, colocated projects in the 1990s. It has spread to large, globally distributed commercial projects, affecting the IEEE, the PMI, the SEI and the Department of Defense. Agile now sits in a larger landscape and should be viewed accordingly. This talk shows that landscape, clarifying how classical agile fits in and what constitutes effective development outside that narrow area.

Bio
Dr. Alistair Cockburn is a world-renowned expert at what is called agile development, the early and regular delivery of business value through improved communications, fast feedback and staged delivery. Dr. Cockburn co-founded the agile development movement, co-authoring the Manifesto for Agile Software Development and the project leadership Declaration of Inter-dependence.

Josh Comments:

This is a great talk which is not really about burying agile, but about recognizing that the basic practices of agile now need to give way to ideas like Software Craftsmanship. He covers much more ground than this, but I’ll just highlight the Software Craftsmanship principles:

“As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:

Not only working software,

              but also well-crafted software

Not only responding to change,

              but also steadily adding value

Not only individuals and interactions,

              but also a community of professionals

Not only customer collaboration,

              but also productive partnerships

That is, in pursuit of the items on the left we have found the items on the right to be indispensable.”

Ashley Johnson and Amr Elssamadisy: Scaling Up by Scaling Down: A (re)Focus on Individual Skills

http://www.infoq.com/presentations/scaling-up-by-scaling-down

Description:

Summary
In this presentation, the causality between performance in the small (individuals and teams) and performance in the large is highlighted and explained. Discover what you can do as an individual regardless of your position in the hierarchy to enable higher performance software development.

Bio
Ashley Johnson splits his time between understanding the challenges that companies face, and consulting with senior IT management to facilitate organizational optimization. Author of the leading book on Agile Adoption, Amr Elssamadisy spends his time helping others learn better ways to develop software and connecting the software development process with its rightful driver - business value.

Josh Comments:

What I like most about this presentation, is how Ashley Johnson incorporates audience participation and experimentation into the course of the presentation. This is the essence of teaching and learning. During my Scrum training with Jeff Sutherland, I was impressed by how Jeff used Scrum to run the training, creating his own task wall with sticky notes which also served as a burndown mechanism. He broke us up into small groups and we worked on problems and exercises that way.

Just to highlight, at one point Ashley asks the audience to break into pairs of two and to come up with a list of “What is required for a high performing team?” This is what the participants came up with on their own:

• Shared vision that matters
• Trust between the team members
• Communication
• Passion
• Empowerment
• Having fun
• Challenging each other while holding each other accountable
• Leadership

D = V * T : The formula in software DeVelopmenT to get features DONE

There’s a hidden formula in software DeVelopmenT that tells how fast a team can get features DONE and Ready-to-Ship.

 

The formula is: D = V * T

It reads as: DONE Features = Velocity multiplied by Time

 

The importance of a software development team’s velocity

The term “velocity” as it applies to software development is simple to explain and to illustrate. Here’s my definition:

Velocity: A team’s velocity is the number of features it can get completely DONE and Ready-to-Ship during a short, fixed time period (2 to 4 weeks)

Velocity is extremely important for business owners and other project stakeholders. Without knowing the velocity of their team, they have no way to reliably plan release dates and coordinate marketing and sales teams. (2) It’s no exaggeration to say that the most important thing a professional software team can do to increase its value to an organization is to become skilled in the arts of estimation and planning. This post introduces the concepts behind  velocity measurement, and provides links for more detailed reading.

 

Are we there yet? Speed racing down the software delivery highway

Building successful software and delivering it on time is an art of numbers. It all boils down to math, like physics or accounting.

Who can forget the familiar high-school formula D = V * T? (also written as D = R * T)

This, of course, is the famous equation for calculating how far you can travel from a given starting point to another when you already know your velocity (or rate) and how long you will be travelling.

 

Distance = Velocity multiplied by Time

For example: if we have know we are traveling at 50 miles per hour and plan to travel for 3 hours, then we know we will travel 150 miles.

 

What happens though if we do not know our velocity, but instead know how far we have traveled and how much time it took to get there? Can we derive our velocity from these other two measurements? Of course we can with simple math. In this case, we have D and T, and can derive V by modifying the formula to be V = D / T.

 

Velocity = Distance divided by Time

For example: if we have traveled 120 miles after 3 hours from point A to point B, then we know our velocity per 1 single hour is 40, or 40 mph. 

 

 

Figure 1: Two US high school math students calculating how far they can travel before returning to math class  in 30 minutes or before being caught by authorities for driving on the wrong side of the road.

 

Measuring velocity in software development to decrease time-to-market and realize faster ROI

I hear what you’re screaming: enough with the PSAT math prep already, how does this apply to releasing software on time? It’s so simple you’ll kick yourself, or your team, for not doing this already.

Agile teams use a formula that works the same way. It’s calculated differently, because most software teams aren’t very mobile while coding, though it would be relaxing to code on a boat.

Because a team cannot reliably know in advance how quickly it can complete a set of features, it must use the second form of the equation to derive its velocity based upon actual observation of progress first.

Thus, the formula for calculating an agile team’s initial velocity still reads as V = D / T, except the D stands for “DONE Features” instead of distance. T, or time, usually stands for 2 to 4 weeks instead of 1 hour. For this article, we’ll assume it means 3 weeks.

 

Initial Velocity = DONE Features divided by Time

For example: if we get 6 features DONE in 3 weeks, then we know our velocity is 6 features per 3 weeks. Simplified, we’ll say 2 features per week.

Here is a simple chart depicting this velocity:

Figure 2: Velocity measurement illustration of six features becoming done during a three-week period

It’s tempting to look at this chart and say the velocity is 2 features per week, and that we can now start using the formula DONE Features = Velocity multiplied by Time to plan ahead. We will use this simplification for the purposes of this article, but keep in mind that this may or may not be true, so be careful! Here are two reasons why:

1. New Requirements Discovered: During the course of any three-week period, teams will discover new requirements frequently. The new requirements could be bugs, change requests from the business team, or important changes required to match the competition. This is a subject for an entire volume on change management!
2. Definition of DONE: It’s extremely important that a team agrees upon what qualifies as a DONE feature. Each team must define what it means by the word DONE. I leave that as an exercise for a future article, but you can find some recommended reading and listening below for reference. (3, 4)

For the rest of this post, we’ll pretend that no new requirements are discovered and we’ll define a feature as DONE if it has successfully passed through each of the following development phases:

1. Requirements Definition
2. Analysis, Design, and sufficient Documentation
3. Coding
4. Unit Testing
5. Code Review (for development standards adherence and security design assessment)
6. Refactoring (to conform to standards and address security deficiencies)
7. Functional Testing
8. User Acceptance Testing (preferably automated)
9. Performance Testing
10. Pilot (beta testing with real or proxy users)
11. Ready-to-Ship

This may sound like a lot of work! And, it certainly is a lot of work. All mission-critical projects consist of a number of features that must go through these steps before they can be considered DONE and Ready-to-Ship.

 

Pitfalls of using early travel velocity to forecast total road trip duration

Returning to our travel example, suppose we are traveling from our city to the mountains for a conference about software estimation and planning. We know the destination is 500 miles away. We also know that the interstate through our city and into the next state has a speed limit of 70 mph. A simple calculation tells us that it would take 7.14 hours to travel 500 miles at 70 mph.

What if you absolutely had to be at the meeting on time? Would you think it’s wise to turn that back-of-the-napkin estimate into a target to which you could commit?

Most people would say it’s insane to expect that would travel into the mountains at 70 mph, the same velocity as the interstate. What’s more, you’d have to take bathroom breaks and food breaks too. You agree with most people.

You decide to email the mailing list for the conference and ask if anyone has ever traveled from your city to the mountain location, and get a response complete with a chart! Your colleague says she kept track of how many miles she traveled during each hour and came up with chart in figure 3 showing that it took just over 9 hours to complete the 500 miles.

 

Figure 3: Chart showing total number of miles driven after each hour in red and number of miles driven during each hour in blue

If we round up the number of hours traveled to an even 10, we’ll just call this 50 mph. The reasons we cannot travel at 70 mph during the entire trip is simple: mountains are more curvy and dangerous, and we have to break for food and the bathroom. Only after completing the trip one time can we look back and use the experience as a way to gauge future trips through the same or similar terrain.

Let’s take a beginner's look now at how agile teams can use historical data, combined with estimation, to produce better delivery date forecasts. This will be covered in more depth in my next post.

 

Producing better software delivery date forecasts using simple, empirical estimation techniques

Similarly, if we know our total number of features is 50, and that our velocity is 2 features-per-week, then it’s tempting to calculate that it should take 25 weeks to complete our project.

Alas, software development is rarely as simple as driving down a straight interstate. Just like the journey into the mountains takes us through a variety of terrain and we must take breaks, all software development takes us through all kinds of unexpected requirements. Stakeholders request new features, markets changes, people get hired, people get fired!

And, most importantly, not all features are the same size or complexity. Because of this, agile teams need to take additional steps to bring predictability to delivery schedules. This is usually done with estimation techniques like Wideband Delphi or Planning Poker. These two techniques have been written about by Steve McConnell and Mike Cohn, respectively. (5, 6, 7)

I will cover Planning Poker in more detail in a future post, but the main idea behind it is that the entire team takes a few hours every three weeks to look ahead at the work to be done and places a relative estimate of size or complexity estimate on each item. They then measure how quickly they can complete each item. So instead our simple count of “50 features”, the team might actually have a number such as 150 “points”, which means that, on average, each feature is roughly 3 points of estimated size or complexity. For now, however, let’s continue to focus on tracking how fast the team moves through 50 features.

Agile teams typically use a chart that is drawn from the top down towards zero, which indicates zero more features outstanding! This is called a burndown chart, and a realistic chart might look as follows in figure 4:

 

Figure 4: Hypothetical burndown chart illustrating how the amount of actual work, in blue, fluctuates up and down as the total number of UNDONE features approaches zero. The initial estimate of 50 features and the target velocity of burning down 2 features per week is shown in red

This chart shows that the team had 50 features remaining to implement at the start of week 0. The initial target velocity of 2 features per week holds up for a few weeks, shown in red, but then it falls off a bit before speeding up to be faster than 2 per week. Not to be outdone, perhaps the business team feels the team can do more work, and new features get added. This causes the time between week 11 and 24 to remain relatively flat before the velocity picks up again.

By the time the initial 50 features are completed, we can calculate that they burned down at a rate of about 1.5 per week. Now, this simple chart does not actually show how many features were added during the duration of the project, though it’s obvious when you see the blue spikes. There are more sophisticated charts that can help illustrate this, but I’ll leave that for next time.

In the meantime, please visit the suggested resources, starting with Mike Cohn’s excellent presentation about Agile Estimation and Planning, to learn more. (1)

Until next time, stay agile not fragile.

 

References and Resources

 

1. “Introduction to Agile Estimation and Planning” – by Mike Cohn, PDF presentation about release planning with agile estimation and planning techniques: http://www.mountaingoatsoftware.com/presentations/106-introduction-to-agile-estimating-and-planning
2. “Nokia Test : Where did it come from?” – by Jeff Sutherland, about how Nokia uses velocity tracking to assess their teams’ productivity and likelihood to generate future ROI: http://jeffsutherland.com/scrum/2008/08/nokia-test-where-did-it-come-from.html
3. “How Do We Know When We Are Done?” – by Mitch Lacey, about how his team defined DONE with the whole team’s participation: http://www.scrumalliance.org/articles/107-how-do-we-know-when-we-are-done
4. “Scrum, et al” – by Ken Schwaber about the history of Scrum, presented at Google : http://www.youtube.com/watch?v=IyNPeTn8fpo
5. Software Estimation: Demystifying the Black Art – by Steve McConnell, book about lessons learned and best practices for software estimation: http://www.amazon.com/Software-Estimation-Demystifying-Practices-Microsoft/dp/0735605351
6. Agile Estimation and Planning – by Mike Cohn, book about how to perform agile estimation and planning using simple estimation techniques and short, fixed time-boxed development iterations: http://www.amazon.com/Agile-Estimating-Planning-Mike-Cohn/dp/0131479415
7. ATL ALT.NET Meetup recorded conversation about Agile Estimation and Planning: http://www.meetup.com/AtlAltDotNet/calendar/9525107/?eventId=9525107&action=detail (direct MP3 link: http://apps.ultravioletconsulting.com/audio/ATLAltDotNet/ATLAltDotNet-2009-01-27-AgileEstimationAndPlanningDiscussion.mp3)