Agile From The Ground Up

Microsoft Moles!

2011-06-27T19:52:00.001-07:00

This is a very cool project from Microsoft Research:

http://research.microsoft.com/en-us/projects/moles/

I'm able to use this to "fake" or "mock" sealed classes inside of the ASP.NET runtime.

For example:

        [TestMethod]
        [HostType("Moles")]
        public void WhenCannotInterpretSdnUserKeyAsIntegerThenMustRedirectToGlobalErrorPageWithProperMessage()
        {
            // Arrange
            var cookies = new HttpCookieCollection { new HttpCookie(SdnUserKeyCookieName, "Gibberish") };
            var context = new MHttpContext();
            var request = new MHttpRequest();
            var response = new MHttpResponse();
            var redirectWasCalled = false;
            var redirectedToLocation = string.Empty;
            var responseEnded = false;
            response.RedirectStringBoolean = (string location, bool endResponse) =>
            {
                redirectWasCalled = true;
                redirectedToLocation = location;
                responseEnded = endResponse;
            };
            MHttpContext.CurrentGet = () => context;
            context.RequestGet = () => request;
            context.ResponseGet = () => response;
            request.CookiesGet = () => cookies;

            // Act
            _sdnAuthenticator.Process(context);

            // Assert
            Assert.IsTrue(redirectWasCalled);
            Assert.AreEqual("/Error.aspx", redirectedToLocation, true);
            Assert.IsTrue(responseEnded);
        }

How cool is that? The moles are implemented as "detours" and replace components of the runtime when configured to do so.

This is great stuff.

From 15 Minute Stand Ups to Standing Work Stations : How to Start a Trend

2011-06-23T07:27:00.001-07:00

As many people involved with the various agile development practices know, one of the common practices is a brief "Daily Stand Up" meeting to discuss project progress, priority changes, and impediments.

When I joined my most recent team, the Epi-X program at CDC, I wanted to try a different kind of Stand Up. This time, I raised my monitors and my keyboard and I began standing for a large portion of the day to do my work. I don't claim any medical expertise, despite working at CDC, so don't take my counsel on this as anything scientific.

All I can say is that there have been some studies written about in popular articles that seem to indicate constant sitting is detrimental to long-term health, including increased risks of obesity and heart disease. I should also say other studies say there are health risks with constant standing as well!

So, for me, it is not that I stand at my workstation all day long. I am in various sit-down meetings and discussions, and have to walk to different areas to speak with people. And, I do lower my monitor and sit from time to time as well.

I'll continue this post later, but in the past week two people in my office have followed suit! So far, we just use boxes to prop up our equipment, but I'm strongly considering investing in a genuine table-top adjustable desk from http://www.ergodesktop.com.

One coworker that adopted this practice also bought himself a foot mat. I have been wearing sandals or occasionally kneeling on my chair.

Remember: Stay agile, not fragile.

MIX 2011 Presentations Reviews

2011-05-20T19:21:00.001-07:00

Glenn Block's presentation on WCF and URIs is very good from MIX 2011. Mike Simpson also forwarded me a recent DotNetRocks episode in which Block discusses the WCF HTTP WebAPI.

MIX 2011 presentation: "There's a URI for That": http://channel9.msdn.com/events/MIX/MIX11/FRM14

DNR episode: "Glenn Block Simplifies WCF with WebAPI": http://www.dotnetrocks.com/default.aspx?showNum=661

In the MIX 2011 presentation, some highlights of things he demonstrates the following:

Microsoft is committed to delivering a first-class HTTP programming model for WCF
The HttpWebResponse<OfTypeT> response type, which has full support of HTTP status codes, demonstarted with response.StatusCode = HttpStatusCode.Created to indicate a successful "resource created" response status from the server to the client.
The use of media types and registering X number of media type processors to handle incoming requests based on "extensions" supplied on the URI.

Gone are the days when an extension like ".aspx" or ".txt" were tightly-coupled to the file-system and to physical files on disk. Instead, these are now fully interceptable and processable by YOUR handler code in the way YOU WANT.

Using an OData producer resource, he showed Google's GMail contacts import dialog pulling down contacts from his WCF service in vCard format, as specified by an extension, and an OData filter expression that limited the results returned to the top 3 results

All in all, I am feeling increasinly confident that Microsoft's direction regarding HTTP and REST is moving in a solid path. While it is of course possible to build REST/HTTP style Web APIs without WCF and without explicit support from Microsoft, the fact that Microsoft is supporting this provides two advantages:

It increases the mind-share and desire amongst .NET developers to understand the web and to leverage these technologies,
It provides an easier path toward enterprise-wide adoption because mangement will begin to understand the benefits and see the backing of MS within their flagship WCF offerring.

Perhaps "hard-core" ALT.NETters and "Restafarians" will say they've been doing things like this without MS support for years, and I would not argue with that. However, most of us work in a heterogeneous world that involves a lot of layers of management and risk-mitigation. So, the extra support at an official level from Microsoft's platform can only help the general adoption curve of web technology.

And, regarding developers and architects, these groups of stakeholders have a lot riding on their technical choices and often face an uphill battle when the major vendor of their company's tools isn't yet "on board".

I and a few of my colleagues at various companies have been building REST-style (though I'd hesitate to call it full REST because of little attention paid to links and hypermedia constraints) for a efw years and have often faced skepticism because of the "That's not what WCF does" responses. Those responses were well-founded at the time, but it's also very nice to see how far MS has come with modernizing WCF to fully support the web and the HTTP specifications for all they offer and are worth.

For me this is a great step in the right direction, and I look forward to evaluating further the use of WCF HTTP WebAPI for backend resources / services.

MIX 2011: WCF, OData, MVC, and MEF Highlighted Presentations

2011-04-16T08:53:00.001-07:00

MIX 11 has finished. Here are the presentations that I will be "diving into" more quickly than others.

My Summary of Summaries:

The topics below caught my eye as priority because:

They build upon OData, and thus upon Atom and REST . REST is the foundational architecture of the WWW, and is still not widely understand throughout the entire development industry

I'm likely to be joining a large project that will be providing enterprise wide alerting, notification, and reporting capabilities to X number of organizational sub-units. It's critical that interfaces to such services be simple, document-based, and that output data be consumable by end-user tools like Excel. OData support is "baked in" to the latest Office products and into Sharepoint 2010.

Regarding MVC and MEF: these two technologies are critical for modular, extensible web applications on the .NET platform. Being able to deploy sub-units independent of an independent application architecture is critical both for ease of maintenance and extensibility, but is also important in highly secure environments that require rigorous application scanning for security threats.

That's more than enough from me. I'll let the experts Castro and Block elaborate :)

Scott Guthrie's key-note, naturally:

http://channel9.msdn.com/Events/MIX/MIX11/KEY01

Why: because it's The Gu. Period. Full-stop.

Pablo Castro on OData Roadmap: Powering the Next Generation of Services:

http://channel9.msdn.com/Events/MIX/MIX11/FRM11

Summary:

At home and work, the way we experience the web (share, search and interact with data) is undergoing an industry-changing paradigm shift from "the web of documents" to the "web of data" which enables new data-driven experiences to be easily created for any platform or device. Come to this session to see how OData is helping to enable this shift through a hands-on look at the near term roadmap for the Open Data Protocol and see how it will enable a new set of user experiences. From support for offline applications, to hypermedia-driven UI and much more, join us in this session to see how OData is evolving based on your feedback to enable creating immersive user experiences for any device.

OData Roadmap: Exposing any Data Source as an OData Service: http://channel9.msdn.com/Events/MIX/MIX11/FRM16

Summary:

Many of the popular OData services, including Netflix, Twitpic and Facebook Insights were built by reusing their existing web API with an OData service. Implementing this type of OData service is not simple but it's also not as hard as you might think. In this session, you'll learn how to build similar services that wrap different types of data sources using the WCF Data Services Toolkit. We'll take a look at the implementations for several of the popular services as examples of how to use the toolkit to create new OData services.

Fun with MVC 3 and MEF: http://channel9.msdn.com/Events/MIX/MIX11/OPN07

Summary:

So you have a team of developers… And a nice architecture to build on… How about making that architecture easy for everyone and getting developers up to speed quickly? Learn all about integrating the managed extensibility framework and ASP.NET MVC for creating loosely coupled, easy to use architectures that anyone can grasp.

OData in Action: Connecting Any Data Source to Any Device

http://channel9.msdn.com/Events/MIX/MIX11/FRM10

Summary:

We are collecting more diverse data than ever before and at the same time undergoing a proliferation of connected devices ranging from phone to the desktop, each with its own requirements. This can pose a significant barrier to developers looking to create great end-to-end user experiences across devices. The OData protocol (http://odata.org) was created to provide a common way to expose and interact with data on any platform (DB, No SQL stores, web services, etc). In this code heavy session we'll show you how Netflix, EBay and others have used OData and Azure to quickly build secure, internet-scale services that power immersive client experiences from rich cross platform mobile applications to insightful BI reports.

Glen Block on WCF Web Apis: There's an URI for That: http://channel9.msdn.com/Events/MIX/MIX11/FRM14

Summary:

Web application developers today are facing new challenges around how to expose data and services. The cloud, move to devices, and shift toward browsers are all placing increasing demands on surfacing such functionality in a web-friendly manner. WCF's Web API makes it easy for developers to expose their services and data to a broad set of clients and to take advantage of rich emerging web standards like Web Sockets.

Delivery and Simplicity : Don't Leave Home Without These Agile Principles

2011-04-05T22:58:00.001-07:00

Bootstrapping Agile from the Trenches

In February of 2006, I was offered the position of Lead Architect for the redevelopment of CDC's Epi-X system, CDC's flagship secure communications platform for emergent disease outbreak notification and bi-directional collaboration between multi-jurisdictional public health authorities. However, on the same day I was offered a position at a private .NET consulting company, Abel Solutions. Realizing that actual redevelopment of Epi-X would be months, if not years away, due to the then very disruptive agency-wide reorganization, I decided to leave so that I could gain more experience in a variety of private sector industries.

During the five years since I left Epi-X, I've worked as a senior software engineer, architect, lead application architect, and as an independent consultant. My first assignment with Abel Solutions was to re-architect and re-develop a very popular web-based electronic commerce & auction system to support more than 1 million registered users and the processing of more than 300 million dollars in annual sales. For a different company, I re-engineered the security, object-relational, and querying architecture of a complicated human resources & payroll processing system used by thousands of companies. Most recently, I helped lead the design and development of both a modular user-interface architecture and the core service-oriented architecture for a new correspondence banking & ACH settlement platform to be used by hundreds of local and regional banks to conduct business more easily with the Federal Reserve and each other.

For the companies sponsoring the first two projects mentioned above, I introduced and lead the successful adoption of Agile management and development practices. For the third, I was recruited specifically to consult both on their adoption of Agile and the design of its new system's user-interface and service-oriented architecture.

I've also consulted with many other private entrepreneurial businesses about technology strategy, and in 2008 founded both the Atlanta Science Tavern and the ATL ALT.NET community groups.

Aligning the Agile Approach to the Business Domain

Let me be the first to state that adopting Agile in the "real world" is not easy. To be successful, you must internalize the values of Agile, especially the very first one which reads:

Our highest priority is to satisfy the customer

through early and continuous delivery

of valuable software.

Did you notice that this specifies nothing whatsoever about writing code? It specifies nothing at all about writing code at all. It specifies delivery of valuable software.

Later on in the principles document, it says:

Simplicity--the art of maximizing the amount

of work not done--is essential.

It says simplicity is essential, not optional, but essential. How many projects have you seen that feature unnecessary complexity? That is the exact opposite of this Agile principle. For more about this problem, see my post that reviews a Skype architect's presentation.

You can read the rest of the Agile principles here: http://agilemanifesto.org/principles.html

I highlight this because a lot of practitioners think that Agile is some kind of magic bullet that will solve all the problems that sequential "waterfall" style development has. This is absolutely not the case. Agile has its own pitfalls that must be addressed as well, and one of them is plainly that development teams don't even understand or truly believe in these two core principles!

The Core of Agile: Communication and Collaboration

As the principles in the Agile Manifesto explain, collaboration and communication are the two most critical underlying themes of agile development. What if, by communicating with your clients successfully you could help them avoid spending millions of dollars custom-developing a solution to a problem that you could solve using low-cost or open-source software?

Would that not be the ultimate fulfillment of the first principle of Agile? I think it most certainly would. And, it would certainly fulfill the later example I highlighted!

Unfortunately, many people, even managers, fail to think this way when they adopt Agile. This is not to say that they don't mean well. It's often just the case that they recognize Agile, and associated development practices like XP and TDD, as a better way of building software, but can lose sight of principle number one: delivery of valuable software.

Internalized Agility = Flexibility

True internalization of Agile values should cause architects, developers, testers, and all manner of managers to adopt an attitude of true collaboration with their stakeholders.

So, keep it mind that being agile doesn't always mean building software. First and foremost, it means delivering valuable software.

New Reading List: Acceptance Testing, Specification by Example

2011-03-29T22:38:00.001-07:00

As usual, there are far, far too many topics that interest me than I will likely be able to comprehend.

I'm extremely interested in recent presentations and work from Gojko Adzic.

I'm going to buy his books Briding the Communication Gap and Specification by Example:

http://www.acceptancetesting.info/the-book/

http://www.manning.com/adzic/

You can see Gojko present about these topics in numerous places, including:

http://www.acceptancetesting.info

http://www.mefeedia.com/watch/32039109

http://skillsmatter.com/podcast/design-architecture/ddd-tdd-bdd

Why do these topics interest me? The easy answer is that I find it very painful, both mentally, and physically to endure lapses of communication in projects that lead to lost time, money, or functionality when I know in my heart such problems can be avoided with proper communication.

Scott Ambler has also written extensively about Executable Specifications at http://www.agilemodeling.com/essays/executableSpecifications.htm.

These ideas bring together the two aspects of system development that matter most to me:

Achieving the correct result for my customer / user
Seeing something valuable running

Nothing is more crucial to my sense of accomplishment when building a system than to see the correct result in action. I know, however, that many teams value "documentation" as a very important communication artifact that serves to mediate between "the business" and "the development team", but I find this to be a source of constant frustration for the very reasons that Gojko lays out. Instead, what is more valuable, and very satisfying, is to execute the documentation, the specification, the requirements in a testable, verifiable way that itself represents real tangible value, not just words on a dead sheet of paper.

Gojko's presentations are excellent and his books look like they will really fill the "gap" in my own arsenal. I am looking forward to devoting significant time to studying these works.

As he explains on his web site, here are the "Key Ideas" in the Bridging the Communication Gap book:

Strangulation: The Pattern of Choice for Misk Mitigating, ROI-Maximizing Agilists When Rewriting Legacy Systems

2011-03-28T21:49:00.000-07:00

"The most important reason to consider a strangler application over a cut-over rewrite is reduced risk. A strangler can give value steadily and the frequent releases allow you to monitor its progress more carefully. Many people still don't consider a strangler since they think it will cost more - I'm not convinced about that. Since you can use shorter release cycles with a strangler you can avoid a lot of the unnecessary features that cut over rewrites often generate." -- Martin Fowler, Chief Scientist, ThoughtWorks, on The Strangler Pattern

When rewriting a system in a new technology, it's tempting to think that the task will be easier and quicker than the first time it was written. Because of this, sometimes business sponsors believe that a "waterfall" or "big bang all at once" approach will work out, but this is rarely the case for any project large enough and important enough to warrant rewriting. It's always important to practice iterative and incremental development to provide for feedback loops. But, it's even more important to do this in the case of a large application rewrite. This article will explain why this is true. There are a few bedrock development principles that project sponsors and team members should put into practice to ensure the success of large scale migrations. Having learned these lessons from experience, these are:

Involve business sponsors and end-users directly (or a user-experience specialist) and the entire support and operations teams during the entire rewrite
Involve permanent quality-assurance professionals from the beginning and during the entire rewrite
Design, code, and test one complete feature rewritten from the existing system as quickly as possible
Thereafter, design, code, test, and pilot user-valued, return-on-investment-generating (ROI) features in small increments
Most importantly, continuously build team member skills, knowledge, and leadership abilities

Lessons Learned in Rewriting Large Legacy Systems

In February of 2006 I joined a small .NET consulting company. Shortly thereafter I was assigned to a brand new project for one of their clients to analyze, design, and develop a new version of an existing electronic commerce platform. The system was a highly successful, niche-market leading auction site with nearly 700,000 registered users at the time. In operation for more than seven years by then, the system was built on classic ASP, C++/COM, and SQL Server 2000. It consisted of about 330 ASP pages. Our client wanted to do two primary things. First, he wanted to add new, value-added features to the system to provide a much better user experience, one that would be similar to eBay. These features would be called "My Auctions". This new set of features would take the place of roughly 30 pages from the existing web site. Second, he wanted to migrate the other 300 pages, without introducing any functionality or usability improvements, to ASP.NET WebForms. Having already personally designed and developed the entire business object back-end COM objects, he wanted all of the new web site to reuse this investment by utilizing COM Interop.

My Recommendation: Perform a Phased, Vertical Migration One Piece at a Time

My first assignment was to analyze the existing ASP and C++ code to and produce a migration strategy recommendation. This strategy document would lay out our company's professional opinion for migrating the system to the .NET platform and the C# language. My recommendation was for our client to perform a vertical migration, which is a migration that incorporates an entire functional slice of a subset of the system (My Auctions) which cuts across all architectural layers (top-to-bottom). In their book The Pragmatic Programmer, Dave Thomas and Andrew Hunt call this a "tracer bullet". This was, in fact, what Microsoft recommended in their best practices guidance documents that I researched about performing large scale system architecture migrations. I recommended that our client hire us to build a new core platform on ASP.NET with C# and get the new, value-added features to market as soon as possible on top of that core platform. Only after these value-added features were in production would we then move on to replacing the rest of the 300 pages with ASP.NET replacements.

My Reasoning: Place Customer Satisfaction, ROI, and Risk Mitigation First

My reasoning was that by creating a new core platform and building the brand new, usability-focused, value-added My Auctions features on top of that, our client would generate a return-on-investment (ROI) much sooner by generating more sales volume with the user-friendly features and would simultaneously mitigate significant risk by testing the viability of the COM Interop strategy. By virtue of the features being value-added, there would be no risk whatsoever for him to deploy them to a parallel web server and get his users to begin pilot testing the system and providing valuable feedback early on in the game when he could still make significant changes prior to committing to replacing the entire system with the new technology.

I've since learned from Dan North at QCon 2010 in S.F. that is called The Strangler Pattern according to Martin Fowler, hence the title of this post!

Client Decision: Let's Do It All At Once

Our client considered my recommendation very carefully, but wanted to take a different approach. Rather than deploy the new My Auctions features independently, side-by-side with the existing system, he wanted to have his in-house staff work on the other 300 pages while our company worked on the value-added features. With more than 330 pages to complete, I estimated that the project would not take less than a year, but would more likely take two years or more to complete. Our client and my manager thought things could be done much faster if we had three or four people working on the system. This was certainly the case early on when I worked side-by-side with another developer in our company. Within four months, he and I had completed the new C# application foundation and the value-added features to the point that they were ready for beta-testing.

And that's when all the fun began!

Planning is Essential; Plans are Useless

As anyone who has worked in the software industry for a number of years knows, the best laid plans never go as you planned. Our client's lead C# developer left his company. Soon after that, my manager at my company was let go, but several months later he was hired by our client to take over the development management of the project. This made sense since he already had strong background on the project since its inception. Shortly after this, our client's HTML, graphics, and CSS developer quit when asked to change his focus to become an ASP.NET developer. They hired a lead C# developer and he got to working on a large slice of the application while I continued to work on another large slice. Five months later, they hired a second C# developer and he began working on several other slices of the application.

Wanting to see the project through to success, I joined the client as a direct employee to continue being the lead architect for the project.

Naturally, There's a Big Trade Show In The Story

What would any development story be without a "Big Trade Show" lurking around the corner? As luck and fate would have it, in early 2008 there was a huge industry trade show, and it was critical that we would be able to demonstrate the new version of the system to the roughly 25,000 customers that would be passing through our booth. And, it would be very important that these customers be able to see their own real items, either ones they were selling or ones they were buying. The problem was, of course, that the system was not ready to replace the production system! Due to security requirements brought about by a changing legal environment, we had to repartition the back-end database for the new system from 2 SQL databases into 6 separate databases just before the trade show. It was deemed too risky to perform this radical "surgery" on the live, production system just two months before the trade show. The new system's schema was about 95% the same as the old system, but there were corrections to long-standing column name problems or foreign-key reference inconsistencies. This was a complicating factor, however, for migration.

We tossed around various ideas, such as:

Perform the "surgery" on the production database to upgrade it to the new system schema, then use views and synonyms to create a "pass-through" database that looked like the old schema, but mapped across to the new DBs and structures.
Do the reverse: create several "pass-through" new databases with views and synonyms that actually resolved to the single existing production database's objects.

We felt that we could mitigate risk entirely by following the second option. What this also allowed us to do was to "override" some of the production system's tables with configuration data specific to the new system. The approach of using synonyms and views ensured that all writes and reads against the pass-through objects would actually resolve into the production database, thus enabling the beta version of the new system to live side-by-side with the legacy system.

The War Room

After some proof-of-concept prototyping, we realized this would be a winning strategy. Over the next couple of weeks, the four of us on the development team gathered daily in our "war room", and worked together to create all the necessary SQL scripts and shell databases, synonyms, views, etc that would be the magic glue. We ensured that we could re-run the scripts at will and automated our quality-control checks and sanity checks to be certain that all mappings would have proper permissions and configurations. After enough practice runs, we felt confident that it was ready to go. We created a single zip file which contained 5 BAK files, and a T-SQL script. We handed them off to our lead database administrator and he ran the scripts. Everything worked just as planned!

Cha-Ching!

At the trade show, everything went off flawlessly! Customers attended our booth and we, the development team, aided them directly in logging into the system and showcasing the new features we had worked so hard to develop. It was a very gratifying feeling to see how our improvised plan came together so well. Most importantly, we had succeeded in mitigating all risks to the money-generating production system, while also achieving the benefit of showcasing the new system to customers with real data. This was very exciting to them because they felt that the new features would greatly help them run their own businesses atop our platform.

Phased Transition From Legacy to New

We had now successfully demonstrated and validated the new, value-added features directly with customers in person. This was a great success. Yet, there was still much to do after the trade show. Features of lesser prominence, those in the 300 other pages set still needed to be developed and tested. This ended up taking a very long time, but we ultimately cycled back to my original recommendation by adopting an incremental replacement strategy.

It worked like this:

We deployed the new system to a new web server, named v2.
The existing, v1 site, remained at www.
We provided a link from v1 to v2 in the header of the v1 site, including advertising the benefits of the new system, but also including disclaimers and calls for assistance in testing and validating the usability of the new system.
This garnered a lot of early-adopters who helped find bugs and inconsistencies, all for free to us!
We monitored the usage patterns of v2 versus v1, to help estimate the load capacity under real-world conditions.

Michael Nygard's book "Release It!" proved prophetic here. In his book he says that "feature complete" is not the same as "production ready."
We learned this because the COM code had to be completely replaced with pure C# code since it could not stand up under load using COM Interop.

This result bore out my original advice to get the new features into production as soon as possible to monitor under real world conditions.

We formally adopted Scrum and Agile practices by identifying business-driven priorities and working through them in sprints.

We did this by closely monitoring the real-world usage of both the existing v1 system and the v2 system and focusing our effort first on the highest traffic pages, such as Viewing, Browsing, and Searching. Of course, Bidding and Payment, while producing less volume, were also mission-critical.
This focus allowed us to prioritize properly. We did not place inordinate emphasis on automating the testing of all areas of the system.

For example: we did not write Selenium test suites for things like Help Pages or Support Pages. Why? They are seldom used! And, they generate no revenue.

Instead, we built comprehensive Selenium test suites for the Big Four: Viewing, Browsing & Searching, Bidding, and Payment.

A Pleasant Surprise!

With the site now operating both in legacy, classic mode at www, and in "beta" mode at v2, the team began to actively monitor the new system's health health and encourage more and more users to jump into using v2. And, because we had focused on developing the value-added My Auctions features in the very beginning of the project, those features sat ready and willing to get into production! Our newest member of the team, who joined about two years after those features were ready and "shelved", took it on his own initiative, to our delight, to start building a mobile version of the core My Auctions features using ASP.NET MVC and the business objects that supported the My Auctions features. He was reluctant to show this prototype to the "higher ups", but the rest of our team encouraged him to do so. Within a few months, his mobile application was released into production before the global "switchover", described below, to the new system. A job well done!

Switching Over Right on Time for a Cool Billion Dollars

Over the course of more than a year, the team monitored the usage of v1 and v2, and began to more aggressively push the late adopters and stragglers into the new system. Eventually a "switchover" was made, and the v2 system took over the place of www. At that point, there was now a link back to v1, which ran from a virtual machine. Several months after this, the VM was retired, and the v1 system, and all of its legacy COM, was no more.

Just after the legacy system was retired for good, the company celebrated its 10th anniversary and 1 billion dollars in sales volume!

Retrospective

In retrospect, I spent nearly three years working on this project and learned a great deal! While I wish that the original plan of seeing the entire migration take place "all at once" could have been successful, I also am pleased that my original recommendation to take a phased, incremental, risk-mitigating, ROI-maximizing approach was very sound. Ultimately, that very approach became necessary due to the "expected" unexpected bumps along the road!

Application to Domains Seeking Non-Financial Returns

I understand that not all projects involve financial reward goals. Before I began working on the project just described, I worked for four years at the US Centers for Disease Control and Prevention. While working there, we were not seeking to generate financial return-on-investment. However, we did seek returns in the form of utility and value to the users and stakeholders of our systems. To assess this properly, it was critical to either observe the real users working with the system or to sit down with them and experience their pain, frustration, and sometimes: delight! Our team did this regularly by conducting evaluations, performing proficiency testing, and through coordinated multi-agency and stakeholder exercises under simulated public health emergency "war games".

Tying This All Back to Agile

While I've written more extensively on Agile in other posts on this blog, this post has not been about the "mechanics" of agile so much as it has been about the why. But, I want to look at just the first principle of the Agile Manifesto and make a brief comment:

Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.

One might look at this principle and ask how can this be done when facing the situation I originally faced that featured my own client asking not for a continuous delivery model, but a "big bang" model? That's a very good question, and it's not one that has any quick-fix answer. My best advice here is that you need to learn the language and goals of both your client and your client's ultimate customers. If your client values financial returns, then ask him or her exactly what it is that generates financial returns.

In my client's case, returns come in when more people purchase items through his system. The next question should be: what is the shortest path we can take to increase that rate? If the client answers that the shortest path is to rewrite the entire system and deploy a big-bang upgrade, then you're going to have to keep breaking that down into smaller and smaller value-added chunks. You might have to suggest straw-men in terms of business-value if your client will not prioritize by business value naturally. Ultimately, like in this story, reality may bear down on the situation and if you have done your best to incrementally develop the system in terms of business value, then you can deliver value, upholding your end of the deal to the utmost of your ability within your realm of control. Sometimes that's the best you can do, until you run your own show!

Thoughts on www.MVCConf.com : Excellent

2011-03-16T23:36:00.001-07:00

I started listening to some of the videos at http://www.MVCConf.com today, including Scott Guthrie's keynote and Glen Block's lecture about WCF's super-enhanced REST support. Glen now calls himself a "REST head". Great!

The development of NuGet and Openwrap is really welcome to me. What must be 14 to 15 years ago now, when I started using PERL, I was very impressed by how well-oiled the machinery of CPAN was for installing packages and modules from the command line, even under Windows. When I started developing C# applications in 2001, I was very disappointed by the lack of a cohesive community of open-source packages available for Microsoft .NET. There have long been open-source projects. But, a package is different from a project. A package is shippable, deployable, consumable. I'm really happy to see this kind of thing coming into the .NET world.

I don't know where the credit goes, I'm sure it goes to many people. But, I definitely think a lot goes to Scott Guthrie for the enhanced openness that Microsoft has demonstrated in past years. I know from reading that this story also ties to Shawn Walker, the creator of DotNetNuke for his advanced forays into open-source based upon .NET.

Yes, I know the "entire stack" is not open-source, and that doesn't bother me that much.

Aligning Business Requirements With Test-Driven-Development: Behavior-Driven-Development

2010-09-05T00:13:00.001-07:00

Test-Driven-Development (TDD) is a topic that has become ubiquitous within software development over the past ten years or so. What is somewhat lesser known at this point is a particular evolution of TDD into what Dan North and Dave Astels have dubbed Behavior-Driven-Development (BDD). There are some specific problems with TDD that Dan and Dave have sought to correct by attempting to popularize BDD. This article will discuss some of the problems as well as some of the remedies.

While the stated goals of both TDD and BDD are similar, there are thought-process differences that should be understood by both technical developers and business people, including business analysts, project managers, product owners, and even CEOs. While one can philosophically argue that TDD “done right” is simply BDD anyway, I’ll trust North and Astels when they confidently state that the language people use is powerful and important and that is why they prefer to practice and speak of BDD now after years of experience.

Article Goal: Explain Why Businesses and Developers Might Want to Try BDD

This article will explain the distinctions between TDD and BDD and will show how using the BDD approach, as designed, should lead to:

Reduced costs due to higher fidelity specifications and communications
Reduced timelines due to less requirements churn and closer business and development alignment
Increased productivity due to the executable nature of BDD specifications

Test-Driven-Development

As a working definition of Test-Driven-Development from http://en.wikipedia.org/wiki/Test-driven_development is:

“Test-Driven-Development (TDD) is a software development process that relies on the repetition of a very short development cycle: first the developer writes a failing automated test case that defines a desired improvement or new function, then produces code to pass that test and finally refactors the new code to acceptable standards. Kent Beck, who is credited with having developed or 'rediscovered' the technique, stated in 2003 that TDD encourages simple designs and inspires confidence.

Test-driven development is related to the test-first programming concepts of extreme programming, begun in 1999,[2] but more recently has created more general interest in its own right.”

Scott Ambler also provides a great summary here: http://www.agiledata.org/essays/tdd.html.

Test-Driven-Development Workflow Diagram

The mantra of Red, Green, Refactor is well-known by developers. It looks like this:

Why and When To Practice Test-Driven-Development

It’s important to look at the complete historical, and current context, of Test-Driven-Development to understand its most-appropriate, and less appropriate usage.

Kent Beck, mentioned above as the most prominent formalizer of the practice, states this in his blog post “Where, Oh Where to Test?” at http://www.threeriversinstitute.org/WhereToTest.html:

“Cost, Stability, Reliability
On reflection, I realized that level of abstraction was only a coincidental factor in deciding where to test. I identified three factors which influence where to write tests, factors which sometimes line up with level of abstraction, but sometimes not. These factors are cost, stability, and reliability.

Cost is an important determiner of where to test, because tests are not essential to delivering value. The customer wants the system to run, now and after changes, and if I could achieve that without writing a single test they wouldn't mind a bit. The testing strategy that delivers the needed confidence at the least cost should win.

Cost is measured both by the developer time required to write the tests and the time to execute them (which also boils down to developer time). Effective software design can reduce the cost of writing the tests by minimizing the surface area of the system. However, setting up a test of a whole system is likely to take longer than setting up a test of a single object, both in terms of developer time and CPU time.”

More recently, he clarified the historical context of TDD with his post “To Test or Not to Test? That’s a Good Question”: http://www.threeriversinstitute.org/blog/?p=187

“Turns out the eternal verities of software development are neither eternal nor verities. I’m speaking in this case of the role of tests.

Once upon a time tests were seen as someone else’s job (speaking from a programmer’s perspective). Along came XP and said no, tests are everybody’s job, continuously. Then a cult of dogmatism sprang up around testing–if you can conceivably write a test you must.

By insisting that I always write tests I learned that I can test pretty much anything given enough time. I learned that tests can be incredibly valuable technically, psychologically, socially, and economically. However, until recently there was an underlying assumption to my strategy that I wasn’t really clear about.”

In this post, Kent goes on to state this about a product he developed called JUnit Max:

“When I started JUnit Max it slowly dawned on me that the rules had changed. The killer question was (is), “What features will attract paying customers?” By definition this is an unanswered question. If JUnit (or any other free-as-in-beer package) implements a feature, no one will pay for it in Max.

Success in JUnit Max is defined by bootstrap revenue: more paying users, more revenue per users, and/or a higher viral coefficient. Since, per definition, the means to achieve success are unknown, what maximizes the chance for success is trying lots of experiments and incorporating feedback from actual use and adoption.”

He later continues:

“When I started Max I didn’t have any automated tests for the first month. I did all of my testing manually. After I got the first few subscribers I went back and wrote tests for the existing functionality. Again, I think this sequence maximized the number of validated experiments I could perform per unit time. With little or no code, no tests let me start faster (the first test I wrote took me almost a week). Once the first bit of code was proved valuable (in the sense that a few of my friends would pay for it), tests let me experiment quickly with that code with confidence.

Whether or not to write automated tests requires balancing a range of factors. Even in Max I write a fair number of tests. If I can think of a cheap way to write a test, I develop every feature acceptance-test-first. Especially if I am not sure how to implement the feature, writing a test gives me good ideas. When working on Max, the question of whether or not to write a test boils down to whether a test helps me validate more experiments per unit time. It does, I write it. If not, damn the torpedoes. I am trying to maximize the chance that I’ll achieve wheels-up revenue for Max. The reasoning around design investment is similarly complicated, but again that’s the topic for a future post.

Some day Max will be a long game project, with a clear scope and sustainable revenue. Maintaining flexibility while simultaneously reducing costs will take over as goals. Days invested in one test will pay off. Until then, I need to remember to play the short game.”

Analysis of Kent Beck’s Cost-Benefit Based Strategy

If you read Kent’s posts carefully you will see that he made careful cost-benefit analyses whenever he decided to use or not use a TDD approach. When he developed JUnit Max, he chose to get the product features to the market as soon as he could, and then once he had an idea what customers wanted, he stabilized many of those features with tests.

So, what motivated Kent was a practical, reasonable, and important concern: financial viability and return-on-investment.

I thik it’s very important to be fully aware of these factors when deciding when and where to use TDD in any project. One of the main drivers of agile or Lean thinking is to optimize the whole, not necessarily individual parts all at once. Because of this, one must think about the larger context of a project.

Financial Investment Necessitates a Nuanced Approach to Development Practices

When thinking about the larger context, it’s easy for a developer to say something like: “We need this code to be well-factored, test-driven, and easy to maintain.” While this absolutely should be the goal of a developer, developers must also be cognizant of the larger context. Almost always, that larger context is that there is a financial investment in the project and the project’s success. And, overwhelmingly more often than not, there is a specific timeline involved. Thus, it cannot ever be solely the developer’s responsibility to set the acceptance criteria for a system’s completion or its acceptable level of quality. Developers and project managers must present risks and tradeoffs to business stakeholders and collaborate to deliver value that is sufficient and as risk-averse as is necessary to the business stakeholders.

Finding the Correct Balance Takes Careful Thought and Collaboration

I cannot give specific recommendations in this article about any particular situation. Rather, I would recommend that all risks and tradeoffs be written down, categorized, and reviewed with key stakeholders in an open and honest fashion. If a product owner would favor a simpler, less glitzy UI in favor of core, sound, well-tested and stable backend services, then the team must focus on delivering this. On the other hand, if a product owner says that customers will be turned off by a UI that is vapid and barren, then the team must work hard to achieve that.

Building the correct features of any system is, as Kent Beck mentions, a very experimental process when the goal is not known. Because of this, it’s important to use practices that help in driving toward the correct features in the quickest fashion. Sometimes that approach is careful, stict adherence to TDD. Sometimes it is not. It takes judgement and careful balancing of all factors between business people and technical people to make the correct decisions.

Problems That Teams Can Face When Practicing TDD

In my experience, projects can get into trouble when project leaders do not make these nuanced decisions about TDD that require careful understanding, analysis, and collaboration amongst the entire team. In a project without a schedule, then it might make sense to practice a strict form of TDD. In projects that have specific dates set in stone, a team must make decisions about when and where to apply strict TDD and when and where to accept technical debt in order to meet those dates.

Delivering Business Value and Mitigating Project Risk Through Behavior-Driven-DevelSopment

It might come as no surprise by the title of this article, that others much more learned in these practices and problems than I have sought remedies. Namely, Dan North and Dave Astels have written and presented widely on Behavior-Driven-Development (BDD) and its close aligntment to business success criteria. The rest of this article will introduce BDD and provide links to sites, videos, presentations, and a sample project in .NET by Rajesh Pillali that provide extensive information.

Introducing Behavior-Driven-Development

As stated on http://behavior-driven.org, the definition of BDD is:

“Behaviour-Driven Development (BDD) is an evolution in the thinking behind Test-Driven-Development and Acceptance-Test-Driven-Planning.

It brings together strands from Test-Driven-Development and Domain-Driven-Design into an integrated whole, making the relationship between these two powerful approaches to software development more evident.

It aims to help focus development on the delivery of prioritised, verifiable business value by providing a common vocabulary (also referred to as a Ubiquitous Language) that spans the divide between Business and Technology.”

Dan North’s original formulation of BDD is here: http://blog.dannorth.net/introducing-bdd/. Dave Astels clearly explains it more succinctly in his own article here: http://techblog.daveastels.com/files/BDD_Intro.pdf

Business Analysts Write Executable Specifications When Teams Practice Behavior Driven Development

Here is an example BDD feature specification from the SpecFlow project’s web site:

There are other “Steps” in the worflow, but here is how the ultimate fulfillment of that specification gets executed and measured:

TODO:

Complete article

Complete Examples

Rajesh Pillali provides an excellent article on CodeProject here: http://www.codeproject.com/KB/architecture/BddWithSpecFlow.aspx

Retrospective of Greg Young’s Final CQRS and DDD Online Course

2010-09-04T23:21:00.001-07:00

I woke up at 4 am so that I could virtually attend via LiveMeeting Greg Young’s final online CQRS and DDD course. While I had some sound issues and had to keep coming back into the session, I can still say: Wow! Greg’s knowledge is excellent and his presentation was great.

You can watch him deliver a similar talk at this Skillsmatter podcast about the business perspectives of this architecture: http://skillsmatter.com/podcast/open-source-dot-net/greg-young-cqrs-event-sourcing-the-business-perspecive

It’s easiest these days for me to learn visually, so in review, here are a couple of slides from another blogger, Dylan Smyth, who blogged about this that summarize the distinction between a CQRS style architecture and a more typical stack:

Rinat Abdullin has a great diagram on his own blog depicting what Greg also calls the Task-Driven UI:

To summarize in my own, short, words:

Typical Architecture

On the left side we have the “typical” architecture which has this kind of interaction model:

User clicks a button in a client.
Client issues a request to the server, asking for Product with ID 50, for example.
Server fetches an entity for Product with ID 50.
Server creates a DTO from this entity and sends it to the client.
The client modifies this DTO.
Client ships the modified DTO back to the server for an update.

CQRS Architecture

He contrasts that the right side, which he says may appear at first to be more complex, but it actually turns out to be simpler and less costly because of the Thin Read Layer in which the operations are optimized for read, not write. This means the translation from data to DTO occurs directly here.

User clicks a button in a client.
Client issues a read request to the server, asking for Product with ID 50.
Server fetches the data for Product with ID 50.
Server creates a DTO from this entity and sends it to the client.
The client operates on a screen and issues a Command to the server, which is not equivalent with sending a modified DTO back to the server.

An important note here is that the "Event Store” is where the commands are serialized, which does not have to be in a relational database. The De-Normalized Data Cache is where a “view” of data created and stored as a result of the Events being stored.

CQRS Diagram

This is an image that Udi Dahan uses to describe CQRS in his own post Clarified CQRS:

Domain-Driven-DESIGN

He also discussed Domain-Driven-Design after talking about CQRS, but unfortunately I did not hear all of that.

Recommend Resources

In addition to the Skillsmatter podcast above, here are some other recommended links about CQRS:

Super Simple CQRS Example from Greg Young: http://codebetter.com/blogs/gregyoung/archive/2010/08/31/super-simple-cqrs-example.aspx
Clarified CQRS, Udi Dahan: http://www.udidahan.com/2009/12/09/clarified-cqrs/
Udi Dahan on CQRS, DDD, and NServiceBus: http://www.infoq.com/interviews/dahan-cqrs-ddd-nservicebus
Command-Query Responsibility Segregation, interview with Udi Dahan: http://www.infoq.com/presentations/Command-Query-Responsibility-Segregation
Greg Young Discusses State Transitions in Domain-Driven Design and DDD Best Practices: http://www.infoq.com/interviews/greg-young-ddd

Review: Learning From Five Years as a Skype Architect

2010-08-12T20:30:00.000-07:00

There is an excellent presentation Andres Kutt recorded at QCon London 2010 about lessons learned from working for fives years as an architect within Skype. Here's the link and full description:

http://www.infoq.com/presentations/Learning-from-Five-Years-Skype

"Andres Kutt discusses his experience as architect at Skype for five years, sharing some of the lessons learned: rules of thumb do not always apply, functionality is important, use simple solutions, buzzwords are dangerous, the architecture needs to fit into the organization, and communication is important.

Bio
Andres Kutt has been with Skype since 2005, leading a growing team of architects. Before that, he worked as IT Deputy Director General for the Estonian Tax and Customs Board, and as a free consultant for financial institutions."

Review
I love his comment at 36 minutes that "80% of what an architect does is communicate". This is so true and important to realize, but not all architects excel at communication. This can cause significant problems on projects. As a steward of the conceptual and functional integrity of a system, the architect must strive to understand and distill the core standards, interfaces, components, connectors, and unifying concepts of the system to the rest of the technical team. They also must be able to speak to business people and other stakeholders using non-jargon language that speaks to the underlying concerns of budgets, ROI, risk management, etc.

Key Points

Rules of Thumb Do Not Apply

His example is that he joined Skype with the belief that "You do not put business logic in the database", but he learned about the Postgre database Skype uses and the distributed cluster and how the team there actually modified the Postgre source code to suit their needs. As a result, he no longer worries about business logic in the database.

Functional Architecture is Important

He says that at least every nine months he gets the whole team together to sit down and all diagram the architecture together and find the problems and fix those problems by analysis and refactoring.

Usually the architecture remains the same, but they fix problems and remove single points of failure.

Unfortunately, he says, many times architects get caught up in the finer grained details of technical implementation architecture.
In his experience, ugly functional architecture always leads to ugly technical architecture.

Simple things work.

"When something takes more than three sentences to explain, it usually doesn't work"
"Always remove all the functionality that is not necessary"
"Always chop things up"
He gives the example of SOAP being bloated and overly complicated. Instead of SOAP, Skype uses REST.

Complicated things do not work well.

He says if you're architecture is getting too complicated you are doing something wrong.

Buzzwords are dangerous.

His example is "Cloud Computing"

Instead of resorting to "buzzwords" or being either awed or fearful of them, seek to understand the motivations behind buzzwords when people use them. What benefits are they interested and what about that buzzword makes them think it will deliver those benefits?

You are there to solve business problems, not the most beautiful, perfect system ever.

He says to not take on technical debt that will cripple the business.

Talk to people in order to understand and discover requirements

Don't expect them to trust you or your team without you trusting them and the business goals they have.
Also do not expect teams to magically understand requirements if they are not also talking with product folks.

UML diagrams do not help much

Comments
The observations Andres makes are right on the money. I've been on so many projects, or went through periods of time on projects, where it seemed like too much management was going on. Managers, by their nature, desire to control and predict everything. They often feel that by limiting the time that one "role" of person, be they BA, QA, TA, DEV, PO, etc need to talk to each other the better the results "should be" and the faster functions "should" get delivered. These managers tend toward thinking of a "pipeline" mentality, such as:

Product Owners and Business Analysts, with the assistance of Technical Architects define functional requirements
Developers and QA then create tasks and test cases to implement these.
And, while the Developers and QA are working, the PO, BA, and TA continue to work on "what's coming next".

While it is a natural and admirable goal that the firsts step above should produce requirements documents and wireframes that are perfectly defined and perfectly refined state of readiness for Developers and QA to implement, it seldom happens this well.

Scott Ambler has a great chart about communications in software projects in this article: http://www.agilemodeling.com/essays/communication.htm.

For me personally, I saw this recently in a project. We were facing several "unknowns" in our requirements that were effecting me and other developers. We quickly gathered several developers, QA folks, and business analysts together. We had started with a whiteboard, getting everyone to focus on shaping the common language and understanding of the problems at hand, but quickly moved to a notepad instance and simple bullet lists. Everyone quickly reached consensus within five minutes of discussion. We all walked away with a common understanding and greater confidence in the requirements.

Agile Software Construction is Like Neighborhood Construction

2009-12-31T11:56:00.001-08:00

In my last article, I commented on Dr. Jan Chong's dissertation presentation "Software Development Practices and Knowledge Sharing: A Comparison of XP and Waterfall Team Behaviors". I raised some questions about the utility of long-term design documents and provided links to Scott Ambler and Mary Poppendieck's writings about unit tests and automated tests as a form of long-term executable specifications.

In this article, I'm going to give a better version of an often heard analogy that "Software construction is like building construction". Usually you hear this when product owners, project managers, or architects say things like, "We can't do that until we've got the foundation in place. Think of a building. You have to have a solid foundation first."

OK. Maybe so, but you only need it for the very first independent feature, and you should be building features as independently as you can. Here's why:

The Simple Building to Software Analogy

Here is a simplified one-to-one comparison that maps the "layers" of a building to the layers of a software application:

Building Component	Software Component
Furnishings	User Interface Layer
Internal Structure	Business Logic Layer
External Structure	Data Access Layer
Foundation	Database
Blueprints	Requirements Specifications

Traditional Versus Agile Software Development

To prepare my final point, I present the following two slides come from the Autumn of Agile screen-cast series mentioned in prior posts.

This first slide depicts the traditional sequential waterfall approach to building an application. The main theme is that waterfall projects typically result in teams developing horizontal layers that end up delivering business value only after all layers are complete.

This next slide depicts how agile (or simply iterative & incremental) projects develop systems. The main idea here is that features are developed vertically. This means that the features are independently valuable, and can represent a portion of the total desired value at any time should the project schedule get cut short.

Neighborhoods Get Built Using Agile, Not Waterfall

Let's assume we have 5 total features in the software system depicted in slides one and two. In the first slide, all 5 features are "finished" at the same time when the user-interface is built on top of the lower layers. In the second slide, a single feature is built from "top to bottom" during each iteration.

Can You Say Fffired?

Imagine a team of developers building a neighborhood like the traditional sequential waterfall approach to building software. Let's assume each iteration is 5 weeks for simplicity's sake. After 5 weeks, the developers would have built 5 separate sets of blueprints for the 5 houses to be built. Imagine that these are 5 or more separate parts of an overall software requirements specification in a software development project. They might describe database table schemas, relationships, multiplicities, etc.

	Week 1	Week 2	Week 3	Week 4	Week 5	Elapsed Weeks
Iteration 5						25
Iteration 4						20
Iteration 3						15
Iteration 2						10
Iteration 1	Blueprints	Blueprints	Blueprints	Blueprints	Blueprints	5

I'm sure you already see how asinine this approach to neighborhood development is.

Nevertheless, let's continue with the next 5 weeks and see how far our team has gotten toward delivering the first "potentially habitable housing increment":

	Week 1	Week 2	Week 3	Week 4	Week 5	Elapsed Weeks
Iteration 5						25
Iteration 4						20
Iteration 3						15
Iteration 2	Foundation	Foundation	Foundation	Foundation	Foundation	10
Iteration 1	Blueprints	Blueprints	Blueprints	Blueprints	Blueprints	5

After 10 weeks of work, our construction company has failed to complete a single inhabitable house. What are we paying these people for? Why are they building a foundation, then moving to the next lot to build the next one? I guess they're just going to get back to it sometime later.

Let's just speed ahead and see what we have after 25 weeks on the project:

	Week 1	Week 2	Week 3	Week 4	Week 5	Elapsed Weeks
Iteration 5	Furnishings	Furnishings	Furnishings	Furnishings	Furnishings	25
Iteration 4	Internal Structure	Internal Structure	Internal Structure	Internal Structure	Internal Structure	20
Iteration 3	External Structure	External Structure	External Structure	External Structure	External Structure	15
Iteration 2	Foundation	Foundation	Foundation	Foundation	Foundation	10
Iteration 1	Blueprints	Blueprints	Blueprints	Blueprints	Blueprints	5

This is great, finally. After 25 weeks we finally have inhabitable houses! Actually we had one inhabitable house after week 20, then one more per week until 25. That almost feels like a privilege at this point to get so many houses finished so quickly in succession.

Not in My Backyard!

Do construction companies ever really build houses like this? Not that I've ever seen. They try to complete a house as quickly as they can after the foundation concrete sets. Of course, that means they may indeed start on a second foundation before they build the remainder of the first house, but you get my point.

Suppose it were 30 houses instead of 5. Could you imagine if this were a new neighborhood being built just behind your backyard. What would you think if you saw the crews building 30 separate foundations, then coming back to the first to frame it, then moving to the next to frame it, etc, etc. You'd think they had last their minds or have absolutely no respect or understanding of the Time-Value-of-Money or concern for a Return-On-Investment.

Neighborhood Construction is More Vertical, Like Agile

As you can see below, our construction crew progresses through each house from "the ground up", which means that they finish one house completely every five weeks. This, in agile terminology, is their velocity per iteration. The learn more about agile teams and velocity, read my article "D = V * T : The formula in software DeVelopmenT to get features DONE".

	Iteration 1	Iteration 2	Iteration 3	Iteration 4	Iteration 5
Weeks Elapsed / Houses Completed	5 : 1	10 : 2	15 : 3	20 : 4	25 : 5
Week 5	Furnishings	Furnishings	Furnishings	Furnishings	Furnishings
Week 4	Internal Structure	Internal Structure	Internal Structure	Internal Structure	Internal Structure
Week 3	External Structure	External Structure	External Structure	External Structure	External Structure
Week 2	Foundation	Foundation	Foundation	Foundation	Foundation
Week 1	Blueprints	Blueprints	Blueprints	Blueprints	Blueprints

Comparing Velocity between Waterfall and Agile Approaches

In the following table, we're looking at how many houses we complete per iteration.

So, the calculation is number of iterations passed / number of houses completed.

Thus, the agile velocity is constant in terms of 5 weeks. We can always count on 1 house becoming complete every 5 weeks.

	Total Completed Under Waterfall	Waterfall Velocity per Iteration	Total Completed Under Agile	Agile Velocity per Iteration
Week 5	0	0	1	1
Week 10	0	0	2	1
Week 15	0	0	3	1
Week 20	0	0	4	1
Week 25	5	1	5	1

Combining Return on Investment with Velocity

Finally, since we know that nobody can live in a house until it's completed, we can calculate the potential accrued Return-on-Investment (ROI) after each iteration for actually having sold houses.

Let's assume we can sell each house for $100,000.

	Potential Waterfall ROI	Potential Agile ROI
Week 5	0	$100,000
Week 10	0	$200,000
Week 15	0	$300,000
Week 20	0	$400,000
Week 25	$500,000	$500,000

Taking into account the Time-Value-of-Money (TVM) we understand that it's certainly more valuable to realize returns via the agile approach to building neighborhoods as opposed to the waterfall method!

Similarly, when we talk about ROI and TVM regarding software projects developed using agile methods, certainly we sometimes mean that we can gain monetary returns earlier by producing "potentially shippable product increments" faster and getting them to the market. However, what we also mean is that by focusing on building features top-to-bottom, we can get the complete experience and feedback necessary to then apply toward the next feature and we prevent the pains of having to refactor code later when it's no longer fresh and when requirements have been changing all round.

Here is a good video on YouTube which depicts this relationship between time and value accrual very well:

Until next time, stay agile, not fragile!

Review: Software Development Practices and Knowledge Sharing: A Comparison of XP and Waterfall Team Behaviors

2009-12-31T11:50:00.000-08:00

Jan Chong, Ph.D., wrote her dissertation on "Software Development Practices and Knowledge Sharing: A Comparison of XP and Waterfall Team Behaviors". I came across a recorded presentation of her discussing her dissertation at The Research Channel here:

http://www.researchchannel.org/prog/displayevent.aspx?rID=16075&fID=345

Here is the description of the recorded presentation:

My dissertation research explores knowledge sharing behaviors among two teams of software developers, looking at how knowledge sharing may be effected by a team's choice of software development methodology. I conducted ethnographic observations with two product teams, one which adopted eXtreme Programming (XP) and one which used waterfall methods. Through analysis of 808 knowledge sharing events witnessed over 9 months in the field, I demonstrate differences across the two teams in what knowledge is formally captured (say, in tools or written documents) and what knowledge is communicated explicitly between team members. I then discuss how the practices employed by the programmers and the configuration of their work setting influenced these knowledge sharing behaviors. I then suggest implications of these differences, for both software development practice and for systems that might support software development work.

Jan's full biography at the time of her dissertation:

Jan is a doctoral candidate in the Department of Management Science and Engineering at Stanford University. She is affiliated with the Center for Work, Technology and Organization. Her research interests include collaborative software engineering, agile methods, knowledge management and computer supported collaborative work. Jan holds a B.S. and an M.S. in Computer Science from Stanford University.

Her complete dissertation is available here: http://www.amazon.com/Knowledge-Sharing-Software-Development-Comparing/dp/3639100840

Highlighted Slides from Recorded Presentation

It's nice to see a formal study that compares these different styles of work. Jan spent time observing both teams for about 9 months. Here is a bit more about her methodology.

Study Methodology

Team Communication Styles

First, for the XP team, communication is more open, faciltated by information radiators and an open workspace:

The waterfall team has team members who work alone, in their own cubicles, and who communicate primarily through an online chat program. Thus, they sometimes "broadcast" information to others inside of the chatroom:

Observed Events and Data Coding

Through analyzing her recordings and notes, Jan classified all the different knowledge-transfer events into the following categories:

Knowledge Seeking Behaviors

This slide compares the actions taken by teammates in explicitly asking for knowledge transfer from others:

Knowledge Offering and Relevance

Below Jan has categorized types of communication that are offered and their respective relevancies.

Recorded Knowledge

It's very interesting to note that the waterfall team recorded knowledge far more for the "long-term", percentage-wise than on the XP team. What is not clear, however, is whether this means that the XP teams simply recorded more personal items in addition to the same type of long-term items or whether they leave out certain long-term items that waterfall teams recorded properly.

Summary Slides

The following three slides are her concluding slides:

Review and Analysis

I'm about to join a great team for an agile project that will be built using Scrum & XP practices. This opportunity is very exciting for me. That excitement is born out of my own "in the trenches" experience and observations on both Agile/Scrum/XP and waterfall. If you're reading this and unfamiliar with the differences between agile and waterfall, I recommend you take both of them for a "test drive" by reading or listening to my article entitled "From Waterfall to Agile Development in 10 Minutes: An Introduction for Everyone". Also read "Don't Get Drowned by Waterfall: Break out the Delusion" and "D = V * T : The formula in software DeVelopmenT to get features DONE"

If you cannot tell by now, my preference is for agile development and not waterfall. Waterfall, at least in its pure form, as you can read in the first article, is a mistake and always has been a mistake for software systems development. While Jan stops short of claiming a preference for one style of development or the other in her analysis, it's important to note that this is because that was not her intention. She is working on improving software methodology as a whole, and seeks to synthesize best practices from actual empirically observed behavior and data.

How Useful is the Long-Term Design Documentation by the Waterfall Team?

Jan observed that the waterfall teams had members who worked alone and went "back to the code" when they needed to understand something or had to work on a new module they had not worked on before. What I'm not clear on is whether she meant "automated test case code" or "implementation code". She also noted that the XP team members consulted one another more often about how things work prior to looking at code. She also noted that the waterfall teams created a higher percentage of recorded knowledge about the long-term aspects of the project. I am assuming this meant written documentation judging by her comments on video.

Question: How often the waterfall developers actually refer back to those long-term design documents, and which of those documents did they originally anticipate being useful to other developers?

The reason I would ask this is that she already noted that the waterfall team members spent a lot of time reading code, and also reading CVS check in messages when others checked in changes, but she didn't address whether they (or anyone) reads the long-term design documents for any useful purpose.

Experience Highest Communication Bandwidth via Face-to-Face at Whiteboard Collaboration

It has been my personal experience as a developer and architect that when working more closely, in a collaborative, open workspace, with other members of a team, I do not need to refer to implementation code or documentation as often anyway. Instead, we rely more a constantly evolved shared language, basic metaphors, automated test cases, and whiteboards to communicate the "gist" of how something works, then we refer to detailed implementation code as soon as we need the details or pinpoint a trouble area. Scott Ambler has written extensively on the subject on agile documentation and communication. Here is a chart he produced based on a survey about the most effective forms of communication:

Source: http://www.agilemodeling.com/essays/agileDocumentation.htm

As you can see, paper documentation is by far the absolute worst form of communication. Face-to-face whiteboard is the most effective. Kevin Skibbe, a friend that I used to work with, is the most effective whiteboard communicator I know. What he explained to me was that when two, or more, people try to communicate via the whiteboard they must focus on developing a shared mental model. Even when it's just face-to-face communication, without a whiteboard, both people are still maintaining independent, non-shared mental models of what the other person is thinking.

Building Long-Term Executable Knowledge / Documentation via Automated Tests

One measurement I don't explicitly noted is the notion of "Executable Knowledge", or more frequently called "Executable Specifications". Scott Ambler writes about this here: http://www.agilemodeling.com/essays/executableSpecifications.htm, and Mary Poppendieck writes about it in her books and presentations: http://www.poppendieck.com/

I suggest that teams, whether waterfall or agile, incorporate this practice into their development in order to produce fewer defects, increase explicit knowledge in the code-base, and reduce the need to continually read implementation code.

We should first think about software systems in light of their intrinsic nature and in terms of the change process required to modify or enhance them with maximum ability to produce the desired result. That result might be faster ROI in the market, increased product quality, or what have you.

What we never want to do however, is result in broken functionality after we release. This is why we build regression test suites that can be executed at will to confirm as much as computationally possible of the encoded knowledge about the business domain and user requirements.

This is how Test-Driven-Development works. Here is how Ambler depicts this view:

That is a nice technical view of things, but what about the view from the business side? Ambler presents a written form of the requirement that would be provided to developers from a business analyst in the article linked above.

Waterfall is Always Wrong Compared to Agile for Building a New System

I'm going to assume here a definition of waterfall that is primarily the standard sequential approach where requirements come first, followed by, detailed design, development, (test, drop scope, rework) (repeat prior three as necessary until completion), and deployment and maintenance.

If a project is about developing a completely new product from the ground up, then adopting the sequential model for a system of any degree of complexity beyond perhaps an estimated month of duration is simply calling for pain. I'm not going to explain why in this article, but I refer you to the three articles I wrote above for complete explanation. But, to summarize:

When you begin a moderate to large custom development project, there are many requirements you simply cannot know until you have started to develop a subset of the entire envisioned project and that subset makes its way into the hands of the critical stakeholders like the project owner and the target users.

For more evidence, refer to Craig Larman's research and explanation about the history of waterfall in this article: http://www.highproductivity.org/r6047.pdf

Waterfall is Still Wrong for Enhancing an Existing System

However, suppose a software system is already released and "in production", should a team then use waterfall techniques to build additional features?

I believe the answer is no. The reason is reflected above in the TDD diagram. You want to reduce the impact of changes and build up a suite of regression tests as you develop the solution. And, you want to seek feedback as early as possible to reduce time spent working on incorrect or undesired functionality.

Waterfall is Especially Wrong for Rebuilding an Existing System

In my experience, and I've been through it a couple of times now, waterfall is notoriously wrong when you are asked to rebuild an existing system in a new technology. The reason is that often the project sponsor will state little more than the fact that they want the existing system functionality essentially duplicated. Unfortunately, much time will be wasted by the team if it tries to clone piece-by-piece the existing product.

A much better approach is to use the existing system as a very high fidelity model. It should then use iterative and incremental agile practices to deliver features that are focused on making and keeping the project potentially shippable as soon as possible. This ensures a complete vertical "slice" of functionality through all the application's layers gets built as soon as possible. To learn more about these practices, see the screen-cast series Autumn of Agile at http://www.autumnofagile.net. The first episode gives a comprehensive and very compelling explanation of why agile presents better business value than waterfall. My article "Don't Get Drowned by Waterfall: Break out the Delusion" references a few key slides from that series.

The Embarrassment of Waterfall's Persistence

Waterfall continues to exist in the technology world because it sounds easy to understand on the surface, but everyone also realizes the intrinsic contradiction in software development: requirements constantly change. I don't blame product owners or business people for waterfall's persistence. I blame ourselves, the developers and project managers. It is our fault for not being more responsive to changing environmental conditions and changing requirements.

But, balancing the ability to change on demand with the requirement to remain stable at all other times is what agility is all about. That is why agile focuses on rigorous empirical testing, visual monitoring, and continuous feedback. That is why automated tests as executable specifications speed up the ability to change while simultaneously increasing quality and confidence.

Stay tuned for my next article which will update the "age old" building construction metaphor that says building is software is like building a building. In many ways it is, but I will make key distinctions that enable agility!

Until then, stay agile, not fragile.

Don’t Get Drowned by Waterfall: Break out of the Delusion

2009-12-07T20:43:00.001-08:00

Many development projects are built with an approach traditionally called “Waterfall”, or “Big bang all at once”. This means that the entire system is defined, designed, developed, tested, then released in that precise order. In its purest form, a system built using waterfall techniques is utterly useless until the time when nearly all of its features are claimed to be “feature complete” and “ready for testing”.

To put it very simply: if a project has 50 features, a team will attempt to build all 50 and only then test all 50 at the same time without any formal, professional QA and automated testing done during the construction of the features. This is a recipe for wildly missed delivery dates, at best, and utter disaster, at worst.

Irrational Justifications for Waterfall

Why in the world would a team try to build a system this way? You might hear something like “The whole system must work before we go live, so we’re going to test everything together”. On the surface, this sounds reasonable. Of course the whole system needs functional, end-to-end testing. However, in any system of non-triviality, this is, at best, an ignorant statement that betrays a lack of experience in developing complex systems. At worst, it’s a statement of learned helplessness or laziness born out of a lack of desire or perceived lack of time to break down the system into smaller, independently testable parts.

The Fantasy of Waterfall

The fantasy vision that people developing a waterfall project have is something like the blue line in the chart below from Steve Bohlen’s Autumn of Agile screencast series. In this vision, late in the project all the components that have not yet worked together in completion during the first 80% of the project get tested all at once and suddenly start to work and the project is released on time on an budget.

Yeah. Right.

Here’s the fantasy vision, compared with an Agile approach in terms of business value accrued over time, especially if the project suddenly is called to halt:

Continuing with the Waterfall delusion, the idea is that all aspects of the system can be developed in isolation from each other, never needing feedback or rework:

The Reality of Waterfall

The reality is shown below. The sharp decrease in the component stability represents the big “Oops, wish we would have thought of that sooner” moments that we all know and love during waterfall projects.

I can think of two instances off the top of my head during recent real-world projects where I’ve seen this:

While working on a large electronic commerce application rewrite, our team warned our senior management that the legacy COM architecture would not scale well under .NET. Despite our warnings and evidence, our management wanted us to “just forge ahead” as if it was more brave or honorable to continue doing something completely stupid rather than sit down with all of us and think through the difficulties necessary to properly solve the problem and devise a plan to reach the market and achieve positive ROI. Psychologists call this cognitive dissonance. I call it lack of planning, ignorant, or maybe just cognitionless dumbness. I eventually left that position and have kept in contact with my old colleagues who eventually had to rewrite the entire COM layer in pure .NET to attain the desired business value (release).
On another recent project I’ve worked on, I was assigned the responsibility of designing and implementing a complex security model on top of the existing custom entity model the new application was already using. Hundreds of entities had already been created in this system, and a large, wide horizontal stretch of pages and controls had been created already. But, no features had yet been designed vertically deep enough to use security. As a result, the system was thus far built entirely with static stored-procedures, but the complex security model required that all SQL statements be appended with additional where clauses and custom filters. Fixing this took weeks to refactor the data-access layer and rework static stored procedures to use more of a Query Object pattern.

Special Note: Microsoft’s Advice for Systems Re-architecture and Migration

Microsoft has a document entitled “Microsoft .NET/COM Migration and Interoperability'”, located here: http://msdn.microsoft.com/en-us/library/ee817653.aspx. It will serve all .NET or COM / C++ developers to read this document.

Microsoft recommends that when you are re-architecting a system using .NET from an older technology, such as COM / C++ that you do not attempt a big-bang, horizontal migration. Instead, they recommend that you create a completely functional, vertical slice of the application first, before expanding horiztonally.

Here is an excerpt:

“You might choose to adopt a vertical migration strategy for a number of reasons:

Planning to re-architect
If you plan to re-architect your application, vertically migrating part of your application to the new architecture provides a good test bed for the new design. The .NET Framework also makes it easier to provide the functionality that newer architectures are built on. For example, you can use HttpHandlers to perform many of the tasks for which you would previously use ISAPI extensions, but with a much simpler programming model.”

Here is a diagram from the same document depicting a vertical migration:

How to Stop The Horizontal Waterfall Madness

If you or your project is on a path of waterfall, horizontal development, then you have your work cut out for you, but it’s not too late. It takes discipline, honesty, and courage to set things upright and vertical.

Here are a few key practical steps:

Stop adding code to the system that is not scheduled for testing in the current or next month. If you do not have a product road-map and product back-log prioritized by business-value and thus don’t know when a feature is going to need that code, then stop adding it, immediately.
Focus instead on the fact that you need a road-map and a prioritized product back-log and define this in terms of users’ needs. If your product owner cannot or won’t prioritize the backlog or features, then simply list them in the order that your users encounter the features or the order that your help-desk team tells you needs most improvement.
Now, identify a vertical slice of the application that you can focus everyone on the team on implementing from top-to-bottom. Determine how to get this slice as close 100% functional as possible.
Work daily with your users, test team, developers, and other stakeholders to create, together, a strategy for standing up and ruthlessly testing that vertical slice.
Once you’ve gotten critical user feedback for usability and functionality, apply those lessons learned to the next vertical slice, and so forth!

Whose Responsibility is it to Stop Waterfall Thinking?

It’s far too easy for us as developers or architects to say that we’re just doing what we’re told by our managers or our business owners. After all, they “Just want the product done”, right?

Certainly, we must listen to what our managers and owners ask of us. It’s very important that we do so. However, what’s more important, to them and to the success of projects, is that we as professionals bring our expertise and our knowledge to the problem-at-hand and that we act with resolve and courage to do the job correctly. Often times this means sitting down with our managers and owners and explaining to them, however uncomfortable it makes them or you, what it really means to do iterative development with high quality. Often we must educate them about the history of waterfall and its miserable rate of success.

Thus, the responsibility is yours. The responsibility is mine. The responsibility is the team’s.

Craig Larman on the History of Waterfall and Iterative & Incremental Development

Horizontal waterfall approaches, despite being popular and widespread today, came after early practitioners used iterative & incremental development techniques to greater success. The sad but true history of waterfall is that the U.S. Department of Defense (DOD) misinterpreted Winston W. Royce's paper on systems development and went on to enforce it as a government standard to which large contractors then adhered. Commercial industry followed suit from the government contractors. The DOD went on the revise their standards, and within the last twenty years they have recommended iterative & incremental practices.

Object-oriented development guru Craig Larman has written several books about iterative & increment project management over the years. He also wrote an extensive article about the history of iterative & increment development (IID) here: http://www.highproductivity.org/r6047.pdf

Here’s a crucial excerpt history from that article:

In the 1970s and 1980s, some IID projects still incorporated a preliminary major specification stage, although their teams developed them in iterations with minor feedback. In the 1990s, in contrast, methods tended to avoid this model, preferring less early specification work and a stronger evolutionary analysis approach.

The DoD was still experiencing many failures with “waterfall-mentality” projects. To correct this and to reemphasize the need to replace the waterfall model with IID, the Defense Science Board Task Force on Acquiring Defense Software Commercially, chaired by Paul Kaminski, issued a report in June 1994 that stated simply, “DoD must manage programs using iterative development. Apply evolutionary development with rapid deployment of initial functional capability.”

Consequently, in December 1994, Mil-Std-498 replaced 2167A. An article by Maj. George Newberry summarizing the changes included a section titled “Removing the Waterfall Bias,” in which he described the goal of encouraging evolutionary acquisition and IID:

Mil-Std-498 describes software development in one or more incremental builds. Each build implements a specified subset of the planned capabilities. The process steps are repeated for each build, and within each build, steps may be overlapping and iterative.

Mil-Std-498 itself clearly states the core IID practices of evolving requirements and design incrementally with implementation:

If a system is developed in multiple builds, its requirements may not be fully defined until the final build…. If a system is designed in multiple builds, its design may not be fully defined until the final build.

Tom Gilb’s Principles of Software Engineering Management was the first book with substantial chapters dedicated to IID discussion and promotion. Computer Meanwhile, in the commercial realm, Jeff Sutherland and Ken Schwaber at Easel Corp. had started to apply what would become known as the Scrum method, which employed time-boxed 30-day iterations. The method took inspiration from a Japanese IID approach used for non-software products at Honda, Canon, and Fujitsu in the 1980s; from Shashimi (“slices” or iterations); and from a version of Scrum described in 1986. A 1999 article described their later refinements to Scrum.

Stop laying down flat. Stand up straight. Have more fun, and kick waterfall to the curb!

Book Review: Effective REST Services via .NET For .NET Framework 3.5

2009-09-18T18:33:00.001-07:00

I’m currently reading Kenn Scribner and Scott Seely’s book Effective REST Services via .NET For .NET Framework 3.5. This is a great book which gives a thorough background on the history of REST, quoting often from Roy Fielding’s Ph.D. dissertation on the subject. The authors do not treat the subject with “kid gloves” or “framework gloves”. Instead, they exlain the HTTP protocol and its purpose, benefits, and its drawbacks.

Warning: I’m not really that good at giving book reviews for the general audience, and I feel like there’s already plenty of information on the web about all these subjects, but for own personal benefit it helps me to summarize subjects in my own words to clarify them. For me, it’s best for me to review a table-of-contents and then just give my best shot at summarizing what I’ve read in brief paragraphs using examples. This misses lots of details, of course, and some of what I remember won’t even come from the book.

For me, this is crucial research material for two projects I am working on. One of these is currently public and is a simple RESTful gaming platform. You can see the progress so far on that at http://apps.ultravioletconsulting.com/Skookzee.

So, don’t mistake this as a book review for anyone other than myself, but I hope you enjoy it anyway.

Chapter Summaries

1: RESTful Systems: Back to the Future

This chapter discusses how early on web “services” consisted simply of web servers and URIs that delivered chunks of data. Sometimes this was done through CGI processing that responded to the invocation of the incoming request, but eventually most web sites became simply file servers. Later, SOAP-based services became popular, but the problem here was that SOAP services are intrinsically Remote-Procedure-Call centric (RPC). The problem with this is that SOAP services typically encapsulate both NOUNS and VERBS, or Resources and Actions, Objects and Methods –- pick your metaphor – behind a single URI.

You might ask why this is a problem. One reason is that the HTTP and URI protocols were designed, as Roy Fielding describes in his dissertation, to work well in a distributed network and to enable distributed caching. Because SOAP specifically, and RPC in general, deal in coupling an invocation-time command to a remote resource, there is no architectural support to cache the results. Indeed, the idea of caching a call to a SOAP web service is something that each service would have to completely encapsulate.

Conversely, the way the web was actually designed was with a few key principles in mind to enable the distributed, cacheable structure of resources. First of all, the URI stands for Universal Resource Identifier, and a resource is a noun, it’s not an action or a method call. SOAP services are about exposing a single noun, the service entry point, and then encapsulating any number of nouns and verbs behind that one single noun.

So, in a RESTful system, each URI refers to a single, unique representation of a remote resource. For example, http://agilefromthegroundup.blogspot.com/2009/09/done-features-equals-velocity.html refers specifically to the HTML representation of my post about development velocity and tracking. There is, however, another URI which refers to this resource in another way: http://agilefromthegroundup.blogspot.com/feeds/posts/default/4352538909387142862. If you click this link you will get the content of that same blog post, but as XML inside of an “<entry>” element.

A good depiction of this distinction is found in the document Architecture of The World Wide Web on the W3C web site:

This becomes important because of REST’s concept of a single, generic interface, or a limited number of verbs/actions/methods. In terms of HTTP this turns out to be the standard set of HTTP methods: GET, PUT, POST, DELETE, HEAD, OPTIONS. Each of these methods must behave in a consistent, expected way when combined with a requested URI.

Another good reference, for quicker reading, is the summary of the web architecture document. This gives a list of prescriptive good practices for design.

For example, assuming I were authenticated and that Blogger supported this ability (I’m not sure whether it does) then issuing a request with each of the common HTTP methods above against the single URI http://agilefromthegroundup.blogspot.com/feeds/posts/default/4352538909387142862 would produce the following results:

Method	Entity Body Contents	Expected Result
GET	Blank	Returns the HTTP headers for and the current representation of the resource, in this case an XML document. But, does not MODIFY the resource in anyway specified from the request.
HEAD	Blank	Returns only the HTTP headers associated with this resource, but not the XML document itself
OPTIONS	Blank	Returns the list of HTTP methods this resource supports, such as GET, PUT, POST, DELETE
PUT	A complete replacement XML <entry> document	A response code indicating whether the replacement was accepted.
POST	Arbitrary content, perhaps a string of content intended to be added as a comment to the blog post	A response code indicating the result, and possibly the URI of a newly created, subordinate resource, such as a direct URI to the comment added to the entry.
DELETE	Blank	A response code indicating whether the resource was deleted.

There is much more to be said about this chapter, but I leave it to you to read it for yourself and enjoy what you learn.

2: The Hypertext Transfer Protocol and the Universal Resource Identifier

This chapter goes into more detail about the HTTP verbs and the standard response codes and header values, then discusses much more about specific HTTP and REST anti-patterns. These are, broadly:

Anti-Pattern Name	Description	Real-World Examples
GET Tunneling	Specifying method information to the server via GET, which is supposed to NOT modify resources from client-specified commands (idempotency)	Flickr used to have GET methods with commands like “delete” embedded in them. Google’s look-ahead pre-fetch tool caused HAVOC on many user’s accounts as a result!
POST Tunneling	Specifying method information inside of the POST entity body.	SOAP based RPC services are a wholesale violation of this. They embed arbitrary command names inside the envelope.
Misused Content-Types
Miusing Status Codes
Misusing Cache
Cookies
Lack of Hypermedia Support
Lack of Self-Description

The concept of URI and Addressability is covered thoroughly. This is so important for so many reasons to get into, but suffice it to say for Search-Engine-Optimization (googleiciousness / bingability) a URI should identify a single resource unambiguously so that search engine indexes can link to it.

Aside from this there is a wide range of computer science concepts wrapped up inside the old trusty URI.

For example, when you look at the URI: http://agilefromthegroundup.blogspot.com/feeds/posts/default/4352538909387142862, you actually have a specification that tells a user agent how to attempt fetching the resource (HTTP protocol), where to start looking for it (agilefromthegroundup.blogspot.com), where within that space to look further (/feeds/post/ etc). What you also have is a contract that is late-bound by definition. The intent of the URI is to provide, of course, a Universal Resource Identifier, which specifies NOTHING about what actually resides behind that identifier. What resides behind it is completely up to the control of the server. It can be absolutely anything. It just so happens to usually be HTML, but of course can be XML, PDF, TXT, etc.

3: Desktop Client Operations

4: Web Client Operations

5: IIS and ASP.NET Internals and Instrumentation

Recommended Resources: Becoming an Expert, Extending Agile, and Individual Improvement

2009-09-12T07:48:00.001-07:00

There are three recent presentations posted from Agile 2009 to InfoQ.com that I highly recommend you listen to and learn from. Here are the links with the descriptions:

Mary Poppendieck: Deliberate Practices in Software Development

http://www.infoq.com/presentations/poppendieck-deliberate-practice-in-software-development

Description:

Summary
In the nature vs. nurture debate, researchers have declared nurture the winner. People who excel are the ones who work the hardest; it takes ten+ years of deliberate practice to become an expert. Deliberate practice is not about putting in hours, it’s about working to improve performance. It does not mean doing what you are good at; it means challenging yourself under the guidance of a teacher.
Bio
Mary Poppendieck started her career as a process control programmer, moved on to manage the IT department of a manufacturing plant, and then ended up in product development, where she was both a product champion and a department manager. Mary tried to retire in 1998, but instead found herself managing a government software project where she first encountered the word "waterfall".

Josh Comments:

I’ve listened to Mary’s talks at Google and read most of her Implementing Lean Software Development book recently. This talk is excellent. She discusses not just software, but also music performance and artistic talent development, citing studies that have shown it typically takes about 10,000 focused hours for musicians to truly reach the level of expert, and that many of them who begin early in life reach this number of hours by age 20! Regarding software development, 10 years of working professionally is about 20,000 working hours, but of course not all of those hours are spent crafting software. I’ve been working professionally about 10 years, and I think I’m near that level of expertise in a broadened sense, but have much, much, much more to learn in the depth direction.

Allistair Cockburn: I Come to Bury Agile, Not to Praise It

http://www.infoq.com/presentations/cockburn-bury-not-praise-agile

Description:

Summary
Agile came from small, colocated projects in the 1990s. It has spread to large, globally distributed commercial projects, affecting the IEEE, the PMI, the SEI and the Department of Defense. Agile now sits in a larger landscape and should be viewed accordingly. This talk shows that landscape, clarifying how classical agile fits in and what constitutes effective development outside that narrow area.
Bio
Dr. Alistair Cockburn is a world-renowned expert at what is called agile development, the early and regular delivery of business value through improved communications, fast feedback and staged delivery. Dr. Cockburn co-founded the agile development movement, co-authoring the Manifesto for Agile Software Development and the project leadership Declaration of Inter-dependence.

Josh Comments:

This is a great talk which is not really about burying agile, but about recognizing that the basic practices of agile now need to give way to ideas like Software Craftsmanship. He covers much more ground than this, but I’ll just highlight the Software Craftsmanship principles:

“As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:
Not only working software,
              but also well-crafted software
Not only responding to change,
              but also steadily adding value
Not only individuals and interactions,
              but also a community of professionals
Not only customer collaboration,
              but also productive partnerships
That is, in pursuit of the items on the left we have found the items on the right to be indispensable.”

Ashley Johnson and Amr Elssamadisy: Scaling Up by Scaling Down: A (re)Focus on Individual Skills

http://www.infoq.com/presentations/scaling-up-by-scaling-down

Description:

Summary
In this presentation, the causality between performance in the small (individuals and teams) and performance in the large is highlighted and explained. Discover what you can do as an individual regardless of your position in the hierarchy to enable higher performance software development.

Bio
Ashley Johnson splits his time between understanding the challenges that companies face, and consulting with senior IT management to facilitate organizational optimization. Author of the leading book on Agile Adoption, Amr Elssamadisy spends his time helping others learn better ways to develop software and connecting the software development process with its rightful driver - business value.

Josh Comments:

What I like most about this presentation, is how Ashley Johnson incorporates audience participation and experimentation into the course of the presentation. This is the essence of teaching and learning. During my Scrum training with Jeff Sutherland, I was impressed by how Jeff used Scrum to run the training, creating his own task wall with sticky notes which also served as a burndown mechanism. He broke us up into small groups and we worked on problems and exercises that way.

Just to highlight, at one point Ashley asks the audience to break into pairs of two and to come up with a list of “What is required for a high performing team?” This is what the participants came up with on their own:

Shared vision that matters
Trust between the team members
Communication
Passion
Empowerment
Having fun
Challenging each other while holding each other accountable
Leadership

D = V * T : The formula in software DeVelopmenT to get features DONE

2009-09-05T14:33:00.001-07:00

There’s a hidden formula in software DeVelopmenT that tells how fast a team can get features DONE and Ready-to-Ship.

The formula is: D = V * T

It reads as: DONE Features = Velocity multiplied by Time

The importance of a software development team’s velocity

The term “velocity” as it applies to software development is simple to explain and to illustrate. Here’s my definition:

Velocity: A team’s velocity is the number of features it can get completely DONE and Ready-to-Ship during a short, fixed time period (2 to 4 weeks)

Velocity is extremely important for business owners and other project stakeholders. Without knowing the velocity of their team, they have no way to reliably plan release dates and coordinate marketing and sales teams. (2) It’s no exaggeration to say that the most important thing a professional software team can do to increase its value to an organization is to become skilled in the arts of estimation and planning. This post introduces the concepts behind velocity measurement, and provides links for more detailed reading.

Are we there yet? Speed racing down the software delivery highway

Building successful software and delivering it on time is an art of numbers. It all boils down to math, like physics or accounting.

Who can forget the familiar high-school formula D = V * T? (also written as D = R * T)

This, of course, is the famous equation for calculating how far you can travel from a given starting point to another when you already know your velocity (or rate) and how long you will be travelling.

Distance = Velocity multiplied by Time
For example: if we have know we are traveling at 50 miles per hour and plan to travel for 3 hours, then we know we will travel 150 miles.

What happens though if we do not know our velocity, but instead know how far we have traveled and how much time it took to get there? Can we derive our velocity from these other two measurements? Of course we can with simple math. In this case, we have D and T, and can derive V by modifying the formula to be V = D / T.

Velocity = Distance divided by Time
For example: if we have traveled 120 miles after 3 hours from point A to point B, then we know our velocity per 1 single hour is 40, or 40 mph.

Figure 1: Two US high school math students calculating how far they can travel before returning to math class in 30 minutes or before being caught by authorities for driving on the wrong side of the road.

Measuring velocity in software development to decrease time-to-market and realize faster ROI

I hear what you’re screaming: enough with the PSAT math prep already, how does this apply to releasing software on time? It’s so simple you’ll kick yourself, or your team, for not doing this already.

Agile teams use a formula that works the same way. It’s calculated differently, because most software teams aren’t very mobile while coding, though it would be relaxing to code on a boat.

Because a team cannot reliably know in advance how quickly it can complete a set of features, it must use the second form of the equation to derive its velocity based upon actual observation of progress first.

Thus, the formula for calculating an agile team’s initial velocity still reads as V = D / T, except the D stands for “DONE Features” instead of distance. T, or time, usually stands for 2 to 4 weeks instead of 1 hour. For this article, we’ll assume it means 3 weeks.

Initial Velocity = DONE Features divided by Time
For example: if we get 6 features DONE in 3 weeks, then we know our velocity is 6 features per 3 weeks. Simplified, we’ll say 2 features per week.

Here is a simple chart depicting this velocity:

Figure 2: Velocity measurement illustration of six features becoming done during a three-week period

It’s tempting to look at this chart and say the velocity is 2 features per week, and that we can now start using the formula DONE Features = Velocity multiplied by Time to plan ahead. We will use this simplification for the purposes of this article, but keep in mind that this may or may not be true, so be careful! Here are two reasons why:

New Requirements Discovered: During the course of any three-week period, teams will discover new requirements frequently. The new requirements could be bugs, change requests from the business team, or important changes required to match the competition. This is a subject for an entire volume on change management!
Definition of DONE: It’s extremely important that a team agrees upon what qualifies as a DONE feature. Each team must define what it means by the word DONE. I leave that as an exercise for a future article, but you can find some recommended reading and listening below for reference. (3, 4)

For the rest of this post, we’ll pretend that no new requirements are discovered and we’ll define a feature as DONE if it has successfully passed through each of the following development phases:

Requirements Definition
Analysis, Design, and sufficient Documentation
Coding
Unit Testing
Code Review (for development standards adherence and security design assessment)
Refactoring (to conform to standards and address security deficiencies)
Functional Testing
User Acceptance Testing (preferably automated)
Performance Testing
Pilot (beta testing with real or proxy users)
Ready-to-Ship

This may sound like a lot of work! And, it certainly is a lot of work. All mission-critical projects consist of a number of features that must go through these steps before they can be considered DONE and Ready-to-Ship.

Pitfalls of using early travel velocity to forecast total road trip duration

Returning to our travel example, suppose we are traveling from our city to the mountains for a conference about software estimation and planning. We know the destination is 500 miles away. We also know that the interstate through our city and into the next state has a speed limit of 70 mph. A simple calculation tells us that it would take 7.14 hours to travel 500 miles at 70 mph.

What if you absolutely had to be at the meeting on time? Would you think it’s wise to turn that back-of-the-napkin estimate into a target to which you could commit?

Most people would say it’s insane to expect that would travel into the mountains at 70 mph, the same velocity as the interstate. What’s more, you’d have to take bathroom breaks and food breaks too. You agree with most people.

You decide to email the mailing list for the conference and ask if anyone has ever traveled from your city to the mountain location, and get a response complete with a chart! Your colleague says she kept track of how many miles she traveled during each hour and came up with chart in figure 3 showing that it took just over 9 hours to complete the 500 miles.

Figure 3: Chart showing total number of miles driven after each hour in red and number of miles driven during each hour in blue

If we round up the number of hours traveled to an even 10, we’ll just call this 50 mph. The reasons we cannot travel at 70 mph during the entire trip is simple: mountains are more curvy and dangerous, and we have to break for food and the bathroom. Only after completing the trip one time can we look back and use the experience as a way to gauge future trips through the same or similar terrain.

Let’s take a beginner's look now at how agile teams can use historical data, combined with estimation, to produce better delivery date forecasts. This will be covered in more depth in my next post.

Producing better software delivery date forecasts using simple, empirical estimation techniques

Similarly, if we know our total number of features is 50, and that our velocity is 2 features-per-week, then it’s tempting to calculate that it should take 25 weeks to complete our project.

Alas, software development is rarely as simple as driving down a straight interstate. Just like the journey into the mountains takes us through a variety of terrain and we must take breaks, all software development takes us through all kinds of unexpected requirements. Stakeholders request new features, markets changes, people get hired, people get fired!

And, most importantly, not all features are the same size or complexity. Because of this, agile teams need to take additional steps to bring predictability to delivery schedules. This is usually done with estimation techniques like Wideband Delphi or Planning Poker. These two techniques have been written about by Steve McConnell and Mike Cohn, respectively. (5, 6, 7)

I will cover Planning Poker in more detail in a future post, but the main idea behind it is that the entire team takes a few hours every three weeks to look ahead at the work to be done and places a relative estimate of size or complexity estimate on each item. They then measure how quickly they can complete each item. So instead our simple count of “50 features”, the team might actually have a number such as 150 “points”, which means that, on average, each feature is roughly 3 points of estimated size or complexity. For now, however, let’s continue to focus on tracking how fast the team moves through 50 features.

Agile teams typically use a chart that is drawn from the top down towards zero, which indicates zero more features outstanding! This is called a burndown chart, and a realistic chart might look as follows in figure 4:

Figure 4: Hypothetical burndown chart illustrating how the amount of actual work, in blue, fluctuates up and down as the total number of UNDONE features approaches zero. The initial estimate of 50 features and the target velocity of burning down 2 features per week is shown in red

This chart shows that the team had 50 features remaining to implement at the start of week 0. The initial target velocity of 2 features per week holds up for a few weeks, shown in red, but then it falls off a bit before speeding up to be faster than 2 per week. Not to be outdone, perhaps the business team feels the team can do more work, and new features get added. This causes the time between week 11 and 24 to remain relatively flat before the velocity picks up again.

By the time the initial 50 features are completed, we can calculate that they burned down at a rate of about 1.5 per week. Now, this simple chart does not actually show how many features were added during the duration of the project, though it’s obvious when you see the blue spikes. There are more sophisticated charts that can help illustrate this, but I’ll leave that for next time.

In the meantime, please visit the suggested resources, starting with Mike Cohn’s excellent presentation about Agile Estimation and Planning, to learn more. (1)

Until next time, stay agile not fragile.

References and Resources

“Introduction to Agile Estimation and Planning” – by Mike Cohn, PDF presentation about release planning with agile estimation and planning techniques: http://www.mountaingoatsoftware.com/presentations/106-introduction-to-agile-estimating-and-planning
“Nokia Test : Where did it come from?” – by Jeff Sutherland, about how Nokia uses velocity tracking to assess their teams’ productivity and likelihood to generate future ROI: http://jeffsutherland.com/scrum/2008/08/nokia-test-where-did-it-come-from.html
“How Do We Know When We Are Done?” – by Mitch Lacey, about how his team defined DONE with the whole team’s participation: http://www.scrumalliance.org/articles/107-how-do-we-know-when-we-are-done
“Scrum, et al” – by Ken Schwaber about the history of Scrum, presented at Google : http://www.youtube.com/watch?v=IyNPeTn8fpo
Software Estimation: Demystifying the Black Art – by Steve McConnell, book about lessons learned and best practices for software estimation: http://www.amazon.com/Software-Estimation-Demystifying-Practices-Microsoft/dp/0735605351
Agile Estimation and Planning – by Mike Cohn, book about how to perform agile estimation and planning using simple estimation techniques and short, fixed time-boxed development iterations: http://www.amazon.com/Agile-Estimating-Planning-Mike-Cohn/dp/0131479415
ATL ALT.NET Meetup recorded conversation about Agile Estimation and Planning: http://www.meetup.com/AtlAltDotNet/calendar/9525107/?eventId=9525107&action=detail (direct MP3 link: http://apps.ultravioletconsulting.com/audio/ATLAltDotNet/ATLAltDotNet-2009-01-27-AgileEstimationAndPlanningDiscussion.mp3)

From Waterfall to Agile Development in 15 Minutes: An Introduction for Everyone

2009-07-19T12:59:00.001-07:00

Good, Fast (Now), and Cheap: Pick All Three

New! Listen to the MP3 Podcast version of this article

Company owners, sales professionals, and software development professionals all want to deliver high-quality products and services to their customers. They want to do this more quickly than their competition, reducing their time-to-market. They also want to generate a return-on-investment (ROI) as soon as possible. To top it off, they want to do all of this at a lower cost of development to maximize profitability.

To successfully deliver products and services, software development professionals have designed what are called “software development methodologies”. Essentially, a methodology is a collection of practices, policies, and conventions used to plan, develop, and deliver a software system over the lifecycle of a project.

Article Goal: Use and Understand Both Waterfall and Agile Methods in the Next 15 Minutes

Two of the most widely known and used types of methodologies are broadly classified as “waterfall” and “agile”. The goal of this article is not to provide an exhaustive theoretical treatise on methodologies, but instead to simulate both waterfall and agile by getting you the reader to apply each of them to process of reading the remainder of this article. By doing this, you will experience a microcosm of what it feels like to work with each type of methodology.

First, let’s get some basic definitions. These from Wikipedia will do just fine:

Waterfall software development

The waterfall model is a sequential software development process, in which progress is seen as flowing steadily downwards (like a waterfall) through the phases of Conception, Initiation, Analysis, Design (validation), Construction, Testing and maintenance.

Agile software development

Agile software development refers to a group of software development methodologies based on iterative development, where requirements and solutions evolve through collaboration between self-organizing cross-functional teams. The term was coined in the year 2001 when the Agile Manifesto was formulated.

Let’s use both of them now and compare and contrast them.

First Let’s Try a Waterfall-like Method to Read About Waterfall

In his book Software Estimation, software development legend Steve McConnell states that the first rule of estimation is to “Distinguish between estimates, targets, and commitments”. What tends to happen on many sequential Waterfall projects is that there is no distinction whatsoever.

Instead, what usually happens is that someone, usually a business owner will say, "We need to have this project done in three months, 90 days. We won’t have time for quality assurance on this project until the end, so don’t make many mistakes. Do you think you can make that date?”

The development staff, or, more likely, the appointed technical lead of the development staff will say, “We should be able to make that date.”

What just happened is that the business owner just heard a commitment from the developer. The developer may actually view that date as more of an estimate or a target. But, the damage is done. The owner now believes the product or service will be ready-to-ship in 90 days.

Now, regardless of the complexity of the work, the development staff will begin laboring to meet an arbitrary date chosen only by the owner and the technical lead. What’s worse, when quality assurance is “put off” to the end on projects like this, inevitably the project will run far longer than the desired and committed date. With no formal feedback loop in process until the end, the owner will likely “check in” periodically with the development team and ask the team lead, “How are we looking for meeting the three month date?” The team lead will tell him or her, “We’re doing pretty good”. What else can they say? They have not had any QA performed against their work, so it’s actually impossible for the development team to determine the correctness or completeness of their work!

You Have Exactly 90 Seconds, Now Go

So, to give you a chance to experience how this feels, I’ll play the role of the business owner who operates a web site that offers podcasts and recorded audio about software methodologies. I come to you, my team lead, and say:

“I need you to record yourself reading the following Wikipedia entry about Waterfall software development aloud, completely, and I need you to do it in one-and-a-half minutes, 90 seconds. I want you to read it carefully because we don’t have time for correcting mistakes. We need it ready for release on our downloads section of the web site. Do you think you can do that?”

Get out your stop-watch and time yourself!

Ready, Set, Read Aloud!

The waterfall model is a sequential software development process, in which progress is seen as flowing steadily downwards (like a waterfall) through the phases of Conception, Initiation, Analysis, Design (validation), Construction, Testing and maintenance.

It should be readily apparent that the waterfall development model has its origins in the manufacturing and construction industries; highly structured physical environments in which after-the-fact changes are prohibitively costly, if not impossible. Since no formal software development methodologies existed at the time, this hardware-oriented model was simply adapted for software development.

Owner: “How we doing?”

Team Lead: “Pretty good…”

Owner: “That’s great!”

The first formal description of the waterfall model is often cited to be an article published in 1970 by Winston W. Royce (1929–1995), although Royce did not use the term "waterfall" in this article. Ironically, Royce was presenting this model as an example of a flawed, non-working model (Royce 1970). This is in fact the way the term has generally been used in writing about software development—as a way to criticize a commonly used software practice.

In Royce's original Waterfall model, the following phases are followed in order:

1. Requirements specification
2. Design
3. Construction (AKA implementation or coding)

Owner: “Are we done yet?”

Team Lead: “How much of the last two paragraphs do we need finished before we release? We can’t make 90 seconds…”

Owner: “We need it all. I’m very disappointed, but this means we have to let the date slip. How much longer is this going to take?”

Team Lead: “Shouldn’t be too much longer…”

4. Integration
5. Testing and debugging (AKA Validation)
6. Installation
7. Maintenance

To follow the waterfall model, one proceeds from one phase to the next in a purely sequential manner. For example, one first completes requirements specification, which are set in stone. When the requirements are fully completed, one proceeds to design. The software in question is designed and a blueprint is drawn for implementers (coders) to follow — this design should be a plan for implementing the requirements given. When the design is fully completed, an implementation of that design is made by coders. Towards the later stages of this implementation phase, separate software components produced are combined to introduce new functionality and reduced risk through the removal of errors.

Owner: “How we doing now? Are we ready for QA?”

Team Lead: “Just a little while longer…”

Thus the waterfall model maintains that one should move to a phase only when its preceding phase is completed and perfected. However, there are various modified waterfall models (including Royce's final model) that may include slight or major variations upon this process.

All Washed Up

If you read aloud closer to how I read aloud, then it probably took you between 2 and 3 minutes (120 and 180 seconds) to read this passage. That’s a lot longer than 1.5 minutes or 90 seconds!

No wonder the owner is disappointed. He asked you if you could complete the assignment in 90 seconds. You told him you should be able to do so, and then you failed to deliver.

There’s got to be a better way to guess how long it’s going to take to do something we’ve never done before, right?

Now Let’s Try an Iterative and Incremental (Agile) Method to Read About Agile!

This time, we’re going to do something different to plan our release. As a business owner, I’ve learned my lesson from above and I realize I need more control over my release date and my teams. Because of this, I’m going to work with you to iteratively plan the release, based upon an actual, observed rate of progress. I’m going to call this rate of progress a “Sentence Velocity”. In later articles, we’ll read about software author Mike Cohn’s writings about user stories, story points, and velocity and how those are crucial measures of progress in an Agile world. But, since we’re starting small, we’re just going to worry about one sentence at a time.

Let’s Plan and Measure as We Go This Time

I’ve numbered each sentence or sentence equivalent sequentially. This time, I tell you:

“I want you to read the following passages from http://www.agilemanifesto.org aloud in 20-second iterations. Read the numbers out loud too. I will keep a tally of how many sentences you've gotten through after each iteration. I’ll divide this tally by the number of iterations, and this will tell me your Sentence Velocity per iteration. This way I can forecast how many iterations total it will take you to finish the whole page.”

On Your Mark, Get Set, Go!

Manifesto for Agile Software Development (1)

We are uncovering better ways of developing software by doing it and helping others do it. (2)

Through this work we have come to value: (3)

Individuals and interactions over processes and tools (4)

Working software over comprehensive documentation (5)

Owner looks at backlog, sees the team has read 5 sentences and calculates that it should take 6 iterations to complete, or 120 seconds.

Customer collaboration over contract negotiation (6)

Responding to change over following a plan (7)

That is, while there is value in the items on the right, we value the items on the left more. (8)

Kent Beck

Mike Beedle

Arie van Bennekum

Alistair Cockburn

Ward Cunningham (9)

Owner again inspects the backlog, this time seeing 9 sentences finished, and calculated the pace is now 7 iterations, or 140 seconds. He plans his marketing, sales, training, and operations teams expectations accordingly.

Martin Fowler

James Grenning

Jim Highsmith

Andrew Hunt (10)

Ron Jeffries

Jon Kern

Brian Marick

Robert C. Martin (11)

Steve Mellor

Ken Schwaber

Jeff Sutherland

Dave Thomas (12)

Principles behind the Agile Manifesto (13)

We follow these principles: (14)

Owner observes the backlog and sees the velocity is still 7. Takes no action.

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. (15)

Welcome changing requirements, even late in development. (16) Agile processes harness change for the customer's competitive advantage. (17)

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. (18)

Business people and developers must work together daily throughout the project. (19)

Owner notes velocity is still 7. No action necessary.

Build projects around motivated individuals. (20) Give them the environment and support they need, and trust them to get the job done. (21)

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. (22)

Working software is the primary measure of progress. (23)

Owner notes velocity is still 7. Reminds other teams to prepare for the release.

Agile processes promote sustainable development. (24) The sponsors, developers, and users should be able to maintain a constant pace indefinitely. (25)

Continuous attention to technical excellence and good design enhances agility. (26)

Simplicity--the art of maximizing the amount of work not done--is essential. (27)

Owner notes velocity is still 7. The backlog has only two items now. Owner reminds everyone of the upcoming successful release.

The best architectures, requirements, and designs emerge from self-organizing teams. (28)

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. (29)

The team delivers the project after 7 iterations, as forecasted as early as the second iteration!

Reflection

When I did this in 20-second-iterations, I recorded the following sentence tallies after each iteration:

Suppose I wanted to calculate after the first iteration how many more iterations it would take me to complete. Well, I would simply divide 29 by my current velocity of 5 and get 5 with a remainder of 4. Well, there’s no such thing as a partial interval, so I'd know I need 6 iterations.

What about after 2? My velocity would now be 9 / 2, or 4.5 “Sentence Points” per iteration. 29 divided by 4.5 is 6.444, which rounds to 7 iterations.

My Sentence Velocity and Duration Forecast

Here’s how my sentence velocity and estimated total number of iterations after each iteration:

Iteration	Sentences Completed	Velocity	Est. # Iterations
1	5	5	6
2	9	4.5	7
3	14	4.67	7
4	19	4.75	7
5	23	4.6	7
6	27	4.5	7
7	29	4.14	7

As you can see, by the second iteration, I had a velocity that reliably predicted my actual number of iterations. In reality, requirements change, and velocity goes up and down, but it averages out over time. Mike Cohn recommends taking a moving average to calculate a team's velocity.

In the next article we’ll more about how Agile teams often create a “burndown chart” to visually indicate how much work remains after each iteration.

Conclusion

We’ve reviewed the two most prevalent software methodology styles, waterfall, and agile, and we’ve applied them to the reading process of this very article. In doing so, we first saw that it’s unwise to make a commitment to deliver a complete product on an arbitrary date. Next, we demonstrated a better way of planning by measuring our rate of progress, our velocity, and using that to forecast our project’s duration.

Until next time, stay agile not fragile.

References

The Waterfall model: http://en.wikipedia.org/wiki/Waterfall_model
Agile development: http://en.wikipedia.org/wiki/Agile_software_development
The Agile Poster from VersionOne.com: http://pm.versionone.com/AgilePoster.html

Tactic Pattern 001: Implement a Personal Task Board

2009-04-15T06:12:00.000-07:00

In A Blink
Create a personal task board to create project visibility and nudge company toward Agile and Scrum

User Story Summary
As a developer new to a team you'd like to start introducing Agile and Scrum so that you can improve productivity and quality.

Problem
How do you introduce an Agile practice to a team when you have no Positional Authority to introduce a process improvement initiative?

Forces

Company is small and not a traditional "software company"
Business stakeholders may be "used to" waterfall and unaware of software being developed in other ways
Developer teammates uncertain about Agile or Scrum or have had bad prior experiences
You may not have Positional Authority to introduce a process improvement initiative

Solution
Create a personal task board and use it as an Information Radiator to communicate the priority and status of your own work to the rest of your team and company.

A simple and effective wall will have horizontal headings like this:

[ Backlog / Stories ] [ Task Backlog ] [ Checked Out ] [ QA / Verify ] [ Complete ]

Underneath the Backlog / Stories column, you should write on sticky notes or index cards the high-level requirement or User Stories that you are attempting to finish. Align these in priority order from top to bottom.

Next to that, create sticky notes for the individual tasks which you deem no shorter than 1/2 day and not longer than 2 days that you'll need to do to complete the requirement.

When you start to work on a particular task, move it to the Checked Out column. When you're ready for someone to code review or perform QA against your work, move it to Checked Out.

Finally, move it to Completed when you're finished.

Notice the Task Backlog column should be wider than the others. That is because you should, ideally, only have one item under Checked Out at a time.

Real World Examples
TODO

Agile From The Ground Up!

2009-04-14T14:19:00.001-07:00

Introducing Agile From The Ground Up!
Welcome to the Agile From The Ground Up blog! My name is Josh Gough. I am a software application architect / developer in Atlanta, GA. I've been practicing iterative and incremental development techniques for most of my career. Over the last couple of years, through research and the good fortune of working with highly talented and educated teammates, I've become increasingly exposed to and involved with Agile practices in general and Scrum in particular.

Background
I've helped several companies adopt Agile practices with Scrum from the "bottom up". This means that I as a developer working on the team was part of the impetus to introduce the process improvement efforts of ushering in Agile and Scrum.

Target Audiences
I'll target articles and postings on this toward the following types of readers:

Company executives and product owners seeking an ROI-producing strategy to provide a better way to plan, measure, and execute the delivery of development projects.
Project managers and team leaders seeking an operational approach to successfully introduce Agile and Scrum to their teams or higher-level company executives.
Developers seeking proven, repeatable tactics to nudge their leaders and companies toward adopting Agile and Scrum to improve results.
Students seeking to understand and apply iterative and incremental development practices from a hands-on, real-world perspective.

I hope you will find this information useful as you attempt to help your teams go Agile From The Ground Up!