Book Review: Effective REST Services via .NET For .NET Framework 3.5

 
I’m currently reading Kenn Scribner and Scott Seely’s book Effective REST Services via .NET For .NET Framework 3.5. This is a great book which gives a thorough background on the history of REST, quoting often from Roy Fielding’s Ph.D. dissertation on the subject. The authors do not treat the subject with “kid gloves” or “framework gloves”. Instead, they exlain the HTTP protocol and its purpose, benefits, and its drawbacks.

Warning: I’m not really that good at giving book reviews for the general audience, and I feel like there’s already plenty of information on the web about all these subjects, but for own personal benefit it helps me to summarize subjects in my own words to clarify them. For me, it’s best for me to review a table-of-contents and then just give my best shot at summarizing what I’ve read in brief paragraphs using examples. This misses lots of details, of course, and some of what I remember won’t even come from the book.

For me, this is crucial research material for two projects I am working on. One of these is currently public and is a simple RESTful gaming platform. You can see the progress so far on that at http://apps.ultravioletconsulting.com/Skookzee.

So, don’t mistake this as a book review for anyone other than myself, but I hope you enjoy it anyway.

Chapter Summaries

1: RESTful Systems: Back to the Future

 

This chapter discusses how early on web “services” consisted simply of web servers and URIs that delivered chunks of data. Sometimes this was done through CGI processing that responded to the invocation of the incoming request, but eventually most web sites became simply file servers. Later, SOAP-based services became popular, but the problem here was that SOAP services are intrinsically Remote-Procedure-Call centric (RPC). The problem with this is that SOAP services typically encapsulate both NOUNS and VERBS, or Resources and Actions, Objects and Methods –- pick your metaphor – behind a single URI.

You might ask why this is a problem. One reason is that the HTTP and URI protocols were designed, as Roy Fielding describes in his dissertation, to work well in a distributed network and to enable distributed caching. Because SOAP specifically, and RPC in general, deal in coupling an invocation-time command to a remote resource, there is no architectural support to cache the results. Indeed, the idea of caching a call to a SOAP web service is something that each service would have to completely encapsulate.

Conversely, the way the web was actually designed was with a few key principles in mind to enable the distributed, cacheable structure of resources. First of all, the URI stands for Universal Resource Identifier, and a resource is a noun, it’s not an action or a method call. SOAP services are about exposing a single noun, the service entry point, and then encapsulating any number of nouns and verbs behind that one single noun.

So, in a RESTful system, each URI refers to a single, unique representation of a remote resource. For example, http://agilefromthegroundup.blogspot.com/2009/09/done-features-equals-velocity.html refers specifically to the HTML representation of my post about development velocity and tracking. There is, however, another URI which refers to this resource in another way: http://agilefromthegroundup.blogspot.com/feeds/posts/default/4352538909387142862. If you click this link you will get the content of that same blog post, but as XML inside of an “<entry>” element.

A good depiction of this distinction is found in the document Architecture of The World Wide Web on the W3C web site:

This becomes important because of REST’s concept of a single, generic interface, or a limited number of verbs/actions/methods. In terms of HTTP this turns out to be the standard set of HTTP methods: GET, PUT, POST, DELETE, HEAD, OPTIONS. Each of these methods must behave in a consistent, expected way when combined with a requested URI.

Another good reference, for quicker reading, is the summary of the web architecture document. This gives a list of prescriptive good practices for design.

For example, assuming I were authenticated and that Blogger supported this ability (I’m not sure whether it does) then issuing a request with each of the common HTTP methods above against the single URI http://agilefromthegroundup.blogspot.com/feeds/posts/default/4352538909387142862 would produce the following results:

Method	Entity Body Contents	Expected Result
GET	Blank	Returns the HTTP headers for and the current representation of the resource, in this case an XML document. But, does not MODIFY the resource in anyway specified from the request.
HEAD	Blank	Returns only the HTTP headers associated with this resource, but not the XML document itself
OPTIONS	Blank	Returns the list of HTTP methods this resource supports, such as GET, PUT, POST, DELETE
PUT	A complete replacement XML <entry> document	A response code indicating whether the replacement was accepted.
POST	Arbitrary content, perhaps a string of content intended to be added as a comment to the blog post	A response code indicating the result, and possibly the URI of a newly created, subordinate resource, such as a direct URI to the comment added to the entry.
DELETE	Blank	A response code indicating whether the resource was deleted.

There is much more to be said about this chapter, but I leave it to you to read it for yourself and enjoy what you learn.

2: The Hypertext Transfer Protocol and the Universal Resource Identifier

This chapter goes into more detail about the HTTP verbs and the standard response codes and header values, then discusses much more about specific HTTP and REST anti-patterns. These are, broadly:

Anti-Pattern Name	Description	Real-World Examples
GET Tunneling	Specifying method information to the server via GET, which is supposed to NOT modify resources from client-specified commands (idempotency)	Flickr used to have GET methods with commands like “delete” embedded in them. Google’s look-ahead pre-fetch tool caused HAVOC on many user’s accounts as a result!
POST Tunneling	Specifying method information inside of the POST entity body.	SOAP based RPC services are a wholesale violation of this. They embed arbitrary command names inside the envelope.
Misused Content-Types
Miusing Status Codes
Misusing Cache
Cookies
Lack of Hypermedia Support
Lack of Self-Description

 

The concept of URI and Addressability is covered thoroughly. This is so important for so many reasons to get into, but suffice it to say for Search-Engine-Optimization (googleiciousness / bingability) a URI should identify a single resource unambiguously so that search engine indexes can link to it.

Aside from this there is a wide range of computer science concepts wrapped up inside the old trusty URI.

For example, when you look at the URI: http://agilefromthegroundup.blogspot.com/feeds/posts/default/4352538909387142862, you actually have a specification that tells a user agent how to attempt fetching the resource (HTTP protocol), where to start looking for it (agilefromthegroundup.blogspot.com), where within that space to look further (/feeds/post/ etc). What you also have is a contract that is late-bound by definition. The intent of the URI is to provide, of course, a Universal Resource Identifier, which specifies NOTHING about what actually resides behind that identifier. What resides behind it is completely up to the control of the server. It can be absolutely anything. It just so happens to usually be HTML, but of course can be XML, PDF, TXT, etc.

3: Desktop Client Operations

4: Web Client Operations

5: IIS and ASP.NET Internals and Instrumentation

Recommended Resources: Becoming an Expert, Extending Agile, and Individual Improvement

There are three recent presentations posted from Agile 2009 to InfoQ.com that I highly recommend you listen to and learn from. Here are the links with the descriptions:

Mary Poppendieck: Deliberate Practices in Software Development

http://www.infoq.com/presentations/poppendieck-deliberate-practice-in-software-development

Description:

Summary
In the nature vs. nurture debate, researchers have declared nurture the winner. People who excel are the ones who work the hardest; it takes ten+ years of deliberate practice to become an expert. Deliberate practice is not about putting in hours, it’s about working to improve performance. It does not mean doing what you are good at; it means challenging yourself under the guidance of a teacher.
Bio
Mary Poppendieck started her career as a process control programmer, moved on to manage the IT department of a manufacturing plant, and then ended up in product development, where she was both a product champion and a department manager. Mary tried to retire in 1998, but instead found herself managing a government software project where she first encountered the word "waterfall".

Josh Comments:

I’ve listened to Mary’s talks at Google and read most of her Implementing Lean Software Development book recently. This talk is excellent. She discusses not just software, but also music performance and artistic talent development, citing studies that have shown it typically takes about 10,000 focused hours for musicians to truly reach the level of expert, and that many of them who begin early in life reach this number of hours by age 20! Regarding software development, 10 years of working professionally is about 20,000 working hours, but of course not all of those hours are spent crafting software. I’ve been working professionally about 10 years, and I think I’m near that level of expertise in a broadened sense, but have much, much, much more to learn in the depth direction.

 

Allistair Cockburn: I Come to Bury Agile, Not to Praise It

http://www.infoq.com/presentations/cockburn-bury-not-praise-agile

Description:

Summary
Agile came from small, colocated projects in the 1990s. It has spread to large, globally distributed commercial projects, affecting the IEEE, the PMI, the SEI and the Department of Defense. Agile now sits in a larger landscape and should be viewed accordingly. This talk shows that landscape, clarifying how classical agile fits in and what constitutes effective development outside that narrow area.

Bio
Dr. Alistair Cockburn is a world-renowned expert at what is called agile development, the early and regular delivery of business value through improved communications, fast feedback and staged delivery. Dr. Cockburn co-founded the agile development movement, co-authoring the Manifesto for Agile Software Development and the project leadership Declaration of Inter-dependence.

Josh Comments:

This is a great talk which is not really about burying agile, but about recognizing that the basic practices of agile now need to give way to ideas like Software Craftsmanship. He covers much more ground than this, but I’ll just highlight the Software Craftsmanship principles:

“As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:

Not only working software,

              but also well-crafted software

Not only responding to change,

              but also steadily adding value

Not only individuals and interactions,

              but also a community of professionals

Not only customer collaboration,

              but also productive partnerships

That is, in pursuit of the items on the left we have found the items on the right to be indispensable.”

Ashley Johnson and Amr Elssamadisy: Scaling Up by Scaling Down: A (re)Focus on Individual Skills

http://www.infoq.com/presentations/scaling-up-by-scaling-down

Description:

Summary
In this presentation, the causality between performance in the small (individuals and teams) and performance in the large is highlighted and explained. Discover what you can do as an individual regardless of your position in the hierarchy to enable higher performance software development.

Bio
Ashley Johnson splits his time between understanding the challenges that companies face, and consulting with senior IT management to facilitate organizational optimization. Author of the leading book on Agile Adoption, Amr Elssamadisy spends his time helping others learn better ways to develop software and connecting the software development process with its rightful driver - business value.

Josh Comments:

What I like most about this presentation, is how Ashley Johnson incorporates audience participation and experimentation into the course of the presentation. This is the essence of teaching and learning. During my Scrum training with Jeff Sutherland, I was impressed by how Jeff used Scrum to run the training, creating his own task wall with sticky notes which also served as a burndown mechanism. He broke us up into small groups and we worked on problems and exercises that way.

Just to highlight, at one point Ashley asks the audience to break into pairs of two and to come up with a list of “What is required for a high performing team?” This is what the participants came up with on their own:

• Shared vision that matters
• Trust between the team members
• Communication
• Passion
• Empowerment
• Having fun
• Challenging each other while holding each other accountable
• Leadership

D = V * T : The formula in software DeVelopmenT to get features DONE

There’s a hidden formula in software DeVelopmenT that tells how fast a team can get features DONE and Ready-to-Ship.

 

The formula is: D = V * T

It reads as: DONE Features = Velocity multiplied by Time

 

The importance of a software development team’s velocity

The term “velocity” as it applies to software development is simple to explain and to illustrate. Here’s my definition:

Velocity: A team’s velocity is the number of features it can get completely DONE and Ready-to-Ship during a short, fixed time period (2 to 4 weeks)

Velocity is extremely important for business owners and other project stakeholders. Without knowing the velocity of their team, they have no way to reliably plan release dates and coordinate marketing and sales teams. (2) It’s no exaggeration to say that the most important thing a professional software team can do to increase its value to an organization is to become skilled in the arts of estimation and planning. This post introduces the concepts behind  velocity measurement, and provides links for more detailed reading.

 

Are we there yet? Speed racing down the software delivery highway

Building successful software and delivering it on time is an art of numbers. It all boils down to math, like physics or accounting.

Who can forget the familiar high-school formula D = V * T? (also written as D = R * T)

This, of course, is the famous equation for calculating how far you can travel from a given starting point to another when you already know your velocity (or rate) and how long you will be travelling.

 

Distance = Velocity multiplied by Time

For example: if we have know we are traveling at 50 miles per hour and plan to travel for 3 hours, then we know we will travel 150 miles.

 

What happens though if we do not know our velocity, but instead know how far we have traveled and how much time it took to get there? Can we derive our velocity from these other two measurements? Of course we can with simple math. In this case, we have D and T, and can derive V by modifying the formula to be V = D / T.

 

Velocity = Distance divided by Time

For example: if we have traveled 120 miles after 3 hours from point A to point B, then we know our velocity per 1 single hour is 40, or 40 mph. 

 

 

Figure 1: Two US high school math students calculating how far they can travel before returning to math class  in 30 minutes or before being caught by authorities for driving on the wrong side of the road.

 

Measuring velocity in software development to decrease time-to-market and realize faster ROI

I hear what you’re screaming: enough with the PSAT math prep already, how does this apply to releasing software on time? It’s so simple you’ll kick yourself, or your team, for not doing this already.

Agile teams use a formula that works the same way. It’s calculated differently, because most software teams aren’t very mobile while coding, though it would be relaxing to code on a boat.

Because a team cannot reliably know in advance how quickly it can complete a set of features, it must use the second form of the equation to derive its velocity based upon actual observation of progress first.

Thus, the formula for calculating an agile team’s initial velocity still reads as V = D / T, except the D stands for “DONE Features” instead of distance. T, or time, usually stands for 2 to 4 weeks instead of 1 hour. For this article, we’ll assume it means 3 weeks.

 

Initial Velocity = DONE Features divided by Time

For example: if we get 6 features DONE in 3 weeks, then we know our velocity is 6 features per 3 weeks. Simplified, we’ll say 2 features per week.

Here is a simple chart depicting this velocity:

Figure 2: Velocity measurement illustration of six features becoming done during a three-week period

It’s tempting to look at this chart and say the velocity is 2 features per week, and that we can now start using the formula DONE Features = Velocity multiplied by Time to plan ahead. We will use this simplification for the purposes of this article, but keep in mind that this may or may not be true, so be careful! Here are two reasons why:

1. New Requirements Discovered: During the course of any three-week period, teams will discover new requirements frequently. The new requirements could be bugs, change requests from the business team, or important changes required to match the competition. This is a subject for an entire volume on change management!
2. Definition of DONE: It’s extremely important that a team agrees upon what qualifies as a DONE feature. Each team must define what it means by the word DONE. I leave that as an exercise for a future article, but you can find some recommended reading and listening below for reference. (3, 4)

For the rest of this post, we’ll pretend that no new requirements are discovered and we’ll define a feature as DONE if it has successfully passed through each of the following development phases:

1. Requirements Definition
2. Analysis, Design, and sufficient Documentation
3. Coding
4. Unit Testing
5. Code Review (for development standards adherence and security design assessment)
6. Refactoring (to conform to standards and address security deficiencies)
7. Functional Testing
8. User Acceptance Testing (preferably automated)
9. Performance Testing
10. Pilot (beta testing with real or proxy users)
11. Ready-to-Ship

This may sound like a lot of work! And, it certainly is a lot of work. All mission-critical projects consist of a number of features that must go through these steps before they can be considered DONE and Ready-to-Ship.

 

Pitfalls of using early travel velocity to forecast total road trip duration

Returning to our travel example, suppose we are traveling from our city to the mountains for a conference about software estimation and planning. We know the destination is 500 miles away. We also know that the interstate through our city and into the next state has a speed limit of 70 mph. A simple calculation tells us that it would take 7.14 hours to travel 500 miles at 70 mph.

What if you absolutely had to be at the meeting on time? Would you think it’s wise to turn that back-of-the-napkin estimate into a target to which you could commit?

Most people would say it’s insane to expect that would travel into the mountains at 70 mph, the same velocity as the interstate. What’s more, you’d have to take bathroom breaks and food breaks too. You agree with most people.

You decide to email the mailing list for the conference and ask if anyone has ever traveled from your city to the mountain location, and get a response complete with a chart! Your colleague says she kept track of how many miles she traveled during each hour and came up with chart in figure 3 showing that it took just over 9 hours to complete the 500 miles.

 

Figure 3: Chart showing total number of miles driven after each hour in red and number of miles driven during each hour in blue

If we round up the number of hours traveled to an even 10, we’ll just call this 50 mph. The reasons we cannot travel at 70 mph during the entire trip is simple: mountains are more curvy and dangerous, and we have to break for food and the bathroom. Only after completing the trip one time can we look back and use the experience as a way to gauge future trips through the same or similar terrain.

Let’s take a beginner's look now at how agile teams can use historical data, combined with estimation, to produce better delivery date forecasts. This will be covered in more depth in my next post.

 

Producing better software delivery date forecasts using simple, empirical estimation techniques

Similarly, if we know our total number of features is 50, and that our velocity is 2 features-per-week, then it’s tempting to calculate that it should take 25 weeks to complete our project.

Alas, software development is rarely as simple as driving down a straight interstate. Just like the journey into the mountains takes us through a variety of terrain and we must take breaks, all software development takes us through all kinds of unexpected requirements. Stakeholders request new features, markets changes, people get hired, people get fired!

And, most importantly, not all features are the same size or complexity. Because of this, agile teams need to take additional steps to bring predictability to delivery schedules. This is usually done with estimation techniques like Wideband Delphi or Planning Poker. These two techniques have been written about by Steve McConnell and Mike Cohn, respectively. (5, 6, 7)

I will cover Planning Poker in more detail in a future post, but the main idea behind it is that the entire team takes a few hours every three weeks to look ahead at the work to be done and places a relative estimate of size or complexity estimate on each item. They then measure how quickly they can complete each item. So instead our simple count of “50 features”, the team might actually have a number such as 150 “points”, which means that, on average, each feature is roughly 3 points of estimated size or complexity. For now, however, let’s continue to focus on tracking how fast the team moves through 50 features.

Agile teams typically use a chart that is drawn from the top down towards zero, which indicates zero more features outstanding! This is called a burndown chart, and a realistic chart might look as follows in figure 4:

 

Figure 4: Hypothetical burndown chart illustrating how the amount of actual work, in blue, fluctuates up and down as the total number of UNDONE features approaches zero. The initial estimate of 50 features and the target velocity of burning down 2 features per week is shown in red

This chart shows that the team had 50 features remaining to implement at the start of week 0. The initial target velocity of 2 features per week holds up for a few weeks, shown in red, but then it falls off a bit before speeding up to be faster than 2 per week. Not to be outdone, perhaps the business team feels the team can do more work, and new features get added. This causes the time between week 11 and 24 to remain relatively flat before the velocity picks up again.

By the time the initial 50 features are completed, we can calculate that they burned down at a rate of about 1.5 per week. Now, this simple chart does not actually show how many features were added during the duration of the project, though it’s obvious when you see the blue spikes. There are more sophisticated charts that can help illustrate this, but I’ll leave that for next time.

In the meantime, please visit the suggested resources, starting with Mike Cohn’s excellent presentation about Agile Estimation and Planning, to learn more. (1)

Until next time, stay agile not fragile.

 

References and Resources

 

1. “Introduction to Agile Estimation and Planning” – by Mike Cohn, PDF presentation about release planning with agile estimation and planning techniques: http://www.mountaingoatsoftware.com/presentations/106-introduction-to-agile-estimating-and-planning
2. “Nokia Test : Where did it come from?” – by Jeff Sutherland, about how Nokia uses velocity tracking to assess their teams’ productivity and likelihood to generate future ROI: http://jeffsutherland.com/scrum/2008/08/nokia-test-where-did-it-come-from.html
3. “How Do We Know When We Are Done?” – by Mitch Lacey, about how his team defined DONE with the whole team’s participation: http://www.scrumalliance.org/articles/107-how-do-we-know-when-we-are-done
4. “Scrum, et al” – by Ken Schwaber about the history of Scrum, presented at Google : http://www.youtube.com/watch?v=IyNPeTn8fpo
5. Software Estimation: Demystifying the Black Art – by Steve McConnell, book about lessons learned and best practices for software estimation: http://www.amazon.com/Software-Estimation-Demystifying-Practices-Microsoft/dp/0735605351
6. Agile Estimation and Planning – by Mike Cohn, book about how to perform agile estimation and planning using simple estimation techniques and short, fixed time-boxed development iterations: http://www.amazon.com/Agile-Estimating-Planning-Mike-Cohn/dp/0131479415
7. ATL ALT.NET Meetup recorded conversation about Agile Estimation and Planning: http://www.meetup.com/AtlAltDotNet/calendar/9525107/?eventId=9525107&action=detail (direct MP3 link: http://apps.ultravioletconsulting.com/audio/ATLAltDotNet/ATLAltDotNet-2009-01-27-AgileEstimationAndPlanningDiscussion.mp3)