An Introduction to Agile Development

An Introduction to Agile Development

David Farley




Introduction

If there has been one constant in our industry it is the complaint that we are unable to reliably deliver software that is of a suitable quality, at reasonable cost and that meets the needs of our users. This complaint has been voiced since the early 1970’s , and it remains a defining issue of our industry.

Studies of productivity vary, but most seem to agree that for all but the smallest projects more than 70% fail to deliver what they promise on time . Why such a dismal success rate?

Agile developers believe that the processes that underlie most software projects are badly misaligned with the reality of software development.

Ultimately most projects are based on an illusion of science; while they aim to be rigorous, in reality they are largely grounded on qualitative measures.

“Let’s imagine what it would be like to use the system and define our requirements on that basis”

“Let’s guess how many people may use this software and how frequently”

“Let’s imagine a design that will be able to deliver these requirements”

“Let’s guess how long it will take us to build all of this”

All of these activities are speculative; they are attempts to look ahead and predict the future based on the guesses of experienced people. The reason that so many projects attempt to predict the future in this way is fear of the cost of change (See Figure 1). It is commonly understood that the later a change is made in the life of a project the more it costs. Traditional projects mitigate this effect by attempting to eliminate risk by foreseeing problems and identifying solutions to them as early as possible (Figure 1).



Agile development practitioners believe that planning too far ahead is doomed to failure, because, however good we are as software developers, we are not prescient and we will make mistakes. 

Agile processes accept that mistakes will happen, that we will, change our minds, misunderstand something and generally behave like human beings. Instead of mitigating risk by trying to anticipate problems, agile projects are designed to avoid the increase in the cost of change completely. By accepting that change will happen and relying on several techniques to reduce the impact of late changes, agile processes alter the cost of change curve (Figure 2).



There are several techniques used on agile projects to effect this reduction in the cost of change. Instead of taking a qualitative approach to development, agile projects try as much as possible to take a quantitative approach. Our estimates for this week are based on what we did last week. Our progress is measured by how much software is actually working. The usefulness of our software is measured by users using the software and telling us if it meets their needs. 

These are pragmatic, measurable things. Of course, there remain times when we must make our guesses based on experience. Agile projects attempt to reduce the impact of getting those guesses wrong by not extrapolating too far ahead, typically only one or two weeks ahead.

Agile proponents try to avoid being too proscriptive about the nature of the processes used – people over process is one of our tenets. We aim to tailor the details of process to the needs of each project and team individually. The process described here is common to many of the projects my company works on and is in part a form of Extreme Programming as described by Kent Beck. Typically our projects have other attributes culled from other agile approaches, but they are beyond the scope of this introductory article.

The remainder of this article describes some of the techniques that allow us to modify the cost of change curve, and establish more quantitative development practices.

Most people can imagine the software system that they want, but it is very difficult to explain such a vision in an unambiguous way. We could try to be more specific about requirements, or more rigorous in our description of how things should be, but a more agile approach is to ensure that the people who understand a problem talk to the people who know how to find a solution.

We write stories about uses of the system to capture small interactions that describe what the user wants. These stories are written in plain English, describing the problem to be solved. The stories don’t attempt to describe how the problem should be solved, and they don’t attempt to describe every detail of the problem.

Requirements

Requirements, expressed as stories, aren’t enough to write code from, they don’t contain enough detail. Instead they represent place-holders for a conversation between a developer and someone who will use the system, or someone who represents the users of the system. 

Stories are key aspects of our approach to agile projects. They represent the smallest piece of business value that makes any sense to the users of the software, and they are the currency of the project. Stories identify requirements, define units of development, are the sole measure used to track the progress of the project and represent the scope of provable deliverable behaviour of the system. 

We often use stories in the form shown here, but the form isn’t proscribed:

e.g.
Retrieve Contact Details
As a salesman I would like to be able to retrieve contact details so that I can make a sales-call with the details to hand.
I will know that I have achieved that when:
I can retrieve the details for a contact where I specify their name,
I can see the details of the contact for the contact(s) I have found.

Requirements in this form are easy to understand by everyone. Any developer can build something that does this without the need for a more detailed specification if they are allowed to have a conversation with a user to clarify what is really needed and they can use the success criteria described in the story as a specification for tests to write to prove that the story meets the requirements.

Once a story has been developed the behaviour is demonstrated to users. At this point the users are free to change their minds. Perhaps the business circumstances have changed,. Perhaps now that the users can see the real software, they perceive the problem differently and they have some new ideas. Perhaps we misunderstood the requirements and developed the wrong thing. In agile projects we accept that all of these changes may happen, and we structure our projects to not only support them, but to expect them, at any stage. Our processes allow people to be humanly fallible!

Documentation

People are the key to software development. Regardless of tools, technologies or techniques, software development is a human activity. Choosing the right people, ensuring that teams work effectively together and encouraging human interactions is what ensures that good software is the result. 

Agile projects encourage people to work together collaboratively and to share responsibility for finding solutions to problems. They avoid processes that encourage the use of documents, diagrams or models that get in the way of human interactions. This is not to suggest that agile developments are document free; they are not. However, the aim is to produce the documents that are useful, and to avoid spending lots of effort developing documents that are either rarely or never used.

One key piece of ‘documentation’ of the behaviour of the system is the collection of automated tests. Software developed in an agile project often has a much higher degree of automated testing than software developed by more traditional processes. This is a result of the pragmatic, quantitative approach – software that passes the tests is finished; software that has no tests is not because we have no measure to show that it is finished.

This body of tests document both the behaviour of the system and its design. We generally find that in helping us to identify problems later in the life of a project, the tests are much more valuable than most design documents. The tests are guaranteed to be in step with the code, and we can therefore use them to confirm that any changes that we make preserve the existing behaviour and design of the system.

Tracking Progress

Stories are used as the atomic unit of progress in agile projects. In more traditional projects, progress is often measured by how many tasks are complete, or progress against a particular task; 

“I am 80% finished on this task, and this task represents 5% of the total project so we are 4% of the way through the entire project”. 

These are very difficult things to measure accurately. Any developer’s estimate of effort remaining is by definition subjective, and the break-down of tasks is based on estimates and design assumptions that are again only qualitative. This is a house of cards on which to build planning assumptions.

In agile projects we measure software in terms of stories delivered – functionally complete, fully tested, and deployed into production (or ready to be deployed into production).

We estimate for small pieces of work over short durations; we do it frequently, every iteration, and we collect statistics on our accuracy. This means that we become practiced, and so consistent, in our estimates. Consistency is more important than accuracy. If we are consistent, even consistently bad, we can use the measures of our accuracy to allow us to understand, and predict over short periods, the rate at which we can build working software.

This approach to tracking has several important implications:

· We can’t sustain lengthy up-front analysis and design phases, because the delay between this and working software, our only measure of real progress, is too long, allowing the cost of change to increase. Instead we must find a way to evolve our designs as we go.

· The systems that we build must be working all the time so that we can avoid nasty surprises. This means that we must be able to prove that any changes we make don’t break existing behaviour and we must be able to reliably produce a working build all the time.

· In consequence, we take automated testing very seriously. Automated testing on agile projects is widespread and pervasive. It provides the principal defence against increases in the cost of change. Our ability to run regression tests quickly and effectively gives us confidence that the code we add or change does not compromise the fitness for release of the software.

· We must build the system often, and we will probably be deploying the system often too, which implies that we need to become very good at build and deployment, usually automating as much as possible to achieve a reproducible, reliable process. 

We use processes like Iterative development, Continuous Integration and Test Driven Development to help us achieve these goals.

Author Email-id: dfarley@thoughtworks.com