Blog

03 Oct 2010

Diminishing ROI with NO Automation in Agile development

My interaction with various clients that are in process of transitioning to Agile often ask this question – Is it ok to skip automated Unit and Acceptance testing and only perform manual testing? The reservation normally arises due to old school of thought where manual testing was tolerated and where CLIs and API needed for testing are normally an after thought. Sometimes it is also because QA engineers are manual testers and have no needed programming skills. And at times people tell me Automation takes time off from active feature development and is an overhead.

No matter what is the reasoning, I always preach to these teams, “Automation is a rule and not an exception” and I try to justify my statement using the example below.

The diagram below, “Agile with No Automation” helps illustrate the impact of skipping automation totally, causing diminishing ROI. The goal of each sprint should be a running software that is fully tested, integrated and potentially ready to ship. To meet this goal all features have to be regression tested at end of each sprint.

For the purpose of this example let’s assume Scrum team is able to do the same amount of work each Sprint and each feature is of same size.

• In Sprint 1, the scrum team completed 3 new features without any automated tests. All testing is done manually to meet the sprint goal.
• In Sprint 2, the team is only able to complete 2 features as the time needed to develop the third feature has to be used to perform manual regression testing of 3 features developed in Sprint 1.
• In Sprint 3, the team will be able to complete only one new feature as the time available for remaining features will be used for manual regression testing of 3 features from Sprint 1 and 2 features of Sprint 2 in addition to performing manual testing the new feature delivered in the current sprint, i.e. sprint 3.

As you see in the above example, every new sprint team is spinning wasted cycles doing manual regression testing the old features which could have been totally eliminated if all tests are automated.

Now, let’s consider this scenario where all tests Unit and Acceptance tests are automated each sprint with illustration in the diagram below, “Agile with Automation”.

• In sprint 1, the scrum team completed 3 features and all test are automated. When the team says done, they are indeed done – everything coded, tested, integrated and working software.
• In the following sprint, i.e. sprint 2, the scrum team will be able to complete additional 3 new features as regression testing is automated. Remember, it is machine time as opposed to laborious human time spent running manual regression tests.
• In sprint 3, the team develops 3 additional new features without worrying about regression testing the old features.

Automation gives developers and quality assurance engineers enough confidence to make the necessary changes in the code base thereby generating a potentially shippable product every sprint.