Comments on: QA and Testing in an Agile Environment http://edgehopper.com/qatesting-in-an-agile-environment/ Brain Droppings on Innovation, Creativity, and Collaboration Sun, 08 Jan 2012 06:03:48 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Corey Baker http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-2/#comment-7789 Corey Baker Wed, 18 May 2011 21:37:27 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-7789 To me, the fundamental tenets of Agile is that resources are cross-functional. This implies that estimates on a story can be done by a single development "unit" that is dedicated to the story until completion. These estimates involve all that is required to develop and ship code, namely requirements vetting, development, and testing. (Note: I have "unit" in quotes, because that will differ based upon organization, automation maturity, defect tolerance, etc.) Once you break-out the responsibilities of work, you no longer can fulfill the development requirements of a story (vetting/development/testing) via a single development unit. At this point, you have broken the development process into multiple sub-processes with implicit dependencies. Once that occurs, you have no capability of force-fitting these multiple, intra-dependent into a set cycle AND maximize resource usage. That's not the way the requirements/code/testing cycle works and trying to accommodate these cycles with intra-sprint schedules is defeating the purposes of Agile. So, what is one to do in the typical organization in which software development and testing is split? IMO, you have three choices: 1. Create a development unit that CAN take care of everything required to deliver code AND keep them dedicated to a story. In most places, that would mean a tester/developer pair together until a story is complete. Unfortunately, this will often result in alternating downtime as the code/test cycles fluctuate back-and-forth. 2. Stagger QA and development, admitting the testing is dependent upon development completion. This requires embracing that "done" has different connotations for different stages of the development process. This shouldn't be an issue by many think it's anathema to Agile. I argue that, in most organizations, even after testing there is a release preparation step required before code is deployed. Thus, even if code is tested successfully, it isn't "done," there are still pieces of the process that remains incomplete. Embracing software development as a more complex process is key to this strategy. 3. Force intra-iteration schedules to try and allow for time needed to manage the ebb and flow of the code/test cycles. This is the suggestion of this post and I believe it can be successful if you have backlog of other activities that can be worked upon during downtime. Basically, this strategy is to use non-standard development activities (hardening of architecture, for example) to fill the gaps caused by the dependencies between testing and coding. Anybody using anything other than these three in shops where testers do not develop as well? To me, the fundamental tenets of Agile is that resources are cross-functional. This implies that estimates on a story can be done by a single development “unit” that is dedicated to the story until completion. These estimates involve all that is required to develop and ship code, namely requirements vetting, development, and testing.

(Note: I have “unit” in quotes, because that will differ based upon organization, automation maturity, defect tolerance, etc.)

Once you break-out the responsibilities of work, you no longer can fulfill the development requirements of a story (vetting/development/testing) via a single development unit. At this point, you have broken the development process into multiple sub-processes with implicit dependencies. Once that occurs, you have no capability of force-fitting these multiple, intra-dependent into a set cycle AND maximize resource usage. That’s not the way the requirements/code/testing cycle works and trying to accommodate these cycles with intra-sprint schedules is defeating the purposes of Agile.

So, what is one to do in the typical organization in which software development and testing is split? IMO, you have three choices:

1. Create a development unit that CAN take care of everything required to deliver code AND keep them dedicated to a story. In most places, that would mean a tester/developer pair together until a story is complete. Unfortunately, this will often result in alternating downtime as the code/test cycles fluctuate back-and-forth.

2. Stagger QA and development, admitting the testing is dependent upon development completion. This requires embracing that “done” has different connotations for different stages of the development process. This shouldn’t be an issue by many think it’s anathema to Agile. I argue that, in most organizations, even after testing there is a release preparation step required before code is deployed. Thus, even if code is tested successfully, it isn’t “done,” there are still pieces of the process that remains incomplete. Embracing software development as a more complex process is key to this strategy.

3. Force intra-iteration schedules to try and allow for time needed to manage the ebb and flow of the code/test cycles. This is the suggestion of this post and I believe it can be successful if you have backlog of other activities that can be worked upon during downtime. Basically, this strategy is to use non-standard development activities (hardening of architecture, for example) to fill the gaps caused by the dependencies between testing and coding.

Anybody using anything other than these three in shops where testers do not develop as well?

]]> By: Ann Konkler http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-2/#comment-1380 Ann Konkler Sun, 14 Dec 2008 19:51:43 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-1380 James, let me attempt to clarify... Devs and QA are busy throughout the iteration. By limiting the story/feature size to no more than "5 points", it ensures that DEV will have multiple stories in the pipeline ready for QA and that there's time to fix any defects. Goes something like this: DEV applies TDD to story 1, they 'stamp it' which signals QA to perfom more robust integration testing, acceptance testing, etc on that particular story. While QA has story 1, DEV begins story 2, stamps it as a signal to QA, and then DEV starts story 3, and so on. Meanwhile, QA may have found bugs in story 1 and story 2; these stories then get 'passed' back to DEV (added to the iteration backlog) and once fixed, they go back to QA for another pass through the cycle until QA signs off on each story. By the end of the iteration, each story may have gone through a couple cycles of DEV (TDD) - QA (find defects) - DEV (fix defects) - QA (validate fix). May sound more waterfall than it is -- DEVs and QA are together as one team, looking at the same backlog, and often pairing together to ensure stories are completed within the iteration. James, let me attempt to clarify…

Devs and QA are busy throughout the iteration. By limiting the story/feature size to no more than “5 points”, it ensures that DEV will have multiple stories in the pipeline ready for QA and that there’s time to fix any defects. Goes something like this: DEV applies TDD to story 1, they ‘stamp it’ which signals QA to perfom more robust integration testing, acceptance testing, etc on that particular story. While QA has story 1, DEV begins story 2, stamps it as a signal to QA, and then DEV starts story 3, and so on. Meanwhile, QA may have found bugs in story 1 and story 2; these stories then get ‘passed’ back to DEV (added to the iteration backlog) and once fixed, they go back to QA for another pass through the cycle until QA signs off on each story. By the end of the iteration, each story may have gone through a couple cycles of DEV (TDD) – QA (find defects) – DEV (fix defects) – QA (validate fix). May sound more waterfall than it is — DEVs and QA are together as one team, looking at the same backlog, and often pairing together to ensure stories are completed within the iteration.

]]> By: Bob Marshall http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-2/#comment-1203 Bob Marshall Wed, 26 Nov 2008 21:50:58 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-1203 @Carlton. Personally, I think that if a team can't do two week iterations, I'd be asking why not (i.e. root cause analysis using the 5 Whys technique, most likely). If it's down to intractable external factors (i.e. not immediately fixable within and by the team) then I suspect that the root causes would suggest iterations longer than two weeks, not shorter. Two-week or shorter iterations can result in stress, it's true, but that generally results from trying to cram in too much into an iteration. Establishing an early indication of team velocity, and then using that along with some more-or-less consistent sizing of backlog items, really helps in countering that tendency. The disadvantages of longer iterations include: - More work in progress (inventory) - Reduced focus (more things to test, work on, ship at the end of the iteration) - Longer (time) between injection of defects and finding them (making process improvement less effective) - generally extension to the mean cycle time - etc Of course, shorter iterations have some disadvantages too - I've found the optimal balance point to generally rest around the two week mark. Actually, these days I think that iterations are a half-way house for people just getting to grips with Agile. For me, single-piece continuous flow is the way to go for more mature teams and organisations (c.f. FlowChain - http://www.linkedin.com/e/gis/121933 ) HTH! - Bob On 11/5/08 6:39 AM, Chris Spagnuolo asked: -------------------- How do you fit QA and testing into an iteration? Lately I've been wrestling with the question of how QA and testing fits into an iteration while keeping both the QA's and the devs effectively utilized. After some thought, I've come up with a "schedule" of activities for the QA's and devs that I think works. Essentially, QA's start the iteration on day one writing test cases (and testing the test cases in the negative) while the devs are writing code. In the middle of the iteration, QA conducts tests on completed code while the devs continue coding and fixing defects uncovered by the tests. In the final days of the iteration, the devs stop writing new code and focus on fixing defects, helping the product owner refine/define upcoming stories, and thinking about design considerations of upcoming stories while QA tests ALL of the code written during the iteration. It's just one way of thinking through this problem. So, my question to the group is, how do you currently address QA/testing in your iterations? And, is it even part of your definition of done? If you want to read more details and see a graphic I've developed around this idea, check out my latest blog post at http://edgehopper.com/qatesting-in-an-agile-environment/ @Carlton.

Personally, I think that if a team can’t do two week iterations, I’d be asking why not (i.e. root cause analysis using the 5 Whys technique, most likely). If it’s down to intractable external factors (i.e. not immediately fixable within and by the team) then I suspect that the root causes would suggest iterations longer than two weeks, not shorter. Two-week or shorter iterations can result in stress, it’s true, but that generally results from trying to cram in too much into an iteration. Establishing an early indication of team velocity, and then using that along with some more-or-less consistent sizing of backlog items, really helps in countering that tendency. The disadvantages of longer iterations include:
- More work in progress (inventory)
- Reduced focus (more things to test, work on, ship at the end of the iteration)
- Longer (time) between injection of defects and finding them (making process improvement less effective)
- generally extension to the mean cycle time
- etc

Of course, shorter iterations have some disadvantages too – I’ve found the optimal balance point to generally rest around the two week mark.

Actually, these days I think that iterations are a half-way house for people just getting to grips with Agile. For me, single-piece continuous flow is the way to go for more mature teams and organisations (c.f. FlowChain – http://www.linkedin.com/e/gis/121933 )

HTH!

- Bob

On 11/5/08 6:39 AM, Chris Spagnuolo asked:
——————–
How do you fit QA and testing into an iteration?

Lately I’ve been wrestling with the question of how QA and testing fits into an iteration while keeping both the QA’s and the devs effectively utilized. After some thought, I’ve come up with a “schedule” of activities for the QA’s and devs that I think works. Essentially, QA’s start the iteration on day one writing test cases (and testing the test cases in the negative) while the devs are writing code. In the middle of the iteration, QA conducts tests on completed code while the devs continue coding and fixing defects uncovered by the tests. In the final days of the iteration, the devs stop writing new code and focus on fixing defects, helping the product owner refine/define upcoming stories, and thinking about design considerations of upcoming stories while QA tests ALL of the code written during the iteration. It’s just one way of thinking through this problem. So, my question to the group is, how do you currently address QA/testing in your iterations? And, is it even part of your definition of done?

If you want to read more details and see a graphic I’ve developed around this idea, check out my latest blog post at http://edgehopper.com/qatesting-in-an-agile-environment/

]]> By: Adam Koszlajda http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-2/#comment-1096 Adam Koszlajda Sun, 23 Nov 2008 18:06:57 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-1096 Chris, My experiences are quite similar to these mentioned above. You have actually two possible approaches depending on organization and projects... Option A That is a perfect situation if you have testers within your team, which writes first the test scenarios or at least you have ready to use UATs. It is quite close to what Gunther writes. At this case it is just enough to establish daily builds and proper communication between developers and fully-committed testers (eg. via Bug Tracking System and organizing Triage Sessions). Unfortunatelly there are still companies, which "can not afford testers" and projects, which does not fit to this model. Option B QA is mostly on developers shoulders and they should write number of Unit Tests. It is quite close to what Tim writes. It is also a good approach to leave them 1-3 days at the end of sprint, so they can stabilize the project and double check that after integration everything works well together. Then you can engage the analyst or the customer who has 1-3 days between sprints to double check the functionality developed by programmers. The time between sprints is necessary to see how many bugs are within your code and estimate how much time will be needed in next sprint to fix them. In each option it is strongly related that the code created by the programmer is reviewed, by the other before it is checked in. Pair programming provides it, but my experience is that Peer programming much better approach :D Chris,

My experiences are quite similar to these mentioned above. You have actually two possible approaches depending on organization and projects…

Option A
That is a perfect situation if you have testers within your team, which writes first the test scenarios or at least you have ready to use UATs. It is quite close to what Gunther writes. At this case it is just enough to establish daily builds and proper communication between developers and fully-committed testers (eg. via Bug Tracking System and organizing Triage Sessions). Unfortunatelly there are still companies, which “can not afford testers” and projects, which does not fit to this model.

Option B
QA is mostly on developers shoulders and they should write number of Unit Tests. It is quite close to what Tim writes. It is also a good approach to leave them 1-3 days at the end of sprint, so they can stabilize the project and double check that after integration everything works well together. Then you can engage the analyst or the customer who has 1-3 days between sprints to double check the functionality developed by programmers. The time between sprints is necessary to see how many bugs are within your code and estimate how much time will be needed in next sprint to fix them.

In each option it is strongly related that the code created by the programmer is reviewed, by the other before it is checked in. Pair programming provides it, but my experience is that Peer programming much better approach :D

]]> By: Carlton Nettleton http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-2/#comment-910 Carlton Nettleton Wed, 19 Nov 2008 01:20:09 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-910 @Bob I am with you that we should have a preference for shorter cycles. I used to think that if a team could not do 2 week iterations, they should do 1 week iterations. Now I am not so sure. In many organizations I have observed this sort of rapid cycling is just thrashing for them and more destructive overall - the work is not done, the quality level is in the can and people end up burned out from the stress. If we were to extend the iteration from 2 weeks to 4 weeks, what would we have to do to make it a success? @Bob

I am with you that we should have a preference for shorter cycles. I used to think that if a team could not do 2 week iterations, they should do 1 week iterations. Now I am not so sure. In many organizations I have observed this sort of rapid cycling is just thrashing for them and more destructive overall – the work is not done, the quality level is in the can and people end up burned out from the stress. If we were to extend the iteration from 2 weeks to 4 weeks, what would we have to do to make it a success?

]]> By: Alan Griffiths http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-874 Alan Griffiths Tue, 18 Nov 2008 14:12:18 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-874 It has already been mentioned that QA and Testing are part of the development process and thus part of the iteration, but one approach that hasn't been mentioned is to have the testers pair with the developers and evolve the tests and code for each story together. It has already been mentioned that QA and Testing are part of the development process and thus part of the iteration, but one approach that hasn’t been mentioned is to have the testers pair with the developers and evolve the tests and code for each story together.

]]> By: Bob Marshall http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-843 Bob Marshall Mon, 17 Nov 2008 22:13:56 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-843 Sorry, Carlton. Have to disagree. Of course, if a Agile team has no influence over what goes on "over the fence" and there's no one in the wider organisation concerned with (or accountable for) concept-to-cash cycle times, then the team may have to accept the sub-optimisation due to the "impossibility" of quick deployment. I have run projects for many years very successfully on a two-week cycle - even for embedded systems development where the hardware and software evolves together, and have recently seen teams leveraging modern languages (e.g. Ruby, Python) and tools to run cycles as short as four *hours*! And yes, these folks' output is "done, done". When Shingeo Shigo first proposed S.M.E.D., people though it was "impossible" too. See: http://en.wikipedia.org/wiki/SMED HTH Sorry, Carlton. Have to disagree.

Of course, if a Agile team has no influence over what goes on “over the fence” and there’s no one in the wider organisation concerned with (or accountable for) concept-to-cash cycle times, then the team may have to accept the sub-optimisation due to the “impossibility” of quick deployment. I have run projects for many years very successfully on a two-week cycle – even for embedded systems development where the hardware and software evolves together, and have recently seen teams leveraging modern languages (e.g. Ruby, Python) and tools to run cycles as short as four *hours*! And yes, these folks’ output is “done, done”.

When Shingeo Shigo first proposed S.M.E.D., people though it was “impossible” too. See: http://en.wikipedia.org/wiki/SMED

HTH

]]> By: Rolf Barbakken http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-787 Rolf Barbakken Sun, 16 Nov 2008 20:30:12 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-787 I think it's important to remember that 1. Your developers should not be the testers. They should test their own code, of course, but the code/release should also be tested by someone else. As a programmer you should know you are probably not the best to find faults with your own work. TDD is good, but will not solve such problems. 2. QA is not (just) testing code. Carlos is right here. 3. QA and QC should be a continous task, and fits very easily into any development cycle if done correctly. With cross-functional teams, both QA and QC has a natural part in it. Sadly, many developers still feel QA/QC is annoying and "its easy to point at faults". As a leader, you should make sure the team take ownership of their deliveries as a team. I think it’s important to remember that

1. Your developers should not be the testers. They should test their own code, of course, but the code/release should also be tested by someone else. As a programmer you should know you are probably not the best to find faults with your own work. TDD is good, but will not solve such problems.

2. QA is not (just) testing code. Carlos is right here.

3. QA and QC should be a continous task, and fits very easily into any development cycle if done correctly. With cross-functional teams, both QA and QC has a natural part in it.

Sadly, many developers still feel QA/QC is annoying and “its easy to point at faults”. As a leader, you should make sure the team take ownership of their deliveries as a team.

]]> By: Carlos Simonini http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-780 Carlos Simonini Thu, 13 Nov 2008 19:31:30 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-780 I agree with the idea of cross-functional teams. Even so, take into account that the members in the team have different skills. A Quality Assurance role (QA) is mixed with a Business Analyst role or with a developer role. By the way, QA and QC (Quality Control/Testing) are not the same thing in a waterfall aproach. QA is focus on prevention and QC is focus on detection. As a QA Analyst who works in an Agile frame, I've been included in the whole process, and, in my opinion, testers should be included in the whole process within the Agile practices too. In that way, the QA/Tester prevents issues because of the misinterpretation of the requeriments. I agree with the idea of cross-functional teams. Even so, take into account that the members in the team have different skills.
A Quality Assurance role (QA) is mixed with a Business Analyst role or with a developer role.

By the way, QA and QC (Quality Control/Testing) are not the same thing in a waterfall aproach. QA is focus on prevention and QC is focus on detection.
As a QA Analyst who works in an Agile frame, I’ve been included in the whole process, and, in my opinion, testers should be included in the whole process within the Agile practices too. In that way, the QA/Tester prevents issues because of the misinterpretation of the requeriments.

]]> By: Carlton Nettleton http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-769 Carlton Nettleton Thu, 13 Nov 2008 04:19:49 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-769 Make the iterations longer. I have noticed that Agile has been trending towards shorter and shorter iterations, but there is a reason why Scrum started (and still has) 30 day Sprints - it allows for a complete piece of work to be DONE during the span of the timebox. In many legacy systems in large enterprises, it is simply "not possible" to get something analyzed coded, tested and deployed (and retested) in 2 weeks. Changing the cycle to 30 days improves your chance of having it "done, done". Make the iterations longer. I have noticed that Agile has been trending towards shorter and shorter iterations, but there is a reason why Scrum started (and still has) 30 day Sprints – it allows for a complete piece of work to be DONE during the span of the timebox. In many legacy systems in large enterprises, it is simply “not possible” to get something analyzed coded, tested and deployed (and retested) in 2 weeks. Changing the cycle to 30 days improves your chance of having it “done, done”.

]]> By: Therese Schoch http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-675 Therese Schoch Mon, 10 Nov 2008 20:34:54 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-675 In my Agile experience the QA team is involved in the iteration planning and estimating from day one and walks in stride with Dev. At the end of a 2 week iteration and/or when Dev is complete with new stories and moving on to the next iteration. QA is in the process of testing all functionality as well as overall performance testing. However, it is fair to say, (at least in my experience), that inevitably a flood of work comes at the tail end of an iteration. Sometimes this is due to defects found and sometimes it is due to the lack of planning and/or estimating early on in the iteration. The key to our team's success is communication, effective planning and accurate estimating. While we all know we can under or over estimate we seem to always pull together to meet our date and deliver quality sotware to our business customers. :-) I have also seen that every IT shop differs and suggest that tailoring your Agile methodology for your company needs is the best approach to take. It may take some experimenting yet find what works for you so you can deliver a quality product to your business customer in a cost effective manner. In my Agile experience the QA team is involved in the iteration planning and estimating from day one and walks in stride with Dev. At the end of a 2 week iteration and/or when Dev is complete with new stories and moving on to the next iteration. QA is in the process of testing all functionality as well as overall performance testing. However, it is fair to say, (at least in my experience), that inevitably a flood of work comes at the tail end of an iteration. Sometimes this is due to defects found and sometimes it is due to the lack of planning and/or estimating early on in the iteration.
The key to our team’s success is communication, effective planning and accurate estimating. While we all know we can under or over estimate we seem to always pull together to meet our date and deliver quality sotware to our business customers. :-)
I have also seen that every IT shop differs and suggest that tailoring your Agile methodology for your company needs is the best approach to take. It may take some experimenting yet find what works for you so you can deliver a quality product to your business customer in a cost effective manner.

]]> By: Sharan Karekatte http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-655 Sharan Karekatte Mon, 10 Nov 2008 16:32:22 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-655 Development without dedicated QA is dangerous, and though it may seem to work well initially quality always suffers in the long term. I'm talking about a well integrated team with absolutely no 'throwing over the wall'. 'Over the wall' seems so 'waterfall'. QA in our teams not only test but also look at the whole Scrum process and suggest process improvements. They also analyze user stories and acceptance criteria to determine value-addition, and question the Product Owners on them to ensure validity and ROI. Developers also practice TDD and write extensive unit tests. Development without dedicated QA is dangerous, and though it may seem to work well initially quality always suffers in the long term. I’m talking about a well integrated team with absolutely no ‘throwing over the wall’. ‘Over the wall’ seems so ‘waterfall’. QA in our teams not only test but also look at the whole Scrum process and suggest process improvements. They also analyze user stories and acceptance criteria to determine value-addition, and question the Product Owners on them to ensure validity and ROI. Developers also practice TDD and write extensive unit tests.

]]> By: Liza Ivinskaia http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-650 Liza Ivinskaia Mon, 10 Nov 2008 14:36:28 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-650 Hi everyone, The best approach I’ve worked with was: 1.When discussing features (we did it at sprint planning meetings) we defined which user acceptance test cases, performance tests etc should be done 2.In the beginning of the sprint we together with developers identified which test cases may be automated and automated them with help of our test tool. 3.After that developers coded and run automated tests. At the same time testers prepared manual test cases, test data, test environment etc. 4.We did not wait with delivery to test until all the features are done. As soon as developers had something that passed through its user acceptance tests they gave it to us so that we could do manual tests, exploratory tests, performance tests etc. 5.We “closed” delivery to test "gates" 3-4 days before the sprint end so that testers could finish all activities 6.We were done when we passed all the test cases we planned. We used this approach in maintenance there we often did not have more than one-two iterations before production deployment and it helped us to be in time with test activities. I believe that the same approach with combination when you have one sprint at the end of the project for test completion (exploratory testing, performance testing, test reporting etc) may be used in other development projects even if you do not have automated tests (but in this case you have to deal with that things will take longer time). Hi everyone,

The best approach I’ve worked with was:

1.When discussing features (we did it at sprint planning meetings) we defined which user acceptance test cases, performance tests etc should be done

2.In the beginning of the sprint we together with developers identified which test cases may be automated and automated them with help of our test tool.

3.After that developers coded and run automated tests. At the same time testers prepared manual test cases, test data, test environment etc.

4.We did not wait with delivery to test until all the features are done. As soon as developers had something that passed through its user acceptance tests they gave it to us so that we could do manual tests, exploratory tests, performance tests etc.

5.We “closed” delivery to test “gates” 3-4 days before the sprint end so that testers could finish all activities

6.We were done when we passed all the test cases we planned.

We used this approach in maintenance there we often did not have more than one-two iterations before production deployment and it helped us to be in time with test activities.

I believe that the same approach with combination when you have one sprint at the end of the project for test completion (exploratory testing, performance testing, test reporting etc) may be used in other development projects even if you do not have automated tests (but in this case you have to deal with that things will take longer time).

]]> By: Keith Sterling http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-603 Keith Sterling Sun, 09 Nov 2008 15:43:18 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-603 Hi Don, does that mean your developers are not writing ANY unit tests. I'd be interested in to know a) How many testing resources this is taking up, do you have a 1:1 ratio b) Do you find you still are completing stories within an iteration c) How do the developers know when they have delivered a story ( i.e when to stop coding ) d) Can you expand further on what you mean by 90% success rate from the dev team Hi Don, does that mean your developers are not writing ANY unit tests. I’d be interested in to know
a) How many testing resources this is taking up, do you have a 1:1 ratio
b) Do you find you still are completing stories within an iteration
c) How do the developers know when they have delivered a story ( i.e when to stop coding )
d) Can you expand further on what you mean by 90% success rate from the dev team

]]> By: Don Clarke http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-538 Don Clarke Fri, 07 Nov 2008 20:07:05 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-538 One of the little things that we have implemented and has improved development productivity for us is that we have pulled unit testing away from the devleopers. Our process works as such Developer works on stories that on their schedule for the sprint. Developer completes his story and assigns it to a tester that is dedicated to the development team. Tester unit tests the functionality and either approves or documents defects and assigns back to the developer if approved the stories are approved for release within the sprint if defects are present it is assigned back to the developer with the expectation that he resolve the defects within the sprint. We are experiencing a 90%+ success rate from the dev team since we implemented this process One of the little things that we have implemented and has improved development productivity for us is that we have pulled unit testing away from the devleopers. Our process works as such

Developer works on stories that on their schedule for the sprint.
Developer completes his story and assigns it to a tester that is dedicated to the development team.
Tester unit tests the functionality and either approves or documents defects and assigns back to the developer
if approved the stories are approved for release within the sprint
if defects are present it is assigned back to the developer with the expectation that he resolve the defects within the sprint.

We are experiencing a 90%+ success rate from the dev team since we implemented this process

]]> By: Patrice Davan http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-536 Patrice Davan Fri, 07 Nov 2008 19:54:19 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-536 Hi Chris, It would depends on how you position your QA and the complexity of the project: is your focus on Component integration/Sub system test or more on System Integration/Product acceptance. By complexity of the project I meant multi-tier application or application doing lot of third party services integration. Component Integration is indeed at the frontieer between Dev and QA in term of ownership for testing. For this type of validation there are strong "QA" requirements for test data set, tools, mock-up/simulator to execute testing. Typically those works is strongly linked to the Test plan creation and could be started from the begining of the sprint. Development of those tools should be "delivered" with the Continuous Integration framework in // of the code. Regarding the System Integration - and I know some will say it is not Agile ;-) - my view is it would mainly be in the second week of your sprint (although it could start earlier).. Typically performance, stability, user acceptance testing would require several days of a stable "final" release even if some of this was done as "exploratory" testing during the first week of your sprint. In term of QA having nothing to do at the begining of the sprint I totaly disagree as they have sevaral "high criticality" tasks, amongst others: - work with stakeholders to ensure we can create test plan that mapped the acceptance criteria and - by experience - it is by doing this we often find flaw in the requirements and the acceptance criteria :-) - anticipate needs in term of libraries and tools for automation - set-up test environment - work on test data set and tools to generate them - define and implement mockup/simulator As being part of the definition of Done: Yes :-) Hi Chris,

It would depends on how you position your QA and the complexity of the project: is your focus on Component integration/Sub system test or more on System Integration/Product acceptance.

By complexity of the project I meant multi-tier application or application doing lot of third party services integration.

Component Integration is indeed at the frontieer between Dev and QA in term of ownership for testing. For this type of validation there are strong “QA” requirements for test data set, tools, mock-up/simulator to execute testing.
Typically those works is strongly linked to the Test plan creation and could be started from the begining of the sprint. Development of those tools should be “delivered” with the Continuous Integration framework in // of the code.

Regarding the System Integration – and I know some will say it is not Agile ;-) – my view is it would mainly be in the second week of your sprint (although it could start earlier).. Typically performance, stability, user acceptance testing would require several days of a stable “final” release even if some of this was done as “exploratory” testing during the first week of your sprint.

In term of QA having nothing to do at the begining of the sprint I totaly disagree as they have sevaral “high criticality” tasks, amongst others:
- work with stakeholders to ensure we can create test plan that mapped the acceptance criteria and – by experience – it is by doing this we often find flaw in the requirements and the acceptance criteria :-)
- anticipate needs in term of libraries and tools for automation
- set-up test environment
- work on test data set and tools to generate them
- define and implement mockup/simulator

As being part of the definition of Done: Yes :-)

]]> By: Marty Hollas http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-528 Marty Hollas Fri, 07 Nov 2008 18:13:33 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-528 I have been leading QA teams in iterative development for a couple of years now, and the most common frustration voiced by QA is the lack of work at the beginning, and then a flood of work at the end that is completely impossible to complete. QA becomes the point of failure in the eyes of others, people loose sight of the team focus and things start to break down. There are many reasons that a team hits this speed bump, but from my experience it starts with poor estimating and planning, poor communication within the team, or a team that hasn't 'gelled'. Typically, QA can start breaking down the stories on day 1 and most of the time can start creating test strategies or test cases. If the estimating and planning was done correctly, at least 1 or 2 dev tasks should be ready to test by the 3rd or 4th day of the iteration. QA first manually tests the features, then starts automating. From there the momentum builds and the flow steadies, until the iteration is complete. Daily updates in the scrum is key to staying on track. An estimate is just that - you may finish earlier, you may finish later, but you always know where you are each day. If dev gets too far ahead of QA, you will blow your iteration plan and set QA up as a bottleneck during your next iteration, while failing to complete stories during the current. If your team is not getting along, or cannot seem to 'gel' after a couple iterations then something has to change. The most successful iterations that I have been involved with were with teams that knew what to expect from each other, had no problems sitting side by side with QA and Dev, and all of them understood that it is a team effort to reach the goal. I have been with teams where Dev and QA do not get along, where developers take each bug as an attack on their work and want no part in QA being a voice in their development approach, and these teams have the hardest times trying to meet iteration milestones. A successful iteration starts with a strongly knit team estimating and planning according to their skill sets, and will end successfully as long as they can work together every day side by side and utilize the leads and PM to remove roadblocks - it just takes time. My 2 cents. Marty I have been leading QA teams in iterative development for a couple of years now, and the most common frustration voiced by QA is the lack of work at the beginning, and then a flood of work at the end that is completely impossible to complete. QA becomes the point of failure in the eyes of others, people loose sight of the team focus and things start to break down.

There are many reasons that a team hits this speed bump, but from my experience it starts with poor estimating and planning, poor communication within the team, or a team that hasn’t ‘gelled’.

Typically, QA can start breaking down the stories on day 1 and most of the time can start creating test strategies or test cases. If the estimating and planning was done correctly, at least 1 or 2 dev tasks should be ready to test by the 3rd or 4th day of the iteration. QA first manually tests the features, then starts automating. From there the momentum builds and the flow steadies, until the iteration is complete.

Daily updates in the scrum is key to staying on track. An estimate is just that – you may finish earlier, you may finish later, but you always know where you are each day. If dev gets too far ahead of QA, you will blow your iteration plan and set QA up as a bottleneck during your next iteration, while failing to complete stories during the current.

If your team is not getting along, or cannot seem to ‘gel’ after a couple iterations then something has to change. The most successful iterations that I have been involved with were with teams that knew what to expect from each other, had no problems sitting side by side with QA and Dev, and all of them understood that it is a team effort to reach the goal. I have been with teams where Dev and QA do not get along, where developers take each bug as an attack on their work and want no part in QA being a voice in their development approach, and these teams have the hardest times trying to meet iteration milestones.

A successful iteration starts with a strongly knit team estimating and planning according to their skill sets, and will end successfully as long as they can work together every day side by side and utilize the leads and PM to remove roadblocks – it just takes time.

My 2 cents.
Marty

]]> By: Alex Forbes http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-520 Alex Forbes Fri, 07 Nov 2008 16:57:11 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-520 I thought the following posts by Damon Poole at AccuRev would be of interest on this subject. http://damonpoole.blogspot.com/search/label/QA You may also be interested in viewing a short demonstration of how an early stage integration between AccuRev and Rally provide agile requirements traceability. http://blog.accurev.com/2008/06/11/agile-requirements-traceability-with-accurev-and-rally/ I thought the following posts by Damon Poole at AccuRev would be of interest on this subject.

http://damonpoole.blogspot.com/search/label/QA

You may also be interested in viewing a short demonstration of how an early stage integration between AccuRev and Rally provide agile requirements traceability.

http://blog.accurev.com/2008/06/11/agile-requirements-traceability-with-accurev-and-rally/

]]> By: Asha Somayajula http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-518 Asha Somayajula Fri, 07 Nov 2008 16:41:13 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-518 To everybody who has agreed to Chris's post. I'd like to know if this would be possible on projects which are matrixed with shared QA/Developer resources and short sprints of ( maybe 2 weeks). Would we have time for reflection and all the completeness criteria that we have as developers/qa/analysts? Can anyone suggest a better approach to handling the aggressiveness in the delivery of the product increment on this ? Thanks, - Asha To everybody who has agreed to Chris’s post. I’d like to know if this would be possible on projects which are matrixed with shared QA/Developer resources and short sprints of ( maybe 2 weeks). Would we have time for reflection and all the completeness criteria that we have as developers/qa/analysts? Can anyone suggest a better approach to handling the aggressiveness in the delivery of the product increment on this ?

Thanks,
- Asha

]]> By: Eric Lakey http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-513 Eric Lakey Fri, 07 Nov 2008 14:30:13 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-513 These are all great points. I really like what Stephen said. "what is the value of fixing several cosmetic defects when you can complete another high value business story" In my current project, we are doing something similar to what Julian suggested. We spend an hour or two at the end of each iteration with all parties to review the functionality being delivered to make sure there were no major disconnects and to identify other bugs/tweaks to features. We then prioritize completion of new features, feature changes, and correction of defects into future iterations. It is important to set the expectation of upper management that the burn rate needs to include bug fix time and feature clarification (aka allowable scope creep). That way, no one assumes that all future development hours in future iterations will be dedicated to new feature development. That is an old-world belief that assumes you know everything up front. If your group is pretty experienced with Agile, this probably isn't an issue, but many of my clients need extra communication about this. These are all great points. I really like what Stephen said. “what is the value of fixing several cosmetic defects when you can complete another high value business story” In my current project, we are doing something similar to what Julian suggested. We spend an hour or two at the end of each iteration with all parties to review the functionality being delivered to make sure there were no major disconnects and to identify other bugs/tweaks to features. We then
prioritize completion of new features, feature changes, and correction of defects into future iterations. It is important to set the expectation of upper management that the burn rate needs to include bug fix time and feature clarification (aka allowable scope creep). That way, no one assumes that all future development hours in future iterations will be dedicated to new feature development. That is an old-world belief that assumes you know everything up front. If your group is pretty experienced with Agile, this probably isn’t an issue, but many of my clients need extra communication about this.

]]> By: Tim Mackinnon http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-512 Tim Mackinnon Fri, 07 Nov 2008 14:17:10 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-512 Controversially I disagree with you guys - the best team I was ever on (Connextra) we weren't allowed to have a tester - this forced us to properly write tests (and use TDD) and naturally made us feel responsible for the quality of the product. Too often I see teams loaded up with QA's and developers throwing code over the wall to them. While the latter scenario sort of works - bug counts seem higher and story execution seems less focused - its a slippery slope to silo'd execution. I have lately seen some teams with 1 or 2 testers who actually champion the use of testing - they are like Scrum Masters for quality - challenging developers to write better tests (and the team/users to write better stories) and helping them slice stories into small pieces that can be verified (they tend to do exploratory testing to identify any testing holes). This seems like a much better way to go - and these teams to have a natural flow that oozes efficiency. Controversially I disagree with you guys – the best team I was ever on (Connextra) we weren’t allowed to have a tester – this forced us to properly write tests (and use TDD) and naturally made us feel responsible for the quality of the product. Too often I see teams loaded up with QA’s and developers throwing code over the wall to them. While the latter scenario sort of works – bug counts seem higher and story execution seems less focused – its a slippery slope to silo’d execution.

I have lately seen some teams with 1 or 2 testers who actually champion the use of testing – they are like Scrum Masters for quality – challenging developers to write better tests (and the team/users to write better stories) and helping them slice stories into small pieces that can be verified (they tend to do exploratory testing to identify any testing holes). This seems like a much better way to go – and these teams to have a natural flow that oozes efficiency.

]]> By: Emmanuel Szabados http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-475 Emmanuel Szabados Thu, 06 Nov 2008 23:14:24 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-475 Some outlines from my experience: - moving teams working few stories at a time was the best: in planning2 teams define what are the first stories from priority order they will start to work on. Therefore, each team members panic the first day of the iteration because some don't know what to do. After ad-hocs team session they find themselves useful and start write test cases, review automated scripts for any changes, pair between tester & developers.... - if you have a big organization, you can create a distinct QA Automation team and have them being mercenaries in other team to help implementing automation like Fitness... Then the team continue to manage their test framework - if you have small group and small budget, get some consultant to write and package simple automated tests using 4GL tools like WGET. Pair some of your engineers with the consultant to share knowledge. - if iterations requires lot of testing: + while we keep testing continuously, we plan to stop development one and half week before the end and have the all team focus on "Bug-out" and very small new code introduction. Sometimes you just want to ensure good quality delivery versus good burn down graph :-) + at some point we call other group to help testing (customer services, even business or directors). Those are "blitz testing". We create small scenarios, ask people to test between 15-30 minutes max and log stuff in word doc with capture screen or chat in Skype... Then we regroup.... Very efficient; People get to better understand the product and new features; Indirect socialization between groups; etc... - we have also a strong unit testing framework which includes more than just unit tests and that is tight to continous build. Every time "unit" tests broke, we regroup and fix it and the one who broke it get a "Toilet Plunger" --> trust me that mark the point :-) - To reduce product bugs, we do also "Kill Bugs" session: take a bunch of engineers, testers, product owners... Every Friday afternoon for 2 month, bring beer and snack in a big conf room; Have all working together in the conf room with their laptops... to: + review the general bug list (close, split, prioritize bugs...) + debug (code, test, close) + hit the product to death and continue recording bugs or fixing it - Since most testers are also coders today (scripts, Ruby....), promote cross functions by paring dev and testers. - Since what developers do the most is also testing their code, extend their own testing by involving them in dependency testing to allow them to get out of their own development box and see the big picture. Wishes: - I wish that all this above takes less time for people to get into it :-) - I'd love to do more TDD but that's very hard to do it right. - better 4GL automated testing tool. Some outlines from my experience:

- moving teams working few stories at a time was the best: in planning2 teams define what are the first stories from priority order they will start to work on. Therefore, each team members panic the first day of the iteration because some don’t know what to do. After ad-hocs team session they find themselves useful and start write test cases, review automated scripts for any changes, pair between tester & developers….

- if you have a big organization, you can create a distinct QA Automation team and have them being mercenaries in other team to help implementing automation like Fitness… Then the team continue to manage their test framework

- if you have small group and small budget, get some consultant to write and package simple automated tests using 4GL tools like WGET. Pair some of your engineers with the consultant to share knowledge.

- if iterations requires lot of testing:
+ while we keep testing continuously, we plan to stop development one and half week before the end and have the all team focus on “Bug-out” and very small new code introduction. Sometimes you just want to ensure good quality delivery versus good burn down graph :-)
+ at some point we call other group to help testing (customer services, even business or directors). Those are “blitz testing”. We create small scenarios, ask people to test between 15-30 minutes max and log stuff in word doc with capture screen or chat in Skype… Then we regroup…. Very efficient; People get to better understand the product and new features; Indirect socialization between groups; etc…

- we have also a strong unit testing framework which includes more than just unit tests and that is tight to continous build. Every time “unit” tests broke, we regroup and fix it and the one who broke it get a “Toilet Plunger” –> trust me that mark the point :-)

- To reduce product bugs, we do also “Kill Bugs” session: take a bunch of engineers, testers, product owners… Every Friday afternoon for 2 month, bring beer and snack in a big conf room; Have all working together in the conf room with their laptops… to:
+ review the general bug list (close, split, prioritize bugs…)
+ debug (code, test, close)
+ hit the product to death and continue recording bugs or fixing it

- Since most testers are also coders today (scripts, Ruby….), promote cross functions by paring dev and testers.

- Since what developers do the most is also testing their code, extend their own testing by involving them in dependency testing to allow them to get out of their own development box and see the big picture.

Wishes:
- I wish that all this above takes less time for people to get into it :-)
- I’d love to do more TDD but that’s very hard to do it right.
- better 4GL automated testing tool.

]]> By: Chris http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-474 Chris Thu, 06 Nov 2008 23:03:22 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-474 Trackback to this post from Leading Software Development: http://www.leadingsoftwaredevelopment.com/2008/11/article-on-dev-and-testing-schedules.html Trackback to this post from Leading Software Development: http://www.leadingsoftwaredevelopment.com/2008/11/article-on-dev-and-testing-schedules.html

]]> By: David Lumsden http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-470 David Lumsden Thu, 06 Nov 2008 22:42:57 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-470 It is easier & quicker for developers to fix bugs on the same day as they create them, or maybe the next day. The help TDD gives here is in proportion to the comprehensiveness of the tests. QA will pick up problems not detected by the developers, & the sooner they do this the better. Chris' basic approach is good; two caveats: 1. to keep developers busy they have to be free to move on to new stories or the defect backlog 2. the shift to agile can be a bigger hurdle for QA than for developers It is easier & quicker for developers to fix bugs on the same day as they create them, or maybe the next day. The help TDD gives here is in proportion to the comprehensiveness of the tests. QA will pick up problems not detected by the developers, & the sooner they do this the better. Chris’ basic approach is good; two caveats:
1. to keep developers busy they have to be free to move on to new stories or the defect backlog
2. the shift to agile can be a bigger hurdle for QA than for developers

]]> By: Sharan Karekatte http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-452 Sharan Karekatte Thu, 06 Nov 2008 18:58:45 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-452 I agree with you all. Here's how we go about integrating QA/Testing tasks into each Sprint: • QA (or Tester, I'll use these terms interchangeably) starts analyzing the User Stories and Acceptance Criteria and follows up with the Product Owner on any open questions, and meets with the Developers to ensure everyone's on the same page. • Once everyone's clear on the requirements, QA starts writing up the test cases. • Developers try and get a testable build ready in the next few days. • QA starts testing, and there's back and forth between QA and the Developers. • QA or Developer tries to give a preliminary demo of what the team has been working on to the Product Owner just to ensure that the team is meeting all of the expectations. • When the complete functionality for a feature set has been delivered for testing, QA starts logging defects in the defect tracking tool from that point on. • When testing is complete for one feature set, QA schedules a QA Review with QA members of other teams. This helps Testers from other teams understand the feature set (under QA Review) in case there is overlap with their team's feature set and also provide feedback to the Tester doing the QA Review. • In some cases, the Tester (doing the QA Review) may also request for another Tester who is more familiar with the feature set to do a verification of test cases and possibly some validation. • The QA Review helps spread the knowledge and also allows for interaction between QA members on different teams. Sometimes, the Product Owner and Documentation Specialist may also attend the QA Review to get a better understanding of the feature set or functionality. • We follow a 'zero defects' policy, meaning, any defects (critical to cosmetic) found during the sprint are resolved. Any defects found after a sprint is over are entered in the defect tracking tool and put back on the Product Backlog. • QA walk-through's of a feature set/functionality with the Product Owner are typically more thorough and detailed (you know how QA folks are). • We try to wrap up most of the walk-throughs before the final sprint review, leaving very little or nothing for the final sprint review, so that we can focus more on the Sprint Retrospective. I agree with you all. Here’s how we go about integrating QA/Testing tasks into each Sprint:
• QA (or Tester, I’ll use these terms interchangeably) starts analyzing the User Stories and Acceptance Criteria and follows up with the Product Owner on any open questions, and meets with the Developers to ensure everyone’s on the same page.
• Once everyone’s clear on the requirements, QA starts writing up the test cases.
• Developers try and get a testable build ready in the next few days.
• QA starts testing, and there’s back and forth between QA and the Developers.
• QA or Developer tries to give a preliminary demo of what the team has been working on to the Product Owner just to ensure that the team is meeting all of the expectations.
• When the complete functionality for a feature set has been delivered for testing, QA starts logging defects in the defect tracking tool from that point on.
• When testing is complete for one feature set, QA schedules a QA Review with QA members of other teams. This helps Testers from other teams understand the feature set (under QA Review) in case there is overlap with their team’s feature set and also provide feedback to the Tester doing the QA Review.
• In some cases, the Tester (doing the QA Review) may also request for another Tester who is more familiar with the feature set to do a verification of test cases and possibly some validation.
• The QA Review helps spread the knowledge and also allows for interaction between QA members on different teams. Sometimes, the Product Owner and Documentation Specialist may also attend the QA Review to get a better understanding of the feature set or functionality.
• We follow a ‘zero defects’ policy, meaning, any defects (critical to cosmetic) found during the sprint are resolved. Any defects found after a sprint is over are entered in the defect tracking tool and put back on the Product Backlog.
• QA walk-through’s of a feature set/functionality with the Product Owner are typically more thorough and detailed (you know how QA folks are).
• We try to wrap up most of the walk-throughs before the final sprint review, leaving very little or nothing for the final sprint review, so that we can focus more on the Sprint Retrospective.

]]> By: Julian Harris http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-441 Julian Harris Thu, 06 Nov 2008 17:41:20 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-441 What you've suggested is what I've done in the past successfully -- you want to push as much of the quality measures into the sprint as possible. How much? Try to keep the release sprint as predictable as possible. If your release sprint increases linearly with the number of previous sprints, then you should consider embedding more of these quality activities into each sprint. In my projects, the QA team are on the same table as the dev team and communicate fluidly. A few retrospectives into one project the team decided to formally agree with a number of checkpoints for each user story: 1. when the test script was written, the devs writing against it would do a once-over (15 minutes or so). 2. When the code was 'ready for testing' the QA and dev would do another quick once-over (another 15-30). This eliminated enormous numbers of 'test/fail/raise bug/code/retest' (90%) and ended up with more getting 'done'. In some environments QA means something closer to 'business acceptance testing' and doing a number of things such as validating the system against an end to end business process etc. We'd usually stagger this one sprint / iteration later against what was delivered in the previous sprint and some of these are very time-consuming activities that can indeed take more than a whole sprint to prepare (e.g. test data for some form of expert system). You'd want to do those as frequently as practicable but accept it maybe every 'few' sprints. What you’ve suggested is what I’ve done in the past successfully — you want to push as much of the quality measures into the sprint as possible. How much? Try to keep the release sprint as predictable as possible. If your release sprint increases linearly with the number of previous sprints, then you should consider embedding more of these quality activities into each sprint.

In my projects, the QA team are on the same table as the dev team and communicate fluidly. A few retrospectives into one project the team decided to formally agree with a number of checkpoints for each user story: 1. when the test script was written, the devs writing against it would do a once-over (15 minutes or so). 2. When the code was ‘ready for testing’ the QA and dev would do another quick once-over (another 15-30). This eliminated enormous numbers of ‘test/fail/raise bug/code/retest’ (90%) and ended up with more getting ‘done’.

In some environments QA means something closer to ‘business acceptance testing’ and doing a number of things such as validating the system against an end to end business process etc. We’d usually stagger this one sprint / iteration later against what was delivered in the previous sprint and some of these are very time-consuming activities that can indeed take more than a whole sprint to prepare (e.g. test data for some form of expert system). You’d want to do those as frequently as practicable but accept it maybe every ‘few’ sprints.

]]> By: Tino Zottola http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-434 Tino Zottola Thu, 06 Nov 2008 15:11:34 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-434 Chris, The timeline described in your blog pretty well describes the optimum use of QA and development resources. During the last day of the cycle we also conduct a post-mortum session with some of our QA people and developers to learn from the mistakes of current development cycle and apply the lessons to the next cycle. We have a distributed QA approach, which is split into two levels: Tier 1 testing is done on each module before it is submitted into the trunk load. We have a 100% pass test case requirement for each module submitted into the trunk. Our developers are responsible for Tier 1 testing and the majority of this testing is automated in the development environment (e.g. Netbeans, Eclipse, etc.) Tier 2 testing is done on each completed trunk load (composed of fully tested Tier 1 modules). This testing is done by the QA people exclusively at the system level. This testing is both automated and manual depending on the product characteristics. Chris,

The timeline described in your blog pretty well describes the optimum use of QA and development resources. During the last day of the cycle we also conduct a post-mortum session with some of our QA people and developers to learn from the mistakes of current development cycle and apply the lessons to the next cycle.

We have a distributed QA approach, which is split into two levels:

Tier 1 testing is done on each module before it is submitted into the trunk load. We have a 100% pass test case requirement for each module submitted into the trunk. Our developers are responsible for Tier 1 testing and the majority of this testing is automated in the development environment (e.g. Netbeans, Eclipse, etc.)

Tier 2 testing is done on each completed trunk load (composed of fully tested Tier 1 modules). This testing is done by the QA people exclusively at the system level. This testing is both automated and manual depending on the product characteristics.

]]> By: Bob Marshall http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-433 Bob Marshall Thu, 06 Nov 2008 14:47:29 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-433 Write the tests first! And I'm not just talking about developers writing units tests (e.g.TDD). Even more important, for those shops that have reached the necessary level of expertise, is having analysts, or better yet, customers, writing the acceptance tests up front too (v.f. Story Driven Development, a.k.a. SDD). In all the projects I've run over the past ten years and more, we have not had "testers" on the team - nor any need for them. Sufficiently professional and motivated development teams should (must) be able to produce defect-free code without the need for testing. All testing to find defects is waste! - Bob Write the tests first! And I’m not just talking about developers writing units tests (e.g.TDD). Even more important, for those shops that have reached the necessary level of expertise, is having analysts, or better yet, customers, writing the acceptance tests up front too (v.f. Story Driven Development, a.k.a. SDD).

In all the projects I’ve run over the past ten years and more, we have not had “testers” on the team – nor any need for them. Sufficiently professional and motivated development teams should (must) be able to produce defect-free code without the need for testing. All testing to find defects is waste!

- Bob

]]> By: Keith Sterling http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-432 Keith Sterling Thu, 06 Nov 2008 14:29:33 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-432 I smell a whiff of waterfall in the description of your process, especially the fact that "In the final days of the iteration, the devs stop writing new code and focus on fixing defects". I'm guessing your probably have a rather flat burndown chart until towards the end of your iteration which is never a good thing. I prefer to see my teams fixing defects as they occur, which means testers finding them as the developers write the code, which means a story not being signed off until both of the above complete. There is nothing wrong with a developer moving onto the next story while the tester completes testing, but this natural stagger in the work, should never be allowed to continue past 1 ( possibly 2 ) stories, otherwise you are just building up a tsunami of defects at the end of each iteration. With this in mind, I have no problem with developers writing new code until almost the last day of the iteration, if the testers are keeping up with the pace. If you are TDD this should never be a problem, however if testing is lagging I generally encourage the developers in my teams to help with the testing so that we get the most stories to a DONE/DONE stage as possible. Testers testing everything at the end of the iteration, again can be interpreted as waterfall, so I try to avoid this and ensure I have good automated test coverage so that the entire test suite is being executed constantly throughout the iteration. I smell a whiff of waterfall in the description of your process, especially the fact that “In the final days of the iteration, the devs stop writing new code and focus on fixing defects”.

I’m guessing your probably have a rather flat burndown chart until towards the end of your iteration which is never a good thing.

I prefer to see my teams fixing defects as they occur, which means testers finding them as the developers write the code, which means a story not being signed off until both of the above complete.
There is nothing wrong with a developer moving onto the next story while the tester completes testing, but this natural stagger in the work, should never be allowed to continue past 1 ( possibly 2 ) stories, otherwise you are just building up a tsunami of defects at the end of each iteration.

With this in mind, I have no problem with developers writing new code until almost the last day of the iteration, if the testers are keeping up with the pace. If you are TDD this should never be a problem, however if testing is lagging I generally encourage the developers in my teams to help with the testing so that we get the most stories to a DONE/DONE stage as possible.

Testers testing everything at the end of the iteration, again can be interpreted as waterfall, so I try to avoid this and ensure I have good automated test coverage so that the entire test suite is being executed constantly throughout the iteration.

]]> By: Hari Koorapati http://edgehopper.com/qatesting-in-an-agile-environment/comment-page-1/#comment-431 Hari Koorapati Thu, 06 Nov 2008 13:08:00 +0000 http://edgehopper.com/qatesting-in-an-agile-environment/#comment-431 Hi, As you said, we can have QA team writing test cases from day 1. I recommend QA team always behind one iteration behind dev team. Once a feature is done after unit and integration testing by dev team, this can be handed over to QA team. QA team starts with System and Acceptance testing with this. If QA finds any bugs, depends on severity of corrections can be taken immediately / at the end of second iteration. Thanks & Regards, Hari. Hi,

As you said, we can have QA team writing test cases from day 1. I recommend QA team always behind one iteration behind dev team. Once a feature is done after unit and integration testing by dev team, this can be handed over to QA team. QA team starts with System and Acceptance testing with this.

If QA finds any bugs, depends on severity of corrections can be taken immediately / at the end of second iteration.

Thanks & Regards,
Hari.

]]>