|
A very experienced QA manager recently emailed me some tough questions.
Here they are, with are my answers. As always, comments and further questions are welcome!
How is Software Quality Assurance and Software Configuration
Management integrated into Extreme Programming?
XP defines two levels of testing. The first is unit testing,
which must be performed by the programmers as they work. Each class implemented must have
programmer-developed unit tests, for everything that "could possibly break".
These tests are to be written during coding of the class, preferably right before
implementing a given feature. Tests are run as frequently as possible during development,
and all unit tests in the entire system must be running at 100% before any developer
releases his code. (By release, we mean transferring from his own code space to the code
integration area. This is handled differently, of course, depending on the code management
tools in place.)
The second level of testing is called functional testing. Each feature of the
system (which is defined by something we call a User Story, rather like a Use Case) must
have one or more functional tests that test it. The functional tests are the
responsibility of what we call the "customer", the body responsible for defining
the requirements.
The implementation and running of functional tests can be done by the Software
QA group, and in fact this is an ideal way to do it.
Within XP, are there any specification baselines, test
baselines, QA Acceptance testing, and CM Release Management/Change Control?
I'm not sure I know what you mean by these terms. Let me take a shot at
how XP works in these areas: if I'm way off base, please ask again.
XP is an inherently incremental process, with software being released to
"production" as frequently as possible. This generally means that programmers
release their work to the common development pool approximately daily, and that means that
if a full system were built on that day, their code would be included in that build. The
time period between full system builds varies depending on the environment: since you have
chosen a particularly difficult integration language (C++), I could imagine that you would
build less frequently. We would recommend, however, that the full system be integrated as
often as possible, at least daily. (This may seem aggressive to you. We'd have to talk
about what is possible in your environment.)
Since XP is incremental, developers are working in short time increments we call iterations:
we recommend about three weeks. Features (user stories) are broken down to the
point of detail that allows a developer and his partner to implement the stories they're
working on in that time period. We like the functional tests for that iteration to
be complete and available no more than half-way through the iteration. (This usually means
that QA is writing tests for the next iteration while this one is going on.)
All through the iteration, programmers can use QA's functional tests to determine
whether they have met the requirements. (They are also using their own unit tests to
determine whether their individual classes are doing what they should. This is usually at
a much finer level of detail.)
Baselines work this way: when the code for a story is released, all the
functional tests for it should be in place, and will ideally be working. Inevitably some
will not, especially with teams just beginning with XP. One of the quality measures in the
process is the daily graph of performance on functional tests. The general shape of this
graph, over the course of the full system release period, is that of two s-curves: the
upper curve is the total number of tests written, the lower curve is the number running at
100%. A healthy project of course shows these curves coming together at 100% by the end of
the schedule.
The code management software needs of course to reflect the requirements scheduled for
release. This is determined by the "customers", as part of the planning
components we call the commitment schedule (overall plan for a major release) and
the iteration plan (plan for a (three week) iteration). The baseline of what is
in the system tracks what is actually requested by the customers. Development doesn't care
whether this is new functionality or a change to old. They don't care whether a given user
story addresses something that was planned for or not. XP is completely flexible with
regard to change management: development merely estimates how long any desired feature
will take, and works on it when "customer" schedules it into an iteration.
(Dependencies of course exist, but we find that far fewer exist than most developers
believe. Drilling into that subject is beyond the scope of this email.)
When do the all the customer sign-offs occur?
Customer sign-off is continuous. Each iteration has its functional tests.
Everyone is fully up to date on which tests are working and which are not. If tests scores
are trailing implementation by too much, the customer will inevitably schedule more work
against older features that are incorrect (or whose requirements have changed). When test
scores are tracking implementation, the customer knows it and is comfortable requesting
new functionality.
Because the test scores are public and visible, everyone has the same level of
understanding of where quality is. Generally scores are showing a good curve toward
release, and everyone gets increasing comfort as the release date shows up. And, of
course, if tests are not tracking, everyone knows that and the priority of getting things
right naturally increases.
The overall idea of this part of the process is to provide the most rapid feedback
possible to everyone, customers and developers alike. That's why we like all the
functional test run every night. Next morning, if anything has been broken the day before,
everyone knows it and can deal with it effectively (since it was only yesterday's work
that could be the problem). The faster the feedback, the faster development of quality
software can proceed.
What are the Quality Assurance and Software Configuration
Management roles and responsibilities with Extreme Programming?
We prefer for there to be a separate organization for functional testing
(probably exactly like your QA function, with testing results made public very quickly).
XP, however, only says that there must be functional tests: it does not specify
organizationally how they must be done. Experience is that testing is best done by a
separate function - but one that is very tightly integrated with development rather than
at the end of a long pipeline.
Configuration management is also up to the team. It is usually necessary to have one or
more individuals responsible for CM. We have no special rules or practices addressing how
a group would manage the requirement to build multiple systems from one code base. Our
main approach would be: for each release configuration, there must be corresponding
functional tests, and these must be run before that configuration is released to the
(real) customer. We would think that development would proceed by running kind of a
"union" of all the functional tests of all the configurations.
We'd probably have to talk more specifically about how your specific
organization needs to build configurations to say much more about that.
Do you use IEEE, SEI, ISO9000 standards as references to
acquire the fundamentals of defining accurate requirements for customers and software
engineering users? How can a person write storyboards without having the basics of
pinpointing and developing sound requirements?
We would agree that those who play the customer role have to know what
they want. We do not, however, recommend any particularly formal requirements writing or
recording mechanism. Instead, what we are working toward (XP is for small teams, after
all) is to have a clear understanding in the heads of customers, developers, and testers
as to what is wanted.
Rather than have, say, an "analyst" sit down with the customer and
laboriously translate his mumblings into something representing what is wanted, and then
having a "designer" take the analysis and build a design, and so on, small teams
function best if the customers and designer/developers talk to one another until they
develop a common vocabulary of what is needed and how it will be done. In XP, we would
like to have a common level of understanding in all heads, each focused on its own
particular interests:
Customers: what's needed, what's the business value, when do we need it?
Developers: what's needed, how can I build this, how can I test my code, how long will it
take?
Testers: what's needed by the customers, how can I test whether developers have done it?
As you can see, the testers' functional tests are what close the loop,
assuring everyone that what was asked for was what we got. The best way to do XP is with a
separate functional testing organization that is closely integrated into the process. It
would be delightful to have that organization run by an experienced QA manager trained in
XP.
Is Extreme Programming not for Software Quality Engineering
and Software Configuration Management practitioners.
XP is a development discipline that is for customers (in their role as
specifiers and their role as investors and their role as testers and acceptors) and for
developers. As such, the Quality Engineering and Configuration Management roles are
critical to the effort. They have to be assigned and played in a way that is consistent
with the mission of the group, the level of criticality of quality, and so on. We'd need
to talk in detail about your situation to see just where the XP terminology connects with
yours, but your QA functions need to be done in any effective software effort, whether in
a separate organization or not. So XP certainly is for software quality engineering and
software configuration management, as part of a healthy overall process.
That said, XP is aimed at smaller projects (like yours) and it sounds like yours has a
much higher level of QE and CM than is often seen in companies of your size. That should
give you a strong leg up in building quality software, and we should strengthen your
contribution to quality as we bring XP into the team.
I've enjoyed trying to answer your questions and hope that we'll get together soon to
talk in more detail. As you have seen in your transition from mil-spec to industry, there
is that need for tailoring and streamlining. What is quite possible is to bring the
benefits of testing orientation into the realm of rapid development. XP believes, and
shows, that more testing, done with rapid feedback, actually speeds development.
Thanks for asking!
|
