Sunday, November 18, 2012

Improving quality of a multi-component system

Suppose you have several product testing teams -- let's call them the "A", "B", "C", "D" [etc] teams.

Suppose each testing team focuses primarily on testing "their" component, but teams can also find defects in other components, so for example the "A" team mainly tests the "A" component, but they might happen to find a problem in the "B" component.

The "A" testing team has a manager, who is probably being rated on how well the "A" team is finding defects in the "A" component. The "B" test manager is rated on how well the "B" team finds defects in the "B" component. And so on.

Each manager will tend to say "I want the bugs in our component to be found by us, so whenever those other turkeys find bugs in our component, you guys explain to me why they found it and we didn't."

What will happen then? Whenever the "B" team finds a bug in the "C" component, say, the "C" team will be motivated to duplicate the portion of "B"'s tests that found the "C bug," and they will spend time on this that they could have spent writing new tests.

So here's a puzzle: how to put a stop to this behavior? Because if things go on like this, the "C" team will waste a bunch of time

  • writing reports for their manager (who is also in a lousy position) explaining why the "B" team found bugs in the "C" component, and
  • duplicating parts of the "B" tests.
In the worst case, everybody will copy everybody else's tests, which means the same tests will be run multiple times, rather than writing and running different tests. How does this improve product quality? (Really, the "C" team should be thinking of other ways to test the "C" component, and the system for that matter, rather than duplicating all the "B" tests that happened to find some "C" bug.)

Here's what needs to happen: The QA Director or VP needs to be told that this nonsense is going on, and they need to put a stop to this sort of internal competition. In particular, they must not ding the "C" manager if the "B" team finds a "C" bug, and so on.

I mean really, if the "B" team finds a bug in the "C" component, the "C" manager's response should not be "Grrr, we should have found that, not those turkeys"; it should be "Terrific! Let's see if we can learn anything from them and write new tests to more thoroughly test our stuff."

Because every bug found inside the company is caught before the customer hits it, and for that we should rejoice. The point of quality assurance is to assure quality of the product as seen by customers; it really shouldn't matter if the "B" team finds "C" bugs or the "D" team finds "B" bugs -- so long as we find and fix the bugs before they get to the customer.

No comments: