Darrin Johnson's Weblog: Need for speed...

W. Edwards Deming has been quoted as saying, "Whenever there is fear, you will get wrong figures." I would like to propose the corollary of "Don't fear figures because they may be wrong." Often it seems we are presented with facts, figures, metrics, etc. which we automatically assume they are accurate and (re)act accordingly. However there are some basic questions that should be asked to critically analyze such data: What was the methodology used to generate the numbers?, Is the underlying data reliable that was used to generate the numbers (e.g. bad data in, bad data out)?, Does the intent of the numbers match up with the data?, etc. Understanding the intent of the numbers is particularly important because it can have a direct impact on both the methodology and the accuracy of the underlying data.

For example, bug metrics are often used to imply the quality of a product from a customer perspective. However in reality most bug databases include bugs that will never been seen by customers or have bugs that are seen by a repeated number of customers. When looking at the data it is important ask the question of whether or not the former should be excluded and the later should be given extra significance. This is particularly important for the downstream use of the data including prioritizing and resourcing which bugs to work on. A priority 2 hit by several customers should probably get resourced before a priority 1 bug that a customer is unlikely to ever see. Again the key is to always have a critical eye for any facts, figures, metrics, etc. that come your way because through validating them you will make better decisions.