We strive to make our software defect free to the extent possible.
And we do this by testing the software under various conditions. Some
of these are formally defined, such as unit and functional testing.
Some are accepted as a standard practice but with more or less room for
modifications. These are stress or reliability testing. Refer to Cos'
blog for more information. And then there are other mechanisms of
testing, closer to real world scenario and one which does not have a
design philosophy. This is what I would like to refer to as user
scenario testing.
As I was discussing to other members of the
Quality organization, I realized that even though this is an important
aspect of testing, it is usually less emphasized and understood. In the
broadest sense what this means is that we use one of the real world
scenarios, perhaps simulate real world load/stress and see if the
application crashes. If it does, then is it because of our software or
is it something else. This sounds definitely wonderful and perhaps
gives some measure of reliability of the platform, software and the
application. Now what if the perceived reliability is good and the
application never crashes? Was it all a waste of time and how do you
quantify this effort.
To me "user scenario" testing is very
important, next to the regular testing. Every software project should
adopt this testing to see the real world usage and then perhaps use the
input to better performance tune and improve usability of the product.
Some of the key points that could be derived from the user scenario
testing are
- Coverage analysis of the software being
tested as used in real world. This need not be code coverage, but could
be a list of classes and how often they are being called. - Performance
of the software being tested and the system as a result of singular to
end-to-end scenario. By this what I mean to say is that we need to look
at the performance of Java and the system when using a simple Swing
based stock ticket app to using the same app in a multi-tier mode where
a database (installed or in memory), caching, business logic layer and
a Swing based display. As a direct impact the stress on the system
would be more. But are the same set of classes being used more often or
are we seeing code path that was not exercised earlier. - Finally
what I think would be most important factor to analyze user scenario is
to perform memory leak analysis of the application as a whole using
instrumented software. This should give us some idea of where the
memory leak is in the most commonly used classes, if any.
Distinguishing between the test application (user scenario test) and
the application being tested is critical in identifying what
information to consider and what to reject.
What I see
with this approach is the equivalence of Japanese car manufacturing
technology. They optimize those parameters of a car manufacturing that
are most critical to overall cost effectiveness of the car and driver
comfort. These are the most used systems of a car. What is the use of
testing the strength of door knob or cup holder as hard as you should
be testing the transmission or the engine. Are they both equally
critical. So if we were to test the engine and transmission of a
software as a real user would use, wouldn't that be a reality check
compared to having a 100% code coverage or running a test in an
infinite loop to test its reliabiliy?
Reference:
