By now, the VMWare "time bomb" issue would have been flogged to death. May be not ...
It will be revisited every time there is a major debacle. However, I tend to pity their QA team. After all, they are a bunch of engineers who will be finger pointed for what was basically a business policy. May be for verifying proprietary stuff, companies should be having a separate QA team to verify things like license, expiry codes etc?
I can assure you, such stuff won't happen with FOSS software.
I had an interesting conversation with our quality architect for automating a new feature that we are working on. We were discussing the choice of programming language to use. Given the complexity of the Solaris Cluster product, we have automation based on Java, C, Perl and shell scripts.
For some of the BUI/GUI tools, Java is the best choice. For others one could either use shell scripts or C programs. C programs are easier to code for automating kernel level components. But the downside is that any bug in the code causes core dumps and it is difficult for people without knowledge of C programming to decode and find workarounds temporarily and continue with testing. This results in delay till the time the QE identifies a fix/workaround.
On the other hand, scripting is the round about way for kernel level components and takes a lot of time to develop. But debugging it is easy and maintenance is lot more easier.
So what will be your choice? If you have some interesting insights or experience, please leave a comment!
One of the most nightmarish experiences for a support engineer is when you have a hysterical customer yelling at you on one end on a possible bug while the sustaining/dev team asks for a reproducible case on the other ! At first, I used to get frustrated ( in my previous job with a different company as a support engineer) but slowly started to get used to it and understood that for a developer, his/her code is never at fault! 
Now that I don't face the customer song anymore directly, I can look back at it with a smile. The Dilbertian view of the the process that we support engineers had is best captured by the song from Melanie's blog:
"
Tell them it's a feature,
Say it's not supported
Change the documentation
Blame it on the hardware
Find a way around it
Say they need an upgrade
Reinstall the software
Ask for a dump!
Run with the debugger
Try to reproduce it
Ask them how they did it, and
See if they can do it again."
I am sure every support engineer has a classical Dilbertian experience and if you happen to be one, do share it!
Ah! Developers!!
After a long break, some gyan for my brethren ....
Theorem :1 Last minute issues irrespective of quality of planning.
It is not possible to release a software product without any last minute nasty surprises and slogging irrespective of how meticulously it was planned.
n(Release of software product without last minute issues) = 0
Corollary:
Last minute issues are not related to the quality of the planning done.
n (Last minute issues) = 1
Theorem 2 : Law of Criticality
The number of non-technical issues is a result of the criticality of the situation. The factor that determines the number of issues is directly derived from the criticality of the release to the team.
n(Non-technical issues) (X Criticality of the situation.
Non-technical issues := Lab outages, facility shutdown, long weekends etc...
Corollary:
Everything will work fine when the project starts and will not work when it is about to end.
n(issues at the start) = 0
n(issues at end of cycle) = (X)
Theorem 3: Law of Voting
If any group (QA, QE, Dev, Management, Rel Team, Marketing) has the option of casting a -ve vote, it will!
Corollary:
Number of votes that are cast for release is always less than the total possible votes.
n(Votes for release) < n (total votes possible)
Theorem 4: Arriving at Quorom
The number of iterations it takes to arrive at quorum is directly proportional to the number of votes possible.
n(Iterations it takes for quorum) (X n(Possible votes)
Corollary
To avoid multiple iterations, keep the number of voters small. aka too many cooks spoil the broth.
Theorem 5: Law of Satisfaction and Sense of Achievement
The sense of achievement and satisfaction is greatest when more number of unexpected hurdles are overcome.
Satisfaction when release doesn't have issues = ΓΈ
Corollary
A software product release without hurdles is worthless!
What say?
Perceptual blindness - "the phenomenon of not being able to see things that are actually there. This can be a result of having no internal frame of reference to perceive the unseen objects, or it can be the result of the mental focus or attention which cause mental distractions. The phenomenon is due to how our minds see and process information"
One of the key things to QA as one of my manager used to say is "To be wide awake to see what is happening" - though this appears to be the right thing to do (and it is) - its interpretation varies and is the reason for many test gaps in my opinion. To understand more, we need to look at how QA is generally done.
Once a feature is coded and unit tests by the developer are done, they are integrated with the code. llly, the QA writes a test plan and specifies the procedure that will be used for verifying the functionality of the feature that is being tested. This may be done either through automated test suites or manual test cases. Typical test plans have what is the expected behaviour and how it should be verified for the particular test case which could be a fault/load injection or something like configuration check.
Given this structured approach to testing, Engineers tend to fall into the trap of not checking how the changes in the specific module which provides the feature affects the system as a whole. Given the narrow scope of verification procedure, Engineers tend to do exactly the same and if that particular behaviour is inline with what is expected, presume the test has succeeded. This is irrespective of whether it is black box or white box testing.
In a reasonably large piece/complex software, integrating features is a big challenge and despite the best intentions and processes, slips are inevitable. So while a feature may work as expected, it can cause trouble in a completely unrelated module which may use a common set of code which could be broken. It is this aspect of testing which is generally ignored. Lack of knowledge of the complete software results in some obvious bugs being missed because of the narrowly defined verification procedures.
One of the greatest benefits of working in Data Services part of Solaris cluster is it being on top of the stack and hence the engineer needs to have a minimum knowledge of everything in the entire stack. The most crucial bugs that I have found are mostly in lower stack of the Solaris cluster which got affected due to the introduction of some other functionality. This is not restricted to cluster alone and includes Solaris/any other application that is being tested with cluster. The same applies to any other software as well.
I'd liken it to the following quote " Never send to know for whom the bell tolls; the bell tolls for thee .."
Never brush aside a bug in a different module as unrelated; Never know what caused it or what it will affect!
This blog copyright 2009 by maddy
