This is a response for not a nice post about quality of NetBeans at nbusers mailing list. I thing that the original post is more misunderstanding and a email from an upset user.
To make some of the things clearer I'd like to describe how we test the Milestones. And how the milestone process should look like. I'm using word "should" because the world isn't perfect...
Milestone process
Let me describe it in words. Release engineers(RE) build the NetBeans builds. When the build is ready quality engineers(QA) start the testing of the milestone. Developer focus on fixing issues during that time. After a time (usually one week) new branch is created. QA identifies show stoppers these are bugs that have to be fixed before the milestone build can be published. When all the show stoppers are fixed the milestone is published and announced to the NetBeans community.
To make some of the things clearer I'd like to describe how we test the Milestones. And how the milestone process should look like. I'm using word "should" because the world isn't perfect...
Milestone process
Let me describe it in words. Release engineers(RE) build the NetBeans builds. When the build is ready quality engineers(QA) start the testing of the milestone. Developer focus on fixing issues during that time. After a time (usually one week) new branch is created. QA identifies show stoppers these are bugs that have to be fixed before the milestone build can be published. When all the show stoppers are fixed the milestone is published and announced to the NetBeans community.
The milestone release process is straightforward and easy to understand.
I guess the core question is however, who defines the quality criteria that a milestone has to meet (e.g. how [too?] liberal are we with regards to P1 - P3 numbers).
Georg
Posted by Georg on July 18, 2008 at 02:46 PM CEST #
The criteria are defined by QE managers of NetBeans. You can look at them at http://wiki.netbeans.org/NB65QualityCriteria
HTH.
Posted by Lukas on July 18, 2008 at 03:00 PM CEST #
Lukas,
thanks for the link. It would be interesting to find out more about the thought process and the rationale behind the bug numbers put forth in NB65QualityCriteria. Do they depend on the size of a module, like get the number of bugs down to the average rate of 6 bugs per 10000 lines for "normally well tested code"?. Or is it kind of f(#loc, #bugs, #testers, #developers, past_experience, #shout_outs_on_nbusers, deadline) with f = unknown magic spell?
Posted by Georg on July 18, 2008 at 09:46 PM CEST #
So, that means if I download Dev 200807170007
Then, Build 20080717 went through the above cycle 7 times, and passing relevant testing the 7th time, right?
Posted by Varun on July 19, 2008 at 12:28 PM CEST #
Hi George,
> I guess the core question is however, who defines the quality criteria that a milestone has to
> meet (e.g. how [too?] liberal are we with regards to P1 - P3 numbers).
The quality criteria come from tough discussion between Dev & QE. We (QE) are trying to keep the quality "under control" for the whole release cycle. So the numbers come from prediction based on history data and goal to achieve defined criteria at the end of release cycle.
> Do they depend on the size of a module, like get the number of bugs down to the average rate
> of 6 bugs per 10000 lines for "normally well tested code"?. Or is it kind of f(#loc, #bugs,
> #testers, #developers, past_experience, #shout_outs_on_nbusers, deadline) with f = unknown
> magic spell?
Well, the numbers we are defining as target/boundary for every release is based on history data (how many bugs we are able to fix, how big income we expect, are we going to touch low-level functionality, what are dependencies and risk for other dependent modules, ....) .
Posted by Marian on July 21, 2008 at 10:10 AM CEST #
@Varun: We don't restart the cycle after every bug fix. There is lot of bug fixes coming every minute. But we do our best to verify all the fixes (Sometimes it's impossible) with the production builds. Therefore, your expectation is correct and almost true.
Posted by Lukas on July 21, 2008 at 10:56 AM CEST #
@Lukas,
Thanks for the info!
Posted by Varun on July 21, 2008 at 02:24 PM CEST #
Marian,
thanks for the explanation. I understand your situation (I once had a boss who was head of development, then became head of the quality team. It turned his attitude towards quality 180 degrees around...).
A quality team must not be afraid of taking a tough stance. You are the user's ambassadors just like the product managers.
Posted by Georg on July 25, 2008 at 10:05 PM CEST #
I think one of the main problems is the backport of bugfixes into the 6.1 release: If there would be more activity on stabilization of the release, user wouldn't be tempted to use the often instable milestone releases.
The current situation is: When you hit on a known bug in the release, you get the simple advice to update to the current development trunk or milestone release "where the bug is fixed".
Posted by jiai on July 26, 2008 at 11:45 AM CEST #
jiai,
the purpose of Milestone builds is different... anyway I see your point, advices ... use latest development build ... are common, but you can change it as well. We started to produce patches more often and cover issues we think users are facing a lot couple a months ago ... see
http://wiki.netbeans.org/NetBeansPatches
We opened patch fixes nomination to the whole community. So if you think that this or that issue is critical and you write down the reason ... we'll take it into account and the issue bug fix will be targeted for next patch (if technically possible).
Thanks for your feedback,
*Marian
Posted by Marian on July 26, 2008 at 09:43 PM CEST #