Roumen's Weblog

Po	Út	St	Čt	Pá	So	Ne
« prosinec 2009
	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30	31

Today

Navigation

Speaker Profile
Roumen's Weblog
Login
Sun Bloggers
Technorati Profile

Am I popular?

Today's Page Hits: 634

Contacts

Name: Roman Strobl
E-mail: roman dot strobl
at sun dot com

NetBeans

Java Sites

Javalobby
The Server Side
Java Tips
Java Blogs
java.net
java.sun.com
java.cz

Blogs

License info

This work is licensed under a Creative Commons License.

Map of visits

« Good for a Chuckle... | Main | NetBeans 5.5 Q-Build... »

Neděle březen 12, 2006

NetBeans Growth Questioned

Ian Skerrett, the director of Eclipse Foundation's Marketing, is questioning growth of NetBeans in his latest blog entry.

Quote: Interesting, one IDE that is claiming great momentum seems to have no growth.

I suppose Ian is talking about NetBeans ;) So Ian, I have some data for you. This data is more relevant than *any* online survey. Why? Because this is reality - the graph shows how many users connect to the NetBeans update center:

Unique Update Center Visitors

Before drawing any conclusions, let me explain this graph. The measure is rather pesimistic. We are not counting people behind firewalls or people who disable their update functionality in NetBeans or people not connected to the internet. We start every month counting with zero users. Then we count all the unique accesses to the update center. Each IDE has it's own unique ID so we're not counting people if they change their IP adresses via DHCP. So this is a very reliable source of statistics for us it's by no means a number for *all* NetBeans users, but it's a reliable number for NetBeans users with connection to the update center. Of course, for each user only one hit is counted in the measured period.

Now Ian's statistics says that NetBeans user base has dropped by 0.2 % between November 2004 and December 2005. Our statistics says that NetBeans user base has approximately doubled between November 2004 and December 2005. If you compare October 2004 and October 2005 - the difference is 3x (this is when we were talking about the 3x growth in a year which I think can be called a momentum).

Who do you believe? An online survey of 621 developers or raw data of unique update center hits? It's up to you...

Btw, I'd be rather not writing such blogs entries, but if somebody would be questioning the growth of NetBeans user base, here's the non-questionable data we can provide.

P.S. If you are wondering how the curve will continue to evolve - the number of downloads of NetBeans 5.0 in one month after it's release is 2.4x the number of downloads of NetBeans 4.1 in one month after it's release. Sorry Ian, but there's no evidence that growth of NetBeans would slow down. Are you still working in the dark?

Comments:

Without a unique ID in each installation, these number are meaningless. Both Statistics and Surveys lies.

Posted by SDiZ on březen 12, 2006 at 05:06 odp. CET #

> Without a unique ID in each installation, these number are meaningless. Both Statistics and Surveys lies.
"Each IDE has it's own unique ID..."

Anyway, I think the point is the claim made by Eclipse foundation is hilarious; if the user base of an IDE drops, it would mean something severe, which is well beyond the momentum about that IDE.

Can't people just think before they speak?

Posted by Alex Lam on březen 12, 2006 at 05:28 odp. CET #

Sorry Roumen, but it's your numbers that are questionable. Not only do they come from Sun (would you believe statistics coming from IBM about Eclipse downloads?) but also because as the previous commenters pointed, visitors have nothing to do with actual users. NetBeans has done a lot of PR these past months, it's not surprising you got a lot of visits. But for actual usage, at least the BZ study is impartial and confirms the trend I've been observing myself. Eclipse is pulling ahead of all the competitors faster than before. By the way, blogs.sun.com is extremely slow.

Posted by John on březen 12, 2006 at 05:47 odp. CET #

John, This isn't visits to a site, its the number of people connecting to the autoupdate center via the NetBeans IDE which is looking for cool plugins or updates. As for "web site hits" we're averaging over 3 million/month, but that number has less meaning than the tool itself looking for updates. And yes, if IBM published data on Eclipse saying how many users it had attempting to update the tool, and followed a set of conservative rules regarding it (i.e. no swag in the #) I'd believe it. Tim: Director of Engineering, NetBeans

Posted by Tim Cramer on březen 12, 2006 at 06:17 odp. CET #

Roumen,

First, let me apologize for the comment in my blog. My intention was to focus on the growth in Eclipse community, not a negative about Netbeans. You have always taken a positive voice in your blog and I will continue to do so in mine. I will post an apology on my blog.

When I saw the numbers from BZ Media I was surprised that Netbeans had not shown any growth. I was not aware of the Netbeans numbers you have shown, so I appreciate you making them public.

I do think the BZ Media numbers are statistically significant. At least if I remember my stats class from university. Of course with any stats it depends on the population being sampled. In this case it is the SD Times readership, so I am perplexed why Netbeans has not done better. Of course one conclusions is that Netbeans users just don't read SD Times? Any other thoughts?

Posted by Ian Skerrett on březen 12, 2006 at 06:43 odp. CET #

Tim: Yes, I'm aware you are talking about update center hits, not web site visits. Still, these numbers are fairly meaningless when it comes to evaluating actual users. I still have yet to see a neutral survey (i.e. not ordered by Sun and with data published for all to see) that would actually poll the number of *active users* for NetBeans compared to other IDE's. The BZ survey is exactly that, and the results speak for themselves: not only is Eclipse far ahead of IDEA and NetBeans, it's actually increasing its lead.

Posted by John on březen 12, 2006 at 06:49 odp. CET #

Netbeans 5.x has been getting lots of buzz in the Java community at least that is what I have seen. While not directly related, Sun Studio 11 runs on Netbeans and is now free, and is getting lots of downloads, i would venture as many as 1 in 8 Solaris 10 downloads will eventually download Sun Studio, thus creating more users for Netbeans.

Posted by James Dickens on březen 12, 2006 at 06:50 odp. CET #

Yow! Roman! These are hot and what I like most is that these are real numbers. Kind of drives a stake in the heart of the bogus survey claims. The conservative nature of these adoption numbers leaves very little doubt that NetBeans growth is very, very hot. Not surprising given all the new features and buzz.

Thanks for sharing, it confirms what I am seeing.
Charles

Posted by Charles Ditzel on březen 12, 2006 at 09:35 odp. CET #

Ian, yes, I will continue with a positive voice in my blog - I am also a believer of positive marketing as you are. Surveys can be very misleading, because they depend on the people who provide the answers. The measure we are using provides a reliable number - it doesn't show the total number of NetBeans users, but it is very reliable if you want to measure the growth because update center visits grow linearly with the total number of NetBeans users. Thus my reaction on your statistics which was pointing out something else.

As for the conclusions from the survey - there can be lots of factors. I think it requires a deeper analysis - it really depends on who reads SD Times, on it's content, etc.

My point is that if you want to measure growth of the user base of an IDE, there are some more reliable ways like the one we are using. No need to post an apology to your blog - this was enough for an apology.

Posted by Roumen on březen 12, 2006 at 10:48 odp. CET #

John, I was *not* evaluating the total number of users, but the *growth* of user base. These numbers are valid - simply because total number of users is the number of users which connect to the update center multiplied by a quocient. This relation is linear. The IDE connects to the update server when it's started unless the user is behind firewall / offline / disables connecting to the update center. Thus these numbers can show the relative change in user base very reliably. It is much better than a survey - a survey mostly depends on which people provide answers and on the company which surveys these people.

Posted by Roumen on březen 12, 2006 at 10:54 odp. CET #

Roumen,

How many times should a unique IDE connect to the Update Center in order to qualify for these statistics? What I mean is - does the "download-run once-move to trash" scenario qualify for this?

Thanks, Kirill

Posted by Kirill Sorokin on březen 12, 2006 at 11:13 odp. CET #

The user needs to connect twice to qualify for every measurement. We start counting every month from zero users, that means if the user connects only in April and then uninstalls the IDE, he won't be counted in May. Thus, the latest month shows how many users connected at least twice in the last month. This is also the reason why the stats are reliable - we start measuring everytime over. So if the user base would start to drop, we would see it.

Posted by Roumen on březen 12, 2006 at 11:19 odp. CET #

Roumen, Sorry, but you are still missing the point: these numbers do *not* represent the number of users, nor do they represent the *growth* in users. At the very worst, they just represent a burst of interest in NetBeans, which is not surprising considering the amount of PR Sun has been doing recently. The bottom line is that we know nothing about the number of NetBeans users. Besides, even growth is fairly meaningless: the less users you have, the easier it is to show a fantastic growth. Again, the BZ numbers are both objective and they measure real users.

Posted by Sid on březen 13, 2006 at 12:00 dop. CET #

I disagree. It is not a burst of interest - the numbers grow continuously since November 2004. About objectivity of BZ numbers: I see on BZ Media's homepage 14 mentions of Eclipse and BZ Media publishes an Eclipse magazine. I am not saying BZ Media is not objective but I am also not completely sure about it.

The growth is *not* meaningless. Growth is what Ian Skerrett was questioning - he wrote "NetBeans seems to have no growth". So I am reacting on this message which was not correct.

Posted by Roumen on březen 13, 2006 at 12:13 dop. CET #

And, for example, is there any data on how many people have used the UC at least ten times? Fifty times? Are the numbers seriously different from the ones on the chart you posted?

Posted by Kirill Sorokin on březen 13, 2006 at 01:49 dop. CET #

And, for example, is there any data on how many people have used the UC at least ten times? Fifty times? Are the numbers seriously different from the ones on the chart you posted?

Posted by Kirill Sorokin on březen 13, 2006 at 01:51 dop. CET #

With my stats background in mind ... We have a sample of 621 readers of SDT Times who have responded. Let's look at some potential bias and questions.

1. Is there a relationship between Eclipse and SD Times readership?

ANSWER: A quick Google search on SD Times Eclipse BZ Media returns:

a.) "BZ Media's flagship publication is SD Times"

b.) "BZ Media has been a steadfast supporter of Eclipse, first through its Eclipse supplements to SD Times, then with its EclipseWorld conference"

I think it's pretty easy to see that there's some bias in an SD Times readership survey with the above top two returns from such a Google search.

Hence, about all one can claim is significant is that a publication which largely endorses Eclipse, (and therefore likely is read by a large number of Eclipse users), finds a "0.2%" decrease in NetBeans. Actually, considering the bias in the survey, (design of experiment), I think one would likely question why the survey results didn't show a much larger decrease in NetBeans.

And, fwiw, I not only have a background in stats, I'm also married to a professional statistician.

Posted by huntch on březen 13, 2006 at 02:17 dop. CET #

Kirill, I do not have this information - but it should be possible to find out. I would have to ask for full logs though. The important thing is that we're counting one hit per person no matter how many times they connect during the month but it has to be at least twice. I don't know if you understood that the IDE connects automatically during start - we don't require users to manually go to the update center in order to count them.

Posted by Roumen on březen 13, 2006 at 07:34 dop. CET #

Huntch, I actually tried a similar experiment as well. If you google: "SD Times" Eclipse you get 33,000 results. If you google: "SD Times" NetBeans you get 796 results. Similarly "BZ Media" Eclipse returns 12,200 results and "BZ Media" NetBeans returns 204 results.

It's hard to derive conclusions but it should be obvious that BZ Media writes a lot about Eclipse and not much about NetBeans - which can be caused by various reasons including that Eclipse is more widely used. BZ Media itself can be really objective - but from the google search it should be obvious that there is much more content produced by BZ Media about Eclipse than about NetBeans (which again is nothing bad, but it's not the best thing if you're trying to make an objective statistics).

Posted by Roumen on březen 13, 2006 at 07:45 dop. CET #

Roumen, you said, "It's hard to derive conclusions but it should be obvious that BZ Media writes a lot about Eclipse and not much about NetBeans". Given the close ties between Eclipse and BZ Media, including the mere fact that EclipseWorld is sponsored by BZ Media, I'd say it's fair to say that a very large number of Eclipse users get their information from BZ Media / SD Times and hence the survey bias. It should also be worthy of note that Ian's blog is an example of marketing 101, 'trying to show Eclipse momentum going into EclipseCon'. Given that MyEclipse, an Eclipse Foundation member and who has representation on the Eclipse Foundation Board is including NetBeans Matisse GUI Builder in a development preview of MyEclipse probably put a damper on the 'showing momentum going into EclipseCon' strategy.

Btw, you recall the 'which IDE do you use' survey on java.net recently? I think you saw quite a different result than the one reported by BZ Media. IIRC, the %'s were very close between Eclipse and NetBeans. So, what does that mean statistically? It suggests that those who visit java.net are pretty evenly split between the two IDEs. Also, wrt the numbers you are quoting.

Posted by huntch on březen 13, 2006 at 08:49 dop. CET #

Hum is this the same SD Times that has a special feature article with the title "Spotlight on Eclipse"?... the same site on witch i made a search with the keywords "NetBeans 5.0"... and had one reference to the beta version? Interesting, I had never even seen that website, but at first looks it appears to be a very Eclipse oriented site. NetBeans has made a leap this past year, while other IDEs have made small steps, and are playing catch up in many areas (Matisse anyone?). The features keep rolling in such a speed that i am having trouble keeping up with that speed :^)... Enterprise pack, C/C++ pack, plugins explosion... flash movies... tutorials... etc... Has an active member of the NetBeans community for the past 18 months i have seen it grow and gain momentum like no other community that i have been a member for the past 12 years that i have been online... there is one thing that my economy teacher taught me, you can work the numbers any way you want, statistics is the art of lying with numbers :^) He was a good teacher.

Posted by Daniel MD on březen 13, 2006 at 02:02 odp. CET #

[Trackback] Welcome to today's IT Blogwatch, in which Microsoft drops EFI boot support in Vista, and Maryland drops Diebold voting machines. Not to mentionb a modern take on turning swords into plowshares ...

Posted by Computerworld Blogs on březen 13, 2006 at 02:52 odp. CET #

I'm not sure how you accurately measure a user base. But one number I have been able to generate from the Eclipse servers is the number of Eclipse 3.1.2 downloads. That number is currently around 830,000. Is it accurate, who knows. Does it measure the right thing, who knows. But it's a fun number to bounce around.

Posted by CDT Doug on březen 13, 2006 at 04:14 odp. CET #

I have to say these statistics mirror my personal experience with NetBeans. Two years ago I thought Sun was flogging a dead horse because Idea and Eclipse were so far ahead. But the 5.0 release of NetBeans is fantastic - especially Matisse and the profiler. Well done!

Posted by Mark McLaren on březen 13, 2006 at 10:40 odp. CET #

Netbeans are far the best and it is growing. If you are in the business you will know. Some people never accepts facts. Even if you give them real facts they deny it. How else could Sadam Hussein be a threat to the world ?

Posted by oldman on březen 15, 2006 at 10:10 dop. CET #

The BZ Media survey is clearly wrong. I wouldn't read anything sinister into it though: any survey just reflects the opinions of the surveyed audience. Congrats on the NetBeans growth. Someone once noted that only a fraction of all Java programmers use an IDE, and the rest are missing out on the great productivity that an IDE can bring. NB has had some good momentum for the past several months. Now we'll see if we can recapture some of that next week. http://www.eclipsecon.org :) (Any NB/Sun folks going BTW? I enjoyed Simon P's talk there even though I disagreed with him on some points. Why don't you come by?)

Posted by Ed Burnette on březen 15, 2006 at 02:33 odp. CET #

I can send comments from Brazil about amount of people that are replacing Eclipse by NetBeans, and the number is too big, and the motivation is very simple: NETBEANS IS A REDY ALWAYS TO WORK. When I am at my customers and they wants a complete IDE, without worry about which other additional plugins they should download before to stabilish an enviroment to begin to work...Then you have this in one hand, ok? And in the other hand you have a complete IDE, with Web and Enteprise Development support, and a lot of other features ready as such WebContainer and UML Support (NB 5.5 preview) as well.

So, I think would be impossible dont imagine in NetBeans growth.... However, surveys like that will have everytime... And each one know theirs own truth.

I've been living in the light :)

does not matter the surveys :)

Posted by Edgar Silva on březen 15, 2006 at 03:17 odp. CET #

Ed, I actually wanted to come for Eclipsecon but unfortunately I have a new passport and will not manage to get a Visa. So maybe next time. If you go to Java One, see you there!
Posted by Roumen on březen 15, 2006 at 08:57 odp. CET #

Post a Comment:

Comments are closed for this entry.

Disclaimer: The contents of my blog represent my personal opinions which may differ from official views of my employer, Sun Microsystems.