NetBeans IDE is scanning/parsing/indexing for ages? Yeah, it's a bug!
"Scanning in progress..." - I believe that everyone have met the issue. You open some larger project and you can not use the NetBeans IDE for a few minutes until the sources are "scanned", whatever it means...
Some unsorted facts first
- scanning is needed and must be performed, otherwise features that need a knowledge of the source code structure (code completion, refactoring, go to ...) could not be used
- only the first scanning should be slow - the result is cached in your userdir ($HOME/.netbeans/$VERSION) and further parsing is incremental (only the part that is modified is scanned again)
- editor infrastructure is very new - Parsing API was introduced in 6.5 for Java and in 6.7 for other languages, therefore, some optimizations are not made yet (for example scanning files in a more intelligent order, allowing to use some features while scanning, etc.). Scanning itself is absolutely no problem - the problem is that the IDE is not very usable while scanning is in progress...
- scanning&indexing issues are the most commonly heard "deal-breakers" for the Netbeans IDE and the NetBeans dev team knows about them
Submitting scanning&indexing issues
If you are in the situation when the scanning&indexing takes too much time or if the IDE freezes during scanning a project, it is an issue. The only way to increase the chance someone is able to fix it is to submit the issue and to have enough patience to handle the issue (reply questions, provide more details, don't get offended by developer's questions/comments...).
Let's summarize what you need to do in order to file a useful bug report on scanning:
- In order to improve logging of the scanning related issues, please run IDE with a command-line switch that improves logging:
-J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE - Include the version of NetBeans, Java and OS in you report (copy information from Help>About) and attach IDE log file ($HOME/.netbeans/$VERSION/var/log/messages.log)
- Describe your problem as precisely as you can but please try to be brief too (these requirements are contradictory). Specifically for scanning issues, include the answers to following questions:
- Does the scanning ever finish?
- If yes, how long did it take?
- Was it only the first scan issue, or is the problem met whenever you open the project?
- If scanning didn't finish, did IDE freeze so that it needs to be killed?
- If yes, did the main window repaint (was AWT blocked)?
- Provide as much information about your project's setup as you can - we cannot reproduce all bugs as we don't have every possible setup:
- What is your project type?
- How big is your project (number of classes, disk size, ...)?
- Does your project use some libraries (how many)?
- What is the structure of your project (i.e.: long "string of dependencies"...)?
- Is your project public, so that everyone can get the sources and open it?
- It very often happens that the performance issues are a result of some specific setup and they can be unrelated to the place where they actually occur - we need to identify possible external cause of the problem:
- Do you use any version control system for your project?
- Is your project located on the network drive?
- When you meet some performance problem and you intend to submit an issue, please don't forget to capture a thread dump or a profiler snapshot so that we can readily see where the activity is.
You already note the biggest single issue above: the IDE is not very useful while scanning. It is essentially an overweight text editor until scanning is done being incapable of Go-To-Type, Go-To-Declaration, or most other Go-To-* navigation and even incapable of Attach-Debugger!
I do understand that scanning an immense body of code like that which I shove at NetBeans takes time. While I hope the NB team continues to optimize performance in this area, this is never going to be fast enough for arbitrarily large code sets. It is critical that NB be able to proceed with navigation, etc, based on (1) data from the scan so far and (2) data from the previous scan.
Note that re-scan/indexing is not very fast in many cases for me. It is faster, but not fast.
The other, lower-priority issue: navigation is sometimes rather slow, e.g. Go-To-Declaration should be all but instantaneous. This is important to keep an eye on, but the big issue is that which you already noted: the IDE must not become a crippled boat anchor during scanning!
Posted by Jess Holle on June 20, 2009 at 08:04 AM PDT #
Note that scanning sometimes starts up when you'd least expect it *and* NetBeans forgets to tell you via the status bar and only tells you when you're attempting an operation which is disabled during scanning.
This is extremely frustrating. Yes, I should file a bug, but it does not happen "on demand", but rather seemingly at random, and I neglected to capture a stack dump, etc, when this last occurred.
[P.S. My projects are large free-form projects interconnected with a large build system and thus not sharable.]
Posted by Jess Holle on June 20, 2009 at 08:09 AM PDT #
"Include the version of NetBeans, Java and OS in you report (copy information from Help>About) and attach IDE log file ($HOME/.netbeans/$VERSION/var/log/messages.log)"
Isn't all of that information in the log file? The easier it is to collect meaningful data, the more data you will get. Let's automate as much as we can. Fir example why not set the logging level to FINE by default in the non-release builds? Daily builds are updated frequently so the less tweaking to the configuration we have to do, the less tweaking we will forget to do :-)
"What is your project type?
How big is your project (number of classes, disk size, ...)?
Does your project use some libraries (how many)?
What is the structure of your project (i.e.: long "string of dependencies"...)?"
The IDE should know all of this, put enough info in the logs (at FINE level anyway).
"Do you use any version control system for your project?"
If it is supported in NB, then the IDE should already know this and it should be in the logs.
Add a bug-reporting mechanism into the IDE so that all of the data is collected automatically - you are already doing things like this for exception reporting. You've already added ability to get profiler snapshots of the IDE... Add one item "Submit problem report" with the ability to automatically grab as much info as it can (being careful to disclose any IP that might go with the report)
Once such a system is in place the feedback is sure to increase substantially - for all problems.
Posted by swpalmer on June 24, 2009 at 07:09 AM PDT #
swpalmer: Thanx for those nice suggestions, all of them are definitely worth considering.
Anyway, it is not in 6.7 yet, so currently you need to do the steps (IDE does not do it). Actually - you might be able to avoid some of the steps I have mentioned... if you capture a thread dump, it can exhibit an activity in the subversion module, for example (which tells us you use svn).
But generally it is better to include as much information in the text description as possible - for a human, it is easier to read a human word than some disgusting log file. If I read "I use subversion for my project", I look for possible subversion activity in the log/dump files to see if it can be related... if you don't write this information, I have just some log file - so I need to ask you a question if you can reproduce the problem with disabled svn, for example...
The similar thing holds for:
> Isn't all of that information in the log file?
Yes, it is, so this is definitely redundant but it is much easier to see this info directly in the text description. Usually when there is some "well known bug", we know exactly at what build it was fixed or that the issue is JDK/OS specific. Providing both information (help>about+log) doesn't cost too much and simplifies reading the issues in the issue tracker...
> For example why not set the logging level to FINE by default in the non-release builds?
For which module? I mentioned FINE logging only for a module related to scanning... If we do it for all modules, the log will be useless because it will not be possible to upload it due to it's size... we would have to ask you to grep the useful information from it - asking you to use one specific switch is easier;)...
Last but not least - it we implement such a system, it won't be bugless and it costs some development time. I am not able to evaluate if it is worth collecting too many details - *there has to be some balance with this.*
Actually, there is one more thing to mention... Some of you might have noticed the "System slowness detected" bubble (automatic profiling when AWT thread is blocked). Jindrich Sedek is preparing an automatic upload of profiler snapshots captured with this system, see: http://wiki.netbeans.org/FitnessViaPostMortemAutoReport
Hopefully, we will be able to deliver some performance improvements for 6.8 - it is already in our plan: http://wiki.netbeans.org/EditorPlan68
Posted by 192.9.112.196 on June 24, 2009 at 07:32 AM PDT #
I don't mean to pile on here, but is there really a need for more bug reports for this? Searching issuezilla for "task scanning" already shows six issues, which mostly cover the same thing (to wit, that the IDE is essentially unusable while task scanning is going on). I'm going to add in a bug for my own particular issue, which is that if you select "open project" while task scanning is active the whole editor completely locks up, but I'm wondering if it's really all that helpful to have a bunch of issues which will just wind up marked as duplicates of another issue. Going by my own experience, it's not like it's at all difficult to reproduce this, all one needs to do is open a large project for the first time. In my own case, NB has been quietly ignoring me for at least 30 minutes now, more than enough time for me to fire up Eclipse and actually get some work done.
Posted by tim on June 24, 2009 at 10:43 AM PDT #
Hi Tim,
this is a question that sometimes occurs and that sounds logical. My reasoning goes as follows:
The scanning takes long -> it means it is an issue. Issues you meet should be submitted if they are to be resolved... There can be 3 cases when asking a question "Is the bug you are facing already submitted?":
* Yes - Vote for it and attach comment and any data that can be useful (requested info for the incomplete issues, for example)
* No - Submit a new issue with data mentioned above
* I don't know - Submit a new issue anyway, the worst thing that happens is that it is evaluated by dev/QE and marked as a duplicate (or you submit a new bug)
Are you sure that the issue you are facing is the same as the issues that are submitted? There can be many issues saying "scanning is slow" but these can have many different causes...
Also, issue's "real priority" is given by its priority (P1 issues must be fixed), number of votes (reason obvious) and by the number of duplicates (a P3 issue with 140 duplicates gets fixed sooner than a P3 issue with 3 duplicates). Of course it is always better to vote for an issue rather than submit a duplicate one but not everyone is able to evaluate the issue based on - for example - the thread dump... Developer should be able to do it...
This post was about "what meaningful data should be provided in order to make a report of a scanning issue useful to us"... It has been also placed on NB wiki (Users FAQ>Bug Submitting).
Once again a list of improvements that are planned for 6.8:
http://wiki.netbeans.org/EditorPlan68
With regards,
--pd
Posted by Petr Dvorak on June 24, 2009 at 04:25 PM PDT #
Maybe one more tip for the brave ones:
http://wiki.netbeans.org/ScanOnDemand
--pd
Posted by Petr Dvorak on June 24, 2009 at 04:29 PM PDT #
Thanks for the clarification, and good luck with the bug. FWIW, reading your post did cause me to get a stack dump of NB from jstack and submit it with my bug report before I killed the process. Cheers!
Posted by tim on June 24, 2009 at 04:32 PM PDT #
You are welcome Tim, actually - I should thank you for submitting the issue and attaching the jstack dump. :)
Posted by Petr Dvorak on June 24, 2009 at 04:46 PM PDT #
> Parsing API was introduced in 6.5 for Java
AFAIK Parsing API was not introduce in 6.5, but in 6.7
Posted by Jan Becicka on June 26, 2009 at 04:54 AM PDT #
Jan, thank you for clarification (I talked to Jan offline) - the Parsing API was implemented for Java in 6.5 (in a special repo) but it was not merged in 6.5 release (as it was only for Java, we decided to wait till 6.7 with the merge)...
Posted by Petr Dvorak on June 26, 2009 at 06:29 AM PDT #
Is this also related to the automatic error checking that seems to be driving everybody nuts with the red-wavy-lines and error icons ???
It's very distracting and annoying to be reminded of the missing semi-colon even before you reach halfway our first line of code.
Zinahe A.
Posted by Zinahe on July 23, 2009 at 08:39 AM PDT #
Zinahe: Not really. Basically, the scanning&indexing has to be done anyway. Yes, the results are used for the error checking too, but the scanning is not done only because of this (I believe - don't know - the red underline works in the similar way the syntax highlighting does)...
Anyway, here is the issue:
http://www.netbeans.org/issues/show_bug.cgi?id=169067
you can help it by voting for it...
Posted by Petr Dvorak on July 23, 2009 at 11:33 AM PDT #
"only the first scanning should be slow - the result is cached in your userdir ($HOME/.netbeans/$VERSION) and further parsing is incremental (only the part that is modified is scanned again)"
Today as I was using NB 6.7 I had to wait several times for scanning to complete before ctrl-clicking on a method call would take me to that method's code... I only had two small source files with changes and it was very frustrating. It was also slow to populate the navigation window while switching between the tabs for those two source files in the editor.
A couple times trying to navigate to a method brought me to the top of the source file instead of to where the method was. In general the IDE simply wasn't responding as fast as I needed it too, and is some cases it was just wrong.
I'm not sure what sort of information is cached by this process, but I find things like getting the call hierarchy to be very slow, even though this scanning is complete. Building a complete database of where every variable and method is referenced should be possible to do faster than compiling the project (at the very least, no slower). Referencing such a database should make things like getting the call hierarchy instantaneous as far as operator perception is concerned.
Posted by swpalmer on July 23, 2009 at 03:56 PM PDT #
> "A couple times trying to navigate to a method brought me to the top of the
> source file instead of to where the method was.
This is the only part of the comment where I can make a competent comment right now (I woke up 5 minutes ago) - this issue is already reported and it is "started":
http://www.netbeans.org/issues/show_bug.cgi?id=164252
Posted by 78.128.197.106 on July 24, 2009 at 01:31 AM PDT #
There are lots of reports in bugzilla now with stack traces, and the dev team appears to be working on many of them.
For anyone who sees this happening, you might find that using a non-freeform project will really change your experience. If you create a non-freeform project (put the project data tree off to the side somewhere so you can try this), and then modify the build.xml to have targets in it that call out to your ant/maven-builds where needed, you might find that netbeans is a lot more usable, while they work through these issues.
I've found a lot less problems in my use of netbeans after making this change to 13 some projects that are the core of what I am working on right now.
Freeform is really not a good thing right now because it doesn't consider all the things that we need it to in how the scanning is triggered and how it reacts to changes in libarary jar files.
Non-freeform projects seem to be tracking a lot more things incrementally so that there aren't near as many opportunities for netbeans to get lots about a change in something like a library jar, and decide it has to rescan the world.
Posted by Gregg Wonderly on August 07, 2009 at 01:46 PM PDT #
Free-form projects may have more problems, but as far as the excessive scanning goes they are a red herring. I don't have a single free-form project and yet during my editing sessions the scanning issue comes up constantly.
Posted by Swpalmer on August 09, 2009 at 07:34 PM PDT #
I don't think that freeform projects cause all the problems. But, there are dependencies, which seemed to be tracked better without freeform projects and this can eliminate some of the scanning in my experience.
There are still bugs and architectural issues that need to be addressed to remove all scanning, or at least hide it from view of the editor and other interactive modules that impact the developers work flow.
Posted by Gregg Wonderly on August 10, 2009 at 02:24 PM PDT #
Last comment here was August. It's now half way through October and still no signs that these issues have been fixed, if anything each new release seems worse as, instead of sorting out Netbeans core functionality, new bells and whistles are added to make for a more impressive features list.
I tried scan on demand, but that seems to have increased the "lost classes" problem where the error bagding throws up dozens of errors because the syntax scan has lost track of classes or whole library jars. Also there's no real documentation I can find for scan on demand. When are the libraries excluded by scanondemand.properties actually excluded? Should I exclude the stable libraries I import from a shared drive?
There seems little point in adding more bug reports. Half the remaining Netbeans user community are clearly getting the same things. Nothing has happened to me that hasn't been reported multiple times. And yet the priority seems to remain at P2,
If it's not fixed fairly soon I'll reluctantly have to join the exedous to Eclipse. It's just wasting too much of my time.
Posted by Malcolm McMahon on October 17, 2009 at 01:14 AM PDT #
In NB 6.7.1 my project can be loaded and the IDE is idle for half an hour and as soon as I actually try to something (ctrl-click an identifier for example) - up pops the "Please Wait. Scanning..." dialog. WTF?
Posted by Scott Palmer on October 17, 2009 at 04:09 AM PDT #
Hi everyone,
sorry for not getting to this post for a longer time. I am sorry to hear that many of you are still facing issues with scanning and indexing. If you watch mailing lists, you might have noticed that some of the users do not suffer from their performance problem in 6.7 and some issues were resolved in 6.8 too. I am sorry for maybe too diplomatic reaction but I have to say that the issues are being resolved continually.
This is a list of performance issues that are opened in the relevant categories (hopefully, I didn't forger any):
http://www.netbeans.org/issues/buglist.cgi?Submit+query=Submit+query&issue_type=DEFECT&issue_type=TASK&component=editor&component=java&subcomponent=compiler&subcomponent=editor&subcomponent=Parsing+%26+Indexing&subcomponent=project&subcomponent=source&issue_status=NEW&issue_status=STARTED&issue_status=REOPENED&email1=&emailtype1=exact&emailassigned_to1=1&email2=&emailtype2=exact&emailreporter2=1&issueidtype=include&issue_id=&changedin=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&issue_file_loc=&issue_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=PERFORMANCE&keywords_type=anytokens&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=Defects%3A+Editor+Technologies&newqueryname=&order=Reuse+same+sort+as+last+time
I subjectively think this issue is the most important one:
Make IDE usable during scanning - http://www.netbeans.org/issues/show_bug.cgi?id=133943
Also, I would like to overcome presumption that the only thing that we do is that we make IDE more "shiny", disregarding bugs. See the "New and noteworthy" document for 6.8:
http://wiki.netbeans.org/NewAndNoteworthyNB68
You will find that there are mainly improvements from the J2EE/Web area (I don't think the best strategy for us is to ignore J2EE6 or JavaFX) and some improvements by the CND team (C/C++). We really are mainly fixing issues in 6.8...
Posted by Petr Dvorak on October 17, 2009 at 06:43 AM PDT #
Sorry if I got a bit into rant mode there, but this really is getting to be a show-stopper for me. And it's not just one project, it's all of our large, multi-source tree projects.
And it's not just that it locks out Netbeans activities, the intensive IO seems to slow everything on the machine, and it takes ages. It's as if it's not even checking last modified times before completely re-analysing everything in sight.
It might be that the anti-virus software which is on our machines as corporate policy is exacerbating the problem.
Personally I'm not wild about making Netbeans into an all-singing-all-dancing behemoth. It's in danger of entering the "bloatware" category. People aren't very often going to be doing, say, Java and PHP at the same time and, to my mind, it would be cleaner to use different IDEs, but my real concern it that the core functionality seems so badly broken that it ought to be an "all hands to the pumps", priority zero issue, and anyone in the Netbeans team who can contribute should be concentrating on that.
My impression is that you've painted yourself into a corner. You may need a more fundamental rethink of this scanning and indexing stuff than you've been prepared to contemplate. None of us likes to throw code away, but sometimes it's the only way forward, Perhaps what you need to do it to really scan on demand, i.e. check the up-to-date status of a class at the point the information is required. Or how about detecting a period of user inactivity to do some scanning. At the least you need to push down the priority. Maybe take a little rest after each file to allow some other IO to get a look in.
And please give us a control panel for it.
Posted by Malcolm McMahon on October 18, 2009 at 01:16 AM PDT #