Tuesday July 28, 2009
NetBeans 6.7.1 was released yesterday, which in addition to a number of important bug fixes in various areas, also now includes support for JavaFX.
It looks very promising; the biggest problems I mentioned a couple of months ago are fixed. I no longer get any bogus or stale error messages. And the preview pane seems to work - I'm not getting exceptions left and right. Also, the preview panel is docked into the navigator area - that seems like a good place for it.
Code completion and import management works as before - which means it might be a little surprising (see my old blog entry for tips), but it works.
Unfortunately, my two other pet peeves are still there: No toggle comment, and no mark occurrences jumping - which is extra problematic since there's no Find Usages for the JavaFX support. I took a quick peek at the sources, and adding it is trivial - so I submitted a patch.
In the meantime, you can download this .nbm plugin file and install it via Tools > Plugins if you want to get these two features. Cmd-/ to toggle comments, and Ctrl-Up/Down to jump between the occurrences in the current file. Note that I couldn't create a separate plugin to do this - I hacked the existing JavaFX editor support. That's because the mark occurrences feature needs to access a method which is package private (and I didn't want to mess around with reflection - I can't spent time on this). This unfortunately means though that you are using a hacked version of the FX editor module. If they release patch updates later you'll be replacing this functionality. But hopefully by then they will have applied the patch!
I noticed by the way that all the source code for the NetBeans JavaFX plugin is available right there in the public NetBeans Mercurial repository - so you can not only grab the source code and build it yourself, but you can fix and contribute patches to make the support better. I want a good JavaFX source formatter! Actually I'll even settle for just a simple indenter. Please, pretty please? P.S. I see that there is already a code formatter class in there - I added a breakpoint and stepped through it a bit but I couldn't see anything obvious, and I unfortunately can't spend any time on this, so hopefully somebody else can step up to the plate?
Monday March 16, 2009 At the Roundup last week I talked to Robert Cooper, who's doing a lot of interesting work with GWT (Google Web Toolkit). In GWT (which compiles Java to JavaScript), you can embed JavaScript directly within the Java files inside comments with special markers. Robert mentioned that IntelliJ has a nice feature for GWT programmers where your native JavaScript snippets get JavaScript highlighting etc.
I figured that would be a trivial thing to add to NetBeans with our language embedding infrastructure. Sure enough, the patch just adds 13 lines of code. Here's a screenshot - showing JavaScript (regular expressions etc) inside Java code (with packages etc) - it's inside comments that start and end with -{ and }-:
If you want this functionality in your own 7.0 build, just download this module nbm, or apply the patch to your own build.
(2009-03-16 10:37:17.0) Permalink Comments [6]
Tuesday January 13, 2009 Python and Ruby are dynamically typed languages. We have a lot of heuristics in our editors to figure out the types - tracking assignments, analyzing your database migrations, etc - to help code completion, go to declaration and other IDE features work correctly. And Martin Krauskopf, who owns the Ruby editor now, has been improving the type inference for Ruby a lot lately.
However, there are cases where we just can't help. What if you're writing a brand new method you haven't called from anywhere, and you're trying to access methods on one of your parameters? We don't know the type of the parameter. In that case, NetBeans will provide "fallback code completion" - it will list ALL methods across ALL known classes in your project, libraries and current platform.
If you're looking for the documentation or parameters for a specific method - that's useful - after typing 2-3 characters you've typically narrowed the result set down to a handful of methods and you can pick the one you're looking for.
But what if you're not sure what the method is called? That exact scenario happened to me a couple of days ago. I was writing some Python code to do some string manipulation, and not being very familiar with Python I wanted to know what the equivalent of Java's indexOf method was in Python. I knew that the parameter in my method was a String, but I didn't have a way of telling the IDE that. I applied my usual workaround of adding a line above the current call:
x = ""and then I could invoke
x. to see the methods available for a String. However, that's definitely not a good way to do it. What if I forget to remove my fake assignment? (Yep, I did that too. I couldn't figure out why my code wasn't working until I debugged and realized I was clearing out the parameter with a bogus assignment... Doh!)
The "obvious" solution for this is supporting type assertions. We've had those for a long time in the Ruby editor
(picture), and obviously for JavaScript using the @type and @param {type} syntax. And the PHP editor just added support for type assertions as well (more here and here).
I've just added this for the Python editor as well. But I've gone one step further which I think makes the feature much more obvious, and easy to use. In code completion, if the editor detects that you are attempting to complete on a simple variable (as opposed to for example a complicated expression), and the type of that variable cannot be resolved, then we add a new special code completion item at the top of the list. Here's what it says (and a more detailed explanation in the documentation popup):
As soon as you press enter, the IDE will insert a special comment referencing the variable you were completing on, and automatically invoke code completion again to complete on the type statement:
The contents of this list has two parts: First, it lists the pseudo-types for many of the core builtin types, like ints, strings, lists and tuples. After that it will list all the known types in the system, such as the user and library classes. If you select one of these and try code completing the original expression again, you can now see that the type resolver knows what to do:
A couple of notes about this. First, note that the type assertions are just comments in your source. You can delete them - they are merely an aid to the IDE. Not only code completion uses these - go to declaration for example will also be more accurate if it is explicitly told the type of a variable it cannot figure out. You can also use multiple type assertions within a method in case the variable type changes during the method. This isn't the case with Ruby yet; the type assertions apply to parameters only, at the method level. But Martin has promised to look into this for Ruby as well so hopefully there will be full parity between the two editors shortly!
P.S. Just like in the PHP editor, mark occurrences and instant rename are type-assertion-aware:
Monday January 05, 2009
In the 7.0 builds, we have a dedicated unit test runner for Python now. Running a file as a test, or running the test project action, will open a test runner with the output docked on its right hand side, instead of just the output window (click for full size screenshot):
Here's the test runner itself:
What you want to see in the output is a single green line which says "All n Tests Passed", where n is hopefully a large number. But when one or more of the tests fail, you see something like the above. Running the Next Error action (Cmd-.) will jump to the next failed test, just like it normally jumps to the next error in the current output window, or the next task in the tasklist.
One thing to notice (and this is new in 7.0) is that we now include the failure message right there in the test summary for each test, so you don't have to drill into an individual test to see the failure message. You can also hover over a test and any output from that test is shown in a tooltip. You can also right click on tests and select "Run Again" to run just one specific test over again.
This is the same window as the one we have for Ruby. In fact, Erno, who wrote the Ruby test runner, has modularized the code now such that it's the same implementation for both languages - we just have language specific plugins to actually hook into the various testing frameworks. Currently, for Python we support the two builtin testing frameworks: unittest and doctest. The above screenshot showed a regular unittest run. Here's a doctest:
One important thing to note is that you don't have to instrument your code with any test running code. As long as you place doctests in your docstrings, or as long as you have any classes extending unittest.TestCase, the test runner will find them. All you have to do is run Test Project:
and when you have done that, both unittests and doctests are found and included in the same test run - here's both failing doctests and unit tests:
Once again, kudos to Erno for writing the excellent test runner support! And as usual, let us know of any problems.
P.S. Jean-Yves Mengant has also done some excellent work recently. In addition to some cool new quickfixes in the Python editor for NetBeans, he has just integrated a multithreaded debugger for Python!
Friday January 02, 2009 I've checked the Ruby code coverage code into the trunk now, so if you grab a recent daily 7.0 build, it should be there. I found a Windows-specific bug in the Python code coverage code as well, so grab a new build if you're on that platform.
I've had some fun with Java2D lately. Instead of the ugly coverage bars I showed in my screenshots a couple of weeks ago, the coverage bars now have proper gradients, drop shadows behind the text etc.:
Another thing I added is a bar in the editor footer (while code coverage is enabled) which lists the coverage percentage for the current file, along with actions to quickly run the tests again, or bring up the coverage report, or clear the results, or finish the coverage mode. As well as warn you when your data is older than your code.
And finally, I updated the built-in and extra color themes to also have color definitions for the coverage highlights. Here's the Dark Pastels color theme with coverage data enabled:
Please let me know of any problems while this code is still fresh in my mind :)
(2009-01-02 11:37:54.0) Permalink Comments [16]
Tuesday December 16, 2008 Here's a new feature for NetBeans 7.0: Code Coverage support!
First a screenshot:
There are a couple of key points about the code coverage support:
I didn't think a screenshot really described this feature well, so I recorded a couple of quick sessions where I use the feature:
Here's a couple more images if you have problems loading the videos:
The Python version is already in the 7.0 builds. The Ruby support isn't integrated yet; it probably needs some tweaking to make it work on Windows.
(2008-12-16 18:09:33.0) Permalink Comments [10]
Friday November 28, 2008
As part of the JavaScript 1.7 work,
I also beefed up the E4X support. E4X allows you to embed XML objects
directly in your JavaScript source. Here's some simple E4X JavaScript:
We've had E4X support in NetBeans in both NetBeans 6.1 and 6.5. But now, in NetBeans 7.0, semantic highlighting is E4X aware, such that the source code looks like this instead:
The whole XML document declaration has a faint purple background, element names are blue and document content is bolded.
(The green here, as for all JavaScript code in NetBeans, indicates variables in global scope.)
(2008-11-28 10:14:09.0)
Permalink
Comments [2]
Tuesday November 25, 2008
I just checked in support for JavaScript 1.7. This means that NetBeans will no longer give you syntax errors if you try to use the new language. Here are some screenshots. First, let's open a file using JavaScript 1.7 constructs:
Obviously the file is using yield which is not recognized as a keyword, and in particular, this is just a syntax error - the parser was expecting a semicolon. But there's a little lightbulb... let's see what it says (type Alt-Enter or click on the lightbulb):
Let's select the option to change the active language to 1.7. I could also open the JavaScript options dialog directly and set it to 1.7 as shown here:
After applying the language change, the editor immediately re-lexes and reparses the file according to the new language. The errors are gone, and yield is now a proper keyword:
Similarly, the let keyword can be used, as well as destructuring assignments, generators and iterators, etc. (See New In JavaScript 1.7 for more information).
If you place the caret on function definitions, it will highlight all exit points, and this includes yields now.
This work isn't done; I'd like it to be smarter about generators, and there may be some issues with scope handling. But at least the editor doesn't get in your way with false error messages now - you can start writing JavaScript 1.7 clean code. (This is with NetBeans 7.0 dev).
In related news, I just heard from Tom Enebo that a lot of Ruby 1.9 language features are now supported by the JRuby trunk parser, so hopefully we can soon start fully supporting Ruby 1.9 features in NetBeans as well.
(2008-11-25 14:46:34.0) Permalink
Friday November 14, 2008
Here are some screenshots of the code completion working for Python.
First, let's try it from an import statement:
Note that the documentation for the module is shown. Next, let's
try completing on imported symbols - again we've got the documentation:
I'm writing a unit test, extending TestCase. Now let's try overriding
some methods - completion after typing def:
Obviously, invoking methods on self. will show the inherited
methods - and their documentation:
Here's completion in my function (not on self) showing
the local variables, imported symbols and builtins. Notice that deprecated
methods are marked with strikethrough (and deprecated modules are in imports
as well, though I didn't show that above).
Notice also that the documentation here looks better - it has links,
colors, etc. This is because NetBeans supports the reStructuredText
Docstring Format (see PEP 287).
System libraries are being documented in this way now, so more of the
documentation will start looking like this.
When NetBeans finds embedded code, it will also syntax highlight the
code fragments in the documentation:
You can use this in your own documentation as well obviously.
Here's a file where I have restructured text. As I'm writing it,
I can -immediately- preview the text myself by just pressing
Ctrl-Shift-Space (Show Documentation - for those who don't know, you
can do this on regular code as well if you want to just get the documentation
for the code at the caret, rather than completion alternatives up to
the caret prefix.)
Here's a final example - completion after raise (or
exception) will show completion for all Error
classes:
I'll be quiet or about a week now -- I'm going on vacation! Not great timing given the release date, but this has been planned since February...
(2008-11-14 09:30:00.0) Permalink Comments [6]
Thursday November 13, 2008
Let's start with a pretty simple module - an import, a function, and a couple of classes:
(Notice by the way the semantic highlighting here - the unused parameters and variables are marked with a slight grey underline,
method names in bold, parameters in orange. More on that another day.)
Now let's go to a different file and import this module. Code completion is context dependent and shows us
which symbols are being exported by the given module. We see not only our functions and classes, the imported symbol sys
is also being defined by module, meaning that if you do from compl4 import * you would have sys defined as well:
Python has a way to deal with this: You can specify a variable named __all__, which lets you define exactly what the public API for
this module should be (more details).
Let's go in and define the names we want exported by our module. I will add __all__ = [ "Bar " ] somewhere in the file, to specify that only Bar should be exported:
Look at the file outline / navigator on the left hand side. As soon as I typed in the __all__ line,
the icons changed to show me that only the Bar class and its method are now "public"; there are little
lock icons on everything else! Let's see what code completion offers when we try to import this module again:
Yep, only the public API is exported now. Let's append to the __all__ list to export the func function:
You don't have to rely on icons to tell you if something is wrong though. Let's say I've forgotten all about this, and I decided
to rename my Bar class:
NetBeans checks that all the symbols you declare in your __all list actually exist, and points out any discrepancies. Here, the public Bar class no longer exists since we renamed it.
Here's another little bug I came across related to this. I was checking the __all__ handling on various library files,
in this case posixpath.py. Notice how in the navigator, all the functions are public, except one...
Sure enough, the code looks like it's registering this conditionally added method... Do you spot the problem? There's a dot missing!
As soon as we add it, we get this:
Of course, this last bug was already detected by the
unresolved symbol detector...
Wednesday November 12, 2008
You didn't think we had forgotten Ruby, did you? Here's a new Ruby editor feature: Detecting duplicate hashkeys. As "Fjan" (who filed the
request) says, when some Rails calls allow dozens or more keys, you can accidentally add a hash key twice, with subtle results. NetBeans now tracks these down and points them out for you.
This is going into 7.0, but you can use this 6.5-compatible module as well.
Martin and Erno have also been hard at work. There are many new features in 6.5 (which I'll cover when 6.5 ships in a week or two). But in the 7.0 trunk, the debugger can attach to arbitrary and remote processes - see Martin's blog entry for more info. And Erno has added support for Shoulda testing and more test runner improvements - details here.
(2008-11-12 18:50:11.0) Permalink Comments [2]
Tuesday November 11, 2008 In my last entry I showed that the Fix Imports action will alert you to an unresolved symbol. This is really helpful since it can help you track down runtime problems.
Obviously, you shouldn't have to go run Fix Imports on every file, periodically, to look for unresolved symbols!
NetBeans has a special semantic checker for this. It is off by default, for two reasons:
To enable it, open up the Editor options dialog, and under Hints, select Python. You should now see the current batch of Python hints. Enable the "Unresolved Symbol" hint:
Now let's open up the datetime.py file again:
As you can see, the file is marked erroneous (by the red square above the scrollbar), and the scrollbar indicates where in the file the error is. Clicking on it to expose the error location you can see an underline under the unresolved symbol, and a tooltip explaining what the problem is. The error is also shown in the tasklist, so you don't have to open each file to look for problems - just let the tasklist scan your project for problems.
Here's another example. I opened up the Django 1.0 sources, and NetBeans pointed out this problem to me, in django.utils.simplejson.tool:
This shows that the import is unused, and that the attempts to call methods on the import do not work. The call would need to be changed to
obj = django.utils.simplejson.load(infile)or the import changed to be
import django.utils.simplejson as simplejson
Finding unresolved symbols means that NetBeans must be able to find all the libraries you are importing. This means you can't just point NetBeans at a plain Python file - your project has to contain all the libraries you are using, and, you have to configure your source path correctly such that NetBeans computes the correct package names for your python modules. (I accidentally pointed to the "django" folder instead of its parent folder at first, which meant django. wasn't part of the package prefix and that meant that all kinds of symbols couldn't be resolved.)
Second, this won't work for dynamically generated symbols (eval etc). I haven't run into this a lot yet - but I'm sure there are areas where the unresolved detection can't find symbols that really do exist.
P.S. Is there a canonical Python term for what I've called "unresolved" here that we should use instead?
(2008-11-11 09:23:59.0) Permalink Comments [4]
Monday November 10, 2008 We're about to release an early access release of Python support for NetBeans. We have ambitious goals for the Python support: NetBeans should be the best IDE for Python development, bar none!
Over the next couple of weeks I'll post screenshots of a bunch of editor related features for Python. Today I'll start with a feature near and dear to my heart: Import Management.
I used to be a die hard XEmacs user. When I started coding in Java, I continued using emacs for a couple of years. And the feature which finally made me abandon emacs for a Java IDE? You guessed it -- import management. If I wanted to use for example the List class, I could just press a keystroke to choose which List class I had in mind, and the import statement was added for it at the top of the file while I could continue working in the middle of the file somewhere.
To be sure, this is not rocket science, and Emacs support for Java has improved a lot over the years. But at the time, this was a huge productivity boost and coupled with other code-intelligence and navigation features was enough to lure my away from my comfort zone in the IDE to what was a less powerful plain text editor, but a more powerful Java coding tool.
Anyway, Python shares some similarities with Java. There are lots of libraries, and you need to import modules before you can use them. And this is where NetBeans' Python support can help.
Let's say I've tried calling the cmp method to compare files. I ran the code and realized I need to import cmp before I can use it. To get the IDE to import the symbol for me, I can just pull up the context menu and select Fast Import (or more realistically, just use the keybinding Alt-Shift-I):
This brings up an import dialog like this:
This tells us there are two definitions of a function named cmp - one in filecmp and one in functions. I'm interested in the first one. The label below the list tells me that I can hold the Alt key or the Shift key when I select the list entry to change the import statement. And below that, there is a preview of the import statement that will be generate. As soon as I press the Alt key, the window changes to this:
Here, the bottom line is showing that we will be doing a wildcard import of all symbols instead of just the target symbol. And finally, let's press Shift:
This will not just import the cmp symbol, it will instead import the whole module and change my current call to use the imported module symbol. When I press Enter my editor looks like this:
This is the import facility I ditched Emacs for ten years ago. But it's not how I manage imports for Java anymore. Instead, there is the Fix Imports action (it's also there in the menu -- Ctrl-Shift-I for short). Unlike Fast Import, which looks at the identifier under the editor caret, Fix Imports will look at all your imports in the file, and, well, fix them! This means sorting them, removing duplicates, even removing unused imports. And obviously, resolving imports automatically using the machinery I showed above. Here's how this works.
Here I've opened the SocketServer.py file, and then I've commented out all the imports:
The file is now broken - it has various references to functions that are not defined/imported anymore. Let's see what happens if I press Ctrl-Shift-I:
Tadaaa! It has figured out the missing imports and has inserted them. This is confirmed by a message in the lower left hand corner:
This just happened automatically because NetBeans could figure out, unambiguously, what the missing imports must be. Let's make it less clear-cut. I'm going to insert a call to get(). Now let's try again:
This time we get a dialog. You can see that it has figured out the sys and os imports - but it is not certain to do about the call to get(). There are two modules defining this function - unicodedata and webbrowser:
The corresponding function signatures are listed to help me decide which one I'm interested in. I can choose one of them, and when I press Ok, the imports are applied.
In some cases it may not find the symbol at all. Let's put in a bogus call:
Here the import manager can't figure out what to do - there is no known matching function. Only when NetBeans is certain about the imports does it proceed to clean up the imports without any user intervention.
This facility can also help find bugs! When I was testing this feature, I happened to invoke it in the datetime.py file which ships with Jython. Here's what it said:
That's right - the code is referencing nonexistent class ValuError. This clearly should be ValueError!
NetBeans has additional facilities for tracking down these types of bugs which I'll cover in another post.
So far I've focused on using the import facility to import new symbols. If you're writing code using the editor you may not need it, since when you use code completion to insert calls, it also automatically inserts import statements as needed. However, another useful aspect of Fix Imports is cleaning up your imports: removing duplicates, sorting alphabetically (configurable), and even deleting unused imports.
If you open up a file with unused imports, NetBeans will show you that immediately:
There are quickfixes here, so if you click on the lightbulbs or press Alt-Enter, you can see the import actions:
The integration with quickfixes means that if you look in the tasklist, it will show files containing unused imports. But when you're editing, you can just press Ctrl-Shift-I to clean up the imports - it will do everything we've covered so far: Add missing imports, remove duplicates, remove unused, and sort the remaining imports.
Import management is also tied into source formatting. If you format your source, you can optionally have it clean up formats. Here are the formatting options for Python:
You can play with the checkboxes and the preview will show you the effect. Here's what happens if we enable sorting system libraries to the top. Note also that removing unused imports is not done by default during formatting (whereas Fix Imports will perform it automatically). You can change that in this formatting preference panel:
As I mentioned, we'll have an early access version out very soon. If you're feeling adventurous, you can always download builds from the continuous build server - http://deadlock.netbeans.org/hudson/job/python/. Please file bugs and ideas here.
(2008-11-10 22:03:52.0) Permalink Comments [4]
Tuesday September 09, 2008 I'm back!
I'm sorry I've been quiet on this blog, but I've just taken a long vacation. It's been a few years since my last long vacation, so I really enjoyed it. On top of that, we bought a house, moved and made some home improvements, and that turned out to be a lot of work. Anyway, I've been back several weeks now and I'm back in full swing. There are a lot of things going on. I'm doing various performance and stability fixes for NetBeans 6.5, I've done some updates in the Ruby support, and I'm also working on Python support. I'll definitely be saying more about that here, soon, but I'm not quite ready yet.
While I was on vacation, a high priority bug came in that I had to look at. It was basically complaining about a big memory leak, and my code looked like the culprit. So I used our memory analyzer. I hadn't used it before, but it was unbelievably easy and powerful, so I thought I'd show a couple of pictures of how to use it here, in case any of you have worried about your own memory leaks or how to fix them. I'll show the actual bug I tracked down, so this isn't a made up fake example.
First, get a heap dump. In my case, the submitter had produced it - but if not, with Java 6 you can just run a jmap command to generate it from a running VM (more info).
Then, go to the Profiler menu in NetBeans and invoke Load Heapdump:

After NetBeans has loaded in the dump file, you can switch to the Classes view to see a sorted view of your memory usage.
There's a filter to let you search for specific classes, but you'll typically want to go for the classes near the top anyway
since that's where you're going to get bang for your buck.

Here I'm concerned with the javascript TokenStream.LexingState objects. So I have picked a class to focus on, and then I
ask NetBeans to show all the instances of my class in the heap:

You can now pick among the various instances in the left hand window and find one you don't believe should still
be in memory. Once you select it, NetBeans will search the heap to find the referenes which point to this instance.

So, you can look at all the objects in the heap of this class, and then you can follow references within the
objects to other objects, or in the References view (on the bottom), you can follow references backwards to see
who is pointing to this object, recursively!
Now, the really good part. I'm wondering why so many of these objects are still live - how are they referenced
from the top? I can right click on the object and ask it to show me:

NetBeans will search for a while, and then show a complete path from a GC root down to my object. Now I just
need to walk back from the GC root and examine each object to decide if it's justified in holding this
reference. In this screenshot I've selected the item which was the problem. As the little red badge
shows (also described in the legend at the bottom of the window), this is a static reference.
This class had created a static map which eventually end up holding a lot of context which should have
been freed up. Best of all, in this case, the culprit wasn't mine so I was able to dispatch the
bug to another developer and go back to my vacation :)

In C, you can have true memory leaks where you've allocated memory and you no longer have a reference to it,
so the memory is truly lost. In Java, with garbage collection, there's no true leak, only "unintentonally held memory".
In other words, this feature isn't just for those of you with memory leaks (where you're running out of memory).
You can use this feature to find out if you're holding more references than you're intending to. Freeing things
up can improve your application's performance and definitely reduce its resource requirements. So please give
it a try! I know there are other tools that can help track down memory issues - I've used some in the past - but
this was unbelievably simple (and by being integrated in my IDE, the Go To Source action on references made it
really tricky to figure out WHY there were references as I was checking each reference back from the GC root).
Saturday July 26, 2008
In my language support blog entry last week I talked briefly about the PHP, Python and Scala work going into 6.5. That's not an exclusive list. We're also adding Groovy support, and it's making great progress. Martin Adamek is posting status updates on his blog; here's Groovy instant rename in action:
The Python work is making good progress too. Here's code completion now for methods and classes, with integrated documentation support:
Here's another screenshot: Python quickfixes:
Anyway, I'm going on "vacation" now for a few weeks. I put that in quotes because I'm moving, and honestly, packing, cleaning and moving is hard work. Coding, now that's fun. I can't wait for my real vacation to start when I get back to work.
(2008-07-26 09:52:57.0) Permalink Comments [2]