Saturday March 31, 2007
Saturday March 31, 2007
Matisse for Groovy: The Early Days
I'm beginning to investigate the rewriting of Matisse for Groovy. What is abundantly clear, thus far, is that it will be impossible to support both Matisse for Java and Matisse for Groovy in the same NetBeans IDE. However, I suspect that SwingBuilder users (i.e., Groovy programmers) are not interested in programming straight Swing simultaneously with SwingBuilder, so that's okay.
The biggest problem is not the rewriting of the templates (e.g., there's a JFrame template that should be rewritten to a SwingBuilder Frame template and so on) that are used to generate the code. But the problem is that Swing containers like JFrame are represented in the IDE by two files simultaneously, a Java file and a Form file. The point is to change that representation from a Java file to a Groovy file. So, when a Groovy file is opened, there should be a Form file in the Design view for the layout and structure of the container, and a Groovy file in the Source view. That is the difficult part that I'm struggling with, meaning that I have to rewrite code in the "Form" module (which is the module that provides the Matisse GUI Builder).
The good news is that I've been able to take the Form module and completely separate it from the rest of the NetBeans sources, so that it is now part of my own module suite. And I've integrated my Groovy support code (e.g., Schliemann) into that Form module. Together with the "Coyote - Groovy Projects" module from the Coyote site (slightly modified as described in earlier blog entries) and the Groovy JAR from the embedded folder in the Groovy download, that's currently all I need for my module suite.
So, I now have three tabs (look at the toggle buttons below, for "Source", "Design", and "Console"). The Design view will be where Matisse will do its thing. Visually, this is how it looks:
Once I get the FormDataObject to work with Groovy instead of Java, assuming that is even possible, things should be well on their way to us having a Matisse for Groovy.
Mar 31 2007, 01:54:18 PM PDT Permalink
10 Comments
Friday March 30, 2007
Code Folding for Groovy via Tokens in Schliemann
My first explorations of how to provide code folding in NetBeans IDE for Groovy, via Schliemann. Two pictures before the journey begins. In this blog entry, I'm going to look at code folding for comment blocks in Groovy (although, I suspect that there may be some default recognition and handling of comment blocks in Schliemann). In the first picture, you see the expanded code fold:
And here it is closed, within the context of other code folds, showing that the content of the fold is shown as a popup when you hover over its collapsed state:
Note: I'm not happy with the syntax colors above, by the way, and am willing to take any suggestions for improvements at all.
I had assumed that Schliemannesque (woo hoo, another new word) code folding is only possible for grammar rules, not tokens, but I was wrong. Here, I declare that everything from "/*" to "*/" should render the token "block_comment":
TOKEN:block_comment: ("/*" - "*/")
Next, I declare that the parsing of the document should skip everything that is defined as a "block_comment":
SKIP:block_comment
Next, lets color everything that is defined by "block_comment" (although, possibly, the SKIP token above may provide this gray color too):
COLOR:block_comment: {
foreground_color: "lightGray";
}
Finally, everything that is defined by the token "block_comment" should be in a code fold:
FOLD:block_comment: {
fold_display_name:"Comment Fold";
expand_type_action_name:"Expand Comment Fold";
collapse_type_action_name:"Collapse Comment Fold";
}
And what do the attributes above signify? Well, fold_display_name defines the text that you see when the fold is collapsed. And expand_type_action_name and collapse_type_action_name define the labels of the menu items for expanding/collapsing all folds defined for the token (which could also be defined in a localizing bundle file):
My approach here is probably not perfect, but this is ALL I had to do, which, especially when compared to the Editor Code Folding API, is not very much at all. (The only reason why there is no NetBeans API tutorial on that API is that I didn't understand it sufficiently, even after receiving a complete sample on the simplest imaginable scenario, which is also the main reason why this API is not discussed in the forthcoming "Rich Client Programming: Plugging into the NetBeans Platform".) The generic languages framework (i.e., Schliemann) takes care of EVERYTHING else—my only responsibility was to specify WHERE the code fold should appear, WHAT the color should be, and WHAT the labels on menu items and so on should contain. The HOW is not my concern, thanks to Schliemann.
I did a similar thing for import statements. The rest will require some more understanding of 'grammar rules' in Schliemann. But it's cool to see that code folding can work for tokens too.
In other news. One of the many reasons that it is cool to be Dutch (or to, at least, understand the language) is that you can go to http://www.uitzendinggemist.nl/ and watch just about everything that's been on (Dutch) TV over the past years. I highly recommend Keyzer & De Boer Advocaten, it is really really good. Also, without being/knowing Dutch, you're missing out on experiencing some fascinatingly interesting personalities, such as Johan Cruijff and Hugo Borst, to name just two. (The latter is a less-camp version of the UK's Russell Brand. Also a football fanatic, with a similarly philosophical bent. Also very 'eigenzinnig', which means 'idiosyncratic'.) I could go on and on recommending programs and praising authentic Dutch people, but will leave it here.
Mar 30 2007, 06:43:32 AM PDT Permalink
Thursday March 29, 2007
How Cool is Swing in Groovy?
Let's say you wanted to create this GUI:
If you were to sketch this out quite roughly in pseudo code, the content and layout would basically be this:
frame (title:'Demo') {
menuBar {
menu('File') {
menuItem 'New'
menuItem 'Open'
}
}
panel {
label 'Label 1'
slider()
comboBox(items:['one','two','three'])
}
}
frame.show()
Anyone, regardless of their insight into programming, would be able to come up with something like the above. Well, how cool is Groovy? You can literally paste the above into a Groovy context, add a few things to load the SwingBuilder library and pack the frame, and you're done:
When I run the above from the IDE, I get the GUI shown at the start of this blog entry. Given how easy this all is in Groovy, I wonder whether the Groovy community even needs a Matisse-like GUI Builder. It all seems pretty intuitive, and with syntax coloring and code completion, plus maybe one or two other fun things (such as the Navigator shown above), coding Swing in Groovy is going to be a breeze.
By the way, the listing above comes directly from section 8.5 of "Groovy in Action". The section is called "Easy GUIs with SwingBuilder" and, based on my first cursory explorations, that title delivers on its promise. (A second "by the way" is that everything you see in the screenshot above,i.e., this support in NetBeans IDE 6.0 for Groovy, is freely available, if you follow the instructions in my blog entry from yesterday. Currently, it is buggier than the underside of a rock in a desert, but that's why it is a development version.)
Mar 29 2007, 06:32:19 AM PDT Permalink
Wednesday March 28, 2007
Breathing Life into a Dead Coyote (Part 2)
In Breathing Life into a Dead Coyote (Part 1), I stated my intention to commit the Schliemann code I've been working on to an upgraded version of the Groovy support module on Coyote. Well, today I did just that. So as not to mess up other things on Coyote, I created a separate directory and put everything in there. By "everything", I mean that I started with the 5.0 version of Groovy, opened the suite in my development build of 6.0, built everything, threw away stuff that prevented compilation from succeeding, threw away the 'syntax' package (based on Lexer), which I replaced with my half-baked Schliemann code, threw away some more stuff (all neatly recorded in the README.txt in the new directory on Coyote) and then eventually ran the suite without a problem. This is the best case scenario of what is currently possible with Coyote for 6.0:
Somewhere along the line I broke something, because the 'Run Script' button is never enabled, now. So that needs to be investigated. But, the point is, I now have the old Groovy data object working together with my Groovy.nbs file. "How is that possible?" you ask in incredulity, tinged with awe. The answer is this example snippet that I received this morning from Hanz Jancura, who is the NetBeans engineer behind Schliemann:
<folder name="Editors">
<folder name="text">
<folder name="css2">
<file name="language.nbs" url="CSS.nbs"/>
<attr name="createDataObject" boolvalue="false"/>
</folder>
</folder>
</folder>
Note the line in bold. This way, you can continue using your old data object, while reaping the benefits of Schliemann.
Update: I found that the above approach does not actually work. To make my data object work together with Schliemann, I had to add this line to my manifest file, below my loader's declaration, to ensure that the data objects are installed in the correct order:
Install-Before: org.netbeans.modules.languages.dataobject.LanguagesDataObject
If you now go to Coyote, you just need to check out the content of one directory, the one indicated by the hand icon below:
That contains everything I've committed. Some stuff may be broken, while other stuff is unstable. But this is a starting point for further integration, specifically integration of the Schliemann-related code, over the next days/weeks. Remember to use a very recent 6.0 daily build, like I'm doing; get the Generic Languages Framework Studio from the development Update Center, as described recently in this blog. Anyone is welcome to join the project and contribute.
Mar 28 2007, 09:29:45 AM PDT Permalink
Tuesday March 27, 2007
Which Keyboard Shortcuts are New/Changed in 6.0?
The world is divided into two groups of people—those who use the mouse and those who use the keyboard. "Aren't you overlooking the very large group of people who don't use either one, because they don't have a computer in the first place?" Well, those people can also be divided into two groups—those who wish they were using a mouse and those who wish they were using a keyboard. OK? Let's not get side tracked.
The second group of people—those who either use a keyboard or who're wishing they were—know they are superior to the first. Keyboards are for hard core people, while mouses (mice?) are for children. No one can seriously contest this claim. So, anyway, without getting distracted by either of these groups of people, armed as they are with sharp edged keyboards and swinging mouses (mice?), let's pose this question, useful to the second group only: "How can one find out which of the keyboard shortcuts in NetBeans IDE 6.0 are new or changed?" Official answer: One can wait for the release of NetBeans IDE 6.0, which will include a new keyboard shortcut card in PDF format, under the Help menu (just like in NetBeans IDE 5.5 and before). "Yes, but this card will not show me the differences, will it? It will only show me those that are current in 6.0, not which ones are new/changed, compared to 5.5!" Good point, my overly precise, freakishly persistent, and strangely unnamed friend. Never fear, though, because there is help at hand, albeit completely unofficial.
In NetBeans IDE 5.0 Easter Egg for Christmas, I reported on an unmapped action called "Export Shortcuts to HTML", which lets you generate an HTML file containing all the actions (some with, some without mapped keyboard shortcuts), for all the profiles defined in the Options window. So, let's use that functionality to our advantage:
- Read NetBeans IDE 5.0 Easter Egg for Christmas, so you know what we're talking about.
- In NetBeans IDE 5.5, map that action to something, and use it.
- Get the latest NetBeans IDE 6.0 development build, map that action to something, and use it.
- Next, open both of the newly generated files in the Favorites window. They're generated to the NetBeans user directory's config folder. So as to not get confused, either rename them to different names, or open their parent folder rather than the bare file, so you can see which parent folder they belong to.
- Then right-click both, choose Tools and then choose Diff, as illustrated here:
- And now... you can use the Diff window to see all the new/changed/deleted actions. In other words, not just the actions that are mapped to keyboard shortcuts are shown, but all actions, for all profiles (you just need to know the order of the profiles, which you can see at the top of the Diff window). Here, for example, you see the first thing that you'll see when you follow this little procedure, i.e., the unmapped action for the new "re-run" feature in the Output window:
In other words, you can also use this little procedure as a roundabout way of discovering the new features in NetBeans IDE, because often a new feature is accompanied by some actions, which may or may not be mapped by default to a shortcut key, but all of which show up in the exported HTML file. Hurray for easter eggs.
Mar 27 2007, 09:48:41 AM PDT Permalink
Monday March 26, 2007
Palette API and 6.0, Testing the Waters... (Part 3)
Revision 1.12 of PaletteItemNode.java, performed in response to issue 97812, allows icons in palette items to come from external files. This may seem a less than exciting fix, however look at the following scenario, which is precisely what I had in mind when creating issue 97812 (by the way, in today's build, I couldn't get the text/x-java content type to work, probably a temporary glitch):
You can now, for the first time, make use of the palette item DTD to let the user select the small icon and large icon of their new snippet. (As explained in the two previous installments of this mini series, here and here, one can now, since recent 6.0 development builds, create a palette for existing editors and let users add to them at runtime.)
Prior to this fix, the icons had to be provided by the module, which meant that the user of the palette was not able to select the icons (except if you were to provide 100s of icons yourself in the module, but even then the user wouldn't be able to select their own icon).
Therefore, if you pass the file's URL to the code that creates the palette item XML file, the result of the XML creation process is now allowed to be as follows (note the two lines in bold below):
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE editor_palette_item PUBLIC
"-//NetBeans//Editor Palette Item 1.1//EN"
"http://www.netbeans.org/dtds/editor-palette-item-1_1.dtd">
<editor_palette_item version="1.1">
<body>
<![CDATA[
blablabla ]]>
</body>
<icon16 urlvalue="file:/C:/Datasource.GIF"/>
<icon32 urlvalue="file:/C:/DatasourceBig.GIF"/>
<inline-description>
<display-name>How to Use a JFileChooser</display-name>
<tooltip>JFileChooser Code</tooltip>
</inline-description>
</editor_palette_item>
That's a pretty cool change, which means that the users of your palette functionality have absolute freedom when defining new palette items, assuming you provide functionality for them to do so. However, if the user accidentally deletes the icons on their disk, they're destroying their palette items. But, that's the side effect of this. Just make sure that your users know that when they select icons from their filesystem for one or more of their palette items, the icons aren't copied anywhere. So, if they delete them or move them (or even just rename them), they're impacting their palette.
Mar 26 2007, 11:24:10 AM PDT Permalink
Saturday March 24, 2007
Breathing Life into a Dead Coyote (Part 1)
The Coyote project on java.net aimed at bringing, amongst others, Groovy support into NetBeans IDE. The project is dead since the release of NetBeans IDE 5.5, but there's enough useful code in there for integration with the Schliemann-related code that I've blogged about recently. In fact, with a little bit of tweaking (mainly involving the throwing out of the syntax support in Coyote), one is left with a project template and file templates for Groovy, which, when combined with most of what I blogged about over the past week, has this result:
Above, I've clicked on an entry in the Navigator and, as a result, the cursor is blinking happily in the related code in the editor. Bear in mind that one can open multiple Groovy documents, thanks to the NetBeans window system, and even undock documents, and that full functionality is available for undocked documents, as shown in the following screenshot, which shows code completion in an undocked Groovy document:
One other difference between Coyote and my code, i.e., in addition to the syntax point made above, is that Coyote assumes you've downloaded Groovy and requires you to set a path in the Options window to the Groovy installation directory, while my code bundles the JAR file that provides all the classes, so that the Options window extension isn't needed anymore. So I threw that out too. One problem with my approach is that the final size of all the modules is larger than Coyote, because I'm bundling the Groovy JAR, but I think that that inconvenience is mitigated by the ease of use (i.e., no need to download Groovy and no need to set anything in the Options window, everything just works out of the box).
I estimate I'll need about a week (bearing in mind that this is a 'weekend project') to finish the integration of the Schliemann code (and the multiview editor code, providing an embedded Groovy Console) with the 5.0 Coyote code. Then I will add my 6.0 version of Coyote to the Coyote project CVS. No point in creating a new project when the existing Coyote project provides all the infrastructure needed. After that, anyone who wants to can contribute modules to the 6.0 version of Coyote. At some point, something similar to the Matisse GUI Builder, in some form (possibly using the approach discussed in yesterday's blog entry, but more likely something a lot simpler), should also be available to Groovy.
On the Schliemann front, I need to learn how to extract information from libaries such as SwingBuilder, so that the related tokens do not need to be defined manually in the NBS file. That's needed for syntax coloring and code completion. I also need to learn more about grammar so that code folding can be added.
Mar 24 2007, 12:40:05 PM PDT Permalink
Friday March 23, 2007
Reusing the Matisse GUI Builder Palette...
Sometimes small changes have large consequences, unforeseen and, as a result, surprising. In Palette API and 6.0, Testing the Waters... (Part 1), I talked about the resolution of issue 90213. As a result, you can create palettes for existing editors, without changing the sources of the editor in question. The clearest illustration of this is that you can now create a module containing a palette and assign it to the Java MIME type, which means it appears when the Java editor is opened in Source view. Very useful, because then you can drag and drop snippets into the source view of a Java document in NetBeans IDE. (There was a very recent, i.e., a few days ago, further issue resolution which I will blog about in a few days, which is also very powerful in the context of creating, rather than using snippets.)
However, a probably unintended, but potentially useful, consequence of the above is that an existing editor can ask palettes created for other editors to be associated with itself. So, it works both ways. Either you create a palette for someone else's editor OR you use someone else's palette in your editor. And this is not confined to editors! This also works for TopComponents. Think about that. You can take any palette anywhere and associate it with your own TopComponent. Before showing how, here are the two major disadvantages or, at the very least, caveats:
- This approach will result in your module depending on the module that provides the palette. If the palette in the other module is provided by someone over whom you have no control, then the palette can change without you having any influence over it!
- This approach will give you a palette, but not necessarily all the functionality defined for the palette in the module where it is created. It is possible to extract some information from the palette, but you are still limited in terms of what you can do with the palette. The module where the palette was originally created probably has a lot of code that is very useful to the palette, code that you cannot extract along with the palette.
A better solution than following the steps below, is to provide the palette yourself, especially if the alternative is reusing a palette that comes from some module over which you have no/little control.
Okay, now that you know the downsides, let's put these thoughts into practice.
- Create a module. Use the New Window Component wizard to create stubs for your TopComponent.
- In the layer.xml file, create a folder structure under the Editors folder. Within it, use the new MIME type structure provided by issue 90213:
<folder name="Editors"> <folder name="abc"> <folder name="xyz"> <file name="PaletteFactory.instance"> <attr name="instanceClass" stringvalue="org.netbeans.spi.palette.PaletteController"/> <attr name="instanceCreate" methodvalue="org.netbeans.modules.form.palette.PaletteUtils.getPalette"/> </file> </folder> </folder> </folder>The abc/xyz could be anything at all. It defines our MIME type. We're not dealing with an editor here, but the principle of MIME type registration of palettes for editors is continued to TopComponents, so... we can use the same approach here, even though we are not dealing with an editor. We just need to specify a MIME type, which could be anything, and must be represented by at least one folder, so that we can associate that folder in the TopComponent. So, instead of abc/xyz, you could have foreign/palette or just one folder, called palette, or anything at all. So the abc/xyz is irrelevant, just so long as you have a MIME type, constructed out of these folders.
The PaletteFactory.instance element, and its attributes, are mandatory. The first attribute, instanceClass, specifies the NetBeans API class that we are returning from the second attribute, instanceCreate. What we need to have returned is an instance of the NetBeans API PaletteController class. Before the fixing of issue 90213, the PaletteController class could only be instantiated within the TopComponent's constructor. Now, however, it can be done via the layer.xml.
And, which PaletteController class are we instantiating? Look closely, friends, because this is where things become interesting! The second attribute has this value:
org.netbeans.modules.form.palette.PaletteUtils.getPalette
In short, what that value implies is that the getPalette method in the org.netbeans.modules.form.palette.PaletteUtils class is going to provide our palette. If you dig deep into the NetBeans sources, you will find that method in that class. If you want to go down this road, the road of having other modules provide your palettes, then you need to know which method provides the palette in the module where the palette is defined. That requires that the writer of the original module should tell you or that you should find out by actually reading the sources of the module in question.
And which module provides the palette above? The same module that provides the Matisse GUI Builder. "Hold on now. Are you saying that at the end of this little procedure I will have the Matisse GUI Builder's palette in my own TopComponent?" Yes. That is exactly what I am saying. Now back up a second and read the two caveats above again. Are you sure you want the Matisse GUI Builder's palette? Are you prepared for it to change, with the result that your module's functionality will change too?
The Matisse GUI Builder does not provide an API, but that is basically how we are using it here. So, that's what I meant when I started this blog entry by burbling about the unintended consequences of small changes.
- In your TopComponent's constructor, associate the palette that you registered in the layer.xml with the TopComponent:
associateLookup( Lookups.fixed( new Object[] { getPaletteFromMimeType("abc/xyz") } ) );So, here we associate our palette with the TopComponent. And here's the magical method that does the work:
private static PaletteController pc; public static PaletteController getPaletteFromMimeType( String mimeType ) { MimePath path = MimePath.get( mimeType ); Lookup lkp = MimeLookup.getLookup( path ); pc = (PaletteController) lkp.lookup(org.netbeans.spi.palette.PaletteController.class); return pc; } - You will need to declare dependencies on 'Common Palette' and 'MIME Lookup API'. Later, you will need the 'Nodes API', among others, too.
- Now you can already install your module! Open the TopComponent and notice... the Matisse GUI Builder palette is now available to your TopComponent. They open and close together, unless you close one of them separately, and then the connection between them is lost, until they're opened together again (as discussed in detail elsewhere in this blog).
Note: You have to open a regular form first, to initialize its palette, only then can you open your own form, which then steals the first form's initialized palette. Otherwise, you get an ugly error.
- Okay, so now you might say: "Big deal. I have that palette. Okay, it looks nice. But there's nothing I can do with it." Not true. Read Chapter 15 of "Rich Client Programming". There you'll read about adding a JPanel to a TopComponent and how to turn it into a "drop panel", which can accept drops.
- "That's nice, but I have no control over that palette, so there's no way for me to know what I am dragging, so I can't know what is being dropped." Wrong again. Readers of the tutorials at http://platform.netbeans.org/tutorials know that palette items are nodes. Therefore, you can get the selected palette item from the PaletteController and then query the node for its name and its icon. That's all you really want to know from a node in this context anyway.
private static PaletteController pc; String selectedName = null; Image dragImage = null; Lookup selectedPaletteItem = pc.getSelectedItem(); if( null != selectedPaletteItem ) { Node selectedNode = (Node)selectedPaletteItem.lookup( Node.class ); if( null != selectedNode ) { selectedName = selectedNode.getDisplayName(); dragImage = selectedNode.getIcon(BeanInfo.ICON_COLOR_32x32); } }Now that you have the name and the image, you're all set to do whatever you want to do with the palette items. Read the tutorials for all the details about drag images, drop targets, drag gestures, and so on. You can also find information in those tutorials about how to specify what happens when a palette item is dropped, in other words, how to get from the knowledge of the name of the node that defines the palette item to the appearance on your TopComponent of a new widget, and how to define that widget.
Above is where things are at this stage of the story. Note that I haven't been able to get the palette associated with the design/visual view of a multiview API implementation. I'm still investigating that, but the palette appears correctly in normal plain TopComponents.
- At this stage, you have a perfectly useful palette. However, your drop target, i.e., a TopComponent, is far from perfect. You need much more functionality there, which is where the Visual Library comes in. Have a look at the NetBeans Visual Library 2.0, and specifically at Toni's Project Jarvis for all the details involved here. Basically, you want to create a scenario where you get guidelines and other features for helping the user position, and reposition, dropped palette items on a surface.
- And then you need the appropriate code to be generated. For Groovy, for example, which is the ultimate aim that I'm going for here, Groovy code would be generated in the source view whenever a palette item is dropped. However, as stated above, currently I haven't got the Matisse GUI Builder palette to appear in the design/visual view of multiview editors yet.
Probably the above approach isn't the best way to do things. However, it is interesting and, as regular readers of this blog know, this blog is defined more by what is interesting than by what is correct. :-) Hopefully, anyway, this gives a new perspective on the flexibility that you now have when working with palettes. Essentially, you can treat a palette as if it is just a component that can be created in one module and then reused anywhere else. Watch those caveats, though...
In other news. Things are going slowly with my Groovy Editor because I've been distracted by Sun's freshly launched interactive game, "Temple of the Sun". It's funny and it's fun and it's free. I've been able to test it out and it's pretty cool. Folks who finish the game, with the highest scores, will be eligible to win some serious cash and prizes. The top winner will receive $5,000 USD. Anyone can play the game, but you'll need Sun Studio software or Solaris Express Developer Edition to complete it. And they're free. Unfortunately for me, Sun employees aren't eligible. But you can play and win, hurray. Go here to download Sun Studio and go here to get a free media kit of Solaris Express, Developer Edition.
Mar 23 2007, 08:37:06 AM PDT Permalink
Thursday March 22, 2007
Getting Further Into Groovy
A lot of things are simple in Schliemann (which is the brand new NetBeans approach for the declarative creation of editor features, as described on the project page here) or, at least, a lot simpler than before. Other things are still pretty hard. The hardest thing I've yet encountered in Schliemann is code folding. But that was already hard before, so there's no change there. :-) It is hard because you need a grammar, and the grammar has to cover everything in your file, not just the pieces that require code folding. But I'll leave that for another day. The good news is that I've been very pleasantly surprised by, especially, hyperlinking and code completion. I haven't completed either of them for Groovy, but have come quite far.
First, the initial screenshot below is interesting. It shows Groovy syntax coloring and code completion, as well as a Navigator for jumping to relevant parts of the code. The way I implemented it for Groovy, the Navigator lists all the components and properties that it identifies in the current document, and clearly distinguishes between them, and then you can scroll through the list, select an entry, and click it. If the document is closed, it opens when you click the entry. Either way, the cursor lands at the entry in the opened document.
And this is literally everything I had to do to create that functionality:
NAVIGATOR:swingCompProps: {
icon: "/org/netbeans/modules/languages/resources/method.gif";
display_name: "<html><b><font color=000099>PROPERTY:</font></b> <font color='green'>$identifier$</font></html>";
}
NAVIGATOR:swingComps: {
icon: "/org/netbeans/modules/languages/resources/class.gif";
display_name: "<html><b><font color=000099>COMPONENT:</font></b> <font color='red'>$identifier$</font></html>";
}
Next, code completion. I like how code completion works in Schliemann. Again, though, (i.e., as I started discovering yesterday with NBS files) it seems that you need to predefine everything, this time in an XML file. Somehow I think I'd prefer having the code completion entries generated at runtime. But maybe I'm wrong. Anyway, code completion isn't completely complete, but below you can see it in action over the 'button' component. Finally, I implemented syntax coloring for most of the syntax, helped by Jim Clarke from the Groovy list and also from Sun, who sent me code for generating the Swing component properties (however, again I wonder whether these shouldn't be generated on the fly). So, right now, this is how my Groovy editor looks (the 'Design' view doesn't contain anything, just a placeholder for some kind of drag and drop interface that I am hoping to implement):
But that's not all. There's also hyperlinking. You can hold down the Ctrl key and then move the mouse over one of the Swing components, as shown here (the 'textField' is underlined in red below because I am holding down the Ctrl key while moving my mouse over it):
Then you click the link and the browser registered in the IDE opens, if it is closed, or receives focus, if it is already open. Then the applicable Javadoc page for the Swing component under the cursor is found and displayed in the browser. (And if you install Rich's JDIC browser from https://jdic.dev.java.net/, the Javadoc will open inside the IDE.) Hyperlinking has really massively improved under Schliemann. Here, one can also see (as with code completion) that one does use Java here and there. For these slightly more intricate features, you create a reference in the NBS file and then you implement the feature in Java. So, in my Groovy.nbs file I have this declaration:
HYPERLINK:swingComps: org.netbeans.modules.groovyschliemann.Groovy.hyperlink
This declaration says: "For the token named 'swingComps', provide hyperlinks by using the 'hyperlink(Context)' method in the org.netbeans.modules.groovyschliemann.Groovy class."
By adding that declaration to my NBS file, the current context is sent to the 'hyperlink(Context)' method, where a hyperlink is created to a specified destination. And this is my extremely simple implementation of the hyperlink, in this case for Groovy:
public static Runnable hyperlink(Context context) {
if (!(context instanceof org.netbeans.api.languages.SyntaxContext))
return null;
//Not sure what's going on here exactly,
//but this retrieves the name from the reference in the NBS file:
org.netbeans.api.languages.SyntaxContext scontext = (org.netbeans.api.languages.SyntaxContext) context;
org.netbeans.api.languages.ASTPath path = scontext.getASTPath();
org.netbeans.api.languages.ASTToken t = (org.netbeans.api.languages.ASTToken) path.getLeaf();
//Here we have a Swing component's name, retrieved from the above three lines:
final java.lang.String name = t.getIdentifier();
if(name.equals("textField")) {
newName = "JTextField";
} else {
newName = "JButton";
} // else { and so on... some way of mapping SwingBuilder to Swing classes... probably a map.
//Here, in a separate thread, we open the related Javadoc
//in the default browser:
return new Runnable() {
public void run() {
try {
org.openide.awt.HtmlBrowser.URLDisplayer.getDefault().
showURL(new java.net.URL("http://java.sun.com/j2se/1.5.0/docs/api/javax/swing/" +
newName + ".html"));
} catch (MalformedURLException ex) {
Exceptions.printStackTrace(ex);
}
}
};
}
Now compare the above code to the code in the Hyperlink Navigation Tutorial (code similar to that found in this tutorial will also be in the forthcoming book). Notice how much simpler Schliemann makes hyperlinking! And all of the above code I learned from reading the JavaScript.nbs file in the NetBeans sources. Also the NBS.nbs file is really useful.
Finally, for today, as shown earlier, I can drag and drop the individual Groovy documents out of the IDE (just like any other document in 6.0) and work with them separately:
For documents that are separated from the IDE, as shown above, the Navigator continues to work. The Navigator shows the entries for whichever document currently has focus, which could be a document far outside of the IDE. The Navigator itself, just like any other window in 6.0, can be extracted from the IDE in this way. When you close the IDE and reopen it, the separated windows are still separated; in other words, the undocked state is persisted across restarts.
About Schliemann, my current main issues are about grammar (needs to be a visual way of designing grammar or there needs to be a way to generate it from 'antlr' files and 'JavaCC' files, and so on). Without grammar, no code folding. My other issue is that I have all these hard coded tokens which seems not very maintainable. It implies I need to change the NBS file whenever new classes/properties are added to the JDK. There's got to be a more dynamic way of working with Schliemann. Aside from these issues, I'm really impressed. Without any documentation, purely from reading the (very readable) NBS files that are already out there, one can extrapolate a lot of interesting information.
In other news. Do you write modules on top of NetBeans? Do you know about a document called "NetBeans API Changes since Last Release"? No?! That's not good! You better start reading it here. There is a LOT of information there and whenever someone makes changes/improvements to the NetBeans APIs, they are supposed to keep this document updated. You should check it regularly to make sure you're aware of the latest changes. I recommend you check it once a week, over a hot cup of morning coffee, together with a bagel and a brownie.
Mar 22 2007, 01:31:15 AM PDT Permalink
Wednesday March 21, 2007
We've Got a Groovy Kind of Editor
Some fun. I seem to have made a Groovy kind of editor... It has a 'Source' view and a 'Console' view. All of it is built on top of the Manifest Multiview sample, which illustrates one of the chapters in the forthcoming NetBeans Platform book. It does not use the unstable XML Multiview API, but just the standard one that's been in the NetBeans Platform forever. Below is my new Groovy editor in NetBeans IDE, with code that comes from Getting to Know Groovy, which I have found to be a very useful article. (This editor is just a draft, by the way. Not all the tokens are colored, and not all tokens are correctly colored, as explained later in this blog entry.)
Above, the "Source" and "Console" are buttons for toggling between the two views. The "File", "Edit, "Actions", and "Help" are the menus from the Groovy Console. (More about that, especially for NetBeans purists who don't agree with having these menu items in the visual view, further on in this blog entry.)
The two sides are synchronized, so that changes made on one side are set to the other side, in the componentActivated() method on both sides. (Some kind of 'Save' action needs to be added, so that the user can choose whether to synchronize the two sides.) However, the synchronization is handy, because it means the user can use the Source view for typing Groovy code, and then use the Console view for testing it!
Quickly looking under the hood, for those who might be interested, the synchronization approach is really simple. This is code in the source view's definition, which checks to see if the Groovy Console (i.e., the visual view embeds the Groovy Console) was started ("isConsoleRunning" should probably be called "wasConsoleStarted", because it is exited after it is started, as explained in recent blog entries), and then retrieves the content of the input area, pasting it into the NetBeans editor (thanks to a recent discussion on dev@openide.netbeans.org, I now know how to replace Registry.getMostActiveComponent(), as shown below):
public void componentActivated() {
super.componentActivated();
String inputFromVisual = null;
//First check that the console is running,
//which implies the the Console view has been opened
//at least once:
if (GroovyVisualView.isconsoleRunninng()) {
//Get the text from the Groovy Console's input area:
inputFromVisual = GroovyVisualView.getConsole().getInputArea().getText();
//Next, check that there is actually something in the input area:
if (!(inputFromVisual.equals(""))) {
//Get the current NetBeans document:
JEditorPane [] jeps = Utilities.actionsGlobalContext().lookup(EditorCookie.class).getOpenedPanes();
for (int i = 0; i < jeps.length; i++) {
JEditorPane jEditorPane = jeps[0];
//Set the text of the current document to the Groovy Console's input area:
jEditorPane.setText(inputFromVisual);
}
}
}
}
In addition, notice that the Source view has some syntax coloring. (It would be nice if the Console view's input area had the same syntax coloring, but that would mean rewriting the Groovy Console, or implementing my own, because the current input area is a JTextArea, which means I can't set a MIME type. Pity it isn't a JEditorPane or JTextPane, like the output area.) That's based on the Groovy.nbs file in the NetBeans sources, to which I added new tokens for Swing components in Groovy. Can anyone give me a COMPLETE list of Swing components in Groovy (such as "menuItem" and "panel" and so on)? Currently, notice that I haven't identified all the tokens yet (which is why I'm appealing for a complete list here), as a result of which Swing components such as "menuBar" are not colored in the illustration above. So, I need all the Groovy tokens (as well as as much of the grammar as possible), please... otherwise this Groovy editor will not be so groovy. The full impact of Schliemann can only be felt once all the tokens are filled out completely, then code completion, code folding and all the other features can be added without much problem.
All of this probably looks a lot more impressive than it really is to implement it. Each part of it is very trivial. NetBeans purists might argue that the Console's menu bar should not be displayed in the visual view. Instead, the menu items should be migrated to the NetBeans IDE's menu items. Well, I disagree, and here's why... in NetBeans 6, with the undocking functionality, one will be able to undock each of the Groovy documents (and any other documents) from the IDE, so that, for example, this effect results:
Here, doesn't it make sense that the menu bar is in the visual view? This way, I could be working on multiple Groovy documents simultaneously, potentially spread across multiple monitors. The more control I have localized to the document, the more sense it makes to undock my document.
Anyway, there's a lot more to be said about all of the above, but clearly... we've got a Groovy kind of editor. By the way, the same editor infrastructure (i.e., this multiview editor) could be used for any scripting language, if it has some kind of console or other visual view. A simple check could be done on the MIME type, and then, depending on the MIME type, the applicable visual view could be loaded on the visual side of the editor.
This is the only Groovy-specific part in the editor, on the visual side, for loading the console into the visual area and the menu bar into the toolbar:
public MultiViewElement createElement() {
Binding bind = new Binding();
console = new Console(this.getClass().getClassLoader(),bind);
console.run();
consoleRunninng = true;
container = console.getFrame().getContentPane();
menubar = console.getFrame().getJMenuBar();
console.getFrame().setVisible(false);
this.revalidate();
console.exit();
return this;
}
public JComponent getVisualRepresentation() {
return (JComponent) container;
}
public JComponent getToolbarRepresentation() {
return menubar;
}
So, in the methods above, we could check the MIME type and load other stuff into the visual view, depending on the language that we're dealing with. Therefore, this isn't just a Groovy editor, but potentially useful for other languages with a console (or something similar) as well.
Mar 21 2007, 05:13:52 AM PDT Permalink
Tuesday March 20, 2007
Schliemannizing Regular Expressions in NetBeans IDE
I've been trying to find tokens to exercise my newly acquired Schliemann skills on. Eventually, after investigating a variety of interesting scripting languages (including one called MUMPs, which would have made a cool subject line for a blog entry: "Schliemann Has Mumps"), I settled on regular expressions, because NetBeans engineer Martin Adamek has a regular expression tester on nbextras.org (Sandip has a regular expression checker there too). In other words, the tokens already exist. (Do you have tokens? Do they need language support features? Please send them along to me and I'll take good care of them...)
So basically, this is as much a porting guide as instructions for creating syntax colors (and more stuff in the future) for regular expressions. Although, if all you're doing is taking tokens from an existing module, and rewriting them the Schliemann way (i.e., regular expressions), then you're not really porting. So little of the original code can be reused, because you need so little code for Schliemann in the first place.
Interestingly, here we're using Schliemann within a JEditorPane, instead of in the NetBeans editor. That's also totally possible, simply by putting the MIME type in the contentType property of the JEditorPane. So, this is what I did:
- Set up the module. Create a MIME resolver and NBS file, wizards are available for both. Register the two files in the XML layer. Assign the MIME type to the JEditorPane contentType, in the GUI Builder's Properties window. Install the module.
Usefully, because nothing you type matches a token (because you have not defined tokens yet), the entire line/s that you enter in the JEditorPane is/are displayed in red, as shown below:
If part of the line were recognized as a token, that part would not be red, as we shall see later.
- Copy your tokens from RegexTokenContext.java. To understand what the tokens are supposed to represent, see RegexSyntax.java
As you define your tokens, keep on checking your progress, by installing the module again and seeing how much of your code is captured by the token definitions. Here you can see we've made some progress already:
Not quite done yet, clearly, but getting there. And all it took was that I defined the following tokens and colors, partly based on Martin Adamek's regular expression tester on nbextras.org:
TOKEN:string:( "\'" )
TOKEN:identifier:( ["a"-"z"] ["a"-"z" "0"-"9"]* )
TOKEN:operator:( "-" | "+" | "*" | "!" | "=" | "," | ";" )
TOKEN:bracket:( "(" | ")" | "<" | ">" | "[" | "]" )
TOKEN:dot:( "." )
COLOR:identifier: {
foreground_color: "blue";
}
COLOR:operator: {
foreground_color: "blue";
}
COLOR:bracket: {
foreground_color: "red";
font_type: "bold";
}
COLOR:dot: {
foreground_color: "black";
font_type: "bold";
}
COLOR:string: {
foreground_color: "magenta";
font_type: "bold";
}
Now I basically have the basis for further support for regular expressions, such as a navigator and other cool language features. Tools for regular expressions are going to become pretty important in 6.0, I reckon, because so much of Schliemann depends on working with regular expressions. So, the modules by Martin and Sandip (or derivatives of these) in this area are going to be helpful to have around.
By the way, thanks especially to Alex Kotchnev (plus a few others, such as Jochen from the Groovy mailing list), I now have a full blown functioning Groovy Console integrated in NetBeans IDE (plus, including the menu bar). Thanks to the NetBeans windowing system, I can even have two consoles (or as many as my heart desires) open at the same time:
Hurray for Groovy. Next, I'm going to try and do something with the Groovy Compiler. Or are there other things that should be looked at more urgently, either in relation to Groovy or in relation to Grails?
In other news. "Schliemannizing" is a brand new word. It means: "to radically simplify the implementation of, while extensively improving, something", as in the sentence: "He Schliemannized his NetBeans module, and now there are only two files left, instead of 200, while at the same time 10 new features have been added." When the word starts being used everywhere, from the depths of your local pub ("they Schliemannized my favorite beer, so now I get a hangover after just a single drop"), to the middle of a Larry King interview ("has your life improved now that your breasts have been Schliemannized?") , to the high point of the Queen's Christmas speech ("we will do all we can to Schliemannize our next Christmas speech, but chances are that it will be equally dull and at least three times as long"), remember... you heard it here first.
Mar 20 2007, 12:34:42 PM PDT Permalink
Monday March 19, 2007
Groovy Console in NetBeans IDE
I've made some progress embedding the Groovy Console in NetBeans IDE 5.5. Before going into details, here's the current result:
Before going into the remaining problems, let's first look at what it took to get to the current result. First, the good people at Groovy did the world a big favor by creating a single JAR file that includes everything needed for working with Groovy in Java. In the Groovy distribution, get embeddable/groovy-all-1.0.jar, and you're good to go. Wrap that in a library wrapper module project, attached to a module suite, add your functionality module to the suite, declare a dependency on the library wrapper module, and you can start coding for Groovy in Java. Next, you want Groovy files to be opened in a TopComponent. Readers of this blog will know how to do that (use the New File Type wizard and the New Window Component wizard, then create an implementation of the NetBeans API OpenSupport class and change the TopComponent to CloneableTopComponent).
Up to this point, everything is completely standard NetBeans API stuff, as described above. Next, you need to embed the Groovy Console in the TopComponent. That's the tricky bit and is still not perfect. Unfortunately, unlike third-party objects such as the FlashPanel, described here, the Groovy Console is not a JPanel. It is something built on top of a JFrame. So, how to embed a JFrame in a TopComponent? Not easy. The Console object is a single, encapsulated object, we can't move its contents around as we would do if we could open its sources as a JFrame in NetBeans IDE. However, there's a handy method on the Console, called Console.getFrame. That's our way in. That method returns a JFrame, from which we can get the content pane, which we then add as a Container to the TopComponent.
However, for some reason we can only get the content pane after the Groovy Console is running. I don't know why this is the case. Therefore, I call Console.run, then I get the content pane and add it to the TopComponent, and then I call Console.exit. So, when you open a Groovy file in the embedded Groovy Console, you first see a quick flashy thing, which is the Groovy Console opening and closing as a JFrame, while the content pane is added to the TopComponent. Not perfect, but workable. Hope it can be fixed somehow.
The next problem, unresolved currently, is the biggest problem. The method Console.setInputArea(JTextArea) is used to send the content of the Groovy file to the input area of the embedded console. However, once the embedded Groovy Console is opened, the sent text isn't visible! But I know it is there, somewhere in the background or hidden or something like that, because when I press Ctrl-Enter, the content of the input area is run and the result is displayed in the output area. So, if the content of my Groovy file is println "hello world", and I then open the Groovy file in the embedded Groovy Console, I don't see anything in the input area. However, when I press Ctrl-Enter, the line println "hello world" is executed. Very strange.
The final issue, currently, is related to the menu items in the Groovy Console that I am transferring to a toolbar in NetBeans IDE. (If you look in the screenshot above, I have transferred the Run, Previous, and Next items so far.) If I could simply grab the entire menu from the Groovy Console and stick it in the TopComponent, I would rather do that, but grabbing the menu seems impossible. However, the real problem is that even though I can press Ctrl-Enter successfully (i.e., the script executes with the required result), I don't know what to call from the Java code. In contrast, for example, to the "Previous" and "Next" buttons, which call Console.historyPrev and Console.historyNext respectively, there doesn't appear to be something like Console.run(inputArea). One would think that Console.runScript would help, but I haven't figured out how to use that, despite googling and crying and so on. I've tried playing with Robot.keyPress, suggested by Michel Graciano on the dev@openide.netbeans.org mailing list, but it hasn't worked for me so far. Therefore, this is currently another big problem for this embedded Groovy Console solution.
However, in combination with the Schliemann solution that I blogged about in the past few days, it seems clear that this embedded Groovy Console points to the possibility of there being good support for Groovy in NetBeans IDE. And, if someone can help me out with the above problems, I would be extremely thankful.
In other news. Quick! Don't miss this opportunity... here on Amazon.com you can now buy the NetBeans Platform book together with "Harry Potter and the Deathly Hallows (Book 7)". Can life get any better than this?
Mar 19 2007, 05:32:05 AM PDT Permalink
Sunday March 18, 2007
Schliemann on Schliemann
An interesting thing to be aware of is that the NBS file type is itself defined in an NBS file. What does this mean? This means that you can brand the NBS file. Let's say, for example, that we don't like the syntax colors of NBS files. Here's how an NBS file looks by default:
So, let's say that we want the syntax colors to be different. We could, as an end user, just go to the Options window and change the text/x-nbs MIME type's colors. However, let's make the change more permanent. First, find the NBS file that defines NBS files in the "this layer in context" folder. (Expand the "Important Files" node in a module project, then the "XML Layer" node, and then the "this layer in context" node.) And there, within Editors/text/x-nbs, we find our language.nbs file for NBS files:
Pretty cool to see the TOKEN keyword defining tokens, including the TOKEN keyword itself!
Notice also that this NBS file, itself an example of the NBS file it describes, has some colors already. Keywords are blue, comments are grey, and strings are purplish. As pointed out yesterday, there are some tokens that automatically are attributed colors. In fact, if you look inside the file above, you will notice that no colors have been defined at all, so only the three special keywords are colored.
Let's, as an experiment, change the color of tokens in NBS files. Here we assign the color red to the 'keyword' token, thus overriding the color attributed to it by default:
COLOR:keyword: {
foreground_color: "red";
}
Just paste the above anywhere in the file above. When you save the file, notice that the Projects window shows the node for the file in bold. In fact, all the folders leading down to the file are in bold. This signifies that you have changed a file provided by the NetBeans sources. But, it gets even more interesting, because when you look in your package, you notice that a copy of the language.nbs file that you changed is now available in your module and that appropriate entries have appeared in the layer, so that your copy of the NBS file will override the one found in the NetBeans sources:
This means you're good to go. You have a modified version of the NBS file for NBS files and you have the relevant registration entry in the layer. Install the module and notice that all the keywords are now red:
In this way, you can provide syntax coloring for NBS files.
Another important point is that you can look at the language.nbs file for NBS files and learn from it. The documentation describing the syntax is a bit sporadic right now. What better way to learn about the syntax than in the file that defines it? For example, by looking at that specific file, I learned that each entry in the navigator is an HTML document. So, this is how each entry is constructed in the navigator for NBS files:
NAVIGATOR:token: {
display_name: "<html><b><font color=000099>TOKEN</font></b>: $identifier$</html>";
}
That's handy info. Better than looking at the half complete NBS files for Groovy and so on, it's probably better to look at the (also half complete, I guess, but probably more reliable) NBS file that defines NBS files. And it might also appeal to one's enjoyment of the "meta" aspect of all of this too!
Mar 18 2007, 05:29:54 AM PDT Permalink
Saturday March 17, 2007
Programming for Non-Programmers and Heinrich Schliemann
To anyone who is not a programmer, programming is seen as a dry activity, comparable to filling out a tax form or applying for a visa. Fundamentally, it is typing after all, and it is typing code, which, compared to words, is not alive. Code cannot express human longing, cannot express hope and passion, does not have the vocabulary for pain, remorse and loss (unless you consider garbage collection a subset of loss). Of course, this is comparing apples with oranges. Code has no need to express those things. But, just because it has no need to express those things and therefore doesn't, a dry activity it is not. It does engage in and interact with language. And this is where the NetBeans APIs are interesting, because one of their strengths lies in their ability to work with language. Code completion, syntax highlighting, and other language support features such as these, are products that help someone work in an editor with a programming language, similar to how a dictionary and a thesaurus are products that help someone with a human language. And the ability to provide these features is central to the NetBeans APIs. It is for this reason that I argue, also in the interview with Roman recently, that in the area of editor functionality, the NetBeans APIs give the NetBeans Platform a very strong reason for existence. Whatever happens in the direction of JSR-296, the editor functionality exposed by the NetBeans APIs will be a fundamental reason for building editors on top of it, rather than on top of an alternative.
So if you are from the "programming is dry" school of thought, set aside that perspective for a minute while I show you some cool new toys that facilitate the interaction between language and programming. Hopefully, at the end of it all, you'll see that there is a fun, creative element to programming, something akin to a jigsaw puzzle or a rubik's cube, rather than the image of the lone programmer hunched over a keyboard with only a flickering green screen for company. The latter image is from an outdated movie anyway, so you're showing your age, my anti-programming friend. For readers of this blog who are programmers, specifically those who are familiar with the plethora of NetBeans APIs involved in providing editor functionality, you're about to be amazed, if you're not yet familiar with something known as NetBeans' "Project Schliemann". You're probably aware of the fact that providing editor functionality involves implementing the Editor Completion API, the Syntax Highlighting API, the Hyperlinking API, the Navigator API, the Code Folding API, and about 10 other APIs, all described in some detail in the upcoming Rich Client Programming: Plugging into the NetBeans Platform. Let's now look at the Schliemann alternative to all of these APIs.
A fundamental point to understand is that Schliemann is a framework for language programming. What does that mean? Common to all frameworks everywhere is that they provide an infrastructure, with the intention that the framework's client can focus on only those things that are important to the client and not those things that are common to each and every other client. For example, as Prakash Narayan said recently in his presentation at the Sun Tech Days in Hyderabad, generally you don't need to worry about your house's infrastructure, such as its plumbing. Except on rare occasions when the plumbing goes faulty, which implies a failure of the infrastructure, you can live very comfortably in your house without worrying about its plumbing. You don't even need to know about the plumbing. All you need is the phone number of a good (and cheap) plumber. (Isn't this simultaneously an argument for modular development and an example of loose coupling?) Similarly, the Struts Web Framework, and the hundreds of other web frameworks, provides the plumbing of your web application and the NetBeans Platform provides the plumbing of your Java desktop application. In the former case, you get a struts-config.xml file to handle the navigation between pages, while the latter gives you a window system on top of which you build your desktop application. Navigation is a fundamental concern for all web applications, while a window system is a fundamental concern for all Java desktop applications that have outgrown the JFrame.
What's the connection between all of this and Schliemann? Typically, when you're providing language support for a programming language, you need to define a multiplicity of Java classes, which extend the NetBeans APIs, for all the various features you're providing. And this needs to be done by every provider of new features for every new programming language. However, all along you, as a provider of the language support features, are really ONLY interested in describing the language. In the realm of human languages, this situation is comparable to someone creating a new language and then being told: "Your language is only official once you have applied in triplicate to the United Nations, the European Union, NATO, and the World Bank. And remember that each has a different submission procedure and each has different forms to fill in!" As a framework, Schliemann provides all this "red tape" right out of the box. You simply need to define your language and then state where you want to use which parts of it. Some parts you might want to give a particular syntax color, other parts you might want to show in the Navigator, and so on. This means the end of bureaucracy (which can be slow!) for language support features, allowing you to focus on your expert area... that of the programming language itself. Read the "Goals", "Overview", and "Why Schliemann?" sections of the Schliemann Project page to see that these are the exact underlying concerns for this project's existence. From personal experience, I've worked on tutorials for providing features such as syntax highlighting in NetBeans IDE, code completion in NetBeans IDE, and so on, as well as in the context of the book, so I know how much work it is just to describe these APIs, never mind learn them. Of course, doing so is necessary in many situations and Schliemann doesn't pretend to be a golden bullet. Schliemann provides a framework only for those programming languages that do not require compilation. Which is why... it is specifically useful for scripting languages. It is a light weight solution, intending to facilitate writers of scripting languages everywhere to quickly incorporate their language into NetBeans IDE. Think of it... what better way to improve the use of your scripting language than to provide support for it in an existing editor? This is as extreme as asking: "What better way to increase the use of your human language than to provide a dictionary?"
So, with that background, let's look at the Schliemann approach to manifest files. Although manifest files do not use a scripting language, they are not compilable, hence provide the necessary requirements for Schliemann's applicability. At the same time, I'm hoping to counter the "programming is dry" anti-pattern, by showing that its engagement with language should be very appealing to the non-programming soul.
When dealing with a programming language in the context of editors, a fundamental concern is tokens. Before the non-programmer's eyes glaze over, let's quickly state that "verb", "noun", and "adjective" are tokens of human languages. Okay? No worries, we're just dealing with parts of speech. So, we need to define the parts of speech of our language. Here's an example of a manifest file:
Name: java/util/ Specification-Title: "Java Utility Classes" Specification-Version: "1.2" Specification-Vendor: "Sun Microsystems, Inc.". Implementation-Title: "java.util" Implementation-Version: "build57" Implementation-Vendor: "Sun Microsystems, Inc."
What parts of speech can we deduce from the above manifest file? We see a colon-separated list of entries. Before the colon we see something normally referred to as the "key", after the colon we see something often called the "value". There's always something before the colon, there's always a colon, and there's always something after the colon. Often, there may be a comment in the manifest file, like this:
#need to check with Bill about this one:
In this case, the comment is just a note to the person creating the manifest, and one hopes that it will be removed at the time that the application is packaged and distributed.
So, here we have the following parts of speech (i.e., tokens): key, colon, value, comment. Sounds like the basis of a language! Wouldn't it be nice if all the keys had one color, all the colons another color, all the values yet another color, and the comments were easily discernible somehow? First, lets looks at the Manifest File Syntax Highlighting Tutorial to see how this is traditionally done in NetBeans IDE. Note that since the writing of that tutorial, things have simplified to some extent, via something known as "incremental lexing" (a topic deserving a separate blog entry, such as the one here, where the creator of the Lexer modules is interviewed in this blog) via the NetBeans Lexer modules, which are described in the upcoming book. Again, the fundamental problem is that, in the tutorial, we're doing a lot more than creating tokens. It would be pretty cool if we could define the tokens using regular expressions, like this:
TOKEN:comment:( "#" [^ "\n" "\r"]* ["\n" "\r"]+ )
TOKEN:key:( [^"#"] [^ ":" "\n" "\r"]* ):<VALUE>
<VALUE> {
TOKEN:whitespace:( ["\n" "\r"]+ ):<DEFAULT>
TOKEN:operator:( ":" ):<IN_VALUE>
}
<IN_VALUE> {
TOKEN:whitespace:( ["\n" "\r"]+ ):<DEFAULT>
TOKEN:value:( [^ "\n" "\r"]* )
}
Maybe slightly dense on first reading, but the above tokens make use of something called state. Don't let your eyes glaze over yet, and remember that everything before a colon is a key and everything after it is a value. Therefore, if we know where we are, we know what token is applicable. (If I know you're in the state "drunk", then I know the token "random insults" is forthcoming.) Here the concept of a "tokenizer", also known as "lexical analyzer", is applicable too. Inside NetBeans IDE is a "lexical analyzer", something that reads a document whenever it is opened in NetBeans IDE. This lexical analyzer reads a document and, if the tokens above are assigned to the type of document in question, the lexical analyzer applies the tokens to the content of the document. In the above example, if the first character it finds in a line is #, the lexical analyzer is entering a comment. That's what the first line above says. If the first character in a line is not a #, the lexical analyzer is in a key. When a colon is reached, as specified in the second line above, the lexical analyzer is moving towards a value. That's what the second line above tells us. Then the <VALUE> section above is entered. When a colon is reached, the line enters the <IN_VALUE> state. That, briefly, is what happens line by line in a document that is opened in NetBeans IDE, if the document is assigned to the above file. To see how that is done, see yesterday's blog entry. All you need to do is define a small XML file called a MIME resolver, which associates a file extension (here it would be "mf") to a MIME type (here it could be "text/mf") and then register it in the XML layer, where the file containing the tokens above is also registered. Note that the file should have the "nbs" file extension, which stands for "NetBeans scripting".
So, now we have tokens (i.e., parts of speech in our manifest file language). In the Manifest File Syntax Highlighting Tutorial, you will see that this is MUCH more difficult when done in Java. A knowledge of regular expressions is all one really needs to create the above tokens (and there are plenty of tutorials on that, such as here in the Swing tutorials).
Okay, great, now we have tokens. So, now what? In the same file above, we can immediately assign colors to tokens:
COLOR:key: {
foreground_color: "blue";
}
COLOR:operator: {
foreground_color: "black";
}
COLOR:value: {
foreground_color: "magenta";
}
Note that we didn't assign a color to the "comment" token. Why? Because tokens with this name receive a grey color automatically by the Schliemann support modules. Another token, "keyword", is automatically dark blue. We could use that keyword here, and even override its color, but here we've used the token name "key" instead. (Note to self: Must find out what all the pre-defined tokens are.)
So, great, we have colors. You can, literally, install the module right now, and you'll have syntax coloring for manifest files. However, Schliemann provides much more. Normally to the left of the editor in NetBeans IDE, you see something called a "navigator", which lets you jump to places in the editor itself. Creating a navigator is not the most difficult part of the NetBeans APIs, but still requires some coding, as shown here in this blog. However, this is all that we now need to do, in the same file where we define the tokens and colors:
NAVIGATOR:key: {
icon: "/org/netbeans/modules/languages/resources/method.gif";
}
That's it! We will now have a navigator, listing the keys in the manifest file, together with the "method.gif" icon from the NetBeans sources. It could not be simpler. However, let's complicate matters and suppose that we don't only want to display the key, but also the colon and value. To do that, we first need to create a set of grammar rules. As in human languages, a grammar rule determines the correct combinations of tokens, and the correct order in which they can be expressed. So, here are our grammar rules, again in the same file as above:
StatementRule = KeyRule OperatorRule ValueRule WhiteSpaceRule; KeyRule = <key>; OperatorRule = <operator>; ValueRule = <value>; WhiteSpaceRule = <whitespace>;
So we create a grammar rule for what a "statement" consists of, which in this case is a rule composed of each of the tokens that we defined earlier. Now we can use the grammar rules in the navigator:
NAVIGATOR:StatementRule: {
icon: "/org/netbeans/modules/languages/resources/method.gif";
}
This is the result, we now have colors and a navigator for manifest files:
Let's note three things in the context of the navigator. Firstly, we have a full-blown navigator, because Schliemann provides a framework. For example, we can click an entry in the navigator, and then the editor will open (if closed) and the cursor will land on the line corresponding to the entry in the navigator. All we needed to supply was the content and the icon. (And, if you supply no icon, a default one is supplied for you.) Secondly, think about debugging... if something is wrong, where are we going to look for the problem? In one of 10 different Java classes? No, only in the definition shown above. That is a big plus. Thirdly, note that adding additional functionality is really easy. For example, here's the same navigator, but this time with a tooltip that appears when the mouse hovers over an entry in the navigator:
NAVIGATOR:StatementRule : {
tooltip: "Click to change value $ValueRule$";
icon: "/org/netbeans/modules/languages/resources/method.gif";
}
The biggest issue with working with language features this way is that one needs to be comfortable with the Schliemann language used in the "nbs" file, which is really a subset of JavaCC and regular expression language. This document provides guidance, but a lot more needs to be done. For example, code completion for "nbs" files and maybe a visual editor as well. However, clearly, for the languages that fall in the ambit of Schliemann's scope, this is really a fantastic innovation that lifts the NetBeans Platform to a new level. Together with the Visual Library, Schliemann shows the continual reinvention and innovation of the NetBeans Platform, the ongoing search for improvements and simplifications, all aiming to make the life of the end user (i.e., a developer) less cumbersome. And, hopefully, I've also shown that this interaction between language and programming makes the latter far from a "dry" activity.
I plan to blog quite a bit more about Schliemann, because here I've only shown syntax coloring and navigator support. Code folding, code completion, hyperlinking, and a variety of other language features are supported or are planned to be supported, in similar ways to syntax coloring and the navigator above. In the process, it is good to note that Schliemann is flexible enough to encompass a variety of approaches, such as the "state" approach to syntax highlighting shown here, but also shown in the fact that Java programming can play a role in adding more detailed sophistication to a bare-bones Schliemann implementation. And why is it called Schliemann anyway? Read yesterday's blog entry to find out!
Mar 17 2007, 10:04:59 AM PDT Permalink
Friday March 16, 2007
Hello Schliemann
Time to meet Heinrich Schliemann. "Schliemann wrote his diary in the language of whatever country he happened to be in," his entry in Wikipedia tells us. "Schliemann had a gift for languages and by the end of his life he was conversant in English, French, Dutch, Spanish, Portuguese, Swedish, Italian, Greek, Latin, Russian, Arabic and Turkish as well as his native German."
As a tribute to Schliemann, there's a NetBeans project for defining programming languages declaratively and integrating them in NetBeans IDE. Undoubtedly, the best resources on this project are:
- The Schliemann Project Page on NetBeans Wiki
- The Schliemann Project Page on netbeans.org
- The Technical Blog by Jan "Hanz" Jancura, who is the Schliemann Project's lead engineer.
If you keep up with the above three resources, you'll end up knowing everything there's worth knowing. But how about a brief introduction, just to get our feet wet? I asked Hanz for exactly that a few days ago and below follows everything he told me. It is based on the daily build of NetBeans 6.0 from the 13th of March. (I heard rumors that the functionality was removed after that, for lack of stability, so I suggest you take the daily build specifically from March 13th, which is what I used without any problems, other than those mentioned below.)
Below, we create a new language support for files with the imaginary "foo" extension. At the end of the story, our "foo" files will look as follows in NetBeans IDE:
In the above screenshot, notice the following:
- syntax coloring
- Navigator window support
- code folding
- indentation
What you don't see is brace completion, and a right-click pop-up menu with items for formatting and collapsing/expanding code folds.
And... not one single Java file was created (as you can see in the Projects window above) to make all of this happen.
Setting Up the Development Environment
- Install NetBeans IDE 6.0 dev build. (As pointed out above, I know the build from the 13th of March works. I don't know this to be true for any other build, although it possibly is true for builds after the 13th.)
- Check that the Generic Language Framework is installed by going to the Module Manager. In the Module Manager, you should see a category called "Languages Support". Within that category you should see "Generic Languages Framework".
- Go to the Update Center wizard, where you will find the Development Update Center. In the Development Update Center, find "Generic Languages Framework Studio". Importantly, this module must have 1.10 (or later) as its specification version, which you can see in the Update Center wizard. After you install this module, you should see "Generic Languages Framework Studio" under the "Languages Support" category in the Module Manager.
Creating Language Support for "Foo" Files
- Create a new module project and name it whatever you like.
- Add a dependency on the "Generic Languages Framework".
- In the New File template, choose the brand new template shown below and create a file called "foo", which will have the "nbs" (NetBeans Scripting) extension:
This NBS template contains some example language definitions, as shown below, which will define our "foo" language:
# # NBS Template # # definition of tokens TOKEN:keyword:( "while" | "if" | "else") TOKEN:operator:( "{" | "}" | "(" | ")" ) TOKEN:identifier:( ["a"-"z"] ["a"-"z" "0"-"9"]* ) TOKEN:whitespace:( [" " "\t" "\n" "\r"]+ ) # parser should ignore whitespaces SKIP:whitespace # definition of grammar S = (Statement)*; Statement = WhileStatement | IfStatement | ExpressionStatement; WhileStatement = "while" "(" ConditionalExpression ")" Block; IfStatement = "if" "(" ConditionalExpression ")" Block; Block = "{" (Statement)* "}"; ConditionalExpression = <identifier>; ExpressionStatement = <identifier>; # code folding FOLD:Block # navogator support NAVIGATOR:WhileStatement:"{$ConditionalExpression}" # brace completion COMPLETE "{:}" COMPLETE "(:)" # indentation support INDENT "{:}" INDENT "(:)" INDENT "\\s*(((if|while)\\s*\\(|else\\s*|else\\s+if\\s*\\(|for\\s*\\(.*\\))[^{;]*)"For info on the syntax of NBS files, see the three resources mentioned at the start of this blog entry, such as here in Hanz's blog. And here is the official Schliemann Language Definition.
- Create a new foo.xml file in your main package, with the following content:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE MIME-resolver PUBLIC "-//NetBeans//DTD MIME Resolver 1.0//EN" "http://www.netbeans.org/dtds/mime-resolver-1_0.dtd"> <MIME-resolver> <file> <ext name="foo"/> <resolver mime="text/foo"/> </file> </MIME-resolver>This file maps the "foo" file extension to the MIME type text/foo.
- Next we register foo.xml and foo.nbs in the XML layer:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE filesystem PUBLIC "-//NetBeans//DTD Filesystem 1.1//EN" "http://www.netbeans.org/dtds/filesystem-1_1.dtd"> <filesystem> <folder name="Services"> <folder name="MIMEResolver"> <file name="foo.xml" url="foo.xml"/> </folder> </folder> <folder name="Editors"> <folder name="text"> <folder name="foo"> <file name="language.nbs" url="foo.nbs"/> </folder> </folder> </folder> </filesystem> - Our new module should be ready now, except that the Schliemann Project itself isn't ready. Therefore, even though you can compile and install the module, you'll probably need to restart the IDE for code folding (and possibly other features) to work. So, after installing the module and restarting the IDE, create a new file called "newfile.foo" and then type or paste in the following text:
while (one) { two if (three) { four } } while (five) { six }
And that's it. You should now see the result shown in the screenshot at the start of this blog entry. If these instructions don't work for you, that's because this is really cutting edge stuff and Schliemann isn't completely complete yet. Why? Because he's still studying... give him some time and he'll speak your language too. (Check out the planned features here.)
In other news. The icon shown in the screenshots above for the new NBS file type reminds me a lot... of Superman... Hmmm... interesting. By the way, speaking of icons, note that by default the icon of your file type (here "newfile.foo") is the same icon as used by the NBS file type. However, you can change that declaratively in the XML layer, as an attribute of the element that registers the NBS file. For example, here I'm reusing the JavaScript icon from the NetBeans sources:
<file name="language.nbs" url="foo.nbs"> <attr name="icon" stringvalue="org/netbeans/modules/languages/javascript/JavaScript.png"/> </file>
Mar 16 2007, 04:43:56 AM PDT Permalink