« Transforming Midi... | Main | Programming for... »

Download NetBeans!

20070316 Friday March 16, 2007

Hello Schliemann

Time to meet Heinrich Schliemann. "Schliemann wrote his diary in the language of whatever country he happened to be in," his entry in Wikipedia tells us. "Schliemann had a gift for languages and by the end of his life he was conversant in English, French, Dutch, Spanish, Portuguese, Swedish, Italian, Greek, Latin, Russian, Arabic and Turkish as well as his native German."

As a tribute to Schliemann, there's a NetBeans project for defining programming languages declaratively and integrating them in NetBeans IDE. Undoubtedly, the best resources on this project are:

If you keep up with the above three resources, you'll end up knowing everything there's worth knowing. But how about a brief introduction, just to get our feet wet? I asked Hanz for exactly that a few days ago and below follows everything he told me. It is based on the daily build of NetBeans 6.0 from the 13th of March. (I heard rumors that the functionality was removed after that, for lack of stability, so I suggest you take the daily build specifically from March 13th, which is what I used without any problems, other than those mentioned below.)

Below, we create a new language support for files with the imaginary "foo" extension. At the end of the story, our "foo" files will look as follows in NetBeans IDE:

In the above screenshot, notice the following:

  • syntax coloring

  • Navigator window support

  • code folding

  • indentation

What you don't see is brace completion, and a right-click pop-up menu with items for formatting and collapsing/expanding code folds.

And... not one single Java file was created (as you can see in the Projects window above) to make all of this happen.

Setting Up the Development Environment

  1. Install NetBeans IDE 6.0 dev build. (As pointed out above, I know the build from the 13th of March works. I don't know this to be true for any other build, although it possibly is true for builds after the 13th.)

  2. Check that the Generic Language Framework is installed by going to the Module Manager. In the Module Manager, you should see a category called "Languages Support". Within that category you should see "Generic Languages Framework".

  3. Go to the Update Center wizard, where you will find the Development Update Center. In the Development Update Center, find "Generic Languages Framework Studio". Importantly, this module must have 1.10 (or later) as its specification version, which you can see in the Update Center wizard. After you install this module, you should see "Generic Languages Framework Studio" under the "Languages Support" category in the Module Manager.

Creating Language Support for "Foo" Files

  1. Create a new module project and name it whatever you like.

  2. Add a dependency on the "Generic Languages Framework".

  3. In the New File template, choose the brand new template shown below and create a file called "foo", which will have the "nbs" (NetBeans Scripting) extension:

    This NBS template contains some example language definitions, as shown below, which will define our "foo" language: 

    #
# NBS Template
#

# definition of tokens
TOKEN:keyword:( "while" | "if" | "else")
TOKEN:operator:( "{" | "}" | "(" | ")" )
TOKEN:identifier:( ["a"-"z"] ["a"-"z" "0"-"9"]* )
TOKEN:whitespace:( [" " "\t" "\n" "\r"]+ )

# parser should ignore whitespaces
SKIP:whitespace

# definition of grammar
S = (Statement)*;
Statement = WhileStatement | IfStatement | ExpressionStatement;
WhileStatement = "while" "(" ConditionalExpression ")" Block;
IfStatement = "if" "(" ConditionalExpression ")" Block;
Block = "{" (Statement)* "}";
ConditionalExpression = <identifier>;
ExpressionStatement = <identifier>;

# code folding
FOLD:Block

# navogator support
NAVIGATOR:WhileStatement:"{$ConditionalExpression}"

# brace completion
COMPLETE "{:}"
COMPLETE "(:)"

# indentation support
INDENT "{:}"
INDENT "(:)"
INDENT "\\s*(((if|while)\\s*\\(|else\\s*|else\\s+if\\s*\\(|for\\s*\\(.*\\))[^{;]*)"

    For info on the syntax of NBS files, see the three resources mentioned at the start of this blog entry, such as here in Hanz's blog. And here is the official Schliemann Language Definition.

  4. Create a new foo.xml file in your main package, with the following content: 

    <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE MIME-resolver PUBLIC "-//NetBeans//DTD MIME Resolver 1.0//EN" 
"http://www.netbeans.org/dtds/mime-resolver-1_0.dtd">

<MIME-resolver>
    <file>
        <ext name="foo"/>
        <resolver mime="text/foo"/>
    </file>
</MIME-resolver>

    This file maps the "foo" file extension to the MIME type text/foo.

  5. Next we register foo.xml and foo.nbs in the XML layer: 

    <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE filesystem PUBLIC "-//NetBeans//DTD Filesystem 1.1//EN" 
"http://www.netbeans.org/dtds/filesystem-1_1.dtd">

<filesystem>
    
    <folder name="Services">
        <folder name="MIMEResolver">
            <file name="foo.xml" url="foo.xml"/>
        </folder>
    </folder>

    <folder name="Editors">
        <folder name="text">
            <folder name="foo">
                <file name="language.nbs" url="foo.nbs"/>
            </folder>
        </folder>
    </folder>
    
</filesystem>

  6. Our new module should be ready now, except that the Schliemann Project itself isn't ready. Therefore, even though you can compile and install the module, you'll probably need to restart the IDE for code folding (and possibly other features) to work. So, after installing the module and restarting the IDE, create a new file called "newfile.foo" and then type or paste in the following text: 

    while (one) {
   two
   if (three) {
      four
   }
}
while (five) {
   six
}

And that's it. You should now see the result shown in the screenshot at the start of this blog entry. If these instructions don't work for you, that's because this is really cutting edge stuff and Schliemann isn't completely complete yet. Why? Because he's still studying... give him some time and he'll speak your language too. (Check out the planned features here.)

In other news. The icon shown in the screenshots above for the new NBS file type reminds me a lot... of Superman... Hmmm... interesting. By the way, speaking of icons, note that by default the icon of your file type (here "newfile.foo") is the same icon as used by the NBS file type. However, you can change that declaratively in the XML layer, as an attribute of the element that registers the NBS file. For example, here I'm reusing the JavaScript icon from the NetBeans sources:

<file name="language.nbs" url="foo.nbs">
   <attr name="icon" stringvalue="org/netbeans/modules/languages/javascript/JavaScript.png"/>
</file>

Mar 16 2007, 04:43:56 AM PDT Permalink

Trackback URL: http://blogs.sun.com/geertjan/entry/hello_schliemann
Comments:

Geertjan, thanks for the blog entry and thanks to Jan Jancura and his team for this. I think Shliemann is going to be yet another feather in NetBean's hat.

I have started a module to support Windows Powershell editing in NetBeans. I was able to quickly get some stuff working, such as syntax coloring. I am struggling with the Powershell grammar (partly due to some gaps in the available reference material) but once I have it I should be able to complete the rest of the support.

See:

http://blogs.sun.com/scblog/entry/getting_started_with_new_generic

I have been interested in this topic of generation of editor support based on language grammar and some additional meta information. There are many research projects such as Harmonia/Pan at UC Berkeley.

http://harmonia.cs.berkeley.edu/harmonia/index.html

IMHO the Schliemann project has hit the sweet spot of how much meta information one needs to supply and how you get a gradually improving support proportional to the effort you put in to write the NBS file. Really cool!

I think expressing the grammar is the only complex part of writing NBS file. I hope someday you help us by writing one of your cool tutorials on that topic.

There is an idea for writing a module for:

  1. Visually expressing/building the grammar - say using a rail road diagram.
  2. Letting a tool figure out the grammar based on example snippets of code (dreaming here :) )

Posted by Sandip on March 16, 2007 at 07:37 AM PDT #

Very good and very nice :).
This is the type of article/feature developers need :D.

Thank you,

Ahmed.

Posted by Ahmed Mohombe on March 16, 2007 at 07:37 AM PDT #

Thanks for your interest Sandip and Ahmed. Sandip, I agree with your points and will blog about these issues soon. FYI, I will be showing how to create language support for Manifest files, probably tomorrow, with the following result (notice the syntax coloring and navigator):

<img border="1" src="http://blogs.sun.com/geertjan/resource/schliemann3.png">

Posted by Geertjan on March 16, 2007 at 09:35 AM PDT #

Sorry. Click here to see the screenshot.

Posted by Geertjan on March 16, 2007 at 09:36 AM PDT #

I wonder if there are any legal issues with using the "Superman" logo.

Posted by 129.49.16.170 on March 20, 2007 at 07:23 AM PDT #

Good question. (But is it really the Superman logo? I was just being facetious in this blog entry. Not sure what defines the Superman logo, nor if this falls within the definition.) It is probably a temporary thing anyway.

Posted by Geertjan on March 20, 2007 at 07:26 AM PDT #

Very nifty! Schliemann seems like a domain specific language for language definitions... very cool. This may be pushing it, but can you do things like syntactic or semantic lookahead in the grammar definition? I've been doing a lot of JavaCC lately and that sort of thing is quite handy...

Posted by Tom Copeland on March 27, 2007 at 06:27 AM PDT #

Hi Tom, you've clearly understood the story. The point is that grammars provided by JavaCC (and ANTLR and so on) cannot be directly applied to an IDE. So, yes, Schliemann or, more correctly, the NBS file format, provided a NetBeans-compliant grammar for a programming language. So, the big question is: "How do I get from JavaCC/ANTLR/etc to an NBS format?" And that question is something I am going to try to help out with, via various forms, such as... interviews with a few people who have acquired some experience with this. So, watch this space, because I'll be making some announcements about this soon.

About look aheads, yes, LL(1) is supported and, from what I understand, LL(4), since a recent bug fix, as well.

Posted by Geertjan on March 27, 2007 at 08:09 AM PDT #

"provided" in my comment above should be "provides"

Posted by Geertjan on March 27, 2007 at 08:10 AM PDT #

Geertjan, sounds great. Thanks for the pointer to Jan's blog, I read through his overview, looks like good stuff. It's a hard problem, that's for sure... just thinking now about lexical states and how they could fit into Schliemann somehow, because they're quite handy. It's a tricky business, since you'd need to annotate which state a token is in, for example: 
<STATE_A>
TOKEN : {
<A : "A">
 | <B : "B">
}
<STATE_B,STATE_C>
TOKEN : {
 <C : "C">
 | <D : "D">
}

and so forth. Anyhow, great stuff, best of luck!

Posted by Tom Copeland on March 27, 2007 at 07:48 PM PDT #

Hi Tom, yes lexical states are supported already. For manifest file syntax highlighting, for example, one state is 'before the colon', the other state is 'after the colon' (and then there is the state 'colon' and the state 'end of line'). That approach is described in Programming for Non-Programmers and Heinrich Schliemann. What you also might find interesting to note is that Schliemann is itself defined in the Schliemann way (see Schliemann on Schliemann).

Posted by Geertjan on March 28, 2007 at 03:32 AM PDT #

Geertjan - Thanks for the info, very cool indeed!

Posted by Tom Copeland on March 28, 2007 at 03:44 PM PDT #

Yeah, powerfull feature I think, these is good collaboration for : 1. Netbeans 2. Schliemann 3. PHP (php.net) 4. Prado (pradosoft.com) 5. Quercus (http://quercus.caucho.com/) So, Netbeans really powerfull and PHP developper be happy. Agus S

Posted by Agus Suhartono on April 03, 2007 at 12:34 AM PDT #

Looks cool. How can we get the source code for this module and build it ourselves? That's one of the best ways I've found to see how something works. It would also be great to see examples of NBS files. Thanks!

Posted by Charlweed on May 08, 2007 at 05:18 PM PDT #

Found it by grepping! FOI it is at the cvs root ./languages, and built in NetBeans with build with depends. No problem! Now to figure it out....

Posted by Charlweed on May 08, 2007 at 06:19 PM PDT #

Wow, this looks great. Good introduction. This will be cool for groovy. But here my question. I downloaded nb6m9 to experiment with schliemann and write a nbs-file for a specific language but I cannot find "Generic Languages Framework Studio". I looked in tools->Plugins->Updates and new plugins but cannot find it. Any hint? Or where can I find the build from the 13th of March?

Posted by Hajo on June 28, 2007 at 12:53 PM PDT #

Hi Geertjan, greetings from Göttingen!

I'll want to play around with schliemann and the Generic Languages Framework (Studio). Is it possible to install the module(s) (yes I know: plugin!) in 6beta1?

best regards,
josh.

Posted by Aljoscha Rittner on September 27, 2007 at 04:42 AM PDT #

Hi Josh! I haven't seen the studio plugin in 6.0 Beta 1. I think you would need to use a daily 6.0 build (any 6.0 daily build) for that, until 6.0 is released. I have a LOT of info in this blog about Schliemann, so there's a lot of info that shohld help you. What language support are you thinking of creating by the way? Just curious.

Posted by Geertjan on September 27, 2007 at 04:45 AM PDT #

I've two little projects. The first is currently a declarative rule language for data distribution in my CRM software.

The second project is an editor for the "Sieve mail filtering language" (see http://www.rfc-editor.org/rfc/rfc3028.txt). I need a filter-editor for my integrated mail client. But it should be a dual-editor with source and design mode.

Ok, I'll test a daily build :-)

Posted by Aljoscha Rittner on September 27, 2007 at 06:18 AM PDT #

I have few editor projects that used to work great with 6.0 -
but they do no work with 6.7.1 release - Is there some new steps I need to invoke after "Install/Reload in development IDE" ?

Posted by Hi Josh on September 04, 2009 at 01:25 PM PDT #

Post a Comment:

Name:
E-Mail:
URL:

Your Comment:

HTML Syntax: NOT allowed