Languages Embedding (in GLF)

During the last month I was working on languages embedding mainly, so I have two screenshots. Languages embedding is not easy and there are many interesting issues. But it is a long story.

The first picture shows HTML with JavaScript and CSS inside a PHP file:



You can see that blocks of PHP code can break one token of other language (JavaScript). It means that you have to preprocess PHP file and connect all blocks of inner code, if you want correctly parse language embedded inside PHP.

Current version of GLF (Generic Languages Framework) supports three types of embedding:

1) Token based embedding: One token of outside language can be reparsed by some other language's parser. For example CSS in HTML. Definitions of both languages can be merged together in this case (theoretically). So, you can define one common tokenizer for both languages, and one grammar.

HTML tokens <token type, token text>:

<tag,"<script>">
<text,"a {\n  color: #454545;\n}">
<end_tag,"<\script>">

HTML text token in this example is replaced by several CSS tokens. Tokens of outside language are not broken. So you can parse outside language without any constraints.

2) Preprocessor (templates) like embedding: Outside language contains blocks of some other language. Borders between languages are recognized by some preprocessor. (PHP, Velocity, Freemaker, EJS, RHTML, ...). EJS (Phobos) example:


There are two kinds of JavaScript in this example - server side and client side. Output of server side JavaScript can generate blocks of client side JavaScript code :-)

3) Last possibility is importing definition of one GLF based language to another one, or to some state of outer language tokenizer. So you can import definition of HTML language to PHP language definition.

So, GLF definition of PHP looks like:

IMPORT:html {
    mimeType:"text/html2";
    state: "DEFAULT";
}

IMPORT:php {
    mimeType:"text/x-php2";
    start:( "" );
    background_color:"#EEEEBB";
}

The first import loads definition of HTML (with all HTML embedded languages) to default state of PHP tokenizer. And second import defines "template" language (blocks between "" should be parsed by php2 language). "text/x-php2" language defines structure of plain PHP.

Posted at 04:40PM Mar 08, 2007 by Jan Jancura in NetBeans  |  Comments[8]

Project Schliemann

What is Schliemann Project about? You have probably already seen some programmers editors like Emacs, Vim or JEdit. They have editing support for many programming languages (~100). But they do not contain special hand coded support (module) for every language. They defines some Generic Languages Framework only. Every programming language is described in some file executed by the engine. Schliemann project implements Generic Languages Framework for NetBeans IDE.

Schliemann engine allows you to describe some programming language and define how to integrate it to the NetBeans. Each programming language is defined in one nbs (NetBeans Scripting) file. New declarative language has been created for that purpose.
So, what can you do in nbs files?

You can define lexical structure of your language. Tokens are described by regular expressions, syntax is similar to JavaCC.
Example:

    TOKEN:keyword: ( "if" | "else" | "while" | "for" | "function" )
    TOKEN:comment: ( "/*" - "*/" )
    TOKEN:line-comment: ( "//" [^ "\n" "\r"]* )
    TOKEN:number: ( ["0"-"9"]+ )
    TOKEN:whitespace: ( [" " "\t" "\n" "\r"]+ )
    TOKEN:identifier: ( ["a"-"z" "A"-"Z"]+ )

Now you can define coloring for your tokens:

    COLOR:line-comment {
        foreground-color: "lightGray";
        font-type: "bold";
    }

Definition of indentation is simple. You can define pairs of brackets that should be indented,

    INDENT "(:)"
    INDENT "{:}"

or some regular expression. If some line of code fulfills this expression, next line will be indented:

    INDENT "\\s*(((if|while)\\s*\\(|else\\s*|else\\s+if\\s*\\(|for\\s*\\(.*\\))[^{;]*)"

Sometimes its hard to define some dynamic behavior in declarative language. Thats why I have added possibility to call Java methods from Schliemann code. Following example defines tooltip value for keywords of our language.

    TOOLTIP:keyword:org.foopackage.FooClass.fooMethod

    public class FooClass {
        public static String fooMethod (Cookie cookie) {
            String keywordName = cookie.getTokenSequence ().token ().text ().toString ();
            if (keywordName.equals ("for")) return "for: for (init, expression, update) defines for cycle";
            else
            ...
        }
    }

Real parser is needed for some more advanced features like Navigator (Structure View) or Code Folding. Thats why Schliemann engine allows you to define grammar of your language. Simple example:

    S = (Statement ";")*;
    Statement = ForStatement | WhileStatement | IfStatement | FunctionDeclaration;
    ForStatement = "for" "(" Expression ";" Expression ";" Expression ")" Block;
    Block = "{" (Statement ";")* "}";
    FunctionDeclaration = "function"  "(" Parameters ")" Block;
    ...

Schliemann engine generates parser for your language, and you can base some other features on its parse tree. Following example defines code folding for blocks defined in our language, and navigator content.

    FOLD:Block
    NAVIGATOR:FunctionDeclaration {
        display-name:"$identifier$";
        icon: "org.foo.FooIcon.gif";
    }

So, Schliemann engine creates code fold for every part of source corresponding to some "Block" nonterminal in parse tree. And it creates special node in Navigator View for each FunctionDeclaration.

This is short preview of some Schiemann Project features. This project is under development, planned for NetBeans 6.0. There are about 20 prototypes of languages based on this engine now. If you would like to try it, you should download development version of NetBeans (or build NetBeans trunk) and download "Languages Support" modules from development auto update.
See NetBeans Wiki for more info.

Posted at 05:38AM Dec 23, 2006 by Jan Jancura in NetBeans  |  Comments[5]

My first post.

This is my first blog post. I work on project Schliemann - Generic Languages Framework for NetBeans. Here I would like to post some news, screen shots and other thoughts about this project and may be some other things too. So, several lines about me first.

In past I was working on Xelfi IDE, and I am one of several NetBeans founders. I am author of NetBeans debugger (Debugger APIs, Java, JSP, Ant debuggers), new Options Dialog UI and architecture and I was working on many other parts of NetBeans too.

Posted at 02:53PM Dec 22, 2006 by Jan Jancura in NetBeans  |  Comments[0]