Duke Listens!: Visit my main blog at MusicMachinery.com

I was really looking forward to the GWT talk at JavaOne (and apparently so were many others, there seemed to be a thousand in the room during the talk).  The talk was presented by two of the core GWT team members.  They did a super job giving the talk - they were relaxed, were polished in their presentation, used humor, didn't get fazed by AV glitches.  They talked a bit about the philosophy behind GWT. They put some stakes in the ground - 'static typing  is good, dynamic typing is not so good', lots of sexy widgets not so important as user experience.   They want to bring software engineering to web development.  They claim (and I agree) that it is hard to engineer a pile of steaming javascript.  Java is much better for all its 'abilities' - maintability, adaptability, etc. 

They had some good metrics about how users react to load time.  If your app takes too long, your users will leave.  They have a 300ms goal for a load time of a rich GWT application - and they are working hard always to reduce load time.  They do this in a number of ways - they use the browser cache to help of course, they make the code they ship over the wire be very compressible, and they reduce the overall number of http requests (compare that to what is typically done with a traditional ajax app, where there may be many trips to the server just to get all of the javascript).  GWT build separage JS bundles for the different browsers, so if you are running in firefox, you don't have any IE specific JS.  One really neat optimization they are doing is called 'image bundles'.  Apps that have lots of static images can take a long time to load since each image is a separate HTTP request.  GWT can bundle all of these images into a single image. So your app only has to do a single HTTP request to get the images. They can then use windowing within this larger image to display the various images.  A very clever technique.  GWT also includes a timing framework so you can easily instrument your app so you can see how long it takes the app to load.

They showed some demos ... I was a bit disappointed here since they just showed the demos that are available on the GWT site. I was also disappointed that they didn't talk much about where they were going, what new features were on the roadmap.

During the Q/A one audience member asked about how to write a GWT app that could be crawled by search engines like google.  The GWT guys knew about this problem and they are indeed working hard on making GWT sites crawlable.

 All in all, a good session.

 

I arrived in SFO at around 1PM (after sitting 100 yards from the gate for a frustrating hour waiting for another plane to leave the gate). I checked into the hotel (the venerable Sir Francis Drake), and walked down to Moscone for the early registration and to see if anything was going on. There were lots of folks there ready to help. It took all of 2 minutes to register. I got my speaker badge, my nice J1 backpack (only a few goodies inside), and my lunch tickets. I was also able to register for CommunityOne - the combined NetBeans and Glassfish event.

I'm looking forward to a fun day tomorrow.  

For one of the Search Inside the Music demos that I'll be showing at the Sun Labs open house next week, I've been building up a database of music artists and related info.  I've been gathering this data  from many different places on the web using a crawler.  It's not a fast process - it takes about a month to build up enough data to make it useful.   My crawler collects the data and writes it out to a set of text files so that later it can be indexed with the our nifty search engine.

I tested the whole process using a small crawl of the web on my Linux laptop.  When I was happy that everything was working fine, I started the crawl running on one of  our large Solaris servers.  Unfortunately, there was a lurking bug ... a Java programming 101 kind of bug that would make the data collected from the month long crawl be wrong.

Music artist names very international.  There's Björk, there's José Feliciano there's Mötley Crüe and Motörhead,  (there's even a whole genre of music called umlaut metal).  My mistake was forgetting that when  writing a text file in Java (using  a PrintWriter for instance), the default encoding used is the encoding of the operating system.  Now for my Linux laptop, the default encoding is UTF-8 which can handle all of the umlauts and accents.  But for our Solaris server, the default encoding is plain, old ASCII.  With its 7 bits, ASCII can't represent any of the rich characters that are needed to represent all of the artist names.  When I indexed my 30 days of data and started looking at the results I was very sad to see 'bj?rk"  and "m?tley cr?e". 

With our open house demo just 5 days from now, there's no way for me to recrawl the data and save it to disk using the proper encoding.  Luckily, when I did the initial crawl, I resolved all of the artists to a MusicBrainz ID.  I'm able to turn this ID back into the canonical name for the artist, so I am able to patch the names without having to do a recrawl.  Whew!

So my lesson for the day is ... don't rely on the default encoding when reading and writing text. Now, back to getting the rest of the demo to work.
 

For an application that I'm writing, I wanted to be able to persist a large number of objects.  I wanted to avoid the complexities of a real database and the opaqueness and fragility of Java object serialization.  What I really wanted was the ability to turn any object into a a bit of XML that I could save and load at will, without having to do a lot of work defining the mapping between the Java object and the XML.  

There are a number of  standard Java APIs to go back and forth between Java and XML, but they all seem a bit complex for what I wanted to do.   Luckily, I stumbled upon XStream.  XStream is a simple API for doing exactly what I wanted:  serialize objects to XML and back again.  The API is quite simple to use.  To serialize an object to a file I use the code:

            XStream xstream = new XStream();
            BufferedWriter writer = new BufferedWriter(new FileWriter(file));
            xstream.toXML(myObject, writer);
            writer.close()

And to load the object I use:

                BufferedReader reader = new BufferedReader(new FileReader(file));
                myObject =  (MyObject) xstream.fromXML(reader);
                reader.close();

It couldn't be any easier than that. 

XStream is fast, simple, and easy to use.  Highly recommended.

 

This morning, I was fixing a bug using Netbeans.   Typical cycle: think, type, build, test.  However, the changes I made didn't seem to fix the bug.  I was nearly 100% sure that I'd identified the problem, but running the test was still showing the failure.  So another cycle of think, type, build and test.  Still, no dice.  Time to add some printlns - think, type, build, test. Woah! none of my debugging prints are showing up - what's going on? Dang!  The project I was working on was not my 'main' project in Netbeans. So when I hit build, it built the wrong project, so my changes were never being compiled. 

Netbeans should be smart enough to know that when you have multiple projects open and you are editing some source and you hit build you are likely to want to build the project related to the source that you just edited, and not the project that has no changes.  I'm probably bit by this at least once a week.  You'd think I'd learn my lesson, but I don't. Sigh.

Processing is a Java-based  programming language and environment for people who want to program images, animation, and sound. Stephan writes that a new book on Processing has been released by MIT Press called Processing A Programming Handbook for Visual Designers and Artists.  It will soon be joined by Processing: Creative Coding and Computational Art.

I've seen some really outstanding programs written in Processing, including Elias's Music Rainbow.
 

 

 

I writing a little web app using the nifty Google Web Toolkit (it is a really nice bit of work) when I received a very troubling warning from the GWT compiler  - I was using Java 1.5 language features (in particular, Generics) while the GWT only supports Java 1.4.  This was going to be a real big problem, since I was hooking the web app to the AST Search Engine which uses Java 1.5 features in its interface.  This could be a deal breaker.  But I soldiered on, thinking that if I confined my 1.5 code to the server side (where I was interacting with the search engine), but kept the client and any Java code that was to be shipped over a wire in 1.4 things might work - and sure enough it did. So if you are considering GWT, don't be put off by the Java 1.4 restrictions - once you are running on the server, you can use 1.5 to your heart's content.  

I saw in a recent post by Roumen that there is now a vi plug-in for Netbeans.  I've installed  it and have given it the once over.  It is really a quite faithful implementation of vi edit mode.   I can now use all of the finger macros that I've trained over the last 20+ years of typing into vi. So far, I'm really pleased with how it integrates with netbeans.  All of the netbeans goodness such as code coloring, completion, and formatting still work just fine with the vi plugin.   With thi jVi plugin, my code editing productivity just went up by 15%.    Thanks and Congrats to the jVi team!

Netbeans 5.5 has a whole bunch of new features designed to help build web services.  Roumen Stroble has put together a 30 minute flash demo showing how to use the netbeans platform to build a web app  that includes a web client, web service and database - it is really a great demo of how far the tools have come to help building these web apps.  I think the speed of development for a web service using netbeans now rivals that of Ruby on Rails or some of the other agile web frameworks. (Despite what Tim Bray says).  Check out the demo at the netbeans site here: Building Applications with Netbeans 5.5

Recently, I wrote a web 2.0 style mashup.  When I deployed, it worked great, except that some users would see browser crashes after a while.  Looking closer, it looked like there was a memory leak, my web app was using up browser memory and not releasing it.  Now, I'm not a bad coder, and it usually takes a pretty basic flaw to have a memory leak in a garbage-collected language like javascript.  Scanning my code, showed no obvious errors.  All my generated javascript objects were neatly discarded. I wasn't accidentally adding things to the DOM, where they could hang out for ever. I had no ever growing lists or tables that would keep references to objects to they wouldn't be GC'd.

But I did have a problem:  A circular reference.   IE has a very basic flaw in its garbage collector. It uses reference counting, and if there's a circular reference, the collector can't collect the objects.  If object A points to object B and B (directly or indirectly) points to A, A and B can never be GC'd. 

One of the Javascript idioms is to use function closures as callbacks. The closures give the callback access to the locally scoped variables where the function closure was declared.  However, this closure forms a circular reference, and without special care, will cause a memory leak on IE.  Every time a callback is added to an object (an image.onload for example), a circular reference is created.  If the object involved happens to be something large like an image (as it was in my case), the leak can become pretty obvious pretty quickly.  It is easy to break the circular reference (by removing the callback from the object when the callback is invoked), but of course, if you don't know about this problem, there's no reason to do it.

The Javascript closure idiom is pervasive in the web 2.0 world.  I wonder how many web apps out there have memory leaks when running on IE due to IEs flaw in its garbage collector.  There's a more detailed description of this problem on Jiberings page on Javascript closures.  (or you can look at the other million or so pages on this topic: google javascript ie memory leak closure.)

One of the really interesting features of Netbeans 6.0 is 'semantic coloring'.  This takes syntax coloring one level further, and colors/highlights code based upon higher level information.  For instance, here's what a deprecated call would look like:

   void myMethod() {
       callToDeprecatedMethod();
   }

There are descriptions of the various other semantic colorings in the Netbeans Java EditorUsersGuide.  Via Charles Ditzel's blog.

There's a nifty summary of all of the JavaOne talks, demos, BOFs for Sun Labs projects here:  Sun Labs, CTO, and Sun Labs Alumni @ JavaOne.  Project DarkStar, Spot, Project DReaM are all well represented, but the slot car thing looks to be the most fun of all. (Image courtesy of Roumen's Weblog):

Get your Game On with Project Darkstar is  a  new article on www.sun.com about Project Darkstar - Sun's new gaming framework designed to allow gamers to produce massively multiplayer online games that can scale to thousands of simultaneous players.  Darkstar allows game developers to focus on the building a fun game instead of having to figure out how to build a scalable infrastructure  ... that means a quicker time to market, and a more reliable and scaleable system.  Scalability is key for MMOs, if an MMO can't keep up with demand it is DOA.

The SDK is free and can be downloaded from games.dev.java.net.

Sun put the Sun Grid Compute Utility online this week.  Good news for all of us with big compute jobs.  Perfect for domains like music information retrieval where a single analysis pass on a large music collection could take years of CPU time.  Instead of waiting  years for the job to complete, the job can be put on the grid and completed in a weekend.  Good Stuff. 

I spent some time a few months ago figuring out what it would take to get our music analysis software to run on the grid.  The main question I needed to answer was "What do I need to do to my big hunk of java code to get it to run well on the Sun Grid so that I could take advantage of all of resources of the Grid"  The answer, I discovered, was that there were no Java APIs for writing software for the grid and getting my Java app to run on the grid was going to be a bit of work. 

The good news is that there's a group in Sun that is trying to fix this, to make sure that Java works and works well on the Sun Grid. This week these folks have issued an early access release of the 'Compute Server' (catchy name, isn't it?).  The goal for the compute server is to create an environment for Java programmers so that they can easily use the Sun Grid Compute utility.  The compute server provides an extremely simple API for a distributing computation across a large number of CPUs and gathering feedback and results from the computation.   The compute server is integrated with NetBeans, so deploying a job to the grid is part of the development work flow.  Here's a diagram that shows how the process works:


Remember, this is an early access release. The goal is to get feedback from developers.  You can download the early access compute server and look at the code (there's some good examples including a grid-enabled program that calculates the digits of PI in just a couple of hundred lines of code).

I've had the lucky opportunity to talk with the  developers building the compute server and have been quite impressed. These are  some really smart programmers with lots of  experience building distributed systems in Java.  I have high hopes for the compute server.

I've been writing software professionally  for about 25 years, which makes me a geriatric programmer.  For 20 of those 25 years I've used vi or vim and make or ant as my main development tools, and I've been very happy with my productivity. Vim coupled with ctags and a bit of configuration gives you syntax coloring, in-the-editor compiling and quick code browsing.  It is a very efficient environment. 

Every few years a new development environment comes along. I usually kick the tires and give the new tool an honest try.  The first IDE I tried was TurboPascal which was fast and cheap, but it was Pascal ... and well,  it just didn't take.  I also remember Visual Cafe (my first Java ide), Eclipse and Netbeans 1 thru 4. 

For the last couple of months now, I've been using Netbeans 5.0, and I think I've finally been convinced to retire the trusty old VIM as my development environment. NB 5.0 does a whole lot of things just right to make coding much easier:

• CVS integration  - everything can be done in the IDE, it's the most intuitive source code control integration I've seen
• Matisse  - It is ultra slick - it is so easy now to create a quick gui panel - and the way the generated gui is represented in the code (using code folds to hide the code you shouldn't touch) is very nice.
• Refactoring tools - things that should be easy (like renaming a package) now are, no more fighting cvs and  manually grepping through hundreds of imports.
• Autocomplete - 80% of my trips to the JDK javadocs have been eliminated
• Project dependencies - you can manage multiple projects and their dependencies in the ide, makes building much easier.
• Ant at the core - uses ant scripts for building so you can build your software without the IDE
• Stable - I've never lost a bit of work with nb5
  

Still there are a few things that could be better:

• Startup time ...  vi/vim is much faster
• Memory usage - after a day of edit / compile /run cycles I tend to need to restart netbeans, its heap tends to get large after a while.
• The editor is not VI. I don't like having to reach for a mouse to do things.
  

After working with Netbeans 5.0 for a few months, I've decided it's a keeper.  So Long VI and thanks for all the :wq