Monday Aug 11, 2008
Software embracing a microkernel architecture has been happening for the last 20 years, yet the great aspects of this particular architecture are too often undermined and not considered when working on a new project. There are various reasons for this, history behind it with poor experiences, but there has also been a lot of lessons learned . So what can one benefit from a microkernel architecture?
Microkernel introduction
In the last two decades, a lot of changes happened in the world of software and Internet has revolutionized quite a few trends. Microkernel architecture used to be adopted for operating system type of implementations pretty much exclusively. I remember about ChorusOS, a Unix variant and first-generation microkernel OS developed in the early 90's which was targeted at high-availability applications in the telecommunications space. Chorus was truly innovative back then and like Mach, another first generation microkernel OS implementation, suffered performance issues due to message processing between the kernel and user spaces (no sharing of address spaces). Minix, another well-known unix-like OS based on a microkernel architecture was created by Andrew S. Tanenbaum in 1987 and influenced the development of Linux although Linus Torvalds went with a monolithic kernel architecture instead.
There has been quite a few debates regarding the benefits and tradeoffs of a microkernel architecture versus a monolithic one. Most Unix operating systems are based on the last one, with no messaging between parts of the system and an easier (and controlled) access between the kernel and user spaces. A microkernel-based implementation typically grants the software with added modularity and flexibility due to the abstraction and extra independency (clear boundaries) between the various modules / parts of a system.
I stated earlier that trends had changed in the last 20 years and although it may sound like stating the obvious, what I really meant to say is that software architectures do evolve due to other factors such as hardware, markets and new requirements. Today one can see a lot more software implementations based on a microkernel architecture, due to an increase demand and new requirements motivating that choice. Hardware (e.g. memory, CPU) has gotten a lot faster and cheaper, there is a lot of hardware environments to port to, the internet and its fast moving pace has increased the demand for more dynamic and flexible software.
Multi-threaded server applications are a lot more popular these days and the overhead of a microkernel-based implementation is no longer an issue when it comes to interactions between various modules involved in a particular processing operation. It used to be true for the most part, when IPC mechanisms had to be used to relay messages between modules running as part of different processes.
Why Microkernel
A microkernel architecture could be summarized to a "highly modular collection of application-neutral abstractions upon which can be built robust, flexible and dynamic applications" and brings the following and non-exclusive benefits to software implementations:
- Modularity
- Flexibility
- Extensibility
- Better footprint management
- Easier development, unit testing, maintenance and portability
The modularity, flexibility and extensibility aspects are what I would consider the most important ones of a microkernel-based implementation. A highly modular architecture allows protocols to be separated from their implementations as well as maintaining a level of independence between the various modules and components / services (enforcement of boundaries) of the system. Flexibility grants you with dynamic loading and unloading of modules and extensibility gives the developers means to extend the current system with new protocol or schemes implementations.
Footprint management can be an important issue in certain environments and the ability to remove unnecessary components as part of your application can help reducing the footprint significantly.
Modules can be developed and unit tested independently of each others and portability is usually easier to handle as non portable logic can be placed at the kernel level directly and not mixed with non-kernel components.
As far as potential performance improvements, I think this is very debatable unless one is dealing with a single-threaded monolithic implementation and in this case it would be slower on SMP machines obviously. I'm a big fan of multi-threaded implementations versus multi-process ones and in this case both architectures would run better on multi-core systems.
Microkernel-based architecture has also reached the world of database systems and most recently, Drizzle, a fork of the MySQL core engine has been announced and is supposedly aimed at Web and Cloud applications. This confirms the need for more modular and flexible implementations at the database management system level.
Microkernels and the likes in Java land
The Java platform and more specifically the language, facilitates the development of microkernel-based implementations. The object-oriented aspect of Java allows abstractions via published interfaces to be defined for the various parts to be implemented. In other words, this enables a clear separation between the protocol various implementations. Java dynamic class loading also allows modules to be loaded on the fly as needed.
To name a few Java applications and frameworks which have adopted a microkernel-based architecture:
- GlassFish V3 has adopted HK2, the Hundred Kilobytes Kernel, with a module subsystem and component model
on top of OSGI
https://hk2.dev.java.net/
For an example of a Spring DM application leveraging OSGI dynamic module system and separation of application logic into modules, I recommend reading this article.
Apache Derby which can also run as an OSGI bundle (since 2005) is used in this example as well.
There is also growing interest in the cloud computing arena with having a more modular, flexible and extensible Apache Hadoop. Being able to plug-in a query language of choice on top of Hadoop's huge datasets is a very attractive value-proposition indeed.
Apache Derby Case Study: How to benefit from a Microkernel architecture
Apache Derby / Java DB has been supporting a highly modular architecture since its inception in 1997 with its first release as part of Cloudscape. Derby's system is composed of a monitor, acting as a kernel of its own, booting and shutting down services as well as discovering appropriate modules to load for a particular service. As an example, different modules can be loaded during boot-up based on the environment such as the JVM version. Modules themselves can also require other modules to be loaded in the environment.
Derby already supports pluggable authentication schemes as well as pluggable raw storages via its module-based subsystem. It promotes a clean separation between the SQL and Store layers which favors the implementation and pluggability of different types of storages. For instance, one could envision a GRAPH storage for Derby to enable a graph-based data model to be accessed via Derby's relational engine. There is already some implementation of an in-memory storage and even a store-less storage type to test Derby's layers separation and submit queries against non-database objects.
The extra benefit of having a clear separation between a protocol and its implementation allows other projects to re-use parts of the system and contribute back to the open source community. For instance, several project have expressed their interest in integrating Derby's raw B-Tree storage as it has proven to be robust over 10 years and has an active open source community.
There are already on-going efforts and proposals to make Derby more flexible and reduce dependencies between modules of the store layer and have a cleaner pluggable storage implementation. When this modular architecture got implemented 10 years ago, there was little demand for specialized and pluggable storage. This has changed with new types of data models and requirements for web applications and cloud computing, translating into innovations at the data management level.
The good news for Derby is that it has already gotten the proper foundations in place to go the extra mile and have a clean microkernel architecture. This would not only benefit the store layer but also other parts of the engine, such as the SQL compiler (parsing and execution plan generation) and the access layer, residing above the store. For instance, one could imagine having Derby's SQL layer plugged-in on top of Hadoop, as an extension to be able to run SQL queries against MapReduce datasets.
In Conclusion
Software implementations which are supporting a microkernel architecture inherit highly modular and flexible characteristics. The performance gap between a monolithic and microkernel architecture is no longer much of an issue, especially in a multi-threaded environment where messaging is done via local interfaces without requiring the use of IPC mechanisms.
The dynamic nature of web applications and requirements to constantly adapt to new standards and protocols will continue to promote this architecture. OSGI has already showed some path and additional web frameworks have become compliant.
Derby with its highly-modular architecture excels in an embedded server mode configuration and is a natural database storage of choice for the Java platform and microkernel-based implementations in the middle-tier.
Thursday Jul 31, 2008
As Alexis Moussine-Pouchkine posted on this blog, the French railways company, SNCF uses GlassFish and Java DB (based on Apache Derby) to power a monitoring application which produces alerts with a real-time graphical representation of key system variables as well as PDF reports.
Don't miss this questionnaire with Franck Leprêtre, Software Architect at SNCF.
Java DB version 10.2.2.0 is bundled in Glassfish V2.
Monday Apr 28, 2008
Java DB & Apache Derby 10.4.1 GA have been released.
10.4.1 introduces some significant features:
Asynchronous replication with manual failover allows you to maintain an up to date copy of your (master) database on a different host (the slave). In the case of a crash on the master database you can perform failover to the copy (slave database) and continue serving client requests against your database. The section titled "Replicating databases" in the Server and Administration Guide has more information.
Table functions let you slice, dice, and zipper together external data. Using table functions and standard SQL, you can pose sophisticated queries against in-memory collections, flat files, web resources, non-relational databases, etc. Table functions also let you efficiently import data from web feeds, sensor logs, and other relational databases. For more information, please see the section titled "Programming Derby-style table functions" in the Developer's Guide.
Java Management Extensions (JMX) for Java DB, allowing local and remote monitoring and management of running Java DB instances (embedded or Network Server). You can use graphical tools such as JConsole or VisualVM to access Java DB's MBeans. You can also write your own Java code to do the same, using standard JMX APIs.
Java DB 10.4.1 also brings performance boosts (better scalability on multi-core machines, network client caches), additional SQL capabilities (unique constraints on nullable columns, limit/offset) and usability improvements (bracketed comments, ij continuation markers).
For more details, please see the New Features and Release Notes references at the Java DB portal.
Thursday Jan 24, 2008
As you probably have heard already, Sun Microsystems has announced an agreement to acquire MySQL.
I have had several people asking me about my view on it and I thought I would offer it here, along with an analogy.
First of all, being part of the Database Technology Group at Sun, I'm very happy that Sun has entered into an agreement to acquire MySQL, the #1 open source relational database system.
Sun is one of the biggest open source contributors and it only made sense that eventually Sun would acquire MySQL to offer its users base various alternatives for their database needs.
Sun will now support 3 major database open source initiatives as MySQL, PostgreSQL and Apache Derby (Java DB).
As Jonathan Schwartz mentions it here in his blog, "This transaction increases our investment in open source, and in open source databases. And increases our commitment to Postgres - and the database industry broadly. The same goes for our work with Apache Derby, and our Java DB."
As far as the analogy with The Three Musketeers, I see these three databases as the database musketeers which are going to be working together, with each of them having very particular and unique capabilities that will satisfy various needs in the entreprise, such as embeddable Java store, data wharehousing, OLTP, Web 2.0, etc. Now I will let you guess who plays the role of D'Artagan and no, don't even go there with Richelieu 
In reality, databases are not one size fits all and it is good to have choices and alternatives for the various applications' requirements out there. So the database musketeers's motto here won't be "One for all, all for one", but rather "No one size fits all", otherwise one could end-up with something like this (joke aside).
Anyhow, here at Sun we're very excited in having MySQL being part of the adventure.
Monday Dec 03, 2007
JavaPolis 2007 is starting on december 10th in Antwerp, Belgium and as expected it is already sold-out.
If you don't happen to know, JavaPolis is one of, if not, the biggest Java conference in Europe. There will be 3,200 attendees this year and some great technical content.
I'm going to be presenting the following sessions on Apache Derby / Java DB:
Throughout these sessions, the audience will get to see how Java DB can be integrated in the various web tiers and throughout various scenarios. I will cover how it can be used to empower web, standalone and server applications, as well as how it differentiates itself from traditional
RDBMS, due to its unique architecture and capabilities for various types of usage.
Kristian Waagan and Jørgen Løland from the Sun Java DB engineering team will be discussing about what's coming up in the next release of Apache Derby - Some very cool stuffs indeed.
Feel free to stop by our booth to ask questions and share your experiences; who knows, you might end-up getting some goodies, it is nearly Christmas after all
See you in Antwerp!
Friday Nov 30, 2007
As you may already know, Apache Derby supports the JDBC API defined for the Connected Device Configuration Foundation Profile, also known as JSR169. The features supported are a subset of the JDBC 3.0 specification. Support for JSR169 is limited to the embedded Derby driver.
John Embretsen has put a great page together with hints and resources about Apache Derby's support for Java ME (also known as J2ME) environments. This also applies to Java DB, which is Sun's supported version of Apache Derby.
If you need to know into which JDBC environment you're running as part of your application, you can use this tip to dertermine for instance if you're running within some JSR169 context.
Thursday Nov 15, 2007
In this podcast, Masood Mortazavi discusses about the advantages of developing and deploying apps with Java DB.
Thursday Nov 15, 2007
Jean-Francois Arcand and myself gave a presentation at ApacheCon US 2007 on how to develop real-time rich internet applications using Comet and Apache Derby / Java DB.
You can download the presentation slides here.
The chat application demo we showed, highlighted how Apache Derby can be used as a web client store service from JavaScript to rapidly handle and persist incoming messages sent via Comet to a web client.
With little coding, the web client store service can persist data locally and asynchronously, allowing web applications to perform searches for online and offline chatting and viewing.
The demo was run with the new (early access) Java SE 6 Update N (formerly known as Consumer JRE). I definitely observed improved JRE cold start-up time and the web client store service extension gets installed in a much more convivial way than ever.
It is good to see Java deployment and runtime being improved to faciliate the delivery and usage of Java-based modules onto a web client. I can envision new (Java) services being made available for people to integrate within their web applications.
Tuesday Oct 16, 2007
In this blog, Byron Nevins shows how to enable Java DB to start as a Windows service using GlassFish's appservService facility. You can also find more information about Java DB bundling in GlassFish here.
Wednesday Sep 26, 2007
+ 
+ 
Arun Gupta has posted some information about using Java DB / Apache Derby as part of a JRuby on Rails application on GlassFish. Java DB is bundled with GlassFish and can be used as a multi-user RDBMS, running embedded in the same JVM as GlassFish or as a standalone database server in its own host and JVM.
Monday Aug 13, 2007
The Apache Derby project is pleased to announce a new feature release
of Derby, 10.3.1.4.
Notes describing the difference between Derby release 10.3.1.4 and the preceding release 10.2.2.0 are available here.
Monday Jul 09, 2007
The Apache Derby community has delivered a release candidate of Derby (10.3) which can now be downloaded.
A non exhaustive list of new features in this release:
- DBA Powers to control who can shutdown, encrypt and upgrade databases.
- Secure Server which makes the Derby Network Server secure by default.
- Language Based Ordering which adds built in language based ordering (collation) and like processing to Derby.
- SSL/TLS to allow secure SSL/TLS communication between a client and a Derby server.
- Various performance improvements in query processing along with reduced CPU usage in Derby embedded core engine.
- Alter Table DROP or RENAME of a column. This means that most dynamic schema modifications are now possible in Derby.
More details at http://wiki.apache.org/db-derby/DerbyTenThreeRelease
NOTE that this is a release candidate, not a GA (General Availability) one, which should be available soon after public testing has completed and reported issues have been addressed.
Monday Jun 25, 2007
Apache Derby's WIKI is full of wonders when it comes to resources. I reckon it is not obvious to navigate the WIKI and find relevant information but if you're interested and curious about some Apache Derby & Java DB hints and tips, then point your browser to this link and bookmark it
Tuesday May 22, 2007
The Java DB (based on Apache Derby) support offering page is now online. Note that you can also consult the Apache Derby derby-user mailing list using Nabble for getting answers or even post your questions to this last one. Apache Derby has a very active community and getting feedback via the mailing lists is a way to participate. In fact, by asking questions on the list, one is in a way adding persistent content that someone might find useful and get to via Nabble at some point
Tuesday May 08, 2007
I heard the news from JRuby core developer Olivier Nutter at CommunityOne as I was setting up for my presentation with Kevin Henrikson from Zimbra.
Yes, so ThoughtWorks Studios will have their collaborative development project management solution named Mingle be the first commercially-distributed Rails application to run on JRuby.
Mingle 1.0 which should be released in June 07', will be running with a Jetty web front-end and an Apache Derby database back-end.
InfoQ blog on the subject reports that performance has not been a major concern. The current runtime easily supports hundreds of concurrent users with a simple cluster, and JRuby performance has been getting faster all the time. According to the team, experimental deployments have included 10 JRuby interpreters running in a single JVM process.
Friday May 04, 2007
As David Van Couvering pointed out in his blog, I will be co-presenting with Kevin Henrikson from Zimbra at CommunityOne and JavaOne 2007 next week. Kevin will especially highlight how they have implemented offline support in their current Zimbra Desktop offering.
We will discuss how today's and tomorrow's Rich Internet Applications (RIA's) can be enabled to run offline and allow the users to keep using the same application, while not being connected to the net.
You can get a sneak peak of what we will talking about based on David Berlind's recent article on the subject of enabling offline Rich Internet Applications using some local storage such as Java DB / Apache Derby and some Local Ajax (LAJAX) or Asynchronous JavaScript technic based on LiveConnect.
Thursday Apr 26, 2007
Here you can find the list of new features and issues to be resolved for the coming Apache Derby 10.3 which is targeted to be released around end of June 07'.
Some exciting additions around security and performance improvements:
- DBA Powers
- SSL/TLS support in Client/Server mode
- Language Based Ordering
- Reduction of CPU usage and improved concurrency in Derby (embedded) core.
- Long awaited ALTER TABLE DROP or RENAME column (great for dynamic schema modifications)
You can find more details here as well as the list of targeted issues to be addressed.
As usual, you can check the latest Alpha documentation corresponding to the Derby Alpha codeline (Trunk).
Thursday Apr 26, 2007
For those of us working on open source software, this article shows an interesting view of what open source hardware is all about.
You can also check OpenSPARC which is Sun's open source hardware initiative, aimed at spawning dramatic innovations around chip design.
All I have to say is that I'm glad to be in Software
/Salute
Wednesday Apr 25, 2007
Come learn what's cool and what's new in Apache Derby, the database for Java applications, from Derby committers and community members. This is a great opportunity to meet with the Apache Derby community and Java DB folks. Of course, there will be Beer, wine, soft drinks, and very nice snacks!
*Tuesday evening, 8 May from 7:00 to 10:00+ pm **
@ Jillian's in the Sony Metreon, right next to the Moscone Center
101 4th Street, Suite 170, San Francisco*
Monday Apr 23, 2007
Simon Phipps has posted a neat picture of Sun's booth at FISL 8.0 (Porto Alegre, Brazil) where I just came back from last week.
As usual and once again, the conference has been terrific - Lots of attendees (5,363) and it is all about free and open source software. It was incredible (even after my second time there) to experience the level of interest from the attendees - Very nice folks and this conference keeps you busy for sure until dusk .
Also, for a "meaterian" like me , it was absolutely awesome to experience the intense BBQ cuisine from Brazil (churrasco)and this area.
I gave a presentation on particular technics on how to enable offline web applications (RIA) using Java DB - I will give a more elaborated talk at JavaOne 2007 where I'm co-presenting with Zimbra on their next-generation suite of collaborative web applications that can operate homogeneously online and offline (Zimbra Desktop), using Apache Derby.
Many thanks to Rafael Vanoni for hosting my talk and speaking Portuguese better than I do
Obrigado a Todos!
Thursday Apr 19, 2007
It has been announced today, Ubuntu Feisty Fawn (7.04) has been released and it includes Sun's complete Java stack.
This is the first time that an entire, production quality Java stack with tooling is being included in an open source GNU/Linux distribution - greatly simplifying the process for Linux developers and users to access Java technology.
The complete and fully supported components of Sun's Java stack which are packaged for Ubuntu are Java SE 6 JDK, GlassFish v1, NetBeans IDE 5.5 and Java DB 10.2 (based on Apache Derby). (FAQ)
You can also check Tom Marble's blog entry for additional information. This is very exciting indeed.
You can 'apt-get' install Java DB as an optional Ubuntu package such as:
# apt-get install sun-java6-javadb
and it should get installed under '/usr/lib/jvm/java-6-sun/db/'
Monday Apr 02, 2007
Try "Orsini Corsica" and click on "Mi sentu furtunatu"
Friday Mar 16, 2007
Jagadish Ramu has written a demo and tutorial article to show how GlassFish and Sequoia can be used together to provide database clustering capability with Apache Derby / Java DB.
Sequoia is an open source database clustering middleware that allows any Java application to transparently
access a cluster of databases through JDBC. No code change is required to enable an application to have failover, replication capabilities.
This demo will have the modified sample application from Netbeans, deployed in GlassFish,
interacting with Sequoia controller for database operations with Apache Derby. The only change needed in the sample is to
configure the properties of jdbc-connection-pool to interact with the Sequoia controller.
|
|
|
|
An improved version of JBoss MicroKernel can be fo...
GlassFish also uses OSGi these days. We use HK2 co...
Nice references. Very good discussion.
<a href=