• The zipped JavaDoc API
  
• The JAR of the reference Java 5 implementation
  
• The zipped NetBeans package containing the sources of ODFDOM v0.6.

tags: api architecture code development netbeans odf opendocument software specification sun xml

ODFDOM is the name of the upcoming free OpenDocument framework sponsored by Sun Microsystems Inc.

It will be the next evolutionary step after AODL and Odf4j. Designed together with their architects with the intent to provide an easy lightwork programming API for the ODF developer community. ODFDOM is meant to be portable to any object-oriented language.

The first pre-version of the Java 5 reference implementation of ODFDOM is planned to become available under LGPL3 in May 2008.

Please find further detailed information in the OOo Wiki.

 

tags: api architecture code development netbeans odf opendocument software specification sun xml

The Problem

Currently an extension can be installed on a system where it does not work. The extension then typically contains libraries which are not suitable for that platform. After installing it, the extension will most certainly not work. Unfortunately the Extension Manager does not warn the user about it. 

An extension can, however, provide native libraries for many platforms. In that case the Extension Manager should install the suitable libraries and silently ignore all others.

The problem is, that an extension can declare for which platform a library is intended but it cannot declare what the target platforms of the whole extension are. People may inadvertently install extensions not suitable for their systems and will not be warned about it or even prevented from doing so.

A similar problem exists for extensions which support a locale different to that used by OOo. For example, an extension could show dialogs using a different language as the one of OOo. User might want to be informed about that when installing the extension.
 

Current handling of different platforms

If one develops an extension for different platforms, having one extension for each platform rather then one extension for all platforms, then these extensions must have different identifiers although they are logically the same. For example extension A and B provide exactly the same functionality but A works only on Windows and B works only on Linux x86. Their extension identifiers must be different.

By providing different identifiers we have somehow attached additional platform information to the identifier. One extension identifier represents exactly one or a set of target platforms. This is a requirement for the online update. Later versions of an extension have the same identifier and the Extension Manager assumes that an update supports also the same platforms as the currently installed version. That is, by using the extension identifier plus the version the Extension Manager can exactly identify the extension.

Although the identifier carries platform information, it cannot be used by the Extension Manager to warn users in case they are going to install an extension on the wrong system. The reason is, that it is not defined how platform information are encoded into the identifier.

First Installation

Extensions are currently installed, no matter what platforms or locales they support. Ideally one would only allow the installation of working extensions. What “working” means is certainly open to debate. As for the platform, it is safe to say, that only an extension can work which supports the platform of the system where it is deployed. But with locales the situation is not quite clear. A very obvious manifestation of an locale is the text used in dialogs. So let us use dialogs as an example. Assuming the extension does not support the locale of the office, then users may see a dialog in a language which they do not understand. This is actually the worst case and for most users this would be acceptable. We need to remember all the possible localizations of OOo. It is certain that extension would rarely support all locales. Therefore there will always be users who will not get localized extensions but they may be happy that there is an extension available at all.

In case an extension does not support the locale of the office, the user should be informed but the installation should be allowed.

What happens when the user switches the office's locale after installation of the extension? To prevent the user from surprises, the Extension Manager could also take all non-active locales of the office into account, when comparing the locales of the extension with those of the office.

Regarding the locales, we need to overcome another difficulty. Locales are named by a string having a language, country and variant parts, for example 'de-DE-hamburg'. There can even be multiple variant parts. When there is no exact match, then one need to find a “near match”. Assuming that the office uses a German locale ('de') and the extension supports only 'de-DE-hamburg' and 'en' then one would probably not want to warn the users, because for most of them 'de-DE-hamburg is just as well and any warning may be confusing.

Adding Support for new Platforms and Locales

A new version of an extension could bring support for additional platforms. This may be preferable over providing separate extensions for each platform. This way one could gradually evolve the extension to support more platforms over the next versions. Let's assume version 1.0 is Windows only and installed on a respective system. Let us further assume that later versions can be obtained by the online update feature of the Extension Manager. Then version 2.0 is published which additionally supports Linux x86. When running the update, version 2 will replace version 1 on the Window system.

In case users are not satisfied with version 2, then they could decide to downgrade to version 1. This cannot be done automatically but the user can simply add the previous version. Because version 1 has the same identifier as version 2, the Extension Manager will replace version 2. Now think of a Linux user, who had installed version 2 from the beginning and for some reason wants to replace it by version 1. Doing so will install a non-functioning extension – remember that version 1 is Windows-only.

In this case it would be desirable if the user is warned or even be prevented from installing the inappropriate extension.

There may, however, be the case where support for a platform will be dropped in a future version. Then users need to be notified and possibly be prevented from installing it. This would be a candidate for the “Show all” view in the update dialog for extensions. That is, the update would be displayed as 'available but not allowed'. In the case of a manual installation, a message box would be displayed.

We have a similar situation when adding / removing locales in future versions. Provided that both, the installed extension and the new version of the extension, do not support the office's locale the user probably does not need to be warned. However, when the installed extension supports the office's locale but the new version does not, then the user needs to be warned. This warning needs to be clearly expressed in the update dialog or in a message box in case of a manual update.

The warning needs also be displayed when installing a previous version.

However, we could consider the loss of a locale as acceptable as long as there is another partly matching locale. Then the warning need not be displayed. For example, there is an office with a locale 'en-GB' and a installed extension which supports 'en-NZ' (New Zealand). If this extension is going to be replaced by a version which only supports 'en-US, 'en' or 'en-GB-south', then a warning is probably unnecessary.

Merging Platforms and Locales

Another problem arises when a future version should unify different platforms. For example, there are two instances of version 1, one for Windows and one for Linux. Both have different extension identifiers. Now the developer decides to only have one extension as version 2, which supports both platforms.

By doing so, the developer would exclude one of the previous versions from online updates. The new version of the extension must have the same extension identifier as the earlier Windows or Linux version. By using the identifier of the Windows extension, one would exclude the previous Linux version from the online update and vice versa.

To get around this situation one could invent an “alternative extension identifier” whose sole purpose is to offer a migration path. The new version would then use either the extension identifier of the previous Windows or Linux version and the identifier which it not uses would be the “ alternative extension identifier”. Therefore the new version of the extension would carry both identifiers. Now it would only be necessary to adapt the Extension Manager and the extension repository to allow updating all previous versions.

By using a combination of extension identifier and platform, one could provide different instances of an extension, which have different target platforms but still the same extension identifier. “Merging” the different instances in a later version would then be no problem, provided that the Extension Manager takes the different platforms into account.

The “alternative extension identifier” would therefore only be used to allow the migration of “legacy extensions”.

In a downgrade scenario we would need to prevent the installation of an extension which is inappropriate for the user's system. This is not possible for those “legacy extensions” because they do not contain yet information about their supported platforms.

Similar scenarios exist for using extensions with different locales. There could be two instances of the same logical extension which supported different locales. To ensure, that the proper version is used during the online update, one needs to provide different extension identifiers. This would have the same implications as those for using different platforms.

Providing different identifiers, however, allows that two instances can be installed in parallel. This is an “unwanted” situation as we will see later.

It's not only the Platform

Similar to the platforms and language one could think of other “features”, which are somewhat related. Currently there does not seem to be other candidates, but we cannot discard the possibility of new future requirements.

The previous paragraphs showed some possible scenarios. One could of course think of combinations. For example, a new version could support additional platforms but drop some locales.

The Combined Identifier

As we have seen, it is necessary that extensions provide additional information in order to warn a user or to prevent the installation of inappropriate extensions. To date we have identified platform and locale as such information but there may be others. The combined identifier would therefore be composed of the extension identifier, platform and locale.

The meaning of the extension identifier (we could perhaps rename it) would be changed from:

“... uniquely identifying and extension”

to: “a unique name for a logical extension.”

That is, there can be several instances of one logical extensions (that is, different files), which have the same extension identifier.

There can be one to many platforms and locales. To facilitate matters, there could be an 'ALL' locale and platform.

Using Combined Identifiers when “Updating”

Currently the extension identifier is used for uniquely identifying an extension. This is important for two scenarios:

• Manual installation. The Extension Manager must determine if an extension is already installed. If so, the extension which is being installed will replace the installed extension.

• Online update. The Extension Manager needs to find exactly one version which represents an appropriate update.

When there are two extensions with the same extension identifier, then they can replace each other. For example, the extensions A and B have the same extension identifier and A is already installed. When the user is installing B, then the Extension Manager compares the identifiers. And when the identifiers are the same, then B will replace A. Depending on the versions, the users will be displayed a warning informing them that they are going to replace the same version, an earlier version or a later version.

The combined identifier would serve the same purpose. But then an extension need not have the same combined identifier but it would be  sufficient to have a “similar” identifier to be a legitimate replacement to an already installed extension. The reason for this is to be as flexible as possible, with respect to adding, removing or merging platforms / locales in later versions.

Before we go on explaining what “similar” exactly means, let us be clear about what would happen, when two extensions are installed which have the same UNO services. UNO services are identified by their service names. If there are two extensions containing the same services and one is already installed, then installing the second extension will overwrite the registration entries of the services of the first extension. When code instantiates a service, then always the services from the last installed extension are used. This could only be prevented with providing different service names. But then, the extension identifiers should be different as well, because the extensions contain different services. Installing the second extension would make the first one “not working”. Therefore we would only allow to install one instance of an extension.

Regarding the combined identifier, the Extension Manager needs to compare the extension identifier, the platform and the locale. Now let's see how they must match in order to determine if one extension can replace another.

Extension Identifier

Because the extension identifier represents the 'logical extension' they must be the same for both extensions.

Platform

Regarding the platforms, there are three important points to consider. First, an extension must support the platform where it is to be installed, otherwise it cannot be installed. Second, an extension may have several target platforms. Third, there cannot be two extension installed with the same extension identifier (remember the UNO services). Taken these into account, then an extension can only replace another extension when they both support the platform where the first one is installed. When this is the case then we disregard all other target platforms. Here are some examples of two extensions, A and B, where B can replace A. We assume that A is installed on Windows and both have the same extension identifier and the same locale.

Example 1:

A supports Windows.
B supports Window, Sparc

Example 2:

A supports Windows, Linux
B supports Windows

Example 3:

A supports all platforms
B supports Windows

Example 4:

A supports Windows
B supports all platforms

In case the extension, which replaces the installed extension, has a different set of target platforms, one could inform the user about it. And of course, the user would be notified if the extension cannot be installed at all.

Locale

Earlier on we said that it would be useful to allow the installation of an extension even if it does not support the office's locale. Since we also want to be flexible and allow adding/removing and merging of locales in later version, we would allow every set of locale to replace the “installed locale”. For example, extension A is installed and is to be replaced by B. A and B have the same extensions identifier and support both the same platforms. Then it would not matter what locales B supported.

In case of a manual installation the user would choose exactly one extension to be installed. In case of the online update there could, however, be a variety of possible updates. For example, extension A is installed and there are three possible updates, B1, B2, B3. All extensions have the same extension identifier and target platforms. B1, B2, B3 have also the same version.

Example 1:

office: en-US
A: en-US

B1: en
B2: en-US
B3: en-GB

In this case B2 would be the perfect match.

Example 2:

office: de
A: en

B1: en
B2: en-US
B3: de

Here B3 would be preferred, although the locale does not match the locale of A.

What complicates matters is that OOo can support various locales while one is active at a time. For example, the office currently uses 'en' and also supports 'de'. The extensions have these locales:

B1: en
B2: en, de

Actually B2 would cover more locales of the office and would therefore be a better replacement. Now consider this case:

Office: en (active), de, fr, en-NZ

B1: de, fr, en-NZ
B2: en

Although B1 matches a lot more of the office's locales than B2, B2 would be preferred because it matches exactly the currently used locale. Selecting B2s would probably be in the user's interest.

There can also be the case that there is no perfect match but only near matches. For example:

Office: en-US (active), de-DE, fr-FR

B1: en-GB, de-AT, fr

B2: en-NZ, de, fr

What would be the preferred update?

Regarding theses scenarios and taking into account, that users hardly ever switch the locale of the office, I suggest to choose the update according to which extension matches best the active locale of OOo. To avoid confusion, developers should either provide only one extension containing all locales or provide 'suitable' sets of locales. For example:

A: en, en-US, en-GB, de, de-DE, de-DE-hamburg

B: es, es-GT, es-MX,

That is, one should not split one language:

A: en, es-GT, de, de-DE, es-MX

B:es, en-US, en-GB, de-DE-hamburg

Migration

Today's extensions should also work in the future. Because they do not yet provide platform and locale information, we would assume that they are suitable for all platforms and support all locales.

To allow the merging of two extensions with different identifiers, we should introduce the 'alternative extension identifier'.

Declaring Platforms and Locales in description.xml 

Platform

A platform specifier should exactly determine the platform related requirements of the components contained in an extension. This includes

• operating system

• used compiler

• processor

Operating systems could be general names like Windows, Linux, Solaris or particular product names, such as Solaris 10, Windows Vista 32, Windos Vista 64.

The compiler part indicates the application binary interface (ABI) as well as other library dependencies, such as C and C++ runtimes, stl library, etc.. The use of a particular compiler may also require a particular bridge in OOo. The name of the compiler could also indicate if 32 or 64 bit code was produced.

The processor types could also be provided as general names, such as Sparc, x86, or particular type names, such as 'Pentium 4'.

The different parts of the platform may partly carry redundant information. For example, a compiler may only produce code for a particular processor and operating system. Some compilers can, however, produce code for different processors. This is similar with operating systems. For example, Windows only runs on x86 processors but Solaris runs on Sparc processors as well.

Operating systems, compilers, processors are developed constantly. Because one cannot see into the future, one can only describe a working constellation at the time of development. If an extension works on a particular platform is only "known to the extension itself". However, the office can make some rough guesses if an extension works. For example, OOo would deduce that an extension cannot work if

• the compiler requires a C++ runtime which is not provided anymore in OOo

• the extension requires Linux but the office runs on Solaris

• the code of the extension is compiled for x86 but the office runs on Sparc

On the other hand, OOo would not know about compatibility problems caused by

• some subtle compiler dependencies (whatever that is)

• different versions within a product line of operating systems

• different versions within a product line of processors.

Only the developer can find out about the latter problems. This, however, requires a huge effort, because the extension needs to be tested, whenever the platform slightly changes, for example, a new version of an operating system becomes available. The list of working platforms needed to be adjusted and new versions of the extension needed to be provided to the users.

So we can have two types of compatibility checks, where each one has its pros and cons:

1. Let OOo guess. An extension declares at a a very basic level the platform requirements, for example (VisualStudio 2008, Windows, x86). OOo recognizes incompatibilities as mentioned earlier. The extension does not need to be updated, because of some platform change. However, an extension may still not work after installing.

2. The extensions declares all working platform constellations. It uses exact product descriptions. The office only checks if the current platform of OOo corresponds to one of the platforms declared by the extension. If there is a match, then it is safe that the extension works, otherwise the extension does not work. The extension must be regularly updated to reflect recent platform developments.

The #2 method assumes that if a particular platform is not declared, then it was found to be not working and must not be installed. That is, the absence of a particular platform indicates that. In other words, we have an implied “negative list”. One could of course weaken this a little. For example, the users could be asked if they still want to install an extension, even if it does not declare explicitly that the user's platform is supported. But then, the extension should explicitly declare the “negative – list”. Only if the users platform is neither declared as working platform nor is contained in the “negative-list” then users would be asked.

By using the “negative list” and asking the user about the installation, the requirement for constantly testing new platforms and providing updates has gone (although it should be done regularly though). In this scenario it would also make sense that OOo makes its “guess checks” as well. For example, an extension declares as operating system Windows XP and there is no negative list of operating platforms. If this extension is now being installed on Linux, then the user would be asked if the installation should proceed. Obviously, that would not make sense. OOo could deduce that Windows XP is some kind of Windows and prevent the installation in the first place.

It appears that the last described solution offers a good compromise between “preventing the user from installing not working extensions” and the maintenance effort for extensions.

The supported platforms and the negative list of platforms could be declared as dependencies. For having OOo do its “guess checks” one could have these dependencies

• Basic Compiler

• Basic Operating System

• Basic Processor

For exactly declaring the platform, we would use

• Specialized Compiler

• Specialized OperatingSystem

• Specialized Processor

And for the negative lists we would use

• Not Working Compiler

• Not Working Operating System

• Not Working Processor

The names given here may not be final.

If one dependency implies another, then one only needs to declare one dependency. For example, currently all Windows operating systems only run on x86 processors. Therefore it would not be necessary to specify the processor unless the extension makes use of some specific processor capabilities.

It would also be unnecessary to declare a “Basic” dependency if there is the respective “Specialized” dependency declared. The terms for compiler, operating systems and processors need to be build-in into OOo so that it can actually do a platform comparison. When introducing a new term one can also provide the corresponding “Basic” term. Then OOo could deduce automatically the “Basic” dependency from a “Specialized” dependency.

When an user upgrades OOo, then extensions may not run anymore. For example, the new version of OOo could have been build with a new compiler which makes the installed extensions unusable. Therefore OOo should check if the dependencies are still fulfilled after an upgrade. Currently the dependencies are only checked during the installation of an extension.

To indicate that an extension is suitable for all platforms one would simply not declare the dependencies.

Locales

Because the locales have no influence on whether an extension can be installed, we would not use a dependency for declaring them. Instead we could introduce some informational tag in the description.xml, which could describe a set of locales. Every locale would be represented by a locale string of the form 'language-country-variant'.

 =================================================

That's for now. If you've made it until here, congratulations and thanks for your patience. If you like, you can discuss this article on dev@extensions.openoffice.org. The subject is not simple, therefore a good idea is always welcome :)

tags: development extensions

tags: development openoffice.org snapshot writer

tags: development framework openoffice.org

During last six  weeks, the PDFImport team in Hamburg experimented with Agile software development, especially using ideas of Scrum. We wanted to know,

• if a development style that adheres to (at least some) agile principles could be used at all in our environment and project,
• if agile development principles  might be promising to enhance our development process.

The Setup

Some of the things we did different than usually (in the hope to reflect agile development principles):

• We put developers and QA (three people altogether) into one physical room, enabling them to communicate any time with fewest effort.
• We practiced continous testing of each step in the development process, as well by unit tests, as by automated document comparisons, as by repeated manual tests.
  
• We used iterations of a fixed length (two weeks). Each iteration was planned by the Product Owner (see below) and the team together.  After each iteration, its results were presented to interested stakeholders - and there was absolutely no possibility of changing this date.
• We had very short, but daily status meetings at a fixed time.
• We installed the institution of a "Product Owner" whose task was to gather, combine and filter all input regarding product features and be the ultimately responsible contact person for the development team regarding which features to reach for, which to leave, and in which sequence.
  The Product Owner gave feedback at least before and after each iteration, but was also asked many times inbetween about the direction to go.
• We tried to give the team the room to focus as much as possible on one task (the PDFImport project) only.
  

Observations 

The experiment is finished for now. About the produced software see Thorsten Behrens' post.
About the "agile" way of working, I'd like to share some observations:

The team absolutely loved the combination of development and QA in space and time! 
Sitting together, being able to get QA feedback on development steps within hours. They loved to be able to communicate directly, e.g.  without the need of involving IssueZilla, because most issues could be resolved immediately.

Continuous testing isn't that easy. Finding ways to automate tests needs time. Often tests broke for reasons within the test or the used tools rather than for the tested code. In the first weeks there needed to be quite some time invested into the infrastructure of building and testing.
On the other hand, the continuous testing found a huge amount of defects that would have been much more costly when found and fixed later.

We found out that an agile development style has high requirements on the building infrastructure:

• New builds are needed at least daily, usually more often.
• Builds need to run absolutely unattended, without any need for human interaction.
  This is challenging, when not only compiling and linking, but as well installing an Office, installing an extension, running smoketest, unit tests and other automated tests - and all those on several platforms.
  

Some team members saw the frequent interaction with the Product Owner (in his role as customer representative) as the single most important factor in having a satisfying result at the end of the experiment. So, asking quite often, if developed features are still moving into in the right direction, seems to be a very good thing.

There were three iterations. While in the first two, many planned work had to be postponed, in the third iteration more than planned could be completed. Doing things in a new way definitley needs time to adapt.

The daily status meetings appeared not to be very useful for communication within the team, because the team communicated the whole day anyway. We kept them nonetheless, because they helped to

• keep aware of status and progress - providing a "beat" in the working rhythm,
  
• communicate blocks to the work, and try to find somebody to address these blocks,
  
• adapt planning, somtimes leading to the involvement of the Product Owner,
• reflect on the way things were done, sometimes inspiring to try things a different way or to use external resources.
  

A complete evaluation of the "agile" experience is still on the way, but one thing can be said: The experiment was promising. We learned a lot and there seem to be at least some practices that may enhance our development style.

tags: development openoffice.org qa software

tags: development openoffice.org