Friday May 29, 2009

What's up at Jülich? The latest 200+ TeraFLOPs of Sun-supplied HPC compute power is now up and running!

The JuRoPa (Jülich Research on PetaFLOP Architectures) system at the Jülich Research Center in Jülich, Germany has just come online this week. A substantial part the system is built with the Sun Constellation System architecture, which marries highly dense blade systems with an efficient, high-performance QDR InfiniBand fabric in an HPC cluster configuration.

We delivered 23 cabinets filled with a total of 1104 Sun Blade x6275 servers or 2208 nodes. Each of these nodes is a dual-socket Nehalem-EP system running at 2.93 GHz. The systems are connected with quad data-rate (QDR) 4X InfiniBand using a total of six of our latest 648-port QDR switches. As usual, we use 12X InfiniBand cables to route three 4X connections, thereby greatly reducing the number of cables and connectors, and increasing the reliability of the fabric. For more detail on the Nehalem-EP blades and other components used in this system, see this blog entry.

I've annotated one of the official photos below. Marc Hamilton has many more photos on his blog, including some cool "underground shots" at Jülich.

Wednesday May 27, 2009

I was hoping to attend CommunityOne in San Francisco next week (June 1-3), but I'll be beavering away here in Boston instead. C1 is the big, blow-out community event that covers all things OpenSolaris for the technical crowd --- developers and users -- with piles of technical sessions, lightning talks, labs and a host of other activities.

There are several registration options, including one free option that gives you access to two Deep Dive technical tracks on Tuesday as well as some free sessions on Monday. The Tuesday tracks are Developing IN OpenSolaris and Deploying OpenSolaris in Your Datacenter. Topics covered:

If you are interested in dropping by the Moscone Center next Monday or Tuesday for these tech talks, complete the free registration here. For details on the entire C1 event, see the event website or the wiki.

Monday May 25, 2009

[UPDATE: A few small errors fixed and some clarifications added. See Comments for details.]

I tried recently to add a package to the OpenSolaris contrib repository, but quickly learned I didn't have enough packaging experience to understand the directions provided at SourceJuicer so I did some homework, asked some questions, and eventually did successfully contribute a package. I've documented in this entry everything I've learned hoping it will be helpful to others who want to build and submit OpenSolaris packages. Specifically, I'll describe how I wrote the spec file for Ploticus (my favorite open source plotting/graphing utility) and how I submitted the package to OpenSolaris.

I used SourceJuicer to submit my package because it is the easiest way for a community member to contribute. Before getting into details, a few words about the overall submission process. Packages are first submitted to the pending repository, which is basically a holding area for packages on their way to the contrib repository, the primary repository for community-contributed packages. Once a package has been validated and successfully built, it can then be moved into /contrib. I'll cover all of this below.

On to the details.

To submit a package to SourceJuicer, you need to supply two files: a text file containing copyright information and a spec file. The spec file contains the information SourceJuicer needs to create a final binary package starting from source code. Ideally the OpenSolaris package will be buildable from the standard, community-released source code without changes, which may require asking the community to adopt changes necessary to build the code for OpenSolaris. In practice, this will often not be necessary since many packages are designed to build on several Unix versions. In cases where changes must be made and those changes have not been accepted by the community, it is possible to specify patches that should be applied to the community source code during the build process. Though not desirable, it is sometimes necessary to do this. I'll supply pointers to information on how to do this below.

Spec files are not an OpenSolaris invention--they have been used for a long time to build RPM packages. This is good news because there are several excellent web resources that document spec files in detail. I recommend Maximum RPM by Edward Bailey as a detailed reference. One complication: It seems that OpenSolaris spec files are not exactly the same as RPM spec files. However, for the purposes of this exercise, don't worry about this -- the Ploticus example below should give you enough information to create a valid OpenSolaris spec file in most cases. However, if you insist on worrying, you can read the information I found here and here. If anyone knows of a better explanation of the differences, let me know and I will include a pointer here.

Okay, lets get to it. I started with a spec file template and created the following file for Ploticus. My commentary includes all of the tips and other information I discovered during the process of writing the spec file for this particular open source package. While I've attempted to give pointers to additional information throughout, this is not meant to be the definitive guide to the full capabilities of spec files. There should, however, be enough information here to allow typical open source apps to be packaged and contributed to OpenSolaris. Consult Maximum RPM for additional details.

spec file	commentary
# # spec file for package: ploticus # # This file and all modifications and additions to the pristine # package are under the same license as the package itself. # # include module(s): ploticus #	This is all boilerplate commentary. Insert the name of your package twice.
%include Solaris.inc	Required for all OpenSolaris packages. For the curious, the source is here.
Name: ploticus	Once you specify the name of your package, you can use the macro %{name} to refer to it later in the spec file. As you will see below, there are other predefined macros available that you will use to write your spec file. You can also define your own macros using the syntax: %define macro_name macro definition
Summary: ploticus -- creates plots, charts, and graphics from data	Summary is a one-line description of the package that will be displayed by the OpenSolaris Package Manager.
Version: 2.41	The version number can be referenced as %{version} later in the spec file, which can often be used to generalize file and directory names. In the case of Ploticus the version number string (e.g. "2.41") happens not to be used as part of its filenames (e.g. ploticus241src) so I do not use %{version} in this example, except in one instance of boilerplate.
License: GPLv2	Free text field describing code's open source license. I've seen all of these used: GPL, GPLv2, GPLv3, BSD, LGPLv2.1, New BSD License. If GPL, be explicit if you can: GPLv2 or GPLv3. The "or later" licenses might be appropriate as well, e.g. GPLv2-or-later, GPLv3-or-later, etc. There is a nice discussion here about the pros and cons of "or later" licenses.
Source: http://voxel.dl.sourceforge.net/sourceforge/ploticus/pl241src.tar.gz	The source tag specifies the location of the source-code tarball (possibly gzip'ed) that should be downloaded to build the package. Because Ploticus is hosted on sourceforge I had to specify a manual download URL rather than that of the automated download site (downloads.sourceforge.net.) Note that the source location can also be specified as an ftp:// address.
URL: http://ploticus.sourceforge.net	The open-source community's web address.
Group: Applications/Graphics and Imaging	The group tag describes the kind of software in the package and will be used by the OpenSolaris Package Manager to categorize the package hierarchically. I chose a group name based on the package classifications listed here.
Distribution: OpenSolaris Vendor: OpenSolaris Community %include default-depend.inc	Boilerplate.
BuildRequires: SUNWxorg-headers, SUNWzlib, SUNWgcc	These are other OpenSolaris packages that must be available on the build system in order to correctly create the binary package. In this case, I am building Ploticus with X-Windows capabilities, so I need to ensure the X client header files are available. I am also enabling a Ploticus compression option so zlib is needed as well. And, to be safe, I've specified which compiler is required. I could have used Sun Studio, but I know for sure that Ploticus compiles with gcc so I've used that. You can find these package names by searching in the Package Manager on your local OpenSolaris system.
Requires: SUNWzlib	This section lists packages that must be installed on the end-user system for the software to work correctly. In this case, Ploticus will be dynamically-linked against zlib so I need to make sure the Package Manager knows about this dependency. When the users asks for Ploticus from the repository, the Package Manager will know it also needs to download and install the SUNWzlib package as well.
BuildRoot: %{_tmppath}/%{name}-%{version}-build SUNW_Basedir: %{_basedir}	This is boilerplate. The intent of BuildRoot is to define a user- and application-specific path that can be used as the root of an area in which your package will be installed on the build server, allowing the build server to support simultaneous builds of multiple packages by multiple users without interference. Note, however, that I do not use BuildRoot in this spec file because this conversation indicates that $RPM_BUILD_ROOT is the officially supported way to refer to the top of a package install area. I don't know if this is true in the OpenSolaris world as well, but most spec files I've seen for OpenSolaris use $RPM_BUILD_ROOT so I have opted to use that as well. Note that while $RPM_BUILD_ROOT (and BuildRoot) refers to the root of the installation area on the build server, the top of the build area itself -- the location where your package will actually be untar'ed and built -- is referred to as %{_builddir}. I do not know how SUNW_Basedir is used.
SUNW_Copyright: %{name}.copyright	This is the name of the copyright file you will upload to SourceJuicer along with this spec file. It must be named as shown (ploticus.copyright in my case.) You will typically find this copyright file on the community's website and/or included within the community's source tarball. In the case of Ploticus, the tarball contains a file in src called Copyright, which I have copied, renamed to ploticus.copyright and then edited to remove html markup. This is the file I will then upload to SourceJuicer. The original src/Copyright file is ignored by SourceJuicer. Update: The preceding was actually not sufficient for my package to be validated. I was asked to append the file GPL.txt, which was also in the tarball's src directory, to ploticus.copyright so that the actual text of the GPL v2 copyright was in the file. The original version of the copyright file (src/Copyright) only refers to the GPL copyleft, it does not include the copyright itself.
Meta(info.upstream): Steve Grubb <ploticus@yahoogroups.com> Meta(info.maintainer): Josh Simons <josh.simons@sun.com>	These fields are specific to OpenSolaris's packaging system. The upstream field contains the name and address of the individual or group that creates and supports the open-source software. The maintainer field contains the name and email address of the individual responsible for the OpenSolaris packaging of the open-source project. The preferred format is as shown in these examples. Additional info fields that can be included are documented here.
%description A free, GPL, non-interactive software package for producing plots, charts, and graphics from data. It was developed in a Unix/C environment and runs on various Unix, Linux, and win32 systems. ploticus is good for automated or just-in-time graph generation, handles date and time data nicely, and has basic statistical capabilities. It allows significant user control over colors, styles, options and details. Ploticus is a mature package, available since 1999, and version 2.40 has more than 12,000 downloads to date.	A more detailed description of the open source software. This description was taken from the Ploticus web page.
%prep %setup -q -n pl241src	Now we begin specifying what actions are required to build the software. The %setup macro cd's into the build directory, removes any cruft left over from earlier builds, unzips the source tarball (which will have been downloaded at this point), and then untars the sources into the build directory. It then cd's into the package's top-level directory. All of this is done with %{_builddir} as the root directory as described earlier. Note that %setup assumes the top-level directory specified in the tarball is named %{name}-%{version}. If this is not true for your package, use the -n option to specify the correct name. For Ploticus, all files in the tarball are in the pl241src directory, so I've used the -n option to specify this. See this page for more details about the %setup macro. The %patch macro, which can also be used in the %prep phase, can be used to apply patches prior to building the binaries if the standard community source code needs to be modified in some way to build successfully on OpenSolaris. See the same page for %patch information. Note that you should try to have your OpenSolaris changes accepted by the community to avoid having to apply these patches. I don't know what the -q option does.
%build cd src make NOX11= XLIBS='-L/usr/openwin/lib -lX11' XOBJ='x11.o interact.o' \ XINCLUDEDIR=-I/usr/openwin/include WALL= ZLIB=-lz ZFLAG=-DWZ \ PREFABS_DIR=/usr/lib/ploticus/prefabs pl	The %build section contains the commands needed to build the package binaries. At the end of the %prep phase we were left sitting in the top-level directory of the source tarball. Since the Ploticus makefile and sources are one level down from this (pl241src/src), I cd into src before invoking the correct make command for OpenSolaris. Assuming the make ran correctly, we exit this phase with the binaries and other files all built on the build server in a sub-directory under %{_builddir}.
%install mkdir -p $RPM_BUILD_ROOT%{_mandir}/man1 cp man/man1/pl.1 $RPM_BUILD_ROOT%{_mandir}/man1/pl.1 mkdir -p $RPM_BUILD_ROOT%{_bindir} cp src/pl $RPM_BUILD_ROOT%{_bindir} mkdir -p $RPM_BUILD_ROOT%{_libdir}/%{name} cp -r prefabs $RPM_BUILD_ROOT%{_libdir}/%{name}	In the install phase, we execute a "make install" or equivalent, moving all files that will be included in the binary package to their final installed locations, but relative to $RPM_BUILD_ROOT rather than to "/" to avoid collisions on the build server. Because the Ploticus "make install" action doesn't do exactly what I need, I instead manually move each required file to its final location. For many projects, something similar to "make DESTDIR=$RPM_BUILD_ROOT install" would be appropriate in this phase. If you are moving files manually, do not assume directories exist -- make them before you use them. And use the predefined directory macros (e.g. %{_mandir} ) to reference standard installation locations. Others are documented here.
%clean rm -rf $RPM_BUILD_ROOT	This is boilerplate clean-up code. Insert other commands as necessary.
%files %defattr(-,root,bin) %attr(0755, root, bin) %dir %{_bindir} %attr(0755, root, bin) %dir %{_mandir} %attr(0755, root, bin) %dir %{_mandir}/man1 %attr(0755, root, bin) %dir %{_libdir} %attr(0755, root, bin) %dir %{_libdir}/%{name} %attr(0755, root, bin) %dir %{_libdir}/%{name}/prefabs %{_bindir}/* %{_libdir}/%{name}/prefabs/* %{_mandir}/*/*	This can be a complicated section so I suggest reading the Max RPM %files section. The %files section specifies the locations and attributes of all files that will be placed onto the end-user's system when the binary package is installed. The %attr directive is used to specify permissions and ownership for files and directories. The %dir directive identifies directories. Multiple directives can be applied to objects by including them on the same line. The first line specifies default mode, default user ID and default group ID for all files created during the build process. The dash ("-") means that a default is not set explicitly for that field. Note that failure to include this line in your spec file will cause an obscure error to be generated when an end-user tries to install your package. That would be very bad. The next four lines specify the directories in which Ploticus-related files will reside. The last three ensure that the Ploticus binary, all of the Ploticus prefabs config files, and the man page will be included in the binary package. Note again the use of macros to specify standard installation directories.
%changelog * Tues Apr 28 2009 - Josh Simons <josh.simons@sun.com> - initial version	Add any changelog information you desire here.

Once you've created your spec file, it is time to feed it to SourceJuicer for syntax and other checking and then iterate as necessary until your spec file is correct and has passed validation. The basic flow is shown in the diagram below.

The first step is to submit the spec file to SourceJuicer along with the project's copyright file. To do so, go to the SourceJuicer Submit page (login required.) Assign a descriptive name to your upload (I used 'ploticus') and then specify your spec file. Use 'add another file' to add your copyright file. Add whatever other files you may need (see 'more help' on the Submit page.) Click Submit and you will see a page like this:

The summary page includes an indication that my spec file successfully passed a syntax check. If an error occurs at this point, make the necessary corrections and use the ReSubmit tab (not shown) at the bottom of this page to upload new versions of your copyright and spec files.

Looking under Reviews, I can see my package has not yet been validated, which means my submission hasn't yet been checked by someone to ensure my copyright file is appropriate, that someone else has not already packaged this program for OpenSolaris, etc.

The next day I receive two email messages with comments from reviewers. When I log back into SourceJuicer and look at the Review tab, I see the two comments that were submitted. The fact that the package is still marked as not validated means I have issues to address:

Clicking on the "[review]" link takes me to the page with detailed information about the Ploticus review. I can also view this page by visiting the MyJuicer tab and then clicking on the appropriate link under My Submissions. This second method is better since it can be difficult to find your review on the main Review page. In any case, the page looks like this:

As you can see from Amanda and Christian's comments, I did not use the correct naming convention for the copyright file I uploaded to SourceJuicer. Rather than "Copyright", the file should have been named "ploticus.copyright" (more generally, %{name}.copyright). Also, Amanda hopes I can remove the html that is for some reason embedded in the standard Ploticus copyright file.

Using this same review page, I submit a clarifying question back to the reviewers to ensure I address their issues. I am not clear on the relationship between the copyright file that is submitted manually to SourceJuicer and the copyright file in the source tarball that is described with the "SUNW_Copyright" tag in the spec file.

Now that I understand the copyright issue and have adjusted my spec file and copyright file appropriately (and also updated the spec file and annotations in this blog entry--meaning you never saw that I had initially called my copyright file "Copyright"), I use the same Review page to Resubmit the spec file and copyright file. Use the tab at the bottom of the Review page to do this:

As of this writing, there is no way to remove a file that has been submitted to SourceJuicer so all three files (Copyright, ploticus.copyright, and ploticus.spec) are associated with the project even though Copyright is now extraneous. Until removal is possible, just ignore the extra files. [UPDATE: As of SJ 1.2.0, files can removed by visiting the MyJuicer review page for the appropriate package.]

I resubmitted the files, the package was subsequently validated, and then it was automatically scheduled to be built on the build server. I did not receive a notification when the build attempt occurred so you need to check status periodically (use the MyJuicer tab). When I checked, I saw my build had completed successfully on the first attempt:

Had the build not succeeded, I would have followed the Log link to view the build log, found the problem, fixed the spec file, and then Resubmitted. The package would then be rescheduled for another build automatically with no need for re-validation.

With the Ploticus build successfully completed, it is now very important to verify that the package installs correctly and that the software actually works. Though I don't cover it here, my first Ploticus package did not work correctly on my test system. I had to make changes to my spec file, rebuild the package, and reinstall it. Therefore, please do install and test your software!

To do the test installation, I first added the pending repository as a package authority on my 2008.11 system. Note carefully the location of this repository; I had expected it to be http://pkg.opensolaris.org/pending, but that is not correct:

% pfexec pkg set-authority -O  http://jucr.opensolaris.org/pending pending

I then started the Package Manager, selected the Pending repository and did a search for Ploticus. Voila! The package is available:

After selecting the package and clicking on Install/Update, the installation proceeds smoothly. I then start a terminal window and verify that Ploticus does, in fact, work correctly:

Once you are sure your package installs and runs correctly, send an email to sw-porters-discuss@opensolaris.org requesting that the package be promoted from the pending repository to the contrib repository. Note that you'll need to subscribe to this mailing list before you can post to it. To subscribe, go here.

Once the package is available in contrib, users will be able to install your package on their systems.

FIN!

[See my later blog entry for additional information about SourceJuicer and OpenSolaris improvements that make package contributions even easier.]