Going on a Warning Hunt

Like the children's story "Going on a Bear Hunt" it's time for a "Going on a Warning Hunt" in the OpenJDK sources.

You'll probably see a set of bugs/rfes being filed and fixed over the next few months related to reducing the warning messages in the OpenJDK builds.

As the warnings get fixed, we also may want to dial up the warnings found with compiler options that ask for more warning diagnostics. We could even include the use of tools like pscan and findbugs, but that's beyond the scope of this hunt. Compiler options to get more warning diagnostics, treat warnings as fatal, and suppress specific warnings:

Compiler	Warnings Requested	Make Warnings Fatal	Disable Specific Warnings
gcc/g++	-W -Wall -Wno-unused -Wno-parentheses	-Werror	-Wno-option
Sun Studio C++	+w (Added recently in jdk Makefiles)	-errwarn=%all	-erroff=tag (Use -errtags to get tags)
Sun Studio C	-v	-errwarn=%all	-erroff=tag (Use -errtags to get tags)
Visual Studio	-W3	-WX	-wdnumber
Java Compiler	-Xlint:all	-Werror	In the source: @SuppressWarnings("warning")

Making warnings fatal can be problematic. Each compiler version or release can introduce new warnings and it's a bit frustrating to get a build failure because you used a slightly different compiler. So we may need to make sure the compiler is a specific version or release before making the warnings fatal. The Java Compiler may be the exception, but it may be a while before we get to the point where we can turn this on.

Fixing warnings in native code can be more risky than fixing them in Java code. Some common things to watch out for:

• Casting pointers to and from integers can be tricky, it shouldn't happen very often and when it does great care needs to be taken in making sure the complete pointer is contained in the integer type. Pointers will be 64 bits in some situations and will require a integer big enough to hold the full 64 bits. The type void* is the so called generic pointer and you may find it necessary to cast a pointer to the generic pointer type before casting it to anything else. Some people will use size_t as a guaranteed integer type that can hold a pointer, others use intptr_t which is the integral type result for pointer subtracts.
• Casts to long. Windows compilers may define long to be 32 bits even in 64 bit mode, so be careful casting any kind of pointer to a long.
• Functions like strlen() return the type size_t which may be a 64 bit integral type. It's not unusual to always cast this type to an int but be careful to consider if it ever makes sense that you might need a larger result than what can be held in a 32 bit integer.
• It's a good idea to always be explicit when you are downsizing an integral value. Passing an int to a function that expects a short probably warrants an explicit cast to int. Many compiler warnings relate to the implicit downsizing conversion the compiler will perform on assignments or arguments passed into function calls.
• No consequence statement warnings happen when the expression statement makes no assignments or function calls. This can happen with debug statements that result in (0). Macros like assert(expression) should result in nothing when not generating debug code.
• Suppressing warnings with options or #pragma directives should probably be a last resort.

So far the bugs I have included for this warning hunt are:

• 6255702: Need to clean up remaining var-args related warnings.
• 6214406: MS VC++ compiler warning at jni.h when using fastcall, enable optimization
• 6323942: jvmti.h causes C compiler warning
• 6375033: Hotspot build warnings need cleanup
• 6496269: Many warnings generated from com/sun/java/util/jar/pack/*.cpp when compiled on Linux
• 6712344: Add -errwarn=%all to Solaris Sun Studio compiles (all warnings being fatal)
• 6722802: Code improvement and warnings removing from the javax.swing.text pa ckage
• 6725543: Compiler warnings in serviceability native code
• 6725625: Compiler warnings in awt code
• 6725818: Compiler warnings in OpenGL code
• 6725819: Compiler warnings in gtk2 glue code
• 6725821: Compiler warnings in medialib code
• 6726309: Eliminate warnings from NIO code
• 6727661: Code improvement and warnings removing from the swing/plaf packages
• 6727662: Code improvement and warnings removing from swing packages
• 6727683: Cleanup use of COMPILER_WARNINGS_FATAL in makefiles

And the hunt is on. Lock and load. :^)

-kto

Posted at 05:07PM Jul 21, 2008 by kto in Java  |  Comments[9]

A Complicated Slide?

This Dilbert comic strip reminded me of a drawing I did a while back to try and explain the critical nature of our bug tracking system, and how all the various systems we have and use connect back to the bug database. Granted, right now the JDK team is using an internal bug tracking system (that will change someday), and the visibility is limited to what you can see via http://bugs.sun.com/.

The complicated drawing below is mine alone and is not an official JDK drawing and is undoubtedly not 100% correct, it is just my attempt to convey some of the connections to our bug tracking system:

And a few terms to define the words used:

• Builds Archives: Promoted builds, nightly builds, and the JPRT build archive.
• Alacrity: A system and set of machines used to run our more critical performance benchmarks in a controlled environment and report on performance results on JDK builds.
• GTEE: The formal test system that verifies our JDK builds pass all our tests (all our many different kinds of tests).
• JPRT: My balliwick of late. JPRT is a set of systems and machines (multiple instances of JPRT) that provide build and basic testing on all platforms and both VMs (client & server). JPRT stands for Java Programmer Rat Trap... I'm kidding, I'm not sure what the name stands for anymore. There was an old HotSpot PRT system and that stood for Performance Reliability Testing, and this newer system works with all the JDK repositories, so I named it JPRT. Anyway, it is a system that developers and integrators can submit repositories or source trees for builds on all 8 of our standard platforms (Solaris SPARC, Solaris X86, Windows, and Linux, both 32 and 64 bit). The builds are archived and JPRT can actually update the bugs and push changesets when the job is successful. This is a system we need to somehow make available externally but we don't know exactly how just yet.
• Testbase: Just represents all the various testbases we have for testing the OpenJDK. Some of these tests are actually in the JDK repositories. These include the JDK TCK (JCK).
• CR Bug Database: Our internal bug database, some of which is visible via the bugs.sun.com site. The whole point of making this drawing (quite a while back) was to demonstrate how the CR number is used in so many places and the bug tracking system is at the center of everything. Someday, this may be bugzilla for the OpenJDK.
• CCC: The committee that reviews any change to a user visible API in the JDK. All developers making changes that involving any change to a user visible API or command line option or file format, must file a CCC request and get approval from CCC before pushing their change into a public repository. This is currently an internal process but will be externalized at some point.
• Integrators: Those blessed souls that deal with the merges between teams and integrate all a teams changes into the master jdk7 repositories.
• Planning: The planning process will always refer to the CRs as part of the planning process.
• Source Repositories: The Mercurial repositories or TeamWare workspaces for some of the older releases.
• WebRevs: An old, extremely valuable, and historic way that Sun Engineers have used to do code reviews. All changes to the JDK are required to be reviewed by at least one other engineer, more than one at critical times of a release. See Jesse's webrev blog on the version of the webrev script we use in the JDK area.

I hope this helps someone and doesn't really confuse people. And no, the rocket ship is not blasting the webrevs. :^)

-kto

Posted at 10:57AM Jun 11, 2008 by kto in Java  |  Comments[2]

OpenJDK Bug States

Until the OpenJDK project converts to Bugzilla, I thought some more information about how our internal bug tracking system works might help people watching the current situation.

The OpenJDK change requests (CRs) or bugs are currently visible at bugs.sun.com.

There are 11 states for a CR and the normal change of state is the following:

1. Dispatched: The initial state for a new CR.
2. Incomplete: Something critical is missing from the CR.
3. Accepted: CR was accepted, the first step in being investigated and fixed.
4. Defer: Work on this CR is on hold.
5. Cause Known: A basic understanding of the problem is known.
6. Fix Understood: A basic fix is now understood.
7. Fix in Progress: The assigned engineer is actively working on the fix or getting it integrated.
8. Fix Available: A changeset or the actual fix has been made available in a team area.
9. Fix Failed: Something went wrong with the fix.
10. Fix Delivered: The changeset or fix has been integrated into the master area and will show up in the next build promotion.
11. Closed: There are many reasons why a bug will be in the closed state. It might be verified as fixed, closed as a duplicate, closed as 'not a bug' or closed as 'will not fix'.

The states 1-9 are considered "unresolved", so until a bug becomes "10 - Fix Delivered" or "11 - Closed", it is still considered unresolved. So being "unresolved" may mean that a fix is available, just not in a position to be made part of a formal build promotion.

The bugs.sun.com interface is read-only and somewhat crude, but you can query the bug information, for example the Unresolved JDK Build Subcategory Bugs and RFEs. Hope this helps explain things.

-kto

Posted at 12:25PM May 11, 2008 by kto in Java  |  Comments[4]

Sun Studio Compilers on LINUX?

With the magic of Mercurial, you can see changesets, like this one: http://hg.openjdk.java.net/jdk7/jdk7/hotspot/rev/485d403e94e1. Which Serguei Spitsyn integrated recently.

But wait, what does this changeset actually mean? Sun Studio on Linux? Does that make sense? YES! It does and it's true. Mind you, it's just Hotspot that can be built with the Sun Studio compilers on Linux right now, but it's an important piece, the Hotspot C++ code is not a trivial pile of code to compile and optimize correctly. My hat is off to the Sun Studio team in making this all happen. What should be interesting now is how well the rest of the tools work like dbx and the analyzer/collector.

More can be read about the Sun Studio Linux Compilers at the Sun Studio Site.

Humm, I guess I'm now on the hook to see if the rest of the OpenJDK can be setup to build with Sun Studio on Linux.... Back to work...

-kto

P.S. No, we are not abandoning gcc/g++, just providing choices on how the OpenJDK can be built.

Posted at 12:47PM May 07, 2008 by kto in Java  |  Comments[7]

InfoQ Article on Git, Mercurial, and Bzr

A while back Sébastien Auvray asked me some questions about the OpenJDK Mercurial conversion. His article was recently published at http://www.infoq.com/articles/dvcs-guide.

Has some interesting information and stats about Git, Mercurial, and Bzr.

-kto

Posted at 12:18PM May 07, 2008 by kto in Java  | 

JDK Build Readme Collection

A collection of links to various JDK Build readme files.

• OpenJDK Build Home Page
• OpenJDK 7 Build README
OpenJDK 6 Build README (inside the source bundle from the download site, file README-builds.html)
• OpenJDK NetBeans Build README
• JDK 7 JRL Build README
• JDK 7 Build README
• JDK 6 Build README
• JDK 5 SCSL Build README

Posted at 04:07PM Apr 30, 2008 by kto in Java  | 

OpenJDK7, Solaris, and Sun Studio 12 Compilers

We are starting work to update the compilers and build OS releases used to build OpenJDK7 and JDK7. The implications of a build system OS change may mean that the JDK7 binary bundles may support fewer OS releases. So this is a bit of a status report, and a bit of a heads up. Hopefully most people will cheer over this news, but we'll see.

Solaris

On Solaris we have already moved to Solaris 10 as the base OS, which means that the build bundles will no longer run on Solaris 8 or Solaris 9 systems. We will also be moving from the Sun Studio 11 compilers to the Sun Studio 12 compilers. Sun Studio 12 is FREE for SDN members (registration is free too). Keep in mind that this doesn't mean that you can't build with Sun Studio 11, just that we will focus on using Sun Studio 12. At some point using older tools or OS releases to build the OpenJDK may become an issue, but it is not our intent to create these issues. There are some places in the JDK where we would like to depend on some newer features of the OS being available, so there are no promises that the OpenJDK7 sources will stay buildable on Solaris 8 or 9, but currently they are.

The Solaris Makefile changes for Sun Studio 12 (SS12) will include changing to new -xarch options as described in the "What's New In The Compilers" document. The old -xarch options will continue to work fine, but generate warning errors, and so the change to the new options is to avoid the warning errors.

Note that since we still use the assembler that comes with the Solaris system at /usr/ccs/bin/as, we still have to supply it with the old option spellings.

We did run into two build issues so far:

• Using -xO4 built .o files with dtrace is giving us some problems. We suspect the optimizer may have tossed some C++ methods that exist only for dtrace probes. We had to change -xO4 to -xO3 -g on three files in hotspot.
• We had to revert the C++ debug format from Dwarf2 to stabs with the option -xdebugformat=stabs. We were getting some ld errors on some Elf sections during link time. We suspect this is a known problem with Elf COMDAT sections and will be watching for a fix.

It may be a while before we push these Makefile changes to the OpenJDK repositories. Changing compilers with the JDK on any platform is not as trivial a task as people might think. With Solaris and the Sun Studio compilers, we have an advantage, knowing the team doing the compilers. But with any compiler change, our biggest issue becomes performance and correctness. Not so much the performance and correctness of the C/C++ generated code, but the combination of this C/C++ code with the Hotspot generated code. It's not unusual to have bad runtime interactions between the C/C++ generated code, and the Hotspot generated code, on any platform and with any compiler revision.

So the next steps will involve running formal benchmarks to verify that we haven't regressed in performance. A somewhat difficult task given the moving target that OpenJDK7 is right now.

Linux

We already know that building the OpenJDK with different Linux releases and gcc compiler versions is working, so with Linux we just need to advance to a slightly newer Linux release as the base OS, and one that gives us the largest set of deployed targets. We still run the risk of performance and correctness issues like with Solaris, so the same benchmark exercises will need to be done. This will mean that the JDK7 binary bundles will no longer run on some older Linux systems. That doesn't mean it can't be built for these versions, just that the binary bundles we supply may be limited in the Linux releases it supports.

Windows

There is some efforts going on to see about changing to the Visual Studio .NET 2008 compilers, and potentially Windows XP as the base OS for 32bit. Again, dito on the performance and correctness issues. This will mean that the JDK7 binary bundles will no longer work on Windows 2000.

Official Build Configurations

The current list of build machines for JDK7 is documented in the JDK7 Build README:

• Solaris 10 SPARC
• Solaris 10 SPARCV9
• Solaris 10 X86
• Solaris 10 X64
• Linux X86 Redhat Enterprise Advanced Server 2.1 update 2
• Linux X64 Suse 8 Enterprise Server - AMD64 Edition
• Windows 2000 X86
• Windows 2003 X64

Official Supported Configurations

The Official JDK6 supported configurations are those configurations that we have tested the JDK6 builds on. The official supported list for JDK7 will need to be adjusted of course, certainly Solaris 8 and Solaris 9 support for JDK7 will be removed.

Other Configurations

The OpenJDK sources can be built on many more systems as documented in the OpenJDK Build README. The intent with the OpenJDK is to allow for it to be built on as many configurations as possible, but that does not necessarily mean that a configuration will become any kind of officially supported JDK7 configuration. The OpenJDK project is an open-source project, and to date, binary builds have not been provided.

Other architectures like IA64 or PowerPC are or may be considered "ports", depending on the need for changing the Hotspot VM to generate code for a different architecture. There are several "ports" being talked about on the OpenJDK porters alias, see the OpenJDK Mailing Lists. Recently, changes to Hotspot were pushed into the OpenJDK source repositories that allowed for Linux SPARC builds of Hotspot, see the changeset Open-source hotspot linux-sparc support. Email exchanges on the distro-pkg alias indicate people are building on Linux IA64, not sure if they have been successful or not, I have no idea how IA64 this would be. Of course, successfully building is only step 1, the next step is to see if it works, and works reliably. The point being that many people are taking the OpenJDK sources or IcedTea and building on many configurations.

The funding needed to take on and support a new official configuration is considerable. We hope that the OpenJDK developers contribute back the changes made to port or build on any configuration. But adding a new configuration to the official JDK7 build configuration list is a high level decision.

-kto

Posted at 03:35PM Apr 30, 2008 by kto in Java  |  Comments[6]

OpenJDK: Dude, Where's My Changeset?

Developers are asking where the changesets are, reminds me of that movie Dude, Where's My Car?, "Dude, Where's My Changeset?"

So it seemed appropriate to bring up the "OpenJDK Integration Wheel" again and talk about how a changeset moves around:

Let's look at the changesets (fixes) pushed to the Swing Team Area. Let's pretend a changeset was pushed to the Swing area and is now publicly available. Not picking on the Swing team, just didn't want to change the picture :^) Feel free to change the example from "Swing" to "Build" and the Build Team Area. And Then? Immediately, anyone in the Swing team can pull these changesets down and have access to them. Other developers from other teams could also pull from this Swing area to get the change, but that's a bit abnormal, certainly possible, just not normal. And Then? Depending on the Integration Schedule these changesets will likely remain in the Swing Team Area until the official Swing integrator pushes those changesets into the Master area (the Integration Process). And Then? Once integrated into the master area, the other teams still may not see these changesets until their own integrators choose to "sync up" or send these new changesets in the Master area into their own team areas. This "sync up" usually happens as a team integrator prepares for the integration, so based on the order in the Integration Schedule, it could be a day or many days before a team will see changes that have been pushed into the master area. And Then? Even after the team area has the changesets, a developer won't see the changesets until he pulls the changesets into his own area.

So you can see how sometimes it's hard to find a changeset. Somewhat independent of the changesets flowing into the various team areas, the Release Engineering Team will use the Master area and attempt to create a promoted build, and if successful will create tags in the Master repositories to record what changesets were included in a promotion.

Some people will find this whole process frustrating, but there are some big advantages. The delay of the changesets getting into the Master area allows for the team most closely associated with the change to test and verify, from that team's perspective. Odds are that bad changesets will get caught by the team, and this protects the other teams. The process the team integrator goes through includes additional testing to verify the changesets being integrated are solid for everyone. It's not unusual for an integrator to run into a problem with his team's changesets, which again protects all the other teams from potential disasters. Granted, regressions will still happen, but really nasty regressions are usually caught early.

Hope this helps explain things.

-kto

Posted at 12:02PM Apr 18, 2008 by kto in Java  | 

How Infrastructure Engineers are like Drug Pushers

I've found myself in a situation where I am working on basic engineering infrastructure almost all the time. Things like fixing build issues, working on Makefiles, writing tools to help developers get their job done, and keeping an automated build&test system up and running. Don't get me wrong, I don't mind it, and rather enjoy doing things that effectively makes everyone more productive. It's a unique job, a bit of the old Jack of all trades position.

But am I like a drug pusher? After an email exchange with a fellow infrastructure engineer, we came up with this list:

Drug Pusher	Infrastructure Engineer
The product can be addictive	The tools can be addictive
Users rarely admit we exist	Developers rarely admit we exist
Users don't talk about us openly	Developers don't talk about us openly
The product make most users feel good	The tools make most developers feel productive
Users go nuts without the product	Developers go nuts without the tools
Users would like these supplies to be free	Developers expect these tools to be free
When the product doesn't work for them anymore, they expect new products that work exactly like the old products, but better	When the tools don't work anymore, new tools are expected, but they must work exactly like the old tools, but better
When the users are under the influence and happy, nothing else matters	When the developers are happy and everything is working fine, nothing else matters

Now if I could just figure out how to make lots of money from infrastructure tools. ;^)

-kto

Posted at 08:38PM Apr 10, 2008 by kto in Java  |  Comments[2]

OpenJDK, Mercurial, and The Changeset View

Why do I have to create a "Merge" changeset when there was nothing to merge?

For most of us old TeamWare users, and maybe other SCM users, the need for all the Mercurial "Merge" changesets (or as some people politely refer to as 'merge turds') seems confusing and messy. If the changes don't involve the same file, it can be hard to understand why you need a Merge changeset.

What did TeamWare look like?

In TeamWare a 'resolve' was necessary only when there was a conflict, meaning that two people changed the same file. The tool 'filemerge' provided a way to easily deal with each file conflict, but merging changes is and will always be a risky business. Everyone has had an experience with a 'bad merge', they are nasty problems. No Source Code Management (SCM) tool completely removes the need for merging, and our only hope is for the merging tools to help us out here. It is probably true that a Distributed SCM like TeamWare, Mercurial, or git may create the need for more frequent merging, but the end result is often the same as a non-Distributed SCM, so maybe with a DSCM the merge work is also distributed? Anyway, I digress.

With TeamWare, the 'resolve' action resulted in multiple revisions in each SCCS file that had a conflict. The TeamWare tool 'vertool' vertool provided a way to pick an SCCS file and view it's revision history. Again, this was on a per-file basis, and although that created some benefits for developers, like being able to 'putback' just one file change, it also made it a little difficult to record the true state of the entire workspace. Here is a snapshot of vertool in action for anyone that hasn't seen it:

Notice the SCCS revision graph, when conflicts happened, the graph gets a little more complicated, but unless the changes are abandoned, it always connects back up to the main trunk of the graph. With TeamWare, every file was controlled with SCCS, and every file had a graph. The connections between files was never formally managed by TeamWare, but TeamWare provided some tools like 'freezept' to allow you to try and manage it.

And with Mercurial ...

The changes come in changesets or grouped changes to files, which are treated and tracked as changes to the repository. Yes, the changes are made to specific files, but the revision tracking is done for the entire repository. When a merge situation in Mercurial happens, and they will be frequent, a new changeset has to be created to potentially carry any file merge changes, but most importantly to identify the merged or joined results of two changesets. All changesets have at least one parent changeset, but Merge changesets have two parent changesets. Everytime you do an 'hg pull' that adds new changesets in your repository, and your repository has changesets that have not been pushed yet, you have created what is called a 'multiple head' situation and you will need a Merge changeset. A 'head' is a changeset with no descendants, the tip changeset is a head and must be the only head if you want to push your changesets to the OpenJDK repositories (we do not allow any multiple head pushes with the OpenJDK repositories). This unfortunately means that people that do frequent "syncs" with their parent repository may be creating many Merge changesets, that's just the way it is, like Taxes, we will need to learn to live with it.

The 'hg view' command of Mercurial can provide some insight into this Merge business. To use 'hg view' you need to:

1. Enable the hgk extension in your ~/.hgrc file.
2. Make sure that the hgk tool in available from your PATH environment variable setting. You may need to download the Mercurial source bundle that matches the version of Mercurial you are using and get the hgk file from the contrib directory.
3. Make sure the wish tool is available from your PATH environment variable setting. Note that Solaris Express has a /usr/bin/wish that works, and the MacOS 10.5 has a /usr/bin/wish that works, but you may need to do a little searching to find a wish that is acceptable to hgk. Solaris 10 and older machines may have one at /opt/sfw/bin/wishx or /usr/sfw/bin/wisk8.[34]

For example, to see what the most recent changesets pushed to the OpenJDK jdk7/jdk repository look like:

   hg clone http://hg.openjdk.java.net/jdk7/jdk yourjdk
   cd yourjdk
   hg view

You should then see something like this:

Looks a little like a Public Transportation System. Notice the groups of changesets created by developers, usually generated one right after the other. If two developers manage to line up (luck), the sequence is simple, but the second one to do a push had to do a pull and create a merge changeset. Layering on top of that is the integrations of the various teams to the master repository, which should appear as major addition to the graph.

Since a changeset is a repository revision this has tremendous benefits. For example, anyone can re-create the state of a repository (all the files) as of any changeset by simply doing:

   hg clone -r 82c85cfd8402 yourjdk trimmedjdk

Creating a separate repository that represents the state of everything as of that specific changeset id (which happens to be a changeset I created, specifically http://hg.openjdk.java.net/jdk7/jdk7/jdk/rev/82c85cfd8402).

I hope this has been helpful to at least a few people. Send me comments if I can clarify this more for people.

For more on the hgk extension go to http://www.selenic.com/mercurial/wiki/index.cgi/HgkExtension. Thanks to Chris Mason for creating this great extension.

-kto

Posted at 12:19PM Mar 27, 2008 by kto in Java  |  Comments[7]