Stories about Java SE in the Embedded and Real-Time world.     Dave Hofert, reporting. Java@Work

Back again after a series of travels, I wanted to start commenting on the progress of our Java Real-Time System (Java RTS) product. Specifically, I wanted to talk about two aspects of Java RTS that set it apart from other real-time solutions on the market.

As you may or may not know, Sun released the Java Real-Time System in July (Press Release). This product is the ONLY fully compliant implementation of the Real-Time Specification for Java also known as JSR-001. Yes, the first JSR - the one that started the JCP.

So, you ask, why did I highlight the ONLY text above? Because I'm trying to correct a perception out there that other RTSJ-compliant versions of real-time Java exist. There are other vendors (who shall remain nameless for the moment) who claim that their products "support RTSJ". In fact, they may use some of the same API calls and they may in fact provide some of the necessary functionality, but they have NOT yet passed the TCK (that I am aware of). This is a big deal -- why? Because only by passing the TCK can customers and developers (probably you) have a choice in which product they'll use. Choice means competition; competition means better products and better value. You, the developer or customer, want this. Note too that JSR-01 defines precisely how Java should support real-time processing. If your vendor doesn't pass the TCK, then they're not using real Java and you are using something that won't allow you to easily migrate your code to another vendor's offering.

So what to do? Well, the first thing is to do is if you are using one of these Java-based products that claim RSTJ compliance is to ask your vendor for proof that they've passed the TCK. I expect at this point that you'll hear some hemming and hawing and then some discussion about h ow their solution is better for you because the vendor didn't have to add in all of these features, etc., etc. Sorry - it's not better. The RTSJ was developed by Java experts and real-time experts and they spent the better part of 6 years coming up with the mechanisms and interfaces specified in RTSJ. [More on this topic in a future post].

Now then, once you are past the compliance issue - what else sets the Java RTS product apart? Primarily the fact that it's based on J2SE and that it can scale in the best traditions of Sun's products. Normally, Java-based real-time systems -- and the majority of C/C++-based real-time systems -- are uniprocessor solutions. Typically they are very focused on the target activity and they don't do much processing outside of the function they are designed for. This is a fine architectural choice, but it does require specialized communication to coordinate and it also means multiple platforms, multiple vendors, and likely multiple development languages. This means complexity - and cost.

The Java RTS product is based currently on Java SE and on Solaris 10 (yes, Solaris has real-time capabilities). Our reference implementation is on a 2-way (2 processor) system which allocates hard real-time threads to one processor and non-real-time tasks to another processor. This enables us to achieve latencies of 20 microseconds on a relatively slow 1 GHz machine. This software stack also allows us to scale -- up to a 12-way system so far. This opens the door to a whole new class of real-time systems where the environment requires a large amount of processing along with some hard real-time capabilities. Example domains are military/aerospace command and control centers, telecom infrastructure, large-scale industrial control systems, etc. But we can also run on a single processor as well. And we're 100% Java - so you can migrate your work across processors as we flesh out the number of platforms supported.

I'll follow up shortly with some other interesting aspects of the RTSJ that also make real-time programming a lot easier with Java RTS.

Without a lot of beating around the bush, I'm sorry to report that Team Jefferson's entry into the DARPA Grand Challenge did not make it to the final round - the race in the desert. I'm extremely saddened because it was clear from scoring in earlier rounds that Tommy would likely have been invited to the race. It all comes down to a series of unfortunate events which is the way things go in the embedded and real-time community. Complex systems (hardware, software, processes, and people) have a way of breaking just when you don't want them to.

The crash and aftermath
The sequence of events that led to Tommy's final ride go something like this:
1. component failure on the throttle control system
2. throttle goes wide open; ignores further instructions
3. DARPA monitor selects e-stop "PAUSE" function - no response. Another option "STOP" could have been selected, but everyone's surprised at this point
4. Tommy avoids one obstacle at speed, but eventually the central equation to physics applies: F=(M)x(A)

Results of Newtonian physics as applied to vehicles moving at 40 MPH into stationary objects

Team Jefferson, to their credit, was undaunted by this turn of events. Over the next 24 hours, they stripped the electronics from Tommy and the front end and motor. A new motor and wheel and suspension parts were sourced and installed. Finally the electronics were reinstalled.

Tommy undergoing repair


Tommy's last stand
By late Tuesday, DARPA officials said that if Tommy could pass a pre-trial inspection at 7 AM on Wednesday, the last day of the trial, then Tommy would be given two last runs to prove it could continue. At 6:30 AM, Tommy arrived in the trailer having come back to life at midnight. Much work remained however. Most importantly, some sensors needed to be bolted on and calibrated. Clearly time was running out.

Reconstructed Tommy heading into final trials

The time for the final trials arrived. Tommy was fully reassembled, but the team needed a few more hours to finish calibration of the systems. The need for calibration quickly became apparent during the first run where Tommy kept drifting to the right on all of the challenges. It eventually became stuck in the tunnel sequence. After just a few quick adjustments, Tommy took its second run but still favored the right side and eventually got stuck in the mountain pass - all obstacles they had cleanly passed before the crash. Tommy's trial was over.

Final Thoughts...
Team Jefferson demonstrated many things during their run at the Grand Challenge. First, they demonstrated the ability and value of Java to control a complex, autonomous vehicle. Tommy was the only vehicle I'm aware of that used Java exclusively to control the vehicle. They also demonstrated the value of COTS (common-off-of-the-shelf) components and the most simple design. Tommy used one programmer on and off over the course of a year where other teams had 10s of programmers contributing code and they followed an approach of simplification to solve problems rather than adding more and more sensor and processing. Finally, Tommy's team amazed the DARPA staff and other teams by demonstrating some real grit in reconstructing their vehicle in time for one last shot at the trial. While they didn't reach the finals, they did pass the trials course extremely well on run 3 and proved that their approach can work. Team Jefferson and it's leader Paul Perrone gave it their all and deserve a long and loud round of applause.

Once the dust settles, we'll revisit Tommy to take a closer look at how Java was used, so stay tuned for a future article on the subject.

As noted in my last entry, the trials course and the Grand Challenge race course are places where things are bound to go wrong. In this entry I'll catch you up on recent progress made by the team on the course. But first a bit more about Tommy to give you more context.

Tommy, Java, and the drive for simplicity
Paul Perrone, the leader and chief programmer for Team Jefferson and Tommy, has quite a background in Java. His experience, partially recounted in a recent interview with java.sun.com, explains how he felt that Java would enable faster development and faster prototyping for his robotic systems. This certainly proved true with Tommy; as noted in my last entry, Paul was able to tweak Tommy's avoidance algorithms to better handle low-lying obstacles like hay bales or rocks. What's probably most amazing about Tommy is it's simplicity. In an informal survey of 12 other teams, I found that on average they had 2-3 main procesors and several smaller controller-type processors. Tommy, on the other hand, uses one main processor (running on a VIA mini-ITX board) and 2 Jstamp systems to control steering and throttle/brakes. Paul's modular control system allocates decision-making and processing among these processors in order to manuever around the course. In fact, Paul started with more processors, but found that his code could be integrated onto one processor and thus eliminate a great deal of complexity which slows down development and troubleshooting. Probably the most telling reaction to this very simple system was made by members of other teams when Tommy was announced on the trials course. When the announcer noted that Tommy had just one main processor - jaws dropped around the infield.

The electronics bay


In this photo you can see the main processor behind the communication card on the 2nd shelf below the laptop. The Jstamp cards are seen to the right of the main processor (1) and on the 3rd shelf directly below (2). The laptop above does logging and handles GPS tracking. Even with a very simple system such as this, things can go wrong as we'll see below...

The bad and not-so-bad-really
Trial 1, Wednesday 9/28: in this trial Tommy got off to a good start and just nicked the gate test, but made it past the parked car obstacle, up the hill, through the speed section and returned to the berms. As noted previously, Tommy ran into a problem where the conflicting goals of staying in bounds and avoiding obstacles caused a hay bale to become stuck under the front tire. So, a good run, but not a complete run -- nothing seriously damaged and on par with most of the other competitors. Some competitors, however, did manage to finish the entire course unscathed.

Trial 2, Friday morning, 9/30: Tommy ran well and passed through the berm section without much difficulty and then successfully completed the tunnel test, power poles, and rough roads section and entered the mountain pass section where the vehicles must hug a barrier wall simulating a mountain pass road in the desert. In this section Tommy rolled to a halt, remained motionless for a few minutes, started moving and then halted again. During the trials, this behavior was demonstrated by many teams as their internal logic systems sorted through the situation the vehicle was in and tried to reach a conclusion about how to proceed. For Tommy however, this was troubling and unexpected. The team called a halt to the trial and investigated the data. It turned out that the DARPA-provided e-stop system had apparently sent spurious pause and stop signals to Tommy. This fact was noted with the judging team and the trial closed on an optimistic note - Tommy had passed the difficult part of the course and should be able to finish if the e-stop system performed correctly.

The good!
Trial 3, Sunday morning, 10/2: Tommy aced the test and passed safely and completely around the course scoring a 49 out of a possible 50 points. The tweaks made earlier in the week worked well, and the e-stop system problems were avoided. Of note, DARPA had moved some of the more challenging obstacles to the front and rearranged the course somewhat, but Tommy covered everything flawlessly. The same could not be said for Tommy's "neighbor" in the next stall. As Tommy was finishing the course, the other team's vehicle was in flames on the rough road section. Clearly Mr. Murphy was loose on the course again...

The ugly
Trial 4, Monday 10/3: The team held a practice session where all systems ran well and the team ended the session early. On the course later that day, Tommy was proceeding well when suddenly, upon entering the tunnel challenge, Tommy began accelerating and shot out of the tunnel, swerved to avoid an obstacle, and then crashed into a barrier wall at a speed of about 40 miles/hr. Race officials had sent a pause signal, but did not send a stop signal before the crash occurred. This was a major setback for the team and Tommy was hauled off of the course to determine how to proceed. Analysis revealed that the most likely cause was a component failure that sent a wide-open signal to the throttle -- none of the other system logic for handling obstacles or paying attention to speed limits on the course could get through. The net result was disheartening: the engine was compromised and needed to be replaced or significantly repaired; some body and frame work needed to be done; but the electronics systems, while knocked around a bit, continued to function and could be reused.

The team made between 100 and 150 calls looking for a new engine and by 5:30 that afternoon, Tommy was loaded on a flat-bed truck to go to a garage in a near-by town for structural repairs. The team was making a statement that this accident wouldn't stop them from pursuing the challenge - rather, it just hardened their resolve.
Tommy heading off for rework

Dateline: Ontario, California, 9/29/05

Tommy and team lead Paul Perrone


First Runs
Thursday dawned sunny and clear but hot weather was clearly on tap here in Fontana, California located well east of Los Angeles. The heat was already on Team Jefferson after their first run late Wednesday afternoon. On that run they made it about 1/2 way through the course but ran into trouble on a section called the "Berms". In this section, hay bales had been formed into a channel narrowing the course down to only 10 feet in width. The Tommy vehicle approached this section correctly, but the GPS guidance given by the race officials meant that Tommy had to strike some of the bales with its right tire in order to stay inbounds. When Tommy did this, one of the bales jumped up and got wedged under the vehicle causing Tommy to halt. The team agreed to stop at this point and the first trial was over.

This was not an unusual result on the trials course. On day one, only a handful of vehicles made it through the course without one stumble or another. The tests that DARPA had developed were working -- and the "berms" section collected the most victims. I did an informal survey of the teams and about 4 out of 5 had trouble in the berms and got stuck there.

Paul Perrone, Team Jefferson's leader, was up at 6 AM on Thursday reviewing data from the run and evaluating what steps to take next. He and the team decided that three things could be changed: 1) a new "cow-catcher" bar for the front wheel would be installed to help push aside low-lying obstacles; 2) Java code used to make decisions about when to stay away from obstacles to the side was made more sensitive (weighted higher); and 3) a roof-mounted laser radar rangefinder (LIDAR) was scheduled to be turned on.

Paul worked on the code all morning in the RV while the mechanical team worked to modify and install the cow-catcher. Additional checks were made on all of the systems as well. Changing the code was not without its risks. Paul's Java-based guidance model is based on trying to prioritize problems that may come up and how to handle them. Increasing the importance of a given sensor or increasing it's sensitivity/response can easily cause instabilities within this complex system. The tuning required was evidenced by one team whose car entered the "tunnel" portion of the trials course and slowly but surely maneuvered itself perpendicular within the tunnel -- the side obstacle detection for that vehicle overwhelmed other rules about how to proceed without a GPS signal.

At 1:30 PM, Tommy was escorted over to practice corral 3 for a 1 hour test run. The corral was constructed out of "jersey barriers" - the concrete barriers often seen on highways. Each corral was overseen by DARPA personnel who had the E-Stop devices necessary to stop and turn off the vehicle for safety reasons. After a few successful runs, some obstacles were put in place - but Tommy started acting a bit strangely and after some analysis Paul realized that the serial port getting data from the front radar wasn't working - and Tommy was running blind. By now the test window was running out and it was back to the garage to decide how to proceed.

A visit from Mr. Murphy
In the garage, the team discovered that the serial card had become partially unseated. It had been bolted down, but the restraining arm was just a bit too high and the board eventually lost contact. Something that had never happened before - just like the external ethernet ports that no longer worked. Some of it may have been due to the heat, some to the dust swirling around in the Santa Anna winds. Some just to the difficulty of getting a lot of diverse equipment to work well together at a point in time during a competition. And, of course, due must be given to Mr. Murphy who ensured that Team Jefferson - as well as every other team here - was kept on their toes.

Another example of Murphy in action was a DC voltage converter that powered some of Tommy's steering motors. This was a ruggedized unit that is typically used to power the large stereo systems found in custom cars. However, on the first day of the trials - it failed -- something it had never done before. Paul had planned to use a sledgehammer to tune it up a bit, but Bo (the electronics lead) suggested that maybe there was a good reason why it failed. After a discussion with a fellow competitor, Bo discovered that there is a thermistor in the unit that shuts down the power conversion at 90 degrees F. Well, in the sunshine, the ambient temperature today was north of 92 degrees F. Therefore, no DC conversion. After a call to the manufacturer, Bo found that if you snipped the lead to the thermistor, the unit would not shut down. Problem solved - but can that converter be trusted?

Luckily, the one thing that was working well was Paul's Java code that manages Tommy on the course. I'll talk more about that in the next entry.

As promised, I wanted to catch you up on the experiences of an autonomous vehicle named Tommy which comes from the same country as our second president, Thomas Jefferson - thus the name.

We're going to do this a bit backwards; given the timeliness of the trials happening right now, I want to catch you up on a particularly interesting use of embedded and real-time Java. Once we get past the race, we'll dive a bit deeper into the technology. So, off to the races...

The Trials course - "power poles" and "rough road" sections.


The Darpa Grand Challenge
The Darpa Grand Challenge is a series of trials and a race for which the winner will recieve $2M cash for accomplishing something never done before: send an autonomous ground vehicle across 150 miles of open desert in under 10 hours. The course is laid out in general terms using GPS technology, but the vehicle must avoid and/or cross any obstacles that it encounters while trying to achieve it's objectives. The last challenge, held in 2003, saw several entrants fail to cross the start line, and the most accomplished entry made it only 7.5 miles.

For 2005, DARPA made the challenge more rigorous. They opened the challenge again and received 195 requests for applications. Only 137 teams actually completed the entries, and of these 118 were chosen for a site visit where a DARPA representative would come out to see the vehicle put through its paces. From this group, DARPA invited 43 teams to the trials where performance on the trials course (partially shown above) would lead to 20 invitations to join the race itself in the desert. The trials course, located at the California Speedway in Fontana, has 15 unique tests designed to probe the vehicles' various obstacle avoidance and maneuverability skills. These tests range from passing through a small gate to narrow pole avoidance and the ability to pass through a tunnel with no GPS data. As proven by the trials I viewed, these tests probed critical elements of the navigational and avoidance skills of the vehicles and most teams failed some of the tests.

So what's the point of the challenge? DARPA hopes to use the technology developed here to create autonomous vehicles that can be used in life-threatening military and civilian scenarios. This is a great motivator on its own merits. However, the challenge is also fascinating because with the 43 teams out there, you have about 43 different approaches to try to solve this complex system problem - and at least one team that chose Java as it's tool of choice.

As a long-time veteran of the computer industry, one of the things that has always delighted me is to see a computer actually doing something in the physical world. I know this delights most of you as well - just go to any tradeshow and watch the queue that forms around anything that moves and is controlled by a computer. Just what is it that attracts us? Anticipation of failure? Fascination about another "intelligence" moving something in our world? The excitement that comes with remote control of something? Your mileage may vary, but probably it's a combination of flashiness and fascination with what technology can accomplish.

On the latter point, I have created this blog to talk about how Java is playing a more important role in the embedded and real-time communities. For me, it is about putting Java to work - about what Java is doing today and what it can be doing tomorrow in systems that must interact with the real world. Clearly Java has done well in the mobile space, but there's a lot more going on in spaces like the military command and control, industrial automation, telecommunications and financial infrastructure, etc. In fact, our first story will be about an autonomous vehicle named Tommy that is competing in the DARPA Grand Challenge.

So, stick around and enjoy the ride. I hope to bring you to some interesting places where we're trying to put Java to work. Also be sure to let me know of any interesting places you've found where Java is working.