Arieh's Weblog

CLIs in Java - raw I/O - redux

Paul states:

What seems to still be missing is the ability to put the invoking terminal in raw mode. What I'd like to do for instance is to write a java clone of 'bash'. I'd like to be able to do something like this:

     % java -jar JavaBash.jar;
     JavaBash> echo hello world;
     hello world
     JavaBash> 

Where JavaBash allows cmd line editing. With this telnetD package, unless I am wrong, you would have to do something like this:

     % java -jar JavaBash.jar -port 2222& 
     % telnet localhost 2222; 
     JavaBash> echo Hello world;

It's not quite the same.

The solution, as I implemented it, was to create a 'tunnel' client, that would be responsible to set the terminal in raw I/O mode. Here is one example (not the same that I used in my previous implementation) to have a local client that can talk to the shell program that would be created.

So, yes, Java does not support it natively, but workarounds are available until they are available.

PS: A possible kludge could be for your Java jar to fork a Thread to Runtime.exec() a telnet invocation to localhost on an ephemeral port on which the other Thread (the Shell) will be listening.

More on Sun Ray usage in Engineering Environments

As a follow-on to this article about Sun Rays in technical environments, I will chime in with some of my own experience.

Many of the points Moazam makes in his article hold true for the Sun Ray systems our department uses.

About 2 years ago, our group (within the changes on campus) was asked to migrate to Sun Rays.

Our Sun Ray servers have been configured to include all of the additional stuff that comes with Solaris (/opt/sfw/*), as well NFS-mounting the development environment for our projects (which includes the JDK, the additional Java members of the 'ecosystem' that we require, as well as other software).

As expected, initially, not many developers jumped to the fray (you'll have to pry my workstation from my dead fingers ;). I was probably one of the only ones who did. After two years of having made the jump, I have gotten to appreciate the advantages of relying on a powerful multi-CPU/multi-user enterprise class server.

The machine is more powerful than the workstation I have (I still have it, but I use it as a crash machine, and it is not bothered with the overhead of hosting my window managers, browser, mail, StarOffice, etc). As a matter of fact, since the Sun Ray server is used by many colleagues, the startup time of the various application(s) is reduced.

I will add that I only have one display (I don't believe current Sun Rays support multiple displays), so that is not a problem for me..

As an aside, I will add that apart from the Sun Ray, I have my own SPARC workstation, an Intel box (running linux), a laptop (dual bootable into linux and Win2k). All four are connected to Belkin Omniview F1D104-USB KVM switch. A pretty convenient setup.

More on Open Source and how to speed up progress

The postings about Java CLI implementations in Java, this one and this one hint again at the strength of leveraging Open Source for the tasks at hand.

As it was stated, support for raw I/O in Java has been waiting for 7 years to be incorporated into the standard libraries. In the project where the Java CLI was implemented, I was not about to wait until the JDK included the functionality required. Instead I went about trying to find the resources on the net that would provide me (as close to) what I needed.

Clay Shirky (RSS Feed) writes here about the motivation of open source projects, which probably illustrates the point of why Java has taken (so far) more than 7 years to deliver what a developer scratching his own itch delivered in far less time (admittedly, perhaps with less than the 100% functionality than the community consensus would have required, yet good enough for most practical uses).

Further, the accessibility of the code allowed me to perform extensions of the classes in the telnet project, reorganizing those subclasses into packages in a manner that there is a cleaner separation between the telnet protocol handling and the Java pseudo-tty code.

The recurring theme is that in these days of technological openness we as developers need to be able to identify where the potential resources to draw from are, and have the Laziness, Impatience and Hubris that Larry Wall talks about.

If we don't we will be waiting for another seven years.

CLIs in Java - Totally doable

I had been waiting for a post like this to expose my own biases with regards to CLIs and Java.

Paul states In Java, it is not possible to perform raw console I/O, only line buffered I/O is possible.

Well, I would begin by stating that the above is quite inaccurate. As a matter of fact, I have implemented, and delivered within the Sun StorEdge Configuration Service such Java-written CLI.

How it was done

The Java program exposes a live interpreter that can be accessible from a telnet session (if so configured), as well as from an ssh session. In essence allowing the users of the application to rely on a fully functional (for both one-shot, scripted, and interactive) embedded shell that is portable and does not rely on any of the hosts' shells.

The CLI provides for sufficient tty capabilities (albeit limited to only 4 types of terminal definitions), including implementation of raw character-at-a-time I/O, which allows to implement features like up-arrow/down-arrow for command line recalls, noecho on password prompting, and many of the features one would expect in a shell accessed via telnet or ssh.

Apart from the "virtual tty" that it supports, the implemented shell also provides for command line history recall, environment variables, etc.

Technologies to Rely on

The implementation takes advantage of the Java Telnet Daemon project.

Another possible implementation can be found within the Jakarta Commons-Net project. However, I did not use it since Daniel Savarese had not made it available under the Apache license at the time.

Conclusion

My experience has shown that what was originally claimed as undoable, is in fact, very doable. In fact because of that feasibility, I argue that embeddability of a functional shell in an application provides the developer of the application with a good measure of independence, and allows to treat the CLI as just another public interface.

The potential hosting of the CLI within the application also reduces the complexity that would have been introduced if they (the CLI and the application) were implemented as different processes.

Passports - Epilogue

So, I mentioned in a previous post the trouble we have gotten into because of not keeping our passports up to date.

Today, a week later, and a significant chunk of change lighter, I picked up the passports that the expediting service delivered.

It is worthwhile to note, that even though we paid for the super-duper expediting - hoping that the passports will be back in two days, the State Department did not agree to deliver/process the passports that quickly, because, in their opinion the date of the request was not close enough to the date of departure.

Anyway, as they say, all's well that ends well (hopefully ;).

The Many Hats of Sun Employees

I don't know if other companies the size of Sun behave/operate/are organized (or is it disorganized) the way Sun does, but I believe that one of the things that makes Sun unique is that Sun employees usually wear many hats.

I will tell you a bit from my own experience.

First and foremost I am a software engineer (actually, at the time I finished college, I am not sure Software Engineering had been invented, I graduated with a Math and Comp.Sci degree). My focus is on Storage primarily on management and RASD (Reliability, Availability, Serviceability, Diagnosability - currently through the StorADE product). I have mentioned before in a previous posting my participation in the Sun StorEdge Component Manager project, as well as in its follow on, Sun StorEdge Configuration Service.

I have been a member of one of those things called ARCs (Architecture Review Committees) which Torrey mentions in one of his articles. One interesting thing about the composition of those committees is that it is made up of engineers that still have responsibility for delivering products, a quality that makes the decisions taken in the committee grounded in reality and not part of an academic elite that may be detached from reality.

Another hat that I wear is that of maintainer of the internal Java Package Names Registry. The job consists of managing the namespace and steering the project teams to select Java Package Names that are appropriate (i.e. do not denote project names, but instead concentrate on the functional aspect that the packages provide).

Over the past year. I have become involved in contributing to the storage aspect of the architectural work related to the N1 System. That activity takes place through various forums that meet regularly to discuss the architecture, design, implementation, technologies. That has put me in contact with people of different organizations in Sun across all the business units, worldwide.

The multi-location, international character that is an inherent part of Sun imposes quite a number of challenges. Imagine trying to schedule a meeting where the participants are scattered across 18 time zones. I am sure that there is blog fodder in that topic, perhaps for another post.

In any case, the work itself is hectic but very fulfilling and challenging.

The many-hats quality that I have tried to describe, is definitely more the norm than the exception here at Sun, as I can see from the colleagues I interact with.

I hope you got a feel for what the many hats of the Sun employee feel like.

Feel free to share similar experiences of your own. .

I am getting old - first time reading glasses

For the first time in my life, I began wearing prescription reading glasses. I have good vision for distance, but I began to realize that that my arms are not long enough to bring the reading material into focus in front of me. I guess it may be a sign that time does not pass in vain (literal translation of the spanish el tiempo no pasa en vano; could not find an english parallel).

I picked a pair of Silhouette 7395 rimless glasses at the recommendation of my wife, who has been wearing them for the past couple of years. They are really light and you practically do not feel they are on.

Now my family and friends say that the glasses make me "look intelligent". At least I can now begin looking the part.

As a new user/wearer of glasses, the first thing that I realized is the change of focus/magnification that I feel whenever I put them on, and take them off, which causes a loss of focus at distances where I can usually see OK without them.

Well, nothing much I can do for now (perhaps next year get progressive, permanent glasses; yeah, the limitations of the VSP plan), except get used to them, and recognize that this is a step in gaining maturity ;).

Versioning in Java Packages

A technical discussion in an internal forum here at Sun and a subsequent conversation with a colleague are the motivation for this post.

The issue in question relates to the determination of versions (at run-time) for Java code. The initial point of discussion was the seeming lack of established convention for the versioned names of jar files. The analogy brought up at the time related to the conventions that exist for naming of .so shared objects in Solaris and Linux (which btw, are different).

Personally, my perspective gravitated more towards the ability to determine versions, but not of the 'jar' file, as much as to determine the version of a specific Java class at run-time, for the purpose of compatibility check, and perhaps even conditional execution.

Work I did back in 1998/1999 within the StorEdge Component Manager project addressed that issue.

The idea is based on the Jar Manifest section in the Java tutorial.
The Java Versioning Specification is available here.

The solution is based on the existence of a Manifest File that is included in the jar file. One thing worthwhile noting is that there may be more than one manifest entry

    Name: com/mycompany/mypackage/
    Specification-Title: My Package Specification 
    Specification-Version: 1.3.0.0
    Specification-Vendor: My Own Company, Inc. 
    Implementation-Title: com.mycompany.mypackage
    Implementation-Version: 1.1.0.1 
    Implementation-Vendor: My Own Company. Inc.

in the jar, thus allowing to specifically version every separate package.

Taking this a bit further, we proceeded to include on every package a Version class that extends from a super-class that defines methods allowing to perform comparisons between versions, to determine equality, earlier-than, later-than, parsing version to extract major/minor revision information, etc.

The common-build-enviroment used provided for automation of the creation of the Version classes, driven off external files where the versioning information appeared as metadata. Further utilities to generate the MANIFEST.MF file and its incorporation into the jar-file were developed.

The solution presented provided a reasonable implementation of what the specification stated. Establishment of the infrastructure and methodology of application was key in effectively using it.

It will be interesting to hear from those of you who may have run into similar issues and to share your own experiences and solutions.

Posting into the future ...

Having seen that the message for which I changed the date of publishing into the future has finally appeared, the mistery of where/if the posting had disappeared has been answered.

What I can conclude is that this feature allows to do something akin to what electronic card web services offer, to schedule a post to take place at a certain date.

One problem that I believe should be resolved, is that the 'in-the-future' post is not visible to the blog owner until that day either, which prevents him/her from modifying/canceling it.

When looking at the post information, two dates appear there, the date of writing, and the date of posting. It appears that 'roller' filters the message to be accessible (to both owner and readers) based on posting date. I am wondering if the author/owner date selection should be based on the writing date.

Any other interesting features that you may have found in roller ?

You've got to love open-source

As I mentioned before, I am working on trying to get JMX and Web Services to interact.

I am working with the Tomcat Servlet Engine and using it in several ways (embedded, inside of a service container, coupled with a stand-alone JMX agent). While debugging my test program, I find that my query trying to enumerate the Connector objects (which are necessary for the generation of the URL from which the Web Services are to be accessed) returns an empty list.

In olden times we would have no recourse, but wait for the vendor to acknowledge the problem, and fix it (as long as within his priority list).

These days, there are definitely better options. Jakarta-Tomcat, is an open source project under the Apache umbrella.

No sooner I am able to download the source code, that a brief investigation provides the clues for the manifested problem.

Using the Bugzilla bug tracking system that Jakarta relies on, I was able to immediately report the problem, submit a proposal for how to fix it, identify the developer assigned (who, in fact is a colleague here at Sun).

I will add that I implemented my own version of the change in my copy of the source tree, and was able to continue my work in short order.

So, yes, open source, long-distance, distributed collaboration works.

Development Processes for Component-based Architectures

In my position(s) at Sun I have been involved in the design and development of software for systems management (primarily storage, although not restricted to it).

In general, the nature of these systems has been that they are distributed, that many different projects end up contributing components that yield a composed system that changes over time with the addition of new components (for example, support for new devices, or additional/enhanced functionality added).

During the development process, I have found that many problems arise due to unexpected incompatibilities between the cooperating/collaborating/dependent pieces. Many of those are triggered by inconsistencies in API revisions, reliance on divergent versions of software used to develop, lack of communication and access to usable software units between teams contributing pieces.

I have found that once one understands the distributed/component-based nature of the software that is created and how it operates in the deployed environment, the implementation of specific development processes/practices that take that (component based) nature into consideration is bound to reduce the problems that will surface later (at the customer's site once the software is deployed), by catching those much earlier on, ending up with a better product that will satisfy customer needs and reduce maintenance costs.

I will enumerate a number of entities/processes that I have seen helping in developing for a component based architecture.

I view the establishment of a Common Build Environment (CBE) as a fundamental piece of the infrastructure to be set.

The goal of the CBE is to establish the shared infrastructure to support the process that the teams developing components will rely on, in order to:

• insure that appropriate revisions of dependent software are used (this may include the version of JDK and Java libraries that are determined to be used, as well as other 'systems-type' software)
• insure that components that have dependencies on other components have access to the latest-and-greatest (as well as to the previous) versions of the latter.
• insure that an individual developer compiles, builds, packages, regression-tests the software unit on which he/she is making changes as part of the ongoing development cycle prior to incorporation into the shared repository
• allow the development of nightly-builds (or appropriate period of choice) for each component, which coupled with validation smoke and regression tests that through their interaction with their dependencies, will allow to catch bugs early on, with reduced impact on the development.
• make sure that components that passed the validation 'smoke' and validation tests are promoted to be visible in the component repository for others to use/access.
• be able to develop integration-tests that allow to validate the system at development time to the extent possible.
• a system/process for tracking the workflow of development, in which a developer submits what I call a check-in request form through which the type of change to be made is described, annotations/approvals of design, code, integration into source trees are incorporated, as well as other types of relevant information. This workflow form (which could be as simple as a plaintext mail message that is forwarded and annotated) can also be used to indicate the correlation of the code change with a problem reported in a bugs database.

While understanding that Software Configuration Management (SCM) systems are able to address many of the above, I believe that organizational and engineering discipline are elements that even though being hard to enforce, if carried forward have the potential for enabling us to develop better software, in a more efficient and predictable manner.

We have all ran across skeptics that will argue that the apparent overhead is too costly in these times of rapid prototyping and development, we have too often seen that you can pay me now, or pay me later (more). To wit, I have seen enough cases when a developer's incorporation of just a tiny change to the source tree, ended up breaking the build and impacting the whole development team for a period of hours.

I will talk more in detail about many of the issues above in later notes.

I would like to hear from you about your thoughts on this topic.

Software Development as a 'wholistic' Activity

Over my career, I have too often seen (lived through) cases where software development is seen as an activity whose primary (and often sole) purpose is to yield a running program.

Such an approach is more often than not the cause of endless difficulties and additions of schedule, cost, effort to the project, and decreased quality perception.
Experience has led me to view the software development process as a wholistic activity.

What do I mean by that ?

The activities of development, which begin from the gathering of requirements (that hopefully ;) satisfy customers' requirements), the conceptual design, the architectural design, and go through aspects such as the actual development process: establishment of code guidelines, test plans, technology selections, common build environments, software configuration and management (scm), integrability of test organizations (early and often) in the process, workflow tracking, have to be considered all along.

Moreso, we should not believe that releasing the product to GA (general availability) is where the process ends.

The infrastructure, processes, methodologies need to be in place so that the services organization, through its call centers, field engineers, and other will be able to handle problems submitted by customers.

Given that we expect our software to evolve, readiness for upgradability (as well as downgradability) needs to be built into the product through the infrastructure and methodologies that the project team has set up.

Training and documentation, a big part in making a product successful are often considered 'after the fact', having impact in the cost, time-to-market, perception of quality that the product will receive.

At one point, the sustaining activity for the product may be transferred to organizations other than the one who may have created the original version(s). In order to facilitiate that transition, the technology selections and the documentation of the process needs to be done in a thorough manner.

At the end of the road, project teams need to understand that support goes on until EOL/EOF.

The human aspect, which may appear here last but not least, is essential. Team dynamics, the selection/emergence of leads, mentors from which more junior team members can learn are fundamental.

It is my belief that by having such a wholistic view, we will be able to create compelling products that satisfy customer needs, be proud of their quality, and do so in an efficient and cost-effective manner.

I am hopeful that others who feel as passionate as I do about software development will follow this thread.

Keep your passports up to date
We are about to travel overseas next week, and yesterday we discovered that the passports of two of our three kids would expire in the middle of the trip.

After some research of how to resolve the problem, we learned that the State Department has an expediting process (for which they charge an extra fee). However, the time limitations did not allow us to do the process through the US Postal Service.

There are agencies that deal with handling the expediting process 'hand-carried' via courier.

Not having much of a choice, we ended spending a significant amount, hopeful that the passports will be back on time.

The lesson learned is obvious (albeit costly) - Keep Your Passport Up-to-Date

Music for Shabbat Evening
I came across the following website - Kol Zimrah (Voice of Song) on which I found many Carlebach melodies I had been looking for.

What I have been doing with Java
I have been developing in Java since 1996, and doing it within Sun since 1997. Back in those times (within Encore Computer Corp. since partially acquired by Sun, hence my employment at Sun), having worked on the port of Java to the 88k platform.

I have been applying Java technology in the area of systems, network and storage management since that time.

I have been involved in CIM/WBEM technologies, having been responsible for a big part of the work for the SNIA SMI-S (Storage Management Initiative Specification) qualification of the 3310 and 3510 disk arrays.

Prior to that I have been a committer to the Jakarta-Tomcat project, having contributed in the area of SSL enabling, localization for the 3.1 code base.

I am probably one of the few people who have done Jiro implementations, and although the technology did not catch on, I certainly learned a few lessons from it.

In JavaOne a couple of years ago, I gave a presentation on the topic of 'Embedding Tomcat for Management'. In the project(s) I am currently involved I am also experimenting with embeddability of the servlet engine in its current incarnation, with the addition of the support of current technologies such as web services, JMX, and others.

I am currently investigating topics like dynamic generation of WSDL and of the associated services.