Arieh's Weblog

Dusting the cobwebs ...
Thanks to Ariel Hendel for providing the motivation to dust the cobwebs of the blog.

Half a year has passed since our move back and things have pretty much been peachy. The kids, each one adapting to their new environment, making new friends, adjusting and for the most, liking it and not complaining too much.

Workwise, the 10 and 9 hour timezone difference relative to pacific and mountain timezones presents a challenge that all who work in this geographically distributed mode of operation can relate.

After a brief trip to the Bay Area two weeks ago, I came back to my new office in the 8th floor at the Sun Israel Development Center in Herzliya. I am glad we left the Ramat Gan offices. While the distance from home is much shorter, traffic is heavier and the commute time not that much better.

I lucked out to have a windowed office facing the mediterranean, with a view to the marina, able to see sailboats and sailboards getting in and out (pix to come later).

Hannukah is here, and we lit the 3rd candle yesterday at some friends' who returned from a sabbatical in Boulder, CO.

Happy holidays (Hannukah, Christmas, Kwanzaa, whatever you celebrate) !!!

And thanks again to Ariel for the nudge ...

Moving back after 17 years ...
After 17 years abroad, we have been back in Israel since mid June. We live in Ra'anana, a suburb of in the northeast section of the greater Tel Aviv area.

The initial two weeks were packed full of logistics and bureaucracy.

We were fortunate enough that we managed to rent an apartment remotely (long distance conversations at odd hours of the day, digital pictures being sent over email, etc), with the kind help of our friends and my folks, who came to check it out and sign the (interim) lease at the time.

We are also lucky that a lot of our friends live in and around Ra'anana.

There was a lot of paperwork to fill, and government offices to visit: establish official residence status, 'returning citizen status', social security, banks, health insurance, register the kids for the next school year, and more.

I can relate to what Jim Grisanzio mentions in his blog about the search for an apartment in Japan, a day after we arrived we went and bought all the appliances (stove, fridge, microwave, tv, washer, dryer). Prices were higher than in the US, but not terribly. We also had to buy wall closets (we will take them with us when we move from here, hopefully to a place we'll own), as well as a bedroom set for us and a bed for our older daughter.

The shipment of what we sent from the US arrived at about the same time we did, but it took a week to deal with the paperwork for releasing from the bonded warehouse (need the certificate of 'returning citizen' to reduce the amount of taxes to be paid) and having it delivered here. As apartments here are much smaller than the house we lived in Colorado, we had to get rid of a lot of our posessions prior to the move, and still, we have a ton of stuff we shipped that we will be donating and/or hand down to friends and relatives.

I have been working in the Sun offices in Ramat Gan, site of the Sun Update Connection team. I met the team on my visit back in January, when a group of us came for the technical due diligence for the Aduva acquisition. I am now working with that team and enjoying every moment of it.

We also purchased a (used) car - an 2002 Opel Astra. That experience was quite different than the way one purchases a car in the US. The shopping, the negotiation, financing, registration. In general, prices are much higher (at least twice) than what a comparable would be in the US (the government here taxes automobiles at about 100%), and that on top of the difference in gas prices.

Just as we began to get into our daily routines, the conflict in the north flared up. While we are not within range of the rockets that have left more than 500000 israelis living in bomb shelters (that is until the Hizballah decides to escalate the war by firing longer range ones), the impact is there, as everybody has relatives/friends who either live in the north, or have a son/daughter in the army serving (or about to begin serving).

While not the best way, this is one hurried-up way of getting integrated into the common national experience.

I hope this posting is a kick in my pants to continue blogging, as I have neglected my blog during the past year.

Susie Marks in the Sun office here has been nagging me about resuming my blogging. There you have it, Susie ..., how about you pick up the gauntlet and begin your own blog too ?

Blog Theme Change, and other upcoming changes
After almost a year and a half without posting, I decided to renew this activity.

This was also a good opportunity to change the blog theme.

And talking about changes ...

I am about to move - about 8000 mi. away -, continuing with my job (thanks Sun for the flexibility provided), at a time-distance of 9 timezones.

My work in the Software organization in Mike Harding's group has been very satisfying and I have had the privilege of working with an incredible set of colleagues. I am sure we will put that to the test with this upcoming long-distance move in the context of logistics and working hours.

To all those in the Broomfield campus I have worked with, I wish you continued success. We'll keep in touch across the oceans. Email address does not change.

Cheers.

Arieh

Monitoring large numbers of files, /dev/poll and others ...

I am in the process of investigating the ability to monitor large numbers of files in a system.

A lot of material has been written about this topic here (Kegel), here (Erenkranz), as well as BSD's Kqueue.

Most of the discussion on those centers on monitoring sockets, pipes, streams, etc.

In that context, there is an open source project File Alteration Monitor that tries to deal with the topic of monitoring files and directories.

I begun my investigation with using the /dev/poll device.

In general, the idea is to open the files being monitored and register their file descriptors with the /dev/poll device. What I wanted is that if any of those files are written to, that the DP_POLL will 'fire' and I will get the event that the file has indeed been written to.

I have not been able to observe the expected behavior. Instead, it seems as 'regular' files operate with poll(7d) in a manner similar to what poll(2) state, namely, that regular files trigger for read, write and error always, therefore not allowing to trigger on writes to the file, for example.

Having read Bart Smaalders blog article on event ports I decided to try using that new feature of Solaris 10.

In my program, I proceeded to open the files I wanted to monitor, get the file descriptors and associate them with the port that I created. I was unable to associate the 'regular' file with the port, with the library call returning a EBADFD for the file descriptor.

I am wondering if I am doing something wrong (probably) or that what I want to do is not possible with the /dev/poll device or the event ports subsystem.

Later at night, from home, I tried to use the kqueue functionality available on BSD systems. I was successful in doing so, getting events on file appends, truncates, etc.

I liked the flexibility that kqueue provides and wonder whether there is similar richness of functionality on event ports or /dev/poll.

They say change is good ...

So, they say change is good ..., and I will now be able to gather one more data point to prove that statement right, or wrong.

I have officially left Network Storage after being there from the time before it was even called Network Storage.

I am now part of the Software organization. The topics with which I am involved are related to what I was doing in my previous job (distributed systems, fault management, technology investigations, advanced development). I am still interacting with many of the colleagues I used to work with, but I am now getting exposed to other areas with which I was not familiar before.

As time progresses, I hope I will be able to share with you more details related to the work I will be involved in.

I am quite enthusiastic about the position and the contributions that I can make to the organization.

Software Engineering - Craft, Art

As time passes by, the experiences of the job (software development, architecture, hacking, learning) continue strengthening the image of software development being part craft, part art.
(I would not want to omit Don Knuth's Programming as Art Turing Award lecture.)

As good craftsmen, we are expected to have a toolbox, on which we can find multiple tools (the set of languages, tools, methodologies, etc.) that we have accumulated over time, from which to choose the appropriate one for the task at hand.

Keeping up-to-date with the technologies is one of the defining factors to keep that toolbox useful, and ourselves relevant.

The context of art is manifested - as mentioned in the reference above - in the precision and elegance of fitting these (the well known tools and techniques - the craft) to the problem domain ....

With the incredible proliferation of new technologies, we find ourselves faced with trying to assimilate a large (and growing) amount of resources from which we would benefit of their addition to that (virtual) toolbox.

We therefore, end up along a specialization tack that potentially limits our horizons.We should be careful to keep ourselves open, and track developments in directions/areas that may not seem obvious.

Learning how to utilize those tools/resources from our toolbox for their own sake is good, but putting them to use in the context of the problem we are trying to resolve should be the scenario under which we are expanding our craft.

As technology is not going to stop evolving any time soon, my best advice to all of us is to continue improving ourselves as craftsmen, while at the same time doing what we can to build those pieces of art that are the software that we create.

Plaintext Diff Change Bars Utility

Yesterday I had to publish an updated plaintext document for review by a technical group that I am part of.

I wanted to provide the latest revision of the document in a manner that the difference (changebars) from the previously published version would be marked, thus facilitating the ability to email the updated documents, with the changed marks. (No, I did not want to publish side-by-side comparisons).

I was surprised when my Google search for "plain text diff change bars" did not come up with a solution to the problem. (Perhaps I erred in what I searched for.)

I decided to come up with a solution for that problem (which I will probably have to turn to in the future). Below you will find the perl program I hacked and provided satisfactory results.

The crux of program is to:

• generate a diff output (sccs diffs)
• traverse the output of diff, operating on the later revisioned file
• generate changebar-ed output

The code follows below. Apologies to perl-haters and perl-purists for the crude hack.

#!/usr/bin/perl -w
#
#  %W% %E%
#
#  chbars:	 generate change bars based on 'sccs diffs'
#
#		 The program will output the document whose name is passed
#		 with change bars in all locations different from the
#		 previous version (or the version specified).
#
#  	chbars [-r{rev}] path
#
use FileHandle;
use Getopt::Long;

use vars qw/ $opt_r $opt_verbose $opt_help $SCCSDIFF_PROG /;

#
#  Usage: display Usage
#
sub usage {

    print <<"E-O-F";
$progname: INFO: Usage: $basename [options] file
    where options are:
	  -r{rev}	the SCCS revision with which we compare
    and
	file 		the file to compare with

E-O-F
    exit;
}

#
#  do verbose printing
#
sub do_verbose {
    return unless $opt_verbose;

    print STDOUT $basename, ": ", "@_\n";
}

#
#  do error printing
#
sub do_error {
    print STDERR $basename, ": ERROR: ", "@_\n";
}

#
#  doSCCSDiff - generate the differences between the two versions of the file 
#
#  Arguments:	file		pathname of file
#
#  Returns:	outputs the file with its change bars
#
sub doSCCSDiff {
    my ( $file ) = @_ if @_;

    &do_verbose ("doSCCSDiff ( path=$file )");

    #  if the argument is not a file, issue an error and return
    #
    if (! -f $file) {
	&do_error ("$file is not a file");
	return;
    }

    &do_verbose ( "$SCCSDIFF_PROG $file");

    my $PATH = new FileHandle "$file", "r";
    die "$basename: open error on file \"$file\": $!" 
					    unless defined $PATH;

    my $DIF= new FileHandle "$SCCSDIFF_PROG $file |";
	die "$basename: open error on $SCCSDIFF_PROG invocation: $!" 
					    unless defined $DIF;

    #  the type of lines 'sccs diffs' outputs is like:
    #
    #  8,9c8,10
    #  221c222
    #
    doDiff( $PATH, $DIF );
}

sub doDiff {

    my ( $file, $dif ) = @_ if @_;

    my $line = 1;

    while( <$dif> )
    {
	if ( /^(\d+|\d+,\d+)(a|c)/o )
	{
	    my @args = split ",", $'; 

	    my $fdifline = $args[0];
	    my $ldifline = $fdifline;
	    $ldifline = $args[1] if ( $args[1] );

	    @args = split ",", $1;

	    my $ofdifline = $args[0];
	    my $oldifline = $ofdifline;
	    $oldifline = $args[1] if ( $args[1] );

	    my $buf;
	    my $dline;

	    # position the $file cursor on the line indicated by fdifline
	    #
	    for ( my $i = $line; $i < $fdifline; $i++ ) {
		$buf = <$file>;
		print "  $buf";
	    }

	    #  output the lines in $file between fdifline to ldifline
	    #
	    for ( my $i = $fdifline; $i <= $ldifline; $i++ ) {
		$buf = <$file>;
		print "| $buf";
	    }

	    $line = $ldifline+1;
	    
	    my $diflines = $ldifline-$fdifline+$oldifline-$ofdifline+1;
	    for ( my $i = 0; $i < $diflines; $i++ ) {
		$dline = <$dif>;
	    }
	}
    }
#
#  MAIN program - variable initialization
#
    $opt_help	= '';
    $opt_verbose= '';
    $opt_r      = '';
    my $result	= '';
    $result = GetOptions( 'r=s', 'verbose', 'help' );

    $progname   = $0;
    $basename	= substr ($0, rindex ($0, '/') + 1);

    $SCCSDIFF_PROG = 'sccs diffs ';
    $SCCSDIFF_PROG .= " -r$opt_r" if ( $opt_r );

#
#  MAIN program - body
#
MAIN: {

    @files = @ARGV;

    #
    #  show the usage if -help passed
    #
    &usage if ($opt_help);

    #
    #  issue usage message if no argument was passed
    #
    &usage unless (@files);

    #
    #  invoke the program on all the arguments passed
    #
    for $arg (@files) {

	&doSCCSDiff( $arg );
    }
}

I am sure I can improve the regular expression on matching the diff-description lines, and the subsequent parsing of the old-file lines and the new-file lines. Possible improvements also include operation on diff/gdiff file comparisons.

Enjoy, provide comments, reuse, modify, whatever ...

Network Advertisements and Lookup Technologies

In the context of the activities I am involved at work, I am researching network registration and lookup services, primarily from the perspective of systems, network and storage administration.

The goal is to be able to identify the resources involved (be them software entities, hardware components) in order to obtain their Service Access Point, (SAP) and utilize them within the application.

The Network Storage world, through its SMI-S initiative, has declared SLP (Service Location Protocol) as the means by which advertisement of the points of presence (CIM Object Managers) is done.

There are limitations to using SLP in that context, some related to the ability to perform multicast and traversal across subnets. Others, to the fact that a dependency exists on the availability of an SLP Daemon on the network. One last limitation is related to the CIM/WBEM implementation, which refers to the SAP being the whole CIMOM.

Jini provides an alternative solution that addresses several of the shortcomings that I can see in the SLP implementation selection. Past limitations of multicast have been removed with the Lincoln Tunnel delivered within the RIO project. One could consider the Java-bound nature of Jini a limitation. Such limitation may be manifested when needing to satisfy a requirement such as the ability to allow components that cannot support a JVM (such as devices with limited capabilities).

JXTA appears to be a technology that can help address the problems I need to tackle. JXTA provides non-Java bindings which would allow the development of JXTA peers embedded in Real Time Operating Systems (where J2SE and even J2ME are not available). JXTA provides also for the ability to not be limited to TCP/IP connectivity (a need that is featured in the storage world, with parallel SCSI, Fibre Channel as some of the non-IP based technologies), as evidenced with the connectivity to Bluetooth.

I will defer to many of the other bloggers who are much more conversant and familiar with it.

I admit that I have not mentioned Zeroconf as one of the possible solutions, mainly because of its focus on the IP Networking area. I have also omitted mention of Apple Rendezvous as a possible solution, for similar reasons.

I'd be interested to hear from your own experiences in this area.

Back from Vacation - Transitioning back to our routine ...

After a great three weeks, I am back at the office. In fact, I landed on Monday at 10:30 and dropped by the office at around 12:30. Mostly, to catch up on the mail that I did not read during the past three weeks.

An interesting observation that this brought up is the speed with which we are able to "change contexts". From many perspectives, coming back after a long vacation is supposed to carry with it some period of acclimatization. Surprisingly, I feel that I engaged back into the work routine/tasks pretty quickly (probably thanks to a reduced jet-lag impact, due to an overnight layover in Toronto, Canada).

By now, I am back to normal, attending the regularly scheduled meetings, carrying on my tasks, and fully engaged.

In general, the impact of the visit, meeting friends, experiencing the changes after a 5 year absence, spending time with parents, relatives, had an invigorating effect, and in many respects helps put things back into a different/more appropriate perspective.

Technology pervasiveness

After two weeks on vacation in Israel, I am beginning to appreciate several differences in the level of acceptance of technology, as it is used in everyday life, and I can draw some comparisons between my everyday life in the US and what I can see here.

Computers

Everybody I have met has a computer, and uses it for more than the plain internet browsing and email reading/writing. Uses include managing digital picture albums, word processing, interactions with banks, government, etc.

In general, Microsoft is very prevalent. Part of it probably attributable to the issues of localization (hebrew having a different character set, and written right-to-left). Linux is practically unknown in the non-technical space, although I am aware of the efforts of Israeli developers in a variety of open source, linux related projects, as well as in several core-linux companies (Qlusters comes to mind).

A recognition that the computing ecosystem needs to evolve appears to be seeping in, and I have heard/read about projects of migration into Java (J2EE, J2SE) away from Microsoft proprietary technologies.

Internet Connectivity/Broadband Penetration

Practically everybody I have met/visited (friends, relatives) is connected with a broadband connection (cable, DSL). Prices appear to be reasonable. The service (I called the phone company offering DSL) was good, the tech people were very helpful and knowledgeable).

Cellphones

Everybody has a cellphone (and I mean it: kids, elderly people). A lot of competition in this area. Functionality offered at the basic level of services seems to me richer than the one I get back in the US. In general people would give you their mobile phone as their primary number ahead of their non-mobile phone.

Will try to share a bit more in a later note. Let me know if you have had similar experiences.

Live, from Jerusalem

Now writing from the Holy City. Visiting friends, and back from a walk on the promenade overlooking the Old City. In general, a very special place.

Some images ...

A kids carnival where both jews and arabs share rides on bouncing houses and other amusements.

Views of the old city, surrounding arab and jewish neighborhoods ...

Visibility, quite clear, apart from some dissipating smog in the horizon (towards the east).

And amazingly, having technology so readily available to be able to pass these comments in quasi-real time.

Sun - Internationally

I visited yesterday the Sun Development Center in Herzliya, Israel.

Even though at 8000+ miles from my home office, really felt at home there. Both as a Sun employee and as a (former) 'native'.

The very helpful people there set me up with access in no time.

Dov's team there has received accolades and industry awards for their work on J2ME. They continue doing great work.

I am sure that the Israel office is one representative example of how Sun operates as one - distributed, international - company (I could have resorted to the 'one happy family' cliche - but did not ;).

A side note. I am writing this from within SWAN, with no appreciable performance problems, and with complete access to my home environment. Kudos to the IT organization.

Vacations and Network Administration

So, on my first day on vacation, 9 time zones away from home (is it really home ?), writing from my parents' (in Israel).

Spent the day visiting family, and yes, fixing computers and networks for the relatives.

First, just fixing configurations of CD-ROM burner, scanner, digital camera at my father in law.

Second, creating a local home network (win XP based) at my brother in law. Downloading OpenOffice, Firefox, and extolling (to my nephew) the virtues of Linux.

Finally, setting up a wireless network at my parents', including the wireless router I brought, installing a network card, replacing the USB DSL modem with an ethernet one.

My goal was to be able to use my laptop, wirelessly. If you are reading these lines, I was successful.

My conclusion is that a bit of network administration background has been very helpful (it probably cut the time on the phone I had with the ISP tech by a significant amount - first time I dealt with DSL). I don't believe I will do any more of that from now on ...

Which brings up the question, how many of us/you engineers/tech guys end up being the family's 'computer experts' ?

What's so great about blogs.sun.com

Apart from the advantages of exposing a more personal face to the rest of the world, blogs.sun.com has facilitated better/more communication between Sun employees.

By creating such a disintermediated medium, it becomes much easier to obtain information about ongoing projects that we may be able to leverage, to help solve problems that some us may be having, create a more cohesive group transcending the organizational hierarchies under which we operate daily.

Not sure if that was one of the intended goals, but I can see its benefits already.

More on CLIs

An interesting addition within the work we did with the embedded CLI stems from the fact that apart from the application embedding its own shell interpreter, it also embeds its presentation (web-based) GUI. (This was accomplished by embedding the Jakarta-Tomcat servlet engine).

The presentation code was designed and implemented to - after all the GUI manipulation, wizard interaction, etc. are done - boil down to the generation of a command in canonical form.

The CLI functional equivalent was chosen to be that canonical form (in essence the argv list for the CLI command). In that way we were able to implement GUI Capture for later replay in a reasonably easy manner, and comply with directives such as anything done on the GUI should be able to be done in a CLI.

The user is able to toggle GUI capture, and at his own discretion, the user is able to dump the accumulated activity to a file. Such file can later be executed as a script or via standard input redirection via the Java shell.