Let the Sunshine In: Software support engineer in Tokyo

I don't think there are lot of applications that writes out gethrtime() return value in trace or log files. Oracle is one of them. So, I wrote a perl to convert those values in my previous blog. There are very visible caveats.

1. It doesn't run on other platforms.
2. Run it on the same machine you took the trace.
3. Run it before you reboot. Because gethrtime() counter starts on boot time.
4. You want to run it right after taking Oracle sql trace. Because:
1. gethrtime() counter rolls over to 0 after 497 days.
2. OS timestamp will drift for several reasons. e.g.) ntp kicking in.

Let's see how the perl works on one of my SPARC server.

1. oracle sql trace output fragment today

   *** SESSION ID:(491.26184) 2009-07-23 23:25:45.437
   =====================
   PARSING IN CURSOR #1 len=109 dep=0 uid=65 oct=3 lid=65 tim=3113736335727 hv=2518377154 ad='b9ccd050'
   SELECT timestamp, id, version, changeType FROM objchange WHERE timestamp > :1  AND type=:2 ORDER BY timestamp
   END OF STMT
   *** 2009-07-23 23:25:55.456
   WAIT #0: nam='SQL*Net message from client' ela= 8 driver id=1297371904 #bytes=1 p3=0 obj#=-1 tim=3113746119816
   

   Run it throught the perl.

   $ perl ~kinoue/oracle.Trace.Tim.pl /tmp/orcl_s000_11620.trc.2
   *** SESSION ID:(491.26184) 2009-07-23 23:25:45.437
   =====================
   PARSING IN CURSOR #1 len=109 dep=0 uid=65 oct=3 lid=65 tim=2009-07-23 23:25:45.43776  hv=2518377154 ad='b9ccd050'
   SELECT timestamp, id, version, changeType FROM objchange WHERE timestamp > :1  AND type=:2 ORDER BY timestamp
   END OF STMT
   *** 2009-07-23 23:25:55.456
   WAIT #0: nam='SQL*Net message from client' ela= 8 driver id=1297371904 #bytes=1 p3=0 obj#=-1 tim=2009-07-23 23:25:55.45667
   

2. oracle sql trace output fragment from 11 days ago

   *** SESSION ID:(481.17143) 2009-07-12 04:22:39.236
   =====================
   PARSING IN CURSOR #6 len=144 dep=0 uid=65 oct=3 lid=65 tim=2118622857650 hv=41747736 ad='b9cb2e10'
   SELECT task.id, task.type, name, '', '', summary, '', '', xmlSize  FROM task WHERE task.type='TaskInstance'
   and task.attr1='READY' order by name
   

   Run it throught the perl.

   *** SESSION ID:(481.17143) 2009-07-12 04:22:39.236
   =====================
   PARSING IN CURSOR #6 len=144 dep=0 uid=65 oct=3 lid=65 tim=2009-07-12 04:22:29.23621  hv=41747736 ad='b9cb2e10'
   SELECT task.id, task.type, name, '', '', summary, '', '', xmlSize  FROM task WHERE task.type='TaskInstance'
   and task.attr1='READY' order by name
   

3. oracle sql trace output fragment from 21 days ago

   *** SESSION ID:(492.30132) 2009-07-02 14:26:52.437
   =====================
   PARSING IN CURSOR #1 len=109 dep=0 uid=65 oct=3 lid=65 tim=1310268562358 hv=2518377154 ad='b9ccd050'
   SELECT timestamp, id, version, changeType FROM objchange WHERE timestamp > :1  AND type=:2 ORDER BY timestamp
   

   Run it throught the perl.

   *** SESSION ID:(492.30132) 2009-07-02 14:26:52.437
   =====================
   PARSING IN CURSOR #1 len=109 dep=0 uid=65 oct=3 lid=65 tim=2009-07-02 14:26:34.43783  hv=2518377154 ad='b9ccd050'
   SELECT timestamp, id, version, changeType FROM objchange WHERE timestamp > :1  AND type=:2 ORDER BY timestamp
   

My machine is not set up for ntp client. Yet, OS timestamp seems to constantly drift for certain seconds. Probably, the drift is caused by machine's internal hardware clock.

ORACLE 10.2 is one of supported RDBMS for Sun Java System Identity Manager. I was comparing oracle's sql trace with Sun IdM's log and noticed sql trace doesn't show user friendly timestamp. So, I googled.

OTN Discussion Forums : Event 10046 and timestamp ...

When setting event 10046 for tuning purpose, I'd like to get the timestamp to compare with other logs (application, OS, ...)
Is it possible ?
How to convert "tim=8317892534" in readable date/time ?

Oracle 10046 tim, e and ela Values use Nanoseconds/1024 not Microseconds, on some Platforms - oracle-l - FreeLists

In recent attempts to correlate 10046 extended trace data from Oracle 10.2, with DTrace data collected under Sun Solaris, I discovered that the tim, ela and e fields in Oracle trace data are not measured in microseconds.

I got hints from above and created this quick & dirty perl. I tested this only on SPARC Solaris10. I think I can elaborate some more in my next blog entry.

#!/usr/bin/env perl
use Time::HR;
use Time::HiRes;

my $diff = Time::HiRes::time() - gethrtime()/1000000000;

line: while (<>) {
    if (m/^([EFPW].*tim=)([0-9]*)(.*)$/o) {
        my $seconds = $2*1024/1000000000 + $diff;
        my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($seconds);
        my $subsec = substr($seconds, index($seconds, '.'));
        printf ("%s%4d-%02d-%02d %02d:%02d:%02d%s %s\n",$1,$year+1900,$mon+1,$mday,$hour,$min,$sec,$subsec,$3);
        next line;
    }
    else {
      print $_;
      next line;
    }
}

How many times have you sent e-mail prematurely on Thunderbird? It's got "Ctrl+Return" shortcut key assigned for "Send Now". It will pop up confirmation dialogue so people like me don't send empty mail. Good... But..., in the dialogue, the default focus is on "Send" button. So, careless people like me will hit 'Return' or 'Space' and send an empty mail anyway.

I tend to use Control key more than an average guy so "Ctrl+Return Return" sequence happens often. So, I decided to change Thunderbird behaviour. I thought I might need to edit Thunderbird binary file. But a little googling found this plugin.

Keyconfig extension - MozillaZine Knowledge Base

The Keyconfig extension (available here) allows you to configure keyboard shortcuts. It works with Firefox, Thunderbird, and (probably) Mozilla Suite/SeaMonkey.

Here's how I downloaded this plugin. I did this because somehow my Firefox couldn't handle this URL well.

curl http://mozilla.dorando.at/keyconfig.xpi > /tmp/keyconfig.xpi

This plugin will add GUI interface for key shortcut config. I just removed the "Ctrl+Return" shortcut. This added one line to prefs.js file under Profiles folder.

user_pref("keyconfig.main.key_send", "!][][");

Now, I'm happy I'm empty e-mail free :)

I've been using this for 2 months for day-to-day work, both on IE and Firefox. This works pretty reliably.

iMacros for Firefox - Script your Firefox Web Browser

Whatever you do with Firefox, iMacros can automate it.
iMacros for Firefox is 100% free for private and commercial use.

This is a sample which toggles Sun Java System Identity Manager 7.1's trace flag. iMacros has recording capability so all but 2 lines were automatically generated.

VERSION BUILD=6210326 RECORDER=FX
TAB T=1
URL GOTO=http://idmserver:8080/idm/login.jsp?nextPage=debug%2Fsession.jsp
' These 2 lines were added by hand becase iMacros didn't record these.
TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:mainform ATTR=NAME:accountId CONTENT=configurator
TAG POS=1 TYPE=INPUT:PASSWORD FORM=NAME:mainform ATTR=NAME:password CONTENT=configurator
TAG POS=1 TYPE=INPUT:BUTTON FORM=NAME:mainform ATTR=VALUE:Login
TAG POS=1 TYPE=INPUT:BUTTON FORM=ACTION:http://idmserver:8080/idm/debug/session.jsp ATTR=NAME:Show_Trace&&VALUE:ShowTrace
TAG POS=1 TYPE=INPUT:CHECKBOX FORM=NAME:formname ATTR=NAME:isLogging CONTENT=YES
TAG POS=1 TYPE=INPUT:BUTTON FORM=NAME:formname ATTR=NAME:Save&&VALUE:Save

I had been using Adobe Reader 8 to read and print out PDF files. I usually print 2 pages per sheet and I want to use all the paper real estate to make it readable. I want NO white space on paper. Usually, "Fit to Printable Area" option works OK. But for PDF files I downloaded today(actually, one of our product document), this didn't work. About 50% of the papar was blank.

So, I googled and learned that free Adobe Reader *might be* crippled in print out capability.
Print a pdf at larger than intended size? - Scot's Newsletter Forums

Without buying the full Adobe Professional version, I don't know any "hack" methods around that.

I googled further and found free PDF viewer which gives users free hand on print out scaling.
PDF Viewer - Software To Open A PDF File - PDF Editing Software

The FREE PDF-XChange Viewer offers all the feature's you would expect such as support of all PDF file formats from Version 1.0 all the way to the latest 1.7 (Acrobat 8)

Last but most important, you may accuse me of printing out PDF to paper in the first place. Yeah, I know. There is  AmazonKindle. But I doubt this is worth the money outside of US. There are other similar devices from SONY and one from UK company and another from German company.

I recently downloaded bunch of StarOffice presentations. I view these on 2nd display and I want them to autoplay. These are XML files. So, I thought I could write XSLT but I'm not good at it. Instead, I thought I could use xml diff tool to apply the change to one file to all other files in batch.



Here are the xml diff/patch tools I tried. The bottom line is: I couldn't make none of these achieve what I want.

1. xmldiff/xmlrev
   

   a python tool that figures out the differences between two similar XML files
   

   It turned out that xmlrev hasn't implemented 'INSERT' yet.
2. Microsoft XmlDiff
   patch tool checks checksum of original file. So, the patch can be applied to only 1 file.
3. diffxml - XML Diff and Patch Utilities
   

   This project aims to provide Open Source XML diff and patch utilities which operate on the hierarchical structure of XML documents.

   This worked for some files but didn't work for files which have a little different structure.
4. Perl: xupdate - search.cpan.org
   

   xupdate - Process XUpdate commands over an XML document

   This page explains well. But it didn't work on OpenDocument xml.
5. another Python tool: 4suite.org
   

   command-line tool for performing XUpdates on XML documents

   I used the same xupdate output from #1.

So, what will I do next? I'm considering these.

1. OpenOffice API or macro
2. Disregard the xml aspect of content.xml file and process it by perl
3. Use OpenOffice XML tool from here and write XSL
   Xml - OpenOffice.org Wiki
   

These are the Wiki pages of the OpenOffice.org XML Project

It is possible on Solaris with the help from dtrace(with a caveat). On Linux/Windows, I don't think this is possible with off-the-shelf kernel. That's why I wrote perl in my previous entry. It can't associate short-lived socket to the owning process. So, I'm still not sure if it was worth my time.

Here's a man page from DTraceToolkit. Notice the caveat, i.e.) this only captures new connections.

$ man -M ~root/DTraceToolkit-0.99/Man tcpsnoop_snv
[...]
DESCRIPTION
     This analyses TCP network packets and prints the responsible
     PID  and  UID,  plus standard details such as IP address and
     port. This captures traffic of newly created TCP connections
     that were established while this program was running. It can
     help identify which processes is causing TCP traffic.

Here's how I show network traffic and owner process on Windows using the perl in my previous entry. This can't show sockets already closed and dis-owned. Also, I sorted by # of bytes rather than by # of packets.

$ perl /c/d/bin/active-port.pl 5 200 | sort -t $'\t' -k 3nr \
> | while read line; do
>   echo $line
>   pid=$(netstat -ano | awk '$2 == "'$(echo $line|cut -d' ' -f3)'" {print substr($0,72)}')
>   [ "$pid" != "" ] && tasklist /v /fi "PID eq $pid" | tail -1 | cut -c1-18,67-76,93-120,150-
> done
50 packets 220.16.82.26:2187 75700 bytes.
wmplayer.exe        10,828 K MY_W2K3\Administrator      0:00:20 Windows Media Player   
                                                 
81 packets other, e.g.) arp 68460 bytes.
41 packets 220.16.82.26:2255 29658 bytes.
firefox.exe        161,272 K MY_W2K3\Administrator      0:34:03 Let the Sunshine In - Mozilla Firefox   
                                
16 packets 220.16.82.26:2256 8475 bytes.
6 packets 220.16.82.26:137 660 bytes.
System                 264 K NT AUTHORITY\SYSTEM        0:36:29 N/A           
                                                          
4 packets 220.16.82.26:2144 364 bytes.
thunderbird.exe    111,220 K MY_W2K3\Administrator      0:16:12 Inbox : MyName@MyMail.COM - Thunderbird    
                    
2 packets 220.16.82.26:61466 298 bytes.

Here's how I show network traffic and owning process on Linux. I only show top 4. One restriction on Linux is that you can't use 'any' interface. Please specify real interface like 'eth0'.

$ sudo perl active-port.pl 1 1000 | head -4 \
> | while read line; do
>   echo $line;
>   echo $line|awk '$3 !~ "^o" {system("lsof -i@" _ $3)}' | tail -1
>   echo
> done
901 packets 129.158.21.152:32772 68016 bytes.
synergyc 3789 kinoue    4u  IPv4   9356       TCP myserver.jp.sun.com:32772->dhcp-jp-20-88.Jpn.Sun.COM:24800 (ESTABLISHED)

77 packets 129.158.21.152:32783 82222 bytes.
ftp     4109 kinoue    4u  IPv4  12469       TCP myserver.jp.sun.com:32783->labserv.Jpn.Sun.COM:53456 (ESTABLISHED)

8 packets other, e.g.) arp 480 bytes.

8 packets other, e.g.)UDP broadcast 1027 bytes.

Ever wanted to know which process is using the network bandwidth most? I googled and learned that, on Linux,  iftop seems to be able to tell which port is the busiest. But 'iftop' doesn't seems to be compilable on cygwin. So, I googled more and found several similar Windows GUI apps. But they are all shareware and they are overkill for my purpose anyway. So, I wrote a perl to parse tcpdump output to aggregate by local ports.

On Windows, it requires 'windump'.
WinDump: tcpdump for Windows

WinDump is the Windows version of tcpdump, the command line network analyzer for UNIX. WinDump is fully compatible with tcpdump

The output will look like this.

$ perl /c/e/tmp/active-port.pl 5 1000
962 packets     other, e.g.) arp                57850   bytes.
10 packets      220.14.81.26:1275               4526    bytes.
10 packets      220.14.81.26:1068               906     bytes.
10 packets      220.14.81.26:1256               570     bytes.
4 packets       220.14.81.26:1273               228     bytes.
2 packets       220.14.81.26:21         128     bytes.
1 packets       220.14.81.26:1026               636     bytes.
1 packets       other, e.g.)UDP broadcast               590     bytes.

Here's perl script. Borrowed code from here and here.

#!/usr/bin/perl -w
#
# Parse tcpdump and say which ports are more active
# Usage example: perl active-port.pl '\\Device\\NPF_{55899BD0-EC52-4066-ABA7-53849ACAA19A}' 100 | sort -t $'\t' -k 3nr
use strict;

if ($#ARGV != 1) {
    die "$0: Need exactly two arguments. Interface name/# and packet count to capture.\n";
}
my $tcpdump = "tcpdump";
my $index=0, my @local_ipaddrs, my $address;

if (defined $ENV{OS} && $ENV{OS} =~ "Windows_NT") {
    use Sys::Hostname;
    $tcpdump = "windump";
    my ($name,$aliases,$addrtype,$length,@addrs) = gethostbyname hostname;
    foreach $address (@addrs) {
        my ($a,$b,$c,$d) = unpack('C4',$address);
        $local_ipaddrs[$index++] = "$a.$b.$c.$d";
    }
} else {
    require IO::Interface::Simple;
    my @interfaces = IO::Interface::Simple->interfaces;
  loop: for my $if (@interfaces) {
      next loop if ( !defined $if->address );
      $local_ipaddrs[$index++] = $if->address;
  }
}

open(TCPDUMP, "$tcpdump -i $ARGV[0] -c $ARGV[1] -nntqep 2>/dev/null |");
eof(TCPDUMP) && die "$0: Could not run $tcpdump: $?\n";
my %activity, my %bytes;
while (<TCPDUMP>) {
    my ($link_level_length, $rest) = ($_ =~ /^[0-9a-f:]{17}\s>\s[0-9a-f:]{17},\s\S+,\slength\s(\d+):\s(.*)/);
    if (my ($addrport1,$addrport2) = ($rest =~ /(\d+\.\d+\.\d+\.\d+\.\d+)\s>\s(\d+\.\d+\.\d+\.\d+\.\d+):/)) {
        my $local;
      local_ipaddrses: foreach $address (@local_ipaddrs) {
          if ($addrport1 =~ /^$address\./) {
              $addrport1 =~ s/(.*)\./$1:/;
              $activity{$addrport1}++; $bytes{$addrport1}+=$link_level_length;
              $local = "true";
              last local_ipaddrses;
          }
      }
      local_ipaddrses: foreach $address (@local_ipaddrs) {
          if ($addrport2 =~ /^$address\./) {
              $addrport2 =~ s/(.*)\./$1:/;
              $activity{$addrport2}++; $bytes{$addrport2}+=$link_level_length;
              $local = "true";
              last local_ipaddrses;
          }
      }
        if (!defined $local) {
            $activity{"other, e.g.)UDP broadcast"}++; $bytes{"other, e.g.)UDP broadcast"}+=$link_level_length;
        }
    }
    else {
        $activity{"other, e.g.) arp"}++; $bytes{"other, e.g.) arp"}+=$link_level_length;
    }
}

foreach my $addr_n_port ( sort { $activity{$b} <=> $activity{$a} }  keys %activity) {
    print "$activity{$addr_n_port} packets\t$addr_n_port\t\t$bytes{$addr_n_port}\tbytes.\n";
}
exit 0;