Let the Sunshine In: Software support engineer in Tokyo

INOUE Katsumi's Blog

Copied from http://blogs.sun.com/LetTheSunshineIn

"The views expressed on this [blog; Web site] are my own and do not necessarily reflect the views of Oracle."

I might add: "Stuffs on this [blog; Web site] are my own and do not necessarily be supported by Oracle. Please do not dial oracle support phone# mention this [blog; Web site]". :)

I'm still learning new policy so I need to move more and more stuffs to wordpress.

I wrote about my experience when I copied Berkeley DB(aka: SleepyCat) data from AMD64 machine to x86 32-bit machine and I needed to remove "__db.001" file to start it up in my past blog. I ran into another occasion where I needed to remove "__db.*" files to start up Sun Java System Calendar Server.

So, I wondered what are these "__db.*" files and found this.

Shared memory regions

system memory used by Berkeley DB is potentially useful past the lifetime of any particular process. Therefore, additional cleanup may be necessary after an application fails because there may be no way for Berkeley DB to ensure that system resources backing the shared memory regions are returned to the system.

I think the action of removing "__db.*" files corresponds to the excerpt above.

Incidentally, this reminded me of ipcrm usage after oracle db crash. Sometimes, oracle db will not restart unless OS admin ipcrm's shared memory. Below, you can see shared memory was allocated on startup by sysdba.

$ ipcs -msp
IPC status from  as of Thu Jan 28 16:54:59 JST 2010
T         ID      KEY        MODE        OWNER    GROUP  CPID  LPID
Shared Memory:
m        102   0xf007cd34 --rw-r-----   ora10g oinstall  5503 29721
Semaphores:
s          7   0xe914e1e0 --ra-r-----   ora10g oinstall

$ tail -12 $ORACLE_HOME/admin/$ORACLE_SID/adump/ora_5503.aud
Redo thread mounted by this instance: 0 
Oracle process number: 0
Unix process pid: 5503, image: oracle@eslab23

Mon Dec 21 15:23:19 2009
ACTION : 'STARTUP'
DATABASE USER: '/'
PRIVILEGE : SYSDBA
CLIENT USER: ora10g
CLIENT TERMINAL: Not Available

I don't know if post-mortem analysis of this shared memory debris is possible in Oracle RDBMS case.
In BDB case, "__db.*" files contain some info. I hope I can write about it in this blog soon.

I found out that all Berkeley DB binaries on Solaris x86 have (or had) problem whereby libraries are searched on /net directory.

# /usr/ccs/bin/elfdump -d /usr/lib/bdb/db_verify | grep 'R*PATH'
       [5]  RUNPATH           0x225               /net/sunserver.loc0a/export/JES/U1/BDB/sbi/20070123.1/builds/solaris_x86/RELEASE/build.4.2.52_solaris_x86_5.10-RELEASE.01.23.2007.0847/usr/lib
       [6]  RPATH             0x225               /net/sunserver.loc0a/export/JES/U1/BDB/sbi/20070123.1/builds/solaris_x86/RELEASE/build.4.2.52_solaris_x86_5.10-RELEASE.01.23.2007.0847/usr/lib

I read this and thought about compiling the attached source. But I couldn't find x86 where SunStudio or gcc was installed.

Changing ELF Runpaths (Code Included) : Ali Bahrami

A recent change to Solaris ELF files makes it possible to change the runpath of a dynamic executable

Then I read this.

Introducing elfedit: A Tool For Modifying Existing ELF Objects : Ali Bahrami

elfedit can indeed modify the runpath in an object, but it is considerably more general than that. elfedit is a tool for examining and modifying the ELF metadata that resides within ELF objects.

So, I extracted elfedit and related libraries from internal Nevada install server onto Solaris 10. Maybe I'm showing my ignorance here. I couldn't make it work on Solaris 10. It could have possibly worked although surely not supported.

So, I decided to hack the binaries. I was a little scared when strings command didn't show it.

# /usr/bin/strings /usr/lib/bdb/db_verify | grep /net

But GNU strings can show it.

# /usr/sfw/bin/gstrings --target=elf32-i386 db_verify | grep /net
/net/sunserver.loc0a/export/JES/U1/BDB/sbi/20070123.1/builds/solaris_x86/RELEASE/build.4.2.52_solaris_x86_5.10-RELEASE.01.23.2007.0847/usr/lib

I used GNU sed to stub out runpath.

# gnused -i.backup 's|\/net/sunserver.loc0a/export/JES/U1/BDB/sbi/20070123.1/builds/solaris_x86/RELEASE/build.4.2.52_solaris_x86_5.10-RELEASE.01.23.2007.0847/usr/lib|                                                                                                                                              |' db_*

Berkeley DB (aka SleepyCat) is used in DSEE and Messaging Server. So, this VM contains many bdb datafiles created by 64-bit bdb library/executables.

Using the Sun Java Communications Suite VMware Image - Sun Java Communications Suite - wikis.sun.com

This image is 64-bit only.

So, when I started this VM on 32-bit machine, DSEE failed to start. I know it. DSEE does not support copying over bdb files over to different platform. Apparently, OpenLDAP doesn't either.

RE: BDB, endian

The ID's do not have a fixed, platform-independent, data-type associated with them, i.e. on 32-bit platforms the ID's are 32-bit integer values and on 64-bit platforms the ID's are 64-bit integer values.

But, in the rare hope that it would work, I edited several text files, among which was dse.ldif. The error finally reduced to __db.001 file. I googled and found a hint at RedHat package management.

I have used RedHat around 15 years and didn't know BerkeleyDB was used there.

Bug 486423 – rpmdb locking broken by other-arch rpmquery

it's deep internals of BDB and contains inherently architecture-dependent data like pointers to memory IIRC

I removed __db.001 files and , finally, DSEE631 32-bit started fine!

But this way of migrating data is not supported. In fact, if index was there or datafile was 8GB large, it may not work.

I have a habit of running VirtualBox or VMware image and run it in headless mode, ... blindly :) ..., without checking config file. I own VMware Workstation so this is possible.

AFAIK, this is OK with VirtualBox. It enables RDP be default. So, one can always connect from remote Windows,.. or SunRay, etc.

Today, I learned VMware doesn't take care of enabling VNC remote access. The VM I tried is from:

Communications Suite 7: Installing the VMware Image - Factotum

Communications Suite 7 VMware image is a simple and portable method for demonstrating Communications Suite 7 features without the overhead of installation and configuration.

I started this vm like:

$ vmrun start SunJava_Comms_Suite7u0_VM/*.vmx nogui

I tried to access it from VNC viewer on Windows. Failed... So, what did I do ?

I could have ACPI shut down this vm pretty much safely like this and configure vnc and restart it.

$ vmrun stop SunJava_Comms_Suite7u0_VM/*.vmx soft

Instead, I chose to ssh into this vm. First, I checked how vm guest network is configured.

$ grep connection SunJava_Comms_Suite7u0_VM/*.vmx
ethernet0.connectionType = "nat"

It's NAT. This means it uses VMNet8 subnet on host. And if guest is set up for DHCP, it would get IP address from dhcp server for vmnet8.

$ grep ethernet.*Address SunJava_Comms_Suite7u0_VM/*.vmx
ethernet0.generatedAddress = "00:0c:29:55:ce:5b"
ethernet0.generatedAddressOffset = "0"
$ grep -B 3 00:0c:29:55:ce:5b /etc/vmware/vmnet8/dhcpd/dhcpd.leases
lease 172.16.128.130 {
        starts 3 2010/01/20 05:56:45;
        ends 3 2010/01/20 05:56:44;
        hardware ethernet 00:0c:29:55:ce:5b;
$ ping -b 172.16.128.255
WARNING: pinging broadcast address
PING 172.16.128.255 (172.16.128.255) 56(84) bytes of data.
64 bytes from 172.16.128.1: icmp_seq=0 ttl=64 time=0.042 ms
64 bytes from 172.16.128.130: icmp_seq=0 ttl=255 time=1.63 ms (DUP!)

Voila! Now, I can ssh.

$ ssh root@172.16.128.130
Warning: Your password has expired, please change it now.

New Password:
Re-enter new Password:
sshd-kbdint: password successfully changed for root

Last login: Tue Dec 15 02:26:34 2009 from 10.1.110.19
Sun Mircosystems Inc.   SunOS 5.10      Generic January 2005

Sun Communications Suite 7
Deployed via the EMRA Toolkit (version 7.0-0.02)
Virtual Machine Built: 12:15:00:44.25-2009

  Product: Messaging Server (64bit) 7u3                Version: 7.3-11.01
  Product: Comms DSsetup 6.4                           Version: 6.4-5.05
  Product: Calendar Server 7                           Version: 7.0-0.26
  Product: Instant Messaging 8u2                       Version: 8.2-02.04
  Product: Convergence 1u3                             Version: 1.0-10.01
  Product: Indexing and Search Service 1.0             Version: 1.0-32.01
  Product: Comms Delegated Administrator 7             Version: 7.0-1.00
  Product: Calendar Server 6.3                         Version: 6.3-19.01

Nice! I didn't know that Solaris can set expiration on root password.

I stumbled upon an interesting tool.

NTrace

NTrace uses a novel approach of instrumenting binary code on the fly by leveraging certain aspects of the Microsoft hotpatching infrastructure. This allows NTrace to be both very robust and fast: On x86, NTrace outperforms DTrace by a factor of almost three!

1. This tool is not yet downloadable.
2. I think this tool is more closer to truss(strace on linux) than to dtrace.
3. The author apparently did a good study on dtrace implementations and explains difference between SPARC and IA-32 in "NTrace: Function Boundary Tracing for Windows on IA-32".
   

Frequently Asked Questions About the Java HotSpot VM

On Windows, the default thread stack size is read from the binary (java.exe). As of Java SE 6, this value is 320k in the 32-bit VM and 1024k in the 64-bit VM.

So, let's see it in Cygwin. Another way to check is with Microsoft's MSVC "dumpbin /headers" command.

$ type objdump
objdump is hashed (/usr/bin/objdump)
$ objdump -p /c/d/Program\ Files/Java/jre1.6.0_07/bin/java.exe | grep stack
SizeOfStackReserve      00050000
SizeOfStackCommit       00001000

These values are in hex. So show them in decimal.

$ objdump -p /c/d/Program\ Files/Java/jre1.6.0_07/bin/java.exe | grep stack | cut -f2 | sed 's/^/0x/' | xargs printf "%d %d\n"
327680 4096
$ objdump -p /c/d/Program\ Files/Java/jre1.5.0_16/bin/java.exe | grep stack | cut -f2 | sed 's/^/0x/' | xargs printf "%d %d\n"
262144 4096

So, it's 256KB in 1.5 and 320KB in 1.6.

My previous entry is apparently a oracle FAQ.

OTN Discussion Forums : tkprof ...

I have 10 udump trace files generated by batch that run in 10 process.
I would like to merge them into one tkprof file.

I don't think my solution of using /dev/stdin is actually supported by oracle. Supported way is to use a Java tool called trcsess on oracle 10g and up.

Just for the fun of it, I tried using /dev/stdin on trcsess command line. It worked! So, here's my hypothesis of the day:
"Any Java program which takes filename as an argument can read /dev/stdin because Java program doesn't call lower library functions."