The Dirtracer Corner - Lee Trujillo: Tips, Tricks and News on Sun Java Systems Directory Server Tracing tool

Alo all,

I received good news from the Big Admin admins today, the new DirTracer and Pkgapp versions are online!

The following versions are now available here on Big Admin. For information on the latest changes in this update, please see my previous blog - DT 6.0.7 Coming soon!

DirTracer 6.0.7

• Sun GDD - Directory Server - Dirtracer v 6.0.7 Solaris (SPARC)
• Sun GDD - Directory Server - Dirtracer v 6.0.7 Solaris on x86 platforms
• Sun GDD - Directory Server - Dirtracer v 6.0.7 SuSE Linux
• Sun GDD - Directory Server - Dirtracer v 6.0.7 RedHat Linux
  

Pkgapp 3.2

• Sun GDD - Pkgapp v 3.2 Solaris
• Sun GDD - Pkgapp v 3.2 SuSE Linux
• Sun GDD - Pkgapp v 3.2 Redhat Linux

I have also released the SLX 02 DirTracer Quickstart on SLX (Sun Learning Exchange) - "This video is a Quickstart guide to getting DirTracer6 running in a short amount of time using Configurator 6. I also step through a real example of how DirTracer runs using the config file created by Configurator."

Finally, I am in production of the latest DirTracer SLX video "Server Impact".  Using Slamd, Cacti and DirTracer I plan on showing exactly what impact a Directory Server will see from DirTracer runs based on different run/problem types.  More on this later.

Cheers!

Lee

Hey all,

Quick update to say Pkgapp 3.2 is now available externally on the Big Admin scripts site.  You can download it here.

Currently I am working on Pkgapp 3.2 ports to Linux.

Big Admin Scripts: Sun Java System Category

Big Admin MOTD

Cheers!

Lee

Hi all!

Pkgapp 3.2 is available internally...yeah!

I finally had time to update the Pkgapp documentation and retest it on various application cores/gcores and it is now ready for prime time.  I have also submitted it to BigAdmin and it should be available in the next few days.  If you need it now, please contact your favorite Sun Support representative and request he send it from the internal download site.  I will also update "this" post once Big Admin has uploaded the new version.

Pkgapp 3.2 has 2 big new features...

1. It includes adm64 libraries that were previously not included making debugging easier on that platform for java issues.
2. It includes a new core truncation feature developed by a member of the Directory Server Sustaining team.  The great news is, the feature not only works on Directory Server cores but any other core/gcore you may be dealing with!

If the core/gcore is good, the user is given the following message:

* Checking if corefile is truncated     [CORE gcore.cshttpd.28734 GOOD Core contains 293 segments ]

If the core is truncated, the user is alerted and required to answer yes/no as to whether they understand the data may be totally unusable.

* Checking if corefile is truncated     [WARNING! core gcore.cshttpd.28734.truncated is truncated (40396864d bytes instead of 40397397d bytes)]
*                                     
* ALERT! The core was found to be truncated, it therefore may not be usable
* pkgapp will continue to gather data and you can still upload the result to Sun
*                                     
* Do you understand the core may not be usable? [yes|no] yes
                                        [Answer was -> yes]
* Continuing...  

If you use/have an RSS Reader such as NetNewsWire or  Google Reader you can see the latest updates to Big Admin using the following links.

Big Admin Scripts: Sun Java System Category

Big Admin MOTD

Next up...I plan to finalize and release DTR and begin production of my first SLX video for Dirtracer users!

Take care,

Lee

Hello all,

I'm please to announce I have released Pkgapp 3.0 for Solaris!

So far Pkgapp 3.0 is only availabe for Solaris but I will begin porting this code to Linux and HPUX.  I wanted to get Solaris first instead of waiting for all to be done...this way at least one unix flavor is available for use.

You can obtain Pkgapp 3 from Big Admin: Pkgapp 3.0 Solaris

Sun Employees can also find information on Sunspace.

Included files:

• Pkgapp version 3 Script
  
• Pkgapp Reference Guide: SUN-GDD_Pkgapp_Reference_Guide3.0.0.pdf
• Pkgapp Readme
• GDD Readme
• GDD License
  

For information on what changed, what the usage looks like and a runtime example (using mysql) see my last post - Pkgapp 3.0 code has been finalized!

Enjoy all!

Lee

Pkgapp3 is the latest update to the Sun GDD tool set. These tools are designed to help customers gather the data needed for the best possible support on their issues.

Pkgapp is a shell tool built by Sun Support Engineering to help gather all OS and application libraries used by Sun Java Systems (and others) servers at runtime.  Pkgapp can be used to gather these libs, etc, from cores produced by crashes or gcores captured manually by administrators.

Pkgapp is part of the GDD (Gathering Debug Data) suite of tools and has been used for years to help Sun Support debug tricky crashes, as well as performance and memory leak problems where cores or gcores are available.

New major features in Pkgapp 3.0:

1) Now handles Java Cores and grabs jstack and jinfo.
 - See the -j and -J options

2) Includes all functionality that was available in both pkg_app 2.7 and pkgcore.

3) Can allow for the temporary files to be retained in /tmp
 - See the -r option

4) Allows pkgapp3 to be placed in a $PATH location; can now be run from any location.

5) Can use relative paths for the following options:
 -c <core file OR pid of a running process>
 -p <full path to, but not including the process binary> (ns-slapd, imapd, httpd etc.)
 -s <Storage; path to store the final tar file in>

6) Retains all lib/binary paths (pkgcore) as well as the pkg_app app/ location for older pkg_app aware applications.

7) Names the final storage path as pkgapp-<date>-<index> to help administrators locate the final tar file faster.
 - Ex: pkgapp3-111708-01/

8) Gathers the following Solaris only proctool commands from the core.
 - pldd
 - pstack
 - pmap
 - pcred
 - pflags
 - pargs

9) Checks to see if the core file is truncated and checks the core files cksum to ensure the file size match.

10) Gathers the following core and pkgapp3 specific data
 - core file cksum
 - core file truncation (good/bad)
 - core file size (ls -l)
 - library list (from the pldd)
 - manifest log (all files included int he final tar)
 - pkgapp3 arguments used.

11) Gathers the following OS specific data
 - coreadm info
 - date/time
 - messages (from /var/adm)
 - pkginfo
 - showrev -p
 - /etc/release (Solaris)
 - ulimit
 - uname -a

12) Creates a coreinfo.sh script to allow Sun Engineers to quickly run "coreinfo" for ns-slapd processes (only).

13) Alerts administrators they must upload 2 files (pkgapp tar & core) separately when they do not use the -i (include core) switch.

14) Will exit with a fatal error if a pldd cannot be properly retrieved from the core and alerts the administrator to run pkgapp3 using a pid as the -c option.

15) Saves the pkgapp3 arguments and runtime log into /var/tmp/pkgapp-history/.  Helps administrators and pkgapp see previous runs.

16) Updated usage and alert messages.

 Here are a couple previews of how the new code works.

 Example: Usage output

* ----------------------------------------------------------------------------------
* Sun Microsystems RSD pkgapp 3.0 Solaris                               [11/24/2008]
* ----------------------------------------------------------------------------------
pkgapp 3.0, a Sun Microsystems data gathering utility.
Usage:
  pkgapp [options] -c <core file | pid> -p <full path (path only) to process binary> -s [path to write tar file]

Required parameters:
 -c <core file OR pid of a running process>
 -p <full path to, but not including the process binary> (ns-slapd, imapd, httpd etc.)

Optional parameters:
 -i (Include a previously generated core file with the final tar.gz)
 -j (Javacore; process a java core)
 -r (Remove all temp files)
 -q (Quiet)
 -d (Debug)
 -J <JstackPath; path to the jstack (jdk) commands>
        defaults to /usr/jdk/instances/...
 -s <Storage; path to store the final tar file in>

usage:  pkgapp -c <name of the core file> -p <path to process binary>
usage:  pkgapp -c <pid of the running app> -p <path to process binary>

Examples: these are examples mixing various parameters

Directory Server
pkgapp -i -r -c ./core.14740 -p /var/mps/ds52p4/bin/slapd/server/64

Messaging Server
pkgapp -c ./core.3496 -p /opt/SUNWmsgsr/lib

Web Server
pkgapp -c ./core.1092 -p /space/iws70/lib -s /var/crash

Calendar Server
pkgapp -r -c ./core -p /opt/SUNWics5/cal/lib

Sendmail
pkgapp -i -c 512 -p /usr/lib

Mysqld
pkgapp -i -r -c ./core -p /support/mysql-5.0.41/bin

Example: Runtime output

* ----------------------------------------------------------------------------------
* Sun Microsystems RSD pkgapp 3.0 Solaris                               [11/24/2008]
* ----------------------------------------------------------------------------------
* OS release                            [5.10]
* Platform                              [SUNW,Sun-Blade-2500]
* Checking [-c] is a core or pid        [using pid 1409]
* Process Root                          [/support/mysql-5.0.41/data]
* Databin parameter [-s] checks         [reset to /var/tmp/dev]
* Databin found                         [/var/tmp/dev/test/cores/storage]
* Databin writable check                [success]
* Databin used/created is               [/var/tmp/dev/test/cores/storage/pkgapp-112408-05]
* Creating temp area                    [/tmp/pkgapp.2928/]
* Pid used, no corefile to check       
* Process binary                        [mysqld]
* Checking usage history                [not recently run]
* mysqld binary bit version             [32]
* Checking path [-p] to binary name     [success, path != binary name]
* Checking path [-p] is a directory     [success]
* Locating mysqld                       [success]
* Checking located mysqld is 32 bit     [success]
* Binary located                        [/support/mysql-5.0.41/bin/mysqld]
* Adding binary to pkgapp.pldd          [success]
* Grabbing pldd                         [success]
* Grabbing pstack                       [success]
* Grabbing pmap                         [success]
* Grabbing pcred                        [success]
* Grabbing pflags                       [success]
* Grabbing pargs                        [success]
* Provide the full path and name to the core file
* If you do not have a core, enter "none"
* Example - /data/cores/core.1445:      /var/tmp/dev/cores/core.1409
                                        [Answer was -> /var/tmp/dev/cores/core.1409]
* Grabbing [-i] core/gcore              [success]
* Javatools [-j] not set                [skipped]
* Grabbing /var/adm/messages            [success]
* Grabbing uname -a                     [success]
* Grabbing date/time                    [success]
* Grabbing showrev -p                   [success]
* Grabbing pkginfo -l                   [success]
* Grabbing /etc/release                 [success]
* Grabbing coreadm                      [success]
* Grabbing ulimit                       [success]
* Grabbing libs                         [success]
* Making lib paths app/                 [success]
* Making lib paths libs/                [success]
* Linking libraries                     [success]
* Libraries linked                      [62 ttl]
*                                      
* Using hostid for naming .tar.gz       [837872d0]
* Writing file                          [pkgapp-837872d0-s4u-2500a-brm04-081124-095809.tar.gz]
*                                      
* Done gathering files                 
* Writing dbxrc & opencore.sh files     [success]
* Writing manifest-081124-095809.log    [success]
* Writing pkgapp-args-081124-095809     [success]
* Creating final tarfile                [success]
* Compressing tarfile                   [success]
* End of runtime logging               
* Saving history info                   [/var/tmp/pkgapp-history/history-081124-095809.log]
* Saving runtime log                    [/var/tmp/pkgapp-history/runtime-081124-095809.log]
* Removing [-r] temp area/files         [removed]
*                                      
* Operations Complete                  
----------------------------------------------------------------------------------
Upload the following file(s) to your supportfiles.sun.com Cores Directory at Sun

1) File(s) located in directory /var/tmp/dev/test/cores/storage/pkgapp-112408-05

                [ pkgapp-837872d0-s4u-2500a-brm04-081124-095809.tar.gz ]



                                Thank you.
                                Sun Software Technology Service Center (STSC)


NOTES:
1) You can check for updates to this script here:
        BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html
2) Release Notes and Guides located here:
        Docs - http://docs.sun.com/app/docs/doc/820-0437
3) GDD information located here:
        Docs - http://www.sun.com/service/gdd/index.xml

4) Please send all Bugs and RFE's to the following address:
        Subject "pkgapp bug/rfe" - gdd-issue-tracker@sun.com

5) Please send all other questions etc to:
        Subject "pkgapp feedback" - gdd-feedback@sun.com

I will announce Pkgapp 3.0 for Solaris's availability in the next few days...stay tuned!

Lee