The Dirtracer Corner - Lee Trujillo: Tips, Tricks and News on Sun Java Systems Directory Server Tracing tool

Hello all,

I'm please to announce I have released Pkgapp 3.0 for Solaris!

So far Pkgapp 3.0 is only availabe for Solaris but I will begin porting this code to Linux and HPUX.  I wanted to get Solaris first instead of waiting for all to be done...this way at least one unix flavor is available for use.

You can obtain Pkgapp 3 from Big Admin: Pkgapp 3.0 Solaris

Sun Employees can also find information on Sunspace.

Included files:

• Pkgapp version 3 Script
  
• Pkgapp Reference Guide: SUN-GDD_Pkgapp_Reference_Guide3.0.0.pdf
• Pkgapp Readme
• GDD Readme
• GDD License
  

For information on what changed, what the usage looks like and a runtime example (using mysql) see my last post - Pkgapp 3.0 code has been finalized!

Enjoy all!

Lee

Pkgapp3 is the latest update to the Sun GDD tool set. These tools are designed to help customers gather the data needed for the best possible support on their issues.

Pkgapp is a shell tool built by Sun Support Engineering to help gather all OS and application libraries used by Sun Java Systems (and others) servers at runtime.  Pkgapp can be used to gather these libs, etc, from cores produced by crashes or gcores captured manually by administrators.

Pkgapp is part of the GDD (Gathering Debug Data) suite of tools and has been used for years to help Sun Support debug tricky crashes, as well as performance and memory leak problems where cores or gcores are available.

New major features in Pkgapp 3.0:

1) Now handles Java Cores and grabs jstack and jinfo.
 - See the -j and -J options

2) Includes all functionality that was available in both pkg_app 2.7 and pkgcore.

3) Can allow for the temporary files to be retained in /tmp
 - See the -r option

4) Allows pkgapp3 to be placed in a $PATH location; can now be run from any location.

5) Can use relative paths for the following options:
 -c <core file OR pid of a running process>
 -p <full path to, but not including the process binary> (ns-slapd, imapd, httpd etc.)
 -s <Storage; path to store the final tar file in>

6) Retains all lib/binary paths (pkgcore) as well as the pkg_app app/ location for older pkg_app aware applications.

7) Names the final storage path as pkgapp-<date>-<index> to help administrators locate the final tar file faster.
 - Ex: pkgapp3-111708-01/

8) Gathers the following Solaris only proctool commands from the core.
 - pldd
 - pstack
 - pmap
 - pcred
 - pflags
 - pargs

9) Checks to see if the core file is truncated and checks the core files cksum to ensure the file size match.

10) Gathers the following core and pkgapp3 specific data
 - core file cksum
 - core file truncation (good/bad)
 - core file size (ls -l)
 - library list (from the pldd)
 - manifest log (all files included int he final tar)
 - pkgapp3 arguments used.

11) Gathers the following OS specific data
 - coreadm info
 - date/time
 - messages (from /var/adm)
 - pkginfo
 - showrev -p
 - /etc/release (Solaris)
 - ulimit
 - uname -a

12) Creates a coreinfo.sh script to allow Sun Engineers to quickly run "coreinfo" for ns-slapd processes (only).

13) Alerts administrators they must upload 2 files (pkgapp tar & core) separately when they do not use the -i (include core) switch.

14) Will exit with a fatal error if a pldd cannot be properly retrieved from the core and alerts the administrator to run pkgapp3 using a pid as the -c option.

15) Saves the pkgapp3 arguments and runtime log into /var/tmp/pkgapp-history/.  Helps administrators and pkgapp see previous runs.

16) Updated usage and alert messages.

 Here are a couple previews of how the new code works.

 Example: Usage output

* ----------------------------------------------------------------------------------
* Sun Microsystems RSD pkgapp 3.0 Solaris                               [11/24/2008]
* ----------------------------------------------------------------------------------
pkgapp 3.0, a Sun Microsystems data gathering utility.
Usage:
  pkgapp [options] -c <core file | pid> -p <full path (path only) to process binary> -s [path to write tar file]

Required parameters:
 -c <core file OR pid of a running process>
 -p <full path to, but not including the process binary> (ns-slapd, imapd, httpd etc.)

Optional parameters:
 -i (Include a previously generated core file with the final tar.gz)
 -j (Javacore; process a java core)
 -r (Remove all temp files)
 -q (Quiet)
 -d (Debug)
 -J <JstackPath; path to the jstack (jdk) commands>
        defaults to /usr/jdk/instances/...
 -s <Storage; path to store the final tar file in>

usage:  pkgapp -c <name of the core file> -p <path to process binary>
usage:  pkgapp -c <pid of the running app> -p <path to process binary>

Examples: these are examples mixing various parameters

Directory Server
pkgapp -i -r -c ./core.14740 -p /var/mps/ds52p4/bin/slapd/server/64

Messaging Server
pkgapp -c ./core.3496 -p /opt/SUNWmsgsr/lib

Web Server
pkgapp -c ./core.1092 -p /space/iws70/lib -s /var/crash

Calendar Server
pkgapp -r -c ./core -p /opt/SUNWics5/cal/lib

Sendmail
pkgapp -i -c 512 -p /usr/lib

Mysqld
pkgapp -i -r -c ./core -p /support/mysql-5.0.41/bin

Example: Runtime output

* ----------------------------------------------------------------------------------
* Sun Microsystems RSD pkgapp 3.0 Solaris                               [11/24/2008]
* ----------------------------------------------------------------------------------
* OS release                            [5.10]
* Platform                              [SUNW,Sun-Blade-2500]
* Checking [-c] is a core or pid        [using pid 1409]
* Process Root                          [/support/mysql-5.0.41/data]
* Databin parameter [-s] checks         [reset to /var/tmp/dev]
* Databin found                         [/var/tmp/dev/test/cores/storage]
* Databin writable check                [success]
* Databin used/created is               [/var/tmp/dev/test/cores/storage/pkgapp-112408-05]
* Creating temp area                    [/tmp/pkgapp.2928/]
* Pid used, no corefile to check       
* Process binary                        [mysqld]
* Checking usage history                [not recently run]
* mysqld binary bit version             [32]
* Checking path [-p] to binary name     [success, path != binary name]
* Checking path [-p] is a directory     [success]
* Locating mysqld                       [success]
* Checking located mysqld is 32 bit     [success]
* Binary located                        [/support/mysql-5.0.41/bin/mysqld]
* Adding binary to pkgapp.pldd          [success]
* Grabbing pldd                         [success]
* Grabbing pstack                       [success]
* Grabbing pmap                         [success]
* Grabbing pcred                        [success]
* Grabbing pflags                       [success]
* Grabbing pargs                        [success]
* Provide the full path and name to the core file
* If you do not have a core, enter "none"
* Example - /data/cores/core.1445:      /var/tmp/dev/cores/core.1409
                                        [Answer was -> /var/tmp/dev/cores/core.1409]
* Grabbing [-i] core/gcore              [success]
* Javatools [-j] not set                [skipped]
* Grabbing /var/adm/messages            [success]
* Grabbing uname -a                     [success]
* Grabbing date/time                    [success]
* Grabbing showrev -p                   [success]
* Grabbing pkginfo -l                   [success]
* Grabbing /etc/release                 [success]
* Grabbing coreadm                      [success]
* Grabbing ulimit                       [success]
* Grabbing libs                         [success]
* Making lib paths app/                 [success]
* Making lib paths libs/                [success]
* Linking libraries                     [success]
* Libraries linked                      [62 ttl]
*                                      
* Using hostid for naming .tar.gz       [837872d0]
* Writing file                          [pkgapp-837872d0-s4u-2500a-brm04-081124-095809.tar.gz]
*                                      
* Done gathering files                 
* Writing dbxrc & opencore.sh files     [success]
* Writing manifest-081124-095809.log    [success]
* Writing pkgapp-args-081124-095809     [success]
* Creating final tarfile                [success]
* Compressing tarfile                   [success]
* End of runtime logging               
* Saving history info                   [/var/tmp/pkgapp-history/history-081124-095809.log]
* Saving runtime log                    [/var/tmp/pkgapp-history/runtime-081124-095809.log]
* Removing [-r] temp area/files         [removed]
*                                      
* Operations Complete                  
----------------------------------------------------------------------------------
Upload the following file(s) to your supportfiles.sun.com Cores Directory at Sun

1) File(s) located in directory /var/tmp/dev/test/cores/storage/pkgapp-112408-05

                [ pkgapp-837872d0-s4u-2500a-brm04-081124-095809.tar.gz ]



                                Thank you.
                                Sun Software Technology Service Center (STSC)


NOTES:
1) You can check for updates to this script here:
        BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html
2) Release Notes and Guides located here:
        Docs - http://docs.sun.com/app/docs/doc/820-0437
3) GDD information located here:
        Docs - http://www.sun.com/service/gdd/index.xml

4) Please send all Bugs and RFE's to the following address:
        Subject "pkgapp bug/rfe" - gdd-issue-tracker@sun.com

5) Please send all other questions etc to:
        Subject "pkgapp feedback" - gdd-feedback@sun.com

I will announce Pkgapp 3.0 for Solaris's availability in the next few days...stay tuned!

Lee

I have finally had time to revamp the now aging pkg_app 2.7.

I have made Pkg_app 3.0 bigger better faster and will now include all libraries gathered by the older (2006) pkgcore.sh many have used in the past.

More later but here is a sneal peak.

--------------------------------------------------------------------------------
* Sun Microsystems RSD pkg_app 3.0b Solaris[08/18/2008]
--------------------------------------------------------------------------------
* OS release                            [5.9]
* Platform                              [SUNW,Sun-Blade-1000]
* Using core                            [/var/tmp/cores/Pkg_app/core.28907]
* Process Root                          [/data/sunone/ds52p5/slapd-config]
* Process binary                        [ns-slapd]
* ns-slapd binary bit version           [64]
* Checking path to binary name          [success, path != binary name]
* Checking path is a directory          [success]
* Locating ns-slapd                     [success]
* Checking located ns-slapd is 64 bit   [success]
* Binary located                        [/data/sunone/ds52p5/bin/slapd/server/64/ns-slapd]
* Adding binary to pkg_app.pldd         [success]
* Grabbing pldd                         [success]
* Grabbing pstack                       [success]
* Grabbing pmap                         [success]
* Grabbing pcred                        [success]
* Grabbing pflags                       [success]
* Grabbing pargs                        [success]
* Grabbing core/gcore                   [success]
* Grabbing /var/adm/messages            [success]
* Grabbing uname -a                     [success]
* Grabbing date                         [success]
* Grabbing showrev -p                   [success]
* Grabbing libs                         [success]
* Making lib paths                      [success]
* Linking libraries                     [success]
*                                      
* Databin parameter check 1             [success]
* Databin parameter check 2             [success]
* Databin Found                         [/var/tmp/mycore]
* Databin used/created is               [/var/tmp/mycore/081808-02]
* Using hostid for naming .tar.gz       [834d2699]
* Writing file                          [pkg_app-834d2699-kaneda-080818-080751.tar.gz]
*                                      
* Done gathering files                 
* Writing dbx files                     [success]
* Creating final tarfile                [success]
* Compressing tarfile                   [success]
* End of runtime logging               
* Operations Complete
--------------------------------------------------------------------------------
Upload this file to your supportfiles.sun.com Cores Directory at Sun
File located in directory /var/tmp/mycore/081808-02


                [ pkg_app-834d2699-kaneda-080818-080751.tar.gz ]


                                Thank you.
                                Sun Software Technology Service Center (STSC)


NOTES:
1) You can check for updates to this script here:
        BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html

2) Release Notes and Guides located here:
        Docs - http://docs.sun.com/app/docs/doc/820-0437

3) GDD information located here:
        Docs - http://www.sun.com/service/gdd/index.xml

4) Please send all Bugs and RFE's to the following address:
        Subject "pkg_app bug/rfe" - gdd-issue-tracker@sun.com

5) Please send all other questions etc to:
        Subject "pkg_app feedback" - gdd-feedback@sun.com

root[/var/tmp/mycore/081808-02]#ls -l
total 38798
drwxrwxrwx   3 root     other        512 Aug 18 08:08 core-lib-data
-rwxrwxrwx   1 root     other        487 Aug 18 08:08 opencore
drwxrwxrwx   3 root     other        512 Aug 18 08:08 os-info
-rwxrwxrwx   1 root     other    19828244 Aug 18 08:08 pkg_app-834d2699-kaneda-080818-080751.tar.gz
drwxrwxrwx   2 root     other        512 Aug 18 08:08 pkgapp-info
drwxrwxrwx   2 root     other        512 Aug 18 08:08 proctool-info
-rwxrwxrwx   1 root     other       1575 Aug 18 08:08 runtime-080818-080751.log

 Beta testing within my group should begin this week.

Cheers!

Lee

Hello all,

Every now and then I run into an issue where the customer used Dirtracer to grab a Gcore (or two) or a Core file from a crash.  Dirtracer users are forgetting to upload the separate pkg_app tar.gz file when one is generated.

It is not a problem per-say as a it is users missing a critical message that is displayed at the end of a Dirtracer run when a successful Pkg_app file has been created.

Lets take a look at the messages I am talking about.

Suppose you define one of the following parameters in the dirtracer.config file:

CORE_LOCATION
GRAB_GCORE

Lets use GRAB_GCORE for a quick test; lets grab one Gcore.

GRAB_GCORE="1"

We then run Dirtracer as normal.

#./dirtracer -f ./dirtracer.config

--------------------------------------------------------------------------------
Sun Microsystems dirtracer 6.0.6 Solaris Sparc                        06/09/2008
--------------------------------------------------------------------------------

Dirtracer run through it's Main Loop and grabs one Gcore.

* Entering Main Performance Gathering Loop
*                                      
* Loop 0 - 080609-131950               
* Loop 1 - 080609-131956               
* Loop 2 - 080609-132001               
* Loop 3 - 080609-132007               
* Loop 4 - 080609-132012               
* togo[5sec]-timer[5]
*                                      
* Grabbing gcore 080609-132018          [success]

Because Dirtracer has noted the presence of the grab gcore parameter it will automatically run Pkg_app to gather the needed OS and DS libraries needed for debugging. 

By default Dirtracer waits 120 seconds for Pkg_app to complete.  If for some reason the Core/Gcore's header has an overrun (can't determine contents normally) or has other errors like it is truncated, Dirtracer will kill the Pkg_app gathering phase.  Users should check the Pkg_app tar contents to see if all libs were gathered and take appropriate actions.

* Packaging files                      
*   Preparing files - pkg_app           [waiting 120 sec.]      [success]

After the main Dirtracer tar file is created it will display the Operations Complete message and inform the user of which files he needs to upload to Sun.

As you can see below Dirtracer informs you there is both a Dirtracer and "separate" Pkg_app file which needs to be uploaded.  Users should pay special attention to the End Messages displayed when Dirtracer finishes.  Most users miss the second file (Pkg_app) requiring separate upload.

Operations Complete
--------------------------------------------------------------------------------
1) Dirtracer capture data located in directory [ /var/tmp/data/060908-04 ]

Upload "only" this file to your supportfiles.sun.com cores directory at Sun

        [ dirtracer-834d2699-mysystem-080609-132018.tar.gz ]

2) pkg_app has captured system libs as the result of a gcore or core being found
                As this file may be large, please ftp this separately to Sun
                pkg_app file located in /var/tmp/data/060908-04/pkg_app_data

                [pkg_app834d2699-mysystem-080609-132051.tar.Z ]

You might ask why is Pkg_app's tar simply not included with the main Dirtracer tar file.  The answer is simple...size.  Dirtracer captures by itself can be big if a customer does not archive their log files on a heavily used system and the final size can be massive if a Core/Gcore is captured.  It simply wasn't an option to include a big Pkg_app file with an already bigger Dirtracer file.

[LT]

Hello all,

I was recently contacted by a Sun Customer Advocate regarding Pkg_app, Dirtracer and Clustered environments.

First some up front info on Pkg_app.  Pkg_app was originally created by Rick Zagorski and I took over its development in 2006.  I took what Rick had coded and added to it specifically (at the time) to help with Directory Server core/gcore files.  This was released as pkg_app 2.0 and was a mash of Ricks code with some additions of mine.  As you know I created Dirtracer (Stracer) in 2004 and continue its development to help customers gather data on their Directory Server problems. 

The Customer was having issues with Pkg_app and Dirtracer when they attempted to use it in a Clustered Environment.   Even though the customer was running an old version of Pkg_app, Pkg_app and Dirtracer are "not" tested nor certified for use in a Clustered Environment.  I've have only had one prior request to have pkg_app/dirtracer be Cluster aware so it has not been on the top of the revision list.

I will be revamping the Pkg_app code to take into account all libraries etc. that Pkgcore gets for use with Java Cores and will look into the Cluster issue.

More news on this later.

[LT]

Today I will revisit Pkg_app but will focus on its uses within Dirtracer.

Before Dirtracer 6.0.4 Customers who would use Dirtracer to gather cores and gcores would have to run Pkg_app manually after the fact.

Since version 6.0.4 Dirtracer has included Pkg_app in the <Dirtracer Install Path>/dirtracertools/ location and with the Quiet (-q) switch in Pkg_app 2.7 I able to embed Pkg_app within Dirtracer to run automatically.

If a Customer uses the following config file parameters Pkg_app will be launched automatically.

CORE_LOCATION="<full path to the core> + SERVER_DOWN="1"

GRAB_GCORE="1" or GRAB_GCORE="2"

Here is an example of the following config:  I used 1 check and 1 second interval for brevity.

NUMBEROFCHECKS="1"
INTERVAL="1"
GRAB_GCORE="1"

See the runtime-<date>-<time>.log:

As you see below Dirtracer completes a quick one second loop, exits the Main Loop and grabs a Gcore.

<SNIP>

*   pms.sh interval(1) x checks(1)      [pms.sh run time (1 sec.)]
*                                       
* Entering Main Performance Gathering Loop
*                                       
* Loop 0 - 080509-075120                
*                                       
* Grabbing gcore 080509-075122          [success]

</SNIP>

Once Dirtracer finishes with the Post Loop gathering, it executed Pkg_app to have it gather all libraries and the ns-slapd binary.  Note the normal Pkg_app processing information is not seen because Pkg_app has been launched with the Quiet (-q) option.

<SNIP>

* Packaging files                       
*   Preparing files - pkg_app           [waiting 120 sec.]      [success]

</SNIP>

In Dirtracer 6.0.4 customers grabbing large cores/gcores with Dirtracer saw what they thought was a pkg_app hang.  It was likely the core/gcore had overflowed the core header and Pkg_app could not process the file correctly.  As a result I created a timer function to monitor processes like Pkg_app.  

If the Pkg_app runs for more than 120 seconds, then Dirtracer will "kill" the pkg_app process and alert the Customer they need to run Pkg_app manually.

<SNIP>

* Packaging files                       
*   Preparing files - pkg_app           [killed]

</SNIP>

If Pkg_app was successful then it will present the Customer with the following message; see 2) below.

<SNIP>

1) Dirtracer capture data located in directory [ /var/tmp/data/051308-01 ]

Upload "only" this file to your supportfiles.sun.com cores directory at Sun

        [ dirtracer-834d2699-kaneda-080513-090202.tar.gz ]

2) pkg_app has captured system libs as the result of a gcore or core being found
                As this file may be large, please ftp this separately to Sun
                pkg_app file located in /var/tmp/data/051308-01/pkg_app_data

                [pkg_app834d2699-kaneda-080513-090347.tar.Z ]

</SNIP>

Currently Dirtracer does not give a major alert if Pkg_app was killed.  The customer should manually run Pkg_app or gather the libraries used by the process.

[LT]