Wednesday Jul 09, 2008
Sometimes, we can use the ready to go debugging scripts to collect more data in a running web server, see
http://docs.sun-DOT-com/app/docs/doc/820-0429/gebck?a=view
In particular, the pkg_app script is often used when we found a core dump file.
This pkg_app script will help collect the needed libraries and others, so we at sun can proceed further to examine the core dump file.
The other situation can be a web server hang, and we will collect gcore <web server pid> and pkg_app libraries to examine the running threads of the hung web server at that time.
You can download the latest pkg_app script at
http://www.sun-DOT-com/bigadmin/scripts/indexSjs.html
http://www.sun.com/bigadmin/jsp/descFile.jsp?url=descAll/sun_gdd__all__pkg_a
the above is for Linux and the below is for Solaris,
http://www.sun.com/bigadmin/jsp/descFile.jsp?url=descAll/sun_gdd__directory___
Then after the download and tar -xf,
apple:/export/home/bits/iws/misc/pkg_app-dir> tar -xf ../SUN-GDD_ALL_pkg_app_solaris.tar
apple:/export/home/bits/iws/misc/pkg_app-dir> ls -lrt
total 62
-rw-r--r-- 1 root other 3581 Sep 12 2007 README.txt
-rw-r--r-- 1 root other 3461 Sep 12 2007 LICENSE.txt
-rwxr-xr-x 1 root other 19285 Oct 2 2007 pkg_app
-rwxr-xr-x 1 root other 3329 Oct 2 2007 README.pkg_app
Then you can follow the steps at
http://docs.sun.com/app/docs/doc/820-0429/gebck?a=view
"
-
Copy the script to a temporary directory on the system where Web
Server is installed.
-
Login as superuser.
-
Execute the pkg_app script in one of the following
three ways:
e.g. in my env,
apple:/export/home/iws7.0u3/https-apple.asia.sun.com/logs> ptree 9736
9734 webservd-wdog -d /export/home/iws7.0u3/https-apple.asia.sun.com/config -r
9735 webservd -d /export/home/iws7.0u3/https-apple.asia.sun.com/config -r /e
9736 webservd -d /export/home/iws7.0u3/https-apple.asia.sun.com/config -r
and a run to show the options,
apple:/export/home/bits/iws/misc/pkg_app-dir> ./pkg_app
--------------------------------------------------------------------------------
* Sun Microsystems RSD pkg_app 2.7 Solaris [07/09/2008]
--------------------------------------------------------------------------------
pkg_app 2.7, a Sun Microsystems data gathering utility.
Usage:
./pkg_app [options] -c <core file | pid> -p <full path to process binary> -s [path to write tar file]
Required parameters:
-c <core file OR pid of a running process>
-p <full path to process binary> (ns-slapd, imapd, httpd etc.)
Optional parameters:
-i (Include the core file with the final tar.gz)
-q (Quiet)
-d (Debug)
-s <Storage; path to store the final tar file in>
usage: ./pkg_app -c <name of the core file> -p <path to process binary>
usage: ./pkg_app -c <pid of the running app> -p <path to process binary>
Examples: these are examples mixing various parameters
Directory Server
./pkg_app -i -c ./core.14740 -p /var/mps/ds52p4/bin/slapd/server/64
Messaging Server
./pkg_app -c ./core.3496 -p /opt/SUNWmsgsr/lib
Web Server
./pkg_app ./core.1092 -p /space/iws70/lib -s /var/crash
Calendar Server
./pkg_app -i -c ./core -p /opt/SUNWics5/cal/lib
--------------------end options above--------------------
then, I looked up where is my webservd binary in web 7.0u3 ,
apple:/export/home/iws7.0u3/https-apple.asia.sun.com/logs> find /export/home/iws7.0u3 -name webservd
/export/home/iws7.0u3/lib/sparcv9/webservd
/export/home/iws7.0u3/lib/webservd
the 1st is 64-bit and 2nd is 32-bit,
e.g.
apple:/export/home/iws7.0u3/lib/sparcv9> file webservd
webservd: ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped
apple:/export/home/iws7.0u3/lib/sparcv9> cd ..
apple:/export/home/iws7.0u3/lib> file webservd
webservd: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, not stripped
so, we will point it to the 2nd one above which is currently in use in my pid=9736.
You can also check out which webservd by
apple:/export/home/iws7.0u3/lib> pmap 9736 | grep webservd
9736: webservd -d /export/home/iws7.0u3/https-apple.asia.sun.com/config -r /
00010000 8K read/exec /export/home/iws7.0u3/lib/webservd
00020000 8K read/write/exec /export/home/iws7.0u3/lib/webservd
which shows the 32-bit binary in above (the 2nd one above).
Then, I ran the pkg_app commands, ./pkg_app -c 9736 -p /export/home/iws7.0u3/lib
Outputs:
apple:/export/home/bits/iws/misc/pkg_app-dir> ./pkg_app -c 9736 -p /export/home/iws7.0u3/lib
--------------------------------------------------------------------------------
* Sun Microsystems RSD pkg_app 2.7 Solaris [07/09/2008]
--------------------------------------------------------------------------------
* OS release [5.8]
* Platform [SUNW,Sun-Blade-1000]
* Using pid [9736]
* Process Root [/export/home/iws7.0u3/https-apple.asia.sun.com/config]
* Process binary [webservd]
* webservd binary bit version [32]
* Checking path to binary name [success, path != binary name]
* Checking path is a directory [success]
* Locating webservd [success]
* Checking located webservd is 32 bit [success]
* Binary located [/export/home/iws7.0u3/lib/webservd]
* Adding binary to pkg_app.pldd [success]
* Grabbing pldd [success]
* Grabbing pstack [success]
* Grabbing pmap [success]
* Grabbing pcred [success]
* Grabbing pflags [success]
*
* Databin Used [/export/home/bits/iws/misc/pkg_app-dir]
* Using hostid for naming .tar.gz [83450f78]
* Writing file [pkg_app83450f78-apple-080709-154653.tar.Z]
*
* Processing file 74 of 74
*
* Done gathering files
* Writing dbx files [success]
* Creating final tarfile [success]
* Compressing tarfile [success]
*
Operations Complete
Upload this file to your supportfiles.sun.com Cores Directory at Sun
File located in directory .
[ pkg_app83450f78-apple-080709-154653.tar.Z ]
Thank you.
Sun Software Technology Service Center (STSC)
NOTES:
1) You can check for updates to this script here:
BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html
2) Release Notes and Guides located here:
Docs - http://docs.sun.com/app/docs/doc/820-0437
3) GDD information located here:
Docs - http://www.sun.com/service/gdd/index.xml
4) Please send all Bugs and RFE's to the following address:
Subject "pkg_app bug/rfe" - gdd-issue-tracker-AT-sun.com
5) Please send all other questions etc to:
Subject "pkg_app feedback" - gdd-feedback-AT-sun.com
------------------------------end Outputs ------------------------------
apple:/export/home/bits/iws/misc/pkg_app-dir> ls -lrt
total 48702
-rw-r--r-- 1 root other 3581 Sep 12 2007 README.txt
-rw-r--r-- 1 root other 3461 Sep 12 2007 LICENSE.txt
-rwxr-xr-x 1 root other 19285 Oct 2 2007 pkg_app
-rwxr-xr-x 1 root other 3329 Oct 2 2007 README.pkg_app
-rw-rw-rw- 1 root other 24871809 Jul 9 15:47 pkg_app83450f78-apple-080709-154653.tar.Z
then send above .Z file to Sun and then we can check out more about the web server running process or core dump file.
The inside of the .Z file is:
apple:/export/home/bits/iws/misc/pkg_app-dir/pkg-app-dump.tmp> ls -lrt
total 110
-rw-rw-rw- 1 root other 0 Jul 9 15:46 pkg_app.info
-rw-rw-rw- 1 root other 0 Jul 9 15:46 pkg_app.errors
-rw-rw-rw- 1 root other 2508 Jul 9 15:46 pkg_app.pldd
-rw-rw-rw- 1 root other 24729 Jul 9 15:46 pkg_app.pstack
-rw-rw-rw- 1 root other 16639 Jul 9 15:46 pkg_app.pmap
-rw-rw-rw- 1 root other 46 Jul 9 15:47 pkg_app.pcred
-rw-rw-rw- 1 root other 2560 Jul 9 15:47 pkg_app.pflags
-rwxr-xr-x 1 root other 443 Jul 9 15:47 opencore
-rw-rw-rw- 1 root other 614 Jul 9 15:47 dbxrc
-rw-rw-rw- 1 root other 1137 Jul 9 15:47 runtime-080709-154653.log
drwxrwxrwx 2 root other 2048 Jul 9 15:50 app
and remember to send in any core dump files or gcore files to Sun support with above .Z file.
Then we can check out more about the running conditions and states inside the webservd at that time.
|
|
|
|
