やっぱり Sun がスキ！

Solaris Crash Analysis Tool 5.1

システムの panic 等で採取される crash dump (/var/crash 配下に格納される)ですが、この中身を覗いてみたくなったこととかありませんか？
今回、ご紹介する Solaris Crash Analysys Tool (以下 SolarisCAT)は、採取された crash dump を解析するためのツールとなり、Solaris 2.6 から Solaris 10 および OpenSolaris をサポートしています。また、採取済みの crash dump 以外にも、稼働中のシステムの情報を採取することも可能です。
(CPU アーキテクチャが異なる環境では、crash dump の解析ができませんのでご注意ください。たとえば、SPARC 上で採取されたものを x86 環境で解析することはできず、その逆も同様となります。)

Solaris CAT は、下記 URL から入手可能です。

Solaris Crash Analysis Tool 5.1
https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_SMI-Site/en_US/-/USD/ViewProductDetail-Start?ProductRef=SCAT-5.1-G-F@CDS-CDS_SMI

パッケージ形式での配布となりますので、download 後に pkgadd コマンドでシステムにインストールすることになります。コマンド群は、/opt/SUNWscat に配置され、マニュアルは HTML ファイル (/opt/SUNWscat/docs) にて提供されます。基本的なインターフェースは、CUI (/opt/SUNWscat/bin/scat コマンド)となり対話形式にて操作することが可能です。 引数を与えずに scat を起動すると下記のメッセージが表示されます。

--
# /opt/SUNWscat/bin/scat            

  Solaris[TM] CAT 5.1b for Solaris 11 64-bit x64
    SV4842, Feb 24 2009

  Copyright 2009 Sun Microsystems, Inc. All rights reserved.
  Use is subject to license terms.

  Feedback regarding the tool should be sent to SolarisCAT_Feedback@Sun.COM
  Visit the Solaris CAT blog at http://blogs.sun.com/SolarisCAT

opening /dev/ksyms /dev/kmem ...symtab...core...done
loading core data: modules...symbols...CTF...done

files:		/dev/ksyms /dev/kmem
user:		Super-User (root:0)
release:	5.11 (64-bit)
version:	snv_114
machine:	i86pc
node name:	macbook
system type:	i86pc
hostid:		4fd7cf4c
dump_conflags:	0x10000 (DUMP_KERNEL) on /dev/zvol/dsk/rpool/dump(1G)
time in kernel:	Thu May 28 11:35:29 JST 2009
age of system:	3 hours 28 minutes 53.88 seconds
CPUs:		2 (3.72G memory)

sanity checks: settings...vmem...sysent...misc...done
SolarisCAT(live/11X)> 
--

試しに、採取された crash dump を覗いてみるには、下記のように実行します。analyze コマンドは、crash dump を自動的に解析し、原因究明への一歩を手助けしてくれる情報を出力してくれます。

--
# /opt/SUNWscat/bin/scat /var/crash/$HOSTNAME/vmcore.0

SolarisCAT(vmcore.0/11X)> analyze

core file:	/var/crash/macbook/vmcore.0
user:		Super-User (root:0)
release:	5.11 (64-bit)
version:	Apr_30_11:53_2009
machine:	i86pc
node name:	macbook
system type:	i86pc
hostid:		4fd7cf4c
dump_conflags:	0x10000 (DUMP_KERNEL) on /dev/zvol/dsk/rpool/dump(1G)
time of crash:	Thu Apr 30 19:48:38 JST 2009 (core is 27 days old)
age of system:	4 minutes 25.10 seconds
panic CPU:	1 (2 CPUs, 3.72G memory)
panic string:	BAD TRAP: type=e (#pf Page fault) rp=ffffff0008da6a30 addr=20 occurred in module "unix" due to a NULL pointer dereference


==== checking for trap information ====

CPU 1 had the panic


==== panic thread: 0xffffff01be4aa000 ==== CPU: 1 ====
==== panic kernel thread: 0xffffff01be4aa000  PID: 0  on CPU: 1 ====
cmd: 
t_procp: 0xffffff01cffe7e10
  p_as: 0x0t_stk: 0xffffff0008da6f10  sp: 0xffffff0008da6790  t_stkbase: 0x0
t_pri: 60(FX)  pctcpu: 0.000000
t_lwp: 0x0  address translation failed for cpupart: 320 bytes @ 0xfed00000
bound psrset: -70932464  last CPU: 1  
idle: 1092044344150 ticks (126394 days 30 minutes 41.50 seconds)
start: Wed Jul 19 11:24:56 .*.*
age: 1092968065422 seconds (12650093 days 8 hours 23 minutes 42 seconds)
tstate:	TS_ONPROC - thread is being run on a processor
tflg:	T_PANIC - thread initiated a system panic
	T_DFLTSTK - stack is default size
tpflg:	TP_TWAIT - wait to be freed by lwp_wait
	TP_MSACCT - collect micro-state accounting information
tsched:	TS_LOAD - thread is in memory
	TS_DONT_SWAP - thread/LWP should not be swapped
	TS_SIGNALLED - thread was awakened by cv_signal()
pflag:	none set

pc:      unix:vpanic_common+0x13b:  addq   $0xf0,%rsp

unix:vpanic_common+0x13b()
unix:panic+0x94()
unix:die+0xdd()
unix:trap+0x175f()
unix:cmntrap_pushed+0x3d()
unix:mutex_enter+0xb()
klmmod:lm_svc+0x45()
nfs:nfssys+0x448()
unix:_sys_sysenter_post_swapgs+0x14b()
-- end of kernel thread's stack --


==== analyzing panic thread stack for trap frames ====

  no trap frames found
--

また、稼働中のシステムの様々な情報も取得することが可能です。
例として、メモリ情報やページ情報を確認するには、下記のように実行します。

[メモリ情報の確認]

--
SolarisCAT(live/11X)> meminfo
                              pages        bytes
physinstalled                976919   4001460224 (3.72G)
physmem                      974870   3993067520 (3.71G)
total_pages                  974869   3993063424 (3.71G)

freemem                      375851   1539485696 (1.43G)
avefree                      381934   1564401664 (1.45G)
avefree30                    374878   1535500288 (1.43G)
availrmem (nonswapable)      683612   2800074752 (2.60G)
availrmem_initial            974869   3993063424 (3.71G)
swapfs_minfree               121858    499130368 (476M)
sw_pending_size                            40960 (40K)

lotsfree                      15232     62390272 (59.5M)
desfree                        7616     31195136 (29.7M)
minfree                        3808     15597568 (14.8M)
throttlefree                   3808     15597568 (14.8M)

lookup failed for symbol segvn_pages_locked: symbol not found
pp_kernel(calculated)        290845   1191301120 (1.10G)
obp_pages                       512      2097152 (2M)
kcage_on:                0

shared memory (SM)                        540672 (528K)
intimate SM (ISM)                              0 (0)
dynamic ISM (DISM)                             0 (0)
locked DISM                       0            0 (0)
total locked SM                                0 (0) (0.00% of memory)
spt_used (ISM)                    0            0 (0)
segspt_minfree                    0            0 (0)

anoninfo: (physical == disk-backed)
  ani_max - total reservable physical swap                 488447 pages (1.86G)
  ani_free - unallocated physical and memory               220853 pages (862M)
  ani_phys_resv - reserved physical                        341070 pages (1.30G)
  ani_mem_resv - reserved memory                                0 pages (0)
  ani_locked_swap - swap locked in reserved mem swap            0 pages (0)

initial virtual swap available for reservation            1341458 pages (5.11G)
  ani_max + MAX((availrmem_initial - swapfs_minfree), 0)
current virtual swap available for reservation             709131 pages (2.70G)
  (ani_max - ani_phys_resv) + MAX((availrmem - swapfs_minfree), 0)

swap device                    pages           free
/dev/zvol/dsk/rpool/swap  488447 (1.86G)  488447 (1.86G)

tmpfs:
tmount          size  mount point
0xffffff01bc091140   456K  /etc/svc/volatile
0xffffff01be771918  6.10M  /tmp
0xffffff01be80a108    52K  /var/run
--

[ページ情報の確認]

--
# /opt/SUNWscat/bin/scat            
SolarisCAT(live/11X)> page summary
reading page table...
reading page_freelists...                                                     
reading page_cachelists...                                                    
reading page_hash...                                                          
                                                                              
total:               974869 (3.71G)
mapped:              545810 (2.08G)
unmapped:            429059 (1.63G)
modified:            238948 (933M)
read-only:                0 (0)
locked in memory:    273714 (1.04G)
soft locked:              0 (0)
shared-locked:       291166 (1.11G)
excl-locked:             63 (252K)
IO_INUSE:                 1 (4K)
FREE pages:          380541 (1.45G)
AGED pages:          230701 (901M)
NORELOC pages:            0 (0)

kernel(kvp):         171113 (668M)
anon:                242316 (946M)
exec:                  9163 (35.7M)
page cache:          168381 (657M)
page cache(mod):       1687 (6.58M)
free (cachelist):    149840 (585M)
NULL vnode !free:      3355 (13.1M) (ramdisk?)

free (freelist):     230701 (901M)

freelist:            227066 pages
       pages      pagesize
  =============== ========
   227059 (886M)  4K
        7 (14M)   2M

  total size:     944713728 (900M)
  constituent:         3577 pages

cachelist:           149429 (583M)

hashed:              740874 (2.82G)
       pages      pagesize
  =============== ========
   719370 (2.74G) 4K
    21504 (84M)   2M

     pages         modified         mapped      vnode
=============== =============== =============== =====
 234056 (914M)        0               1 (4K)    0
 171113 (668M)        0          141084 (551M)  unix(data):kvp+0x0  
 114705 (448M)        0          114705 (448M)  unix(data):zvp+0x0  
  24522 (95.7M)       0             510 (1.99M) 0xffffff01cd37f100  
   8688 (33.9M)       0             158 (632K)  0xffffff01cf1ca580  
   7914 (30.9M)       0            3791 (14.8M) 0xffffff01d0d33600  
   6160 (24M)         0            5214 (20.3M) 0xffffff01ccebe080  
...

     pages         modified         mapped      vnode ops
=============== =============== =============== =========
 285819 (1.09G)       0          255789 (999M)  0
 265316 (1.01G)  237261 (926M)   241334 (942M)  *genunix(bss):swap_vnodeops
 152844 (597M)        0           46491 (181M)  *zfs(bss):zfs_fvnodeops
  22657 (88.5M)       0            2168 (8.46M) 0xffffff01bb20b380
  12487 (48.7M)       0               0         *nfs(bss):nfs4_vnodeops
   1687 (6.58M)    1687 (6.58M)      27 (108K)  *tmpfs(bss):tmp_vnodeops
      3 (12K)         0               0         *specfs(bss):spec_vnodeops

--

Posted at 03:00午後 5 28, 2009 by masahiko in Sun  |  投稿されたコメント[0]