Alan Hargreaves' Weblog

What does "panic: sync initiated" mean?

Twice in the last week I've received crashdumps for analysis with this panic string, and customers concerned that they need to replace hardware.

Let's just walk through an example.

On the system console we something like

panic[cpu0]/thread=2a10007dd20: sync initiated

sched: software trap 0x7f
pid=0, pc=0xf0050c7c, sp=0x2a10007cd41, tstate=0x8800001404, context=0x0
g1-g7: 1003a07c, 10418000, 10418000, 6b, 0, 0, 2a10007dd20

00000000fffa9d00 unix:sync_handler+150 (1041ae50, 10000000, b, 2a10004bd20, 2a10007dd20, 0)
  %l0-3: 0000000010027ff8 0000000000000003 0000000000000000 000002a1000abd20
  %l4-7: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00000000fffa9de0 unix:vx_handler+8c (f0000000, 10418348, 10418240, fff9b840, f003bda1, 0)
  %l0-3: 000000001002866c 0000000000000016 000000000000000a 000000001000a408
  %l4-7: 000000000000004e 00000300009af5ec 0000000000000006 000002a10007d6f0
00000000fffa9e90 unix:callback_handler+20 (fff9b840, fffde280, 0, 0, 0, 0)
  %l0-3: 0000000000000016 00000000fffa9741 0000000000000009 000000001000a408
  %l4-7: 000000001041b5b8 0000000000000000 0000000000000000 000002a10007d7b0

syncing file systems... 7 2 done
dumping to /dev/dsk/c0t0d0s1, offset 53805056

So we see that we are in the sync handler, which kind of makes sense given the panic string. What this actually means is that someone has typed

    sync

at the ok prompt.

OK, so how did we get to the ok prompt? Let's have a look at the stack.

    unix:client_handler+0x2c()
    unix:p1275_sparc_cif_handler(0x2a10007d6a8) - frame recycled
    unix:prom_enter_mon+0x38()
    unix:debug_enter+0x15c()
    unix:abort_sequence_enter(0x0) - frame recycled
    kb:kbdinput+0x304(0x300003edd68, 0x1046b000?)
    kb:kbdrput+0x13c(, 0x30000d92300)
    unix:putnext+0x1cc(0x300007a6e48?, 0x30000d92300)
    su:async_softint+0x58c(0x3000013c008)
    su:asysoftintr+0x70()
    unix:intr_thread+0xa4()
    unix:prom_rtt+0x0()

The important one here is abort_sequence enter().

This only gets called when the abort sequence has been entered on the console. If it's a serial console (like this one), this means <BREAK> or the alternate abort sequence if KEYBOARD_ABORT=alternate in /etc/default/kbd has been detected. On a Sun console, it's STOP-A.

So, what do we actually have here?

We have the system receiving the abort sequence, dropping the system to the ok prompt and then someone forcing a crashdump by typing sync.

Generally this would be a deliberate action, done to capture the system state of something like a hang condition for analysis.

The only other thing that occurs to me as a possibility is that a serial console is connected to something that dropped the DTR signal (eg a VDU or a terminal server).

There are a number of infodocs that discuss how to deal with this.

For example:

• Infodoc 16093: Configuration of serial consoles (headless) on Solaris[TM]
• Infodoc 13031: Annex Network Terminal Server (NTS) PSD/MFAQ
• Infodoc 15721: NTS as console server

The upshot of this is, with a panic: sync initiated core, it's unlikely you have broken hardware.

Posted at 08:06PM Oct 29, 2004 by Alan Hargreaves in Solaris  |  Comments[3]

Sun Education has a DTrace Course Available

How about that? For anyone who wants to do some training, Sun Education is running a two day course titled

Dynamic Performance Tuning and Troubleshooting with DTrace

The course code is SA-325-S10, and you can find out more about it at the Sun Education site.

The course content as listed on the site is...

• Describe the features of DTrace
• Describe the DTrace architecture

• Examining performance problems using DTrace
• Use DTrace to obtain information about system calls
• Create D Scripts that Use Arguments

• Use DTrace to profile an application
• Use DTrace to access application variables
• Use DTrace to find transient system call errors
• Use DTrace to determine the names of files being opened

• Use DTrace to access kernel variables
• Use DTrace to obtain information about read calls
• Use DTrace to do anonymous tracing
• Use DTrace to do speculative tracing
• Explain the privileges necessary to run DTrace operations

• Describe how to lessen the performance impact of DTrace
• Describe how to use and tune DTrace buffers
• Debug DTrace scripts

Posted at 07:19PM Oct 28, 2004 by Alan Hargreaves in Solaris Express  |  Comments[1]

Solaris Express 10/04 is coming

Solaris Express 10/04 (s10_69) should be available for download around October 29!

Some of the new features for the release include:

• Solaris Service Manager(Greenline)
• JDS Release 3
• Oban (Solaris Volume Manager for SunCluster)
• Ipsec/IKE NAT traversal
• OpenSSL and PKCS#11 for OpenSSL
• Hardware Random Number Generation support
• Failed Account Login Account Locking
• Track Uptime across reboots
• Solaris 1394 Mass Storage Driver-x86
• Remove Netscape4.7xin Solaris *Mozilla 1.4 removal
• Xorg replaces Xsun as the default X server on x86
• DTrace USDT (User-land Statically Defined Tracing) and plockstat Provider
• New Realtek NIC driver

As usual, the documentation will be available at docs.sun.com.

Greenline and the Xorg transition are likely to cause a little frustration. Greenline because it is such a complete mindset change, and Xorg in getting the configurations correct. I am expecting to see some of the respective engineers blogging on at least these two over the next few weeks.

Posted at 05:19PM Oct 26, 2004 by Alan Hargreaves in Solaris Express  |  Comments[13]

Two new articles in On#Sun

This issue I have two articles in On#Sun.

Why are Sun engineers so excited about Solaris 10?

Part 1 of a two part article briefly discussing some of the reasons why we are exited. In this part I touch on

• Fire Engine
• DTrace
• Zones
• KMDB

It was going to be a larger one part article, but after writing it I was informed of a two page limit and had to break it up.

Open Solaris

Hopefully answering some of the questions that folk have been asking on the topic.

On a side note, isn't it funny how they can get the spelling of your name right in one article, but not in another; in the same issue!

Posted at 08:17PM Oct 17, 2004 by Alan Hargreaves in Solaris Express  |  Comments[1]

Jake is student of the week

I came home the other night to find that Jake had been awarded "student of the week" in his kindergarten class.

This means that at a school assembly, he was awarded a blue ribbon with his name on it and the reason for receiving it.

I'm working from memory, but I think the reason went something like ...

Jacob is working hard to talk more quietly.

Jake has a tendancy to speak quite loudly when he gets excited. It's good to see him working on it, maybe the person he is imitating here (me) should also spend a bit of time on it ;)

Posted at 09:43PM Oct 15, 2004 by Alan Hargreaves in Kids  |  Comments[1]

Jonathan's in town

Jonathan is in Sydney this week and I managed to get to hear him speak.

I must say his enthusiansm is infectious. One of the challenges that he presented to the folk present was "Why are you here?". That is, why are you still working for Sun. I'm happy to say that some of the reasons that he gave echo mine very closely.

• I truly love the work that I am doing. If I didn't I'd move elsewhere.
• I have faith in Sun. I think that we now have some incredible technology, both in our hardware and software offerings. I believe that we can now provide solutions that our competitors simply cannot match.
• I know that we have some incredible people working for this company. Some true visionaries.
• By doing my damndest for Sun, in the area that I work in (Support Services), I am doing my part to bring this company back into profitability, which then helps me to provide for my family's future.

Everything that I am seeing is telling me that we do have an incredible future in front of us. Sun has been written off before, and each time it has re-invented itself and come back stronger. I feel like we are nearing the end of the re-invention and our competitors are going to have to watch themselves, or end up eating our dust!

On a side note, it was great to actually meet Jonathan, shake his hand and congratulate him on the work he is doing. Keep it up mate, and enjoy your time down-under!

Posted at 03:14PM Oct 11, 2004 by Alan Hargreaves in General  |  Comments[2]