Musings of a socket scientist

Yesterday the Sun office in Somerset, NJ was home (for a day) to the high school students from the NJ Governor's School of Engineering and Technology Program. I had the opportunity to share my thoughts on the topic of Open Source and in particular, Sun's contribution to it, in a talk titled "Open Source, Open Minds" to these students.

Over 3 sessions, I got to interact with about 30 students. As I went through my presentation I was amazed by how much they already knew. For example, some of them not only know what IDE meant but actually had downloaded netbeans and used it! There were some who even correctly guessed the open source business model (generating revenue through service, support and training).

Their eyes lit up when we had Sun giveaways that were doled out whenever they asked great questions or answered questions really well. I have to thank Teresa Giacomini(from the OpenSolaris group) for these cool openSolaris giveaways. The openSolaris wifi locator was probably the coolest and a hit amongst the students. The T-shirts were not any less popular- one of the students even asked me if he could buy an openSolaris T-shirt online someplace :-)

My partners in crime were Wayne Abbott and John Cecere who handled 3 sessions each; be sure to check out their blogs in case they have some interesting insights from the experience yesterday.

A couple of pictures from yesterday are below. Unfortunately, Wayne was the one taking the pictures, so he's not in here!

Speculative tracing is a useful way to trace code paths in case of errors in the code path. I've also found it useful to trace codepaths, in general (which is useful in understanding the code). For example, if the the data path taken by network packet, as it starts from the top of the network stack (stream head) and all the way down to the driver, is of interest, then a simple script like the one below does the trick:

#!/usr/sbin/dtrace  -Fs
#pragma D option nspec=10

fbt:sockfs:sostream_direct:entry
{
        self->spec = speculation();
        speculate(self->spec);
}

fbt:::
/self->spec/
{
        speculate(self->spec);
}

fbt:ce:ce_wput:entry
/self->spec/
{
        commit(self->spec);
        self->spec = 0;
}

With some basic knowledge of what you want as the entry and exit points of the path, you can create a speculation at the entry point (sostream_direct, a routine in the stream head, in the case above) and commit the speculation at the exit point (ce_wput, the cassini network drivers write route, in the example above). The output for an ftp data packet (i.e the TCP data path) looks like this:

# /d/spec_w.d
dtrace: script '/d/spec_w.d' matched 39427 probes
CPU FUNCTION
  0  -> sostream_direct
  0    -> mcopyinuio
  0      -> allocb_cred
  0        -> allocb
  0          -> kmem_cache_alloc
  0            -> kmem_cache_alloc_debug
  0              -> verify_and_copy_pattern
  0              <- verify_and_copy_pattern
  0              -> dblk_constructor
  0                -> kmem_cache_alloc
  0                  -> kmem_cache_alloc_debug
  0                    -> verify_and_copy_pattern
  0                    <- verify_and_copy_pattern
  0                    -> getpcstack
  0                    <- getpcstack
  0                    -> kmem_log_enter
  0                    <- kmem_log_enter
  0                  <- kmem_cache_alloc_debug
  0                <- kmem_cache_alloc
  0              <- dblk_constructor
  0              -> getpcstack
  0              <- getpcstack
  0              -> kmem_log_enter
  0              <- kmem_log_enter
  0            <- kmem_cache_alloc_debug
  0          <- kmem_cache_alloc
  0        <- allocb
  0        -> crhold
  0        <- crhold
  0      <- allocb_cred
  0      -> uiomove
  0        -> xcopyin_nta
  0        <- xcopyin_nta
  0      <- uiomove
  0    <- mcopyinuio
  0    -> tcp_wput
  0      -> msgdsize
  0      <- msgdsize
  0      -> mutex_owned
  0      <- mutex_owned
  0      -> conn_trace_ref
  0        -> mutex_owned
  0        <- mutex_owned
  0        -> getpcstack
  0        <- getpcstack
  0      <- conn_trace_ref
  0    <- tcp_wput
  0    -> squeue_enter
  0      -> servicing_interrupt
  0      <- servicing_interrupt
  0      -> mutex_owned
  0      <- mutex_owned
  0      -> tcp_output
  0        -> tcp_timeout
  0          -> conn_trace_ref
  0            -> mutex_owned
  0            <- mutex_owned
  0            -> getpcstack
  0            <- getpcstack
  0          <- conn_trace_ref
  0          -> timeout
  0          <- timeout
  0          -> timeout_common
  0            -> mutex_owned
  0            <- mutex_owned
  0          <- timeout_common
  0        <- tcp_timeout
  0        -> dupb
  0          -> kmem_cache_alloc
  0            -> kmem_cache_alloc_debug
  0              -> verify_and_copy_pattern
  0              <- verify_and_copy_pattern
  0              -> getpcstack
  0              <- getpcstack
  0              -> kmem_log_enter
  0              <- kmem_log_enter
  0            <- kmem_cache_alloc_debug
  0          <- kmem_cache_alloc
  0        <- dupb
  0      <- tcp_output
  0      -> tcp_send_data
  0        -> ire_trace_ref
  0          -> th_trace_ire_lookup
  0            -> mutex_owned
  0            <- mutex_owned
  0          <- th_trace_ire_lookup
  0          -> kmem_zalloc
  0            -> kmem_cache_alloc
  0              -> kmem_cache_alloc_debug
  0                -> verify_and_copy_pattern
  0                <- verify_and_copy_pattern
  0                -> getpcstack
  0                <- getpcstack
  0                -> kmem_log_enter
  0                <- kmem_log_enter
  0              <- kmem_cache_alloc_debug
  0            <- kmem_cache_alloc
  0          <- kmem_zalloc
  0          -> th_trace_rrecord
  0            -> getpcstack
  0            <- getpcstack
  0          <- th_trace_rrecord
  0        <- ire_trace_ref
  0        -> ire_to_ill
  0        <- ire_to_ill
  0        -> putnext
  0          -> mutex_owned
  0          <- mutex_owned
  0          -> mutex_owned
  0          <- mutex_owned
  0          -> ce_wput

Similarly,if the inbound data path is of interest, that is, as a packet is recieved on the network interface and travels all the way up to the stack, then reversing the roles in the script as show below does the trick:

#!/usr/sbin/dtrace  -Fs
#pragma D option nspec=10

fbt:ce:ce_intr:entry
{
        self->spec = speculation();
        speculate(self->spec);
}

fbt:::
/self->spec/
{
        speculate(self->spec);
}

fbt:genunix:strrput:entry
/self->spec/
{
        commit(self->spec);
        self->spec = 0;
}

And the output looks like this:

  0                              -> ce_intr
  0                                -> ddi_get_devstate
  0                                <- ddi_get_devstate
  0                                -> pci_dma_sync
  0                                  -> ddi_get_instance
  0                                  <- ddi_get_instance
  0                                  -> ddi_get_soft_state
  0                                  <- ddi_get_soft_state
  0                                  -> ddi_driver_name
  0                                    -> ddi_driver_major
  0                                    <- ddi_driver_major
  0                                    -> ddi_major_to_name
  0                                    <- ddi_major_to_name
  0                                    -> mod_major_to_name
  0                                    <- mod_major_to_name
  0                                  <- ddi_driver_name
  0                                  -> ddi_get_instance
  0                                  <- ddi_get_instance
  0                                  -> pci_debug
  0                                  <- pci_debug
  0                                  -> pci_debug
  0                                  <- pci_debug
  0                                  -> pci_debug
  0                                  <- pci_debug
  0                                  -> pci_pbm_dma_sync
  0                                  <- pci_pbm_dma_sync
  0                                <- pci_dma_sync
  0                                -> ce_dupb
  0                                  -> desballoc
  0                                  <- desballoc
  0                                  -> gesballoc
  0                                    -> kmem_cache_alloc
  0                                      -> kmem_cache_alloc_debug
  0                                        -> verify_and_copy_pattern
  0                                        <- verify_and_copy_pattern
  0                                        -> dblk_esb_constructor
  0                                          -> kmem_cache_alloc
  0                                            -> kmem_cache_alloc_debug
  0                                              -> verify_and_copy_pattern
  0                                              <- verify_and_copy_pattern
  0                                              -> getpcstack
  0                                              <- getpcstack
  0                                              -> kmem_log_enter
  0                                              <- kmem_log_enter
  0                                            <- kmem_cache_alloc_debug
  0                                          <- kmem_cache_alloc
  0                                        <- dblk_esb_constructor
  0                                        -> getpcstack
  0                                        <- getpcstack
  0                                        -> kmem_log_enter
  0                                        <- kmem_log_enter
  0                                      <- kmem_cache_alloc_debug
  0                                    <- kmem_cache_alloc
  0                                  <- gesballoc
  0                                <- ce_dupb
  0                                -> dvma_sync
  0                                <- dvma_sync
  0                                -> pci_fdvma_sync
  0                                  -> pci_debug
  0                                  <- pci_debug
  0                                <- pci_fdvma_sync
  0                                -> pci_dma_sync
  0                                  -> ddi_get_instance
  0                                  <- ddi_get_instance
  0                                  -> ddi_get_soft_state
  0                                  <- ddi_get_soft_state
  0                                  -> ddi_driver_name
  0                                    -> ddi_driver_major
  0                                    <- ddi_driver_major
  0                                    -> ddi_major_to_name
  0                                    <- ddi_major_to_name
  0                                    -> mod_major_to_name
  0                                    <- mod_major_to_name
  0                                  <- ddi_driver_name
  0                                  -> ddi_get_instance
  0                                  <- ddi_get_instance
  0                                  -> pci_debug
  0                                  <- pci_debug
  0                                  -> pci_debug
  0                                  <- pci_debug
  0                                  -> pci_debug
  0                                  <- pci_debug
  0                                  -> pci_pbm_dma_sync
  0                                  <- pci_pbm_dma_sync
  0                                <- pci_dma_sync
  0                                -> hcksum_assoc
  0                                <- hcksum_assoc
  0                                -> canputnext
  0                                <- canputnext
  0                                -> putnext
  0                                  -> mutex_owned
  0                                  <- mutex_owned
  0                                  -> mutex_owned
  0                                  <- mutex_owned
  0                                  -> ip_rput
  0                                  <- ip_rput
  0                                  -> ip_input
  0                                    -> ire_cache_lookup
  0                                      -> ire_trace_ref
  0                                        -> th_trace_ire_lookup
  0                                          -> mutex_owned
  0                                          <- mutex_owned
  0                                        <- th_trace_ire_lookup
  0                                        -> kmem_zalloc
  0                                          -> kmem_cache_alloc
  0                                            -> kmem_cache_alloc_debug
  0                                              -> verify_and_copy_pattern
  0                                              <- verify_and_copy_pattern
  0                                              -> getpcstack
  0                                              <- getpcstack
  0                                              -> kmem_log_enter
  0                                              <- kmem_log_enter
  0                                            <- kmem_cache_alloc_debug
  0                                          <- kmem_cache_alloc
  0                                        <- kmem_zalloc
  0                                        -> th_trace_rrecord
  0                                          -> getpcstack
  0                                          <- getpcstack
  0                                        <- th_trace_rrecord
  0                                      <- ire_trace_ref
  0                                    <- ire_cache_lookup
  0                                    -> ip_tcp_input
  0                                      -> ipcl_classify_v4
  0                                        -> conn_trace_ref
  0                                          -> mutex_owned
  0                                          <- mutex_owned
  0                                          -> getpcstack
  0                                          <- getpcstack
  0                                        <- conn_trace_ref
  0                                      <- ipcl_classify_v4
  0                                    <- ip_tcp_input
  0                                    -> ire_refrele
  0                                      -> ire_untrace_ref
  0                                        -> th_trace_ire_lookup
  0                                          -> mutex_owned
  0                                          <- mutex_owned
  0                                        <- th_trace_ire_lookup
  0                                        -> th_trace_rrecord
  0                                          -> getpcstack
  0                                          <- getpcstack
  0                                        <- th_trace_rrecord
  0                                        -> ire_trace_free
  0                                        <- ire_trace_free
  0                                        -> kmem_free
  0                                        <- kmem_free
  0                                        -> kmem_cache_free
  0                                          -> kmem_cache_free_debug
  0                                            -> kmem_log_enter
  0                                            <- kmem_log_enter
  0                                            -> getpcstack
  0                                            <- getpcstack
  0                                            -> kmem_log_enter
  0                                            <- kmem_log_enter
  0                                            -> copy_pattern
  0                                            <- copy_pattern
  0                                          <- kmem_cache_free_debug
  0                                        <- kmem_cache_free
  0                                      <- ire_untrace_ref
  0                                    <- ire_refrele
  0                                    -> squeue_enter_chain
  0                                      -> servicing_interrupt
  0                                      <- servicing_interrupt
  0                                      -> mutex_owned
  0                                      <- mutex_owned
  0                                      -> tcp_input
  0                                      <- tcp_input
  0                                      -> tcp_rput_data
  0                                        -> tcp_find_pktinfo
  0                                        <- tcp_find_pktinfo
  0                                        -> tcp_process_options
  0                                          -> tcp_parse_options
  0                                          <- tcp_parse_options
  0                                          -> tcp_mss_set
  0                                          <- tcp_mss_set
  0                                          -> tcp_maxpsz_set
  0                                            -> setmaxps
  0                                              -> claimstr
  0                                              <- claimstr
  0                                              -> strqset
  0                                                -> mutex_owned
  0                                                <- mutex_owned
  0                                              <- strqset
  0                                              -> releasestr
  0                                                -> cv_broadcast
  0                                                <- cv_broadcast
  0                                              <- releasestr
  0                                            <- setmaxps
  0                                            -> mi_set_sth_maxblk
  0                                              -> allocb
  0                                                -> kmem_cache_alloc
  0                                                  -> kmem_cache_alloc_debug
  0                                                    -> verify_and_copy_pattern
  0                                                    <- verify_and_copy_pattern
  0                                                    -> dblk_constructor
  0                                                      -> kmem_cache_alloc
  0                                                        -> kmem_cache_alloc_debug
  0                                                          -> verify_and_copy_pattern
  0                                                          <- verify_and_copy_pattern
  0                                                          -> getpcstack
  0                                                          <- getpcstack
  0                                                          -> kmem_log_enter
  0                                                          <- kmem_log_enter
  0                                                        <- kmem_cache_alloc_debug
  0                                                      <- kmem_cache_alloc
  0                                                    <- dblk_constructor
  0                                                    -> getpcstack
  0                                                    <- getpcstack
  0                                                    -> kmem_log_enter
  0                                                    <- kmem_log_enter
  0                                                  <- kmem_cache_alloc_debug
  0                                                <- kmem_cache_alloc
  0                                              <- allocb
  0                                              -> putnext
  0                                                -> mutex_owned
  0                                                <- mutex_owned
  0                                                -> mutex_owned
  0                                                <- mutex_owned
  0                                                -> strrput
 

Notice, how the packet travels up the stack starting from ce (the network driver), to IP (ip_* functions), to TCP (tcp_* functions) before finally landing into the stream head (strrput).

Technorati Tag: OpenSolaris
Technorati Tag: Solaris
Technorati Tag: dtrace

If you have ever used an ifconfig command to plumb a network interface like: >ifconfig ce0 plumb and wondered what the internal STREAMS plumbing looks like, then read on. This blog entry is especially for those who are interested in a simplified view of the network interface plumbing and data flow in Solaris. The plumbing, for those interested in looking at code, happens in plumb_one_device(). First, ifconfig opens the network device: Fig 1: ifconfing opening a stream to the network interface.

---

It then pushes IP on top of the network device. This forms the stream over which IP packets are sent and recieved: Fig 2: IP pushed on top the network device.

---

Assuming that a resolver is needed, an ARP module is then pushed on this stream to form the IP-ARP stream. This allows IP to send queries direclty to ARP and process responses from it: Fig 3: ARP pushed on top of IP.

---

If ARP is needed (as in this example), then another stream to the device is opened: Fig 4: Opening another stream for ARP

---

And ARP pushed on top of it. This forms the ARP-device stream over which the actual ARP queries are sent out and responses processed from the wire: Fig 5: ARP pushed on top of the network device.

---

To allow ifconfig to exit, these two streams are PLINKed underneath a multiplexor stream: Fig 6: PLINK below the mux

---

Since PLINK is used, ifconfig can close the corresponding file descriptors and exit: Fig 7: ifconfig exits

---

The above diagram describes the "lower half" of setup. To understand how a packet actually travels from the application out to the wire and vice versa, one needs to see the other half (i.e "upper half") of the setup that happens when a socket is opened by the application: Fig 8: Final plumbing showing data path

---

On the outbound side (packet going from application to the wire), the packet travels upto IP where a routing lookup is done to determine the correct "network interface" (i.e "lower half") to send the packet out on. Accordingly the packet is placed in the correct output queue. On the inbound side (packet coming in from the wire), the packet is "fanned out" to the correct "upper connection" based on five-tuple (src IP addr, dest IP addr, src port, dst port, protocol) match. The bi-directional arrow highlights both the paths.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

Race conditions can be difficult to debug since they represent
a sequence of events that have to happen in a particular order for the
condition to manifest itself. This blog describes the debugging
process which revealed the sequence of events that lead to the race,
how this also allowed to reproduce the problem (with some
instrumentation)  and ultimately allowing the verification of the fix.

Bug 4986151 was filed some time ago stating that a  internal test
suite was hanging while executing the following command:

ifconfig qfe0 inet6 up

The key, though, was that this hang would happen once in a while (maybe
once in 30 times).

Thirumalai (our resident networking guru) quickly debugged the dump
and pointed out that there was a missing "reference release" that was
causing the hang. Thinking that this would be an easy fix, I happily took
this bug  on; only to have Thirumalai throw a curveball- "You need to find
out the exact set of circumstances that caused this and reproduce this
so you can verify the fix" :-). So much for an easy fix...

But I was up for the challange, so dove into the dump with mdb.
First I took a look at the state of the "network interface" qfe0 under
question:

> 30003802fe8::print ill_t ill_ipif|::print ipif_t ipif_flags         
ipif_flags = 0x8020000

ILL in the ill_t stands for "IP lower level" and represents "inteface
endpoint" as seen by IP. Each ILL can have "ipifs" which represent
the logical instances (eg: qfe0:1) of the physcical interface.

The address of the ILL (i.e 30003802fe8) is obtained by walking the
ILLs via mdb as seen below

So it can be seen that the IFF_NOFAILOVER and the IFF_NOLOCAL
flags are set, looking at the values in if.h

The IFF_NOLOCAL flag is set by ifconfig during the duration of DAD
(duplicate address detection) when bringing the interface 'up'.

Looking closely at the ifconfig source it can been seen that DAD calls
SIOCSLIFFLAGS (the hung ioctl in question) twice. Once to set the
IFF_NOLOCAL|IFF_UP flags (so that it can do the DAD operation by
sending/recieving packets) and then the second time to restore the
original flags (once DAD is completed). SIOCSLIFFLAGS is an ioctl
used to set flags on a network interface.

It is the second SIOCSLIFFLAGS that is actually hung. To understand
this one needs to understand that 'ifconfig <interface> up' actually
triggers first a down on the interface and then an up (in the kernel).

> 30003802fe8::print ill_t ill_dl_up
ill_dl_up = 1

The ill_dl_up being set to TRUE and the ipif_flags being set to
IFF_NOLOCAL indicate that the first SIOCSLIFFLAGS operation was
successful completely.

The ill_dl_up is set in latter part of the up
operation i.e in ip_rput_dlpi_writer (called in response to the
DL_BIND_REQ sent down by ill_dl_up).

When the second SIOCSLIFFLAGS is issued, first the down on the
interface is initiated. The ipif_down function, clears the IPIF_UP
flag, destroys the multicast memberships, deletes IREs associated
with the interface, clears NCEs (neighbor cache entries) and then checks if
the ipif is quiescent (i.e no outstanding refrences on the ILL).

It is the last step that causes the command to hang since the ILL has an
has an outstanding reference count and the ioctl message is queued for
later processing (once the reference count drops to zero but it never does
because of the missing reference release). Note all of what's stated in the
paragraph can be observed to be true by looking at the state of the ipif/ill
the dump (indicating that the down had progressed till the ipif_is_quiescent
check)

> 30003802fe8::print ill_t ill_ipif_up_count
ill_ipif_up_count = 0
30003802fe8::print ill_t ill_ipif|::print ipif_t ipif_flags
ipif_flags = 0x8020000 (note: IFF_UP is 0x1 which is not set)
30003802fe8::print ill_t ill_ipif|::print ipif_t ipif_multicast_up
ipif_multicast_up = 0
30003802fe8::print ill_t ill_nce_cnt                       

ill_nce_cnt = 0
So, to summarize, here is the chronological sequence of events that trigger the
hang:

thread I (the ifconfig thread):

1> 'ifconfig qfe0 inet6 up' triggers the DAD. DAD sets IFF_NOLOCAL
and IFF_UP (via SIOCSLIFFLAGS) in order to send/recieve packets
related to DAD.

2> Setting the IFF_UP flag actually causes the DL_BIND exchange
between IP and the qfe driver and this allows packets to be
sent/recieved.

3> ifconfig completes DAD

<thread II (see below) executes somewhere here>

4> ifconfig issues the second SIOCSLIFFLAGS
to restore the flags. This first intiates the ipif to be brought down
(via ipif_down) and this involves (in order):

a> clearing the IPIF_UP flag and the ill_ipif_up_count
b> destroying the mcast memberships
c> deleting the IREs associated with the interface
d> bringing NDP down (i.e deleting the NCEs)
e> checking if the ipif is quiescent
f> queuing the request if ipif is not quiescent (and returns EINPROGRESS)


thread II (the inbound ICMP thread):

1> An ICMP packet arrives on qfe0 and is one to which there is a
reply, so it ends up in ip_output_v6. However, there is no IRE_CACHE
for this destination so ip_newroute_v6 is called. Note, IRE stands for
Internet Route Entry and every address in the system has an an
internal IRE structure associated with it. IRE_CACHE represent
those "resolved entries" i.e there is complete information available
to send a packet whose destination address is available as an IRE_CACHE.
ip_newroute_v6 intiates the process to create such a complete
entry if one doesn't exist by contacting an external resolver like
ARP for example to get the MAC address of the destination in
question.

2> An ire_ftable_lookup_v6 is done to find a matching route that can
reach the destination

3> A corresponding destination ILL is looked up

4> An attempt is made to locate an appropriate source IRE but this
fails (ipif_select_source_v6 returns NULL since IFF_NOLOCAL is set
on the ipif) and the code branches to the icmp_err_ret failure path

5> In the failure path, ip_hdr_complete_v6 fails since there is no
local/loopack v6 IREs and the reference release on the ILL is not
done. That is,  it leaves the ILL with an outstanding reference
count.

Thread II executes somewhere between step 3 and 4 of thread
I.

To reproduce the problem (and to confirm the above theory), ifconfig
is intrumented such that a delay is introduced between the first
and the second SIOCSLIFFLAGS during the dad operation (in do_dad).
This opens the race window in which the failure path can be
exercised. To exercise the failure path a local packet (say a v6 ping)
is sent to remote link local destination.

On solar3 (the test system) we run the instrumented ifconfig.

solar3# /ifconfig.test qfe0 inet6 plumb
solar3# /ifconfig.test qfe0 inet6 up
ifconfig sleeping for 30 secs
In the 30 sec window a ping is intitiated:
solar3# ping fe80::a00:20ff:fec4:c593
^C

The ping is interrupted after a few seconds.

With ip debug turned on, the following messages are seen:

----

Mar 15 17:20:35 solar3 ip: [ID 844449 kern.notice] ip_newroute_v6: ire_ftable_lookup_v6() returned ire 300017d0c28, sire 0
Mar 15 17:20:35 solar3 ip: [ID 357465 kern.notice] ipif_select_source_v6(qfe0, fe80::a00:20ff:fec4:c593) -> NULL
Mar 15 17:20:35 solar3 ip: [ID 787518 kern.notice] ip_newroute_v6: no src for dst fe80::a00:20ff:fec4:c593
Mar 15 17:20:35 solar3 , 
Mar 15 17:20:35 solar3 ip: [ID 481217 kern.notice] ip_newroute_v6: interface name qfe0
Mar 15 17:20:35 solar3 ip: [ID 558731 kern.notice] ip_newroute_v6: no route
Mar 15 17:20:35 solar3 ip: [ID 354122 kern.notice] ip_hdr_complete_v6:
no source IRE
snipping several ping retries>

Mar 15 17:20:59 solar3 ip: [ID 326427 kern.notice] ipif_down: need to wait, adding pending mp qfe0 ill 3000164d4a8 Mar 15 17:20:59 solar3 ip: [ID 519129 kern.notice] ipif_down returns 150 err ---

and the ifconfig hangs as expected. Looking at the the live system:

solar3# mdb -k
Loading modules: [ unix krtld genunix sd ip usba nca nfs random ptm logindmux ]
> ::walk ill
3000164d728
3000164d4a8
3000164d4a8::print ill_t ill_isv6 ill_refcnt
ill_isv6 = 1
ill_refcnt = 0x4

We see that the ill_refcnt is non-zero. It is 0x4 in this case since
ping tries 20 times to send a packet before giving up (and the ping
was interrupted a few seconds after it is started)

Technorati Tag: OpenSolaris
Technorati Tag: Solaris
Technorati Tag: mdb

Welcome, and thanks for stopping by!

I am an engineer with the Solaris network performance team. The network performance team works on various performance related issues in the solaris networking (TCP/IP) stack.

I have been at Sun for a little over 8 years now- this being my first job out of grad school. I started on the Solaris 64-bit port project and then moved onto doing some work on PPP (point to point protocol). My interests later took me onto Mobile IP.

My most recent contirbution to solaris 10 was the virtual IP source address selection feature which facilitates IP multipathing in conjunction with routing protocols like OSPF. I plan to post some details on this in a later blog.

At the moment, I have been involved with making our IP forwarding table lookup faster using some creative lookup algorithms. I've also written a high speed traffic generator in the recent past (more on that later as well). I have also started to look into asynchronous sockets for solaris.

In life outside of work, besides spending time with my angels (my wife, 2 year old daughter and lab/terrier), I love volunteering. I have been involved with a non-profit Charityfocus) since it's inception (1997) where we have tried to leverage web technology for the 'greater good' through various projects. I also have been volunteering at the pathways hospice, providing companionship to terminally ill patients, for over 3 years.