Weblog

Thursday June 30, 2005

softcall processing in solaris

Points to note before i move to softcall processing (this is for sun4u only):

1. There exists an intr_vector array (each element of which is a 'struct intr_vector'
(refer to ivintr.h)). HW drivers register the interrupts that the HW
generates with this array. The index into this array will be the interrupt
number. And subsequently whenever that piece of HW interrups the CPU, it will
send this number.

2. A part of this array is used for softints. Softints are something which
are generated by the CPU itself (using SOFTINT register).

3. Again there exists an array called softcalls[], each element of which is
a 'struct softcall'  (refer to common/os/softint.c)

Softcall is a mechanism whereby kernel can execute a function at PIL-1.
How do we do this?

1. The first step is to make an entry in the above array (i.e intr_vector[])
using add_softintr(). add_softintr() returns the index of this entry.

	e.g: siron_inum=add_softintr(PIL_1, softlevel1,0)

2. From now on, we make a call to softcall() to execute a function at PIL-1.
What softcall does is: it adds an entry for this function in a softcall array -
and then makes a call to siron().

	e.g: softcall(callout_execute, ct);

What siron() does is, it calls setsoftint() 
	
	e.g: setsoftint(siron_inum)

what setsoftint() does is, it checks whether the pending flag is set
for this number (in this case its siron_inum) in the intr_vector, and if its
not set, then it sets a bit in the SOFTINT register/sets the pending flags, 
which triggers a level-1 interrupt when the PIL on the cpu goes below 1.

(one more thing it does is, takes off one free intr_req structure from the
freelist, fills it up with the interrupt number (i.e siron_number in this case)
and adds it to the list corresponding to this PIL - i.e to 
cpu.machcpu.intr_pool[])

3. Now when a level-1 interrupt occurs, 
	pil_interrup() {
		Takes off the first intr_req from the intr_pool[], note the
		intr number and return the intr_req struct to the free pool.

		intr_thread() {
			SERVE_INTR(this is a macro) {
				looks up the interrupt number in the intr_vector
				and calls the handler
				(i.e softlevel1 at index siron_inum)

					softlevel1() {
						
						softint() { }
					}
				}
			}
		}
	}

	what softint() does is, goes through the softcall array(i.e softcalls[])
	and calls all the handlers registered (e.g callout_execute() at line:28)

Caveats:

1. siron() doesn't call setsoftint() if a global flag siron_pending is already
set. This gets unset in softlevel1(). For this reason, we can't guarantee on 
which cpu a real-time callout runs.

(explained in detail: some other thread running on a different CPU may have called
softcall() and this resulted in siron_pending getting set and then it sets a
bit in SOFTINT registe of that cpu;
subsequently when clock runs and calls softcall(callout_execute), this results in
adding an entry to the softcall table, but as siron_pending is already set, this
just returns).
Subsequently when a level-1 interrupt happens on the first cpu, it will process
besides other entries, RT callouts as well.

2. One more thing to note is we are calling setsoftint() with siron_inum, and if
pending flag is alread set for this index in intr_vector[], it just retuns.

For this reason, in cyclics, it creates its own interrupt numbers for each cpu
for level1 and level10 interrupts.

Refer to cbe_configure() of sun4u/io/cbe.c
This way we don't depend on a softint that has to happen on a differet CPU.

3. cyclics also makes use of SOFTINT feature to generate soft interrupts, but
it doesn't go through the softcall array but what it does is:
	add_softintr(PIL_1, (intrfunc)cbe_level1, 0);
	cbe_softint() -> setsoftint(data->cbe_level1_inum);
	cbe_level1() -> cyclic_softint() -> (*handler)(arg)
More on this later..