If you are using T2000 servers :

While tuning & benchmarking an application on T2000, Sol 10, using QFS 4.5 and ST6140 we doubled I/O throughput (MB/s) and therefore application performance by 100%. The Sun StorageTek 6140 has a theoretical bandwidth of 800 MB/s, by increasing the pci-max-read-request and tuning filesystem and disk volume blocksizes the T2000 was able to exploit our storage to about 700MB/s. This was a I/O intensive app, thus the same performance gains may not be seen in other environments. I think that it could be useful anyway in general applications if we increase it from the default. However, do not increase to it's maximum value as this could do more harm than good.

A relatively unknown, "qlc.conf" parameter, specifically the "pci-max-read-request" has a very low default setting, this restricts I/O performance and therefore makes the T2000 server slow and also does not exploit the full capabilities of Sun storage. This can be fixed by setting the parameter to a higher value. This is only applicable to T2000 using PCIe.

Details below:

My recommendation: Set pci-max-read-request=2048 on all T2000 servers.

Parameter explanation:

Set in " /sysconfig/drv/qlc.conf "

#Name: PCI max read request override;
#Type: Integer, bytes; Range: 128, 256, 512, 1024, 2048, 4096
#Usage: This field specifies the value to be used for the PCI max read request, overriding the value programmed by the system.
#NOTE: The minimum value is 128 bytes; if this variable does not exist or is not equal to 128, 256, 512, 1024, 2048 or 4096, the ISP2xxx
# defaults to values specified by the system.
Background:
When you write data to a target, you are reading from PCI. The issue is that T2K defaults this to 128 bytes. To get best performance this has to be set to 4096. But its not that simple. SUN PCI folks warn that if we set this too high then we are eating up resources from other devices on the bus such as ethernet. There is currently a project going on in PCI team to make this process automatic. In the meantime folks have set this to 2048 without seeing any issues.

Other things to check for anyone optimizing I/O throughput on general Sun servers. These are the Solaris parameters that need attention if you want to improve throughput. Only change and test during development, not on production systems. As always use good project management and change control procedures when implementing any kernel changes.

Allow Solaris and Solaris disk drivers to perform the maximum size I/Os you anticipate. Setting these numbers lower than what the application and file system prefer, results in fragmentation of I/Os. Setting these numbers higher consumes more memory and may cause some very old disk or channel hardware to become unstable, but for recent large memory footprint machines and disks this is typically not an issue. The system must be booted for these changes to take effect:

Set the maximum I/O size for Solaris by editing the /etc/system file.
# /etc/system
# this sets maximum physical I/O size to 8MB
set maxphys = 0x800000

Set the maximum I/O size for fibre channel disks by editing the /kernel/drv/ssd.conf file:
ssd_max_xfer_size = 0x800000;

You have to set up /kernel/drv/sd.conf for individual SCSI disks:
name="sd" class="scsi"
sd_max_xfer_size=0x800000
target=3 lun=0;

Solaris Queue depth, how many SCSI commands can be queued to a LUN.
Set this too low and you are not getting max performance, too high and you can get device overruns, which then cause recovery processes to redrive/retry I/O and causes a slowdown, so normally stay with 64 (NB check with storage supplier as 3rd party equipment often is not optimized for multithreading as is Sun Storage).

ssd_max_throttle=64
(change sd_max_throttle for SCSI disks)

There may be other configuration files and option settings for non-Sun host bus adapters and drivers. See the third-party documents for more.

Caveat, 3rd party arrays often do not multi-thread and scale as well in performance as does Sun Storage. It is possible with large queues to “overrun” 3rd party storage. We have often had to reduce the queue depth (ssd_max_throttle) for slower 3rd party devices. I will not get into this as 3rd party or aftermarket suppliers will often disagree with me.

This is a personal recommendation, which will hopefully avoid me from documenting elsewhere and having to explain this several times a month.

This is not a Sun Engineering, or official Sun Solaris patch, fix etc.