Touch me!

Some Oracle experts say that paging is normal or even unavoidable in Oracle installation. I'll try to explain why this is not true.

• Paging increases response time. Average page-in time for single 8Kb page is measured in tenths of a second, and these tenths should be added to response time of your application as many times as many page-in events are occuring during query parsing, execution and results fetching.
• Paging sometimes dramatically increases response time. If you are using large pages for anything except non-pageable SGA, your memory becomes fragmented and in a case of paging-in of a single large page OS will be relocating small pages to construct a "hole" in a memory large enough to accomodate this large page. Page relocation consumes lot of CPU, requires excessive locking and consumes bus banwidth while moving pages across physical memory. Typical page relocation time vary depending on hardware configuration and memory pressure, but one cad add several seconds to response time of application if you are planning to keep your memory on a disk and relocate 100+ small pages to provide space for one large page being paged-in.
• Paging can be easily avoided by adequately sizing memory consumption of your application. In a case of Oracle database do not forget about total PGA of your processes. And remember: pga_aggregate_target is not a limit! There are no way to limit total PGA size by means of Oracle.
• The last, but not the least: frequent paging causes deterioration of disks containing swap area (typically - boot disks).

When planning memory configuration of Oracle instances one should consider the following:

• Solaris will start paging if amount of free memory drops below lotsfree, which has default value of 1/64 of physical RAM installed in machine. Main indicator of paging is any non-zero value in sr column of vmstat output. Also you can observe vmstat -p output for non-zero values in api, apo, apf, epi, epo or epf columns. While non-zero *po values indicate that your system is paging out, non-zero *pi values indicate that your system has been paging out somewhere in the past and you currently have problems with response time of your application.
  
• Solaris on SPARC uses up to 12% of physical memory for filesystem cache. Although these pages are first candidates to be freed when system is under memory pressure, they can not be freed in just a nanosecond. Filesystem cache will interfer with your application if you are heavily using filesystem and running near or below lotsfree memory. Either you shoud reserve 12% of memory for this purpose, or you should avoid using filesystem cache by mounting your heavy-loaded filesystems with forcedirectio flag. However, skipping filesystem cache can potentially influence performance of your database, so please consult your DBA before applying this changes. In most cases, datafiles and redo logs are already located on raw devices or on filesystems mounted with forcedirectio flag and the most significant contributors to filesystem cache activity are filesystems used to contain archived logs.
• Solaris kernel will typically take 4-6% of physical memory on large NUMA machine running Oracle, dependent on load profile. On a small machine with 8-16Gb RAM it will consume ~1-1.5Gb, also dependent on load profile.
  
• Several dagnostic tools, like custom DTrace scripts can allocate vast amounts of memory, so one should reserve some memory for such tools too. 2-3 Gb for a large NUMA machine should be reasonable value.