Sangpill Kim's Blog

Recently, I've got a chance to investigate "the memory utilization of xvm server"

On a X4450 system, I installed SXCE 86 OS and installed 32 Windows XP guest OSs by zfs cloning.

- System Spec -

I created one Windows XP guest as a master os image and cloned it to 32 guests. After creating them, I've started guests one by one. As I increase the number of running guests, I monitored memory utilization of system by 'echo "::memstat" | mdb -k' command and got results below. The results shows physical memory status not virtual and Kernel, Free(freelist), Balloon memories are significant values who vary as the number of running guests changes. 'Balloon' is a memory assigned to hypervisor to be used for guest os.

- memory status -

26 was the maximum number I could run. As I try to run another guest, I encountered this error message.

- Error message -

Since zfs is major memory consumer of kernel, I limited zfs_arc_max to 512MB.

Increasing the number of running guests by 26, I could get this graph.

The graph shows some simple phenomenons.

• The xVM Server assigns physical memory for guest os not virtual memory.
• Balloon memory seems come from Free memory. They are in inverse proportion.
  
• To add guest OS, available Free memory, larger than the memory to be assigned to guest, must exists.
• Running guest OS requires not only Balloon memory for guest but also Kernel memory for hosts. Required Kernel memory is estimated about 250MB per a guest.
• In case of running large number of guests(>17 guests), Kernel memory does not grow any more and decrease its size as the number of running guests grows.
• Though Kernel memory decreases, free memory keeps its size (17~21 guests).
• Kernel memory & Free memory shrinks together over 21 guests.
  

Before the test, I had no idea about the increase of Kernel memory.  However, in fact, running guest os requires not only Balloon memory but also additional Kernel memory in the host. And the maximum number of runnable guests depends on the available Free memory on the system. In case of lack of Free memory, running additional guests is not possible. Those are the conclusion of my test.

Anyway the results are very interesting as the number of guests goes over 17. The Kernel memory does not increase, even it shrinks though Free memory is still enough. Between 18~21, Kernel memory decreases instead of Free memory, and then Free memory & Kernel memory decrease together over 21.

In general, the kernel will not free any memory it allocates for itself, but it also wants to keep a certain amount free for application usage. When the amount of free memory drops below a certain threshold, it will begin to free its own memory.  As more memory is used beyond this point, both the kernel memory and free memory will decrease, although the rate of decrease for each will be unpredictable.¹⁾

I went further. I started shutting down guests one by one, then, I could get this graph.

Simple but interesting result. There were no changes in memory status at all. xVM guest OS do not return the memory after shutting down. I thought this is a bug at first, however, it is a correct operation. When a domU is shut down, the memory should remain free to allow quicker creation of future domUs.²⁾

* 1), 2) by Ryan Scott in SW Eng group