Virtual Ryan

Changing a Domain's Memory Size Under Xen

I know I promised this yesterday, so I'm sorry if you were up all night waiting for it :)

I was planning on writing about the using the balloon thread and giving some explanations behind the code, but it was going way too long. So, I'll make one post today explaining how to change memory sizes of domains, and another one Monday explaining some of the code behind it.

Now, on to the show...

I'm in charge of a seemingly random assortment of things for the Solaris on Xen work. One of these things is called the balloon thread. The balloon thread is a kernel thread that works with the hypervisor to dynamically grow and shrink the amount of memory in a domain. To activate this code, you can use the 'xm mem-set' command. An example on dom0:

# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  3891     2     r-----     66.5
# xm mem-set 0 3500
# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  3500     2     r-----     67.8

This is on my desktop, with 4G of RAM. At the beginning, the control domain (dom0) is given 3891 MB of RAM. The rest of the memory is held by the hypervisor, and is not visible to any domain. I issued the command 'xm mem-set 0 3500', which means change the amount of memory in domain #0 to 3500 MB. As you can see on the second 'xm list', it worked.

The tools can do this automatically during domain creation, as well. I still have 3500 MB in my dom0, which means I have around 400 MB free. Let's say I want to create a domU with 768 MB:

# head -3 guest.py
name = "example1"
vcpus = 2
memory = "768"
# xm create guest.py
Using config file "./guest.py".
Started domain example1
# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  3251     2     r-----     83.2
example1                                   1   767     2     -b----     28.8

You can see that more memory was automatically removed from dom0 and given to domain #1.

You may ask, "Why does it say 767 MB when we asked it to have 768 MB?" Well, some of the memory for any domain is given to the hypervisor for disk and networking purposes. It will be constantly exchanged as packets and disk blocks are moved around. So, the value given in 'xm list' is usually within 1 or 2 MB of the requested amount.

Let's lower the memory in our domU and give it back to dom0:

# xm mem-set example1 500
# xm mem-set Domain-0 3500
# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  3501     2     r-----     89.3
example1                                   1   499     2     -b----     38.3

As you can see, it's now easy to dynamically add and subtract memory from a running Xen domain.

There is one catch: a domain cannot go above the amount of memory that it originally booted with. For our above example, I can change the domU's memory allocation to anything below 768 MB, but it will not go above 768 MB. There are some complexities in the Solaris kernel VM system causing this, which I'll explain on Monday when I give more details on the code implementing the Solaris thread.

Update, 11:45 AM:I forgot to mention... if you prefer libvirt over xm commands (and really, you should, because it's easier to maintain), you can use virsh:

# virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # dominfo 0
Id:             0
Name:           Domain-0
UUID:           00000000-0000-0000-0000-000000000000
OS Type:        linux
State:          running
CPU(s):         2
CPU time:       448.5s
Max memory:     no limit       
Used memory:    3984324 kB

virsh # setmem 0 3500000

virsh # dominfo 0
Id:             0
Name:           Domain-0
UUID:           00000000-0000-0000-0000-000000000000
OS Type:        linux
State:          running
CPU(s):         2
CPU time:       450.3s
Max memory:     no limit       
Used memory:    3499008 kB

virsh # 

The most noticeable difference is that the setmem takes memory sizes in kB, not MB like xm mem-set.

Posted at 10:19AM Jul 20, 2007 by rscott in Sun  |  Comments[3]