Tino Rachui's Blog: Some technical content

I recently gave an interview on Sun VDI and MySQL Cluster to Lenz Grimmer, MySQL Community Relations Manager. It has been published on dev.mysql.com. Check it out: http://dev.mysql.com/tech-resources/interviews/tino-rachui-sun-vdi-cluster.html

-Tino

In case you have missed this. There exist a document "Virtualization for MySQL on VMware: Best Practices and Performance Guide" which provides a comparison of virtualized MySQL Server instances running on VMware ESX with non-virtualized MySQL Server instances running on bare-metal. Additionally the document contains "Best Practice" configurations for running MySQL Server on VMware ESX. Please note that the document is about MySQL Server and not MySQL Cluster which has to run on bare-metal.

Why is this all important in the context of VDI 3.x? You should have heard meanwhile that Sun VDI 3.x is making use of MySQL to host the VDI configuration. Secondly for the VDI Single Host configuration which we have introduced with VDI 3 Patch 2 you have to use a MySQL Server. With the ability to run MySQL Server in a virtualized environment you also gain the possibility to run the whole VDI Single Host configuration in a virtualized environment. You have to play with it a little bit though as we haven't tested this particular configuration extensively in our labs yet.

In an earlier posting I have tried to explain "Why you need at least 3 VDI hosts". It's also clearly documented that when you are using the standard VDI configuration you need to run at least your first two Secondary VDI hosts on bare-metal. This restriction is implied by MySQL Cluster see MySQl Cluster FAQ 23.10.13. Running MySQL Cluster in a fully virtualized environment is not supported. As VDI is - well - all about virtualization certain people almost naturally try to install and run VDI itself in a virtualized environment until they realize that it is unsupported. While running MySQL Cluster and hence VDI in a fully virtualized environment remains an unsupported configuration this doesn't mean it strictly doomed to fail. Check out this interesting posting on the topic in the MySQL forum: "Cluster is capable of running within a VM but it is not recommended for production purposes because Cluster is sensitive to latency. I'd worry that additional latency would be imposed by the VM layer. You may need to adjust some timeouts to account for the additional latency...".

By adjusting certain MySQL Cluster parameter you may actually be able to get it to work successfully. Take a look at  "Defining MySQL Cluster Data Nodes". MySQL Cluster parameters you may need to adapt to stabilize the MySQL Cluster in a virtual environment are for instance:

• ArbitrationTimeout
• HeartbeatIntervalDbDb
  
• HeartbeatIntervalDbApi
  
• TimeBetweenEpochsTimeout
  
• TimeBetweenInactiveTransactionAbortCheck
  
• TimeBetweenWatchDogCheck 
• TransactionDeadlockDetectionTimeout

It's worth a try for the brave hearts at least. ;)

Check out this cool demonstration of Sun's VDI 3 in combination with Sun's Identity Management solution. 

http://slx.sun.com/1179272877

Have fun,

Tino

VDI 3 is out for quite a few days meanwhile and the feedback we've got so far reaches from positive to total excitement. That's a nice and really satisfying fee for all the hard work necessary to get this product done in the given time frame. Several things have been written already about VDI 3 and How it all came together so I don't want to repeat such general information here. I'd like to rather concentrate on a specific aspect of VDI 3 namely the MySQL database integration. Why I want to do that?

1. I'm the responsible engineer for the MySQL database integration (you now have somebody to directly blame for any shortcomings in this particular area ;))
   
2. Feedback we've got so far on various tech and feedback aliases suggests there are a few questions open around this area and a few misunderstandings exist as well (apparently the VDI 3 documentation has still some gaps to be filled in this regard). 

VDI 3 has a far more sophisticated data model than his predecessor VDI 2. This data model almost naturally fits into a relational data model hence we changed the path from using the Sun Ray or Sun SGD data store (both LDAP databases) and decided to use a relational database instead. Around the time of this decision Sun announced to acquire MySQL. Given this turn of events and the reputation of MySQL it was not exactly difficult to choose a specific relational database for VDI 3. Besides that what other requirements influenced the VDI 3 database design?

1. No single point of failure (SPOF). As VDI 3 is a system destined to serve hundreds of users 24x7 hours the system shouldn't have a SPOF.
   
2. No MySQL knowledge required to get started with VDI 3. People should be enabled to get started with VDI 3 in production without the requirement to learn about relational databases in general and MySQL in particular and yet be able to quickly setup a high available (HA) system.
3. Flexibility. VDI 3 is supposed to serve small to large deployments. What will be optimal for smaller deployments will most likely not be optimal for larger deployments too. So VDI 3 should offer the flexibility to satisfy a broad range of deployment scenarios.
   

All three are important requirements but as so often in life you cannot treat everything as equally important but have to prioritize. For us 1. was the most important requirement leading our initial choice of a particular MySQL version to use for VDI 3. To make a long story short we've chosen MySQL Cluster which is a "...high-availability, high-redundancy version of MySQL adapted for the distributed computing environment...". While this choice perfectly satisfies requirement 1 it doesn't help (rather contradicts) a solution for requirement 2 because setting up MySQL Cluster, though the MySQL folks tried hard to make it as easy as possible, requires quite some knowledge and experience. To compensate this disadvantage we have invested a lot of energy and effort into the VDI 3 installation process. The result is that now with VDI 3 if you choose to use the embedded database a complete, high available MySQL Cluster will be automatically configured for you. You don't need to know anything about the details and theory behind it. There is one noticeable implication of using MySQL Cluster: You need a minimum of three physical hosts to configure a HA MySQL Cluster. This is the recommended deployment option for the embedded VDI database scenario. Why it is necessary to have three physical hosts will be answered here.

Feedback we've got from the field so far reveals that this is a particularly painful implication. A lot of customers want to start small and cheap. Requiring a minimum of three physical VDI hosts if you only want to run a handful of virtual desktops can be hard to explain to somebody. How to escape this dilemma? I will show some possibilities below.

But first let me say something about the third requirement "Flexibility". The embedded VDI database is fine for a lot of smaller to medium deployments but it might not be the perfect match for each and every situation or customer. So we've added the possibility to make use of an existing MySQL database known as the remote database option in the VDI 3 configuration. This hopefully satisfies requirement 3. Here are two of the most likely reasons making the remote database option the better choice:

• People have a MySQL database running already and want VDI to integrate into that database.
• People need more freedom in setting up the MySQL database for a variety of reasons e.g. performance, security etc.

So now on to the deployment options and what you can do about the requirement of three physical VDI hosts. I will enumerate a couple of supported scenarios. First of all you can choose the remote database option. This allows you to either use MySQL Server >= version 5.0 with InnoDB or MySQL Cluster version >= 6.2.15 in whatever way is supported by MySQL. One minimal setup enabled by this option is the following one:

• All in one host scenario [*]. In this scenario you will run everything on one physical host. Everything means Sun VDI 3, VirtualBox and the storage. Please note that the requirement for this deployment options is to run Solaris 10 >= update 7 on this host. Concerning the database you would have a locally installed MySQL Server and connect to it selecting the remote database option during VDI 3 configuration. It should be pretty clear however that this kind of deployment offers zero redundancy meaning it is one big SPOF (the cheapest one though :)). Furthermore you would have to buy a separate MySQL support contract if you need this part of the solution to be supported formally.
  [*] Requires VDI 3 patch 1, soon to be released!
  

• Shared VirtualBox, VDI Primary host. In this scenario the Sun VDI 3 Primary and one of your VirtualBox hosts share one physical machine. Make sure the shared host has enough capacity to deal with these two roles at the same time.
  
• Running the VDI 3 Primary in a virtual machine. This option is useful when you are using VMware as virtualization platform. Running MySQL Cluster completely in a virtualized environment is not supported see MySQL FAQs. Given the fact that the MySQL Cluster management node requires only little resources the MySQL folks have agreed that it is an acceptable and supported scenario to run it in a virtual machine. The two VDI 3 secondary hosts running the MySQL Cluster data nodes nevertheless need to run on bare metal.
  

That's it for now, a lot of stuff. I hope a could clarify a few things concerning the VDI 3 database integration.

It's certainly no news when I say that this is going to be a challenging year in many respects. But it's also going to be a great year for Sun VDI I think. Not only but also because we will release Sun VDI 3.0 early this year. The new version has completely been reworked from bottom to top. It will be extremely easy to install, has an exciting new user interface (web based and command line), has LDAP and Active Directory integration, uses MySQL Cluster as database for its configuration data and so on and so forth... 

Stay tuned! 

Thumper (aka Sun Fire X4500) is an incredible device everybody should have one. You can't afford one?  Well build one yourself. You think I'm kidding? Then look at this amazing demonstration of what is possible with ZFS and a bunch of USB sticks. It's here on YouTube http://uk.youtube.com/watch?v=1zw8V8g5eT0&eurl=http://blogs.interdose.com/sebastian/topics/future/ (sorry it's in German only).

This is the last posting on how recycling works with SVDC for the time being. The previous posting of this small series ended with the question "What if things are not working as expected and you need to know why not as soon as possible?".

There exist a couple of tools and sources of information to help you tracking down possible root causes:

    * VMware Infrastructure Client (VIC)
    * ESX Managed Object Browser
    * Windows Event Log Viewer
    * VDA Agent log files

Before you start using any of these tools or information reflect and understand on what conditions SVDC is actually supposed to recycle your VMs.

Ask yourself "What are the power settings of your guest OS?" and "What are the power settings of the VM hosting your guest OS?". Review the conditions that SVDC takes into account for recycling to get a clear picture of what actually should lead SVDC to recycle a specific VM. When you have done this it makes sense to actually start using the above listed tools and information sources for further investigation.

Lets go through some prominent cases and see how to check what is going on (this is how I investigate recycling problems before resorting to use the debugger).

• You have a VM that is assigned to a user and the VM is in power state 'suspended' but the VM will not be recycled after the idle time has elapsed. You probably have had a look at the VM status already using the VIC in order to know that it is actually 'suspended'.
  
  If a VM is in use and suspended SVDC makes the recycling decision based on the suspend time stamp that VC has set for that VM (see my previous postings for details).
  
  Probably the first thing to do would be to have a look into the VDA Agent log files which you will find in the 'log' subdirectory of the installation directory of the VDA Agent on your VC host. If you find a warning therein similar to
  
  "Virtual Machine <name> suspended without suspend time"
  
  then you know that you've come across the VC 2.5 problem I've mentioned in my previous posting. Otherwise you might want to check if the VDA Agent is actually running. Use the "Services" control panel on Windows to verify that. Finally you should check if something in the logs suggests communication problems between VDA Agent (running on the VC host) and VDA Service (running on your Sun Ray host(s)) or even more subtle issues like problems to connect to VC via the VMware SDK etc.
  

• You have a VM that is assigned to a user, the VM is in power state 'running' but no user is logged in to the guest OS anymore.
  
  The VM will not be recycled after the idle time has elapsed. In this current case the VM should actually be recycled based on the time stamp written by the VDA Tools.
  
  If you have multiple ESX servers in your setup use the VIC to identify which one is running your VM. Open a web browser and browse to the Managed Object Browser of this ESX host
  
  http://<ip-esx-server>/mob
  
  Log in using the root credentials of your ESX host. Follow these links
  
  content->ha-folder-root->ha-datacenter->ha-folder-vm
  
  on this page you'll find a list of 'childEntity' entries (a bunch of numbers) referring to the VMs running on that ESX host. Pick the one of the machine your're interested in. You have to look for it by clicking on the links and find that VM with the name of the one you are seeing when using the VIC.
  
  After you've found it click on the 'config' link near the top of the page. In the section 'extraConfig' you'll find a 'extraConfig["guestinfo.logoffTime"]'. Click on this link to see what the value of this property is. A value of "0" means the VM will not be recycled by SVDC any other value means this VM is subject to recycling.
  
  If everything is working correctly in the case we are talking about the value you find should not be "0". If it is nevertheless than you need to dig deeper. Connect to the guest OS via RDP using the Sun Ray Windows Connector (SRWC) or the console in the VIC. Verify that the VDA Tools are actually started. Open the Windows "Event Viewer" (in the control panel under Administrative Tools) and go to the 'Application' section. See if you can find any error messages written by the VDA Tools suggesting that they were having problems setting the 'guestinfo.logoffTime' property.

• You have a VM that is assigned to a user, the VM is in power state 'running' and the guest OS has entered the 'standby' mode. The VM will not be recycled after the idle time has elapsed. Use the same procedure as suggested in the second case to investigate possible causes for the problem.
  

These are some problems I've heard about quite frequently. The list is certainly not complete nor will all possible root causes be tracked down using the methods and tools I've described it's a good start however.

Feedback appreciated!

With my previous posting I've tried to set the stage in order to understand how VM recycling works with SVDC 1.0. I've listed the conditions that SVDC checks to decide whether or not a VM should be recycled. So how does SVDC identify these conditions?

It is easy for SVDC to detect that a VM is in power state "suspended" and get the time stamp when it actually went into that state. VC maintains this information and it can be access using the VMware SDK (alas VC 2.5 doesn't provides the suspend time of a VM correctly anymore, this seems to be a regression to previous versions).

It's not possible though to determine the rest of the conditions using the VMware SDK so the another component of SVDC - the VDA Tools - are necessary to help. The VDA Tools are a Windows service that need to be running on the guest OS. They detect if the guest OS enters the 'standby' mode or the user logs off his Windows session. If either of these conditions is met they will write a time stamp that SVDC uses for its recycling decisions.

It might help the understanding to imagine what would happen without the VDA Tools?

Assume a user did log off his Windows session. In that moment you could consider this VM as being 'unused'. If you have configured your guest OS power options to "Never" go into 'standby' mode VC would never put this VM into power state "suspended" obviously and consequently this VM would never be recycled meaning it'll be assigned to the last user forever. That would be identical to a static VM assignment and somehow contradicts the idea of a dynamic pooling.

A similar situation arises with the combination VM power options "leave VM powered on when guest OS enters standby" and guest OS power options have been set to lets say "System Standby after 5 min".  Again the VM would always be in power state 'running' and there would be no chance for SVDC to find out that the guest OS is in 'standby' hence the VM would never be recycled.

If everything is setup correctly (and I hopefully don't need to mention to this audience again that time synchronization across your VDI environment is of paramount importance for reliable behavior) recycling with SVDC is working great. But alas this world is not perfect and things sometimes don't work as expected what to in this case, how to find out asap what is going wrong?

Stay tuned for my next posting.

Recycling is not only an excellent way to protect our environment but also a good way to balance resources in your VDI environment. A couple of interesting postings about this topic has been publish already by Dirk G. in his blog White men can't jump.

I assume you have read them in addition to the official Sun Virtual Desktop Connector 1.0 (SVDC) documentation in order to have the necessary context to understand the topic this blog post is about.

So why am I writing about recycling again? The answer is that question about this area keep coming from many people using the SVDC.

Answering all of these questions individually takes a lot of time so I decided to use a blog as means to address these questions in particular and future technical questions around SVDC in general. I don't aim at duplicating the information that have been given on the topic of recycling elsewhere I rather try to complement them with a little more focus on in-depth technical details on how it is working and how to diagnose issues that you might be facing along the way.

The explanations that will follow assume that you are running Windows XP as guest Operating System (OS) inside your Virtual Machines (VM). There exist some limitation for Window Vista please have a look into the official SVDC 1.0 documentation or find some details on White men can't jump. 

One crucial point in order to understand how VM recycling with SVDC 1.0 works or how to diagnose possible problems in this area is to understand the differences as well as relationships between the VM power options and the guest OS power options (Windows XP assumed).

With the Windows XP  "Power Options" control panel you can influence how Windows should behave when a certain period of inactivity has elapsed. On this control panel there exist a 'System standby' drop down box enabling you to specify a value between 'never' and 'After 5 hours'. When you have chosen a value not equal to 'never' Windows will go into a low power consumption mode after the specified time of inactivity (not mouse moves, keystrokes etc.) has elapsed. The state that Windows then enters is often also referred to as 'standby' mode.

The Virtual Machine Power Options on the other hand determine "How the VM should respond when the guest OS is placed on standby". In other words what should VMware Virtual Center (VC) do with a VM when the guest OS running inside that VM went into the 'standby' mode. With VC 2.5 there are two possibilities (please consult the official VMware VirtualCenter/ESX documentation for details):

1. Suspend the virtual machine. This options probably makes sense if you want to save resources. If the guest OS inside the VM has entered the 'standby' mode VC will put the VM into suspend mode so that it doesn't consume CPU and memory resources for instance.
2. Put the guest OS into standby mode and leave the virtual machine powered on. This option could make sense if you want to be able to enable something like Wake On LAN (WoL) for instance.
   

A second point that is essential in order to really understand how recycling is working with SVDC is to know and grasp the conditions that SVDC uses in order to determine if a VM can be recycled or not. I will explain later how SVDC actually identifies each of these conditions. But first let me name the conditions that must be met so that a VM is subject to recycling (please note that these conditions are combined by 'AND' and 'ORs'):

• The VM is assigned to a user (for SVDC 1.0 this means it resides in the 'used' folder of the corresponding pool) AND
• the VM is in power state 'suspended' OR
• (the VM is in power state 'running' AND no user is logged in to the guest OS) OR
• (the VM is in power state 'running' AND the guest OS is in 'standby' mode)
  

When a VM will really be recycled after the aforementioned conditions have been met and what happens to it then can be controlled with the pool settings "Idle Time" and "Recycle policy" (again please consult the official SVDC documentation for details).

These are the preconditions in order to understand VM recycling. Large blog entries are tedious and this one is quite long already so I will break down the whole topic into multiple postings. Next time I will explain how SVDC identifies the conditions mentioned above in order to come to a decision if a VM has to be recycled or not.

In case anybody is interested here's a quick overview of my professional background at Sun Microsystems, Inc.:

I'm a software engineer in the Sun Ray development team. I joined this group in 2006. It's a really challenging and interesting job. Before I belonged to the StarOffice/OpenOffice.org team at Sun for almost six years. 

I' aiming at providing mostly technical content around Sun Ray and the Sun Virtual Desktop Connector that others hopefully might find useful.