Speeding Up Migration with Page Sharing
May 8, 2009 Memory, Virtualization
Update: Looking at this now, I’ve definitely become fully convinced that it is a good idea. Clearly you need to be a little careful that your pages match up at each end, but as long as you keep an intelligent cache at each end, you definitely should be able to significantly reduce the amount of migration traffic due to duplicate memory contents.
Kevin Lawton (author of old-school virtualization tool Bochs), recently wrote an article (and followup) on how you could speed up VM migration in a data center by exploiting things like page duplication between the source and destination machines. The idea is that a lot of VMs have common memory pages that you wouldn’t actually need to copy over. He references some of my own work that looked at the amount of sharing that actually occurs between VMs, the first (but hopefully not last) time I’ve ever seen a link to one of my papers in a random blog post in my RSS feed!
The only problem I see has to do with detecting when pages are truly identical. When you do page sharing between VMs on a single host, you detect the similarity by producing a short (32 or 64 bit) hash for each page in memory. If you scan two pages and they produce the same hash, then those pages are very likely identical. I say “very likely“, because it could just be a hash collision, and you need to actually scan all the bits in each page before you can truly know that the two pages are identical.
When you do this on a single machine, it’s not too big a deal to scan two pages and compare their bits, but if you are trying to verify that pages on the source and destination of a migration are identical, it is a big problem. Obviously you can’t just copy the page over the network to do the comparison, since that is what you were trying to avoid in the first place. I guess the only solution is to use a longer hash value (thus reducing the chance for collisions) and really hope that you don’t have a malicious VM at the destination that is trying to corrupt your memory by purposefully creating memory pages that will collide with your content.
I think it is a neat idea that would generally work in practice, but you will need a pretty smart cache at each end to make sure your really keeping the pages consistent.
Another idea would be to use a “post copy” based approach that tries to get the VM started up on the destination machine as quickly as possible, deferring copying most memory pages until after it has already started. You might be able to use this to quickly unload a host that is approaching the overload limit, although the migrated VMs may see a larger performance penalty because of how the migration technique works.