vSphere ESX4: Hot Add CPUs for Linux Guests

Sometime ago within my post entitled vSphere ESX4: Hot Add Memory for Linux Guests, I promised to blog about hot add CPU support to the same VM.  As you may have guessed, I didn’t get around to writing it, and have subsequently found the area well covered elsewhere.

wila has knocked together a very clear procedure here http://communities.vmware.com/docs/DOC-10493 which kind of makes my post unnecessary.

My earlier post has been referenced as a resource by lamw for his helpful guide here: http://communities.vmware.com/docs/DOC-10492

Add to FacebookAdd to NewsvineAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to Furl

vSphere ESX4: Hot Add Memory for Linux Guests

I was asked recently if I could hot-add some RAM to a client’s virtual machine over at virtualDCS. Most of the information I’d found online related to Windows Server versions, but I needed to hot add resources to a Linux VM. VMware.com was lacking in detail about the hot-add compatibility with client operating systems, so I realised I’d better lab it up and see how it works for myself.

The first problem I had, was that the virtual machine I’d cloned from my clients live VM, was originally built using ESX3.5. Hence, it was VM version 4, and hot add hardware is not supported unless the VM hardware is upgraded to version 7. In order to enable hot-add features, I had to first upgrade VMware Tools, and then shut down the VM again to upgrade the virtual hardware to version 7.

Once this had been done, I made sure the VM General Options (VM > Edit Settings > Options > General Options) was set to the correct OS type. This important, as the interface will only display the Memory/CPU Hotplug options for supported OSes. In my case I was running CentOS 5.3 x86_64, so selected Other Linux 2.6.

General Options

Next I enabled the Hot Add CPU and Memory as below, but was unable to check the radio button for Hot Remove CPU, which is interesting in relation to what I found when playing with Hot Add CPUs (discussed in an upcoming post).


I found that the CentOS build I was using (2.6.18-128.el5) recognises hot added memory automatically. A colleague (thanks Stu) recommended I read the Linux Hotplug Memory docs which made the rest fairly obvious.

My VM was running with 512MB RAM, so I added some more via the vCenter console, so my VM now had 1GB RAM allocated to it. (BTW: even though vCenter appears to let you do this for the 32bit guest version, it doesn’t actually work. The task is reported as successful, but when you check the VM properties again, you’ll see the RAM was not added.)

When memory is hotplugged, the kernel recognizes new memory, makes new memory management tables, and makes sysfs files for new memory’s operation.
If firmware supports notification of connection of new memory to OS, this phase is triggered automatically. ACPI can notify this event. If not, “probe” operation by system administration is used instead.

Now comes the interesting part. Within


there are a number of folders all named ‘memoryX’ where X represents a unique ‘section’ of memory. How big each section is, and hence how many folders you have is dependent on your environment, but you can check the file


to view the size of sections in bytes. Basically, the whole memory has been divided up into equal sized chunks as per the SPARSEMEM memory model.

In each section’s folder there is a file called ‘state’, and in each file is one of two words; online or offline.
Locate the memoryX folder(s) which account for the hot added memory by working out the section sizes above, or (like me), just check the contents of the state files:

#cat /sys/devices/system/memory/memoryX/state

Once you locate the offline sections, you can bring them online as follows:

#echo online > /sys/devices/system/memory/memoryX/state

Validate the memory change is seen, using:


That’s it! Quite simple really.

UPDATE: I noticed that William Lam (lamw on the VMware communities) created a nice script to automate the discovery and online process.  It’s very neat and can be downloaded here:

# William Lam
# http://engineering.ucsb.edu/~duonglt/vmware/
# hot-add memory to LINUX system using vSphere ESX(i) 4.0
# 08/09/2009

if [ "$UID" -ne "0" ]
 echo -e "You must be root to run this script.\nYou can 'sudo' to get root access"
 exit 1

for MEMORY in $(ls /sys/devices/system/memory/ | grep memory)
 echo "Found sparsemem: \"${SPARSEMEM_DIR}\" ..."
 STATE=$(cat "${SPARSEMEM_STATE_FILE}" | grep -i online)
 if [ "${STATE}" == "online" ]; then
 echo -e "\t${MEMORY} already online"
 echo -e "\t${MEMORY} is new memory, onlining memory ..."
 echo online > "${SPARSEMEM_STATE_FILE}"

Add to FacebookAdd to NewsvineAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to Furl