vSphere ESX4: Hot Add Memory for Linux Guests

I was asked recently if I could hot-add some RAM to a client’s virtual machine over at virtualDCS. Most of the information I’d found online related to Windows Server versions, but I needed to hot add resources to a Linux VM. VMware.com was lacking in detail about the hot-add compatibility with client operating systems, so I realised I’d better lab it up and see how it works for myself.

The first problem I had, was that the virtual machine I’d cloned from my clients live VM, was originally built using ESX3.5. Hence, it was VM version 4, and hot add hardware is not supported unless the VM hardware is upgraded to version 7. In order to enable hot-add features, I had to first upgrade VMware Tools, and then shut down the VM again to upgrade the virtual hardware to version 7.

Once this had been done, I made sure the VM General Options (VM > Edit Settings > Options > General Options) was set to the correct OS type. This important, as the interface will only display the Memory/CPU Hotplug options for supported OSes. In my case I was running CentOS 5.3 x86_64, so selected Other Linux 2.6.

Next I enabled the Hot Add CPU and Memory as below, but was unable to check the radio button for Hot Remove CPU, which is interesting in relation to what I found when playing with Hot Add CPUs (discussed in an upcoming post).

I found that the CentOS build I was using (2.6.18-128.el5) recognises hot added memory automatically. A colleague (thanks Stu) recommended I read the Linux Hotplug Memory docs which made the rest fairly obvious.

My VM was running with 512MB RAM, so I added some more via the vCenter console, so my VM now had 1GB RAM allocated to it. (BTW: even though vCenter appears to let you do this for the 32bit guest version, it doesn’t actually work. The task is reported as successful, but when you check the VM properties again, you’ll see the RAM was not added.)

When memory is hotplugged, the kernel recognizes new memory, makes new memory management tables, and makes sysfs files for new memory’s operation.
If firmware supports notification of connection of new memory to OS, this phase is triggered automatically. ACPI can notify this event. If not, “probe” operation by system administration is used instead.

Now comes the interesting part. Within

/sys/devices/system/memory

there are a number of folders all named ‘memoryX’ where X represents a unique ‘section’ of memory. How big each section is, and hence how many folders you have is dependent on your environment, but you can check the file

/sys/devices/system/memory/block_size_bytes

to view the size of sections in bytes. Basically, the whole memory has been divided up into equal sized chunks as per the SPARSEMEM memory model.

In each section’s folder there is a file called ‘state’, and in each file is one of two words; online or offline.
Locate the memoryX folder(s) which account for the hot added memory by working out the section sizes above, or (like me), just check the contents of the state files:

#cat /sys/devices/system/memory/memoryX/state

Once you locate the offline sections, you can bring them online as follows:

#echo online > /sys/devices/system/memory/memoryX/state

Validate the memory change is seen, using:

#free

That’s it! Quite simple really.

UPDATE: I noticed that William Lam (lamw on the VMware communities) created a nice script to automate the discovery and online process.  It’s very neat and can be downloaded here:

#!/bin/bash
# William Lam
# http://engineering.ucsb.edu/~duonglt/vmware/
# hot-add memory to LINUX system using vSphere ESX(i) 4.0
# 08/09/2009

if [ "$UID" -ne "0" ]
 then
 echo -e "You must be root to run this script.\nYou can 'sudo' to get root access"
 exit 1
fi

for MEMORY in $(ls /sys/devices/system/memory/ | grep memory)
do
 SPARSEMEM_DIR="/sys/devices/system/memory/${MEMORY}"
 echo "Found sparsemem: \"${SPARSEMEM_DIR}\" ..."
 SPARSEMEM_STATE_FILE="${SPARSEMEM_DIR}/state"
 STATE=$(cat "${SPARSEMEM_STATE_FILE}" | grep -i online)
 if [ "${STATE}" == "online" ]; then
 echo -e "\t${MEMORY} already online"
 else
 echo -e "\t${MEMORY} is new memory, onlining memory ..."
 echo online > "${SPARSEMEM_STATE_FILE}"
 fi
done