Mounting an persistent EBS volume at boot time is no trivial task. Since the introduction of the Nitro based instance types, EBS volumes no longer appear under the device name specified during the attach command. Instead, the device name will be determined by the kernel and depend upon the order in which it is attached. As a result, switching EC2 instance type or volumes, may cause an incorrect volume to be mounted or not be mounted at all. In this blog we present the details for mounting an EBS volume on any EC2 instance type.
In cloud migrations, we often run into an application that require a persistent volume to store state. For instance, WebsphereMQ. In these cases, We provision the relevant EC2 instances with a detachable network interfaces and EBS volumes: In that way, we can still destroy and replace the instance while preserving the ip address and data.
In order to mount such an EBS volume on the EC2 instance, we need to: - Attach the disk - Wait for the disk - Format the disk - Mount the disk
Attaching the disk is quite easy. In CloudFormation we specify the volume id and device name as one of the volumes of the ec2 instance:
EC2Instance: Properties: Volumes: - Device: /dev/xvdd VolumeId: !Ref 'Storage'
Even though the disk is part of the ec2 instance definition, the ec2 instance is booted before the attachment has completed. To avoid the boot to complete before the volume is mounted, we need to wait for the block device to appear:
while [[ ! -b $(readlink -f /dev/xvdd) ]]; do echo "waiting for the disk to appear..">&2; sleep 5; done
On NVMe based instances, the disk will not appear as the specified device name ‘/dev/xvdd’, but rather
as ‘/dev/nvme[0-9]n1’. When the disk is attached, the device manager will create a symbolic link ‘/dev/xvdd’
pointing to the actual device. Unfortunately, the mount command does not accept symbolic links as device.
By using the
readlink command we always get the actual device name, whether it is on a Nitro-based instance or
a classic instance.
Now the device has appeared, we format the disk only if it is unformatted:
blkid $(readlink -f /dev/xvdd) || mkfs -t ext4 $(readlink -f /dev/xvdd)
To ensure that we can create a mount instruction without specifying a physical device name, we label the disk:
e2label $(readlink -f /dev/xvdd) wmq-data
Now the disk is attached, formatted and labelled, we generate the fstab entry:
grep -q ^LABEL=wmq-data /etc/fstab || echo "LABEL=wmq-data /var/mqm ext4 defaults" >> /etc/fstab
Remove all the old-style mount entries in /etc/fstab:
sed -e '/^[\/][^ \t]*[ \t]*\/var\/mqm[ \t]/d' /etc/fstab
Finally we mount the disk, if it was not mounted:
grep -q "$(readlink -f /dev/xvdd) /var/mqm " /proc/mounts || mount /var/mqm
When the machine reboots, the /etc/fstab entry will already have mounted the disk for us.
When you want to add your own EBS volume mount, you can use the generate-mount-ebs-volume-bootcmd script to generate the required commands. Type:
$ ./generate-mount-ebs-volume-bootcmd /dev/xvdd /var/mqm wmq-data
The first parameter is your device name, the second the mount point and the third the label. It will generate a bootcmd snippet, you can add to your machine’s cloud-config.
bootcmd: - mkdir -p /var/mqm - while [ ! -b $(readlink -f /dev/xvdd) ]; do echo "waiting for device /dev/xvdd"; sleep 5 ; done - blkid $(readlink -f /dev/xvdd) || mkfs -t ext4 $(readlink -f /dev/xvdd) - e2label $(readlink -f /dev/xvdd) wmq-data - sed -e '/^[\/][^ \t]*[ \t]*\/var\/mqm[ \t]/d' /etc/fstab - grep -q ^LABEL=wmq-data /etc/fstab || echo 'LABEL=wmq-data /var/mqm ext4 defaults' >> /etc/fstab - grep -q "^$(readlink -f /dev/xvdd) /var/mqm " /proc/mounts || mount /var/mqm
It has to be hard-coded, as the write-files module of cloud-config is run after the bootcmd.
Mounting a persistent EBS volume at boot time is no trivial task, and with the Nitro based images it has become even more difficult. By using the
readlink command, this script will work both on NVMe based instances types and on older instance types. I do hope that AWS will provide a simpler and more robust way of mounting EBS volumes at boot time in the near future.