Moving VMs

These days I needed to rebalance the VMs I have running and move the VMs off from an OracleVM server to a VMware ESX 3i environment. Most of these VMs are running some
version of Linux.

I have a number of ways to move the filesystem from the VM in the old server to the new one.

Before you start you will need to make sure that you have created a VM with sufficient disk space to be able to "copy" all used disk space to the new VM. You can do a "df -h" and start adding up the Used column.

Note that you must not have any process that keeps state in memory running such as databases, application servers, etc. Please shutdown these down before doing the transfer.

These are the steps:

1. Create the new VM

After the VM instance has been created, boot into a minimal Linux distro, in my case I used SystemRescueCD version 1.3.5.

2. Make the partitions on the new Virtual Disk

Next create the partitions on the new virtual disk. You can use "fdisk" for this remember to make a "swap partition" (type 82) and also make the "/" or "/boot" partition as Active, otherwise it may not boot.

Also, the boot partition (or the root partition if you don't want a boot partition) must be as
closer as possible to the beginning of the virtual disk due to the typical limitation of "grub" and boot BIOS.

3. Put a filesystem on them

After the partitions have been created, you need to format them for the appropriate filesystem.
Generally most older Linux distros use EXT3 for the root and boot partitions. There is caveat here in that the newer Linux distro use a 256 bytes INODE size which will prevent it from booting using a older "grub" version. The older ext3 partitions use 128 bytes.

To fix this, please use the following command:

# mkfs.ext3 -I 128 /dev/sda1

assuming that the new root/boot partition is on partition 1.

For the swap partition, you need to use:

# mkswap /dev/sda2

assumming that the swap partition 2.

4. Mount them and copying the filesystem

The next step is when we start to clone the file system onto the new virtual disk.
Mount the new partition on the SystemRescueCD filesystem. I used a new directory "temp" under the "/mnt" directory.

# mkdir /mnt/temp
# mount /dev/sda1 /mnt/temp

To copy the filesystem from the old VM you must ensure that the new VM can reach it via the network, and so you need to configure the network interface on the SystemRescueCD.

# ifconfig eth0 X.X.X.X
# ping old_server_ip_address

You have a number of different ways to clone the filesystem:

a) using "dump" and "restore"

# cd /mnt/temp
# (ssh root@src_server dump -f - /) | restore -rf -

This assumes that you can log in to the old server via SSH on the root user. If not, you may need
to enable this.

Note that this is only for the root ("/") filesystem. Repeat the process for each of the mounted filesystem on the old server. You need not have the same set of mounted partitions on the new virtual disk and you can consolidate all of them into a single large partition, including the /boot partition.

# cd /mnt/temp/boot
# (ssh root@src_server dump -f - /boot) | restore -rf -

b) Using RSYNC

Sometimes I find that the dump/restore fails. In that case, rather than restarting from beginning I use RSYNC to only transfer the missing files. For example if the /u01 partition failed halfway you can run this command to continue the transfer:

# cd /mnt/temp
# rsync -avz root@src_server:/u01 .

c) Using tar

Previously I used "tar" to do the cloning and it still works.

# cd /mnt/temp
# (ssh root@src_server tar -c /) | tar -C /mnt/temp -xv

d) Using cpio

# (ssh root@src_server "find ./source -depth -print | cpio -cvo" ) | cpio -icvmdI

e) Using nc (netcat) command

I have not tried this method but it is suppose to work:

On the source machine:
# cd /source
# tar -czpsf - . | pv -b | nc -l 3333

On the destination machine (new VM):
# cd /mnt/sda1
# nc src_server 3333 | pv -b | tar -xzpsf -

5. Making the new VM bootable

If you are using "grub" for booting the VM, you may need to perform the following
steps.

a) copy the minimal devices to the new /dev directory
# cd /mnt/temp/dev
# cp -a /dev/sda* .
# cp -a /dev/core .
# cp -a /dev/null .
# cp -a /dev/zero .
# cp -a /dev/console .
# cp -a /dev/mem .
# mkdir pts

b) fixing back the /etc/fstab file
You may need to edit the /etc/fstab file to point to the correct devices to
mount

c) running in a chroot environment

# chroot /mnt/temp /bin/bash
# mount -t proc none /proc
# mount -t sysfs none /proc
# mount -t devpts none /dev/pts

d) installing grub

# cd /boot/grub
# grub
grub> root (hd0,0)
grub> find /boot/grub/stage1
grub> setup (hd0)

6. Fixing back the kernel and/or the initrd.img

In some cases, you need to install an appropriate kernel into the new VM if the
old one cannot be use (e.g. if you are running a Xen DomU kernel).

Or, you may have a kernel that has a default initrd.img without the correct disk
driver (e.g. mptscsi or sd_mod) module. You may need to re-create it and fix back
the /boot/grub/grub.conf configuration to reflect the correct initrd.img.

7. Reboot the new VM

UPDATE:

In some cases the server will complain that the filesystem is using unsupported features especially on EXT3 filesystem. You need to then disable these features and you need to use the "debugfs" utility. For example on the Centos servers, the older versions do not support "resize_inode" option. In this case, you need to turn this off by using:

# debugfs -w /dev/sda1
debugfs: feature -resize_inode
debugfs: close
#