lunedì 27 luglio 2015

Migrating a live system from ext3 to ext4 filesystem

This article is meant to serve as a guide for migrating a live system from ext3 to an ext4 filesystem, including migration of files to use extents, a major feature in ext4. It describes the entire migration procedure, including common pitfalls involving a migration of a live system, as opposed to doing a fresh install.

Reasons for conversion

The explanation of advantages and disadvantages of ext4 is beyond the scope of this article. If you are not affected by the limitations of ext3, and not willing to take risks, it may not be worth it. On the other hand, on successful completion of the migration procedure your system may perform faster, experience shortened file system checks, and have increased reliability with no ill effects.

Trying ext4 without conversion

An interesting property worth noting is that you can mount ext3 filesystems as ext4 without doing the conversion first. All you need to do is to modify your /etc/fstab to say "ext4" instead of "ext3" for all your filesystems, and reboot. This way you will be able to go back to ext3 at any time.
This may be a nice test before attempting full conversion, but this will only enable ext4 features that are compatible with ext3. And that means that no major feature of ext4, like extents, will be enabled.

Migrating to ext4

Word of warning

Do a backup before attempting this procedure. It may render your system unbootable, and may destroy your data.
Your filesystems will become incompatible with ext3, so you need to make sure that you have a complete toolkit available with ext4 support before doing a conversion. This includes a bootloader, e2fsprogs, mount, and a recent kernel.

Prerequisites

  • e2fsprogs 1.41.6
  • mount 2.16
  • linux-image 2.6.30
  • grub 1.96+20090808
All of these packages are available in Debian unstable or experimental. Lower versions may work, except for e2fsprogs - this is indeed the lowest version.

Converting a non-root filesystem to ext4

As long as you are converting a filesystem that can be unmounted, it is fairly simple procedure. In this example we will be converting a /dev/sdc1 partition mounted as /home directory.
First, unmount the partition.
umount /dev/sdc1
Next, run a filesystem check on it to make sure it is in sane condition. We are still on ext3.
fsck.ext3 -pf /dev/sdc1
Enable new features of ext4 on the filesystem.
tune2fs -O extents,uninit_bg,dir_index /dev/sdc1
Option "extents" enables the filesystem to use extents instead of bitmap mapping for files, "uninit_bg" reduces file system check times by only checking used portions of the disk, and "dir_index" allows storing the contents of large directories in a htree for faster access. Option "dir_index" is also supported by ext3, so you may already be using it, but it makes no harm to specify it here.
Run a filesystem check. It will find errors. It is normal. Let it fix them. You may want to run the check twice to make sure that the filesystem is now clean.
fsck.ext4 -yfD /dev/sdc1
The "-D" parameter will actually enable the "dir_index" option by rebuilding directory index. It can be rebuilt (optimized) at any later time by running the check with the parameter.
Now edit your /etc/fstab file to say "ext4" instead of "ext3" for /home. Other options may differ for your system.
/dev/sdc1 /home ext4 defaults 0 2
Try to mount your new ext4 filesystem.
mount /home
If it succeeds, congratulations. If not, do not panic. You have not lost your data. And you have a backup after all, right? Make sure you have all the latest tools listed in prerequisites. Get them form Debian unstable or experimental if needed. Upgrade and try again.

The /boot partition

If your /boot is a separate partition, all is good and great. Just leave it as ext3. Latest development grub versions do have support for ext4, but it still may and will fail for some given snapshot of grub.
As ext3 can be mounted ext4 without conversion, you can just edit your /etc/fstab to say "ext4" instead of "ext3" for boot partition.
/dev/sdb1 /boot ext4 defaults 0 1
Most features of ext4 will not be used, but that makes little difference for /boot, as it is only used early at boot time. And since this is essentially still an ext3 partition, grub will have no problem booting it.
If, on the other hand, you do not have a separate /boot partition, you should consider creating one. Otherwise you must be really careful, and not enable features not supported by grub, or make sure that you are using a version of grub that supports all of them.

Converting a root filesystem to ext4

Converting a root filesystem is a bit more tricky because you cannot unmount it, as your system is running on it. Nevertheless it is still possible to do it without using an external bootable media. You should do this in a single-user mode.
First step is to modify your /etc/fstab file to say "ext4" instead of "ext3" for root partition. This is important because you will be operating on a read-only filesystem later, and will not be able to make the change, and this would result in your system unable to mount a root filesystem on next boot.
Let us assume that root partition is /dev/sda1, so your /etc/fstab should look something like this.
/dev/sda1 / ext4 defaults 0 1
Now remount the root filesystem read-only.
mount -o remount,ro /
Then run a filesystem check on the root filesystem.
fsck.ext3 -pf /dev/sda1
It will tell you to reboot the system. That may be a good idea, so simply boot into single-user mode and remount it read-only again. It is fine even though we have already modified /etc/fstab, because ext3 can be mounted as ext4 without conversion.
Next, enable all the ext4 features on the root filesystem.
tune2fs -O extents,uninit_bg,dir_index /dev/sda1
And run run a filesystem check on the root filesystem again. It will find and fix errors. This is normal.
fsck.ext4 -yfD /dev/sda1
You can now reboot to your new ext4 system, and enjoy faster filesystem check times, better performance, and all the improvements of ext4. Well, almost; read the next section.

Migrating files to extents

It may seem that the migration from ext3 to ext4 is now complete, and it is almost true. Except that any old files created before the conversion will continue using the bitmap mapping of ext3 instead of extents of ext4.
Files will eventually migrate to the new format as they are updated during normal system operation, because on next write they will be saved using extents. Unfortunately many frequently used files (like application binaries) are often read and rarely written to. The outcome is that the files will remain using the old format for a long time, and you will not be able to experience full potential of ext4.
A utility called e4defrag, which would be able to migrate files, is being developed. But unfortunately it is far complete, and is not suitable for use on real data as of time of writing.
Fortunately it is possible to migrate separate files to extents by using chattr utility, which comes with e2fsprogs package. It allows you to set an attribute on a file which causes the kernel to rewrite the file using extents. It even possible to do this on a mounted and working file system. In fact, that is the only way to do it.
Please not that this feature is still experimental and has not been tested thoroughly. Performing such operation may be dangerous. It may also flood your system log with warnings and errors. You should first test this on small number of insignificant files.
You can check if a file (or a directory) is using extents with lsattr.
lsattr /home/user/foo/bar
If it not using extents, the output will be something like this.
------------------- /home/user/foo/bar
The dashed line here is simply a placeholder for various attributes a file can have. This means that it has no attributes.
Now set the attribute to use extents on the file.
chattr +e /home/user/foo/bar
Now list the attributes again, and you will notice that the output looks like this.
-----------------e- /home/user/foo/bar
Note the "e", which means that it is now using extents.
Check your system log and look for scary things. Not finding any is a good sign. That means you can continue.
Modifying attributes with chattr can be done on multiple files. Although digging trough the entire directory system is not really feasible, so you can use some of the shell magic to accomplish the task.
find /home -xdev -type f -print0 | xargs -0 chattr +e
find /home -xdev -type d -print0 | xargs -0 chattr +e
The first command will run "chattr +e" on all files in /home, and the second will do the same for directories.
It is possible to run this on the root directory and convert everything at once. But running this on one file system at a time with "-xdev" parameter will prevent it from diving into file systems that do not support extents. It may also be useful to run filesystem check on a partition after conversion.
It may make sense to perform this in single-user mode to minimize the chances of something interfering with a task. You may also want to shutdown syslog, as it may generate a lot of warnings.