Solving The Right HA Problem

High Availability, HA for short, refers to an applications ability to continue operating after a hardware failure. HA comes in many different shapes and sizes, but two methods in production today are the presence of multiple machines performing the same task, and pairs of machines in a master-slave setup. Sometimes the master-slave setup is extended to include several slaves, but the main idea is that if the master should go away, the slave will pick up where the master left off, with no interruption in service.

In it’s conception, HA was meant to protect against the physical failure of a machine. With the rise of virtualization technologies in the data center, the concept of protecting against machine failure has been extended to include virtual machines as well as physical. If a physical machine fails, software like VMware and Xen have the ability to seamlessly transfer the virtual machines to another host in the cluster. Virtualization then delivers on its promise of guaranteeing HA by solving a lower problem of protecting the machine, independent of the application running on the machine.

However, virtualization technology has its trade-offs, especially in environments that support web applications. The first is disk space. Since each virtual machine is a unique and self contained operating system, minimum disk requirements must be met per virtual machine. This creates large files that hold virtual file systems, most of which is unrelated to the operation of the application that the virtual machine supports. It is not uncommon for a single virtual machine to be 20GB in size.

There are ways to mitigate this problem. A virtual machine could be created with a disk that automatically grows in size as needed, known as thin-provisioning. This is a dangerous practice though, because it allows the sysadmin to over-allocate his available disk space. If one virtual machine suddenly grows too big, and inadvertently fills the available disk, all of the virtual machines could freeze until more disk is available.

Another problem with virtualization is again related to the point that each virtual machine is an independent operating system. Each operating system must be managed to ensure compliance with standards, logging, user accounts, and all of the myriad other tasks associated with running a production operating system. Again, there are ways to mitigate this problem. Systems like CF engine and Puppet are good examples of system automation, and are a necessity after the data center grows beyond a certain point (where that point is seems unique to the each datacenter).

Each shortcoming of virtualization has been addressed with a solution of one type or another, but they each address the problem of managing operating systems. Remember, the purpose of HA is to ensure that the application remains available, not the operating system. The operating system is an auxiliary requirement, needed of course for the application to run, but receiving an undue amount of attention unrelated to the original purpose of HA. HA is not meant to ensure that the operating system remains available, it is meant for the application residing on the operating system.

VMware and it’s ilk are the current darlings of the datacenter, due in no small part because they do in fact deliver on promises of HA. However, they deliver at the cost of misdirection away from the actual problem.

It is my opinion that virtualization technology solves the problem of HA at the wrong layer. HA should be implemented higher in the stack, at the application layer. Consider a much older technology implemented in FreeBSD, jails: FreeBSD jails pair a filesystem, and IP address, and an application into a tightly controlled bundle. Jails allow you to add multiple applications to a single operating system, while keeping them independent of one another. This allows you to fully utilize the hardware, and removes a layer of abstraction that exists in virtualization. Jails do not provide HA. Building an architecture using jails forces the designer to implement HA farther up the stack. Designers still need to provide for hardware failure, the same as an environment built using VMware does, but they do so using existing mature technologies.

For example, a web cluster uses multiple, nearly identical load balanced web servers. In this scenario, HA for the web site is ensured by providing the same data on all web servers. If the web cluster is a database driven application, the database could ensure HA using a master-slave replication setup. This simple setup ensures HA without the overhead and additional complication of virtualization.

Where jails really shine is in the management of each physical server. Each jail mounts the operating system as a read-only filesystem, with symbolic links to unique read-write directories for each jail. That means that there is only a single operating system to manage for each jail, one OS to patch, one OS to configure. Coupling jails with zfs provides an amazingly simple method of backups, and application distribution. A typical jailed application residing in a zfs filesystem will require less than 500MB of storage space, compared to a VMware image of 20GB. Multiplied by several hundred virtual machines or jails, the difference in size becomes a significant difference in cost.

When building a new environment from scratch, it may be tempting to agree with the vendors and shell out tens of thousands of dollars for VMware licensing and associated hardware. When you do, know that you are solving the problem, but at a much higher cost than necessary.