Some time ago (a year or so), I changed the hardware of my own server. I decided to take the opportuny to change my VM system  from XEN 3 to OpenVZ (for a lots of reasons that are not in the scope of this article ;).

I finaly managed to take some time to write something about this story on my blog. This is far from being a guide : it’s more a suite of notes…

Chosen Hardware and linux distro

As I’m a big fan of Archlinux, this is the distro I choosed (simplicity, stability, up to date, …)

For the hardware, as my server is running 24/7, I wanted a quite “green” solution ;). I opted for an Atom based architecture which is powerfull enought for my own usage (3 containers) and as a limited power consumption:

• Motherboard : Asrock A330 GC including an atom 330 at 1.6Ghz (overclocked to 1.8 ghz)
• Memory : 4 Gb of RAM
• Main system disk : 8Gb compaq flash mounted on an IDE adapter (more than 40 Mb/s for a very light power usage)
• VM disks : 2.5 inch 40 Gb drive for containers (VM) and logs partition, two 3.5 inch, 5400 rpm, 2Tb for data storage and backup (soft mirrored: I hate raid as a faulty controller could make 2 unusable disks…)
• Dual gigabit network cards (bonded and dispached using software bridge between containers)

Basic container (VM) Creation

Partitions on OpenVZ Host

The host of all container use the Compaq flash for the main system disk with 3 partitions : /, /boot and /var.
The 2.5 inch 40Gb drive as one partition dedicated to /var/log (to avoid premature death of the compaq flash) and the rest of the disk is used as LMV volume for containers.

This solution as severals advantages :

• The compaq flash insure best file access for the base system (with is more important than brute speed for the container host)
• LVM for containers offer the best option (IMHO) for re-sizing and snapshot-ting containers (think about backup)

Creating a template container

Then, when the container is started, I just need to ajust the hostname in /etc/rc.conf

Network settings

Bonding configuration on OpenVZ host and containers

To enabled bonding and vzbridge (for container disptching) on archlinux I had to comment out all network settings on /etc/rc.conf and set NETWORKS Variable like the following :

For bridge setting, you first need to install bridge utils (pacman -S bridge-utils). Then the config file is :

At this time, the host server as a network access using DHCP with a bonded interface (2x1Gbps here).

Use bonded interface in VM + DHCP

First, you need to create the file /etc/vz/vznet.conf in order to make openvz to configure veth network using bidge-utils for containers (vm). This file just define the path to a script that come with openvz (don’t forget to set the executable bit on it):

Then we juste need to add a network interface for each container. Example for a container with ID 99 (you can set the mac adress as you wish):

We are almost done. We need to configure network on the container… but  for dhcp to work we have to comment the 2 lines In the archlinux tempate setting that begin with ADD_IP and DEL_IP (/etc/vz/dist/arch.conf).

For final touch, on the container rc.conf file, ajust settings as the following:

• set eth0=”dhcp”
• check that “eth0” is defined in the INTERFACES variable
• check that lo=”lo 127.0.0.1″ is declared
• check that ROUTES=(!gateway)

Resources allocations for containers

Summary of OpenVZ ram parameters

Mistakes in any of openvz memory parameter can lead to bad performance and/or unexpected process kill. If I had to remember something I would say :

• Privvmempages : allocable memory (allocate != used). Barrier = limit, whereas limit is not used
• vmguardpages : garanteed memory, even if host is out of memory (and so swap begins to be used)
• oomguardpages: threshold before any process kill
• kmemsize =  non swappable kernel memory used for processes (check usage thought userbean counter)

To make things clear, a process kill will occur when :
limit vmguardpages < oomguardpages + socket buffer +kmemsize

Adding peripherals

This is pretty simple : we just use the DEVNODES variable. For example adding an harddrive :

Allowing HTTPS/SSL and SSHD usage on a container

By default, containers don’t have “random” device needed for SSL to generate key (they need some entropy found in random devices). As OpenVZ is not compatible with udev, special devices must be created manualy.

So, to make apache works with HTTPS, we need to add /dev/random and /dev/urandom to the container (here, ID 99) :

Then, on the container configuration file, we need to add the urandom device:

Allowing OpenVPN on a container

Yes it is possible to use a container as an OpenVPN server… this is one of the advantages OpenVZ has over other containers system (eg: lxc).

Like for SSL, we need to add some devices (here with tun. It should be similar for tap):

Then to allow the container to use iptables (need for openVPN routing rules), we need to set the following on the container config file :

Finally, when OpenVPN is installed on the container, we need to add a routing rule to make the container act as a router to the lan for any connected client.

In the following example, the vpn network is 192.168.100.0 and the IP of the container is 192.168.1.3:

That’s all folks !

I don’t have anything else to write. Everything is working very well since more than one year. This allow to have multiple virtual server with a minimum overhead on the server (remember that atom n330 has vm extension)

I will try to publish some of the script I used for maintenance (automatic backup of all container, automatic ssh ip ban, etc.).

The only thing which is sad is that upgrading Archlinux can be tricky sometime as the container does not run the initial init scripts of the distribution…