Building your own IaaS Part 2 – DHCP Server

This blog post is part 2 of the “Buiding your own IaaS” series. Part 1 discussed different virtual machine hosting models, and focused on a service which provides physical machines preinstalled with virtualization software. In this blog post I’ll discuss the DHCP server in a virtualized environment.

The DHCP server holds a pool of IP addresses, and when a new machine joins the network it dynamically assigns an IP address to that machine. An alternative is static IP addresses, where each machine has a preconfigured IP address. Since the context of our work is scaling out (dynamically starting virtual machines) having static IPs won’t cut it. There are no APIs for setting the static IP address of a virtual machine before it is started.
Alas, DHCP has some pitfalls, which I had to learn the hard way. Like any good developer, I wrote tests and ran them periodically. Basically each test would start (one or more) virtual machines, run a few commands and then shutdown the machines. This time, the tests were running on a XenServer and not a public IaaS, meaning there was no pay-per-machine cost incurred. However, after a few days, the tests started failing. “Fate, it seems, is not without a sense of irony”. The VMs couldn’t allocate new IP addresses..

So I put on the “I didn’t do anything” face, and went into our sysadmin’s office…

Me: Hey, I just dropped by to ask if the DHCP server is down.

Admin: (Holding a screw driver fixing some server.) No, it’s fine, why?

Me: Ohh.. I don’t know… perhaps it has exahsted it’s IP address pool ?

Admin: That rarley happens. What’s the problem. (still holding the screw driver while logging into the DHCP server).

Me: Well… my tests started failing for no apparent reason, and it feels like a network prob…

Admin: It looks like a DHCP denial of service attack. What have been up to? hey! come back…

You see, each time the test would clone a new virtual machine, that machine would get a random MAC address (ethernet id). The DHCP would then welcome the new machine, assign an ip address and reserve it for a few days in case it comes back online (which never happens, since the next test would start an identical VM with a different random MAC address). After a few days of continous testing, the pool of IP addresses exahsted, and the DHCP server could not allocate new IP addresses.

The first workaround is to release the IP address before terminating a virtual machine. That could be done by SSHing the machine and executing:

dhclient –r;shutdown -h now

The second workaround is to store a pool of MAC addresses in the configuration file and before starting a new VM, assign it with an unsued MAC address from the pool. There is no API for preconfiguring a VM’s IP address, but there is an API to preconfigure the virtual machine’s MAC address.

Note: Whatever you do, just don’t start a DHCP server inside your company. Unless properly configured you will cause conflicts with the existing DHCP server, and this kind of stunt is more difficult to troubleshoot than a DoS

attack. The instructions below are meant to be used in a subnet isolated from other networks.
Also not described here are security measures and single point of failure mitigations, that are essential for production ready deployments.
Below is the hardware design of our lab from part 1. On the right hand side is the xenserver machine which we already ordered from SoftLayer. It would run a number of virtual machines. Each VM gets a dynamic IP address from the DHCP server, running on a separate physical machine. That machine also serves as a web proxy, so the VMs could download files from the internet, and it would serve as a convinient desktop (VNC) for remote management.

The benefit of having one physical machine (“Computing Instance” in SoftLayer terminology), is that it’s public network settings can be configured from the online portal (internet routing, optional public IP address, firewall settings, vulenarbility scans, etc…)

Step 1 – order a physical machine

Step 2 – determine IP address range for virtual machines

You will find the netwok settings of physical machines under “Hardware->Configuration” menu. Notice that each “Computing Instance” and “Bare Metal Instance” has two static IP addresses. One for the operating system, and a second one for managing the hardware regardless of the operating system (see IPMI).
Here is our subnet details. The total number of IPs is 32 = 2^(32-27)

Subnet: 10.24.82.192/27
Netmask: 255.255.255.224
Network: 10.24.82.192 (used for routing)
Gateway: 10.24.82.193
Broadcast: 10.24.82.223

Then we list existing and planned static IP addresses:

XenServer: 10.24.82.194
XenServer IPMI: 10.24.82.195
DHCP: 10.24.82.197
DHCP IPMI: 10.24.82.198
Load Balancer: 10.24.82.199, 10.24.82.200
Petclinic Virtual Load Balancer: 10.24.82.201

That leaves the virtual machines that would run on the XenServer the following free IP range 10.24.82.202-10.24.82.222 (21 ip addresses).

Step 3 – install and configure the DHCP daemon

So once you are logged into the dhcp machine, install dhcp daemon.
(Reference: https://dakeung.com/2009/07/03/quick-guide-how-to-setup-a-linux-dhcp-server-centos-5-2/ )

yum install dhcp

Edit the dhcp configuration (replace with your network details).

nano /etc/dhcpd.conf
ddns-update-style interim;
ignore client-updates;
subnet 10.24.82.192 netmask 255.255.255.224 {
range 10.24.82.202 10.24.82.222;
default-lease-time 259200;
max-lease-time 518400;
option subnet-mask 255.255.255.224;
option broadcast-address 10.24.82.223;
option routers 10.24.82.193;
}

Make sure the machine is in the correct timezone:
setup
(choose timezone, in our case it’s New York)
And start the DHCP daemon

/etc/init.d/dhcpd start

The DHCP server is now ready to provide IP addresses for new virtual machines.

Step 4 – install HTTP proxy

yum install squid

Edit the squid configuration file

nano /etc/squid/squid.conf
# add the following to allow incoming connections
# replace the IP/Mask with your network details
acl our_networks src 10.24.82.192/27
http_access allow our_networks

Check squid configuration and start it

chkconfig squid on</ p>
/etc/init.d/squid start

Step 5 – install desktop and VNC server

Install gnome desktop, firefox and vnc server (enter a STRONG! password for the vnc login)

yum groupinstall “GNOME Desktop Environment”
yum install vnc-server
yum install firefox
vncserver

Edit the vnc startup configuration file

nano ~/.vnc/xstartup
# Uncomment the following two lines
unset SESSION_MANAGER
exec /etc/X11/xinit/xinitrc

You can later kill the vnc session with

vncserver –kill :1

That’s all for today.
In our next blog post we will finally start a new virtual machine.
Itai