When doing quick maintenance tasks on a server, you can use the following approach to keep your site available:

  • Failover the backnet IP address of the host to another host
  • Use arping to tell the network that this IP was switched
  • Remove the IP from the host that needs maintenance

In case you do not have a full high availability setup available, you can use ipswitch, a small tool I wrote to assist with this kind of simple failover tasks.

You can install it using

$ gem install ipswitch

ipswitch connects to the remote hosts using SSH, and uses the “ip” command to maintain IP addreses.

Migrate an IP without downtime

Let’s say you have an application server called rails-1.example.com, and you need to do some maintenance task on it (e.g. install a kernel update and reboot the machine). While you could modify all of your reverse proxys to exclude this server from their configuration, it is much simpler to just assign the (backnet) IP address of rails-1.example.com to one of your other application servers.

With ipswitch, this can be done using one command:

$ ipswitch migrate rails-1.example.com rails-2.example.com

rails-1.example.com: Getting IP for interface eth0
rails-1.example.com: Found IP 192.168.1.2/24
rails-2.example.com: Adding IP address 192.168.1.2/24 to interface eth0
rails-2.example.com: Running arpping
rails-2.example.com: 3 packets transmitted, 0 packets received, 100% unanswered (0 extra)
rails-1.example.com: Removing IP address 192.168.1.2/24 from interface eth0

Note: ipswitch automatically detects and uses the (primary) IP of the specified interface (default: eth0)

When your maintenance tasks are done, migrate the IP address back to the original host

$ ipswitch migrate rails-2.example.com rails-1.example.com --ip 192.168.1.2/24

rails-1.example.com: Adding IP address 192.168.1.2/24 to interface eth0
rails-1.example.com: Running arpping
rails-1.example.com: 3 packets transmitted, 0 packets received, 100% unanswered (0 extra)
rails-2.example.com: Removing IP address 192.168.1.2/24 from interface eth0

Note: As eth0 now has two IP addresses, you need to specify which once to migrate. If the original node reclaimed the IP automatically (e.g. due to a reboot), ipswitch still works.

Simple tasks

You can also use ipswitch to just add/remove IP addresses from your nodes

$ ipswitch add --ip 192.168.1.1/24 example.com
$ ipswitch del --ip 192.168.1.1/24 example.com

Assign an IP to an interface other than eth0 and do not use arping to broadcast IP

$ ipswitch add --interface eth1 --ip 192.168.1.1/24 --no-broadcast example.com

Managing IPv6 addresses is also possible

$ ipswitch add --family inet6 --ip fe80::abcd:66ff:fede:9999 example.com
$ ipswitch del --family inet6 --ip fe80::abcd:66ff:fede:9999 example.com

Caveats

Currently only works when using SSH keys for authenticating at your hosts. The following features are planned, and pull requests are apprechiated :)

  • Password support
  • SOCKS proxy support)
  • Ability to specify SSH keyfile
  • Support for sudo