When doing quick maintenance tasks on a server, you can use the following approach to keep your site available:
- Failover the backnet IP address of the host to another host
- Use arping to tell the network that this IP was switched
- Remove the IP from the host that needs maintenance
In case you do not have a full high availability setup available, you can use ipswitch, a small tool I wrote to assist with this kind of simple failover tasks.
You can install it using
$ gem install ipswitch
ipswitch connects to the remote hosts using SSH, and uses the “ip” command to maintain IP addreses.
Migrate an IP without downtime
Let’s say you have an application server called rails-1.example.com, and you need to do some maintenance task on it (e.g. install a kernel update and reboot the machine). While you could modify all of your reverse proxys to exclude this server from their configuration, it is much simpler to just assign the (backnet) IP address of rails-1.example.com to one of your other application servers.
With ipswitch, this can be done using one command:
$ ipswitch migrate rails-1.example.com rails-2.example.com
rails-1.example.com: Getting IP for interface eth0 rails-1.example.com: Found IP 192.168.1.2/24 rails-2.example.com: Adding IP address 192.168.1.2/24 to interface eth0 rails-2.example.com: Running arpping rails-2.example.com: 3 packets transmitted, 0 packets received, 100% unanswered (0 extra) rails-1.example.com: Removing IP address 192.168.1.2/24 from interface eth0
Note: ipswitch automatically detects and uses the (primary) IP of the specified interface (default: eth0)
When your maintenance tasks are done, migrate the IP address back to the original host
$ ipswitch migrate rails-2.example.com rails-1.example.com --ip 192.168.1.2/24
rails-1.example.com: Adding IP address 192.168.1.2/24 to interface eth0 rails-1.example.com: Running arpping rails-1.example.com: 3 packets transmitted, 0 packets received, 100% unanswered (0 extra) rails-2.example.com: Removing IP address 192.168.1.2/24 from interface eth0
Note: As eth0 now has two IP addresses, you need to specify which once to migrate. If the original node reclaimed the IP automatically (e.g. due to a reboot), ipswitch still works.
You can also use ipswitch to just add/remove IP addresses from your nodes
$ ipswitch add --ip 192.168.1.1/24 example.com $ ipswitch del --ip 192.168.1.1/24 example.com
Assign an IP to an interface other than eth0 and do not use arping to broadcast IP
$ ipswitch add --interface eth1 --ip 192.168.1.1/24 --no-broadcast example.com
Managing IPv6 addresses is also possible
$ ipswitch add --family inet6 --ip fe80:🔡66ff:fede:9999 example.com $ ipswitch del --family inet6 --ip fe80:🔡66ff:fede:9999 example.com
Currently only works when using SSH keys for authenticating at your hosts. The following features are planned, and pull requests are apprechiated :)
- Password support
- SOCKS proxy support)
- Ability to specify SSH keyfile
- Support for sudo