Use and automate letsencrypt certificates (ACME) in an high availability environment

Mozilla launched a “free, automated and open” certificate authority called Let’s encrypt. As the name suggests, it provides free certificates trusted by all (major) browsers and operating systems. I’m using it heavily (on this blog, for example).

This blog post shows how Syncthing can be used to deploy letsencrypt certificates in an environment with multiple servers (e.g. in a round-robin scenario) without adding a single-point-of-failure.

Read more →

Online resizing LVM guest-partitions on OpenNebula/libvirt guests

Today I found out howto resize guest partitions on OpenNebula (or in general with libvirt, which OpenNebula uses underneath for KVM virtualization).

I’m using a LVM storage for virtual machines. So resizing them is pretty easy.

Read more →

Nested if workaround for Nginx to allow a specific ip address access to a disabled site

When doing maintenance on a web application, you probably have a custom 503 site, showing your customers that the servers are currently lying on the operating table.

At the dynamic ridesharing service flinc, we touch a certain file on our reverse proxies (e.g. using capistrano deploy:web:disable) when maintenance begins. Nginx then serves a static “we’ve disabled the site for maintenance” site, instead of the actual content.

But wouldn’t it be nice to test your web application before going live for your customers? It sure would. Unfortunately, this is not as simple as a task as you might think, because you cannot nest if directives in an Nginx location and if is evil.

Read more →

iptables-ng cookbook for chef

Today, I released iptables-ng, a cookbook to maintain iptables rules on different machines using chef.

But why another cookbook? There are two fairly often used around

Well, I wanted a tool which can do all the following:

  • Configure iptables rules in a consistent and nice way for all distributions
  • Be configured by using LWRPs only
  • Be configured by using node attributes only
  • Respect the way the currently used distribution stores their rules
  • Provide a good-to-read and good-to-maintain way of deploying complex iptables rulesets
  • Provide a way of specifying the order of the iptables rules, in case needed
  • Only run iptables-restore once during a chef run, and only if something was actually changed
  • Support both, ipv6 as well as ipv4
  • Be able to assemble iptables rules from different recipes (and even cookbooks), so you can set your iptables rule where you actually configure the service
Read more →

ipswitch - migrate IP addresses without downtime

When doing quick maintenance tasks on a server, you can use the following approach to keep your site available:

  • Failover the backnet IP address of the host to another host
  • Use arping to tell the network that this IP was switched
  • Remove the IP from the host that needs maintenance

In case you do not have a full high availability setup available, you can use ipswitch, a small tool I wrote to assist with this kind of simple failover tasks.

You can install it using

$ gem install ipswitch
Read more →

apt-get cleanup commands

Just a short post about some useful cleanup commands for Debian and Ubuntu systems. There are (to my knowledge) no build in task solving the following things

  • Remove old kernels (while keeping the currently running and the latest)
  • Purge removed packages (especially after autoremoving unneeded dependencies)
Read more →

Howto use chef with ssl

By default, the connections between the chef-client and the chef-server are not secured. This is a short post on howto encrypt and verify your connections.

As of chef-11 (unlike chef-10), SSL is enabled by default. But (naturally, as Opscode cannot create trusted certificates for your domain) the certificates are not verified. This essentially means that the connection is not secure at all.

Unless you only use chef in a trusted network, you should invest some time in securing your clients connections.

Read more →

Chef deploy_revision and Capistrano git_style

One thing that was annoying me for a long time, was that, using Capistrano deployment, you cannot spawn a new vanilla virtual machine, and bring it to a fully up-and-running state with just one Chef command.

make deploy_revision compatible with Capistrano, so deployments can happen with Capistrano, until we’ve decided to fully migrate to Chef, or to stick with the push deployment

Read more →

Migration from rvm to chruby on production

On our rails and worker servers at flinc, we recently migrated the ruby version management from rvm to chruby.

Besides the usual arguments against rvm, like preferring unpatched cd commands, there was another reason:

The fnichol’s chef rvm cookbook has some issues.

Read more →