chr4

Make keepalived play nicely with netplan/ systemd-network

Mon, 21 Jan 2019 00:00:00 +0000

TL;DR: Use use_vmac directive in keepalived when using multicast and your carrier allows multiple MAC addresses per interface. If that’s not an option, migrate away from netplan/systemd-networkd, e.g. to ifupdown.

Saturday night, 06:00: I was torn from my sleep by the lovely voice of my colleague, uttering the words dreaded among the ops community: “We’re down, Chris.”

After opening the lid of my notebook, I started digging into the issue.

We’re running several high availability services with the help of keepalived. When a service goes down, keepalived switches over services to a standby instance. This is usually done by assigning a dedicated virtual IP address (VIP) to a service which can then be migrated on the fly between instances. This is achieved by keepalived using the Virtual Router Redundance Protocol (VRRP) as described in RFC5789, and has proved itself to be very reliable in the last years.

Not so today. The primary server somehow lost its VIP, without any notice in the log files. Just like that. I reassigned the VIP manually, verified that we’re back up and went back to sleep.

We then did a post-mortem on Monday. Obviously, we wanted to know what the heck had happened, and how we could prevent it from happening again in the future.

DHCP #

Our OpenStack hoster relies on DHCP to configure the attached network interfaces. I usually prefer static configured interfaces on servers, and eyed the configuration with some suspicion for a while. We’ve had some minor issues with static routes and DHCP in the past, but the environment was pretty stable, so we went with the hoster’s original base images and their configuration.

Ubuntu upgrade #

Keeping this in mind, we’ve been implementing one bigger change in our infrastructure: We’ve upgraded some instances (including the database node that had the outage) from Ubuntu 16.04 LTS (xenial) to Ubuntu 18.04 LTS (bionic). One of the major changes between the two releases is the change of the default networking stack from ifupdown to netplan. Could this have caused the issue?

netplan #

After digging around a bit more, I found out that calling netplan apply (to apply a network configuration) resulted in dropped VIPs.

Apparently, keepalived doesn’t monitor the VIP assigned to it, and neither initiates a failover, nor tries to reinstate the dropped VIP after it was removed from the interface. This is a known issue, and was fixed by this commit which was released with keepalived-2.0.0.

Unfortunately, Ubuntu ships with keepalived-1.3.9, and the keepalived developers do not provide an official repository for more recent versions. After considering providing packages myself, I came to the conclusion that this actually wouldn’t fit the actual problem, as keepalived would just note the removed VIP and failover to another machine. But I wanted to fix the underlying problem itself, instead of just coping with the symptoms.

systemd-networkd #

The default renderer for netplan on Ubuntu is systemd-networkd, a systemd component. On every netplan apply, the systemd-networkd is restarted, applying the new configuration. This apparently also happens when a DHCP lease is renewed and results in the VIPs being removed, because systemd-networkd is unaware of them, as they are assigned by keepalived and are not configured in /etc/netplan.

This seems to be a feature, and I kind of agree why: If you have a messy interface configuration and you connect to a network, it makes sense to make sure the interface is in a defined state.

But having a downtime every time a DHCP lease is renewed obviously also is not an option.

So, how do I fix the problem permanently in a clean way? I was thinking about the following approaches:

1. Migrating to a static IP configuration #

As I was critical of using DHCP anyway, I considered migrating to static IPs. While this would fix the issues with DHCP lease renewals, we’d still have the same problem on other occasions of systemctl restart systemd-networkd (e.g. automatic security updates). I was reluctant to change that much of the networking stack, as I believe this should be done by the underlying hypervisor.

Update 2020-02-07: Bear in mind that apparently systemd-networkd still renews DHCP leases upon (automatic) updates (and purges the VIP from the interface) even when DHCP is configured within another service (e.h. ifupdown). A possible workaround here is to make sure systemd-networkd service is stopped and disabled (e.g. via systemctl stop systemd-networkd; systemctl disable systemd-networkd). When using static configuration via ifupdown, restarting systemd-networkd doesn’t seem to interfere.

2. Dummy network interface (plz don’t!) #

NOTE: An earlier version of the blog post suggested to use a dummy network interface. The Linux networking stack defaults to “Weak ES (End System) Model” as specified in RFC1122), enabling it to handle incoming packages for IP addresses configured on another interface than the physical interface the package came in. Unfortunately, the gratuitous ARP packages (ipv4) and unsolicited neighbour adverts (ipv6) can’t be sent out properly with the dummy network interface, resulting in shaky failovers. Furthermore, this solution didn’t work at all using ipv6.

3. MACVLAN #

After a very helpful discussion with a keepalived maintainer, the recommended way of solving this is to use MACVLANs. This is a built-in feature of keepalived and can be simply enabled with the use_vmac directive in the configuration file. Recent versions of keepalived take care of all required sysctl settings for this, so it should just work.

The MACVLAN interface (e.g. vrrp.51@eth0) will be created by keepalived, and it will also take care of the VIP on that interface. Then, systemd-networkd can configure the original interface at will, without hurting the dedicated interface for the virtual IP when it’s reloaded (or a new DHCP lease comes in).

Bonus: The interface (incl. the master VIP) are also now displayed in Ubuntu’s default message-of-the-day (motd) upon login.

Here’s the relevant section from the keepalived documentation:

# Use VRRP Virtual MAC.
# NOTE: If sysctl net.ipv4.conf.all.rp_filter is set,
# and this vrrp_instance is an IPv4 instance, using
# this option will cause the individual interfaces to be
# updated to the greater of their current setting, and
# all.rp_filter, as will default.rp_filter, and all.rp_filter
# will be set to 0.
# The original settings are restored on termination.
use_vmac [<VMAC_INTERFACE>]

This is the recommended way of implementing VIPs with systemd, and should work with ipv4 and ipv6 setups!

Unfortunately, using Virtual MAC is only possible when keepalived is running in multicast mode, as pointed out by the maintainer.

While I was able to get multicast running on OpenStack (a security group allowing protocol 112 is necessary), most providers only allow a single MAC address per interface, which leads to unstable failovers. Apparently, MACVLAN also is not an option on AWS, as it doesn’t allow multicast at all.

When multicast is not allowed, or multiple MAC addresses are no option, there’s another method available: IPVLAN

4. IPVLAN #

An IPVLAN interface is similar to MACVLAN, but receives packets from the underlying interface based on IP address filtering instead of MAC address filtering.

You can create it using the following command (NOTE: This won’t persist on reboots)

ip link add link eth0 keepalived0 type ipvlan mode l2
ip link set keepalived0 up

# If you intent to use the interface with IPv4, a dummy IP address might be required
ip address add 1.2.3.4/32 keepalived0

Modify the following lines in your keepalived.conf to use the interface:

# Set the base interface here
interface eth0

# Attach the VIP to the IPVLAN device
virtual_ipaddress {
    10.1.0.1/32 dev keepalived0
}

This should work and also send unsolicited neighbour adverts and gratuitous ARP packets correctly.

Disclaimer: While IPVLAN worked fine with IPv6, I wasn’t able to get this to work for IPv4 in an OpenStack environment. Incoming IPv4 ICMP/ TCP packets are received by the parent interface, but never replied to. When attaching the interface to a namespace and use keepalived’s net_namespace pinging works, but I cannot access the high availability service which resides in the default namespace. Also, both nodes go into MASTER mode and apparently VRRP doesn’t work, as the interface needs to be set to the IPVLAN interface inside the namespace. The respective kernel documentation is pretty scarce and only provides examples including namespaces.

Unfortunately, I found no way so far to persist the IPVLAN interface with systemd-networkd. I’ve tried creating the respective systemd .netdev and .network files, with no luck (the interface is just not created). Also, netplan doesn’t support MACVLAN/ IPVLAN, but at least there a ticket with the feature request. I assume though, that creating the interface with netplan/ systemd would also mean, that the interface then will be purged again. So this also is not a solution.

There’s a commit that adds an use_ipvlan option to keepalived to solve this issue and to make the use of IPVLAN as seamless as MACVLAN in keepalived.

Until this is ready, there are two options:

Use networkd-dispatcher to create the IPVLAN device after systemd-networkd brings the interface up by placing the IPVLAN creation from the above snipped in etc/networkd-dispatcher/routable.d/50-keepalived. See netplan FAQ for more information. Because seemingly the hook scripts for networkd-dispatcher are global, and I cannot restrict them to a single interface, I personally went for the second approach for now:
Migrate back to ifupdown for all instances using keepalived. I was reluctant to so, as I didn’t want to exchange the network stack on every new instance, but the migration went smoother than expected and I was able to automate it quite well using Saltstack. While ifupdown doesn’t support IPVLAN natively, it’s easy to implement it using the up directive and just add the necessary commands to /etc/network/interfaces.d/<yourinterface>.cfg:

iface eth0 [...]
  address [...]
  gateway [...]

  # Create keepalived IPVLAN interface
  up ip link add link eth0 keepalived0 type ipvlan mode l2
  up ip link set keepalived0 up

While this is not the perfect solution, it makes me feel comfortable enough to run this in production.

I hope this ensures I can sleep in on Saturdays.

A new hope #

There’s hope. Besides the mentioned keepalived commit that adds an use_ipvlan option, I’ve filed an systemd issue explaining the problem to the systemd maintainers. Even though they currently have more than 1.000 open issues there’s already a pull request adding an option to systemd to prevent automatic purging of addresses.

Update Oct 2020: While Ubuntu 20.04 LTS (focal) has systemd-243 bundled, the new KeepConnection attribute is not supported by netplan. I’ve added a bug report, but with no reply so far.

cache_warmer - tool to warm-up HTTP caches

Mon, 18 Dec 2017 00:00:00 +0000

I’ve written a small tool to warm-up HTTP caches, e.g. services like nginx.

Source code as well as compiled releases are available at Github.

Comparison to other tools #

While most tools in this area are designed to apply a certain load to a web server (often for a specified time), cache_warmer is designed to explicitly GET a set of URLs from a file to warm-up a cache.

Usage #

Warm-up URLs from a file using a mobile User-Agent

$ cache_warmer --mobile urls.txt

Spawning 4 threads to warm cache with 10000 URIs
10000 / 10000 [===================================================================] 100.00 % 133.99/s
Processed 10000 URLs

X-Cache-Status header statistics:
        Miss: 8991
        Hit: 1009

HTTP Status Code statistics:
        Ok: 9850
        NotFound: 150

Total time taken: 450.002s

Features #

Multi-Threaded #

cache_warmer is multi-threaded. Threads can be specified using the --threads option.

HTTP keep-alive #

It uses keep-alive by default, which can be disabled with --no-keep-alive.

Captcha detection #

It supports a --captcha-string option, which scans the response body for certain strings to detect (and abort) when running into e.g. captchas.

Base URI #

If your URL file only contains the base URL (like /products/spoons), you can add a --base-uri to prepend the host and scheme, e.g. --base-uri https://example.com

Request delay #

Outgoing requests can be toned down with the --delay flag.

X-Cache-Status header support #

If your backend sets the X-Cache-Status header, you’ll get nice statistics about your cache hit rates at the end of the run.

When using nginx, such a header can be added with this directive:

add_header X-Cache-Status $upstream_cache_status;

Custom User-Agent #

cache_warmer defaults to a Googlebot-like User-Agent. You can use the corresponding mobile User-Agent when specifying the --mobile flag.

In case you need a custom User-Agent, you can set it with --user-agent 'Your-User-Agent'.

Attach arbitrary cookies with the --cookie key=value option.

Compile #

# Compile locally
cargo build --release

# Cross compile to Linux using Docker
docker-compose up

sslsecure.vim - Highlight insecure SSL/TLS cipher suites and protocols as errors in your editor

Thu, 27 Apr 2017 00:00:00 +0000

When configuring or programming SSL/TLS servers, at some point a SSL/TLS cipher suite and a list of supported protocols have to be chosen. Unfortunately, not all configuration options are safe. :(

Meet sslsecure.vim! A plugin for the Vim editor, that marks insecure SSL/TLS cipher suites and protocols as errors. See all potentially insecure options right in your editor!

Features #

Mark insecure SSL ciphers as errors
Mark insecure SSL protocols as errors
Works with all configuration files (web servers, mail servers, …)
Works with all source code (independently on the used programming language)
Works on top of regular syntax highlighting

Encryption works. Properly implemented strong crypto systems are one of the few things that you can rely on.

– Edward Snowden

Screenshots #

Enough words, see sslsecure.vim in action!

Web Servers #

Nginx:

Apache:

Lighttpd:

Mail Servers #

Postix:

Exim:

Dovecot:

Load Balancers #

Haproxy:

FTP Servers #

ProFTPd:

Databases #

PostgreSQL:

MariaDB/ MySQL

Programming languages #

C (OpenSSL):

Go:

Rust (Rustls)

Jana:

Additional notes on runtime cipher expanding #

Cipher suites are expanded upon runtime. This is especially important when using +CHIPER statements in your suite, as insecure ciphers might be included upon runtime.

Example: The following (suggested by Cipherli.st) cipher suite doesn’t seem to contain any insecure ciphers when specifying:

EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH

However, depending on your OpenSSL/ LibreSSL version, this suite expands upon runtime to the following (note the insecure DSS and SHA ciphers!):

$ openssl chipers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH' | vim

The plugin currently doesn’t save you from this pitfall. It’s therefore recommended, not to use the + statement in your cipher suite. There’s an issue that discusses whether this plugin should also highlight all + statements.

Feedback #

I’m neither a mathematician, nor a cryptographer. If you are one and you have feedback to this plugin, find a flaw, please open an issue or contact me.

Installation #

Just plug it into your favorite Vim package manager:

" Plug
Plug 'chr4/sslsecure.vim

" Dein.vim
call dein#add('chr4/sslsecure.vim')

" Vundle
Plugin 'chr4/sslsecure.vim'

Using sslsecure.vim and nginx.vim together #

sslsecure.vim was inspired by my other plugin nginx.vim, which also automatically tries to detect insecure settings, but is nginx specific. Both plugins can be used alongside - nginx.vim automatically detects when sslsecure.vim is installed, so errors are not highlighted twice.

Edit: Gracias a Redeszone por un revisión muy bien sobre esta extensión!

Edit: Christian Rebischke packaged this plugin for Archlinux users in the Archlinux User Repository (AUR)

Edit: Auch ein Danke an Heise für den Artikel über dieses Plugin!

nginx.vim (with better syntax highlighting)

Fri, 14 Apr 2017 00:00:00 +0000

I’m editing nginx configuration files. A lot. Naturally, I’ve tried several plugins for my favorite editor vim - but ran around a lot of problems:

Most of the plugins available are outdated.
Even syntax highlighting of the current vim plugin distributed with the nginx release has some deficits.
I’ve been tired of copying around secure ssl_cipher directives, etc.

So, I’ve created a new, super-cool and mega-advanced vim plugin for nginx!

Ladies and gentlemen: Please welcome, chr4/nginx.vim!

Edit: This plugin was integrated into Vim and Neovim upstream!

Features #

The plugin is based on the recent vim plugin distributed with nginx-1.12.0 and additionally features the following syntax improvements:

Highlight IPv4 and IPv6 addresses
Mark insecure ssl_protocols as errors
Inline template syntax highlight for ERB and Jinja
Inline syntax highlight for LUA
Improve integer matching
Syntax highlighting for proxy_next_upstream options
Syntax highlighting for sticky options
Syntax highlighting for upstream server options
More to come!

Furthermore:

Remove annoying delimiters, resulting in strange word boundaries

Screenshots #

A server block with highlighted insecure SSL options:

An upstream block with highlighted options:

Embedded highlighting for ERB and Jinja templates:

Embedded LUA syntax highlighting:

Snippets #

The plugin comes with useful snippets which can be accessed using e.g. vim-snipmate.

Select a decent cipher for your requirements (all of them can provide SSLLabs A+ ratings)

ciphers-paranoid<tab>: Even-more-secure ciphers (elliptic curves, no GCM), not compatible with IE < 11, OpenSSL-0.9.8, Safari < 7, Android != 4.4
ciphers-modern<tab>: High security ciphers (elliptic curves), not compatible with IE < 11, OpenSSL-0.9.8, Safari < 7, Android < 4.4 (recommended)
ciphers-compat<tab>: Medium security ciphers with good compatibility (No IE on WinXP) but TLSv1 and SHA required
ciphers-old<tab>: Low security ciphers (using weak DES and SHA ciphers, TLSv1), but compatible with everything but IE6 and Java6
ssl-options<tab>: Bootstrap secure SSL options

Example:

# High-security ciphers (elliptic curves), less compatibility
# No IE < 10, OpenSSL-0.9.8, Safari < 7, Android < 4.4
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';

Or add a robots.txt file with robots.txt<tab>:

# Tell bots to not index this site
location /robots.txt {
    default_type text/plain;
    return 200 'User-agent: *\nDisallow: /\n';
}

It also has auto-completion for location and server blocks with location<tab> resp. server<tab>, and many more!

Installation #

Just plug it in using your favorite vim package manager

For example vim-plug (which I’m currently using):

Plug 'chr4/nginx.vim'

" Optionally, if you like Jinja template syntax highlighting
Plug 'lepture/vim-jinja'

For further information and installation options, please consult the README.

Feedback and further improvements welcome! Just file an issue or pull request on Github, or contact me

Cross-compile and link a static binary on macOS for Linux with cargo and rust

Wed, 15 Mar 2017 00:00:00 +0000

One Go feature which I’m using regularly is cross-compiling Go code to other platforms (usually from macOS to linux-amd64).

In Go, this is a built-in feature that “just works”. The following command produces a statically linked ELF binary which can simply be copied and run on a Linux machine:

$ GOARCH=amd64 GOOS=linux go build
$ file api
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

~~Rust doesn’t have built-in support for such a feature.~~ Edit: Rust core-commiter Steve Klabnik pointed out, that actually what I’m showing off here is the built-in Rust support :)

I’ve been looking for a while how to achieve this using Cargo and Rust. And (spoiler-alert!) I’ve found one, which is relatively simple to set up.

Assuming you have rustup and Homebrew installed, it’s actually quite simple to setup cross-compiling on macOS!

Start with installing the required target. In our case (Linux x86_64) this is x86_64-unknown-linux-musl. The lightweight standard library musl is able to produce statically linked libraries (contrary to the x86_64-unknown-linux-gnu target, which uses GNU libc.

$ rustup target add x86_64-unknown-linux-musl

What’s still missing is the linker. This was always the part where I go into trouble when trying this earlier. Finally, this blog post from Graham Enos guided me to a Homebrew tap by Filippo Valsorda, which provides a complete macOS to Linux cross-compile toolchain.

$ brew install filosottile/musl-cross/musl-cross

That was the heavy-lifting. Now just tell Cargo where to find the linker. You can place the following into your projects .cargo/config or configure it globally in your ~/.cargo/config

[target.x86_64-unknown-linux-musl]
linker = "x86_64-linux-musl-gcc"

Edit: As of rust-1.32.0, it might be required to add a symlink to make it work, which I came accross here

ln -s /usr/local/bin/x86_64-linux-musl-gcc /usr/local/bin/musl-gcc

That’s it! You can now cross-compile Linux binaries with Cargo!

$ cargo build --release --target x86_64-unknown-linux-musl

You can find the resulting statically linked ELF binary in the target/x86_64-unknown-linux-musl/release directory. This file can be copied and run on any Linux x86_64 machine, just like the cross-compiled Go binary!

$ file target/x86_64-unknown-linux-musl/release/api
ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, not stripped

Writing An Interpreter In Rust (Part 2)

Sat, 17 Dec 2016 00:00:00 +0000

Note: This is a follow-up post to Writing An Interpreter In Rust

Thanks for all your feedback, explainations and pull requests! I’m pretty overwhelmed by the feedback and I can confirm that the Rust community is very friendly and extremely helpful.

In this post, I want to quickly review some of the changes to my implementation of the Monkey interpreter I’ve made with your help. I’ve implemented a small benchmark (Disclaimer: I’ve just put next_token() into a #[bench], I’m not sure whether that’s the best practise for this kind of tests), and the results are really impressive so far!

First implementation:

$ cargo bench
[...]
test tests::bench_next_token ... bench: 30,614 ns/iter (+/- 2,177)

Current implementation:

$ cargo bench
[...]
test tests::bench_next_token ... bench: 3,863 ns/iter (+/- 627)

The new implementation is almost ten times faster than the first one! Let’s have a look at the changes:

Use built-in functions #

The first (small) step was to replace hand-written custom functions with built-in Rust functions when possible. For example, the function is_digit() can be replaced with Rust’s built-in is_numeric(). Unfortunately, there’s no built-in function to check whether a character is either a letter or an underscore, but our custom function is_letter() could at least make use of the built-in function is_alphabetic().

fn is_letter(ch: char) -> bool {
    // This was the old statement:
    // 'a' <= ch && ch <= 'z' || 'A' <= ch && ch <= 'Z' || ch == '_'

    ch.is_alphabetic() || ch == '_'
}

In another commit, a loop could be improved using the built-in function is_whitespace().

Improve the enum #

Someone pointed out in one of the many Reddit comments that Rust can store different types in a single enum. With this, it was possible to drop the Token struct completely and merge TokenType and Token into a single enum. We can store Ident and Integer as a String, while keeping the other elements plain and simple:

#[derive(Debug, PartialEq)]
pub enum Token {
    Illegal,
    EndOfFile,

    // Literals are stored as strings
    Ident(String),
    Integer(String),

    // [...]
}

See this commit to review the whole changeset.

Custom iterator #

The biggest change was made using Rust’s internal Peekable<Chars> iterator instead of rolling our own. The following struct was the original Lexer:

pub struct Lexer<'a> {
    input: &'a str,
    position: usize,
    read_position: usize,
    ch: Option<char>,
}

Two people pointed out, that running a custom iterator in the way I did actually is not very efficient. With their help, and Jay Klickliter’s implementation as a reference, the struct could be reduced to a single Peekable<Chars>:

pub struct Lexer<'a> {
    input: Peekable<Chars<'a>>,
}

See this commit for the complete adaption.

Thanks #

Thanks again for all the valuable feedback I received via different channels! Your comments and pull requests are still welcome!

The feedback and pull requests of the following four Rustaceans was especially helpful, so let me give you guys a special shoutout!

Taryn Hill
Jay Kickliter
Cameron Derwin
Tohie

You can find the Writing An Interpreter In Rust project on Github and my contact details here.

Writing An Interpreter In Rust

Fri, 09 Dec 2016 00:00:00 +0000

Last month, Thorsten Ball released his first book: Writing An Interpreter In Go. It’s an awesome book that teaches its readers how to write their own programming language, step by step. It comes bundled with the complete Go code, including tests.

While reading it, I was looking for a challenge. I’m a huge fan of Rust, a safe systems programming language by Mozilla, so I thought it might be a good exercise to port the Go implementation of the programming language “Monkey” to Rust.

Here’s an example of what Monkey code looks like (you can find more examples here):

// Bind values to names with let statements
let version = 1;
let name = "Monkey programming language";
let myArray = [1, 2, 3, 4, 5];
let coolBooleanLiteral = true;

// Use expressions to produce values
let awesomeValue = (10 / 2) * 5 + 30;
let arrayWithValues = [1 + 1, 2 * 2, 3];

I’ve finished implementing the first chapter of the book (the lexer), and decided to share my work. The code is not very idiomatic yet, so I’d welcome pull requests to improve it.

To checkout and run my version of the Monkey lexer, here’s how to do it (in case you do not have Rust or Cargo installed, you can get it via rustup):

$ git clone https://github.com/chr4/writing_an_interpreter_in_rust
$ cd writing_an_interpreter_in_rust
$ cargo tests
$ cargo run

While I followed the original source code quite literally, there are a few things that can’t be done in Rust in the same way it’s implemented in Go. Furthermore, I tried to write idiomatic Rust code, which resulted in quite a few adaptions.

Let me walk you through the challenges that I faced, and the resulting changes.

Porting token.go #

The first step was to to migrate was the token constants. Here’s how it was implemented in Go:

const (
    ILLEGAL = "ILLEGAL"
    EOF     = "EOF"

    // Identifiers + literals
    IDENT = "IDENT" // add, foobar, x, y, ...
    INT   = "INT"   // 1343456

    // [...]
)

This can be easily ported to Rust by using constants as well:

pub const ILLEGAL: TokenType = "ILLEGAL";
pub const EOF: TokenType = "EOF";

// Identifiers + literals
pub const IDENT: TokenType = "IDENT"; // add, foobar, x, y, ...
pub const INT: TokenType = "INT";     // 1343456

// [...]

However, there’s a more idiomatic way of handling this in Rust. Rust supports Enums. So let’s use enum instead:

#[derive(Debug, PartialEq)]
pub enum TokenType {
    Illegal,
    EndOfFile,

    // Identifiers + literals
    Ident,   // add, foobar, x, y, ...
    Integer, // 1343456

    // [...]
}

Note: For our implementation, the Debug and PartialEq traits are derived.

This looks way cleaner. Have a look at this commit to see the full changeset necessary to migrate from constants to an Enum.

Porting lexer.go #

The first problem I came across when porting the lexer was, that the Go version uses the integer 0 (as a byte) to indicate when the end of file is reached. Let’s have a look at the Go readChar() function presented in the book:

func (l *Lexer) readChar() {
    if l.readPosition >= len(l.input) {
        l.ch = 0
    } else {
        l.ch = l.input[l.readPosition]
    }
    l.position = l.readPosition
    l.readPosition += 1
}

The caller then checks whether l.ch is 0. If that’s the case, it generates an EOF token:

switch l.ch {
case '=':
    // [...]
case 0:
    tok.Literal = ""
    tok.Type = token.EOF
default:
    // [...]
}

~~In Rust, a char can’t take an integer value.~~ It’s more idiomatic to use an Option<char> and return None in case the end of the file is reached. It might be even more idiomatic to return a Result with an EOF error, but choosing Option<char> seemed to be the quicker solution for now.

Edit: A Reddit user pointed out that this statement is not true. You can assign an integer to a char casting it using the following syntax: let character = 97 as char;. Thanks!

match self.ch {
    Some(ch @ '=') => {
    [...]

    // Handle EOF
    None => {
        tok.literal = String::new();
        tok.token_type = token::TokenType::EndOfFile;
    }
}

The full changeset is implemented this commit.

Furthermore, I’ve changed the peekChar() function to peek_char_eq(ch: char) -> bool, which seemed like a quick solution for not having to return another Option<char> type.

fn peek_char_eq(&mut self, ch: char) -> bool {
    // Return false on EOF
    if self.read_position >= self.input.len() {
        return false;
    }

    let peek_ch = self.input
        .chars()
        .nth(self.read_position)
        .unwrap();

    peek_ch == ch
}

Porting the REPL #

Porting the REPL was pretty straightforward. Read a line from STDIN and then run the lexer on it. The only pitfall here is that STDOUT needs to be flushed manually so the PROMPT >> is actually displayed:

use std::io::{self, BufRead, Write};

pub mod lexer;
pub mod token;

fn main() {
    let stdin = io::stdin();

    loop {
        // Stdout needs to be flushed, due to missing newline
        print!(">> ");
        io::stdout().flush().expect("Error flushing stdout");

        let mut line = String::new();
        stdin.lock().read_line(&mut line).expect("Error reading from stdin");
        let mut lexer = lexer::Lexer::new(&mut line);

        loop {
            let tok = lexer.next_token();
            println!("{:?}", tok);
            if tok.token_type == token::TokenType::EndOfFile {
                break;
            }
        }
    }
}

Running the tests and REPL #

This is the exiting part. Let’s see whether the implementation (still) works!

Run the tests:

$ cargo test
    Finished debug [unoptimized + debuginfo] target(s) in 0.1 secs
     Running target/debug/deps/writing_an_interpreter_in_rust-e35a8839d5e6ef77

running 3 tests
test lexer::new_token_test ... ok
test token::lookup_ident_test ... ok
test lexer::next_token_test ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured

     Running target/debug/deps/writing_an_interpreter_in_rust-0dfac4ec669be969

running 3 tests
test token::lookup_ident_test ... ok
test lexer::new_token_test ... ok
test lexer::next_token_test ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured

   Doc-tests writing_an_interpreter_in_rust

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

BAM! That worked. Now let’s try the REPL:

$ cargo run
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/writing_an_interpreter_in_rust`

>> let add = fn(x, y) { x + y; };
Token { token_type: Let, literal: "let" }
Token { token_type: Ident, literal: "add" }
Token { token_type: Assign, literal: "=" }
Token { token_type: Function, literal: "fn" }
Token { token_type: LeftParenthesis, literal: "(" }
Token { token_type: Ident, literal: "x" }
Token { token_type: Comma, literal: "," }
Token { token_type: Ident, literal: "y" }
Token { token_type: RightParenthesis, literal: ")" }
Token { token_type: LeftBrace, literal: "{" }
Token { token_type: Ident, literal: "x" }
Token { token_type: Plus, literal: "+" }
Token { token_type: Ident, literal: "y" }
Token { token_type: Semicolon, literal: ";" }
Token { token_type: RightBrace, literal: "}" }
Token { token_type: Semicolon, literal: ";" }
Token { token_type: EndOfFile, literal: "" }
>>

That’s it! This is my current implementation of chapter 1 of Writing An Interpreter In Go! As mentioned, I’m still fairly new to Rust, and I’d welcome pull requests or hints on howto make our Monkey interpreter in Rust more idiomatic. Feedback welcome!

You can find my contact details here.

Note: There’s a follow up post reviewing changes I’ve made with the help of the Rust community. They improved the overall performance by the factor 10!

Use and automate letsencrypt certificates (ACME) in an high availability environment

Mon, 14 Nov 2016 00:00:00 +0000

Mozilla launched a “free, automated and open” certificate authority called Let’s encrypt. As the name suggests, it provides free certificates trusted by all (major) browsers and operating systems. I’m using it heavily (on this blog, for example).

This blog post shows how Syncthing can be used to deploy letsencrypt certificates in an environment with multiple servers (e.g. in a round-robin scenario) without adding a single-point-of-failure.

ACME #

Let’s encrypt automated the process of requesting and authenticating a certificate using a protocol called ACME. The client requesting a new certificate uses a .well-known path on its webserver where it places a challenge, and Let’s encrypt retrieves this challenge for authentification.

The actual process is a little more complicated, though. If you want to know how it works in detail, I recommend Let’s encrypt’s excellent ACME documentation.

The problem in high availability setups #

When using multiple servers for SSL termination (e.g. in the load-balancing scenario described in the picture below, where SSL termination is handled by the nginx instances) each one requires a certificate for the domain(s) they are serving.

In a setup that e.g. uses a round-robin, we can’t guarantee that the incoming request for the ACME challenge ends up on the server actually requesting the certificate. Furthermore, each server needs to request (and renew) its own certificates.

The cleanest solution I found for this problem is to share the .well-known challenge directory (and maybe even the certificate) between multiple servers.

Syncthing to the rescue! #

The tool I found best to syncronize those directories was Syncthing. It is one of the most exiting tools for file-sharing, as it is completely decentralized and works without any central server (but can be configured to use one, if required), is fully peer-to-peer, open-soure, written in Go and cross-platform.

Syncthing fulfills all items on my wishlist:

Traffic between the instances is encrypted
The setup is automatically deployable
Instances can be easily added or removed
No single-point-of-failure (all nodes connect to each other, syncronizing the same directory between all machines)
No additional services required

I chose it to syncronize the /etc/nginx/certs directory. It shares the dhparams, SSL certificates and the ACME challenges between all nginx instances. Here’s what the shared directory looks like:

$ tree -a
.
├── .stfolder
├── acme
│   └── .well-known
│       └── acme-challenge
│           ├── 8xdoeH5OLPUij4xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
│           ├── cWaLNpzt_8v--xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
│           └── _wsvWIOyvP-45vt-xxxxxxxxxxxxxxxxxxxxxxxxxxx
├── dhparam.pem
├── www.example.com.crt
└── www.example.com.key

3 directories, 7 files

Implementation #

We’re using Chef to automate our infrastructure at flinc, but the process should be easily adaptable to a automation tool of your choice.

Syncthing is easily deployed, as there’s an official repository available:

Install Syncthing #

# Use the official Syncthing apt repositories
apt_repository 'syncthing-release' do
  uri 'http://apt.syncthing.net'
  distribution 'syncthing'
  components %w(release)
  key 'https://syncthing.net/release-key.txt'
end

package 'syncthing'

Set up systemd service #

A systemd.service is quickly crafted from the provided example:

[Unit]
Description=Syncthing - Open Source Continuous File Synchronization for %I
Documentation=man:syncthing(1)
After=network.target

[Service]
User=%i
ExecStart=/usr/bin/syncthing -no-browser -no-restart -logflags=0
Restart=on-failure
SuccessExitStatus=3 4
RestartForceExitStatus=3 4

[Install]
WantedBy=multi-user.target

Generate Syncthing configuration and keys #

We want to centrally manage our instances, so Syncthing certificates are stored centrally in Chef’s encrypted data bags, alongside their device IDs and API keys. Here’s how to generate and extract everything that’s required:

First, generate a new key-pair and save the device ID and API key for each node:

NODE=1.nginx.example.com
syncthing --generate=$NODE |grep ID |awk '{ print $5 }' > $NODE/device_id
grep apikey $NODE/config.xml |cut -d\> -f2 |cut -d\< -f1 > $NODE/apikey
rm $NODE/config.xml

The resulting key.pem and cert.pem will then be deployed into the .config/syncthing directory on the target machine.

After using Syncthing’s web-interface to configure the share, the resulting config.xml was then used to craft the following ERB template:

<configuration version="16">
    <!-- This is our shared folder. Scan it every 5s, so updates are syncronized quickly -->
    <folder id="vm623-hxlsp" label="letsencrypt" path="/etc/nginx/certs/" type="readwrite" rescanIntervalS="5" ignorePerms="false" autoNormalize="true">

        <!-- Share the folder between all nodes -->
        <% @nodes.each do |name, config| %>
          <device id="<%= config['id'] %>"></device>
        <% end %>

        <!-- Share settings. Default settings, with simple versioning -->
        <minDiskFreePct>1</minDiskFreePct>
        <versioning type="simple">
            <param key="keep" val="5"></param>
        </versioning>
        <copiers>0</copiers>
        <pullers>0</pullers>
        <hashers>0</hashers>
        <order>random</order>
        <ignoreDelete>false</ignoreDelete>
        <scanProgressIntervalS>0</scanProgressIntervalS>
        <pullerSleepS>0</pullerSleepS>
        <pullerPauseS>0</pullerPauseS>
        <maxConflicts>10</maxConflicts>
        <disableSparseFiles>false</disableSparseFiles>
        <disableTempIndexes>false</disableTempIndexes>
    </folder>

    <!-- Make sure all nodes are connected to one another -->
    <% @nodes.each do |name, config| %>
      <device id="<%= config['id'] %>" name="<%= name %>" compression="metadata" introducer="false">
          <address><%= config['address'] %></address>
      </device>
    <% end %>

    <gui enabled="true" tls="false" debugging="false">
        <address>127.0.0.1:8384</address>
        <apikey><%= @apikey %></apikey>
        <theme>default</theme>
    </gui>
    <options>
        <listenAddress>default</listenAddress>

        <!-- Disable announcement, as we're automatically adding all servers above -->
        <globalAnnounceServer>default</globalAnnounceServer>
        <globalAnnounceEnabled>false</globalAnnounceEnabled>
        <localAnnounceEnabled>false</localAnnounceEnabled>
        <localAnnouncePort>21027</localAnnouncePort>
        <localAnnounceMCAddr>[ff12::8384]:21027</localAnnounceMCAddr>

        <maxSendKbps>0</maxSendKbps>
        <maxRecvKbps>0</maxRecvKbps>
        <reconnectionIntervalS>60</reconnectionIntervalS>
        <relaysEnabled>false</relaysEnabled>
        <relayReconnectIntervalM>10</relayReconnectIntervalM>
        <startBrowser>false</startBrowser>
        <natEnabled>false</natEnabled>
        <natLeaseMinutes>60</natLeaseMinutes>
        <natRenewalMinutes>30</natRenewalMinutes>
        <natTimeoutSeconds>10</natTimeoutSeconds>
        <urAccepted>1</urAccepted>
        <urUniqueID></urUniqueID>
        <urURL>https://data.syncthing.net/newdata</urURL>
        <urPostInsecurely>false</urPostInsecurely>
        <urInitialDelayS>1800</urInitialDelayS>
        <restartOnWakeup>true</restartOnWakeup>
        <autoUpgradeIntervalH>12</autoUpgradeIntervalH>
        <keepTemporariesH>24</keepTemporariesH>
        <cacheIgnoredFiles>false</cacheIgnoredFiles>
        <progressUpdateIntervalS>5</progressUpdateIntervalS>
        <symlinksEnabled>true</symlinksEnabled>
        <limitBandwidthInLan>false</limitBandwidthInLan>
        <minHomeDiskFreePct>1</minHomeDiskFreePct>
        <releasesURL>https://upgrades.syncthing.net/meta.json</releasesURL>
        <overwriteRemoteDeviceNamesOnConnect>false</overwriteRemoteDeviceNamesOnConnect>
        <tempIndexMinBlocks>10</tempIndexMinBlocks>
    </options>
</configuration>

Deploy Syncthing configuration #

Here’s how we deploy Syncthing keys and configuration from encrypted data bags to the nginx nodes (Note: It probably makes sense to use run Syncthing as the same user as nginx, as Syncthing needs to deploy a key that should only be readable by nginx and noone else):

# Set this to the home directory of your user (probably the same user running nginx)
user = 'nginx'

# Populate node information from data bag
node_config = {}
node_list.each do |node_name|
  config = Chef::EncryptedDataBagItem.load('syncthing', node_name, data_bag_secret)
  node_config[node_name] = {}
  node_config[node_name]['id'] = config['device_id']

  # Set address to "dynamic" if it's ourselves
  node_config[node_name]['address'] = if node.name == node_name
    'dynamic'
  else
    "tcp://#{node_name}.#{node['domain']}:22000"
  end
end

# Deploy Syncthing certificate (from data bag)
local_config = Chef::EncryptedDataBagItem.load('syncthing', node.name, data_bag_secret)
%w(key cert).each do |k|
  # Show an error message if key couldn't be retrieved
  Chef.fatal("#{k}.pem is empty!") unless local_config[k]

  file "/home/#{user}/.config/syncthing/#{k}.pem" do
    mode 0o600
    owner  user
    group  user
    content local_config[k]
  end
end

# Deploy Syncthing configuration
template "/home/#{user}/data/.config/syncthing/config.xml" do
  mode   0o600
  owner  user
  group  user
  source 'syncthing.config.xml.erb'
  variables nodes: node_config, apikey: local_config['apikey']
end

# Restart Syncthing upon configuration/ key changes
service "syncthing@#{user}" do
  subscribes :restart, "template[/home/#{user}/.config/syncthing/config.xml]"
  subscribes :restart, "template[/home/#{user}/.config/syncthing/key.pem]"
  subscribes :restart, "template[/home/#{user}/.config/syncthing/cert.pem]"
  action [:enable, :start]
end

Restrict Syncthing to private backnet #

We have a dedicated backnet for all environments. Syncthing should only be allowed on this specific backnet (in our case eth1). I’m using the iptables-ng cookbook to manage iptables.

# Allow Syncthing in backnet only
iptables_ng_rule '50-syncthing' do
  rule ['-i eth1 --protocol tcp --dport 22000 --match state --state NEW --jump ACCEPT',
        '-i eth1 --protocol udp --dport 21025 --match state --state NEW --jump ACCEPT']
end

Get the certificates and automate renewal #

To actually request the certificate, the acme cookbook got you covered, which uses the ruby ACME library acme-client under the hood.

# Get some bonus points for generating your own Diffie-Hellmann parameters:
execute 'openssl dhparam -out /etc/nginx/certs/dhparam.pem 2048' do
  creates '/etc/nginx/certs/dhparam.pem'
  notifies :restart, 'service[nginx]'
end

# Make sure acme-client gem is installed
include_recipe 'letsencrypt::default'

# Create a webroot for acme challenges
directory '/etc/nginx/certs/acme' do
  owner user
  group user
end

# Deploy nginx site to answer ACME challenges
template '/etc/nginx/conf.d/letsencrypt.example.com.conf' do
  mode      0o644
  source   'letsencrypt.nginx.erb'
  notifies :restart, 'service[nginx]', :immediately
  not_if   'test -f /etc/nginx/certs/www.example.com.crt'
end

letsencrypt_certificate 'www.example.com' do
  alt_names %w(example.com)
  owner     user
  group     user
  fullchain '/etc/nginx/certs/www.example.com.crt'
  key       '/etc/nginx/certs/www.example.com.key'
  method    'http'
  wwwroot   '/etc/nginx/certs/acme'
  notifies  :restart, 'service[nginx]'
end

# Remove temporary letsencrypt site
file '/etc/nginx/conf.d/letsencrypt.example.com.conf' do
  notifies :restart, 'service[nginx]', :immediately
  action :delete
end

The temporary letsencrypt.nginx.erb

server {
    # This is for HAproxy with proxy_protocol, adapt if necessary
    listen [::]:80 ipv6only=off proxy_protocol;

    # Serve well-known path for letsencrypt
    location /.well-known/acme-challenge {
        root /etc/nginx/certs/acme;
        default_type text/plain;
    }
}

Also make sure to include something like this to your actual nginx site configuration, so challenges of automatic renewals can be answered:

server {
    # This is for HAproxy with proxy_protocol, adapt if necessary
    listen [::]:80 ipv6only=off proxy_protocol;

    # Use remote_addr from proxy_protocol
    real_ip_header proxy_protocol;
    set_real_ip_from 10.13.37.0/24;

    # Serve well-known path for letsencrypt
    location /.well-known/acme-challenge {
        root /etc/nginx/certs/acme;
        default_type text/plain;
    }

    location / {
        return 301 https://<%= @domain %>$request_uri;
    }
}

server {
    # This is for HAproxy with proxy_protocol, adapt if necessary
    listen [::]:443 ssl http2 ipv6only=off proxy_protocol;

    [...]
}

Wrap up #

That’s it! We can now automatically request and renew free Let’s encrypt SSL certificates in our high availability setup! Syncthing will happily keep the certificates and challenges in sync, even if some nodes go down. More nodes can be added by simply adding the credentials to the syncthing data bag, and the configuration of all nodes will adapt automatically.

If you have some feedback, feel free to contact me. I’m also available for hire as a freelancer.

pg-cert-check: A tool to monitor postgresql database SSL certificates

Mon, 24 Oct 2016 00:00:00 +0000

I recently wrote pg-check-cert, a small tool to check whether a postgresql server’s SSL certificate is about to expire. It was featured in PostgreSQL Weekly Issue 164.

This script connects to a postgresql instance, checks the certificate and displays the amount of days left before it expires. It’s intended to be used for monitoring your postgresql certificates, using a monitoring tool like Zabbix or Nagios.

Why openssl is not enough #

I used to monitor my postgresql certificates using openssl. Unfortunately, the openssl s_client option does not support the postgresql handshake, and can therefore only look at the .crt file to monitor the expiration date. As postgresql needs to be restarted after the .crt file was replaced, the actual file might be updated, but postgresql is still using the old certificate in-memory, until the server is restarted (as of postgresql-9.5, reload is not sufficient to read in the new certificate).

Installation #

Precompiled versions (linux-amd64, osx-amd64) are available on the release page. Download the file, extract it and move it to e.g. /usr/local/bin.

Build #

go build -o pg-check-cert *.go

Usage #

pg-check-cert localhost:5432

Thanks #

thusoy/postgres-migm for the inspiration.
buf.go and conn.go are taken from lib/pq, see copyright notice in the respective files.

entren - I just stumbled upon a traffic analyser I wrote in C when I was like 17

Wed, 03 Aug 2016 00:00:00 +0000

Around the year 2000-2002 (back in the days when Snort was still super young (and I haven’t heard of it yet), I decided to write a small network traffic analyser, which could serve as a “poor man’s intrusion detection system”. It was basically a C daemon configured with a ini-like configuration file, watching for network events.

If I remember correctly, I wanted a way to detect the (then pretty new and fancy) nmap stealth scan mechanisms (like half open, xmas, etc), and counter them with alerts and on demand firewall rules.

I just stumbled upon the old C source on my ~~harddrive~~ ssd on my macOS machine and tried make just to see what would happen. Back then I operated some FreeBSD/ OpenBSD and Linux servers, and I didn’t really expect much to happen besides tons of errors.

As it apparently still compiles and runs today (even on macOS), I’ve decided to upload the code to Github.

Checkout the configuration examples. Detecting the various nmap portscan types still works (see configuration below)!

# rule to detect tcp connect/half open portscans
[tcp]

    # tcp connect / half open
    tcp_flags = syn !ack

    # after 20 packages in 60 seconds a scan
    count = 20
    time  = 60

    # we need the portscan mode
    portscan_mode = 1


    #############################################################
    # possible actions against the attacker
    # (create a new firewall rule to block his ip for 30 seconds)
    #
    #    command1     = ipfw add deny ip from %sip to any
    #    delay        = 30
    #    command2     = ipfw del deny ip from %sip to any
    #
    #############################################################

    # the string for syslog
    logstr = portscan from: %sip [tcp connect/half open]



# rule to detect "null" scans
[tcp]

    # null scan
    tcp_flags = !syn !ack !psh !rst !urg !fin

    # after 10 packages in 60 seconds a scan
    count = 10
    time  = 60

    # we need the portscan mode
    portscan_mode = 1


    #############################################################
    # possible actions against the attacker
    # (create a new firewall rule to block his ip for 30 seconds)
    #
    #    command1     = ipfw add deny ip from %sip to any
    #    delay        = 30
    #    command2     = ipfw del deny ip from %sip to any
    #
    #############################################################

    # the string for syslog
    logstr = portscan from: %sip [null scan]



# rule to detect "xmas" portscans
[tcp]

    # nmap xmas scan
    tcp_flags = fin urg psh

    # after 10 packages in 60 seconds a scan
    count = 10
    time  = 60

    # we need the portscan mode
    portscan_mode = 1

    #############################################################
    # possible actions against the attacker
    # (create a new firewall rule to block his ip for 30 seconds)
    #
    #    command1     = ipfw add deny ip from %sip to any
    #    delay        = 30
    #    command2     = ipfw del deny ip from %sip to any
    #
    #############################################################

    # the string for syslog
    logstr = portscan from: %sip [xmas scan]



# rule to detect fin portscans
[tcp]

    # fin scan
    tcp_flags = !syn !ack !rst !urg !psh fin

    # after 50 packages in 30 seconds a scan
    count = 50
    time  = 30

    # we need the portscan mode
    portscan_mode = 1


    #############################################################
    # possible actions against the attacker
    # (create a new firewall rule to block his ip for 30 seconds)
    #
    #    command1     = ipfw add deny ip from %sip to any
    #    delay        = 30
    #    command2     = ipfw del deny ip from %sip to any
    #
    #############################################################

    # string for syslog
logstr = portscan from: %sip [fin]

Running rabbitmq on hosts with numeric hostnames

Wed, 08 Jun 2016 00:00:00 +0000

I encountered the following issue when running RabbitMQ on a host with the hostname 1.rabbitmq.staging:

$ sudo apt-get install rabbitmq-server
[...]
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
invoke-rc.d: initscript rabbitmq-server, action "start" failed.
dpkg: error processing package rabbitmq-server (--configure):
 subprocess installed post-installation script returned error exit status 1

dmesg shows a segfault happened:

$ dmesg
[...]
[ 7891.350558] async_4[29072]: segfault at 0 ip 000000000055ff71 sp 00007f413997de40 error 4 in beam.smp[400000+234000]

When manually starting rabbitmq-server, I get the following message:

$ rabbitmq-server
ERROR: epmd error for host "1": badarg (unknown POSIX error)

The Bug #

I found a bug report for the Ubuntu package rabbitmq-server, but as of today has no replies. Apparently, it’s not possible to run rabbitmq-server on a host that has a numeric hostname. I assume this is a bug in either Erlang or RabbitMQ and will eventually be fixed, hopefully.

RabbitMQ interprets everything after the first . in the hostname as domain, and therefore comes to the conclusion 1 is the full hostname.

The Workaround #

The following workaround fixed it for me.

On Ubuntu, I added the following to /etc/rabbitmq/rabbitmq-env.conf:

# RabbitMQ cannot cope with hostnames containing only numbers.
# "1.rabbitmq.env" will be interpreted as hostname "1", and then fail.
# The workaround for this is manually adapting the HOSTNAME variable in rabbitmq-env.conf
# and making sure that it's resolvable in /etc/hosts
HOSTNAME=rabbitmq

Make sure to also add an entry for rabbitmq to your /etc/hosts file, otherwise rabbitmq-server will complain that it’s not resolvable:

# Resolve rabbitmq to 127.0.2.2, so we're not interferring with other localhost hostnames
127.0.2.2 rabbitmq

After that, starting rabbitmq-server works like a charm.

Note: Modifying HOSTNAME like this might cause other issues. I didn’t test it on clustered setups yet. Let me know if you know more!

Homebrew betrayed us all to Google

Tue, 26 Apr 2016 00:00:00 +0000

Homebrew is arguably the best package manager for OSX around. It’s a great project, I’ve been using it for years, and it’s doing what it’s supposed to in a very clean manner. Unfortunately, the team decided to track the behaviour of its users via Google Analytics.

This is bad.

Open Source is about trust. Trust is underminded by things like tracking.
Do not track your users. In the rare case you really need anonymous data, ask your users first.
Never use Google products (or any other “big data” company that relies on making money out of the data you provide) to track your users.
Using Google’s tracking and then calling it “anonymous” is a lie. Google collects tons of information of its users and even non-users. There’s no way to know what data Google will relate internally. Even if you don’t get to see all of the collected information, Google still has them.
Opt-out is never an excuse. It always excludes most users (which either don’t care, or have more severe things to care about than protecting their privacy in every random app they’re using).

Read on to lean howto fix the issue for at least yourself.

Edit: I’ve been contacted by a Homebrew maintainer and have been asked to remove the link to the Github issue. His arguments about keeping the issue to the topic convinced me, and I was sad to hear that he apparently received tons of personally insulting emails. When I asked for a more appropriate way to provide feedback, he didn’t give any and stated that they won’t make the tracking procedure opt-in. They also have removed all comments regarding complaints about their tracking from the issue. I wrote a final comment in the comment section, which was deleted immediately. For completeness, I’m quoting it here:

While I can understand you want to keep this Issue productive, I find it disturbing how Homebrew deals with massive complaints from an open-source-loving community (see HN, Reddit, etc. discussions). Don’t just delete this away and hope everybody just forgets about this.

I know this will probably be deleted in a minute and you guys will probably ban me, but I’m still writing this so people get notified via email/ etc.

There are people out there for whom this is a serious issue. I beg you do not take this lightly. I want to tell people “Homebrew is good, use it” and not “Homebrew is kinda good, but before you use it make sure to paste this-and-this in your terminal and also block this and that, otherwise you’ll be tracked”.

Fixing the issue #

Homebrew ships with an option to opt-out of the tracking.

Set the environment variable HOMEBREW_NO_ANALYTICS=1
Or run git config --file="$(brew --repository)/.git/config" --replace-all homebrew.analyticsdisabled true

A more general approach (as there are other programs out there, which are less nice than Homebrew) is to block known tracking hosts. This can be either done using something like Little Snitch, or by setting up an adblocking /etc/hosts file. The latter option is free and pretty straightforward. You can use the following script to collect well-known tracking services and tell your computer to not resolve them.

#!/bin/sh
#
# This code is GPLv3

HOSTSFILE=hosts
TMPFILE=/tmp/aosp-hosts-file

echo "Updating adblocking hosts file..."

curl "http://hosts-file.net/download/hosts.txt" > $TMPFILE
curl "http://winhelp2002.mvps.org/hosts.txt" >> $TMPFILE
curl "http://pgl.yoyo.org/adservers/serverlist.php?mimetype=plaintext&hostformat=hosts" >> $TMPFILE
curl "http://someonewhocares.org/hosts/hosts" >> $TMPFILE

echo "# This is a generated hostfile for adblocking.
# Sources:
# - http://hosts-file.net/download/hosts.txt
# - http://www.mvps.org/winhelp2002/hosts.txt
# - http://pgl.yoyo.org/adservers/serverlist.php?mimetype=plaintext&hostformat=hosts
# - http://someonewhocares.org/hosts/hosts

# Localhost entries
127.0.0.1 localhost
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
" > $HOSTSFILE

# Streamline all entries to use 0.0.0.0
gsed -i 's/127\.0\.0\.1/0\.0\.0\.0/' $TMPFILE

# Remove other localhost entries
gsed -i '/localhost/d' $TMPFILE

# Remove comments
gsed -i 's/#.*$//g' $TMPFILE

# Replace tabs with spaces
gsed -i 's/\t/ /g' $TMPFILE

# Also block ipv6
gsed -i 's/\(0\.0\.0\.0\)\(.*\)/\1\2\n::0\2/' $TMPFILE

# Sort and remove duplicates and empty lines
sort -u $TMPFILE |grep -v '^\s*$' >> $HOSTSFILE

dos2unix $HOSTSFILE

rm $TMPFILE

This script will download known tracking services or other malicious domains from three different sources, removes duplicates and makes sure both ipv4 and ipv6 addresses are blocked. Save it as get-hosts.sh and you can use it like so:

# Install dependencies, but make sure tracking is disabled
export HOMEBREW_NO_ANALYTICS=1
brew install dos2unix
brew install gnu-sed

# Generate hosts file
chmod +x get-hosts.sh
./get-hosts.sh

# Inspect the hosts file if your paranoid, then deploy it
sudo mv hosts /etc/hosts

What Homebrew should do #

Homebrew should (really!) make the tracking opt-in. A “really ask your user whether it’s ok” opt-in.
It should really consider other methods for the goals it wants to achieve (like finding out “most-used” packages), and make sure they are really necessary.

Dualstack multiple IP addresses with systemd-networkd

Wed, 24 Feb 2016 00:00:00 +0000

I’m using systemd-networkd on Archlinux on one of my servers to configure the static IP addresses. While this seems pretty straight-forward, there’s a big issue that you can bump into when trying to configure multiple IP addresses. As this took me some time to figure out and it’s not well documented, I decided to leave a blog post for future me (and possibly others).

Apparently, the configuration described in the Archlinux wiki is not feasible when trying to configure multipe addresses. You need to use the alternative method using one [Network] block. Using multiple Address= lines in the [Address] block is apparently not valid. Unfortunately, systemctl restart systemd-networkd doesn’t complain about an invalid configuration, but instead leaves you without any network config at all - Which then requires a rescue system to boot when you’re on a server without physical access.

So here’s the working configuration I ended up with:

# cat /etc/systemd/network/eno0.network

[Match]
Name=eno1

[Network]
DNS=2.x.x.x
DNS=2.x.x.x

Address=1.x.x.x/26
Address=2001:xxxx:xxxx::1/64
Address=2001:xxxx:xxxx::2/64

Gateway=1.x.x.x

Update: According to this post I wasn’t reading through man systemd.network carefully enough. It’s also possible to just use multiple [Address] sections with one IP each.

Increase password entropy on developermail.io

Wed, 29 Jul 2015 00:00:00 +0000

I recently co-founded an email SaaS for developers called developermail.io where tech-savy people can configure their email mailboxes using git. We just released a new feature, which enables you to use high-entropy passwords with our services.

In this blogpost I’ll quickly show you howto generate more secure passwords for your developermail.io account and mailboxes.

Increase mailbox password entropy #

On developermail.io, we generate strong passwords for our users. You can now set the length of those passwords in the YAML configuration files.

To increase the length of the generated password for your mailbox, add the password_length attribute to your config.yaml:

developermail.io:
  you:
    # This will set the password length for the mailbox
    # you@developermail.io to 30 characters
    password_length: 30

Simply git push your configuration, and the changes are live!

git add config.yaml
git commit am 'Increase password length for mailbox'
git push

Once your configuration was pushed and updated successfully, you can trigger a password change to retrieve a new password. This can be done using a curl request. See the official documentation for details.

Change your password using the credentials for the mailbox

curl -X POST -u "you@developermail.io" "https://developermail.io/api/mailboxes/change-password"

Alternatively, you can also trigger a password change using your account login (the one you also use for git access)

curl -X POST -u you "https://developermail.io/api/mailboxes/change-password" --data '{"user": "chr4", "domain": "developermail.io"}'

You’ll retrieve a JSON response including your new high-entropy password:

{
  "error": false,
  "message": "Password successfully changed.",
  "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "username": "chr4@developermail.io"
}

Increase account password entropy #

As your account has access to the mailbox configurations, do not forget to also adapt your account password length. The process is analog to the mailboxes covered above.

First, set the account_password_length attribute to the desired length in your settings.yaml. In case you do not have a settings.yaml file in your repository, simply create it.

# This sets the password length for your account to 50
account_password_length: 50

git add settings.yaml
git commit -m 'Increase password length for account'
git push

Once the new settings are pushend (and therefore live), you can request a new password using a curl request. See the official documentation for details. Use your account credentials for this request (the same you’re using for git)

curl -X POST -u "examplecom" "https://developermail.io/api/accounts/change-password"

If you entered the correct password, you’ll receive a JSON response including your brand new, high entropy password:

{
  "error": false,
  "message": "Password successfully changed.",
  "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "username": "examplecom"
}

Howto secure openssh-6.x

Sat, 13 Sep 2014 00:00:00 +0000

Since OpenSSH 6.x came out, a lot of new ciphers where introduced. I was wondering, which ones where the best and what I should use, and I read a few articles on the internet to find out.

I’m certainly not a cryptographer, so if you have any suggestions howto further improve the configuration below, feel free to contact me.

As a general statement, one should avoid ECDSA and use Ed25519 instead, and due to the fixed key length of DSA that ssh-keygen uses, DSA should also be avoided. RSA keys should be at least 2048 bits long, perhaps 4096 bits is the better choice.

Note: Most of the settings covered in this post are incompatible with openssh-5.x. Consider upgrading!

Availablility of openssh-6.x #

Ubuntu 14.04 ships with openssh-6.6
Archlinux ships with newest openssh, due to its rolling release package management
MacOS Mavericks ships with openssh-6.2, you can install openssh-6.6 using Homebrew
Debian Wheezy ships openssh-6.0 (Note: Some covered settings are not compatible with OpenSSH < 6.4)
RHEL 5.x ships openssh-5.4 :(

You can configure your ssh to prefer good ciphers on both, the client and the server side.

Securing the ssh client configuration #

There’s two files you can configure your ssh client with

/etc/ssh/ssh_config (Global configuration, for all users)
~/.ssh/config (Your users configuration)

Place the configuration for all hosts at the bottom of the file, and override this default settings with entries for individual hosts/networks with entries placed above (This is the way how the configuration file is read).

So we should start with settings for individual hosts. Here’s the settings I use for Github, as Github doesn’t support recent ciphers unfortunately :(

Host github.com
  # Github doesn't support decent ciphers, using the best available
  Ciphers       aes256-ctr
  MACs          hmac-sha2-512
  KexAlgorithms diffie-hellman-group14-sha1
  IdentityFile  ~/.ssh/id_rsa

In the same way, you can add cipher (as well as other) specifications for other hosts, e.g.:

# This is a host with OpenSSH < 6.4
Host myoldhost.com
  User              katie
  HostKeyAlgorithms ssh-rsa
  Ciphers           aes256-ctr
  MACs              hmac-sha2-512

And finally, here’s the global defaults, using only secure ciphers.

Host *
  # Use only secure ciphers
  # Never use ECDSA/DSA, prefer Ed25519, use RSA as fallback
  # In case you need to support openssh-server versions < 6.4,
  # you need to add ssh-rsa, aes256-ctr and hmac-sha2-512 :(
  #
  # Update 2015-08-14: Remove ssh-rsa-cert-v00@openssh.com, was deprecated by openssh-7.0
  HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,ssh-rsa-cert-v01@openssh.com
  Ciphers           chacha20-poly1305@openssh.com,aes256-gcm@openssh.com
  MACs              hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com
  KexAlgorithms     curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

  # Prefer Ed25519 over RSA, never use DSA/ECDSA
  IdentityFile ~/.ssh/id_ed25519
  IdentityFile ~/.ssh/id_rsa

  # Display randomart images of hostkeys
  VisualHostKey yes

Securing openssh-server #

Server side config resides in /etc/ssh/sshd_config. I’m mostly covering the security/cipher related configuration settings here. Basically, the configuration resembles the client configuration for most of the settings.

# Using a non-standard ssh port is just security by obscurity
# Port 22

# In general, please use pub/priv authentication instead of passwords
PasswordAuthentication no

# Speedup login process on machines that have no proper DNS settings, protect
# privacy, no real security drawback
UseDNS no

# Use only secure ciphers
# Never use ECDSA/DSA, prefer Ed25519, use RSA as fallback
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

# Add aes256-ctr for compatibility with older clients
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com

# Add hmac-sha2-512 for compatibility with older clients
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

Issues #

Ruby net/ssh library #

The Ruby net/ssh library cannot deal with the new ciphers yet. I tried to fix the library, but couldn’t quite fix all the issues. Please see this pull request, and feel free to contribute!

Popular programs using this library are among others Vagrant and knife ssh.

There best workaround I found was overriding the SSH_AUTH_SOCK variable when using those programs, resulting in ignoring the unknown keys in ssh-agent:

SSH_AUTH_SOCK='' vagrant up

OpenSSH versions < 6.x #

When dealing with OpenSSH clients/servers < 6.x, you might add more exceptions into your ssh_config resp. sshd_config. The settings I use for Github above might be a good starting point. ssh -v usually gives good hints which ciphers you need to enable.

Chef sshd cookbook #

If you want to deploy ssh configurations for multiple hosts, you might want to have a look on my sshd cookbook for Chef.

conf.d like directories for zsh/bash dotfiles

Wed, 10 Sep 2014 00:00:00 +0000

I don’t like messy dotfiles.

The thought of having tons of random configuration entries in files like my .zshrc really bothers me, so I implemented something that works like a conf.d like directory structure for my shell dotfiles.

I also still use the bash shell in certain situations, and I want a more or less consistent environment, no matter which shell I use (bash, zsh).

With the following setup, it’s possible to have the following:

A directory called zshrc.d, which includes multiple, zsh related configuration files.
A directory called bashrc.d, including multiple configuration files concerning bash.
A directory called rc.d, including configuration items needed by both shells.
A directory called login.d, including elements included by .bash_profile resp. .zlogin.

This keeps your dotfiles nice and clean, and also allows you do have additional files on systems where you need them, without them being included on all your systems (e.g. your $GOPATH).

Here are the dotfiles, that do nothing but include all files in the described directories:

# .bashrc
if [[ $- != *i* ]] ; then
  # shell is non-interactive. be done now!
  return
fi

# Load all files from .shell/bashrc.d directory
if [ -d $HOME/.shellrc/bashrc.d ]; then
  for file in $HOME/.shellrc/bashrc.d/*.bash; do
    source $file
  done
fi

# Load all files from .shell/rc.d directory
if [ -d $HOME/.shellrc/rc.d ]; then
  for file in $HOME/.shellrc/rc.d/*.sh; do
    source $file
  done
fi

# .bash_profile

if [ -f $HOME/.bashrc ]; then
  source $HOME/.bashrc
fi

# Load all files from .shell/login.d directory
if [ -d $HOME/.shellrc/login.d ]; then
  for file in $HOME/.shellrc/login.d/*.sh; do
    source $file
  done
fi

# .zshrc

# Load all files from .shell/zshrc.d directory
if [ -d $HOME/.shellrc/zshrc.d ]; then
  for file in $HOME/.shellrc/zshrc.d/*.zsh; do
    source $file
  done
fi

# Load all files from .shell/rc.d directory
if [ -d $HOME/.shellrc/rc.d ]; then
  for file in $HOME/.shellrc/rc.d/*.sh; do
    source $file
  done
fi

# .zlogin

# Load all files from .shell/login.d directory
if [ -d $HOME/.shellrc/login.d ]; then
  for file in $HOME/.shellrc/login.d/*.sh; do
    source $file
  done
fi

No you can include all sorts of useful functions, settings and aliases by putting individual files in your .shellrc/* folders!

You can find my full .shellrc directory on Github. Have a look at the README.md for more information!

Happy housecleaning!

gittree: bash/zsh function to run git commands recursively

Wed, 10 Sep 2014 00:00:00 +0000

I’m using Androids repo tool from time to time when dealing with large groups of git repositories. In most situations, it is too bloated though. Some git “batch” commands I found very useful, like repo status, checking the status of all git repositories recursively.

To mimic this (and other) behaviour in a simple way, I created the following bash/zsh function (put this in your .bashrc or .zshrc, or another file where you define functions in your dotfiles)

# Recursively traverse directory tree for git repositories, run git command
# e.g.
#   gittree status
#   gittree diff
gittree() {
  if [ $# -lt 1 ]; then
    echo "Usage: gittree <command>"
    return 1
  fi

  for gitdir in $(find . -type d -name .git); do
    # Display repository name in blue
    repo=$(dirname $gitdir)
    echo -e "\033[34m$repo\033[0m"

    # Run git command in the repositories directory
    cd $repo && git $@
    ret=$?

    # Return to calling directory (ignore output)
    cd - > /dev/null

    # Abort if cd or git command fails
    if [ $ret -ne 0 ]; then
      return 1
    fi

    echo
  done
}

This allows me to quickly get an overview of multiple git repositories, and check e.g. whether they are all clean, or push them all at once:

dockerfiles % gittree status

./base-go
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

./dovecot
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

./nginx
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

  feeling-dirty

nothing added to commit but untracked files present (use "git add" to track)

./dnsmasq
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

gittree works with all git commands, try out e.g.

gittree status
gittree diff
gittree push
gittree --no-pager log --graph --pretty=format:'%h -%d %s (%cr) <%an>'

It’s a pretty simple script and for sure not perfect, but it get’s the job done!

If you have any suggestions how to improve it, please file a pull request on this file on Github.

Online resizing LVM guest-partitions on OpenNebula/libvirt guests

Thu, 28 Aug 2014 00:00:00 +0000

Today I found out howto resize guest partitions on OpenNebula (or in general with libvirt, which OpenNebula uses underneath for KVM virtualization).

I’m using a LVM storage for virtual machines. So resizing them is pretty easy.

Note: Replace the XX below with your virtual machines ID, as you can find out with e.g. onevm list or virsh list

host ~ $ sudo lvresize -L 50G /dev/volgrp/lv-one-XX
  Extending logical volume lv-one-XX to 50.00 GiB
  Logical volume lv-one-XX successfully resized

Then, you need to tell libvirt about the new size. This can be done with the following command: Note: The –path argument is the device name of the partition used by the guest. You can find it out by using virsh domblklist one-XX

host ~ $ sudo virsh blockresize one-XX --path vdX --size 50G
Block device 'vdX' is resized

Now simply login your guest machine and adapt the filesystem (no reboot required)

guest ~ $ sudo resize2fs /dev/vdX   # For ext filesystems
guest ~ $ sudo xfs_growfs /dev/vdX  # For xfs filesystems

Unfortunately, the new size will not display in the SIZE attribute of the image in OpenNebula (as seen with oneimage show XX). Apparently this is only set when creating the image.

As far as I know, this only works when using the virtio or scsi block drivers. In OpenNebula you can e.g. set DEV_PREFIX="vd" in your image configuration to use virtio.

Many thanks to Humble Chirammal for pointing out virsh blockresize.

Nested if workaround for Nginx to allow a specific ip address access to a disabled site

Tue, 29 Oct 2013 00:00:00 +0000

When doing maintenance on a web application, you probably have a custom 503 site, showing your customers that the servers are currently lying on the operating table.

At the dynamic ridesharing service flinc, we touch a certain file on our reverse proxies (e.g. using capistrano deploy:web:disable) when maintenance begins. Nginx then serves a static “we’ve disabled the site for maintenance” site, instead of the actual content.

But wouldn’t it be nice to test your web application before going live for your customers? It sure would. Unfortunately, this is not as simple as a task as you might think, because you cannot nest if directives in an Nginx location and if is evil.

Because of hose limitations, we need another creative approach. As try_files is also forbidden in if directives and only return and rewrite are safe commands to use let’s abuse the unused HTTP status code 418 to work around this.

location / {
    chunked_transfer_encoding off;
    error_page 418 = @try_files;
    error_page 503 = @maintenance;

    # allow out office ip even when in maintenance mode
    if ($remote_addr = 10.11.12.13) {
        return 418;
    }

    # we are in maintenance mode when this file is present
    if (-f $document_root/maintenance.html) {
        return 503;
    }

    return 418;
}

Basically, as long as the maintenance.html is not present or we are sitting in our office, we return the 418 status code. For customers, we’re serving 503 in case the site is disabled.

So let’s define the locations for normal operation (try serving the static files directly, and redirect all other requests to our upstream)

location @try_files {
    try_files $uri @proxy;
}

location @proxy {
    proxy_pass https://flinc;
}

And finally, our maintenance site. We’re serving maintenance.html instead of the requested site. As POST requests to static websites are not allowed and therefore all our disabled forms will return a 405 instead of the wanted 503, let’s correct those as well.

location @maintenance {
    # return 503 for non-GET requests
    # (returns 405 by default, because POST/etc to a static website is not allowed)
    if ($request_method != GET) {
        return 503;
    }

    try_files $uri /maintenance.html;
}

The last location directive makes sure that the static assets required for the maintenance page (images, stylesheets, etc) stored in /maintenance are served normally.

# don't hand out 503 headers when accessing maintenance assets
location /maintenance {}

iptables-ng cookbook for chef

Fri, 13 Sep 2013 00:00:00 +0000

Today, I released iptables-ng, a cookbook to maintain iptables rules on different machines using chef.

But why another cookbook? There are two fairly often used around

Well, I wanted a tool which can do all the following:

Configure iptables rules in a consistent and nice way for all distributions
Be configured by using LWRPs only
Be configured by using node attributes only
Respect the way the currently used distribution stores their rules
Provide a good-to-read and good-to-maintain way of deploying complex iptables rulesets
Provide a way of specifying the order of the iptables rules, in case needed
Only run iptables-restore once during a chef run, and only if something was actually changed
Support both, ipv6 as well as ipv4
Be able to assemble iptables rules from different recipes (and even cookbooks), so you can set your iptables rule where you actually configure the service

Respect the way the distribution handles iptables #

Every distribution has its own way of maintaining iptables rules. While iptables-restore provides a generic format/tool available to save and restore iptables rules, every distribution uses it differently.

Debian and Ubuntu #

Debian based systems provide a package called iptables-persistent, which provides a system service to save and load iptables rules. Debian Squeeze (and earlier) use an outdated version of iptables-persistent, which doesn’t support ipv6.

iptables-persistent stores the rules in these files:

# Debian
/etc/iptables/rules.v4 # ipv4
# ipv6 [not supported]

# Ubuntu
/etc/iptables/rules.v4 # ipv4
/etc/iptables/rules.v6 # ipv6

# RHEL #

Redhat (and clones like CentOS) provide a system service by default which loads iptables rules from the following files:

# RHEL
/etc/sysconfig/iptables  # ipv4
/etc/sysconfig/ip6tables # ipv6

# Gentoo #

Gentoo also provides system services by default, but stores the rules here:

# Gentoo
/var/lib/iptables/rules-save  # ipv4
/var/lib/ip6tables/rules-save # ipv6

# Archlinux #

Archlinux uses iptables-restore too, and stores its files here:

# Archlinux
/etc/iptables/iptables.rules  # ipv4
/etc/iptables/ip6tables.rules # ipv6

The new iptables-ng cookbook #

The iptables-ng cookbook supports all of those platforms transparently for the user, as well as trying to be as compatible as possible to the way of the currently used distribution.

It automatically saves the assembled rules to the locations specified above, and if the system doesn’t support its own service provider, it will fall back to apply to rules manually using iptables-restore. (This enables e.g. ipv6 support in Debian, yeee!)

It furthermore provides LWRPs for all its functions, while it also can be configured using only node attributes.

Examples #

Let me show you some examples

# Set default policy to DROP (whitelist mode)
# Unless it is already specified
%w{input output forward}.each do |chain|
  iptables_ng_policy chain.upcase do
    policy 'DROP [0:0]'
    action :create_if_missing
  end
end

# Drop invalid packages
iptables_ng_rule '10-input-drop-invalid' do
  rule '--match state --state INVALID --jump DROP'
end

# Drop ICMP timestamp packages (ipv4 only)
iptables_ng_rule '14-drop-icmp-timestamp' do
  rule [ '--protocol icmp --icmp-type timestamp-request --jump DROP',
         '--protocol icmp --icmp-type timestamp-reply --jump DROP' ]
  ip_version 4
end

# Masquerade (nat table automatically is only applied to ipv4)
iptables_ng_rule '99-masquerade' do
  chain 'POSTROUTING'
  table 'nat'
  rule  '-o eth0 -j MASQUERADE'
end

# Delete a rule
iptables_ng_rule '10-unwanted-rule' do
  action :delete
end

All rules will be written to /etc/iptables.d directory, resulting in the following directory structure:

├── filter
│   ├── FORWARD
│   │   └── default
│   ├── INPUT
│   │   ├── 10-input-drop-invalid.rule_v4
│   │   ├── 10-input-drop-invalid.rule_v6
│   │   ├── 14-drop-icmp-timestamp.rule_v4
│   │   └── default
│   └── OUTPUT
│       └── default
├── mangle
│   ├── FORWARD
│   │   └── default
│   ├── INPUT
│   │   └── default
│   ├── OUTPUT
│   │   └── default
│   ├── POSTROUTING
│   │   └── default
│   └── PREROUTING
│       └── default
├── nat
│   ├── OUTPUT
│   │   └── default
│   ├── POSTROUTING
│   │   ├── 99-masquerade.rule_v4
│   │   └── default
│   └── PREROUTING
│       └── default
└── raw
    ├── OUTPUT
    │   └── default
    └── PREROUTING
        └── default

At the end of the chef run, they will be assembled into a sinble iptables restore script. Finally, if the ruleset was changed, iptables-ng takes care of reloading the rules, using the systems provider (e.g. /etc/init.d/iptables). If the system doesn’t provide it’s own way of loading the rules, iptables-ng will fall back to iptables-restore.

Fore more examples, have a look at the README

Get it! #

You can grab your copy from these locations

ipswitch - migrate IP addresses without downtime

Tue, 13 Aug 2013 00:00:00 +0000

When doing quick maintenance tasks on a server, you can use the following approach to keep your site available:

Failover the backnet IP address of the host to another host
Use arping to tell the network that this IP was switched
Remove the IP from the host that needs maintenance

In case you do not have a full high availability setup available, you can use ipswitch, a small tool I wrote to assist with this kind of simple failover tasks.

You can install it using

$ gem install ipswitch

ipswitch connects to the remote hosts using SSH, and uses the “ip” command to maintain IP addreses.

Migrate an IP without downtime #

Let’s say you have an application server called rails-1.example.com, and you need to do some maintenance task on it (e.g. install a kernel update and reboot the machine). While you could modify all of your reverse proxys to exclude this server from their configuration, it is much simpler to just assign the (backnet) IP address of rails-1.example.com to one of your other application servers.

With ipswitch, this can be done using one command:

$ ipswitch migrate rails-1.example.com rails-2.example.com

rails-1.example.com: Getting IP for interface eth0
rails-1.example.com: Found IP 192.168.1.2/24
rails-2.example.com: Adding IP address 192.168.1.2/24 to interface eth0
rails-2.example.com: Running arpping
rails-2.example.com: 3 packets transmitted, 0 packets received, 100% unanswered (0 extra)
rails-1.example.com: Removing IP address 192.168.1.2/24 from interface eth0

Note: ipswitch automatically detects and uses the (primary) IP of the specified interface (default: eth0)

When your maintenance tasks are done, migrate the IP address back to the original host

$ ipswitch migrate rails-2.example.com rails-1.example.com --ip 192.168.1.2/24

rails-1.example.com: Adding IP address 192.168.1.2/24 to interface eth0
rails-1.example.com: Running arpping
rails-1.example.com: 3 packets transmitted, 0 packets received, 100% unanswered (0 extra)
rails-2.example.com: Removing IP address 192.168.1.2/24 from interface eth0

Note: As eth0 now has two IP addresses, you need to specify which once to migrate. If the original node reclaimed the IP automatically (e.g. due to a reboot), ipswitch still works.

Simple tasks #

You can also use ipswitch to just add/remove IP addresses from your nodes

$ ipswitch add --ip 192.168.1.1/24 example.com
$ ipswitch del --ip 192.168.1.1/24 example.com

Assign an IP to an interface other than eth0 and do not use arping to broadcast IP

$ ipswitch add --interface eth1 --ip 192.168.1.1/24 --no-broadcast example.com

Managing IPv6 addresses is also possible

$ ipswitch add --family inet6 --ip fe80::abcd:66ff:fede:9999 example.com
$ ipswitch del --family inet6 --ip fe80::abcd:66ff:fede:9999 example.com

Caveats #

Currently only works when using SSH keys for authenticating at your hosts. The following features are planned, and pull requests are apprechiated :)

Password support
SOCKS proxy support)
Ability to specify SSH keyfile
Support for sudo

apt-get cleanup commands

Sun, 04 Aug 2013 00:00:00 +0000

Just a short post about some useful cleanup commands for Debian and Ubuntu systems. There are (to my knowledge) no build in task solving the following things

Remove old kernels (while keeping the currently running and the latest)
Purge removed packages (especially after autoremoving unneeded dependencies)

Remove old kernels #

Debian and Ubuntu don’t remove old kernels when upgrading. Although this of course makes sense to keep the system bootable in case of a broken kernel, it can fill up /boot pretty quickly. Usually it should be sufficient to keep the currently running kernel, as well as the latest one. The rest can be safely deleted. This can be done with the following command:

Edit: Also purge other kernel packages, like modules and source

apt-get purge \$( \
    dpkg --list | \
    egrep 'linux-(image|image-unsigned|headers|modules|modules-extra|source|doc)-[0-9]' | \
    awk '{print \$3,\$2}' | \
    sort -nr | \
    tail -n +2 | \
    grep -v \$(uname -r) | \
    awk '{print \$2}' | \
    tr '[:space:]' ' ' \
)

A short explanation

apt-get purge             remove packages (and purge configuration) selected by the following lines
dpkg --list               list installed packages
egrep 'linux-(...)-[0-9]' grep installed kernels, modules, source, headers, ...
awk '{print $3,$2}'       we need the version, as well as the package name
sort -nr                  sort by version
tail -n +2                filter out latest kernel
grep -v $(uname -r)       filter out currently running kernel (failsafe)
awk '{print $2}'          cut everything but the package name
tr '[:space:]' ' '        make sure all whitespace characters are real spaces

Purge removed packages #

If you remove a package using

apt-get remove <packagename>

The packages configuration will be retained. Also, when autoremoving unneeded dependencies, apt-get by default removes packages instead of purging them.

To cleanup your system and purge all packages that are removed from the system and their dependencies, use this command

apt-get autoremove -y; apt-get purge -y $(dpkg --list |grep '^rc' |awk '{print $2}')

Explanation

apt-get autoremove -y  remove all dependencies no longer required
apt-get purge -y       purge packages selected with the following lines
dpkg --list            list installed packages
grep '^rc'             grep packages removed, but not purged (rc)
awk '{print $2}'       cut everything but the package name
tr '[:space:]' ' '     make sure all whitespace characters are real spaces

Chef cookbook #

Furthermore, I created the apt_cleanup Chef cookbook, which provides recipes to do all those tasks automatically.

For a convenient auto-cleanup, the following recipes can be used

apt_cleanup::default #

Includes all other cleanup recipes

apt_cleanup::remove_old_kernels #

Removes all old kernels, but the most recent as well as the currenlty used one.

apt_cleanup::remove_unneeded_packages #

Runs apt-get autoremove to remove packages not required anymore.

apt_cleanup::purge_removed_packages #

Purges already removed packages, to get rid of e.g. old config files.

apt_cleanup::clean_apt_cache #

Runs apt-get clean to remove .dpkg files from /var/cache/apt/archives.

Saltstack formula #

UPDATE 15th Oct. 2018: I’ve also created a salt formula to take care of cleaning up.

Either run it to clean up packages immediately:

# Clean up packages on this system
sudo salt-call state.apply apt.cleanup.now

# Clean up packages on all nodes
sudo salt '*' state.apply apt.cleanup.now

# Setup systemd timer to automatically clean packages once a day
sudo salt-call state.apply apt.cleanup

# Respectively, to install it on all nodes
sudo salt '*' state.apply apt.cleanup

Unlike Chef, Saltstack doesn’t run periodically. To run the cleanup scripts regularily (e.g. daily), the apt.cleanup state installs a systemd service and timer to cleanup your system automatically on a daily basis. Feel free to use the provided apt-cleanup.service and apt-cleanup.timer files independently of Saltstack!

Howto use chef with ssl

Thu, 01 Aug 2013 00:00:00 +0000

By default, the connections between the chef-client and the chef-server are not secured. This is a short post on howto encrypt and verify your connections.

As of chef-11 (unlike chef-10), SSL is enabled by default. But (naturally, as Opscode cannot create trusted certificates for your domain) the certificates are not verified. This essentially means that the connection is not secure at all.

Unless you only use chef in a trusted network, you should invest some time in securing your clients connections.

Installing a certificate on your server #

First, grab yourself a certificate for your chef server. This post will use a certificate from cacerg.org.

Installing the certificate to the chef-servers nginx is a little annoying. chef-11 (at least when installed using the official packages from opscode) uses a bundled nginx, instead of a system-wide one. Therefore, you need to change the configuration in /var/opt/chef-server/nginx/etc which feels a little unsafe, as it might get overwritten automatically by upgrades.

Fortunately, as Jeff Blaine pointed out in an email to me, that there’s a chef-server.rb config file which takes care of these problems.

First, set up a resource to reconfigure chef-server when something is changed:

execute 'chef-server-ctl reconfigure' do
  action :nothing
end

Deploy your certificate using the certificate cookbook. After that, notify the reconfigure resource, so it reconfigures in case we renew the certificate later.

# deploy certificate to chef-appliance directory
certificate_manage node['hostname'] do
  cert_file         "#{node['fqdn']}.crt"
  cert_path         '/var/opt/chef-server/nginx/ca'
  data_bag          'certificates'
  data_bag_secret   '/etc/chef/certificate_data_bag_secret'
  create_subfolders false
  notifies          :run, 'execute[chef-server-ctl reconfigure]'
end

Configure the /etc/chef-server/chef-server.rb using a template. Feel free to add additional settings, I chose to disable the webgui. Again, we notify the reconfigure resource to automatically reconfigure chef in case the configuration is changed.

#
# Generated by Chef for <%= node['fqdn'] %>.
# Local modifications will be overwritten.
#
chef_server_webui['enable']  = <%= @webgui.to_s %>
nginx['ssl_certificate']     = '<%= @ssl_certificate %>'
nginx['ssl_certificate_key'] = '<%= @ssl_certificate_key %>'

template '/etc/chef-server/chef-server.rb' do
  mode      00644
  source    'chef-server.rb.erb'
  variables ssl_certificate:     '/var/opt/chef-server/nginx/ca/chef.yourcompany.com.crt',
            ssl_certificate_key: '/var/opt/chef-server/nginx/ca/chef.yourcompany.com.key',
            webgui:              false
  notifies  :run, 'execute[chef-server-ctl reconfigure]'
end

Tell your chef-clients to verify peers #

If you use the chef-client cookbook you can tell your clients to verify the certificate by overriding the following attributes in your wrapper cookbook (support for verify_mode was added in chef-client-0.3.0). Note, that you need to specify ssl_ca_path manually, as it is not set by default.

override['chef_client']['config']['chef_server_url'] = 'https://chef.yourcompany.com'
override['chef_client']['config']['ssl_verify_mode'] = 'verify_peer'
override['chef_client']['config']['ssl_ca_path'] = '/etc/ssl/certs'

Or do it manually in your client.rb

chef_server_url 'https://chef.yourcompany.com'
ssl_verify_mode :verify_peer
ssl_ca_path     '/etc/ssl/certs'

If you are using cacert, you may need to fix the cacert root certificate (see this gist. You can use the cacert cookbook to do so automatically.

include_recipe 'cacert:cacert.org'

Be careful with the remote_file resource! #

What I also stumbled across is, that the remote_file resource doesn’t check certificates for https addresses either. So you might be very cautious if you use it for sensitive data, or to download scripts from the internet that will be executed later. I’m sure you do not want a script run by root modified on the way.

Chef deploy_revision and Capistrano git_style

Wed, 31 Jul 2013 00:00:00 +0000

One thing that was annoying me for a long time, was that, using Capistrano deployment, you cannot spawn a new vanilla virtual machine, and bring it to a fully up-and-running state with just one Chef command.

make deploy_revision compatible with Capistrano, so deployments can happen with Capistrano, until we’ve decided to fully migrate to Chef, or to stick with the push deployment

Attributes #

Let’s manage some things with attributes, so we can adjust them centrally later, in case needed.

default['rails']['app-root'] = '/var/www/example.com'
default['rails']['owner'] = 'deploy'
default['rails']['group'] = 'deploy'

# The shared directories that we're going to need later
default['rails']['shared_directories'] = %w{
  shared
  shared/config
  shared/pids
  shared/sockets
}

Workaround to maintain the git repository with a unprivileged user #

Due to a bug in git, we cannot use the “user” and “group” attributes in the deploy resource. This would result in the following error

STDERR: fatal: unable to access '/root/.config/git/config': Permission denied

Therefore, we’re chowning the app directory manually after the deploy. See CHEF-3940.

execute 'chown app root' do
  command "chown -R #{node['rails']['owner']}:#{node['rails']['group']} #{node['rails']['app-root']}"
  action :nothing
end

bundle install #

We cannot start up the application without an installed bundle. The following example mimicks the Capistrano way of “bundle install”, using chruby

execute 'bundle install' do
  user node['rails']['owner']
  cwd "#{node['rails']['app-root']}/current"

  # bundler needs LC_ALL set, to prevent "invalid byte sequence in US-ASCII" error
  # HOME also needs to be set to allow a user install
  environment 'LC_ALL' => 'en_US.UTF-8',
              'HOME'   => ::File.dirname(node['rails']['app-root'])

  command [ "/usr/local/bin/chruby-exec #{node['rails']['ruby-string']} --",
            "bundle install --gemfile #{node['rails']['app-root']}/current/Gemfile",
                           "--path #{node['rails']['app-root']}/shared/bundle",
                           "--deployment --quiet --without development test" ].join(' ')
  action :nothing
end

The deploy resource #

Now to the deploy resource. This was a little tricky, as we’re using capistrano_fanfare’s git_style deployment strategy in our deploy.rb

set :deploy_via, :git_style

This has several advantages to timestamped deploys, and actually Chef’s deploy_revision is using a similar approach. Unfortunately, there are some differences:

The directory where the repository is checked out. Capistrano (with :git_style) uses current, whereas Chef uses shared/cached-copy.
Release management. Capistrano creates empty folders in releases using timestamp-SHA directory names, Chef creates SHA directories, with the complete repository content (but without .git)
Chef symlinks currentto releases/current-SHA, and actually works with the releases/current-SHA directory during deployment, so we need to setup a temporary link to current
Chef places an empty file with SHA of the current commit as the filename in the repository_cache directory. This gives Capistrano hickups when checkout out a certain commit (ambigious statement)

I was addressing those issues in the following way:

deploy_revision node['rails']['app-root'] do
  repository 'git@github.com:chr4/rails-app.git'

  # checkout the repository to current, as capistrano :git_style would do
  repository_cache "../current"

  before_symlink do
    # create shared directories
    node['rails']['shared_directories'].each do |dir|
      directory "#{node['rails']['app-root']}/#{dir}" do
        owner node['rails']['owner']
        group node['rails']['group']
        mode  00755
      end
    end

    # remove the release directory (not a real git repo)
    directory release_path do
      recursive true
      action :delete
    end

    # create a workaround symlink
    link release_path do
      to "#{File.dirname(release_path)}/../current"
    end
  end

  # create symlinks to shared directory
  purge_before_symlink %w{log tmp/sockets tmp/pids}
  create_dirs_before_symlink %w{tmp}

  symlinks( { 'sockets' => 'tmp/sockets',
              'pids'    => 'tmp/pids',
              'log'     => 'log'} )


  # remove the workaround symlink after restarting services
  after_restart do
    link release_path do
      action :delete
    end

    # remove strange SHA file, which is hindering capistrano
    link "#{File.dirname(release_path)}/../current/#{File.basename(release_path)}" do
      action :delete
    end
  end

  # owner workaround (see comment above, CHEF-3940)
  notifies :run,     'execute[chown app root]', :immediately

  # install dependencies and start up application
  notifies :run,     'execute[bundle install]', :immediately
  notifies :restart, 'service[unicorn]'

  # only run deployment once
  # succeeding deploys will be done using capistrano (for now)
  not_if "test -e #{node['rails']['app-root']}/current"
end

The not_if statement allows us to continue using Capistrano to deploy like before, until we might decide to fully migrate to a continous deployment using Chef.

Migration from rvm to chruby on production

Wed, 10 Jul 2013 00:00:00 +0000

On our rails and worker servers at flinc, we recently migrated the ruby version management from rvm to chruby.

Besides the usual arguments against rvm, like preferring unpatched cd commands, there was another reason:

The fnichol’s chef rvm cookbook has some issues.

Issues I experienced using the rvm cookbook #

rvm::user_install #

The rvm::user_install recipe fails, because it’s trying to find rvm in a wrong path (the global one). This can be fixed by setting a symlink before installing rvm

link '/bin/rvm' do
  to '/home/vagrant/.rvm/bin/rvm'
end

include_recipe 'rvm::user_install'

rvm_ruby '1.9.3' do
  user 'vagrant'
end

file '/bin/rvm' do
  action :delete
end

rvm_gem #

The rvm_gem LWRP doesn’t chown the target directly correctly (needs to belong to the user), when using user_install). It needs to be fixed with a manual chown

rvm_gem 'bundler' do
  user 'vagrant'
  ruby_string '1.9.3@mygemset'
end

directory '/home/vagrant/.rvm' do
  owner vagrant
  group vagrant
  recursive true
end

bash defunct processes (zombies) #

Some LWRP/recipe in fnichols cookbook leave zombies behind, shown when having a look at the ps tree. Those zombies where occurring on every server where the fnichol rvm cookbook was used.

$ ps axfu
root      8887  0.1  1.1 153668 60756 ?        Sl   Jul07   8:33 chef-client
root     30498  0.5  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     30545  0.7  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     30634  0.7  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     30757  0.6  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     30897  0.8  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     31149  0.7  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     31236  0.8  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     31359  0.7  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     31499  1.0  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     31622  0.9  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>
root     31745  0.8  0.0      0     0 ?        Z    16:05   0:00  \_ [bash] <defunct>

As great as a tool as rvm is on your workstation machine, on a (production) server, you usually do not need multiple ruby versions and gemsets, but a clean and stable environment.

Migrating to chruby #

The new setup is using fnichol’s ruby-build cookook, as well as the chruby cookbook from Atlanta.

For system processes (like chef-client), I think it’s a good idea to stick to the systems ruby (therefore setting it as a default), while the application uses chruby to select its ruby version. This is how the chruby chef recipe in my wrapper looks like:

# install ruby_build
include_recipe 'ruby_build'

# build and install ruby
ruby_build_ruby node['app-rails']['ruby-string'] do
  prefix_path "/opt/rubies/#{node['app-rails']['ruby-string']}"
end


# install chruby
node.default['chruby']['default'] = 'system'
include_recipe 'chruby'


# install bundler
gem_package 'bundler' do
  gem_binary "/opt/rubies/#{node['app-rails']['ruby-string']}/bin/gem"
end

migrating rvm wrapper to chruby-exec #

For starting up services or tasks like “bundle install”, you can use chruby-exec, which works just as smooth as an rvm wrapper.

chruby-exec 1.9.3-p448 -- bundle install

whoami

Mon, 01 Jan 0001 00:00:00 +0000

$ whoami
chr4

$ finger chr4
Login: chr4                             Name: Chris Aumann
Directory: /home/chr4                   Shell: /bin/zsh
On since Thu Jan 5 05:13 1998 on pts/0 from 2001:1af8:4100:a016:2::3818
No mail.
No Plan.

Hi, I’m Chris.

I’m a freelance Devops Architect, Infrastructure Developer, SRE and Security Engineer.

Companies I work/ worked for:

Email SaaS you can configure with git developermail.io (Co-Founder, decommissioned)
On demand responsive transport and autonomous driving service ioki
Dynamic ridesharing service flinc (before they where aquired by Daimler)
Job search engine kimeta
Fintech startup vaamo (before they where aquired by Moneyfarmi)
IBM Financial Services Germany
LSE Leading Security Experts

You can find me/ contact me:

on Keybase
as @_chr4 on Twitter
on the social coding platform Github where you can also find my Chef cookbooks
on the professional network Xing
on the Opscode Chef Community where I share my chef cookbooks
via email: (sorry, no email without JavaScript, because hoping spambots are still stupid) [gpg]

Support #

I’m always glad getting little “thank-you” emails or feedback about an article!

chr4

Make keepalived play nicely with netplan/ systemd-network

DHCP #

Ubuntu upgrade #

netplan #

systemd-networkd #

1. Migrating to a static IP configuration #

2. Dummy network interface (plz don’t!) #

3. MACVLAN #

4. IPVLAN #

A new hope #

cache_warmer - tool to warm-up HTTP caches

Comparison to other tools #

Usage #

Features #

Multi-Threaded #

HTTP keep-alive #

Captcha detection #

Base URI #

Request delay #

X-Cache-Status header support #

Custom User-Agent #

Cookie support #

Compile #

sslsecure.vim - Highlight insecure SSL/TLS cipher suites and protocols as errors in your editor

Features #

Screenshots #

Web Servers #

Mail Servers #

Load Balancers #

FTP Servers #

Databases #

Programming languages #

Additional notes on runtime cipher expanding #

Feedback #

Installation #

Using sslsecure.vim and nginx.vim together #

nginx.vim (with better syntax highlighting)

Features #

Screenshots #

Snippets #

Installation #

Cross-compile and link a static binary on macOS for Linux with cargo and rust

Writing An Interpreter In Rust (Part 2)

Use built-in functions #

Improve the enum #

Custom iterator #

Thanks #

Writing An Interpreter In Rust

Porting token.go #

Porting lexer.go #

Porting the REPL #

Running the tests and REPL #

Use and automate letsencrypt certificates (ACME) in an high availability environment

ACME #

The problem in high availability setups #

Syncthing to the rescue! #

Implementation #

Install Syncthing #

Set up systemd service #

Generate Syncthing configuration and keys #

Deploy Syncthing configuration #

Restrict Syncthing to private backnet #

Get the certificates and automate renewal #

Wrap up #

pg-cert-check: A tool to monitor postgresql database SSL certificates

Why openssl is not enough #

Installation #

Build #

Usage #

Thanks #

entren - I just stumbled upon a traffic analyser I wrote in C when I was like 17

Running rabbitmq on hosts with numeric hostnames

The Bug #

The Workaround #

Homebrew betrayed us all to Google

Fixing the issue #

What Homebrew should do #

Dualstack multiple IP addresses with systemd-networkd

Increase password entropy on developermail.io