I’ve written a small tool to warm-up HTTP caches, e.g. services like nginx.

Source code as well as compiled releases are available at Github.

Comparison to other tools

While most tools in this area are designed to apply a certain load to a web server (often for a specified time), cache_warmer is designed to explicitly GET a set of URLs from a file to warm-up a cache.

Usage

Warm-up URLs from a file using a mobile User-Agent

$ cache_warmer --mobile urls.txt

Spawning 4 threads to warm cache with 10000 URIs
10000 / 10000 [===================================================================] 100.00 % 133.99/s
Processed 10000 URLs

X-Cache-Status header statistics:
        Miss: 8991
        Hit: 1009

HTTP Status Code statistics:
        Ok: 9850
        NotFound: 150

Total time taken: 450.002s

Features

Multi-Threaded

cache_warmer is multi-threaded. Threads can be specified using the --threads option.

HTTP keep-alive

It uses keep-alive by default, which can be disabled with --no-keep-alive.

Captcha detection

It supports a --captcha-string option, which scans the response body for certain strings to detect (and abort) when running into e.g. captchas.

Base URI

If your URL file only contains the base URL (like /products/spoons), you can add a --base-uri to prepend the host and scheme, e.g. --base-uri https://example.com

Request delay

Outgoing requests can be toned down with the --delay flag.

X-Cache-Status header support

If your backend sets the X-Cache-Status header, you’ll get nice statistics about your cache hit rates at the end of the run.

When using nginx, such a header can be added with this directive:

add_header X-Cache-Status $upstream_cache_status;

Custom User-Agent

cache_warmer defaults to a Googlebot-like User-Agent. You can use the corresponding mobile User-Agent when specifying the --mobile flag.

In case you need a custom User-Agent, you can set it with --user-agent 'Your-User-Agent'.

Attach arbitrary cookies with the --cookie key=value option.

Compile 

# Compile locally
cargo build --release

# Cross compile to Linux using Docker
docker-compose up