I’ve written a small tool to warm-up HTTP caches, e.g. services like nginx.
Comparison to other tools
While most tools in this area are designed to apply a certain load to a web server (often for a specified time), cache_warmer is designed to explicitly
GET a set of URLs from a file to warm-up a cache.
Warm-up URLs from a file using a mobile User-Agent
$ cache_warmer --mobile urls.txt Spawning 4 threads to warm cache with 10000 URIs 10000 / 10000 [===================================================================] 100.00 % 133.99/s Processed 10000 URLs X-Cache-Status header statistics: Miss: 8991 Hit: 1009 HTTP Status Code statistics: Ok: 9850 NotFound: 150 Total time taken: 450.002s
cache_warmer is multi-threaded. Threads can be specified using the
It uses keep-alive by default, which can be disabled with
It supports a
--captcha-string option, which scans the response body for certain strings to detect (and abort) when running into e.g. captchas.
If your URL file only contains the base URL (like
/products/spoons), you can add a
--base-uri to prepend the host and scheme, e.g.
Outgoing requests can be toned down with the
X-Cache-Status header support
If your backend sets the
X-Cache-Status header, you’ll get nice statistics about your cache hit rates at the end of the run.
When using nginx, such a header can be added with this directive:
add_header X-Cache-Status $upstream_cache_status;
cache_warmer defaults to a Googlebot-like User-Agent. You can use the corresponding mobile User-Agent when specifying the
In case you need a custom User-Agent, you can set it with
Attach arbitrary cookies with the
--cookie key=value option.
# Compile locally cargo build --release # Cross compile to Linux using Docker docker-compose up