Hosting setup
Hosting setup of this blog
This site is generated by Zola and stored in Fastly Object Storage (an S3-compatible object store) and a Fastly Compute service serves it. The Object Storage credentials are stored in a Secret store and other configuration lives in a Config store.
The Compute service uses the Fastly Rust SDK for implementing the service. The SDK contains APIs for reading configuration and secrets, fetching content from Object Storage, manipulating headers, 404 handling, and more.
Goals
For the Compute service I have several goals:
- Runnable on multiple domains without source modification
- Content has surrogate keys attached for purging on updates
- Content fetched from Object Storage is cached
- Include HTTP preloading headers
The first two goals allow easy updates to the site service and content. I can test the service on a testing domain to avoid breaking the blog domain. When I want to publish new content I can clear old cached pages from all POPs.
The last two goals keep the content fast. Fetching content from Object Storage will be slower than serving it from the local cache. HTTP preloading allows faster fetches of custom fonts and styles.
Service basics
I used the Compute starter kit for static content as the base for my service.
This provides a basic service that can fetch from an S3-compatible backend like
Object Storage. It accepts only GET requests, generats CORS responses,
and responds to /robots.txt. It doesn't meet all my goals above.
Object Storage
I followed the Object Storage guide to create a blog.segment7.net bucket
and configure aws s3 to use it from my laptop. I then used zola build to
generate content and aws s3 sync --delete to update the bucket with the
rendered blog.
I then updated the starter kit configuration and service to use my object storage bucket as a backend and verified it worked.
Config and Secret stores
In order to easily test changes to the service I want to deploy independent
versions of the service to separate domains, but I don't want to have to change
the service config for every deployment. The starter kit embeds configuration
in config.rs so I would need a version with different config
when testing. I would eventually make an error when switching the config back
and forth in the service, breaking my blog.
The Config and Secret stores allow setting and editing configuration independent of the rust code, and can be updated independently of the service.
I created a Config store blog_segment7_net_config and added bucket_name,
&c. entries for bucket access. Then I created a Secret store
blog_segment7_net_secret and added entries for a read-only Object Storage
access_key_id and secret_access_key.
Next I edited the service to use the TLS client servername to determine the
stores to load from. Converting the TLS client servername
(blog.segment7.net) to a store name (blog_segment7_net_config) will allow
creating a second service that uses the same service implementation but
different configuration store names.
let store_prefix = req
.get_tls_client_servername()
.map(|name| name.replace('.', "_"))
.ok_or_else(|| Error::msg("missing request tls client servername"))?;
let config = ConfigStore::open(&format!("{store_prefix}_config"));
let bucket_name = config
.get("bucket_name")
.ok_or_else(|| Error::msg("missing bucket_name in config store"))?;
// … rest of bucket_ config entries
let auth = SecretStore::open(store_name)
.map_err(|_| Error::msg("secret store missing"))?;
let access_key_id = auth
.get("access_key_id")
.ok_or_else(|| Error::msg("missing access_key_id in secret store"))?;
// let secret_access_key = …
I then set up a second test service, bucket, Config, and Secret stores with equivalent names and entries. The service code will automatically use the config test stores and load content from the test bucket.
HTTP Preloading
HTTP preloading allows the browser to fetch CSS and fonts before parsing the HTML and CSS. The starter kit already has a section for appending headers to HTML content so I can add my extra headers there.
Zola includes a cache-busting query string parameter on the CSS so I chose to
store the preload information in the config store preloads entry as one
preload per line, with the kind and URL space-separated:
style https://blog.segment7.net/main.css?h=6114f82d54f9eea8b588
font https://blog.segment7.net/fonts/AtkinsonHyperlegibleNext-Light.woff2
font https://blog.segment7.net/fonts/AtkinsonHyperlegibleNext-Regular.woff2
font https://blog.segment7.net/fonts/AtkinsonHyperlegibleNext-SemiBold.woff2
font https://blog.segment7.net/fonts/IosevkaTerm-Extended.woff2
I regenerate the preloads when I rebuild and upload the blog to make sure the
main.css query string is up-to-date. (Due to purging I don't need the query
string but I don't know if I can disable it.)
I set the preloads only for HTML responses:
if let Some(header) = beresp.get_header(header::CONTENT_TYPE)
&& header
.to_str()
.map(|value| value.starts_with("text/html"))
.unwrap_or(false)
{
// other headers …
if let Some(preloads) = config.get("preloads") {
for preload in preloads.split('\n') {
let parts: Vec<&str> = preload.splitn(2, " ").collect();
let url = parts[1];
let kind = parts[0];
let value = if kind == "font" {
format!("<{url}>; rel=preload; as={kind}; crossorigin")
} else {
format!("<{url}>; rel=preload; as={kind}")
};
beresp.append_header(header::LINK, value);
}
}
}Purging
The starter kit already sets a TTL for various content types. In order to allow the longest possible TTL values, but still allow updates in case of new content, typos, or errors I added surrogate keys to backend requests. If the object is not cached it will be fetched from the backend and the set TTL and surrogate keys will be set in the cache when fetched. The surrogate key can later be used to purge the object.
For objects that exist in the store I set only the blog.segment7.net
surrogate key. For objects that don't exist in the store where I will return a
404 response I set the blog.segment7.net and 404 surrogate keys. This
allows purging either the entire site (purge blog.segment7.net) or only 404s
(purge 404).
const SURROGATE_HIT: HeaderValue = HeaderValue::from_static("blog.segment7.net");
const SURROGATE_404: HeaderValue = HeaderValue::from_static("blog.segment7.net 404");
#[fastly::main]
fn main(mut req: Request) -> Result<Response, Error> {
// …
// Set the cache TTL for the expected result of the request.
let ttl = get_cache_ttl(bereq.get_path());
bereq.set_ttl(ttl);
bereq.set_stale_while_revalidate(600);
bereq.set_surrogate_key(SURROGATE_HIT);
// Send the request to the backend
let mut beresp = bereq.send(config::BACKEND_NAME)?;
if is_not_found(&beresp) && !original_path.ends_with("index.html") {
// Lookup /path/index.html if /path was not found in object storage…
}
// If backend response is still 404, serve the 404.html file from the bucket.
if is_not_found(&beresp) {
// Copy the original request and replace the path with /404.html.
bereq = req.clone_without_body();
bereq.set_path(&format!("{}/404.html", bucket_name));
bereq.set_surrogate_key(SURROGATE_404);
beresp = bereq.send(config::BACKEND_NAME)?;
beresp.set_status(StatusCode::NOT_FOUND);
}
// …
}
Now after I aws s3 sync … I can fastly purge -s $service_id -k blog.segment7.net and have all pages update.