Berkeley smorgasbord — part II

OpenBSD makes life pretty easy when it comes to securing your web-app. In this short post, we will go through the steps to:

1. secure traffic to ‘foobar.com’ (our web-app) and ‘cdn.foobar.com’ (our assets host)
2. still just have plaintext traffic to ‘barbaz.com’
3. make sure our TLS certificate gets updated automatically via ‘cron’

We will be building upon our previous post for those who came to this all just now. It’s assumed that the state-of-the-server is in place from this previous post, meaning, our ‘foobar’ web-app is all humming away via ‘relayd,’ ‘httpd,’ Erlang/OTP, etc.

It is also assumed that ‘letsencrypt.org’ CAA DNS records are in place, that port 443 is open on your box, and that doas is being employed for root-access.

Order of operations

It’s important to do several of these steps in a particular order. Otherwise, some chicken/egg troubles will stop you in your tracks. First, as our normal user, ‘joe,’ let’s back up our ‘relayd.conf’ and ‘httpd.conf’ files:

cp /etc/relayd.conf ~/relayd.conf.bak
cp /etc/httpd.conf ~/httpd.conf.bak

Next, we need to improve upon ‘httpd.conf’ and get it in its final state that can handle ‘https,’ answer acme-client challenge questions over plaintext, etc.

Here’s the improved file, that we’ll also just keep in ~ for simplicity:

‘httpd.conf.master’

int_ip="127.0.0.1"

server "foobar.com" {
  alias "www.foobar.com"
  alias "cdn.foobar.com"

  listen on $int_ip port 8080
  # make /var/www/acme available
  location "/.well-known/acme-challenge/*" {

    # /var/www/acme
    root "/acme"

    # remove .well-known and acme-challenge from the path
    # before looking for the file in /var/www/acme
    request strip 2
  }
  # redirect all other non-acme-client requests
  location "/*" {
    block return 301 "https://$SERVER_NAME$REQUEST_URI"
  }
}
server "www.foobar.com" {
  listen on $int_ip port 8443
  block return 301 "https://foobar.com$REQUEST_URI"
}
server "cdn.foobar.com" {
  listen on $int_ip port 8001
  location "/assets/*" {
    root "/htdocs/foobar/priv/assets"
    request strip 1
  }
}

server "barbaz.com" {
  listen on $int_ip port 8080
  location "/robots.txt" { root "/htdocs/local" }
  root "/htdocs/barbaz.com"
}
server "www.barbaz.com" {
  listen on $int_ip port 8080
  block return 301 "http://barbaz.com$REQUEST_URI"
}

This file is ready to be used in conjunction with acme-client — and we don’t have to change anything in ‘relayd.conf’ for this to all work.

But before we copy over our new ‘httpd’ configuration, it’s time to do a little prep-work before using the acme-client tool. First, make sure the apt directories are in place:

doas mkdir -p -m 700 /etc/ssl/private # may be there
doas mkdir -p -m 755 /var/www/acme    # may be there

Let’s now configure ‘/etc/acme-client.conf’ to our liking:

‘acme-client.conf’

authority letsencrypt {
  api url "https://acme-v01.api.letsencrypt.org/directory"
  account key "/etc/acme/letsencrypt-privkey.pem"
}

authority letsencrypt-staging {
  api url "https://acme-staging.api.letsencrypt.org/directory"
  account key "/etc/acme/letsencrypt-staging-privkey.pem"
}

domain foobar.com {
  alternative names { www.foobar.com cdn.foobar.com }
  domain key "/etc/ssl/private/foobar.com.key"
  domain certificate "/etc/ssl/foobar.com.crt"
  domain full chain certificate "/etc/ssl/foobar.com.fullchain.pem"
  sign with letsencrypt
}

Now although ‘relayd’ isn’t ready to serve our web-app over TLS, that doesn’t mean we can’t get our certificates with acme-client. We can, in fact.

For a brief moment, we can expose our plaintext domains and resolve each ‘well-known/acme-challenge’ which puts our certificates in place. Then, we just put our old ‘httpd.conf’ back in place while we rework the rest of our OpenBSD server configurations.

Follow along with some care:

doas cp ~/httpd.conf.master /etc/httpd.conf
doas su
root> httpd -n
root> rcctl reload httpd
root> acme-client -vFAD foobar.com
root> exit
doas cp ~/httpd.conf.bak /etc/httpd.conf
doas httpd -n
doas rcctl reload httpd

So, what did we just do? We briefly took our web-app out of service; got our security certificates (see ‘/etc/ssl’); then brought back our web-app and ‘httpd’ service as it once was.

Hacking for service

There’s a bit of hacking needed to get ‘relayd’ to work with TLS and our web-app. But there’s nothing to fear. The complications aren’t too egregious.

First, let’s create a work-in-progress ‘relayd.conf’ in our ~ directory, like we did with ‘httpd.conf.master.’ Here’s the new file:

‘relayd.conf.master’

relayd_addr="0.0.0.0"
plaintext_port="80"

app_port="8000"
table <app_hosts> { 127.0.0.1 }

cdn_port="8001"
table <cdn_hosts> { 127.0.0.1 }

web_port="8080"
table <web_hosts> { 127.0.0.1 }

tls_web_port="8443"
table <tls_web_hosts> { 127.0.0.1 }

log state changes
log host checks
log connection errors

http protocol "httpfilter" {
  match header append "X-Forwarded-For" \
    value "$REMOTE_ADDR"
  match header append "X-Forwarded-By" \
    value "$REMOTE_ADDR:$SERVER_PORT"
  match header set "Keep-Alive" value "$TIMEOUT"

  ### TCP performance options
  tcp { nodelay, sack, socket buffer 65536, backlog 100 }

  ### Return HTTP/HTML error pages
  return error
}

http protocol "httpsfilter" {
  match header append "X-Forwarded-For" \
    value "$REMOTE_ADDR"
  match header append "X-Forwarded-By" \
    value "$REMOTE_ADDR:$SERVER_PORT"
  match header set "Keep-Alive" value "$TIMEOUT"

  match response header remove "Server"
  match response header set "X-Frame-Options" \ 
    value "SAMEORIGIN"
  match response header set "X-XSS-Protection" \
    value "1; mode=block"
  match response header set "X-Content-Type-Options" \ 
    value "nosniff"
  match response header set "Referrer-Policy" \ 
    value "strict-origin"
  match response header set "Content-Security-Policy" \
    value "style-src 'self' *.foobar.com \
    https://fonts.googleapis.com; \
    font-src 'self' data: *.foobar.com \
    https://fonts.gstatic.com; \
    default-src 'self' *.foobar.com"
  match response header set "Feature-Policy" \
    value "accelerometer 'none'; camera 'none'; \
    geolocation 'none'; gyroscope 'none'; magnetometer \
    'none'; microphone 'none'; payment 'none'; usb 'none'"
  match response header set "Strict-Transport-Security" \
    value "max-age=31536000"

  ### TCP performance options
  tcp { nodelay, sack, socket buffer 65536, backlog 100 }

  ### Return HTTP/HTML error pages
  return error

  match request quick header "Host" value "cdn.foobar.com" \
    forward to <cdn_hosts>
  match request quick header "Host" value "foobar.com" \
    forward to <app_hosts>
}

relay "proxy" {
  listen on $relayd_addr port $plaintext_port
  protocol "httpfilter"

  # main server table
  forward to <web_hosts> port $web_port
}

relay "tlsproxy" {
  listen on $relayd_addr port 443 tls
  protocol "httpsfilter" # N.B. http/s/filter

  # N.B. the order as such

  forward to <tls_web_hosts> port $tls_web_port

  # additional server tables used by custom rules
  forward to <cdn_hosts> port $cdn_port check tcp interval 60
  forward to <app_hosts> port $app_port
}

There’s quite a few changes in there since the last time we put a plaintext ‘relayd.conf’ together. But in a nutshell, we are just expanding our rules to include TLS and moving the web-app from the plaintext protocol to the secure one.

N.B. in the ‘httpsfilter’ protocol, you may want to add the following match-rule if you wanted all your subdomains to be forced (in the client) to be over TLS:

match response header set "Strict-Transport-Security" \
  value "max-age=31536000; includeSubDomains"

In our instance, we actually didn’t want it this way, but our case may be exceptional.

More nuances

relayd is limited in that it expects your certificates to be in a specific location. As well, it expects them to be named after the IP addresses upon which it is listening. This is the state of affairs as of OpenBSD 6.5. This may or may not change in the future.

But we can get around this, using soft-links. Follow along with some care:

cd /etc/ssl
doas ln -s foobar.com.pem 0.0.0.0.pem
doas ln -s foobar.com.fullchain.pem 0.0.0.0.crt # N.B. not a typo
doas su
root> cd /etc/ssl/private
root> doas ln -s foobar.com.key 0.0.0.0.key
root> exit

With that, we are ready to put our new ‘relayd.conf’ into the front lines and start serving our web-app over TLS. We can do that as follows:

doas cp ~/relayd.conf.master /etc/relayd.conf
doas cp ~/httpd.conf.master /etc/httpd.conf
doas su
root> relayd -n
root> httpd -n
root> rcctl reload httpd
root> rcctl reload relayd
root> exit

And there you have it: ‘https.’ You can employ curl to make sure all your domains are redirecting accordingly. For example:

...
curl -I -XGET https://foobar.com
curl -I http://foobar.com
...

You should see some 200s and 301s. But that’s just child’s play; you can do some deep dives with the following resources:

1. https://securityheaders.com
2. https://www.ssllabs.com/ssltest

These can help you troubleshoot and tighten things up where you see fit.

Steps for securing the future

A nice way to have your certificates updated for you, is just let ‘cron’ take that on. Here’s a line one should add to root’s crontab -e:

15  3  *  *  *  acme-client foobar.com && rcctl reload relayd

With that, your certificates are scheduled for updates. Easy.

One note of importance, if you’re adding a subdomain, or changing anything at all, it’s necessary to update certificates deterministically with the correct flags.

For example, say you wanted to add ‘www1.foobar.com’ to your project?

Here’s how it’s done.

Make the change to your ‘acme-client.conf’ in the ‘alternative names’ block. Then add the new alias to your ‘httpd.conf,’ not forgetting to execute doas rcctl reload httpd so plaintext traffic can reach your ‘amce’ challenge reply at ‘www1.’

After all this has been done, you’ll want to perform the following:

doas mv /etc/ssl/foobar.com.crt /etc/ssl/foobar.com.crt.bak
doas acme-client -vFAD foobar.com

If you don’t do this, errors will come looking for you. If it all went well, you are in the clear to remove the ‘foobar.com.crt.bak’ file.