Reverse Proxy Multiple Domains Using Caddy 2

During lockdown, I’ve spent a bit of time improving our home network. The bigger picture of which I’ll write about in a future post. But for now, I came across some challenges with running Caddy 2 as a reverse proxy for multiple domains used internally.

If you’ve stumbled across this looking for the end config file for Caddy, then you can skip there.

Background

A few months back I kitted out my home with some Ubiquiti UniFi gear to fix our crappy Wifi at home, following inspiration from Troy Hunt and Scott Helme.

In order to administrate UniFi devices, you’ll need the UniFi Cloud Key which runs the Controller software to do just that. Although if you have a spare Raspberry Pi lying around, you can download the software for free and run it on there - this is what I did.

I’ve also wanted to protect my home network with a self-hosted DNS server, such as PiHole. I won’t go into depth about how that was done, but you can follow Scott Helme’s guide on how you can set the same up.

Both of these services can be accessed through web browsers at the IP address and ports where they are being hosted, such as http://192.168.1.10:8093/admin/ in the case of PiHole. Having to remember the IP address and the port can be a pain. We can front these services with a rememberable domain name which points to these services - of which I’ve written about in a previous post.

Securing with HTTPS

The web is evolving, and there is no reason why we should access services via insecure HTTP, that includes services that are only running on an internal network such as a home network. Web browsers nowadays give you a warning when you are connecting to website over an unencrypted connection.

Caddy is a web server similar to Apache, nginx, et al., but it is different in that it enables HTTPS by default and upgrades requests from HTTP to HTTPS. Managing certificates for HTTPS is a pain - so Caddy does that too, so long as you can prove you own the domain you are hosting requests at. We can use Caddy in a reverse proxy mode, allowing us to access services at endpoints such as https://pihole.domain.local in our browsers and forward them to the corresponding IP address hosting the service.

A reverse proxy is a service that simply forwards client requests onto the server on the clients behalf.

Proving Domain Ownership

Caddy uses Let’s Encrypt (LE) to provide certificates for domains. Since domains can be exposed publicly, we will have to prove ownership of the domain to have LE issue certificates on our behalf - so we’ll have to purchase the domain from a registrar. I talked about how to do this for this website in the past.

LE supports several challenge methods in order to prove you own the domain. This helps mitigates attacks by adversaries by claiming they own a domain such as natwest.co.uk - allowing them to create phishing attacks and steal banking information.

Since my network is only visible internally for the moment (i.e. the domain will only resolve to an IP address on my network) - I cannot use HTTP or TLS since these require the domain to resolve to a public IP address to a web server hosting a challenge file requested by LE. Therefore the only option I have is DNS challenge, where a randomly string generated by LE is placed into the TXT record of a DNS record to confirm ownership.

Building Our Caddy

For this exercise I’ll be using the latest version, Caddy 2, which allows for plugins to be built into the binary depending on your use case - including DNS challenge. This plugin isn’t included by default, so we’ll need to build our own Caddy binary. The tool to do this is called xcaddy.

UPDATE 2020-09-30

Looks like Caddy now comes with a nice web interface for downloading a Caddy binary with whatever plugins you desire. I just tested out the Linux arm 7 platform with just the github.com/caddy-dns/cloudflare plugin, and it was able to run my Caddy configuration below perfectly!

Once you’ve got the binary downloaded, copy it to the Pi then skip to Caddy Configuration.

To build using xcaddy, you need to make sure you have Go installed on your machine.

Note that I am building Caddy on my laptop, but running it on a Pi, so I will have to specify the architecture that Pi is running on so that Go can correctly build it.

# Download xcaddy
go get -u github.com/caddyserver/xcaddy/cmd/xcaddy

# Build custom Caddy binary for Raspberry Pi
GOOS=linux GOARCH=arm GOARM=7 xcaddy build --with github.com/caddy-dns/cloudflare

# Copy the new binary across to the Pi
scp caddy pi:/home/pi/caddy/

Caddy Configuration

The configuration I’m using can be seen below. Some things to note:

• I’m using Cloudflare as the DNS name servers for the domain, even though I purchased my domain from namecheap
  • This repeats an exercise I’ve done previously
  • I’ve done this for two reasons:
    • Caddy at the time of writing does not have a namecheap DNS challenge plugin
    • It’s a proven method I know already
• A CLOUDFLARE_API_TOKEN is required to have Caddy set the TXT record DNS challenge received from LE
  • Guide for the same
• Caddy is reverse proxying traffic to services running locally on the Pi
• Caddy is not verifying the certificate being hosted by the UniFi Controller (insecure_skip_verify = true)
  • The controller self-signs a certificate, and the reverse proxy has no means of establishing a chain of trust to verify the certificate
  • It’s not a best practice to not verify the chain of trust, however I’m happy to accept the risk for now

Click here to see documentation on Caddy JSON config files.

Updating DNS Records

Remember that the domain names aren’t actually publicly accessible. At a basic level we can update the /etc/hosts file of the machine we’re running on to add a record telling our machine how to resolve the domain.

sudo sh -c "echo \"192.168.1.10 pihole.joannet.casa\n192.168.1.10 unifi.joannet.casa\" >> /etc/hosts"

However, we’re already using PiHole as our own DNS server right? We can add the records there instead.

The IP addresses you see above are pointing to the host running Caddy, the Raspberry Pi.

Verifying Caddy

Once the config file is built, you can perform a test run to confirm everything is working by executing this command.

sudo ./caddy run --config config.json

We need to execute using sudo so that we can expose the service to restricted ports 80 and 443 (HTTP and HTTPS respectively).

Now we have a memorable domain name fronting the service, and Firefox is happy that we’re encrypting the connection too. The certificate being produced in seen below.

Enabling Caddy Service

Since we’re not using the standard Caddy installation method, we will need to specify a service unit file so that Caddy starts up at the same time as the host - which is what PiHole and UniFi are doing currently.

First check to see if there is a stale service there already.

$ ls -la /etc/systemd/system/caddy.service
lrwxrwxrwx 1 root root 9 Jun  4 09:14 /etc/systemd/system/caddy.service -> /dev/null

If you get the above then remove the symlink so that we can create a file there.

rm /etc/systemd/system/caddy.service

Then populate the same file with the below, remembering the change the location of the Caddy config file to where it exists on your machine.

[Unit]
Description=Caddy Reverse Proxy
Wants=network-online.target
After=network.target network-online.target
 
[Service]
ExecStart=/usr/local/bin/caddy run --config /home/jdheyburn/homelab/caddy/config.json
Restart=on-abort
 
[Install]
WantedBy=multi-user.target

Finalise the new service with the two commands, enabling it on host startup and starting the service right now.

sudo systemctl enable caddy.service
sudo systemctl start caddy.service

Conclusion

For now I have all the above running bare-metal on one Pi instance, which produces a huge single point of failure in my network. In the future I’d like to see how converting these to Docker containers and having them distributed on multiple Pis would increase the resiliency of these services.

Until then, these basic but essential services are being hosted at easy to remember domains, transported over an encrypted connection, for me to easily administer the network for when it gets more complex over time.