TLS termination on load balancer with re-encryption, mixed with SNI
routing
Sometimes a site (site1.example.com) needs to terminate the TLS connection
on the load balancer itself, but you also need the traffic to be encrypted
when it leaves the load balancer:
But you still have IPv6 traffic bypassing the load balancer:
And you still have another site (site2.example.com) that uses SNI routing,
with E2EE, when going over IPv4:
But you still have IPv6 traffic bypassing the load balancer:
The solution is to route all HTTPS traffic through the SNI "sorter" and then
setup another port that has TLS termination for the server name that has to
terminate at the load balancer.
The configuration will look like this.
On loadbalancer:
$ sudo vi /etc/haproxy/haproxy.cfg
In the HTTPS section, in frontend fe_https, change one line.
Then also add one new frontend and two new backends:
### HTTPS ###
frontend fe_https
mode tcp
bind *:443
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
# use_backend be_webserver1_8443 if { req_ssl_sni -i site1.example.com } # remove this line
use_backend be_loopback_site1_8008 if { req_ssl_sni -i site1.example.com } # add this line
use_backend be_webserver2_8443 if { req_ssl_sni -i site2.example.com }
default_backend be_webserver1_8443
backend be_webserver1_8443
mode tcp
server webserver1 [2001:db8:2::2]:8443 check send-proxy
backend be_webserver2_8443
mode tcp
server webserver2 [2001:db8:2::3]:8443 check send-proxy
# new backend, specific for site1
backend be_loopback_site1_8008
mode tcp
server fe_loopback_site1_8008 [::1]:8008 check send-proxy
# new (internal) frontend, specific for site1
frontend fe_loopback_site1_8008
mode tcp
bind [::1]:8008 accept-proxy ssl crt /etc/haproxy/keys/site1.example.com/haproxy.pem
default_backend be_webserver1_8443_local_ca
# new backend for webserver1, alternative setup with local CA
backend be_webserver1_8443_local_ca
mode tcp
server webserver1 [2001:db8:2::2]:8443 check send-proxy verify required sni str(site1.example.com) ca-file /etc/ssl/certs/ca-certificates.crt
It might seem counter intuitive to have an alternative backend pointing to
webserver1 on port 8443, but with the local CA instead. But it works, because
on the webserver end, which key and certificate to use, are set for each
site configuration, not per port.
Here's the flow of the request (as described by the configuration above):
- The request for site1.example.com is catched by the frontend fe_https
- fe_https then routes the request to a backend called be_loopback_site1_8008
- be_loopback_site1_8008 then routes the request to a new frontend,
fe_loopback_site1_8008, that is only reachable via a loopback IPv6 address.
- fe_loopback_site1_8008 terminates the TLS connection with a certificate/pemfile.
This pemfile has three items (simple concatenation): the private key,
the public certificate and the intermediate certificate
(aka "fullchain.pem").
- fe_loopback_site1_8008 then sends the request (now un-encrypted) to the new
backend called be_webserver1_8443_local_ca
- be_webserver1_8443_local_ca then reencrypts the request, by "calling"
webserver1, but with a local CA certificate. This works by specifying
where the intermediate certificate for the local CA, can be found.
As you can see in the configuration, when you have requests that goes from
HAProxy to HAProxy, the backend "send-proxy" is catched in the frontend with
"accept-proxy".
Also, the last line of the configuration is quite specific:
server webserver1 [2001:db8:2::2]:8443 check send-proxy verify required sni str(site1.example.com) ca-file /etc/ssl/certs/ca-certificates.crt
Let's break it down:
check checks that the backend webserver1 is available.
send-proxy sends the original IPv4 address to the webserver logs.
verify required forces verification of the certificate of webserver1.
sni str(site1.example.com) forces verification of a specific SNI.
ca-file /etc/ssl/certs/ca-certificates.crt tells which file holds the intermediate certificate for the backend local CA.
And no, HAProxy doesn't allow a long line to be broken up into a multi-line configuration.
Also, as you might've picked up by now. You more or less need to setup a
backend and a frontend for each site that TLS terminates on the load
balancer. So each site will take up a port. The good news are that you can
use IPv6 internally, so you don't need to sacrifice a lot of ports, you could
just as well let each site have it's own, local, IPv6 address, on the
loopback interface.
My suggestion is that you keep port 8008 for loopback stuff. It is an IANA
registered port for http-alt. Also it will help you not to mix it up with
8080 and 8443.
I also suggest that you allocate a ULA IPv6 range on your loopback device
(See RFC 4193).
Also, concerning keys...
There are three places where you need to have a private key (and certificate):
- On webserver1 port 443
- On HAProxy port 443
- On webserver1 port 8443
In this article I use Let's Encrypt, but almost any other SSL/TLS certificate
vendor should be fine.
What I do is that I run Let's Encrypt's certbot on webserver1. Since certbot
does the "ACME standard" challenge over port 80, which was already defined
previously, in HAProxy, to reach the correct backend webserver. Then I have
a crontab script that copies the key and certificate over to HAProxy (i.e.
scp or rsync) and reuse they key there.
I don't think this is a "beautiful" way of solving TLS termination with
re-encryption, but it is a way.
Here's my crontab script (just make sure the dir /etc/haproxy/keys exists
on loadbalancer1):
#!/bin/sh
certbot renew
rm /etc/letsencrypt/live/site1.example.com/haproxy.pem
touch /etc/letsencrypt/live/site1.example.com/haproxy.pem
chmod 0600 /etc/letsencrypt/live/site1.example.com/haproxy.pem
cat /etc/letsencrypt/live/site1.example.com/privkey.pem >> /etc/letsencrypt/live/site1.example.com/haproxy.pem
cat /etc/letsencrypt/live/site1.example.com/fullchain.pem >> /etc/letsencrypt/live/site1.example.com/haproxy.pem
rsync -e 'ssh -i ~/.ssh/my_keyfile.key' -vac /etc/letsencrypt/live/site1.example.com/haproxy.pem root@loadbalancer1:/etc/haproxy/keys/site1.example.com/
The apache configuration for port 443 (on webserver 1) is:
<VirtualHost *:443>
ServerAdmin webmaster@example.com
DocumentRoot /home/wwws/site1.example.com
ServerName site1.example.com
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCertificateFile /etc/letsencrypt/live/site1.example.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/site1.example.com/privkey.pem
SSLCACertificateFile /etc/letsencrypt/live/site1.example.com/chain.pem
ErrorLog /var/log/apache2/site1.example.com-error.log
CustomLog /var/log/apache2/site1.example.com-access.log combined
<Directory /home/wwws/site1.example.com>
Require all granted
</Directory>
</VirtualHost>
As you can see, this configuration hasn't changed. However, the apache
configuration on port 8443 is different:
<VirtualHost *:8443>
ServerAdmin webmaster@example.com
DocumentRoot /home/wwws/site1.example.com
ServerName site1.example.com
# HAProxy with send-proxy to retrieve original IP-address, Apache 2.4.31
# sudo a2enmod remoteip
RemoteIPProxyProtocol on
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCertificateFile /etc/apache2/ssl/site1.example.com/site1.example.com.crt
SSLCertificateKeyFile /etc/apache2/ssl/site1.example.com/site1.example.com.key
SSLCACertificateFile /etc/apache2/ssl/site1.example.com/myCA.crt
ErrorLog /var/log/apache2/site1.example.com-error.log
CustomLog /var/log/apache2/site1.example.com-access.log combined
<Directory /home/wwws/site1.example.com>
Require all granted
</Directory>
</VirtualHost>
The private key and certificate come from a local CA. This article will
not go into detail on how to create a CA. If you want to know more, you
can read my article on how to
create your own CA.
You also need to install the CA certificate on loadbalancer1.
First, put your myCA.crt in /usr/local/share/ca-certificates/ on
loadbalancer1. Then run this command:
$ sudo update-ca-certificates
You're finished!
But...
You probably want to test this...
You might even need to debug this... A lot...
Read the next section!
