Using pfSense and HAproxy as an SSL offloading Reverse proxy

I wanted to offload ssl from my eucalyptus cluster, and homogenize the URI of the disparate eucalyptus endpoints so I could easily move parts of the cluster around to different hardware in my lab. I thought it would be pretty straightforward to accomplish using pfSense, but I ran into a few snags that made this more challenging than I had hoped. I’m writing this in the hopes that it saves someone else some time.

This post is influenced heavily by @Loredo‘s Geeking out with HAproxy and pfSense

Pre Requisites

I assume you’ve already got a working knowledge of pfSense, and a functional pfSense router to perform the offloading in the first place.

You’ll need at least one IP that is bound to your firewall that tcp:80/443 are not being used for some other service, as HAProxy will need to bind to them. Setting this up is outside the scope of this post, but it’s worth mentioning as a requirement

I also assume you also have your ssl certificate installed on your pfSense router

Assumptions

  1. I will assume that your public endpoint is called eucalyptus.example.com.
  2. I will assume that your public endpoint ipv4 address is 123.234.123.234.
  3. I will assume that you have one host which all the services live on, and its’ private address is 10.10.10.18.

Anyplace where you see these assumptions, you will want to change them to values relevant to your setup.

Install HAProxy

This is pretty straight forward. System -> Packages -> HAProxy 1.5 (not using 1.6dev but not for any real reason)

Basic Proxy Settings

set the internal stats port to 2200

In the general settings tab, you want to add this to the Global Advanced Pass Through field:

# Modern browser compatibility only as mentioned here:
# https://wiki.mozilla.org/Security/Server_Side_TLS
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
tune.ssl.default-dh-param 2048# Time-to-first-Byte (TTFB) value needs to be optimized based on
# the actual public certificate chain see
# https://www.igvita.com/2013/10/24
# /optimizing-tls-record-size-and-buffering-latency/
tune.ssl.maxrecord 1370

Create Backends – Round 1

Dummy backend

Head over to the backend section of the HAproxy configuration gui, and create a new instance called “none”. We’ll use this backend later as our default destination. It’s actually doing nothing but being a placeholder for everything that doesn’t match elsewhere.

Use the following settings:

Mode Name Forwardto Address Port SSL
disabled dummy_backend Address+Port 127.0.0.1 88881 no

Health check method: none

Click on “save”.

: 8888 was picked at random. Any port which does not listen on localhost will work just fine

SSL redirect backend

Next we’ll create a redirect backend to shovel tcp:80 traffic to our (not yet created) SSL frontend. Here are the options in the pfSense HAproxy gui to change

Mode Name Forwardto Address Port SSL
inactive ssl-redirect Address+Port 127.0.0.1 8081 no

Backend pass through: redirect scheme https code 301
Health check method: none

Click on “save”.

Create Frontends – Round 1

SSL Redirect frontend

Now we need to tell HAproxy to send the tcp:80 traffic to our redirect backend.

Name Eucalyptus_ssl_redirect
Description Redirect HTTP traffic to HTTPS
Status Active
External Address 123.234.123.234
Port 80
SSL Offloading not checked
Backend server pool ssl-redirect
Type HTTP/HTTPS(offloading)

Click on “save”.

SSL Frontend

Now lets create out main SSL Frontend. This will be the frontend that all the other services are a slave to.1

We’re going to use our dummy backend here, as we’ve not yet created any of the per-service backends

Name Eucalyptus_HTTPS_Frontend
Description Eucalyptus HTTPS Front door
Status Active
External Address 123.234.123.234
Port 443
SSL Offloading checked
Backend server pool dummy_backend
Type HTTP/HTTPS(offloading)

1) Despite the fact that there are several additional sub-backends, if you do not assign an ACTIVE  backend server pool to the main frontend interface, nothing will work.

HTTPS shared frontend template

We’re going to make a few of these shared frontend instances; One for each of the eucalyptus backend services. Lets make a cookie cutter one to use as a template

Switch to the frontend section and create a new instance using the following settings:
Name Eucalyptus_HTTPS_dummy
Description Eucalyptus Dummy HTTPS Endpoint
Status Active

Check the Shared Frontend checkbox, and set the Primary frontend to be the mantle_HTTPS_Frontend ‘front door’ endpoint.

Set the Backend server pool to be the dummy_backend backend we created a few steps ago.

Advanced settings

Client Timeout 7200000
Use ‘forwardfor’ option yes
Advanced pass through:
# Remove headers that expose security-sensitive information.
rspidel ^Server:.*$
rspidel ^X-Powered-By:.*$
rspidel ^X-AspNet-Version:.*$

SSL Offloading

Certificate: Ideally choose a wildcard cert you uploaded to the Cert Manager before, e.g. from StartCom/StartSSL. We are choosing “*.example.com” here. Put additional certificates as required.
Add ACL for certificate CommonName: No
Advanced SSL options: no-sslv3

Click on “save”.

Create Backends – Round 2

Eucalyptus APIs Backend Skeleton Template

Now lets switch over and create the backend server pool template. We have to set this up before we can create our per-service front and back ends.

  •  Navigate to the backend section of the HAproxy configuration gui
  • Create a new instance called euc_APIs_backend
  • Create backend servers for each endpoint found in your eucarc.
    • Create the first backend server (ec2)
    • Settings:
Mode Name Forwardto Address Port SSL
active ec2_server Address+Port 10.10.10.18 8773 no
Health check method http
HTTP check method GET
Http check uri /services/Heartbeat
  • Duplicate the ec2_server server for each of the disparate API Endpoints
    • s3
    • aws_iam
    • euare
    • token
    • aws_auto_scaling
    • aws_cloudformation
    • aws_cloudwatch
    • aws_elb
    • aws_simpleworkflow
  • Enable the statistics
    • Set the Stats URI to /euc_haproxy?stats
  • Click on “Save”.

Create the individual API Backends from the API Backend Skeleton

Now that we have a template to work from, create the specific endpoint backends we will connect the frontends to.

  • Clone the backend template for each of the following services
    • ec2
    • s3
    • aws_iam
    • euare
    • token
    • aws_auto_scaling
    • aws_cloudformation
    • aws_cloudwatch
    • aws_elb
    • aws_simpleworkflow
  •  For each endpoint backend, perform the following steps
    • Update the Name of the backend to reflect the specific api instead of euc_APIs_backend.
    • Remove the servers which are not relevant to the endpoint in question.
  • Click on “Save”.

You should now have 10 eucalyptus service backends.

You can remove the euc_APIs_backend and the dummy backends. Next we’ll create the eucaconsole backend.

EucaConsole Backend

Now we’re going to create the EucaConsole Backend. It’s just a touch different

  • Navigate to the backend section of the HAproxy configuration gui,
  • Create a new instance called eucaconsole_Bknd
  • Create a backend server for the server where your eucaconsole entity runs with the following settings:
Mode Name Forwardto Address Port SSL
active eucaconsole_server Address+Port 10.10.10.18 8888 no
Health check method http
HTTP check method OPTIONS
Http check uri
  • Enable statistics on the backend
    • Set the Stats URI to /euc_haproxy?stats

Create Frontends – Round 2

Per-Service Frontend endpoints

Now lets go create the per-service frontend listeners. We’re going to end up with several shared listeners attached to the main Eucalyptus_HTTPS_Frontend; one for each endpoint.

This will be where the SSL offloading and routing is handled.

  • Navigate to the frontend tab
  • Duplicate the Mantle_HTTPS frontend
  • This will bring you to a window where you can edit the cloned frontend. Here’s what you need to change there
Edit each new frontend
  • Change the NameDescription and check the Shared Frontend checkbox.
    Name Eucalyptus_HTTPS_ec2
    Description Eucalyptus EC2 Public HTTPS Endpoint
    Status Active
    Shared Frontend
  • Set the Primary Frontend to be the Mantle_HTTPS_Frontend we created earlier
  • Set the Backend Server Pool to the endpoint relevant to the frontend in question. In this case we selected the ec2_backend.
  • Create an ACL to match traffic matching the relevant URI for the endpoint in question. Refer to the table below for Endpoints and their URIs
  • Click on “Save”.

 

 Name / Endpoint Expression URI / Value
EC2_URL Path starts with: /services/compute
S3_URL Path starts with: /services/objectstorage
AWS_IAM_URL Path starts with: /services/Euare
EUARE_URL Path starts with: /services/Euare
TOKEN_URL Path starts with: /services/Tokens
AWS_AUTO_SCALING_URL Path starts with: /services/AutoScaling
AWS_CLOUDFORMATION_URL Path starts with: /services/CloudFormation
AWS_CLOUDWATCH_URL Path starts with: /services/CloudWatch
AWS_ELB_URL Path starts with: /services/LoadBalancing
AWS_SIMPLEWORKFLOW_URL Path starts with: /services/SimpleWorkflow

Create Eucalyptus eucaconsole HTTPS endpoint

Last but not least, we’ll make the eucaconsole frontend listener. This will be the same as the shared frontend listeners we made in the previous step, without an ACL.

  • Navigate to the frontend tab
  • Duplicate the Eucalyptus_HTTPS frontend
  • This will bring you to a window where you can edit the cloned frontend. Here’s what you need to change there
Edit the eucaconsole endpoint
  • Change the NameDescription and check the Shared Frontend checkbox.
    Name Eucalyptus_HTTPS_eucaconsole
    Description Eucalyptus console Public HTTPS Endpoint
    Status Active
    Shared Frontend
  • Set the Primary Frontend to be the Mantle_HTTPS_Frontend we created earlier
  • Set the Backend Server Pool to the eucaconsole backend.
  • Click on “Save”.

Test it out!

That’s it! This should work. Validate that you can reach your endpoints. Check the stats page, to see where connections are being routed.

It’s important to note that you must have an active backend associated with the main shared frontend, otherwise no traffic will be routed to the shared frontends.

 

Leave a Reply