Recently at work, we were trying to get an application that uses websockets working on an AWS instance behind an ELB (load balancer) and nginx on the instance.

If you’re either not using a secure connection or handling the cryptography on the instance (either in nginx or Flask), it works right out of the box. But if you want the ELB to handle TLS termination it doesn’t work nearly as well… Luckily, after a bit of fiddling, I got it working.

Update 2018-05-31: A much easier solution, [https://aws.amazon.com/blogs/aws/new-aws-application-load-balancer/](just use an ALB):

WebSocket allows you to set up long-standing TCP connections between your client and your server. This is a more efficient alternative to the old-school method which involved HTTP connections that were held open with a “heartbeat” for very long periods of time. WebSocket is great for mobile devices and can be used to deliver stock quotes, sports scores, and other dynamic data while minimizing power consumption. ALB provides native support for WebSocket via the ws:// and wss:// protocols.

First, we have a basic application. For my purposes, I wrote a quick Websocket chat app: ws-chat. The particular implementation details aren’t as important. We’ll start with the nginx config file:

upstream webserver {
    server 127.0.0.1:8000;
}

upstream wsserver {
    server 127.0.0.1:9000;
}

server {
    listen 80 proxy_protocol;

    location / {
        if ($http_x_forwarded_proto = "http") {
            return 301 https://$host$request_uri;
        }

        proxy_pass http://webserver;
    }

    location /ws/ {
        proxy_pass http://wsserver;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Straight forward enough. We have two backend services: a web server running on port 8000 (a simple Flask server that just servers a single HTML page) and the websocket backend running on port 9000. Alternatively, these could be the same codebase. The important parts are that you allow the Websocket upgrade header to pass through to establish the connection and that you tell nginx to listen using the proxy_protocol, an extra header that passes through extra information:

PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"

This seems like it wouldn’t be necessary, except that without proxy_protocol AWS ELBs seem to strip something important to the connection.

Next, we need to configure the load balancer. One complication here is that telling the load balancer to forward HTTPS traffic to HTTP will not work for the websockets. Instead, you have to configure it to forward TCP (SSL) to TCP. This will still work for HTTP/HTTPS traffic (as HTTP is just a specific protocol over TCP and HTTPS is just HTTP with a TLS layer), but it will also allow the non-HTTP websocket traffic to pass through successfully. Something like this:

(Don’t forget to set the certificate :))

Finally, you have to configure the ELB also to speak proxy protocol. This part is slightly more annoying, since (at least now), there’s no way to configure this through the AWS console. You have to use the AWS CLI.

First, create the new policy (assuming you have an environment variable ELB_NAME defined):

aws elb create-load-balancer-policy \
    --load-balancer-name $ELB_NAME \
    --policy-name $ELB_NAME-proxy-protocol \
    --policy-type-name ProxyProtocolPolicyType \
    --policy-attributes AttributeName=ProxyProtocol,AttributeValue=True

Then, attach it to the load balancer. You will have to run this once for each port that the instance is listening on:

aws elb set-load-balancer-policies-for-backend-server \
    --load-balancer-name $ELB_NAME \
    --instance-port 80 \
    --policy-names $ELB_NAME-proxy-protocol

Make sure that you’re using https:// for the web traffic and wss:// for the websocket and you’re golden. Encrypted websockets behind an AWS ELB. Now if only they would expose the proxy protocol options in the console…