Deploy behind nginx

This guide demonstrates a way to load balance connections across multiple websockets server processes running on the same machine with nginx.

We’ll run server processes with Supervisor as described in this guide.

Run server processes

Save this app to app.py:

#!/usr/bin/env python

import asyncio
import os
import signal

from websockets.asyncio.server import unix_serve


async def echo(websocket):
    async for message in websocket:
        await websocket.send(message)


async def main():
    # Set the stop condition when receiving SIGTERM.
    loop = asyncio.get_running_loop()
    stop = loop.create_future()
    loop.add_signal_handler(signal.SIGTERM, stop.set_result, None)

    async with unix_serve(
        echo,
        path=f"{os.environ['SUPERVISOR_PROCESS_NAME']}.sock",
    ):
        await stop


if __name__ == "__main__":
    asyncio.run(main())

We’d like nginx to connect to websockets servers via Unix sockets in order to avoid the overhead of TCP for communicating between processes running in the same OS.

We start the app with unix_serve(). Each server process listens on a different socket thanks to an environment variable set by Supervisor to a different value.

Save this configuration to supervisord.conf:

[supervisord]

[program:websockets-test]
command = python app.py
process_name = %(program_name)s_%(process_num)02d
numprocs = 4
autorestart = true

This configuration runs four instances of the app.

Install Supervisor and run it:

$ supervisord -c supervisord.conf -n

Configure and run nginx

Here’s a simple nginx configuration to load balance connections across four processes:

daemon off;

events {
}

http {
    server {
        listen localhost:8080;

        location / {
            proxy_http_version 1.1;
            proxy_pass http://websocket;
            proxy_set_header Connection $http_connection;
            proxy_set_header Upgrade $http_upgrade;
        }
    }

    upstream websocket {
        least_conn;
        server unix:websockets-test_00.sock;
        server unix:websockets-test_01.sock;
        server unix:websockets-test_02.sock;
        server unix:websockets-test_03.sock;
    }
}

We set daemon off so we can run nginx in the foreground for testing.

Then we combine the WebSocket proxying and load balancing guides:

• The WebSocket protocol requires HTTP/1.1. We must set the HTTP protocol version to 1.1, else nginx defaults to HTTP/1.0 for proxying.

• The WebSocket handshake involves the Connection and Upgrade HTTP headers. We must pass them to the upstream explicitly, else nginx drops them because they’re hop-by-hop headers.

  We deviate from the WebSocket proxying guide because its example adds a Connection: Upgrade header to every upstream request, even if the original request didn’t contain that header.

• In the upstream configuration, we set the load balancing method to least_conn in order to balance the number of active connections across servers. This is best for long running connections.

Save the configuration to nginx.conf, install nginx, and run it:

$ nginx -c nginx.conf -p .

You can confirm that nginx proxies connections properly:

$ PYTHONPATH=src python -m websockets ws://localhost:8080/
Connected to ws://localhost:8080/.
> Hello!
< Hello!
Connection closed: 1000 (OK).