https://manishrjain.com/reverse-proxy-caddy-nginx

Manish R Jain
Search Feed About Archives Outserv

HTTPS Reverse Proxy: Caddy outperforms NGINX 4x

In my setup, where I have an HTTPS load balancer, sending traffic
over HTTP to the server, aka reverse proxy, Caddy outperforms NGINX
by 4x.

That's 400% better performance via Caddy over an HTTPS connection
than NGINX. I'm using RPC Bench for this test.

Summary

With 4 concurrent requests, NGINX was able to run 100 calls/second,
with a 90%-ile latency of 48ms. Under the same conditions, Caddy ran
400 calls/second, with a 90%-ile latency of 16ms.

90%-ile response size is the same in both, which shows that the
results were identical and legit. Also, the rpc-bench script spot
checks for validity of results, and crashes in case it finds any
inconsistencies.

I ran the tests multiple times, with the same results to ensure that
the backend wasn't somehow caching things better. I ran NGINX, then
Caddy, then NGINX, then Caddy, and that loop 3 times to ensure the
results were correct. Because I found it unbelievable that Caddy
could outperform NGINX 4x over HTTPS.

I double checked my methodology, then triple checked it. But, there's
no other way to see this, except Caddy is indeed way faster when
dealing with HTTPS connections.

    What is happening?

    Caddy is 2x outperforming NGINX in my reverse proxy test  With
    Caddy, there's practically no difference in HTTPS vs HTTP
    performance.

    If legit, this is clearly a David vs Goliath story. @mholt6

    -- Manish R Jain | mrjn.eth (@manishrjain) November 23, 2022

Actually: My Tweet is inaccurate. It's not 2x faster, it's 4x faster!

Machine Specs

The machine I used for this test is a Digital Ocean, CPU-optimized
droplet, with 4 vCPUs, and 8 GB RAM, running Ubuntu 22.04 LTS.

{
  "name":"ubuntu-c-4-8gib-sfo3-01",
  "size":"c-4-8GiB",
  "region":"sfo3",
  "image":"ubuntu-22-10-x64"
}

$ inxi

CPU: quad core Intel Xeon Platinum 8358 (-MCP-) speed: 2600 MHz Kernel: 5.15.0-53-generic x86_64
Up: 16h 54m Mem: 528.5/7949.5 MiB (6.6%) Storage: 50 GiB (6.3% used) Procs: 120 Shell: Zsh
inxi: 3.3.13

$ lsb_release -a

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:    22.04
Codename:   jammy

Results with NGINX

Here's the relevant portion of NGINX config. The HTTPS certs were
auto issued by following instructions in this link.

server {
    root /var/www/html;
    server_name xxxxxxxxxxxxxxxxxxxxx;
  access_log off;

    location / {
        proxy_pass http://xxxxxxxxxxxxxx:80;
    }

  listen [::]:443 ssl ipv6only=on; # managed by Certbot
  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/xxxxxxxxxxxxxxxxxxxxx/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/xxxxxxxxxxxxxxxxxxxxx/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
  if ($host = xxxxxxxxxxxxxxxxxxxxx) {
      return 301 https://$host$request_uri;
  } # managed by Certbot


    listen 80 default_server;
    listen [::]:80 default_server;

    server_name xxxxxxxxxxxxxxxxxxxxx;
  return 404; # managed by Certbot
}

Based on nginx performance tips I ensured that worker_processes auto;
was already set in /etc/nginx/nginx.conf. I also turned off
access_log and gzip compression.

Num Queries: 7190 | Num 429:    0 | Data: 951 MiB [   1m0s @ 117 calls/sec ]
-----------------------
Latency in milliseconds
 -- Histogram:
Min value: 13
Max value: 101
Count: 7190
50p: 48.00
75p: 48.00
90p: 48.00

-----------------------
Resp size in bytes
 -- Histogram:
Min value: 1488
Max value: 1030245
Count: 7190
50p: 131072.00
75p: 262144.00
90p: 262144.00

Method: eth_getBlockByNumber | DONE

Results with Caddy

sudo caddy reverse-proxy --from <domain>:443 --to <server>:80

Num Queries: 25138 | Num 429:    0 | Data: 3.3 GiB [   1m0s @ 411 calls/sec ]
-----------------------
Latency in milliseconds
 -- Histogram:
Min value: 4
Max value: 44
Count: 25138
50p: 16.00
75p: 16.00
90p: 16.00

-----------------------
Resp size in bytes
 -- Histogram:
Min value: 1479
Max value: 1387712
Count: 25138
50p: 131072.00
75p: 262144.00
90p: 262144.00

Method: eth_getBlockByNumber | DONE

---------------------------------------------------------------------

Tags

caddy nginx baremetal networking
---------------------------------------------------------------------

Date

November 23, 2022
---------------------------------------------------------------------

Previously

Share One IP Across Servers with Keepalived As you might have picked
up, I run baremetal servers. They provide the best
performance-to-cost ratio. A problem I've been trying to solve for
some