Reverse proxy partially works on docker swarm - linux

Question

I setup a docker swarm with 3 nodes :
s1 : manager + worker
s2 : worker
s3 : worker
I deployed a nginx as a reverse proxy to a docker swarm service on each node with publishing port as mode=host to get the real ip. Nginx works "fine", i'am able to serve static content, use over https, etc ...
The part which doesn't work is the reverse_proxy :
if the nginx and the service are on the same node, everything works
if the nginx and the service aren't one the same node, i can only GET / because others requests ( like /css/style.css ) will fails with 499 ( from nginx point )
nginx network is an overlay network swarm-scopped and ip forwarding is enabled.
Here is my nginx configuration :
server {
listen 80;
server_name service.foo.bar;
location / {
proxy_pass http://service:80;
}
}
server {
listen 443 ssl;
server_name service.foo.bar;
ssl_certificate /ssl/service.foo.bar/fullchain.pem;
ssl_certificate_key /ssl/service.foo.bar/privkey.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
proxy_pass http://service:80;
}
}
Here is how i deployed my nginx :
docker service create --name nginx --mount /etc/nginx/nginx.conf:/etc/nginx/nginx.conf --mode=global --publish mode=host,published=80,target=80 --publish mode=host,published=443,target=443 --network nginx nginx
If i curl the node who hosts the service :
* TCP_NODELAY set
* Connected to service.foo.bar port 80 (#0)
> GET /app/polyfills.d92dcdb0a986e964fec8.js HTTP/1.1
> Host: service.foo.bar
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.21.6
< Date: Mon, 13 Jun 2022 15:38:36 GMT
< Content-Type: application/javascript
< Content-Length: 257750
< Connection: keep-alive
< cache-control: public, immutable, max-age=604800
< expires: Mon, 20 Jun 2022 15:38:36 GMT
< permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), camera=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), sync-xhr=(self "https://haveibeenpwned.com" "https://2fa.directory"), usb=(), vr=()
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< referrer-policy: same-origin
< x-xss-protection: 0
<
/*! For license information please see polyfills.d92dcdb0a986e964fec8.js.LICENSE.txt */
[...]
If i curl a node which doesn't host the service :
* TCP_NODELAY set
* Connected to service.foo.bar port 80 (#0)
> GET /app/polyfills.d92dcdb0a986e964fec8.js HTTP/1.1
> Host: service.foo.bar
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.21.6
< Date: Mon, 13 Jun 2022 15:38:25 GMT
< Content-Type: application/javascript
< Content-Length: 257750
< Connection: keep-alive
< cache-control: public, immutable, max-age=604800
< expires: Mon, 20 Jun 2022 15:38:25 GMT
< permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), camera=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), sync-xhr=(self "https://haveibeenpwned.com" "https://2fa.directory"), usb=(), vr=()
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< referrer-policy: same-origin
< x-xss-protection: 0
<
* transfer closed with 257750 bytes remaining to read
* Closing connection 0
curl: (18) transfer closed with 257750 bytes remaining to read
nginx log say :
nginx.0.scembp2e9iqp#s3 | 2022/06/13 15:38:36 [warn] 23#23: *114 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/5/00/0000000005 while reading upstream, client: #ip, server: service.foo.bar, request: "GET /app/polyfills.d92dcdb0a986e964fec8.js HTTP/1.1", upstream: "http://10.0.4.56:80/app/polyfills.d92dcdb0a986e964fec8.js", host: "service.foo.bar"
My nodes are connected each others over wireguard, this is my routing table :
default via #ip dev ens3
#ip dev ens3 scope link
10.252.1.0/24 dev wg0 proto kernel scope link src 10.252.1.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.19.0.0/16 dev docker_gwbridge proto kernel scope link src 172.19.0.1
Here is my wireguard configuration :
[Interface]
Address = 10.252.1.1/24
ListenPort = 51820
PrivateKey = ***
[Peer]
PublicKey = ***
AllowedIPs = 10.252.1.2/32
Endpoint = #s2
[Peer]
PublicKey = ***
AllowedIPs = 10.252.1.3/32
Endpoint = #s3
This is my firewall configuration :
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:FILTERS - [0:0]
:DOCKER-USER - [0:0]
-F INPUT
-F DOCKER-USER
-F FILTERS
-A INPUT -i lo -j ACCEPT
-A INPUT -j FILTERS
-A DOCKER-USER -i ens3 -j FILTERS
-A FILTERS -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FILTERS -p icmp --icmp-type echo-request -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A FILTERS -p udp --dport 51820 -j ACCEPT
-A FILTERS -s 10.252.1.0/24 -j ACCEPT
-A FILTERS -j REJECT --reject-with icmp-port-unreachable
COMMIT
Any ideas ? Am i missing something ?