Traefik on Docker Swarm, with fixed IP — Static/Reserved IP addresses for swarm services

Docker allows specific IP addresses on a host to be assigned to certain containers.  Traefik directs access to certain services, abstracting containers.  Docker Swarm marshals hosts into a “swarm”, but does away with fixed IP addresses.  Putting Traefik on Docker Swarm means that specific services can not (easily) be targeted by specific IP addresses, and it also means that Traefik on DS ‘takes over’ the host’s IP stack, applying its default configuration to all IPs on the host, unworkable in complex systems that require neutech to play nicely.

Traefik on Docker Swarm, with fixed IP -- Static/Reserved IP addresses for swarm services

 

A really good description of Traefik, Docker — from Elton here https://blog.sixeyed.com/arming-a-hybrid-docker-swarm-part-4-reverse-proxying-with-traefik/

reverse proxy quickly becomes a must-have when you’re running a container orchestrator with more than a couple of services. Network ports are single-occupancy resources, you can’t have multiple processes listening on the same port, whether they’re different apps or different containers.

You can’t run different web apps in containers all listening on port 80. Docker lets you map ports instead, so if the app inside a container expects traffic on port 80 you can actually publish to a different port on the host – say 8080 – and Docker will receive traffic on port 8080 and send it to port 80 in the container.

That doesn’t work for public-facing services though. Non-standard ports are only suitable for private or test environments. Public HTTP clients expect to use port 80, and HTTPS to use port 443. That’s where a reverse proxy comes in.

You run the proxy in a container, publishing ports 80 and 443. All your other services run in containers, but they don’t publish any ports – only the reverse proxy is accessible. It’s the single entrypoint for all your services, and it has rules to relay incoming requests to other containers using private Docker networks.

 

Below is some info and explored ways to surmount this ongoing major obstacle for all legacy installations.

 

From the Original Poster of the best thread I’ve found yet, here https://github.com/moby/moby/issues/24170

From June 2016 and still a problem —

There are some things that I want to run on docker, but are not fully engineered for dynamic infrastructure. Ceph is an example. Unfortunately, its monitor nodes require a static ip address, otherwise, it will break if restarted. See here for background: ceph/ceph-container#190

docker run has an --ip and --ip6 flag to set a static IP for the container. It would be nice if this is something we can do when creating a swarm service to take advantage of rolling updates and restart on failures.

For example, when we create a service, we could pass in a --static-ip and --static-ip6 option. Docker would assign a static ip for each task for the life of the service. That is, as long as the service exists, those ip addresses would be reserved and mapped to each task. If the task scales up and down, then more ip addresses are reserved or relinquished. The ip address is then passed into each task as an environment variable such as DOCKER_SWARM_TASK_IP and DOCKER_SWARM_TASK_IP6

One reply further down the thread (above), regarding “macvlan”, one proposed solution —
I don’t see macvlan mentioned here anywhere, so here goes! I think this would be immensely useful for setting up the same macvlan network on all nodes in a cluster, then being able to give a service a static IP address that can then roam between machines in the cluster but keep the same IP address via macvlan (so --ip would have to be a per-network option, which is probably obvious but worth being explicit about).

(As it stands, services can’t easily both talk to other services in the cluster and do multicast with other machines on the host network, which is what this sort of thing would enable.)

(Edit: there’s a great post about how macvlan in a cluster might work that describes exactly the thoughts I had at #25303 (comment) — using either an IP address range or an interface glob to auto-match the appropriate parent interface for each host)

macvlan may be a better solution instead of host networking in this case but it’s not supported by services. Also, a macvlan network need to be setup on each host in the cluster as it is implemented right now. Maybe some cluster wide setup may be possible to achieve by giving the “docker network” command an interface match condition, like IP-network address or interface name. Then a macvlan network may be setup on all hosts matching the condition in just one command. For the “docker service” command the network specification can be handled as a constraint so that containers are not scheduled on a host that does not have the specified network defined.

OverNode is a parallel project, supposedly to get swarm-like behavior from independent containers —

I have created and published the Overnode tool, which is basically multi-host docker compose without swarm, but it has some great features. It allows to assign static IP addresses too.  An example configuration for zookeeper / kafka can be found here: https://github.com/overnode-org/overnode/tree/master/examples/databases/kafka

image

Realization that available IP addresses should equal possible replicas — not sure how this would map across hosts —

I’m really surprised to see that people have suggested such exotic solutions to the problem, yet noone thought of the simplest one:

If the sole problem that prevents us from using --ip and --ip6 in a swarm is the possibility of multiple replicas, then simply require an amount of static IPs equal to the number of replicas.

Possible example:

replicas: 4

ipv4_address: { 192.168.1.1, 192.168.1.2, 192.168.1.3, 192.168.1.4 }

Scaling up/down should assign/deassign the addresses sequentially.

My addition (here https://github.com/moby/moby/issues/24170 ), which summarizes good possible ideas, links to relevant info —

Still evaluating how to best meet this aging need. Maybe others are facing this, too.

Our simple need is
(1) on a single host having multiple NICs, each NIC having multiple IP addresses (ipv4) and numerous services in complex arrangements — despite its awesomeness, Docker is still a duckling in established systems
(2) make Docker Swarm answer/respond on only a few, specific IP addresses, not all IP addresses on the host; Traefik runs as reverse-proxy in its own swarmed container — Traefik is primary target of specific IP addresses (trying to put Traefik in its own container outside of swarm causes its “internal” API and dashboard to become inaccessible)
(–) problem seems related to Traefik’s need to itself run swarmed for api/dashboard to work
(–) problem also seems related to Docker Swarm’s difficulty (or outright inability?) to bind to specific IP addresses on host with complex networking

Options
(A) Don’t do it this way; instead, rent fresh host whose IP stack can be totally taken over by Docker Swarm
(B) Investigate macvlan which is claimed workable by those in a parallel universe ( see moby/libnetwork#2249 )
(C) See also https://stackoverflow.com/questions/27937185/assign-static-ip-to-docker-container
(D) Consider https://serverfault.com/questions/893647/publish-docker-swarm-services-on-specific-ip-addresses
(E) See also #29816
(F) See also https://success.mirantis.com/article/how-do-i-change-the-docker-gwbridge-address
(G) #25303
(H) https://stackoverflow.com/questions/52301256/traefik-ha-in-docker-swarm-mode
(I) https://tiangolo.medium.com/docker-swarm-mode-and-traefik-for-a-https-cluster-20328dba6232 (dockerrocks)
(J) https://tiangolo.medium.com/docker-swarm-mode-and-distributed-traefik-proxy-with-https-6df45d0c0fc0

We’re seeing Docker Swarm ‘take over’ all the hosts’s IP addresses via iptables (despite having defined a Docker Network thusly:

#!/bin/bash
#! define custom docker network, intentionally limited to one public IP on host
docker network create
–subnet=172.20.20.0/24
–gateway=172.20.20.1
–driver overlay
–attachable
–opt “com.docker.network.bridge.name”=”net2020”
–opt “com.docker.network.bridge.host_binding_ipv4″=”1.2.3.4”
net2020
#! end

In other words, it’s not the host IP binding but that Docker iptables manipulation captures all port 80/443/etc to all IPs on the host. Therefore, this looks most immediately applicable, (from D) Quoting user name NewsNow1 —–

We had a need to publish separate docker swarm services on the same ports, but on separate specific IP addresses. Here’s how we did it.

Docker adds rules to the DOCKER-INGRESS chain of the nat table for each published port. The rules it adds are not IP-specific, hence normally any published port will be accessible on all host IP addresses. Here’s an example of the rule Docker will add for a service published on port 80:

iptables -t nat -A DOCKER-INGRESS -p tcp -m tcp –dport 80 -j DNAT –to-destination 172.18.0.2:80

(You can view these by running iptables-save -t nat | grep DOCKER-INGRESS).

Our solution is to publish our services on different ports, and use a script that intercepts dockerd’s iptables commands to rewrite them so they match the correct IP address and public port pair.

For example:

service #1 is published on port 1080, but should listen on 1.2.3.4:80
service #2 is published on port 1180, but should listen on 1.2.3.5:80

We then configure our script accordingly:

###! cat /usr/local/sbin/iptables
#!/bin/bash

REGEX_INGRESS=”^(.DOCKER-INGRESS -p tcp) (–dport [0-9]+) (-j DNAT –to-destination .)”
IPTABLES=/usr/sbin/iptables

SRV_1_IP=1.2.3.4
SRV_2_IP=1.2.3.5

ipt() {
echo “EXECUTING: $@” >>/tmp/iptables.log
$IPTABLES “$@”
}

if [[ “$*” =~ $REGEX_INGRESS ]]; then
START=${BASH_REMATCH[1]}
PORT=${BASH_REMATCH[2]}
END=${BASH_REMATCH[3]}

echo “REQUESTED: $@” >>/tmp/iptables.log

case “$PORT” in
‘–dport 1080’) ipt $START –dport 80 -d $SRV_1_IP $END; exit $?; ;;
‘–dport 2080’) ipt $START –dport 80 -d $SRV_2_IP $END; exit $?; ;;
*) ipt “$@”; exit $?; ;;
esac
fi

echo “PASSING-THROUGH: $@” >>/tmp/iptables.log

$IPTABLES “$@”

N.B. The script must be installed in dockerd’s PATH ahead of your distribution’s iptables command. On Debian Buster, iptables is installed to /usr/sbin/iptables, and dockerd’s PATH has /usr/local/sbin ahead of /usr/sbin, so it makes sense to install the script at /usr/local/sbin/iptables. (You can check dockerd’s PATH by running cat /proc/$(pgrep dockerd)/environ | tr ” ’12’ | grep ^PATH).

Now, when these docker services are launched, the iptables rules will be rewritten as follows:

iptables -t nat -A DOCKER-INGRESS -d 1.2.3.4/32 -p tcp -m tcp –dport 80 -j DNAT –to-destination 172.18.0.2:1080
iptables -t nat -A DOCKER-INGRESS -d 1.2.3.5/32 -p tcp -m tcp –dport 80 -j DNAT –to-destination 172.18.0.2:2080

The result is that requests for http://1.2.3.4/ go to service #1, while requests for http://1.2.3.5/ go to service #2.

The script can be customised and extended according to your needs, and must be installed on all nodes to which you will be directing requests, and customised to that node’s public IP addresses.

 

Some posts suggest removing docker_gwbridge network, then recreating it with docker network create -o “com.docker.network.bridge.host_binding_ipv4″=”192.168.1.151” docker_gwbridge, then running in host mode. Because docker_gwbridge is used by all services running on node’s engine (swarm or not), this “solution” precludes adjusting some services and some IP addresses and (presumably) forces all services on the one IP — it’s too much, but in the opposite direction.

When seemingly simple efforts encounter such incredible problem it indicates diametric opposition to intention, means, purpose — like squirting a garden hose upstream into a river.   Clearly better to ‘go with the flow’ of how DOCKER SWARM was designed and built (so far) than to force it using iptables machinations.  The best compromise I’ve found yet is to use docker-compose on each host needing assignable specific IP addresses from amongst a local pool that must otherwise remain untouched.  Swarm is currently best when allowed to take over the host’s IP stack, i.e., assign its ports to all available IP addresses.  By the time working iptables machination is devised, tested and installed, docker will likely have released an updated engine whose swarm mode much more easily allows capturing specific host IP addresses from amongst many.

Leave a Comment