Setting up Docker Swarm in VMware NSX

Setting up Docker Swarm is pretty simple. BUT VMWare NSX is a little annoying, in that it blocks the VXLAN transport port (TCP Port 4789) at the hypervisor level. I’m sure this seemed GREAT for security, but it majorly messes up any application USING VXLAN inside the transport zone. Suck as Docker Swarm inside a cloud provider who uses VMWare NSX. As long as you know about this, you can work around it, however, as you can specify an alternate VXLAN port when you initialize your swarm! So let’s do that!

We will be bringing up a swarm on a cluster today with one manager and four nodes – each host has two network interfaces – we’ll be using ens160 in 10.129.2.0/24 for our transport network. we use the –data-path-port parameter to set the VXLAN port.

Note: Our manager, and all nodes, already need Docker installed, incase this isn’t obvious 😀

root@prod-swarm-manager-1:~# docker swarm init --data-path-port 4788 --advertise-addr 10.129.2.21
 Swarm initialized: current node (p9ojg9edmipi7saldcbrcnhyt) is now a manager.
 
To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-42rg6zgs3onagtyamztitzgqb21z9hmwnwfdqoabmew4ppk2i5-2r0upkukt2asdfsdf3234512ad 10.129.2.21:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. 

And we now have a swarm (with one node) up. Time to add more nodes!

root@prod-swarm-node-1:~# docker swarm join --token SWMTKN-1-42rg6zgs3onagtyamztitzgqb21z9hmwnwfdqoabmew4ppk2i5-2r0upkukt2asdfsdf3234512ad 10.129.2.21:2377
This node joined a swarm as a worker. 

root@prod-swarm-node-2:~# docker swarm join --token SWMTKN-1-42rg6zgs3onagtyamztitzgqb21z9hmwnwfdqoabmew4ppk2i5-2r0upkukt2asdfsdf3234512ad 10.129.2.21:2377
This node joined a swarm as a worker. 

root@prod-swarm-node-3:~# docker swarm join --token SWMTKN-1-42rg6zgs3onagtyamztitzgqb21z9hmwnwfdqoabmew4ppk2i5-2r0upkukt2asdfsdf3234512ad 10.129.2.21:2377
This node joined a swarm as a worker. 

root@prod-swarm-node-3:~# docker swarm join --token SWMTKN-1-42rg6zgs3onagtyamztitzgqb21z9hmwnwfdqoabmew4ppk2i5-2r0upkukt2asdfsdf3234512ad 10.129.2.21:2377
This node joined a swarm as a worker. 

We should now have our swarm up and running – run docker node list, to see!

root@prod-swarm-manager-1:~# docker node list
 ID                            HOSTNAME                 STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
 p9ojg9edmipi7saldcbrcnhyt *   prod-swarm-manager-1     Ready               Active              Leader              19.03.1
 lnp2b2ijurmtamp0if4aner7y     prod-swarm-node-1        Ready               Active                                  19.03.1
 caxka5zdq0nb9lilcvss1fv82     prod-swarm-node-2        Ready               Active                                  19.03.1
 k0ar3rgjzoz1jjncfpr5xd9t1     prod-swarm-node-3        Ready               Active                                  19.03.1
 oa0ym3ytsgf5svbs2rz205jwr     prod-swarm-node-4        Ready               Active                                  19.03.1

We do, perfect! We now want to manage the swarm with a nice web interface, so lets bring up swarmpit.

root@prod-swarm-manager-1:~# docker run -it --rm \
>   --name swarmpit-installer \
>   --volume /var/run/docker.sock:/var/run/docker.sock \
>   swarmpit/install:1.7
Unable to find image 'swarmpit/install:1.7' locally
1.7: Pulling from swarmpit/install
e7c96db7181b: Pull complete 
5297bd381816: Pull complete 
3a664477889c: Pull complete 
a9b893dcc701: Pull complete 
48bf7c1cb0dd: Pull complete 
555b6ea27ad2: Pull complete 
7e8a5ec7012a: Pull complete 
6adc20046ac5: Pull complete 
42a1f54aa48c: Pull complete 
717a4f34e541: Pull complete 
f95ad45cac17: Pull complete 
f963bb249c55: Pull complete 
Digest: sha256:04e47b8533e5b4f9198d4cbdfea009acac56417227ce17a9f1df549ab66a8520
Status: Downloaded newer image for swarmpit/install:1.7
                                        _ _   
 _____      ____ _ _ __ _ __ ___  _ __ (_) |_ 
/ __\ \ /\ / / _` | '__| '_ ` _ \| '_ \| | __|
\__ \\ V  V / (_| | |  | | | | | | |_) | | |_ 
|___/ \_/\_/ \__,_|_|  |_| |_| |_| .__/|_|\__|
                                 |_|          
Welcome to Swarmpit
Version: 1.7
Branch: 1.7

Preparing dependencies
latest: Pulling from byrnedo/alpine-curl
8e3ba11ec2a2: Pull complete 
6522ab4c8603: Pull complete 
Digest: sha256:e8cf497b3005c2f66c8411f814f3818ecd683dfea45267ebfb4918088a26a18c
Status: Downloaded newer image for byrnedo/alpine-curl:latest
DONE.

Preparing installation
Cloning into 'swarmpit'...
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 17028 (delta 1), reused 1 (delta 0), pack-reused 17022
Receiving objects: 100% (17028/17028), 4.39 MiB | 3.05 MiB/s, done.
Resolving deltas: 100% (10146/10146), done.
DONE.

Application setup
Enter stack name [swarmpit]: prod-swarmpit
Enter application port [888]: 
Enter database volume driver [local]: 
Enter admin username [admin]: 
Enter admin password (min 8 characters long): SYJpt6FQ@*j2ztPZ53^yF@!q5VRkZRyr*h$ydWGYE67$RWaHWat5Q$g6#zQtA3q^8QgQeSAMBEPT2^z8t2y#GKb5^X%e
DONE.

Application deployment
Creating network prod-swarmpit_net
Creating service prod-swarmpit_db
Creating service prod-swarmpit_agent
Creating service prod-swarmpit_app
DONE.

Starting swarmpit............DONE.
Initializing swarmpit...DONE.

Summary
Username: admin
Password: SYJpt6FQ@*j2ztPZ53^yF@!q5VRkZRyr*h$ydWGYE67$RWaHWat5Q$g6#zQtA3q^8QgQeSAMBEPT2^z8t2y#GKb5^X%e
Swarmpit is running on port :888

Enjoy :)

And bingo! If I hit up the manager host on port 888, I can login and view the swarm state!

Leave a Comment