Using Swarm with Calico on Docker Machine

Development

Reading Time: 11 minutes

In 2015, Docker announced their plugin system and revealed a list of early network and storage integrators. During Docker Global Hackday #3, we started playing with Swarm and Calico — however the tooling and integration at that stage made it difficult to implement.

Thanks to the hard work of contributors to both projects, the barrier to entry for implementing Swarm and Calico has been greatly reduced. In this article, we’ll look into implementing a Docker Swarm and using Calico as a network plugin, using Docker Machine.

Swarm allows a set of Docker hosts to be clustered, presenting a container API which abstracts scheduling of containers. By using a swarm instead of a set of hosts, much of the complexity around managing application availability and distributing resources is taken care of.

Project Calico provides a layer 3 network implementation, aimed at scalable datacenter deployments. Compared to traditional network overlays, Calico provides a more efficient implementation with minimal packet encapsulation. This allows better usage of node resources and a simple yet powerful network stack for your infrastructure.

While Calico is a great SDN solution for many cases, it has drawbacks in advanced cases. You can read more about why you should use Calico here.

To prototype using Swarm with Calico, I’ll be using Docker Machine creating VMs on VirtualBox. There may be some minor changes when using a different driver, but the core process should be the same.

Many of the steps taken here are not production safe — keep an eye out for warnings around implementation specifics in this document, as well as in the relevant provider docs.

Implementing Docker Swarm without Calico

Let’s start by creating a non-Calico Docker swarm using the standard Docker Machine interface. This is purely an exercise for comparison. We’ll be following the documentation listed in the Swarm docs.

You’ll need an existing Docker instance to get started, purely to create a Swarm discovery token, so if need be, create a Docker Machine instance just for this. Let’s create a Swarm discovery token:

$ docker run --rm swarm create
c25aa882df76a92ae962f4b4fc26168d

Next we can launch the necessary Swarm containers using this token to coordinate the swarm.

$ docker-machine create -d virtualbox --swarm --swarm-master --swarm-discovery token://c25aa882df76a92ae962f4b4fc26168d swarm-master
Running pre-create checks...
Creating machine...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Configuring swarm...
To see how to connect Docker to this machine, run: docker-machine env swarm-master

$ docker-machine create -d virtualbox --swarm --swarm-discovery token://c25aa882df76a92ae962f4b4fc26168d swarm-agent-00
Running pre-create checks...
Creating machine...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Configuring swarm...
To see how to connect Docker to this machine, run: docker-machine env swarm-agent-00
….
$ docker-machine create -d virtualbox --swarm --swarm-discovery token://c25aa882df76a92ae962f4b4fc26168d swarm-agent-NN
...

At this point, you should have a running Swarm cluster. Let’s take a look at it.

$ docker-machine ls
NAME             ACTIVE   DRIVER       STATE     URL                         SWARM
swarm-agent-00   -        virtualbox   Running   tcp://192.168.99.101:2376   swarm-master
swarm-agent-01   -        virtualbox   Running   tcp://192.168.99.102:2376   swarm-master
swarm-master     *        virtualbox   Running   tcp://192.168.99.100:2376   swarm-master (master)
$ eval $(docker-machine env --swarm swarm-master)
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                     NAMES
8f4cef9a02b3        swarm:latest        "/swarm join --advert"   9 minutes ago       Up 9 minutes        2375/tcp                                  swarm-agent-01/swarm-agent
01a3712457c7        swarm:latest        "/swarm join --advert"   10 minutes ago      Up 10 minutes       2375/tcp                                  swarm-agent-00/swarm-agent
5220dca465d2        swarm:latest        "/swarm join --advert"   12 minutes ago      Up 12 minutes       2375/tcp                                  swarm-master/swarm-agent
1205877156ee        swarm:latest        "/swarm manage --tlsv"   12 minutes ago      Up 12 minutes       2375/tcp, 192.168.99.100:3376->3376/tcp   swarm-master/swarm-agent-master
$ docker info
Containers: 4
Images: 3
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 3
 swarm-agent-00: 192.168.99.101:2376
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.021 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.1.13-boot2docker, operatingsystem=Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015, provider=virtualbox, storagedriver=aufs
 swarm-agent-01: 192.168.99.102:2376
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.021 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.1.13-boot2docker, operatingsystem=Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015, provider=virtualbox, storagedriver=aufs
 swarm-master: 192.168.99.100:2376
  └ Status: Healthy
  └ Containers: 2
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.021 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.1.13-boot2docker, operatingsystem=Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015, provider=virtualbox, storagedriver=aufs
CPUs: 3
Total Memory: 3.064 GiB
Name: 1205877156ee

docker ps now shows containers running on all nodes. You can interact with containers on the swarm as a whole using the Docker CLI.

Integrating Calico

Using Swarm with Calico requires a few changes from the standard swarm creation process.

Each node within the Swarm cluster must be configured to offload the cluster store to a location also accessible to the Calico cluster. Each host also needs to create a Calico node using the calicoctl binary. This configures the host and creates a set of Docker containers to maintain the cluster.

Because of these changes, we can’t create a swarm using the Docker Machine helpers. We will need to create individual machine hosts and then manually create Swarm and Calico clusters. We’ll be following the guide from the Calico docs for implementing Calico as a Docker network plugin. We hope to see more direct plugin support around the docker-machine tool in the future.

Sign up for a free Codeship Account

Create a machine cluster

To implement a Calico cluster, we’ll need a set of unclustered Docker hosts.

$ docker-machine create -d virtualbox node-00
Running pre-create checks...
Creating machine...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
To see how to connect Docker to this machine, run: docker-machine env node-00

$ docker-machine create -d virtualbox node-01
Running pre-create checks...
Creating machine...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
To see how to connect Docker to this machine, run: docker-machine env node-01
...
$ docker-machine create -d virtualbox node-NN
…
$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM
node-00   *        virtualbox   Running   tcp://192.168.99.100:2376   
node-01   -        virtualbox   Running   tcp://192.168.99.101:2376   
node-02   -        virtualbox   Running   tcp://192.168.99.102:2376

Set up a cluster store

We need to use an external cluster store, etcd, in order to synchronize the Calico and Swarm clusters.

We can run this etcd store on our cluster. In a production environment, this should be a scaled cluster supporting HA. However in this example, we’ll just run a single instance. We’ll run this single instance of etcd on node-00, which is on IP 192.168.99.100.

$ eval $(docker-machine env node-00)
$ docker run -d -p 2379:2379 quay.io/coreos/etcd -advertise-client-urls http://192.168.99.100:2379 -listen-client-urls http://0.0.0.0:2379
...
$ curl 192.168.99.100:2379/v2/keys
{"action":"get","node":{"dir":true}}

Set up a Calico cluster

Thanks to tooling and the fact that we’re using boot2docker as a base OS in Docker Machine, Calico requires very little setup.

We simply need to download calicoctl and use it to set up the cluster on each host. The ETCD_AUTHORITY variable will remain the same on each host, however NODE_IP should be the IP of the host.

$ docker-machine ssh node-00
...
docker $ wget http://www.projectcalico.org/latest/calicoctl
Connecting to www.projectcalico.org (64.91.234.195:80)
Connecting to www.projectcalico.org (64.91.234.195:80)
Connecting to github.com (192.30.252.128:443)
Connecting to github-cloud.s3.amazonaws.com (54.231.114.122:443)
calicoctl            100% |*******************************|  5428k  0:00:00 ETA
docker $ chmod +x calicoctl
docker $ sudo ETCD_AUTHORITY=192.168.99.100:2379 ./calicoctl node --libnetwork --ip=$NODE_IP
Pulling Docker image calico/node:v0.14.0

Calico node is running with id: 3cb0b50060d2bf423bd22f51ec30b9408bf5d199f631a70da7ca340902e7e134
Pulling Docker image calico/node-libnetwork:v0.7.0

Calico libnetwork driver is running with id: a8a9e8bd63e2540de76fb89fc40e31652b2df08392366046d3304626387e5b01
docker $ sudo  ETCD_AUTHORITY=192.168.99.100:2379  ./calicoctl status
calico-node container is running. Status: Up 7 minutes
Running felix version 1.3.0rc6

IPv4 BGP status
IP: 192.168.99.100    AS Number: 64511 (inherited)
+--------------+-----------+-------+-------+------+
| Peer address | Peer type | State | Since | Info |
+--------------+-----------+-------+-------+------+
+--------------+-----------+-------+-------+------+

IPv6 BGP status
No IPv6 address configured.

This needs to be run on every host, using the relevant host IP. Once this is done, calicotl should list all connected nodes.

docker $ sudo ETCD_AUTHORITY=192.168.99.100:2379 ./calicoctl status
calico-node container is running. Status: Up 19 seconds
Running felix version 1.3.0rc6

IPv4 BGP status
IP: 192.168.99.102    AS Number: 64511 (inherited)
+----------------+-------------------+-------+----------+-------------+
|  Peer address  |     Peer type     | State |  Since   |     Info    |
+----------------+-------------------+-------+----------+-------------+
| 192.168.99.100 | node-to-node mesh |   up  | 23:38:29 | Established |
| 192.168.99.101 | node-to-node mesh |   up  | 23:38:30 | Established |
+----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 address configured.

Configure your Docker hosts

Before we can set up a Swarm, we need to reconfigure each host to use the same cluster store as Calico.

This is complicated by the fact we are using boot2docker. However we can still reconfigure our Docker initialization parameters with an external cluster store. We can do this by adding a --cluster-store argument to /var/lib/boot2docker/profile.

 $ cat <<EOF > profile.new

EXTRA_ARGS='
--label provider=virtualbox
--cluster-store=etcd://192.168.99.100:2379
'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem

EOF
$ sudo chown root:root profile.new && sudo mv profile.new /var/l
ib/boot2docker/profile && sudo /etc/init.d/docker restart

Now your Docker hosts are configured to use the same cluster store as Calico.

Setting up a Swarm

Just like with the simple Swarm example, we first need to create a cluster token and then Swarm nodes and a manager.

If you plan on deploying Swarm in a production environment, be sure to read the Swarm docs on discovery methods and use something other than the standard Swarm token service.

Starting Swarm nodes is fairly standard. However we need to configure the manager with TLS enabled and mount the generated boot2docker certificates when starting the container.

Keep in mind that in this configuration the Swarm manager is a single point of failure. For production environments, be sure to follow the Swarm docs on HA deployments.

$ docker run --rm swarm create
938891a526e627d6ab11dd2e92cb8694
$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM
node-00   -        virtualbox   Running   tcp://192.168.99.100:2376   
node-01   -        virtualbox   Running   tcp://192.168.99.101:2376   
node-02   *        virtualbox   Running   tcp://192.168.99.102:2376  
$ eval $(docker-machine env node-00)
$ docker run -d swarm join --addr=192.168.99.100:2376 token://938891a526e627d6ab11dd2e92cb8694
$ eval $(docker-machine env node-01)
$ docker run -d swarm join --addr=192.168.99.101:2376 token://938891a526e627d6ab11dd2e92cb8694
$ eval $(docker-machine env node-02)
$ docker run -d swarm join --addr=192.168.99.102:2376 token://938891a526e627d6ab11dd2e92cb8694
$ docker run -dp 2377:2375 -v /var/lib/boot2docker:/var/lib/boot2docker swarm manage --tlsverify --tlscert /var/lib/boot2docker/server.pem --tlscacert /var/lib/boot2docker/ca.pem --tlskey /var/lib/boot2docker/server-key.pem token://938891a526e627d6ab11dd2e92cb8694

At this point, we have a Swarm manager with control over all nodes in the cluster, available via 192.168.99.102:2377. From here, we can follow the standard steps to test Docker Swarm and Calico listed in the Calico tutorial.

Rather than using the -H flag, you can also just redefine the DOCKER_HOST variable to specify the Swarm endpoint. Be sure to check your etcd cluster store to make sure it’s still running; restarting the Docker service may not have automatically started it.

$ eval $(docker-machine env node-02)
$ export DOCKER_HOST=tcp://192.168.99.102:2377
$ docker ps
fe3fcff4e0d2        calico/node-libnetwork:v0.7.0   "./start.sh"             About an hour ago   Up 56 minutes                                                                     node-02/calico-libnetwork
83a472b3ba65        calico/node:v0.14.0             "/sbin/start_runit"      About an hour ago   Up 56 minutes                                                                     node-02/calico-node
dbe7968c4707        calico/node-libnetwork:v0.7.0   "./start.sh"             About an hour ago   Up 53 minutes                                                                     node-01/calico-libnetwork
4bddb4e3d6b4        calico/node:v0.14.0             "/sbin/start_runit"      About an hour ago   Up 53 minutes                                                                     node-01/calico-node
a8a9e8bd63e2        calico/node-libnetwork:v0.7.0   "./start.sh"             About an hour ago   Up 54 minutes                                                                     node-00/calico-libnetwork
3cb0b50060d2        calico/node:v0.14.0             "/sbin/start_runit"      About an hour ago   Up 54 minutes                                                                     node-00/calico-node
3c9092601f5a        quay.io/coreos/etcd             "/etcd -advertise-cli"   2 hours ago         Up About a minute   2380/tcp, 4001/tcp, 192.168.99.100:2379->2379/tcp, 7001/tcp   node-00/boring_panini

It is also simpler to manage your Calico cluster by directly executing commands inline via docker-machine ssh, which can be aliased in Bash or scripted. But keep in mind that any time your Docker Machine needs to be restarted, you’ll need to re-download the calicoctl binary.

$ docker-machine ssh node-00 sudo ETCD_AUTHORITY=192.168.99.100:2379 ./calicoctl pool show

+----------------+---------+
|   IPv4 CIDR    | Options |
+----------------+---------+
| 192.168.0.0/16 |         |
+----------------+---------+
+--------------------------+---------+
|        IPv6 CIDR         | Options |
+--------------------------+---------+
| fd80:24e2:f998:72d6::/64 |         |
+--------------------------+---------+

Using Calico

Now that Calico is set up, what does that mean for your infrastructure and your applications?

First of all, Calico is not an entirely self-managing service. Be sure to read the documentation thoroughly to ensure you are applying the correct base iptables rules and configuring your nodes in a sensible manner. It’s important to understand what restrictions Calico brings to your infrastructure, as well as what it does and does not protect against. The Calico docs provide information on these concerns.

With the Calico libnetwork driver in place, you can manage networks and container IPs via the docker network interface. This means that after standard Calico node configuration, most operational changes required can be implemented through the libnetwork driver in much the same way such changes would be made using an overlay network.

Conclusion

Calico can greatly enhance your Docker infrastructure by facilitating scale and providing a more full-featured SDN than a standard overlay.

With recent updates in Docker and Calico tooling, setting up and maintaining a cluster is far simpler, and running one in a highly available manner in production is well documented. As further tooling and integrations develops, we expect this process to be even more simplified and configurable.

Calico provides a simple-to-implement container network, a highly scalable alternative to the standard overlay. By integrating with these networks via libnetwork and using a standard control interface, you can make it easy to switch Calico out for another network layer or connect different geographic regions as a single homogenous network.

Before settling on Calcio, be sure to read up on its benefits and those of its alternatives such as Weave and the standard overlay.

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles.
Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.



We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.

  • Under “Integrating Calico”, you may have mistyped the `calicoctl` executable as `clalicoctl`.

    • Well spotted, thanks for letting us know, we’ll fix that asap.