Building Minimal Docker Containers for Go Applications

Development

Reading Time: 6 minutes

There are several great official and community-supported containers for many programming languages, including Go, but these containers can be quite large. Let’s walk through a comparison of methods for building containers for Go applications, then I’ll show you a way to statically build Go apps for containerization that are extremely small.

Part One: Our “app”

We need something to test for our app, so let’s make something pretty small: we’re going to fetch google.com and output the size of the html we fetch:

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
)

func main() {
    resp, err := http.Get("https://google.com")
    check(err)
    body, err := ioutil.ReadAll(resp.Body)
    check(err)
    fmt.Println(len(body))
}

func check(err error) {
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }
}

If we run this, it will just print out some numbers. For me, it was around 17k. I purposefully decided to do something with SSL for reasons I promise to explain later.

Part 2: Dockerize

Following the official Docker image for Go, we would write an “onbuild” Dockerfile like this:

FROM golang:onbuild

The “onbuild” images assume your project structure is standard and will build your app like a generic Go app. If you want more control, you could use their standard Go base image and compile yourself:

FROM golang:latest 
RUN mkdir /app 
ADD . /app/ 
WORKDIR /app 
RUN go build -o main . 
CMD ["/app/main"]

This is nice if you have a Makefile or something else nonstandard you have to do when you’re building your app. We could download some assets from a CDN or maybe add them in from another project, or maybe we want to run our tests within the container.

As you can see, Dockerizing Go apps is pretty straightforward, especially since we don’t have any services or ports we need access to or to export. But there’s one big drawback to the official images: they’re really big. Let’s take a look:

REPOSITORY SIZE     TAG         IMAGE ID            CREATED              VIRTUAL SIZE
example-onbuild     latest      9dfb1bbac2b8        19 minutes ago       520.7 MB
example-golang      latest      02e19291523e        19 minutes ago       520.7 MB
golang              onbuild     3be7ee2ec1ae        9 days ago           514.9 MB
golang              1.4.2       121a93c90463        9 days ago           514.9 MB
golang              latest      121a93c90463        9 days ago           514.9 MB

The bases are 514.9MB and our app adds just 5.8MB to that. Wow. So for our compiled application we still need 514.9MB of dependencies? How did that happen?

The answer is that our app was compiled inside the container. That means the container needs Go installed, and that means it needs Go’s dependencies, which means we need a package manager and really an entire OS. In fact, if you look at the Dockerfile for golang:1.4, it starts with Debian Jessie, installs the GCC compiler and some build tools, curls down Go, and installs it. So we pretty much have a whole Debian server and the Go toolkit just to run our tiny app. What can we do?

Try Codeship – The simplest Continuous Delivery service out there.

Part 3: Compile!

The way to improve is to do something a little… off the beaten path. What we’re going to do is compile Go in our working directory, then add the binary into the container. That means a simple docker build won’t work. We need a multi-step container build:

go build -o main .
docker build -t example-scratch -f Dockerfile.scratch .

And Dockerfile.scratch is simply:

FROM scratch
ADD main /
CMD ["/main"]

So what’s scratch? Scratch is a special docker image that’s empty. It’s truly 0B:

REPOSITORY          TAG         IMAGE ID            CREATED              VIRTUAL SIZE
example-scratch     latest      ca1ad50c9256        About a minute ago   5.60 MB
scratch             latest      511136ea3c5a        22 months ago        0 B

Also, our container is just that 5.6MB! Cool! But there’s one problem:

$ docker run -it example-scratch
no such file or directory

Huh? What does that mean? Took me a while to figure it out, but our Go binary is looking for libraries on the operating system it’s running in. We compiled our app, but it still is dynamically linked to the libraries it needs to run (i.e., all the C libraries it binds to). Unfortunately, scratch is empty, so there are no libraries and no loadpath for it to look in. What we have to do is modify our build script to statically compile our app with all libraries built in:

CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .

We’re disabling cgo which gives us a static binary. We’re also setting the OS to Linux (in case someone builds this on a Mac or Windows) and the -a flag means to rebuild all the packages we’re using, which means all the imports will be rebuilt with cgo disabled. These settings changed in Go 1.4 but I found a workaround in a GitHub Issue. Now we have a static binary! Let’s try it out:

$ docker run -it example-scratch
Get https://google.com: x509: failed to load system roots and no roots provided

Great, now what? This is why I chose to use SSL in our example. This is a really common gotcha for this scenario: to make SSL requests, we need the SSL root certificates. So how do we add these to our container?

Depending on the operating system, these certificates can be in many different places. If you look at Go’s x509 library, you can see all the locations where Go searches. For many Linux distributions, this is /etc/ssl/certs/ca-certificates.crt. So first, we’ll copy the ca-certificates.crt from our machine (or a Linux VM or an online certificate provider) into our repository. Then we’ll add an ADD to our Dockerfile to place this file where Go expects it:

FROM scratch
ADD ca-certificates.crt /etc/ssl/certs/
ADD main /
CMD ["/main"]

Now just rebuild our image and run it, and it works! Cool! Let’s see how big our app is now:

REPOSITORY          TAG         IMAGE ID            CREATED              VIRTUAL SIZE
example-scratch     latest      ca1ad50c9256        About a minute ago   6.12 MB
example-onbuild     latest      9dfb1bbac2b8        19 minutes ago       520.7 MB
example-golang      latest      02e19291523e        19 minutes ago       520.7 MB
golang              onbuild     3be7ee2ec1ae        9 days ago           514.9 MB
golang              1.4.2       121a93c90463        9 days ago           514.9 MB
golang              latest      121a93c90463        9 days ago           514.9 MB
scratch             latest      511136ea3c5a        22 months ago        0 B

We’ve added a little more than half a meg (and most of this is from the static binary, not the root certs). This is a really nice little container — it’ll be really easy to push and pull between registries.

Conclusion

Our goal in this post was to whittle down the container size for a Go application. Go is special in that it can create a statically linked binary that fully contains the application. Other languages can do this, but certainly not all of them. If we were to apply this technique of reducing container size to other languages, it would depend on what their minimal requirements are. For example, a Java or JVM app could be compiled outside a container and then be injected into a container that only has a JVM (and its dependencies). This is at least smaller than a container with the JDK present.

I’m really looking forward to the strides the community makes in creating both minimal OSes for container guests, as well as aggressively trimming down the requirements for all kinds of languages. The great thing about the public Docker hub is these can be shared with everyone easily.

PS: If you liked this article you might also be interested in one of our free eBooks from our Codeship Resources Library. Download it here: Turning Your App into Separate Containers for Better CI/CD

Posts you may also find interesting:

Are you interested in more articles like this one? Please let us know in the comments!

Discuss this article on HackerNews: https://news.ycombinator.com/item?id=9432429

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles.
Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.



We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.

  • You made me want to use Go… anything similar for nodejs? :D

    • Ha! I dunno. You’d have to bundle up a lot of dependencies before injecting. But you’d want to inject a statically compiled nodejs binary then all your js code. Good luck.

  • I am doing something similar, but using alpine linux as base – https://github.com/gliderlabs/docker-alpine. It’s still really small.

    You can then install additional stuff with package manager, like for instance the CA certificates.

    You can see how I build static binary and shippable container at

    https://github.com/t0mk/dnscock

    • Awesome! Alpine looks really cool. I didn’t know about it.

  • Pingback: 1p – Building Minimal Docker Containers for Go Applications | Profit Goals()

  • Pingback: 1p – Building Minimal Docker Containers for Go Applications | blog.offeryour.com()

  • Just seconding alpine – you don’t have to worry about the cgo stuff there either. My containers are ~20MB with it, but I think it’s worth the slightly increased space to have all the packages I need out of the box.

    You can get pretty efficient with it too: https://github.com/gliderlabs/registrator/blob/master/Dockerfile

    • Whoa really cool! I searched for small linux distros but somehow didn’t find alpine. I agree I would not mind dropping static compilation for a real image like alpine given the small space usage.

    • marcosnils

      @reactor5:disqus I’ve tried to run a program with CGO support in the image you proposed but I’m getting an error that the binary is not able to locate the dynamic libraries needed.

      How’s this supposed to work?

      • Worked for me by installing `gcc` and `libc-dev` in Alpine. Whatever else you need you should either install or copy to the container before build.

  • marcosnils

    You should definitely need to take a look at this: https://github.com/jamiemccrindle/dockerception

    • We’re well aware of it. We are working on something similar *wink*.

  • Adriaan de Jonge

    Last year, I used a slightly different approach to achieve the same: http://blog.xebia.com/2014/07/04/create-the-smallest-possible-docker-container/

    • Yeah cool! I may have stumbled on this while working on this myself. The issue is that Go 1.4 changed the way static compilations worked, so I wanted to share the technique again.

  • José Coelho

    You should use ENTRYPOINT instead of CMD in the last Dockerfile, you can’t pass arguments otherwise

    • Yes, that is how Docker works. For the purpose of this article this is irrelevant.

  • Benjamin Thomas

    I don’t understand why you’d wanna use docker to ship a binary? Also a bunch of containers will share the same cached commits so the space issue would only be a problem if you only have 1 running/installed container on the server.

    Or am I missing something?

    • I would use docker to ship a binary because it’s a common interface that developers and hosting providers standardize on. This boils down to “why use Docker”. A binary is just how I happen to wrap up my app.

      Your second point assumes a long running server maintaining a cache. I prefer using an immutable system and not relying on the cache to be present. AFAIK elasticbeanstalk’s docker platform does not keep a cache. I don’t know if ECS does.

  • dimastopel

    Thanks Nick, great post.

    Specific question. My go code uses user.Current() from os/user. When I use your technique I get the following error: user: Current not implemented on linux/amd64

    Did you or other fellows in the comments encounter this? Any solution?

    Thanks,

    • Like the SSL issue, when you want something from the OS you need an OS :-). You’d have to look at (1) what you need the user for and (2) how the os stores and retrieves the user in go.

      But yeah, this will happen because we’ve removed the OS. If it’s too much trouble it’s not worth it.

      • dimastopel

        Will look into it. Thanks!

  • jimcumming

    It would be great if you could work up a full example of a JVM app, maybe something that runs on tomcat. I’d be really interested to see how that works.

  • Great article. Thanks!

  • sillypog

    Because you’re compiling the binary outside of a container, aren’t you relying on each developer’s machine having the same versions of the linked libraries? It might be better to compile the code inside a different container and have it write the executable to a shared volume on the host, which you could then load in during your scratch build.

    • Scott Miller

      You could do that, but that’s also what a CI pipeline is for.

  • rossjimenez
  • Pingback: Running minimal golang apps in kubernetes cluster | Bite-Code()

  • Vince Yuan

    It helped me. Thanks!

  • Pingback: Veratyr comments on "Super small Docker image based on Alpine Linux"()

  • Kyle C. Quest

    There’s another way to create small images using your first approach. Run DockerSlim [1] on your 520MB image and you’ll end up with an image very close to your compiled application size (5.8). That’s it and you won’t have to deal with any of the Alpine image gotchas :)

    [1] http://dockersl.im

  • George Blazer

    What does go build -o main . look like if you have third party dependencies?

  • Dave Mazzoni

    Great post — prefect for creating standalone apps for docker. So I’ve compiled a go binary and included it in a scratch docker image. However, I cannot find a way to pass arguments to my (go) program at “docker run” time. What am I missing?

  • Pingback: Go RESTful Services Tutorial: Moving forward and improving your app()

  • Pingback: 创建超小的Golang docker 镜像 | 心似浮雲常自在 意如流水任東西()

  • mingderwang

    Have problem to use in this way when go need to get sqlite3.

  • bluealert
  • fletcher91

    Thanks for your article, we’ve been using this method for a while now and it works great.

    I’ve build a small golang CLI program which does all these steps automatically and cleans up after itself :) https://github.com/fletcher91/docker-go

  • Pingback: slimage:为 Go 语言应用创建最小化的 Docker 镜像 - 莹莹之色()

  • Amir Keibi

    I wonder if you have tried deploying the images you’ve built this way (from `scratch`) to Kubernetes.

  • Carl Mosca

    Very nice, thank you.

  • Pingback: 用 Docker Multi-Stage 編譯出 Go 語言最小 Image | 小惡魔 - 電腦技術 - 工作筆記 - AppleBOY()