Arnaud Rebillout

Linux/FOSS software engineer

Powered by Pelican • Source at GitLab
© 2015-2020 Arnaud Rebillout

Docker + Shippable for Painless Continuous Integration

Fri 13 January 2017

Here's my situation in a glimpse:

A very normal situation for a developer, isn't it?

Automating the build is a part of what is called Continuous Integration. You probably heard the term already, it's very trendy lately. It's trendy, and also fairly easy to setup if you use a major hosting service for your project, such as GitHub, Bitbucket or GitLab.

In this article, I'll describe the steps I followed to setup automatic builds for my application Goodvibes, an internet radio player for the GNU/Linux OS.

Goodvibes is hosted on Github.

Builds are automatically triggered thanks to Shippable, one of the numerous CI services available out there. Builds are performed on different environments: Debian, Ubuntu, Fedora at the moment.

These build environments are provided under the shape of docker images. I had to create these images myself, and I stored them on Docker Hub, so that Shippable can easily retrieve it.

One last word: the workflow described here can be reproduced without the services mentioned here. If you have your own GNU/Linux server, and are keen on git hooks and shell scripting, you can probably reproduce such an automated process fairly easily.

Introducing the Players

Player A: GitHub

GitHub is used to host the source code of my project. But it's also the key player in the automation process, thanks to the magic of git & web hooks. Changes in git repositories will trigger actions of the two other players: Shippable and Docker Hub.

Player B: Shippable

Shippable is in charge of automating the build. It supports C language, which means that there will be very little scripting to do.

Shippable already provides a base environment to build C applications. But you're not obliged to use it. The good thing is that Shippable uses docker images for its build environment. It means that using another environment is extremely simple: you just need to provide a different docker image.

Player C: Docker Hub

Docker Hub will host my build environments, aka docker images. That's all it does, and that's all I need. A perfect match.

The Big Picture

Let me know describe briefly the process as it will happen once everyting is setup.

Is it more or less clear? If you don't understand a word, it's time to quit reading. Otherwise, hold on!

To achieve that, we will go through the following steps:

Getting Started with Docker

I won't go into much details here, since the official documentation is just PER-FECT. I advise you to read it, and follow the tutorials.

First thing first, installation. It's very easy. If you're using Debian, the documentation is available here: https://docs.docker.com/engine/installation/linux/debian/

In a nutshell, it looks like that:

apt-get install apt-transport-https ca-certificates gnupg2

apt-key adv \
    --keyserver hkp://ha.pool.sks-keyservers.net:80 \
    --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

echo 'deb https://apt.dockerproject.org/repo debian-stretch main' \
    > /etc/apt/sources.list.d/docker.list

apt-get update
apt-get install docker-engine

systemctl daemon-reload
systemctl start docker

A few words of caution:

To avoid running Docker with sudo, you might want to create a group:

groupadd docker
gpasswd -a user docker
systemctl restart docker

Please note that:

Once Docker is up and running, it's time to start to play with it. Please follow the documentation here:

After you finish playing with Docker, you might want to cleanup behind you. There's a good summary of the commands you might need here:
https://www.calazan.com/docker-cleanup-commands/

Creating Docker Images for Your Build

Basically, what I need is a minimal environment that allows to build my C application.

Docker Hub provides bare, minimal images of some major distros, like Debian, Ubuntu, Fedora and such. These images are a base layer, I just need to top it up with the packages I need:

Once my images are ready, tested, validated, I must create a Dockerfile: this file contain the steps needed to build the image.

Dockerfiles are easy to create, easy to understand, but that shouldn't prevent you frome reading the "best practices" section from the Docker documentation: https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/

Dockerfiles are needed since we will later on create "automated build" on Docker Hub. Please focus here, otherwise it might be confusing. I'm talking about automated build of the docker images, in which I will later on run automated builds of my application. You get it ;)?

Please don't get lost in all this automation and read on.

Storing Dockerfiles on GitHub

Now that you have your Dockerfiles ready, you need to store them in a git repository, on GitHub or Bitbucket (both are supported by Docker Hub).

Please be logical in your naming. My application repository is goodvibes, so I created a repository named goodvibes-docker. As simple as that.

As an example, I invite you to have a look there:
https://github.com/elboulangero/goodvibes-docker

As you can see, I just created a sub-directory for each docker image I want to build, containing a single Dockerfile.

If you're a bit lost, don't worry, everything will become clear in the next step, where Docker Hub kicks in.

Creating a Docker Hub Account

Now that your Dockerfiles are ready and hosted on the net, the next step is to have a Docker Hub account, where you will store your docker images.

So do that. Create your account on Docker Hub, then link it with your GitHub account. Once it's done, you now want to create Automated Builds. That is, you want your images to be automatically generated according to your Dockerfiles.

It might be time for you to read some more documentation now:
https://docs.docker.com/docker-hub/builds/

The procedure is quite straightforward, there's little to explain. By default, the automatic build settings expect your Dockerfile to be at the root of your git repository. If it's not the case (and it's not in my case), just click on the Customize link when you can, and set the location of your Dockerfile.

Then repeat the procedure for each docker image you intend to build, so that in the end you have an automated build for each Dockerfile.

It seemed to me that the first build had to be triggered manually. Just check that in the Build Details section. If nothing's happening, just go to Build Settings and trigger a build manually.

Now, you want Docker Hub to be smart, and automatically rebuild your images when it's needed. When is it needed by the way?

To handle the first situation, it all happens on Docker Hub. Just enter the Build Settings section of your automated build, then the Repository Links, and add a link to the base image. Done.

For the second situation, it happens on GitHub. You will have to manually link the Docker service to your Github project. In order to do that, open GitHub, navigate to the Settings part of the repository containing your Dockerfiles, then choose Integration & services, and there you add Docker.

When it's done, Docker should automatically be notified when your Dockerfiles are modified, and rebuild your images accordingly.

As an example, you can have a look at my docker images here:
https://hub.docker.com/u/elboulangero/

Create a Shippable Account

We're almost there. We have our docker images hosted and ready, automatically built, easy to update, it's all wonderful. What's next? Setup the continuous integration service, which is Shippable in this tutorial.

Creating an account there is straightforward, nothing special.

Then, link it with your GitHub account, and then enable the projects you want to monitor. You might need a few clicks on the GitHub side as well, or maybe not, I don't remember. Anyway, this is simple, you won't get lost.

The very last step of this tutorial is coming: you now need to add a .shippable.yml file to your project, where you describe the steps for the build. I strongly advise you, one more time, to check the doc:
http://docs.shippable.com/ci/overview/

Shippable provides a default build image, Ubuntu something. If you don't want to use it, no problem. You can instead indicate the custom images you want to run. If these images are hosted on Docker Hub, it's very straightforward.

As an example, you can have a look at my .shippable.yml file:
https://github.com/elboulangero/goodvibes/blob/master/.shippable.yml

Finished

So let me draw you the whole picture again. I've you've been reading seriously up to now, you already have the picture in your mind:

   ----------------------
  | dockerfiles @ github |
   ----------------------
                  \
             source for
                    \
              ---------------------
             | images @ docker hub |
              ---------------------
                    /
               pull from
                  /
         -------------------
        | build @ shippable |
         -------------------
              /
          trigger
            /
 ------------------
| project @ github |
 ------------------

Easy to understand once all the pieces of the puzzle fit.

I hope you enjoyed the reading, now it's time to dock and ship!