Docker + Shippable for Painless Continuous IntegrationFri 13 January 2017
Here's my situation in a glimpse:
- I've been writing an application in C
- I target the GNU/Linux OS
- I want to ensure it builds on every major distros
- I want an automated solution
A very normal situation for a developer, isn't it?
Automating the build is a part of what is called Continuous Integration. You probably heard the term already, it's very trendy lately. It's trendy, and also fairly easy to setup if you use a major hosting service for your project, such as GitHub, Bitbucket or GitLab.
In this article, I'll describe the steps I followed to setup automatic builds for my application Goodvibes, an internet radio player for the GNU/Linux OS.
Goodvibes is hosted on Github.
Builds are automatically triggered thanks to Shippable, one of the numerous CI services available out there. Builds are performed on different environments: Debian, Ubuntu, Fedora at the moment.
These build environments are provided under the shape of docker images. I had to create these images myself, and I stored them on Docker Hub, so that Shippable can easily retrieve it.
One last word: the workflow described here can be reproduced without the services mentioned here. If you have your own GNU/Linux server, and are keen on git hooks and shell scripting, you can probably reproduce such an automated process fairly easily.
Introducing the Players
Player A: GitHub
GitHub is used to host the source code of my project. But it's also the key player in the automation process, thanks to the magic of git & web hooks. Changes in git repositories will trigger actions of the two other players: Shippable and Docker Hub.
Player B: Shippable
Shippable is in charge of automating the build. It supports C language, which means that there will be very little scripting to do.
Shippable already provides a base environment to build C applications. But you're not obliged to use it. The good thing is that Shippable uses docker images for its build environment. It means that using another environment is extremely simple: you just need to provide a different docker image.
Player C: Docker Hub
Docker Hub will host my build environments, aka docker images. That's all it does, and that's all I need. A perfect match.
The Big Picture
Let me know describe briefly the process as it will happen once everyting is setup.
- my C project is hosted on GitHub, remember?
- when I push a change, Shippable is notified (thanks to the hooks magic)
- Shippable wants to build. For that:
- it pulls the docker images from Docker Hub
- then it builds in these contained environments
Is it more or less clear? If you don't understand a word, it's time to quit reading. Otherwise, hold on!
To achieve that, we will go through the following steps:
- prepare docker images that allow building our app
- store it on Docker Hub
- link our Github project to Shippable to automate the build
Getting Started with Docker
I won't go into much details here, since the official documentation is just PER-FECT. I advise you to read it, and follow the tutorials.
First thing first, installation. It's very easy. If you're using Debian, the documentation is available here: https://docs.docker.com/engine/installation/linux/debian/
In a nutshell, it looks like that:
apt-get install apt-transport-https ca-certificates gnupg2 apt-key adv \ --keyserver hkp://ha.pool.sks-keyservers.net:80 \ --recv-keys 58118E89F3A912897C070ADBF76221572C52609D echo 'deb https://apt.dockerproject.org/repo debian-stretch main' \ > /etc/apt/sources.list.d/docker.list apt-get update apt-get install docker-engine systemctl daemon-reload systemctl start docker
A few words of caution:
apt-keycommand might be outdated, please check the official documentation
- make sure to change
debian-stretchto match your debian version
To avoid running Docker with
sudo, you might want to create a group:
groupadd docker gpasswd -a user docker systemctl restart docker
Please note that:
- you should replace
userwith your username
- log out and log in is required after adding a group
Once Docker is up and running, it's time to start to play with it. Please follow the documentation here:
After you finish playing with Docker, you might want to cleanup behind you.
There's a good summary of the commands you might need here:
Creating Docker Images for Your Build
Basically, what I need is a minimal environment that allows to build my C application.
Docker Hub provides bare, minimal images of some major distros, like Debian, Ubuntu, Fedora and such. These images are a base layer, I just need to top it up with the packages I need:
- the C compilation toolchain (gcc/clang, make, autotools and so on)
- the libraries required by my application (glib, gstreamer, gtk+, ...)
Once my images are ready, tested, validated, I must create a Dockerfile: this file contain the steps needed to build the image.
Dockerfiles are easy to create, easy to understand, but that shouldn't prevent you frome reading the "best practices" section from the Docker documentation: https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/
Dockerfiles are needed since we will later on create "automated build" on Docker Hub. Please focus here, otherwise it might be confusing. I'm talking about automated build of the docker images, in which I will later on run automated builds of my application. You get it ;)?
Please don't get lost in all this automation and read on.
Storing Dockerfiles on GitHub
Now that you have your Dockerfiles ready, you need to store them in a git repository, on GitHub or Bitbucket (both are supported by Docker Hub).
Please be logical in your naming. My application repository is
so I created a repository named
goodvibes-docker. As simple as that.
As an example, I invite you to have a look there:
As you can see, I just created a sub-directory for each docker image I want to build, containing a single Dockerfile.
If you're a bit lost, don't worry, everything will become clear in the next step, where Docker Hub kicks in.
Creating a Docker Hub Account
Now that your Dockerfiles are ready and hosted on the net, the next step is to have a Docker Hub account, where you will store your docker images.
So do that. Create your account on Docker Hub, then link it with your GitHub account. Once it's done, you now want to create Automated Builds. That is, you want your images to be automatically generated according to your Dockerfiles.
It might be time for you to read some more documentation now:
The procedure is quite straightforward, there's little to explain. By default, the automatic build settings expect your Dockerfile to be at the root of your git repository. If it's not the case (and it's not in my case), just click on the Customize link when you can, and set the location of your Dockerfile.
Then repeat the procedure for each docker image you intend to build, so that in the end you have an automated build for each Dockerfile.
It seemed to me that the first build had to be triggered manually. Just check that in the Build Details section. If nothing's happening, just go to Build Settings and trigger a build manually.
Now, you want Docker Hub to be smart, and automatically rebuild your images when it's needed. When is it needed by the way?
- when the base image (Debian, Ubuntu and so on) is updated
- when you modify your Dockerfiles
To handle the first situation, it all happens on Docker Hub. Just enter the Build Settings section of your automated build, then the Repository Links, and add a link to the base image. Done.
For the second situation, it happens on GitHub. You will have to manually link the Docker service to your Github project. In order to do that, open GitHub, navigate to the Settings part of the repository containing your Dockerfiles, then choose Integration & services, and there you add Docker.
When it's done, Docker should automatically be notified when your Dockerfiles are modified, and rebuild your images accordingly.
As an example, you can have a look at my docker images here:
Create a Shippable Account
We're almost there. We have our docker images hosted and ready, automatically built, easy to update, it's all wonderful. What's next? Setup the continuous integration service, which is Shippable in this tutorial.
Creating an account there is straightforward, nothing special.
Then, link it with your GitHub account, and then enable the projects you want to monitor. You might need a few clicks on the GitHub side as well, or maybe not, I don't remember. Anyway, this is simple, you won't get lost.
The very last step of this tutorial is coming: you now need to add a
file to your project, where you describe the steps for the build. I strongly advise
you, one more time, to check the doc:
Shippable provides a default build image, Ubuntu something. If you don't want to use it, no problem. You can instead indicate the custom images you want to run. If these images are hosted on Docker Hub, it's very straightforward.
As an example, you can have a look at my
So let me draw you the whole picture again. I've you've been reading seriously up to now, you already have the picture in your mind:
---------------------- | dockerfiles @ github | ---------------------- \ source for \ --------------------- | images @ docker hub | --------------------- / pull from / ------------------- | build @ shippable | ------------------- / trigger / ------------------ | project @ github | ------------------
Easy to understand once all the pieces of the puzzle fit.
I hope you enjoyed the reading, now it's time to dock and ship!