Build container images in GitLab CI (iptables-legacy at the rescue)

It's 2023 and these days, building a container image in a CI pipeline should be straightforward. So let's try.

EDIT: All this blog post is now obsolete! In May 2024 GitLab landed a major infra upgrade. At the time of this writing, the ephemeral VMs used in shared runners are now running Google COS-105 with kernel 1.5.154+, it has iptables-nft support, and the commands podman build (or buildah build) work out of the box (at least for me :).

For this blog post we'll focus on GitLab SaaS only, that is, gitlab.com, as it's what I use for work and for personal projects.

To get started, we just need two files in our Git repository:

• a Containerfile (or Dockerfile if you prefer to name it this way) that defines how to build a container image.
• a .gitlab-ci.yml file that defines what the CI should do. In the example below, we want to build a container image and push it to the GitLab registry associated with the GitLab repo.

Here is our Git tree:

$ ls -A
Containerfile  .git  .gitlab-ci.yml

$ cat Containerfile 
FROM debian:stable
RUN  apt-get update
CMD  echo hello world

$ cat .gitlab-ci.yml 
build-container-image:
  stage: build
  image: debian:testing
  before_script:
    - apt-get update
    - apt-get install -y buildah ca-certificates
  script:
    - buildah build -t $CI_REGISTRY_IMAGE .
    - buildah login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
    - buildah push $CI_REGISTRY_IMAGE

A few remarks:

• We use buildah, but we could have used podman.
• However we don't use docker, because its client/server design makes it cumbersome to use in a CI environment: it requires a separate container to run the Docker daemon, plus setting the DOCKER_HOST variable. Why bother?

Now let's push that. Does the CI pass? No, of course, otherwise I wouldn't be writing this blog post ;)

The CI fails at the buildah build command, with a rather cryptic error:

$ buildah build --tag $CI_REGISTRY_IMAGE .
[...]
STEP 2/3: RUN  apt-get update
error running container: did not get container start message from parent: EOF
Error: building at STEP "RUN apt-get update": netavark: code: 4, msg: iptables v1.8.8 (nf_tables): Could not fetch rule set generation id: Invalid argument

The hint here is nf_tables... Back in July 2021, GitLab did a major update of their shared runners infrastructure, and broke nftables support in the process, as it seems. So we have to use iptables instead.

Let's fix our .gitlab-ci.yml, which now looks like that:

$ cat .gitlab-ci.yml 
build-container-image:
  stage: build
  image: debian:testing
  before_script:
    - apt-get update
    - apt-get install -y buildah ca-certificates
    - |
      # Switch to iptables legacy, as GitLab CI doesn't support nftables.
      apt-get install -y --no-install-recommends iptables
      update-alternatives --set iptables /usr/sbin/iptables-legacy
  script:
    - buildah build -t $CI_REGISTRY_IMAGE .
    - buildah login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
    - buildah push $CI_REGISTRY_IMAGE

And push again. Does that work? Yes!

If you're interested in this issue, feel free to fork https://gitlab.com/arnaudr/gitlab-build-container-image and try it by yourself.

It's been more than a year since this change, and I'm surprised that I didn't find much about it on the Internet, neither mentions of the issue, nor of a workaround. Maybe nobody builds container images in GitLab CI, or maybe they do it another way, I don't know. In any case, now it's documented in this blog, hopefully some will find it useful.

Happy 2023!