Speed up gitlab-ci using kubernetes and Docker

Gitlab-ci is a very usefull tool, that allow automate a part of the development process. It allow to run automaticly linters, tests, deployment, usw. Because of this, many companies are using it. The problem is, that even if tests are fast, it require many minutes to run. Minutes, in what the developer can easly lose his concentration or worser, distracting other developers by talking (inecesarry) to them (making lose theyer concentration). So, even a small speed improvement on the gitlab-ci would  increase productivity. 

In this tutorial, we will see, how to install gitlab runner on kubernetes and how this can help us to improve the speed of our pipelines.

Install gitlab runner on kubernetes

As all the things in the software world, there exist more than one way to do it. One way to install it, would be to use the kubernetes integration of GitLab, but I would recomend to do it yourself, using the helm packet manager. The reason, why I prefer to do it manually, is that is that GitLab does not show logs, when the installion fails. Also, doing it manually, allow you to costumize/configure the gitlab runner, instead using it out of the box.

helm repo add gitlab https://charts.gitlab.io
helm install gitlab-runner --name gitlab-runner \
  --namespace gitlab \
  --set gitlabUrl=https://gitlab.com/  \     # change this, if necessary 
  --set runnerRegistrationToken= \           # can be found on settings -> CI/CD -> runners
  --set runners.locked=false \               # Allow to use the runner in multiple projects
  --set concurrent=5                         # How many paralel instances can be runned

Notice: see values.yaml for more options.

If you want to build docker images on the gitlab-ci, then you have to set privileged to true (–set runners.privileged=true). If you do it, pleas do not forget to use overlay2. 

Notice: if you see that your pipelines fail, after moving to kubernetes, change the host name to localhost. E.g mysql -> localhost

Notice: Running docker images on privileged modus, allow privilegion escalation. So, by carryfull, who can run the pipelines or install the runner on a seperat cluster.

Noctice: If you are not using overlay2, then the docker build process can become very slow.

Use a custom docker image

Normally, we run the gitlab-ci by using a docker image and installing on it all what we need to run our tests, linters, deployment, usw. The packet, that are installed on when we run the pipeline, need time to been downloaded, extracted and installed (or compiled). This can by very slow. So, if we use a custom docker image, where all our dependecies are installed, then the time consuming installing step colud by avoided. For example, a Dockerfile for rails, could look as the following:

# Dockerfile
FROM ruby:2.5.1-alpine

RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.5/main' >> /etc/apk/repositories && \
  apk add --no-cache --update \
  build-base zlib-dev libxml2-dev libxslt-dev tzdata \
  mariadb-dev \
  mariadb-client \
  nodejs yarn git \
  imagemagick-dev=6.9.6.8-r1 \
  imagemagick=6.9.6.8-r1

COPY Gemfile Gemfile.lock ./
RUN bundle

We can build the image by executing:

docker build -t cache-image .

Now, we have to push the image to some repository. One posibility would by the oficial DockerHub but I normally prefer to submit to the registry of our GitLab project. The registry can by enabled under Settings -> General -> Permissions -> Container registry. Then, on the left asside, a new menu point, named Registry, should appear. Any on the bootom should appear diferent posibilies about how the image name could by.  

docker build -t registry.gitlab.com/<yourproject> . # Adjust the image name as described above
docker login registry.gitlb.com                     # If you are useing a self hosting versin, then adapt the url. It should appear on the GitLab registry page 
docker push registry.gitlab.com/<yourproject>

By last, you have to use your image on the gitlab-ci. For rspec (ruby test), it could look as following:

# gitlab-ci.yml

test:
  image: registry.gitlab.com/<yourproject>
  script:
    - bundle install
    - bundle exec rspec

Deployment (using docker)

If you use kubernetes or an other deployment system, based on docker containers, then maybe you are allready using the gitlab-ci to build your docker images and to deploy them. My experience has been, that the build step can by very slow (sometimes requiring 1 hour or more).  Afortunatly, there exist many ways to speed up it significantly. One of them is to use the --cache-from flag, that docker build offers. This flag is used to specify a older builded images, so that if the files required for some steps have not changed, the step from the old image is used, insted of executing it again. 

docker build --cache-from registry.gitlab.com/<yourproject>:latest -t registry.gitlab.com/<yourproject>:<version> .

To maximize the usage of cached stepts, we have to put the most unlike to change steps, on the beginning of the Dcokerfie. For a rails app, this colud looking as the following:

# Dockerfile

FROM ruby:2.5.1-alpine

# System packate, normally, are not changed very frecuently
RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.5/main' >> /etc/apk/repositories && \
  apk add --no-cache --update \
  build-base zlib-dev libxml2-dev libxslt-dev tzdata \
  postgresql-dev \
  nodejs yarn git \
  imagemagick-dev=6.9.6.8-r1 \
  imagemagick=6.9.6.8-r1

WORKDIR /app

# Gems,ruby packages, also do not change very frecuently
COPY Gemfile Gemfile.lock  ./
RUN bundle install --frozen --jobs $(nproc)

# Npm packages, also do not change very frecuently
COPY yarn.lock package.json ./
RUN yarn install

COPY . ./

RUN bundle exec rails assets:precompile DATABASE_URL=postgres://postgres:postgres@postgres:5432/development


CMD ["bundle", "exec", "puma"]

In the dockerfile above, there is still a big performance problem: The asset precompile step is still very slow. Many framworks like rails, solve this by only compiling the files that have changed but this is a problem here because there does not exist perevios compiled files. A way to solve this, is using  the docker build feature named “multi step build”. This feature allow us to copy files from one docker image to a other one. So, we could copy the asset files from a older image to the new one. This is also very usefull, if we are working with a compiled language like Elixir, Java or C++, where recompiling the enterly programm can be very expensive. Altough, I only use is with precompiling javascript assets and java because on Elixir and c++ it is a litle bit error prone (maybe offering a optional clean build step in the gitlab-ci, would solve the problem). 

# Dockerfile
FROM registry.gitlab.com/<yourproject>:latest

FROM ruby:2.5.1-alpine

# System packate, normally, are not changed very frecuently
RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.5/main' >> /etc/apk/repositories && \
  apk add --no-cache --update \
  build-base zlib-dev libxml2-dev libxslt-dev tzdata \
  postgresql-dev \
  nodejs yarn git \
  imagemagick-dev=6.9.6.8-r1 \
  imagemagick=6.9.6.8-r1

WORKDIR /app

# Gems,ruby packages, also do not change very frecuently
COPY Gemfile Gemfile.lock  ./
RUN bundle install --frozen --jobs $(nproc)

# Npm packages, also do not change very frecuently
COPY yarn.lock package.json ./
RUN yarn install

COPY . ./
COPY --from=0 /app/public/ /app/public/

RUN bundle exec rails assets:precompile DATABASE_URL=postgres://postgres:postgres@postgres:5432/development


CMD ["bundle", "exec", "puma"]

Finally, we have to write/adjust our gitlab-ci.yml file. For a rails project, it could look as the following:

# gitlab-ci.yml

stages:
  - build
  - test
  - deploy

variables:
  GIT_STRATEGY: none

before_script:
  - cd /app

build:
  image: docker:latest
  stage: build
  variables:
    GIT_STRATEGY: clone
    DOCKER_DRIVER: overlay2
  services:
    - docker:dind
  before_script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
  script:
    - docker pull $CI_REGISTRY_IMAGE
    - docker build --build-arg REVISION=$CI_COMMIT_SHA --cache-from $CI_REGISTRY_IMAGE -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

test:
  image: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  variables:
    POSTGRES_DB: database
    DATABASE_URL: postgres://postgres:postgres@postgres/$POSTGRES_DB
    RAILS_ENV: test
  services:
    - postgres:11-alpine
  script
    - cd app
    - bundle exec rspec
tag latest:
  stage: deploy
  image: docker
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker pull $RELEASE_IMAGE
    - docker tag $RELEASE_IMAGE $CI_REGISTRY_IMAGE
    - docker push $CI_REGISTRY_IMAGE
  only:
    - master

###################################################################
# This step use the kubernetes packete manager name helm. If you  #
# use a other tool or even only ssh, then adapt this step          #
###################################################################
deploy production:
  stage: deploy
  image: dtzar/helm-kubectl
  environment:
    name: production
    url: <my url>
  only:
    - master
  script:
    - mkdir ~/.kube
    - echo $KUBE_CONFIG | base64 -d > ~/.kube/config # set under settings -> CI/CD -> Enviroment variables
    - helm init --upgrade
    - helm upgrade <deployment name> chart
        --install
        --set image.tag=$CI_COMMIT_SHA