CI/CD: Using GitLab + Docker + Ansible

How we built an efficient CI/CD pipeline

September 20, 2018

2021 Update: We now have shared how we use GitLab + Ansible to deploy to Docker Swarm.

At CALLR, we have been using GitLab and Ansible internally for quite a time. In this post, I’ll explain how we use both tools together with Docker to build an efficient CI/CD pipeline.

Quick reminder

GitLab is a web-based Git repository manager with CI/CD pipeline features.
Ansible is an automation tool for provisioning, configuration management, and application deployment.
Docker is a program (and much more) that runs containers.

Workflow

CI/CD Pipeline using GitLab — Original image: GitLab CI

We use GitLab CI at the center of our CI/CD system.

On the CI/CD pipeline, GitLab CI uses runners to run jobs (build, tests, deployment…). Runners can be docker containers, virtual machines, local shells, among others options (see executors).

We use Ansible (independently of GitLab) to configure and deploy most of the components of our platform.

It was only logical to use both tools for better deployments.

The CI pipeline: GitLab + Docker

For most projects, the first tasks are:

run tests (unit tests, syntax checks)
build

Those jobs are run on docker runners (each run gets a clean environment).

In the following examples, we will show you how we do it for a PHP project.

Syntax check

For the syntax check, we use PHP Parallel Lint which permits fast linting.

Syntax check:
  image: gitlab.callr.tech:4567/devops/docker-image-api:0.9.3
  stage: test
  script:
    - parallel-lint -j 20 --exclude ./vendor ./

Unit tests

For the unit tests, we include an “ssh” job which starts ssh-agent and loads a private key from a GitLab secret variable. This is needed because composer needs to load dependencies that are hosted on our private GitLab.


# the "ssh" job will start an ssh-agent 
# and load a private key from a gitlab secret variable
# this is needed to fetch dependencies that are 
# hosted on private git repositories
.ssh: &ssh
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$GITLAB_SSH_PRIVATE_KEY")
  - mkdir -p ~/.ssh
  - echo "$GITLAB_SSH_HOSTKEYS" > ~/.ssh/known_hosts

Unit tests:
  image: gitlab.callr.tech:4567/devops/docker-image-api:0.9.3
  variables:
    PHPUNIT_FLAGS: --exclude-group no-mocks
    COMPOSER_FLAGS: --prefer-dist -a
  stage: test
  before_script: *ssh
  script:
    - make vendor
    - make test

First, we run make vendor to run composer install. Then we run make test which starts phpunit with some flags.

Here is the corresponding part of the makefile:

COMPOSER_CMD=composer
PHPUNIT_CMD=bin/phpunit

PHP_FLAGS?=
PHPUNIT_FLAGS?=
COMPOSER_FLAGS?=

vendor:
	@$(COMPOSER_CMD) install $(COMPOSER_FLAGS)
	@$(COMPOSER_CMD) services-map
.PHONY: vendor

test: $(PHPUNIT_CMD)
	@php $(PHP_FLAGS) $(PHPUNIT_CMD) --colors $(PHPUNIT_FLAGS)
.PHONY: test

For our unit tests, we use the flag --exclude-group no-mocks to skip tests that would not work in a CI environment.

Build

Build:
  image: gitlab.callr.tech:4567/devops/docker-image-api:0.9.3
  artifacts:
    expire_in: 1 month
    paths:
      - build/
  stage: build
  variables:
    COMPOSER_FLAGS: --prefer-dist -a --no-dev
  before_script: *ssh
  script:
    - make build

For the build, we also include the “ssh” job, and then make build will run composer install with --prefer-dist -a --no-dev.

--prefer-dist: Composer will install from dist if possible. This can speed up installs substantially on build servers.
-a: Autoload classes from the classmap only. Implicitly enables: Convert PSR-0/4 autoloading to classmap to get a faster autoloader. This is recommended especially for production, but can take a bit of time to run so it is currently not done by default.
--no-dev: Skip installing packages listed in require-dev.

Then, we create a clean build subdirectory in which we copy the required files to run the PHP program.

build: vendor
	@rm -rf build/
	@mkdir build
	@cp -r htdocs system vendor Makefile bin composer.* build
	@git log -1 --pretty=%B > build/.gitlastcommitmsg
.PHONY: build

Finally, we write the last commit message in a hidden file .gitlastcommitmsg. You will see why later :)

The build directory is saved as an artifact for the following jobs.

The CD pipeline: GitLab + Ansible

For deployments with Ansible, we use a shared job like this:

.deploy: &deploy
  - >
    cd /var/ansible &&
    sudo -E ansible-playbook ci_api.yml
    --diff
    --private-key="/var/ansible/ssh_keys/gitlab/gitlab-ci"
    -i "inventories/$ANSIBLE_INVENTORY"
    -l "$ANSIBLE_SUBSET"
    -e BUILD_PATH="\"$CI_PROJECT_DIR/build/\""
    -e CI_COMMIT_REF_NAME="\"$CI_COMMIT_REF_NAME\""
    -e CI_COMMIT_SHA="\"$CI_COMMIT_SHA\""
    -e CI_ENVIRONMENT_NAME="\"$CI_ENVIRONMENT_NAME\""
    -e CI_PIPELINE_ID="\"$CI_PIPELINE_ID\""
    -e CI_PROJECT_URL="\"$CI_PROJECT_URL\""
    -e CI_RUNNER_DESCRIPTION="\"$CI_RUNNER_DESCRIPTION\""
    -e CI_SERVER_NAME="\"$CI_SERVER_NAME\""
    -e GITLAB_USER_EMAIL="\"$GITLAB_USER_EMAIL\""
    -e GITLAB_USER_ID="\"$GITLAB_USER_ID\""
    -e GIT_LAST_COMMIT_MSG="\"`cat $CI_PROJECT_DIR/build/.gitlastcommitmsg`\""

This will run an Ansible playbook, on inventory $ANSIBLE_INVENTORY with a subset $ANSIBLE_SUBSET, along with a few variables.

The actual deployment jobs are:

Deploy to staging:
  stage: deploy
  script: *deploy
  when: manual
  environment:
    name: staging
  tags:
    - deployment
  variables:
    GIT_STRATEGY: none
    ANSIBLE_INVENTORY: staging
    ANSIBLE_SUBSET: all

Deploy to production:
  stage: deploy
  script: *deploy
  when: manual
  allow_failure: false
  only:
    - master
  environment:
    name: production
  tags:
    - deployment
  variables:
    GIT_STRATEGY: none
    ANSIBLE_INVENTORY: prod
    ANSIBLE_SUBSET: all

GIT_STRATEGY is set to none because we do not need the source code anymore, the build we have done in the previous task is passed as an artifact.

We use the deployment tag here to select specific runners that are shell runners, installed on servers along with Ansible.

GitLab CI will place the artifact (the build) on the server on directory $CI_PROJECT_DIR.

In our example, the .deploy job runs the ci_api.yml playbook, and passes a few GitLab CI variables to Ansible (via -e).

Ansible

Here is the deployment role we use for PHP projects that are not dockerized.

First, some checks:

# Checks

- name: Failing if BUILD_PATH is not defined
  fail:
    msg: "Missing BUILD_PATH variable"
  run_once: true
  delegate_to: localhost
  when: BUILD_PATH is undefined or BUILD_PATH|trim == ''

- name: Failing if CI_COMMIT_SHA is not defined
  fail:
    msg: "Missing CI_COMMIT_SHA variable"
  run_once: true
  delegate_to: localhost
  when: CI_COMMIT_SHA is undefined or CI_COMMIT_SHA|trim == ''

- name: Failing if GITLAB_USER_ID is not defined
  fail:
    msg: "Missing GITLAB_USER_ID variable"
  run_once: true
  delegate_to: localhost
  when: GITLAB_USER_ID is undefined or GITLAB_USER_ID|trim == ''

- name: Failing if GIT_LAST_COMMIT_MSG is not defined
  fail:
    msg: "Missing GIT_LAST_COMMIT_MSG variable"
  run_once: true
  delegate_to: localhost
  when: GIT_LAST_COMMIT_MSG is undefined or GIT_LAST_COMMIT_MSG|trim == ''

- name: Check if API code has been pushed here
  stat: path={{ BUILD_PATH }}
  register: deploy_path
  run_once: true
  delegate_to: localhost
  check_mode: no

- name: Failing if API code has not been pushed
  fail:
    msg: "Code has not been pushed to deploy server"
  run_once: true
  delegate_to: localhost
  when: deploy_path.stat.exists == false

We make sure that some variables are passed and are not empty, and that the BUILD_PATH exists.

Now, let’s deploy the code.

The code is deployed on /var/www/api/$CI_COMMIT_SHA: each deployment is on its own directory, we move a current symlink to point to the latest deployment.

The goal is to be as close as possible to an “atomic” switch.

pa2-api10 api # l
total 8.0K
drwxr-xr-x 7 root root 4.0K Sep 11 15:36 30e10af2b6a2a7c8f1c0ee13c82690a80531be66/
lrwxrwxrwx 1 root root   53 Sep 14 09:23 current -> /var/www/api/fd33d7ca0f1cabeca1267b79b8ce70da1a0258ed/
drwxr-xr-x 7 root root 4.0K Sep 14 09:23 fd33d7ca0f1cabeca1267b79b8ce70da1a0258ed/

Here is how we do it with Ansible:

- name: Check if API exists
  stat: path=/var/www/api/{{ CI_COMMIT_SHA }}
  register: api_expected_commit
  check_mode: no

- name: Copy build
  synchronize:
    src: "{{ BUILD_PATH }}/"
    dest: "/var/www/api/{{ CI_COMMIT_SHA }}/"
    archive: yes
    delete: yes
    owner: no # don't preserve owner
    group: no # don't preserve group
  diff: no
  when: api_expected_commit.stat.exists == false

- name: Remember current /current link
  stat:
    path: "/var/www/api/current"
  register: api_before_current_link

- name: Link /current to latest build
  file:
    src: "/var/www/api/{{ CI_COMMIT_SHA }}"
    dest: "/var/www/api/current"
    state: link

At this point, /var/www/api/current points to /var/www/api/{{ CI_COMMIT_SHA }} which is the build we just deployed.

But we are not done yet. Because this is a PHP project and we use opcache, we must clear it:

- name: Clear opcache
  uri:
    force_basic_auth: yes
    body: >
            {"jsonrpc":"2.0","id":42,"method":"system.opcache_reset","params":[]}
    headers:
      Host: api.callr.com
      Content-Type: application/json-rpc
    method: POST
    url: http://localhost/
    user: "{{ config.credentials.api.scripts.user }}"
    password: "{{ config.credentials.api.scripts.pass }}"

To clear the opcache, we must call the PHP function opcache_reset(), but it has to be run by one of the PHP-FPM process (the PHP CLI uses a distinct opcache).

Our solution here is to make a JSON-RPC call on our code to call this function.

At this point, the code is deployed, the /current symlink points to the code, and the opcache is cleared.

Time to announce the deployment!

- name: Send slack notification
  slack:
    token: "{{ slack_token }}"
    channel: "#ops-prod"
    attachments:
      - text: "API deployed from Gitlab CI\n"
        color: "#39932A"
        fields:
          - title: "Environment"
            value: "{{ CI_ENVIRONMENT_NAME }}"
            short: true
          - title: "Git branch/tag"
            value: "{{ CI_COMMIT_REF_NAME }}"
            short: true
          - title: "Hosts"
            value: "{{ ansible_play_hosts|join(', ') }}"
            short: false
          - title: "Git commit hash"
            value: "{{ CI_COMMIT_SHA }}"
            short: false
          - title: "Deploying user"
            value: "{{ GITLAB_USER_EMAIL }} (id:{{ GITLAB_USER_ID }})"
            short: true
          - title: "CI Runner"
            value: "{{ CI_RUNNER_DESCRIPTION }}"
            short: true
          - title: "Last commit message"
            value: "{{ GIT_LAST_COMMIT_MSG }}"
            short: false
          - title: "Pipeline"
            value: "{{ CI_PROJECT_URL }}/pipelines/{{ CI_PIPELINE_ID }}"
            short: false
  delegate_to: localhost
  run_once: true

We use the slack module to send a notification on a specific channel.

On Slack, it looks like this:

Slack message from Ansible — Slack message sent by Ansible

This is very useful for everyone to keep track of what is being deployed.

To finish, we delete old versions, and keep only the previous deployment.

- name: Get deprecated API versions (older than 5 minutes)
  shell: >
    find /var/www/api/ -maxdepth 1 -mindepth 1 -mmin +5
    | grep -v '/var/www/api/current'
    | grep -v '/var/www/api/{{ api_before_current_link.stat.lnk_target | basename }}'
    | grep -v `readlink /var/www/api/current`    
  register: api_deprecated_versions
  ignore_errors: yes # because rc == 1 if no file is found
  check_mode: no

- name: API versions to delete
  debug: msg="{{ api_deprecated_versions.stdout.split('\n') }}"
  check_mode: no

- name: Delete deprecated API versions
  file: path="{{ item }}" state=absent
  with_items: "{{ api_deprecated_versions.stdout.split('\n') }}"
  diff: no
  when: api_deprecated_versions.rc == 0

This way, /var/www/api is always clean: two versions, and current symlink.

On GitLab, you get the full output of Ansible:

Running with gitlab-runner 10.1.0 (c1ecf97f)
  on th2-deploy01 (00ae9e6e)
Using Shell executor...
Running on th2-deploy01.inf.callr.tech...
Skipping Git repository setup
Skippping Git checkout
Skipping Git submodules setup
Downloading artifacts for Build (13018)...
Downloading artifacts from coordinator... ok        id=13018 responseStatus=200 OK token=o_zscJ7t
 
$ cd /var/ansible && sudo -E ansible-playbook ci_api.yml --diff --private-key="/var/ansible/ssh_keys/gitlab/gitlab-ci" -i "inventories/$ANSIBLE_INVENTORY" -l "$ANSIBLE_SUBSET" -e BUILD_PATH="\"$CI_PROJECT_DIR/build/\"" -e CI_COMMIT_REF_NAME="\"$CI_COMMIT_REF_NAME\"" -e CI_COMMIT_SHA="\"$CI_COMMIT_SHA\"" -e CI_ENVIRONMENT_NAME="\"$CI_ENVIRONMENT_NAME\"" -e CI_PIPELINE_ID="\"$CI_PIPELINE_ID\"" -e CI_PROJECT_URL="\"$CI_PROJECT_URL\"" -e CI_RUNNER_DESCRIPTION="\"$CI_RUNNER_DESCRIPTION\"" -e CI_SERVER_NAME="\"$CI_SERVER_NAME\"" -e GITLAB_USER_EMAIL="\"$GITLAB_USER_EMAIL\"" -e GITLAB_USER_ID="\"$GITLAB_USER_ID\"" -e GIT_LAST_COMMIT_MSG="\"`cat $CI_PROJECT_DIR/build/.gitlastcommitmsg`\""

PLAY [Deploy API Code] *********************************************************

[...]

PLAY RECAP *********************************************************************
[...]
th2-apicore20              : ok=12   changed=5    unreachable=0    failed=0   
th2-apicore21              : ok=12   changed=5    unreachable=0    failed=0   
th2-apicore22              : ok=12   changed=5    unreachable=0    failed=0   
th2-apicore23              : ok=12   changed=5    unreachable=0    failed=0   

Job succeeded

That’s it!