Skip to content

GitLab Pipelines: How They Work and How to Create Them

GitLab CI/CD (Continuous Integration/Continuous Deployment) uses pipelines to automate the steps of your software development lifecycle, such as building, testing, and deploying your code.

What is a Pipeline?

A pipeline is the top-level component of CI/CD. It represents the entire process from code commit to deployment.

  • Pipelines are configured in a file called .gitlab-ci.yml at the root of your project repository.
  • They are triggered by events like code pushes, merge requests, schedules, or manually.
  • A pipeline consists of one or more stages.
  • Each stage consists of one or more jobs.

Running Pipelines

Pipelines are hosted on runners, or servers that execute the jobs. These are hosted on our proxmox servers and reference both harbor and the container bakery. They are called via tags in the .gitlab-ci.yml file. We use the dind tag which means Docker-IN-Docker and allows us to run docker containers

Stages and Jobs

  • Stages: Stages define the order of execution. All jobs within a stage run in parallel. The next stage only starts if all jobs in the previous stage complete successfully. Common stages include build, test, deploy.
  • Jobs: Jobs are the fundamental building blocks of a pipeline. They define what to do. Each job runs independently and executes scripts defined in the .gitlab-ci.yml file. Jobs are executed by runners.

How Pipelines Work

  1. Trigger: A pipeline is triggered by an event (ex. a git push).
  2. Read Configuration: GitLab reads the .gitlab-ci.yml file.
  3. Create Pipeline: GitLab creates a pipeline instance based on the configuration.
  4. Execute Stages: Stages are executed sequentially.
  5. Execute Jobs: Within each stage, jobs are sent to available runners for execution. Jobs in the same stage run concurrently.
  6. Job Execution: A runner picks up a job, checks out the code, and runs the defined scripts in an isolated environment.
  7. Status Update: The runner sends the job's status (success, failure) back to GitLab.
  8. Pipeline Completion: The pipeline completes when all stages and jobs have finished. The overall pipeline status depends on the status of its jobs.

Creating a Pipeline (.gitlab-ci.yml)

The .gitlab-ci.yml file uses YAML syntax to define your pipeline. Here is the pipeline that is used to build this website:

# Define default settings for all jobs unless overridden
default:
  # Specify the Docker image to use for the job's environment
  # This image is hosted on our harbor registry (192.168.2.65) and contains mkdocs
  image: 192.168.2.65/gitlab/mkdocs:latest

  # Specify tags to select specific runners capable of running this job
  # 'dind' means 'Docker in Docker', meaning the runner can build/run Docker containers
  tags:
    - dind

# Define variables that can be used throughout the pipeline
variables:
  # Set the Docker storage driver to overlay2 (good for performance)
  DOCKER_DRIVER: overlay2

# Define a job named 'test'
test:
  # Assign this job to the 'test' stage
  # If no stages are explicitly defined, GitLab uses default stages: .pre, build, test, deploy, .post
  stage: test
  # Define the commands to execute for this job
  script:
  - mkdocs build --verbose --site-dir test   # Build the mkdocs project, outputting verbose logs and saving to a 'test' directory
  #  - mkdocs build --strict --verbose --site-dir test
  # Define artifacts to be saved from this job's execution
  artifacts:
    paths:
    - test # Saves the 'test' directory containing the built documentation
  rules:
    # This rule means the job runs IF the current branch name ($CI_COMMIT_REF_NAME)
    # is NOT the project's default branch ($CI_DEFAULT_BRANCH, usually 'main' or 'master')
    - if: $CI_COMMIT_REF_NAME != $CI_DEFAULT_BRANCH

# Define a job named 'pages' (special job name for GitLab Pages deployment)
pages:
  stage: deploy
  # Define the commands to execute for this job
  script:
    # Build the mkdocs project with verbose logs (default site-dir is 'site')
    - mkdocs build --verbose
  # Define artifacts to be saved from this job's execution
  artifacts:
    # Specify the paths of files/directories to save
    paths:
      # GitLab Pages automatically serves content from a directory named 'public'
      - public # The mkdocs build output needs to be moved/configured to be in 'public' for Pages
  # Define rules to control when this job runs
  rules:
    # This rule means the job runs IF the current branch name ($CI_COMMIT_REF_REF_NAME - typo, should be $CI_COMMIT_REF_NAME)
    # IS the project's default branch ($CI_DEFAULT_BRANCH)
    # This ensures deployment only happens on the main branch
    - if: $CI_COMMIT_REF_REF_NAME == $CI_DEFAULT_BRANCH # Deploy only on the main branch

Parameters in .gitlab-ci.yml:

  • stages: Defines the sequence of stages.
  • job_name: Defines a specific job.
  • stage: Assigns a job to a stage.
  • script: Defines the commands executed by the job.
  • image: (Commonly used) Specifies the Docker image to use for the job.
  • tags: Used to select specific runners.
  • only/rules: Control when a job runs based on branches, tags, etc.
  • artifacts: Define files to be saved from the job's execution.
  • dependencies: Specify jobs that must complete before this job starts (within the same stage).

By creating and pushing this .gitlab-ci.yml file to your repository, GitLab will automatically detect it and start running pipelines based on your configuration.