Designing Reusable GitLab CI Pipelines with Templates at Scale

If you’ve ever maintained multiple repositories, you’ve probably duplicated the same CI logic over and over again. A small change in the CI pipeline requirements suddenly becomes a tedious task across dozens of projects. Miss an update in even one repository, and its pipeline begins drifting away from the desired state.

At my workplace, we previously used self-hosted Forgejo to host our codebases. While it worked well for source control, our CI logic was duplicated across repositories, making it increasingly difficult to maintain consistency. As we began migrating to GitLab for its more advanced CI/CD capabilities, we started leveraging GitLab CI templates to address this problem.

The core idea is simple: CI should be reusable, consistent, and easy to maintain. GitLab CI templates enable teams to centralize CI logic outside individual repositories and reference it from projects through reusable templates.

This approach provides a significant boost to maintainability and standardization. CI logic is now centralized and enforced across the organization, ensuring every service follows the same quality and deployment standards. Teams can confidently ship changes knowing that the pipeline enforcing quality, security, and build checks remains consistent across all repositories.

While implementing this system, I encountered several challenges, particularly around documentation gaps and practical implementation details. In this blog, I’ll walk through how we approached the problem, how our self-hosted GitLab is integrated into our infrastructure, and how we orchestrate CI pipelines across 20+ microservices. I’ll also cover how we migrated these services from Forgejo to GitLab without causing development downtime.

Before diving deeper into the individual components, the following diagram provides a high-level overview of how our GitLab infrastructure and CI execution environment are organized.

flowchart LR

    Dev[Developer Push / Merge Request]

    GitLab[GitLab Instance<br>Production AWS Account]

    Runner[GitLab Runner Manager<br>UAT AWS Account]

    EKS[EKS Cluster]

    Pod[Ephemeral CI Job Pod]

    S3[S3 Bucket<br>Pipeline Logs]

    Dev --> GitLab
    GitLab -->|Pipeline Trigger| Runner
    Runner -->|Schedule Job| EKS
    EKS --> Pod
    Pod -->|Upload Logs| S3
    Pod -->|Job Status| GitLab

This separation ensures that the GitLab control plane (repository hosting and pipeline orchestration) remains isolated from the CI execution infrastructure where untrusted build workloads run.

The GitLab Architecture

For security and infrastructure isolation, our GitLab deployment separates the code hosting platform and CI execution infrastructure into different AWS accounts.

The GitLab instance itself runs in the Production AWS account, which handles all developer-facing operations such as repository hosting, merge requests, comments, and pipeline orchestration.

The GitLab CI Runners operate in a separate UAT AWS account. This isolation ensures that CI workloads which execute arbitrary code from repositories do not run inside the same account that hosts the core GitLab service.

The CI runners are deployed inside an Amazon EKS cluster, where they run as Kubernetes workloads.

GitLab Runners follow an ephemeral execution model. They continuously poll the GitLab instance for new job requests. When a pipeline is triggered, the runner manager receives the job and schedules it for execution within the Kubernetes cluster.

Instead of using persistent build agents, each CI job runs inside a fresh ephemeral pod created by Kubernetes. Once the job completes, the pod is automatically destroyed.

This approach provides several benefits:

• Security isolation: CI workloads are separated from the GitLab instance and run in a different AWS account.
• Scalability: Kubernetes dynamically spins up pods based on job demand.
• Clean execution environments: Every pipeline job runs in a fresh environment without leftover state from previous jobs.

Pipeline logs are persisted to an Amazon S3 bucket, allowing developers to access job outputs even after the ephemeral pods are terminated.

With this architecture, responsibilities are clearly separated. This separation ensures that the CI infrastructure can scale independently while maintaining strong isolation between code hosting and job execution environments.

GitLab describes a similar architecture for hosted runners where CI jobs are executed in isolated runner infrastructure separate from the GitLab instance.

https://docs.gitlab.com/administration/dedicated/architecture/

GitLab Runner

GitLab Runner is the application responsible for executing CI/CD jobs in a pipeline. On every code push to the repository, automated tasks can be defined in the .gitlab-ci.yml file. These tasks can include running tests, building artifacts, and deployments. When a job is scheduled, the runner receives the job payload and token required to fetch repository sources and artifacts. The runner then forwards it to the job executor, which is responsible for creating Kubernetes workloads to execute the job.

GitLab supports both GitLab-hosted runners and self-managed runners. We used self-managed runners as they give us full control over infrastructure and maintenance. GitLab Runners use executors to receive job payloads from the GitLab instance, execute them on our Kubernetes cluster, and return the job outputs and status.

The following sequence diagram represents the reference architecture described in the GitLab documentation:

sequenceDiagram
    participant GitLab
    participant GitLabRunner
    participant Executor

    opt registration
        GitLabRunner->>GitLab: POST /api/v4/runners with registration_token
        GitLab-->>GitLabRunner: Registered with runner_token
    end

    loop job requesting and handling
        GitLabRunner->>GitLab: POST /api/v4/jobs/request with runner_token
        GitLab-->>GitLabRunner: job payload with job_token

        GitLabRunner->>Executor: Job payload

        Executor->>GitLab: clone sources with job_token
        Executor->>GitLab: download artifacts with job_token

        Executor-->>GitLabRunner: return job output and status
        GitLabRunner-->>GitLab: updating job output and status with job_token
    end

Runner Pod

A runner job pod is created by the executor to execute the CI job. Essentially, the GitLab Runner itself runs as a Kubernetes deployment that continuously polls jobs from the GitLab instance. When a job is received, the runner uses the executor to spin up a new ephemeral pod with all the job context received from the poll.

This way, a new pod is created for every job being executed, ensuring that each CI job runs in an isolated environment.

This job pod typically consists of multiple containers:

• Build container: executes the CI job defined in .gitlab-ci.yml
• Helper container: handles Git operations such as cloning the repository and uploading artifacts
• Service containers: optional containers defined in the pipeline for dependencies such as databases used during testing

Once the job execution completes, the pod is automatically destroyed, ensuring that no state persists between CI runs.

The following sequence diagram represents the reference architecture from the GitLab documentation:

sequenceDiagram
    participant G as GitLab instance
    participant R as Runner on Kubernetes cluster
    participant Kube as Kubernetes API
    participant P as Pod

    loop
        R->>G: POST /api/v4/jobs/request
        G-->>R: CI job data
    end

    Note over R,G: POST /api/v4/jobs/request

    R->>Kube: POST to Kube API
    Note over R,Kube: Create a pod to run the CI job

    Kube->>P: Execute job
    Note over P: CI build job = Prepare + Pre-build + Build + Post-build

    P->>G: Job logs

With the GitLab architecture and runner execution model understood, we can now move to how the CI pipelines are structured using GitLab CI templates.

The CI Pipelines

With more than 20 services and repositories to migrate, maintaining CI logic across dozens of repositories quickly becomes tedious as codebases evolve.

This is exactly the problem that GitLab CI templates solve.

GitLab CI templates allow teams to define CI job specifications in a central repository and reuse them across multiple services during CI runs.

In the following sections, we will dive deeper into how this system is designed and how it is implemented in practice.

Design

A standard way to write the CI pipelines in GitLab is to include .gitlab-ci.yml file at the project root, define all the required jobs in that file and GitLab automatically detects this file and executes the jobs according to the defined pipeline rules.

As discussed earlier, this makes it difficult to maintain CI pipelines across the organization as codebases evolve. GitLab therefore provides an alternative: CI templates.

The idea is simple: keep all CI logic in a central repository and have other repositories reference those job definitions.

flowchart LR
    A[Central CI Templates Repository]

    B[Service Repository 1]
    C[Service Repository 2]
    D[Service Repository 3]
    E[Service Repository N]

    B -->|include CI template| A
    C -->|include CI template| A
    D -->|include CI template| A
    E -->|include CI template| A

    B -->|pipeline triggered| F[GitLab CI Runner]
    C -->|pipeline triggered| F
    D -->|pipeline triggered| F
    E -->|pipeline triggered| F

    F --> G[Ephemeral Job Pod]

To address this, we created a dedicated CI templates repository and it is owned by the Platform team for the development and evolution of CI Pipelines over time.

This gives the Platform Team:

• Confidence that any code touching production is going through the standard code quality checks across the organisation.
• All services remain consistent in terms of organisational coding standards.
• Updating any CI Job reflects the changes across the services and repositories immediately.

Let’s now break down the parts of Standard GitLab CI Job implementation and then we will see how templating can help us reference the job from any repositories/services.

CI Pipeline Stages

In real-world systems, CI pipelines are broken into stages and each stage represents a specific concern through which code changes must pass. Let’s look at the standard, must have CI Pipeline stages and respective Jobs.

• Stage 1: Lint & Test

  • Lint + Format + Type Check:

    Ensures the codebase has no linting errors and follows the language’s idiomatic coding standards. This step enforces consistent formatting, static checks, and type validation where applicable.

  • Test Code with Coverage:

    Ensures both newly added and existing code meet an organization-defined test coverage threshold.

    This job typically runs integration tests, while unit tests are usually executed locally by developers and may also run as part of this job.

    End-to-end (E2E) tests are usually excluded at this stage and instead executed in staging environments.

• Stage 2: Analyze

  • Diff Coverage:

    Measures test coverage specifically for newly added or modified code, ensuring that new changes maintain acceptable coverage levels.

  • Static Code Quality Check (SonarQube):

    We use SonarQube, a static code analysis platform that detects bugs, security vulnerabilities (SAST), and code smells to maintain high code quality standards.

• Stage 3: Build

  • Build Artifacts:

    This stage builds the artifacts required for deployment or distribution, such as Docker images, compiled binaries, or packaged services.

• Stage 4: Security Scan

  • Security Scan (Trivy):

    We use Trivy to scan container images, file systems, and Infrastructure as Code (IaC) templates for known vulnerabilities.

• Stage 5: Push

  • Push Artifacts:

    The generated artifacts are pushed to their respective registries or storage systems for later deployment.

• Stage 6: Cosign

  • Cosign Image Signing:

    Built container images are signed using Cosign to ensure artifact integrity.

    This guarantees that only verified and signed images are eligible for deployment to production environments.

flowchart LR
    A[Lint + Test]
    B[Analyze]
    C[Build]
    D[Security Scan]
    E[Push Artifacts]
    F[Cosign Sign]

    A --> B --> C --> D --> E --> F

GitLab CI Job: Implementation Dissection

A typical GitLab CI job template looks like the following. Let’s take a closer look at each section and understand how GitLab interprets them during pipeline execution.

.job_name:
  stage: 
  extends:
    - 
    - 

  image: 
  services:
    - name: 
      alias: 
      command:

  variables:
     VAR1:
     VAR2:

  before_script:
    - 
    - 
    - 
    
  script:
    - 
    - 
  
  after_script:
    -
    -  

  artifacts:
    paths:
      - 
    expire_in:

The template above shows a standard structure of a GitLab CI job defined in the CI templates repository. Let’s break down each YAML section to understand what it represents in the job execution context.

Dissection:

• .job_name:

  GitLab job definitions start with a job name field. Note the . before the job name, this makes the job hidden. Hidden jobs are not executed directly by GitLab. Instead, they are used as reusable templates.

  In our case, these hidden jobs live in the CI templates repository, and service repositories reference them from their .gitlab-ci.yml files using the extends keyword.

• stage:

  The stage field defines the pipeline stage in which the job runs. Multiple jobs can belong to the same stage, and all jobs within a stage are executed in parallel by default.

  If a job needs to wait for another job to finish (even within the same stage), GitLab provides the needs: keyword to explicitly define execution dependencies.

• extends:

  The extends keyword allows a job to inherit configuration from other job templates. This is a core feature that enables CI templating and reuse.

  A job can extend one or multiple templates, allowing configuration layering such as:

  • language-specific setup
  • base job configuration
  • environment-specific settings

  This mechanism allows the platform team to maintain shared job behavior centrally while still enabling flexibility for individual repositories.

• image:

  The image field defines the container image used to execute the job. GitLab CI jobs run inside containers when using Docker or Kubernetes executors.

  This ensures that each job runs in a consistent and reproducible execution environment, independent of the host machine.

• services:

  The services section defines additional containers that run alongside the job container. These are commonly used for dependencies such as:

  • databases
  • message queues
  • caching systems

  For example, integration tests may require a temporary database instance that runs as a service container during the job execution.

• variables:

  The variables section defines environment variables available to the job during runtime.

  These variables can be used for:

  • configuration values
  • authentication tokens
  • environment-specific parameters

  GitLab also supports global variables defined at project, group, or instance level, which are injected into jobs automatically.

• before_script:

  The before_script section contains commands executed before the main job logic runs.

  Typical use cases include:

  • installing dependencies
  • preparing runtime environments
  • authenticating with external services
  • downloading required tools

• script:

  The script section defines the primary commands executed by the job. This is the core logic of the CI job where tasks such as:

  • running tests
  • building artifacts
  • compiling binaries
  • executing build steps

  are performed.

• after_script:

  The after_script section contains commands that run after the main job script completes, regardless of whether the job succeeded or failed.

  This is often used for:

  • cleanup tasks
  • log collection
  • sending notifications

• artifacts:

  The artifacts section defines files or directories that should be persisted after job execution.

  These artifacts can be downloaded from the GitLab UI or passed to later jobs in the pipeline.

  Typical examples include:

  • build artifacts
  • test reports
  • coverage reports
  • compiled binaries

  The expire_in field defines how long GitLab should retain the artifacts before automatically deleting them.

Extending Templates

With the CI job structure understood, the next step is to see how service repositories consume these templates.

Instead of defining CI jobs directly inside each repository, service-level pipelines reference the shared CI templates repository and extend the required job templates.

This allows service repositories to compose pipelines while keeping the CI logic centralized and reusable.

The example below shows how a service repository’s .gitlab-ci.yml references the central templates repository and extends predefined job templates.

include:
  - project: platform/ci-templates
    ref: main
    file:
      - templates/lint.yml
      - templates/build.yml
      - templates/security.yml

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: always


lint:
  extends:
    - .lint_template
    - .language_lint_template


test_with_coverage:
  extends:
    - .test_template
    - .coverage_template

  services:
    - !reference [.postgres_service, services]
    - !reference [.redis_service, services]

  variables:
    COVERAGE_THRESHOLD: 80
    TEST_ENV: ci


diff_coverage:
  extends:
    - .diff_coverage_template

  needs:
    - test_with_coverage


sonarqube_scan:
  extends:
    - .sonarqube_template

  needs:
    - test_with_coverage


docker_build:
  extends:
    - .docker_build_template

  needs:
    - diff_coverage


trivy_scan:
  extends:
    - .trivy_scan_template

  needs:
    - docker_build


docker_push:
  extends:
    - .docker_push_template

  needs:
    - docker_build
    - trivy_scan


cosign_sign:
  extends:
    - .cosign_template

  needs:
    - docker_push

In this setup, the service repository does not implement the CI logic itself. Instead, it includes template files from the centralized CI templates repository and extends the required job definitions.

Each job inherits its configuration from reusable templates such as .lint_template, .docker_build_template, or .trivy_scan_template. This allows the platform team to maintain CI logic centrally while service repositories simply compose the required pipeline stages.

The .gitlab-ci.yml above introduces several GitLab CI features that allow service repositories to compose pipelines using centralized templates.

include

The include keyword allows a pipeline to import CI configuration from other repositories or files.

include:
  - project: platform/ci-templates # defines repository to get templates from
    ref: main # defines branch name/commit sha/HEAD etc for template versioning
    file:
      - templates/lint.yml
      - templates/build.yml
      - templates/security.yml

This enables service repositories to reuse CI templates maintained in a central templates repository.

workflow

The workflow keyword controls when a pipeline should be created.

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: always

In this example, pipelines are triggered only for merge request events.

extends

The extends keyword allows a job to inherit configuration from reusable templates.

lint:
  extends:
    - .lint_template
    - .language_lint_template

GitLab merges the referenced templates to construct the final job definition.

needs

The needs keyword defines explicit job dependencies.

needs:
  - docker_build

This allows GitLab to build a directed job dependency graph, enabling faster pipeline execution by running jobs as soon as their dependencies are satisfied.

variables

The variables section defines environment variables available during job execution.

variables:
  COVERAGE_THRESHOLD: 80
  TEST_ENV: ci

These variables are accessible to all commands executed in the job.

!reference

!reference allows a job to reuse specific sections from another job or template.

services:
  - !reference [.postgres_service, services]

This is useful when reusing sections like services, variables, or before_script without overriding previously defined values.

It is important to note that !reference is not a standard YAML keyword, but a GitLab-specific extension processed during pipeline compilation.

Experiences and Learnings

While building this templated CI system, we encountered a few subtle behaviors in how GitLab processes include, extends, and !reference.

include

The include keyword allows pipelines to reference CI templates from external repositories.

GitLab fetches included templates only when a new pipeline is created. If a failed job or pipeline is retried, GitLab does not fetch updated templates and instead uses the configuration compiled during the original pipeline creation.

This means that when testing template changes, simply retrying pipelines will not apply updates. A new pipeline must be triggered to pull the latest template changes.

extends

The extends keyword allows jobs to inherit configuration from reusable templates.

When a job redefines the same sections in .gitlab-ci.yml, those values generally override the inherited configuration. However, some sections such as variables are merged instead of overridden, allowing both template and service-level variables to coexist.

!reference

The !reference keyword allows reusing specific sections from another job or template.

This is useful when redefining sections like services, variables, or before_script, where redefining the section normally overrides inherited values. Using !reference allows selectively importing configuration without losing previously defined settings.

It is worth noting that !reference is not a standard YAML keyword, but a GitLab-specific extension.

Wrap Up

GitLab CI templates allow platform teams to centralize CI logic and maintain consistent pipelines across multiple services. This becomes especially valuable in organizations managing a large number of repositories.

One trade-off is that developers may lose some direct control over CI configurations and may require platform team involvement for certain changes. While this can slightly slow down some workflows, the benefit is a more maintainable, standardized, and reliable CI system across the organization.