CI/CD Pipelines on Linux

Why This Matters

A developer pushes a code change at 3pm on Friday. The change breaks the application, but nobody notices until Monday morning when customers start complaining. It takes the team four hours to figure out which of the 47 commits merged over the weekend caused the issue.

Now imagine the alternative: the developer pushes the change, and within five minutes an automated pipeline builds the code, runs 200 tests, and rejects the change because three tests fail. The developer sees the failure immediately, fixes it, pushes again, and all tests pass. The code deploys automatically to staging for further validation. Nobody's weekend is ruined.

That is CI/CD -- Continuous Integration and Continuous Delivery (or Deployment). It is the backbone of modern software engineering, and it runs almost entirely on Linux.

Try This Right Now

If you have Git installed, you can simulate a basic pipeline with a shell script:

$ mkdir -p ~/cicd-demo && cd ~/cicd-demo
$ git init

$ cat > app.py << 'EOF'
def add(a, b):
    return a + b

def subtract(a, b):
    return a - b
EOF

$ cat > test_app.py << 'EOF'
from app import add, subtract

def test_add():
    assert add(2, 3) == 5

def test_subtract():
    assert subtract(5, 3) == 2
EOF

$ cat > pipeline.sh << 'PIPELINE'
#!/bin/bash
set -e
echo "=== Stage 1: Lint ==="
python3 -m py_compile app.py && echo "PASS: Syntax OK"

echo "=== Stage 2: Test ==="
python3 -c "
from app import add, subtract
assert add(2,3) == 5, 'add failed'
assert subtract(5,3) == 2, 'subtract failed'
print('PASS: All tests passed')
"

echo "=== Stage 3: Build ==="
echo "Building artifact..."
tar czf app-$(date +%Y%m%d%H%M%S).tar.gz app.py
echo "PASS: Build complete"

echo "=== Pipeline Succeeded ==="
PIPELINE
chmod +x pipeline.sh

$ ./pipeline.sh

Expected output:

=== Stage 1: Lint ===
PASS: Syntax OK
=== Stage 2: Test ===
PASS: All tests passed
=== Stage 3: Build ===
Building artifact...
PASS: Build complete
=== Pipeline Succeeded ===

That is a CI/CD pipeline in its most primitive form. Real tools add automation, parallelism, artifact management, and much more -- but the concept is the same.

CI/CD Concepts

Continuous Integration (CI)

Continuous Integration means developers merge their code changes into a shared repository frequently -- at least daily. Each merge triggers an automated build and test process.

┌────────────────────────────────────────────────────────────┐
│               CONTINUOUS INTEGRATION                        │
│                                                            │
│  Developer A ──push──┐                                     │
│                      │    ┌──────────┐    ┌──────────┐     │
│  Developer B ──push──├──► │  Build   │──► │  Test    │     │
│                      │    └──────────┘    └──────────┘     │
│  Developer C ──push──┘         │              │            │
│                                │              │            │
│                           Pass? ──► Merge     │            │
│                           Fail? ──► Reject  Fail? ──► Fix  │
│                                                            │
└────────────────────────────────────────────────────────────┘

The goal: catch bugs early, when they are cheap to fix. A bug found in CI costs minutes to fix. The same bug found in production costs hours, money, and customer trust.

Continuous Delivery (CD)

Continuous Delivery extends CI by automatically preparing code for release to production. Every change that passes CI is deployable, but a human still decides when to deploy.

Continuous Deployment

Continuous Deployment goes one step further: every change that passes all automated tests is deployed to production automatically, with no human intervention.

┌──────────────────────────────────────────────────────────────┐
│                                                              │
│  CI only:       Code → Build → Test → ✓ (done)              │
│                                                              │
│  CI + Delivery: Code → Build → Test → Stage → [Human] → Prod│
│                                                              │
│  CI + Deploy:   Code → Build → Test → Stage → Prod (auto)   │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Think About It: When would you choose Continuous Delivery over Continuous Deployment? Think about regulated industries, user-facing applications, and infrastructure changes.

Pipeline Stages

A typical CI/CD pipeline has these stages:

┌─────────┐   ┌─────────┐   ┌─────────┐   ┌──────────┐   ┌──────────┐
│  Lint   │──►│  Build  │──►│  Test   │──►│  Package │──►│  Deploy  │
│         │   │         │   │         │   │          │   │          │
│ Syntax  │   │ Compile │   │ Unit    │   │ Docker   │   │ Staging  │
│ Style   │   │ Bundle  │   │ Integr. │   │ Tarball  │   │ Prod     │
│ Checks  │   │ Resolve │   │ E2E     │   │ RPM/DEB  │   │          │
└─────────┘   └─────────┘   └─────────┘   └──────────┘   └──────────┘
     │              │             │              │              │
     ▼              ▼             ▼              ▼              ▼
  Fail fast    Fail fast     Fail fast     Artifacts       Rollback
  on errors    on errors     on errors     stored          on failure

Lint

Check code quality before doing anything expensive:

  • Syntax validation
  • Style compliance (PEP 8, ESLint, shellcheck)
  • Security scanning (static analysis)

Build

Compile code, resolve dependencies, bundle assets:

  • make, gcc for C/C++
  • mvn package for Java
  • npm run build for JavaScript
  • go build for Go

Test

Run automated tests at multiple levels:

  • Unit tests: Test individual functions
  • Integration tests: Test components working together
  • End-to-end tests: Test the full application flow

Package

Create deployable artifacts:

  • Docker images
  • Tarballs
  • RPM/DEB packages
  • Application archives

Deploy

Release to target environments:

  • Staging for validation
  • Production for users

Git-Based Workflows

CI/CD pipelines are triggered by Git events. The most common workflow:

┌────────────────────────────────────────────────────────────────┐
│                   BRANCHING WORKFLOW                            │
│                                                                │
│  main ──────●──────────────●──────────────●──── (always stable)│
│              \            / \            /                      │
│  feature/     \──●──●──●──   \──●──●──●──                      │
│  login           │  │  │        │  │  │                        │
│                  CI CI CI       CI CI CI                        │
│                  runs  │        runs  │                        │
│                     Merge PR      Merge PR                     │
│                                                                │
│  Every push to a branch triggers CI.                           │
│  Merging to main triggers CD (deploy to staging/production).   │
│                                                                │
└────────────────────────────────────────────────────────────────┘

Self-Hosted Git: Gitea and Forgejo

Before choosing a CI/CD tool, you need a Git hosting platform. For self-hosted, open-source options:

Gitea

Gitea is a lightweight, self-hosted Git service written in Go. It is fast, requires minimal resources, and provides a GitHub-like interface.

# Install Gitea (binary method)
$ wget -O /tmp/gitea https://dl.gitea.com/gitea/1.21/gitea-1.21-linux-amd64
$ sudo mv /tmp/gitea /usr/local/bin/gitea
$ sudo chmod +x /usr/local/bin/gitea

# Create system user
$ sudo adduser --system --shell /bin/bash --group --home /home/gitea gitea

# Create required directories
$ sudo mkdir -p /var/lib/gitea/{custom,data,log}
$ sudo chown -R gitea:gitea /var/lib/gitea
$ sudo mkdir -p /etc/gitea
$ sudo chown root:gitea /etc/gitea
$ sudo chmod 770 /etc/gitea

Forgejo

Forgejo is a community fork of Gitea, focused on remaining fully community-governed. It is API-compatible with Gitea and uses the same configuration.

Distro Note: Both Gitea and Forgejo are available as single binaries, Docker images, or distribution packages. On Debian-based systems, check if your distro ships a package. Otherwise, the binary or Docker approach works everywhere.

Open-Source CI/CD Tools

Woodpecker CI

Woodpecker CI is a community fork of Drone CI, fully open source under the Apache 2.0 license. It integrates tightly with Gitea, Forgejo, GitHub, and GitLab.

Installing Woodpecker CI

Woodpecker is typically deployed with Docker alongside your Git server:

# docker-compose.yml for Woodpecker + Gitea
version: '3'
services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:latest
    ports:
      - "8000:8000"
    volumes:
      - woodpecker-server-data:/var/lib/woodpecker/
    environment:
      - WOODPECKER_OPEN=true
      - WOODPECKER_HOST=http://your-server:8000
      - WOODPECKER_GITEA=true
      - WOODPECKER_GITEA_URL=http://gitea:3000
      - WOODPECKER_GITEA_CLIENT=your-oauth-client-id
      - WOODPECKER_GITEA_SECRET=your-oauth-secret
      - WOODPECKER_SECRET=a-shared-secret

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:latest
    depends_on:
      - woodpecker-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_SERVER=woodpecker-server:9000
      - WOODPECKER_SECRET=a-shared-secret

volumes:
  woodpecker-server-data:

Writing a Woodpecker Pipeline

Pipelines are defined in .woodpecker.yml in your repository:

# .woodpecker.yml
steps:
  lint:
    image: python:3.11
    commands:
      - pip install flake8
      - flake8 --max-line-length=120 .

  test:
    image: python:3.11
    commands:
      - pip install pytest
      - pip install -r requirements.txt
      - pytest tests/ -v

  build:
    image: python:3.11
    commands:
      - python setup.py sdist bdist_wheel

  deploy:
    image: alpine
    commands:
      - echo "Deploying to staging..."
      - apk add --no-cache openssh-client
      - scp dist/*.whl deploy@staging:/opt/app/
    when:
      branch: main
      event: push

Each step runs in an isolated container. If any step fails, the pipeline stops.

Jenkins

Jenkins is the oldest and most widely used open-source CI/CD server. It is written in Java and has a massive plugin ecosystem.

Installing Jenkins

# On Debian/Ubuntu
$ sudo apt update
$ sudo apt install -y openjdk-17-jre
$ curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
    /usr/share/keyrings/jenkins-keyring.asc > /dev/null
$ echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
    https://pkg.jenkins.io/debian-stable binary/" | sudo tee \
    /etc/apt/sources.list.d/jenkins.list > /dev/null
$ sudo apt update
$ sudo apt install -y jenkins
$ sudo systemctl enable --now jenkins

Jenkins runs on port 8080 by default. The initial admin password is at:

$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Jenkinsfile: Pipeline as Code

Modern Jenkins uses a Jenkinsfile in the repository root:

// Jenkinsfile
pipeline {
    agent any

    stages {
        stage('Lint') {
            steps {
                sh 'python3 -m py_compile app.py'
            }
        }

        stage('Test') {
            steps {
                sh 'python3 -m pytest tests/ -v --junitxml=results.xml'
            }
            post {
                always {
                    junit 'results.xml'
                }
            }
        }

        stage('Build') {
            steps {
                sh 'tar czf app.tar.gz *.py requirements.txt'
                archiveArtifacts artifacts: 'app.tar.gz'
            }
        }

        stage('Deploy to Staging') {
            when {
                branch 'main'
            }
            steps {
                sh './deploy.sh staging'
            }
        }
    }

    post {
        failure {
            echo 'Pipeline failed! Check the logs.'
        }
    }
}

GitLab CI/CD

GitLab includes CI/CD built directly into the platform. Pipelines are defined in .gitlab-ci.yml:

# .gitlab-ci.yml
stages:
  - lint
  - test
  - build
  - deploy

lint:
  stage: lint
  image: python:3.11
  script:
    - pip install flake8
    - flake8 --max-line-length=120 .

test:
  stage: test
  image: python:3.11
  script:
    - pip install pytest
    - pip install -r requirements.txt
    - pytest tests/ -v --junitxml=report.xml
  artifacts:
    reports:
      junit: report.xml

build:
  stage: build
  image: python:3.11
  script:
    - python setup.py sdist bdist_wheel
  artifacts:
    paths:
      - dist/

deploy_staging:
  stage: deploy
  script:
    - echo "Deploying to staging..."
    - scp dist/*.whl deploy@staging:/opt/app/
  only:
    - main
  environment:
    name: staging

GitLab CI/CD uses runners -- agents that execute pipeline jobs. You can use shared runners provided by GitLab or install your own:

# Install GitLab Runner
$ curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
$ sudo apt install -y gitlab-runner

# Register the runner
$ sudo gitlab-runner register

Hands-On: Build a Simple CI Pipeline

Let us create a complete, working pipeline using shell scripts to understand the mechanics before using a CI tool.

Step 1: Set up a project with tests:

$ mkdir -p ~/cicd-project/tests
$ cd ~/cicd-project
$ git init

Step 2: Create the application:

$ cat > ~/cicd-project/app.py << 'EOF'
"""Simple calculator application."""

def add(a, b):
    """Add two numbers."""
    return a + b

def subtract(a, b):
    """Subtract b from a."""
    return a - b

def multiply(a, b):
    """Multiply two numbers."""
    return a * b

def divide(a, b):
    """Divide a by b."""
    if b == 0:
        raise ValueError("Cannot divide by zero")
    return a / b
EOF

Step 3: Create tests:

$ cat > ~/cicd-project/tests/test_app.py << 'EOF'
import sys
sys.path.insert(0, '.')
from app import add, subtract, multiply, divide

def test_add():
    assert add(2, 3) == 5
    assert add(-1, 1) == 0
    assert add(0, 0) == 0

def test_subtract():
    assert subtract(5, 3) == 2
    assert subtract(0, 5) == -5

def test_multiply():
    assert multiply(3, 4) == 12
    assert multiply(0, 5) == 0

def test_divide():
    assert divide(10, 2) == 5.0
    assert divide(7, 2) == 3.5

def test_divide_by_zero():
    try:
        divide(1, 0)
        assert False, "Should have raised ValueError"
    except ValueError:
        pass

if __name__ == "__main__":
    test_add()
    test_subtract()
    test_multiply()
    test_divide()
    test_divide_by_zero()
    print("All tests passed!")
EOF

Step 4: Create the pipeline script:

$ cat > ~/cicd-project/run-pipeline.sh << 'PIPELINE'
#!/bin/bash
set -euo pipefail

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

stage() {
    echo -e "\n${YELLOW}=== $1 ===${NC}"
}

pass() {
    echo -e "${GREEN}PASS: $1${NC}"
}

fail() {
    echo -e "${RED}FAIL: $1${NC}"
    exit 1
}

# Stage 1: Lint
stage "LINT"
python3 -m py_compile app.py && pass "Syntax check" || fail "Syntax error in app.py"

# Stage 2: Test
stage "TEST"
python3 tests/test_app.py && pass "All tests" || fail "Tests failed"

# Stage 3: Build
stage "BUILD"
BUILD_ID="build-$(date +%Y%m%d-%H%M%S)"
mkdir -p artifacts
tar czf "artifacts/${BUILD_ID}.tar.gz" app.py tests/
pass "Artifact created: artifacts/${BUILD_ID}.tar.gz"

# Stage 4: Deploy (simulated)
stage "DEPLOY"
echo "Would deploy artifacts/${BUILD_ID}.tar.gz to staging"
pass "Deployment simulated"

echo -e "\n${GREEN}=== PIPELINE SUCCEEDED ===${NC}"
PIPELINE
chmod +x ~/cicd-project/run-pipeline.sh

Step 5: Run it:

$ cd ~/cicd-project && ./run-pipeline.sh

=== LINT ===
PASS: Syntax check

=== TEST ===
All tests passed!
PASS: All tests

=== BUILD ===
PASS: Artifact created: artifacts/build-20250615-143022.tar.gz

=== DEPLOY ===
Would deploy artifacts/build-20250615-143022.tar.gz to staging
PASS: Deployment simulated

=== PIPELINE SUCCEEDED ===

Step 6: Now break something and see the pipeline catch it:

$ echo "this is not valid python" >> ~/cicd-project/app.py
$ cd ~/cicd-project && ./run-pipeline.sh

=== LINT ===
FAIL: Syntax error in app.py

The pipeline stopped at the lint stage, exactly as it should.

Artifacts, Variables, and Secrets

Artifacts

Artifacts are files produced by a pipeline that need to be preserved -- compiled binaries, test reports, Docker images, packages.

# GitLab CI example
build:
  stage: build
  script:
    - make build
  artifacts:
    paths:
      - build/myapp
    expire_in: 30 days

Environment Variables

Pipelines use environment variables for configuration:

# Woodpecker CI
steps:
  deploy:
    image: alpine
    environment:
      - APP_ENV=staging
      - APP_PORT=8080
    commands:
      - echo "Deploying to $APP_ENV on port $APP_PORT"

Secrets Management

Secrets (API keys, passwords, deploy keys) should never be in the pipeline file or the repository. CI/CD platforms provide secret storage:

  • GitLab: Settings > CI/CD > Variables (masked, protected)
  • Jenkins: Credentials plugin
  • Woodpecker: Repository secrets in the UI or via API
  • GitHub Actions: Repository or organization secrets
# GitLab CI -- using a secret variable
deploy:
  script:
    - echo "$DEPLOY_KEY" > /tmp/deploy.key
    - chmod 600 /tmp/deploy.key
    - scp -i /tmp/deploy.key build/app deploy@prod:/opt/
  after_script:
    - rm -f /tmp/deploy.key

Safety Warning: Never echo or print secret variables in pipeline output. Most CI platforms mask them automatically, but do not rely on that alone. Also be cautious about secrets in pull request pipelines -- a malicious PR could add a step that exfiltrates secrets.

Deployment Strategies

How you deploy to production matters as much as what you deploy.

Rolling Deployment

Update servers one at a time (or in batches). If something goes wrong, stop and roll back.

┌─────────────────────────────────────────────────┐
│             ROLLING DEPLOYMENT                   │
│                                                  │
│  Time 0: [v1] [v1] [v1] [v1]  (all on v1)       │
│  Time 1: [v2] [v1] [v1] [v1]  (updating...)     │
│  Time 2: [v2] [v2] [v1] [v1]  (updating...)     │
│  Time 3: [v2] [v2] [v2] [v1]  (updating...)     │
│  Time 4: [v2] [v2] [v2] [v2]  (complete)        │
│                                                  │
│  Pros: No extra infrastructure needed            │
│  Cons: Mixed versions during deployment          │
│                                                  │
└─────────────────────────────────────────────────┘

Blue-Green Deployment

Maintain two identical environments. Deploy to the inactive one, test, then switch traffic.

┌─────────────────────────────────────────────────┐
│           BLUE-GREEN DEPLOYMENT                  │
│                                                  │
│  Before:                                         │
│  Users ──► Load Balancer ──► BLUE (v1) ← active  │
│                              GREEN (idle)        │
│                                                  │
│  Deploy v2 to GREEN, test it:                    │
│  Users ──► Load Balancer ──► BLUE (v1) ← active  │
│                              GREEN (v2) ← ready  │
│                                                  │
│  Switch:                                         │
│  Users ──► Load Balancer ──► BLUE (v1) ← idle    │
│                              GREEN (v2) ← active │
│                                                  │
│  Rollback = switch back to BLUE                  │
│                                                  │
│  Pros: Instant rollback, no mixed versions       │
│  Cons: Need double the infrastructure            │
│                                                  │
└─────────────────────────────────────────────────┘

Canary Deployment

Route a small percentage of traffic to the new version. If it works, gradually increase.

┌─────────────────────────────────────────────────┐
│            CANARY DEPLOYMENT                     │
│                                                  │
│  Phase 1:  95% ──► v1 (stable)                   │
│             5% ──► v2 (canary)   ← monitor       │
│                                                  │
│  Phase 2:  75% ──► v1                            │
│            25% ──► v2            ← still OK?     │
│                                                  │
│  Phase 3:  50% ──► v1                            │
│            50% ──► v2            ← looking good  │
│                                                  │
│  Phase 4: 100% ──► v2           ← full rollout   │
│                                                  │
│  Pros: Minimal blast radius for bugs             │
│  Cons: More complex routing and monitoring       │
│                                                  │
└─────────────────────────────────────────────────┘

Think About It: You are deploying a database schema change. Which deployment strategy would you use? Why are database migrations particularly tricky for blue-green and canary deployments?

Debug This

A pipeline keeps failing with this error:

Step 3/5: Test
ERROR: ModuleNotFoundError: No module named 'requests'
Pipeline FAILED at test stage

The .gitlab-ci.yml looks like this:

test:
  stage: test
  image: python:3.11
  script:
    - pytest tests/ -v

What is wrong?

Answer: The pipeline does not install dependencies before running tests. The requests module (used by the application) is not installed in the clean Python container. Fix:

test:
  stage: test
  image: python:3.11
  script:
    - pip install -r requirements.txt    # Install dependencies first!
    - pytest tests/ -v

Every pipeline stage starts with a clean environment. Dependencies must be installed explicitly every time, or cached between runs.

What Just Happened?

┌──────────────────────────────────────────────────────────────┐
│                    CHAPTER 69 RECAP                           │
│──────────────────────────────────────────────────────────────│
│                                                              │
│  CI/CD automates the build-test-deploy cycle.                │
│                                                              │
│  CI = merge frequently, test automatically                   │
│  CD (Delivery) = always deployable, human triggers deploy    │
│  CD (Deployment) = deploy automatically on every merge       │
│                                                              │
│  Pipeline stages: Lint → Build → Test → Package → Deploy     │
│                                                              │
│  Open-source CI/CD tools:                                    │
│  • Woodpecker CI: lightweight, Docker-native, Gitea-friendly │
│  • Jenkins: veteran, plugin-rich, Jenkinsfile-based          │
│  • GitLab CI/CD: integrated with GitLab, .gitlab-ci.yml      │
│                                                              │
│  Self-hosted Git: Gitea or Forgejo                           │
│                                                              │
│  Deployment strategies:                                      │
│  • Rolling: update one at a time                             │
│  • Blue-green: switch between two identical environments     │
│  • Canary: send small traffic to new version first           │
│                                                              │
│  Never put secrets in pipeline files or repositories.        │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Try This

Exercise 1: Expand the Pipeline

Take the shell-based pipeline from the hands-on section and add:

  • A code coverage stage that reports what percentage of code is tested
  • An artifact cleanup stage that removes builds older than 7 days

Exercise 2: Write a Woodpecker Pipeline

Create a .woodpecker.yml for a project of your choice. Include lint, test, build, and deploy stages. Use when conditions to only deploy on the main branch.

Exercise 3: Set Up Gitea

Install Gitea on your local machine (using Docker or a binary). Create a repository, push code to it, and explore the web interface.

Exercise 4: Jenkins Exploration

Install Jenkins, create a freestyle project, and configure it to poll a Git repository and run tests on every push.

Bonus Challenge

Build a pipeline that:

  1. Runs tests in parallel (unit tests and integration tests at the same time)
  2. Only builds a Docker image if all tests pass
  3. Deploys to staging automatically
  4. Requires manual approval before deploying to production
  5. Sends a notification (even just an echo) on success or failure

Think about how each CI/CD tool handles parallel stages and manual gates.