Creating Zstream Builder Images

Step-by-step guide for creating new RHEL zstream builder container images

This document provides explicit instructions for creating new RHEL zstream builder container images (RHEL 9+). It is designed to be followed by both humans and AI systems with minimal interpretation required.

Automation Note

This document has two phases, each ending with a merge request that requires human review. When following as an AI agent:

  • Execute steps within a phase without asking for confirmation
  • The build step (Phase 1, Step 3) takes several minutes - this is expected
  • STOP at the end of each phase when you see STOP: Human action required
  • Report the MR URL and wait for the user to confirm the merge before continuing
  • Only proceed to the next phase after the user confirms the previous MR is merged

Overview

When a new RHEL zstream is released (e.g., 9.7, 10.1), a corresponding builder container image must be created. This requires changes in two repositories:

  1. containers (gitlab.com/cki-project/containers) - Container image definitions and the shared image matrix
  2. kernel (gitlab.com/redhat/rhel/src/kernel) - Kernel repository that uses the builder image

The process has two phases:

  1. Phase 1: Create Container Image - Add the image definition, register it in the shared image matrix, build locally, and submit an MR
  2. Phase 2: Update Kernel Repository - Update the kernel repository to use the new builder image

The image matrix in .gitlab/ci_templates/cki-images.yml is the single source of truth for all CKI container images. It is included by downstream repositories (pipeline-definition, deployment-all) automatically - no separate registration step is needed.

Input Parameters

The process requires the following parameters:

  • {major}: RHEL major version (e.g., 9, 10, 11)
  • {minor}: RHEL minor version (e.g., 7, 1, 0)
  • {issue_url}: URL to the GitLab issue or Jira ticket tracking this work
  • {containers_fork_remote}: Git remote name for your containers fork
  • {kernel_repo_path}: Path to the kernel repository (e.g., ../kernel-rhel)
  • {kernel_upstream_remote}: Git remote for upstream kernel repo (e.g., rhel-9)
  • {kernel_fork_remote}: Git remote for your kernel repo fork (e.g., rhel-9-fork)

Note on kernel repo remotes: The kernel repository typically uses separate remotes for each RHEL major version:

  • Upstream: rhel-{major} → git@gitlab.com:redhat/rhel/src/kernel/rhel-{major}
  • Fork: e.g., rhel-{major}-fork → user’s fork of the upstream repo

Zstream branches on each remote are named {major}.{minor} (e.g., 9.6, 9.7, 10.1).

Standard MR Workflow

All phases in this guide create merge requests by pushing to a personal fork. This requires having a fork of each repository with a git remote configured.

The workflow pattern is:

  1. Navigate to the target repository (if not already there)

  2. Update to latest main: git checkout main && git pull

  3. Create a feature branch: git checkout -b {branch_name}

  4. Make changes and stage files: git add {files}

  5. Commit with issue reference: git commit -m "{message}\n\n{issue_url}"

  6. Push to your fork and create MR targeting upstream. Where {fork_remote} is the git remote pointing to your personal fork:

    git push \
    -o merge_request.create \
    -o merge_request.target=main \
    -o merge_request.title="{title}" \
    -o merge_request.description="Related issue: {issue_url}" \
    {fork_remote} {branch_name}

  7. Return to original directory (if navigated away)

Each phase below provides the concrete commands with all values filled in.

Prerequisites

Before starting, verify the following:

1. Issue URL

You must have a GitLab issue or Jira ticket URL to associate with this work.

Verification: Confirm {issue_url} has been provided.

  • If provided: Proceed to next prerequisite.
  • If NOT provided: STOP and ask the user for the issue URL before continuing.

2. Network Access

You must have access to Red Hat internal network (VPN) to reach download.devel.redhat.com.

Verification command:

curl -s -o /dev/null -w "%{http_code}" https://download.devel.redhat.com/

Success criteria: Returns 200 or 301.

3. UBI Base Image Availability

The UBI base image for the target version must exist.

Verification command:

podman pull registry.access.redhat.com/ubi{major}/ubi:{major}.{minor}

Success criteria: Image pulls successfully.

4. Repository File Exists

Check if files/rhel{major}.z.repo exists in the containers repository.

Verification command:

test -f files/rhel{major}.z.repo && echo "EXISTS" || echo "MISSING"
  • If EXISTS: No action needed, proceed to fork setup.
  • If MISSING: A new repo file must be created in Phase 1 (see 1.0 Create Repo File).

5. Containers Fork Setup

You need a fork of the containers repository with remotes configured.

Verification command:

git remote -v

Identify your fork remote: Look for a remote pointing to gitlab.com/{your-username}/containers. Note this remote name as {containers_fork_remote}.

Example output:

fork    git@gitlab.com:your-username/containers.git (fetch)
origin  git@gitlab.com:cki-project/containers.git (fetch)

In this example, {containers_fork_remote} = fork.

6. Kernel Repository Fork Setup

You need access to the kernel repository with remotes configured for the target RHEL major version.

Verification command:

cd {kernel_repo_path} && git remote -v && cd -

Identify your remotes: Look for:

  • Upstream remote pointing to gitlab.com/redhat/rhel/src/kernel/rhel-{major} (note this as {kernel_upstream_remote})
  • Fork remote pointing to your fork (note this as {kernel_fork_remote})

Example output:

rhel-9       git@gitlab.com:redhat/rhel/src/kernel/rhel-9 (fetch)
rhel-9-fork  git@gitlab.com:your-username/rhel-9 (fetch)

In this example:

  • {kernel_upstream_remote} = rhel-9
  • {kernel_fork_remote} = rhel-9-fork

7. Tools

Required: podman.

Verification command:

podman --version

Success criteria: Returns version information.

Decision Logic: AWSCLI Flag

Based on the major version, determine the AWSCLI installation flag:

Major Version Flag Value
9 SYSTEM
10+ MATCHING

Decision rule:

IF major == 9 THEN flag = "SYSTEM"
ELSE IF major >= 10 THEN flag = "MATCHING"

Phase 1: Create Container Image

This phase creates the container image definition, registers it in the shared image matrix, builds it locally for validation, and submits an MR to the containers repository.

1.0 Create Repo File (new major versions only)

If this is the first zstream for a new major version and files/rhel{major}.z.repo does not exist, create it based on the existing pattern.

Use files/rhel9.z.repo or files/rhel10.z.repo as a template:

  1. Copy the template: cp files/rhel{existing}.z.repo files/rhel{major}.z.repo
  2. Replace all version numbers with the new major version
  3. The __Z__ placeholder will be substituted with the minor version at build time

Note: This step is only needed for the first zstream of a new major version (e.g., 11.0). Skip to 1.1 if the repo file already exists.

1.1 Create the Container Image File

Create the file builds/builder-rhel{major}.{minor}/Containerfile.in.

First, fetch the base image digest:

skopeo inspect --no-tags docker://registry.access.redhat.com/ubi{major}/ubi:{major}.{minor} | jq -r .Digest

Then create the file with the following content (replacing {digest} with the sha256 digest from the previous command):

/* renovate: datasource=docker depName=registry.access.redhat.com/ubi{major}/ubi */
#define _BASE_IMAGE registry.access.redhat.com/ubi{major}/ubi
#define _BASE_IMAGE_TAG {major}.{minor}@{digest}
#define _AWSCLI_PIP_{flag} 1
#define _RHEL_VERSION {major}
#define _RHEL_MINOR_VERSION {minor}

#include "setup-from-rhel"
#include "builder"
#include "cleanup"

Example: RHEL 9.7

File: builds/builder-rhel9.7/Containerfile.in

/* renovate: datasource=docker depName=registry.access.redhat.com/ubi9/ubi */
#define _BASE_IMAGE registry.access.redhat.com/ubi9/ubi
#define _BASE_IMAGE_TAG 9.7@sha256:e9a31af6530caffa3551f266c51a0d43b602e8f76a0dc12826dbeebceb487c92
#define _AWSCLI_PIP_SYSTEM 1
#define _RHEL_VERSION 9
#define _RHEL_MINOR_VERSION 7

#include "setup-from-rhel"
#include "builder"
#include "cleanup"

Example: RHEL 10.1

File: builds/builder-rhel10.1/Containerfile.in

/* renovate: datasource=docker depName=registry.access.redhat.com/ubi10/ubi */
#define _BASE_IMAGE registry.access.redhat.com/ubi10/ubi
#define _BASE_IMAGE_TAG 10.1@sha256:15f0a6e2b448ac7aa24425c9ab541e6402dc32b782844a285ccaeffb1c938fe1
#define _AWSCLI_PIP_MATCHING 1
#define _RHEL_VERSION 10
#define _RHEL_MINOR_VERSION 1

#include "setup-from-rhel"
#include "builder"
#include "cleanup"

1.2 Add Image to the Shared Matrix

Add the new image name to the shared image matrix in .gitlab/ci_templates/cki-images.yml.

Location: Find the .cki_images section with parallel: matrix: containing the list of IMAGE_NAME values.

Action: Add builder-rhel{major}.{minor} to the internal pipeline images section alongside other builder-rhel* images with CONTAINERS_AUTOMATIC_DEPLOYMENT: 'false'.

Example: To add builder-rhel10.1, find the section containing builder-rhel10.0 and add the new image:

      - IMAGE_NAME:
          - builder-rhel8
          - builder-rhel8.2
          # ... other images ...
          - builder-rhel10
          - builder-rhel10.0
          - builder-rhel10.1  # Add this line
        CONTAINERS_AUTOMATIC_DEPLOYMENT: 'false'

This matrix is automatically included by downstream repositories (pipeline-definition, deployment-all) - no additional registration is needed.

Verification command:

grep -q "builder-rhel{major}.{minor}" .gitlab/ci_templates/cki-images.yml && echo "FOUND" || echo "MISSING"

Success criteria: Returns FOUND.

1.3 Build the Container Image Locally

Run the build using the CKI buildah container:

podman run \
    --rm \
    --pull=newer \
    --security-opt label=disable \
    -e IMAGE_NAME=builder-rhel{major}.{minor} \
    -w /code \
    -v .:/code:z \
    -v ~/.local/share/containers:/var/lib/containers:z \
    quay.io/cki/buildah:production \
    cki_build_image.sh

Note: The --security-opt label=disable flag is required to allow buildah to access the shared container storage. This disables SELinux label enforcement for the container, which is needed because the host’s container storage files have SELinux contexts that would otherwise block access.

Success criteria: Command exits with code 0 and outputs build completion message.

Note: The local build creates the image with tag latest-amd64 (or the architecture you’re building on).

1.4 Validate the Built Image

Run validation tests on the built image:

podman run --rm localhost/builder-rhel{major}.{minor}:latest-amd64 bash -c '
    set -e
    echo "Checking gcc..." && gcc --version
    echo "Checking make..." && make --version
    echo "Checking python3..." && python3 --version
    echo "Checking build tools..." && which bison flex bc
    echo "Running compile test..."
    echo "int main(){return 0;}" > /tmp/test.c
    gcc -o /tmp/test /tmp/test.c
    /tmp/test
    echo "SUCCESS: All validation tests passed"
'

Success criteria: Output ends with SUCCESS: All validation tests passed.

1.5 Create and Submit Merge Request

1.5.1 Create a Branch

git checkout -b add-builder-rhel{major}.{minor}

1.5.2 Stage and Commit

git add builds/builder-rhel{major}.{minor}/Containerfile.in .gitlab/ci_templates/cki-images.yml
git commit -m "Add builder-rhel{major}.{minor} container image

{issue_url}"

1.5.3 Push to Fork and Create MR

git push \
    -o merge_request.create \
    -o merge_request.target_project=cki-project/containers \
    -o merge_request.target=main \
    -o merge_request.title="Add builder-rhel{major}.{minor} container image" \
    -o merge_request.description="Related issue: {issue_url}" \
    {containers_fork_remote} add-builder-rhel{major}.{minor}

Success criteria: Command outputs MR URL.

STOP: Merge containers MR and verify image publication

The containers MR must be reviewed, merged, and the CI pipeline must complete to build and publish the container images.

Report the MR URL to the user and wait for confirmation that:

  1. The MR has been merged
  2. The container images have been published

Verification commands (replace {mr_id} with the MR number):

# Check MR pipeline completed (image tagged with MR number)
skopeo inspect docker://quay.io/cki/builder-rhel{major}.{minor}:mr-{mr_id}

# Check production deployment completed (may require manual deployment)
skopeo inspect docker://quay.io/cki/builder-rhel{major}.{minor}:production

Both images must be available before proceeding to Phase 2.

Phase 2: Update Kernel Repository

After the container image is published, update the kernel repository to use the new builder image. New zstream branches use the ystream builder image (e.g., builder-rhel9) as a fallback until the zstream container is available. This phase updates builder_image to the newly created zstream image.

2.1 Checkout the Zstream Branch

cd {kernel_repo_path}
git fetch {kernel_upstream_remote}
git checkout -b use-builder-{major}.{minor} {kernel_upstream_remote}/{major}.{minor}

2.2 Update builder_image in .gitlab-ci.yml

File: .gitlab-ci.yml

Location: Find the trigger pipeline section for the RHEL major version.

Action: Change builder_image from the ystream fallback to the zstream image:

Before:

.trigger_rhel{major}_pipeline:
  trigger:
    branch: rhel{major}
  variables:
    builder_image: quay.io/cki/builder-rhel{major}

After:

.trigger_rhel{major}_pipeline:
  trigger:
    branch: rhel{major}
  variables:
    builder_image: quay.io/cki/builder-rhel{major}.{minor}

2.3 Create and Submit Merge Request

The kernel repository requires a specific commit message format:

git add .gitlab-ci.yml
git commit -s -m "gitlab-ci: use rhel{major}.{minor} builder image

ubi{major}.{minor} has been released.

JIRA: INTERNAL
Upstream Status: RHEL only

Part of {issue_url}"
git push \
    -o merge_request.create \
    -o merge_request.draft \
    -o merge_request.target={major}.{minor} \
    -o merge_request.title="gitlab-ci: use rhel{major}.{minor} builder image" \
    -o "merge_request.description=ubi{major}.{minor} has been released.\n\nJIRA: INTERNAL\n\nUpstream Status: RHEL only\n\nPart of {issue_url}\n\nSigned-off-by: $(git config user.name) <$(git config user.email)>" \
    {kernel_fork_remote} use-builder-{major}.{minor}
cd -

IMPORTANT: The kernel repository enforces DCO/Signoff checks on both the commit message AND the MR description. The -s flag on git commit adds the Signed-off-by line to the commit. The MR description MUST also contain a matching Signed-off-by: line (shown in the push command above) or the cki-kwf-bot will reject the MR with CommitRules::NeedsReview / Signoff::NeedsReview. Use literal \n for newlines in the description since push options cannot contain raw newlines.

Success criteria: Command outputs MR URL.

STOP: Wait for pipeline and mark MR ready

  1. Wait for the MR pipeline to complete successfully
  2. Mark the MR as ready (remove draft status)
  3. The kernel stream maintainers will review and merge the MR on their own schedule

Report the MR URL to the user. The onboarding process is complete once the MR is submitted. The kernel maintainers will merge it when ready, after which the new builder image will be in use by the kernel CI pipelines.

Complete Example: Creating builder-rhel9.7

Assuming:

  • {issue_url} = https://gitlab.com/cki-project/cki-lib/-/issues/123
  • {containers_fork_remote} = fork
  • {kernel_repo_path} = ../kernel-rhel
  • {kernel_upstream_remote} = rhel-9
  • {kernel_fork_remote} = rhel-9-fork

Example: Phase 1

git checkout main && git pull

# Prerequisites check
curl -s -o /dev/null -w "%{http_code}" https://download.devel.redhat.com/
podman pull registry.access.redhat.com/ubi9/ubi:9.7
test -f files/rhel9.z.repo && echo "Repo file exists"

# Get the base image digest
DIGEST=$(skopeo inspect --no-tags docker://registry.access.redhat.com/ubi9/ubi:9.7 | jq -r .Digest)

# Create the Containerfile.in
mkdir -p builds/builder-rhel9.7
cat > builds/builder-rhel9.7/Containerfile.in << EOF
/* renovate: datasource=docker depName=registry.access.redhat.com/ubi9/ubi */
#define _BASE_IMAGE registry.access.redhat.com/ubi9/ubi
#define _BASE_IMAGE_TAG 9.7@${DIGEST}
#define _AWSCLI_PIP_SYSTEM 1
#define _RHEL_VERSION 9
#define _RHEL_MINOR_VERSION 7

#include "setup-from-rhel"
#include "builder"
#include "cleanup"
EOF

# Update cki-images.yml - add builder-rhel9.7 to the internal images list
# Find the section with builder-rhel9.6 and add builder-rhel9.7 after it
sed -i '/- builder-rhel9.6$/a\          - builder-rhel9.7' .gitlab/ci_templates/cki-images.yml

# Verify the change
grep -q "builder-rhel9.7" .gitlab/ci_templates/cki-images.yml && echo "CI file updated" || echo "CI update failed"

# Build
podman run \
    --rm \
    --pull=newer \
    --security-opt label=disable \
    -e IMAGE_NAME=builder-rhel9.7 \
    -w /code \
    -v .:/code:z \
    -v ~/.local/share/containers:/var/lib/containers:z \
    quay.io/cki/buildah:production \
    cki_build_image.sh

# Validate
podman run --rm localhost/builder-rhel9.7:latest-amd64 bash -c '
    set -e
    gcc --version && make --version && python3 --version
    which bison flex bc
    echo "int main(){return 0;}" > /tmp/t.c && gcc -o /tmp/t /tmp/t.c && /tmp/t
    echo "SUCCESS: All validation tests passed"
'

# Create MR
git checkout -b add-builder-rhel9.7
git add builds/builder-rhel9.7/Containerfile.in .gitlab/ci_templates/cki-images.yml
git commit -m "Add builder-rhel9.7 container image

https://gitlab.com/cki-project/cki-lib/-/issues/123"
git push \
    -o merge_request.create \
    -o merge_request.target_project=cki-project/containers \
    -o merge_request.target=main \
    -o merge_request.title="Add builder-rhel9.7 container image" \
    -o merge_request.description="Related issue: https://gitlab.com/cki-project/cki-lib/-/issues/123" \
    fork add-builder-rhel9.7

# STOP: Get MR reviewed and merged, wait for CI to publish images
# Verify images are available (replace 123 with actual MR number):
#   skopeo inspect docker://quay.io/cki/builder-rhel9.7:mr-123
#   skopeo inspect docker://quay.io/cki/builder-rhel9.7:production

Example: Phase 2

cd ../kernel-rhel
git fetch rhel-9
git checkout -b use-builder-9.7 rhel-9/9.7

# Edit .gitlab-ci.yml
# Find .trigger_rhel9_pipeline and change:
#   builder_image: quay.io/cki/builder-rhel9
# to:
#   builder_image: quay.io/cki/builder-rhel9.7

git add .gitlab-ci.yml
git commit -s -m "gitlab-ci: use rhel9.7 builder image

ubi9.7 has been released.

JIRA: INTERNAL
Upstream Status: RHEL only

Part of https://gitlab.com/cki-project/cki-lib/-/issues/123"
git push \
    -o merge_request.create \
    -o merge_request.draft \
    -o merge_request.target=9.7 \
    -o merge_request.title="gitlab-ci: use rhel9.7 builder image" \
    -o "merge_request.description=ubi9.7 has been released.\n\nJIRA: INTERNAL\n\nUpstream Status: RHEL only\n\nPart of https://gitlab.com/cki-project/cki-lib/-/issues/123\n\nSigned-off-by: $(git config user.name) <$(git config user.email)>" \
    rhel-9-fork use-builder-9.7
cd -

# STOP: Wait for pipeline to complete, then mark MR ready (remove draft)
# Kernel maintainers will merge on their own schedule - onboarding complete!

Troubleshooting

Build fails with network errors

Ensure you are connected to Red Hat VPN and can access download.devel.redhat.com.

UBI image not found

The UBI image for the target version may not be published yet. Wait for Red Hat to publish the UBI image before creating the builder.