Engineering Practice · CI/CD DetailsDocument version v2026100501
SRE · Engineering Practice · Implementation

CI/CD Details

Document version v2026100501 (quote this version when discussing or citing) · 中文版. For engineers in every department and for SRE. The Release Contract & Branching Template says what; this document says how. Sections marked contract describe what SRE does with a delivery and apply to everyone; sections marked template describe a recommended app-repo pipeline that teams may adopt or replace.

1. At a glance

SRE deploys a tag + commit hash

The delivery is one tag and its full commit hash per repo. SRE checks that they match, then deploys exactly that.

Build once

The artifact is built from the delivered tag and identified by its commit. Gray and production run the same artifact.

Environment settings are injected at deploy time

At build time CI has no environment's settings, so the image must not contain domains, API addresses or toggle values.

Approval in GitLab, deployment by the platform

SRE approves deployment requests in GitLab; K8s is then synced by GitOps and VMs by a deployment job, both invisible to the requester.

2. From delivery to deploy (SRE side) contract

  1. D-1The Release Manager opens one deployment request per repo: tag, full commit hash, release plan, revert method.
  2. CheckSRE (or a job) resolves the tag in the app repo and compares it with the delivered hash (git rev-parse <tag>^{commit}). A mismatch is not deployed until the Release Manager confirms a corrected pair.
  3. GrayRelease day: SRE approves the gray change for the whole set; the platform deploys; SRE observes real traffic. Gray runs from 11:00 to 14:00, any number of rounds; a problem in any repo → the whole set is withdrawn from gray and re-delivered. Gray ends at 14:00; not passed → next release day.
  4. ProductionThe whole set passed gray: SRE approves the production change for each repo in release-plan order, with the same artifact digest as gray.
  5. RollbackEach repo back to its previous version in reverse release-plan order, or a config-only change that switches a toggle off.

The environment repo is the git repo that records which version each environment runs and with which settings (currently fleet-delivery). The GitOps tool (Argo CD) syncs each environment to what the repo says. It covers services on K8s today; VM services will be added later. A deployment request is an MR on it: SRE's approval is the deployment decision, and the merge is the deployment.

Environment repo pathContentApproved byAfter approval
workloads/<svc>/<self-verification env>/Rendered K8s manifestsAuto-mergedGitOps sync
workloads/<svc>/gray/Same, version = delivered tag (gray, real traffic)SRE (CODEOWNERS)GitOps sync
workloads/<svc>/production-<cluster>/Same, version = the tag that passed graySRE (CODEOWNERS)GitOps sync
workloads/<svc>/production-vm/version.yaml (tag) and config filesSREA deployment job on an on-prem runner runs git checkout <tag> on the VM and restarts it
Same path, settings onlyFeature toggles, environment variables; artifact unchanged, marked config-onlySRE, on the Release Manager's requestSame; services that read env get a rollout restart
RollbackSRE, in reverse release-plan order, deploys each repo's previous version. K8s services: set the production version back (the Revert button on the deployment MR, or a new MR back to the previous tag); VM services: for now the existing deployment scripts, to be moved into the environment repo later. Can become a manual job rollback-release that takes the release plan and rolls back repo by repo in reverse orderAnnounced and executed by SREEach repo back to its previous version, one by one
Why the approval point is in GitLab

Argo CD syncs automatically and has no concept of approval; most departments are not on K8s either. Approval in a GitLab MR puts K8s and VMs through the same gate.

VMs and K8s look the same

SRE always sees "the production version of some service changes from A to B". Moving a service from a VM to K8s changes the content of the same directory; the process stays the same.

Platform prerequisites

The environment repo's production directories are not yet connected to the production clusters; the ApplicationSet's target cluster must be derived from the directory; main needs CODEOWNERS and code owner approval; the tag ↔ hash check and rollback-release jobs are not built yet. SRE completes these.

3. Pipeline (app repo) template

A shared CI/CD component that fits the branching template; each repo only provides its test command, Dockerfile and service name. Three paths: ordinary MR / push, rc tag, and the release tag after the merge into master.

MR / PUSH
test

the stack's test command

MANUAL
deploy-dev

deploy to a self-verification environment

MR → MASTER
check-merge

content == rc in production

RC TAG
tag-verify

rc is on release/* or hotfix/*

RC TAG
build

build once, record the digest

RC TAG
request-deploy

open the deployment request: tag, hash, digest

RELEASE TAG
tag-verify

== master tip, content == rc in production

RELEASE TAG
retag

add alias X.Y.Z to the same image

RELEASE TAG
release-notes

add digest and MR list to the Release

RELEASE TAG
cleanup

delete release and merged feat branches

Figure 1 — Top: every MR / push. Middle: the Release Manager creates the rc tag; build once and open the deployment request (from here on, section 2). Bottom: after production is verified, the Release Manager merges into master and creates the release tag; the release tag neither builds nor deploys. Gold marks the guards.
JobWhen it runsWhat it does
testMR, branch pushThe stack's test command (section 5).
deploy-devManualBuilds an image of the current commit and deploys it to a self-verification environment chosen by the developer.
check-mergeMR → masterGuard, see section 4. Pure git, no API calls.
tag-verifyFirst step of the tag pipelineGuard, see section 4. If it fails nothing after it runs.
buildrc tagBuilds the image, registry tag = commit, digest passed on. The only build for gray / production.
request-deployrc tagOpens (or updates) the deployment request on the environment repo with tag, full commit hash and digest; the Release Manager adds the release plan and revert method. The rest is SRE's (section 2).
retagRelease tagcrane tag / skopeo copy adds the alias X.Y.Z to the image of the rc in production; nothing is redeployed.
release-notesRelease tagUpdates the GitLab Release: digest, the matching rc, the Milestone's MR list.
cleanupRelease tagDeletes release/X.Y.Z (or hotfix/*) and the merged feat/* branches.

4. Guard jobs (app repo) template

Two jobs that keep the template's promises without relying on memory: "production runs the tag that was delivered" and "master only takes verified code". They run in the team's repo and protect the team; SRE's own check is the tag ↔ hash comparison in section 2.

GuardConditionsWhen it fails
check-merge
MR → master
① The source is release/* or hotfix/*; ② the source's latest commit is exactly an rc tag (the one in production, with no commits after it); ③ master is an ancestor of that commit.①: a branch that should not enter master. ②: something was pushed after production; deliver a new rc. ③: something else reached master meanwhile; merge master in first and create a new rc.
tag-verify
tag pipeline
rc tag: the commit is on release/* or hotfix/*. Release tag: the commit is the tip of master, its file tree is identical to some rc (git diff --quiet rc final), and that rc is its ancestor.The tag was created in the wrong place. Delete and recreate it before delivery; nothing was deployed.
Why compare file trees rather than commits: the release is merged into master with a merge commit, so master's latest commit is not the rc's commit, but the content is the same. The release tag reuses the artifact of the rc in production.

5. Three stacks template

Whatever the pipeline, four things differ per stack: how to test, how to build, what the artifact is, and how settings get in. "Settings injected at deploy time" is part of the contract (build once); the rest is guidance.

Next.jsJava Spring BootLaravel
testnpm ci && npm test, npm run lint, tsc --noEmit./mvnw -B verify or ./gradlew testcomposer install, vendor/bin/phpunit
buildmulti-stage: npm ci → next build (output: 'standalone') → runtime image carries only .next/standalonemulti-stage: mvnw package → JRE image with the jar; or buildpacks / jibmulti-stage: composer install --no-dev, config:cache → php-fpm + nginx image
ArtifactContainer image; shared components published as npm packagesContainer image; shared components published as jarsContainer image; shared components published as composer packages
Settings injectionServer side reads env at runtime. Client side must not put environment values in NEXT_PUBLIC_* (baked into the bundle at build time): use same-origin relative paths, a server-provided /api/config, or an env.js generated at container startRuntime env → application.yaml placeholders, or a ConfigMap mounted as application-prod.yamlRuntime env → config/*.php (after config:cache only config is read)
Feature togglesServer side reads env, or the client fetches /api/config; the OpenFeature JS SDK can unify bothRead via Environment at runtime, or Togglz / OpenFeature; @ConditionalOnProperty is evaluated only at startupconfig('features.x') from env; with a package, point its driver at env, not the DB
Restart on setting change?Reading env: yes; reading a mounted file: noYes (default); with Spring Cloud Kubernetes reload: noYes (config:cache)
HealthCustom /api/healthActuator /actuator/health/readinessLaravel 11 /up
DB migrationUsually none; Prisma / Drizzle as on the rightFlyway / Liquibase, expand / contractartisan migrate in a Job / init container; migrations are not rolled back
Version observabilityImage label org.opencontainers.image.revision=<commit>; the app exposes /version returning the commit, so SRE can confirm in gray and production that the delivered hash is running.
Values that truly can only be decided at build time (for example Next.js NEXT_PUBLIC_*) are treated as code, not environment settings: they follow the commit and do not differ by environment.

6. GitLab settings template

App repos (template — recommended group-level defaults)

SettingValue
Protected branchesmaster, release/*: push = No one, merge = Maintainers.
Merge methodMerge commit. Move work between branches by merging whole branches through MRs, not by cherry-pick.
SquashProject setting "Allow", unchecked by default (GitLab sets this per project, not per target branch). Use it for feat/*, hotfix/* → release/* only; release/* → master with squash is blocked by check-merge / tag-verify. A squashed feat branch is not recognised by git as merged, so cleanup relies on "delete source branch after merge".
Protected tagsv*: only Maintainers (the Release Manager) can create.
Push rulesBranch names ^(feat|hotfix|release)/; commit message contains a ticket number #\d+; author email @fmt.com.tw (existing).
MRDelete source branch after merge; pipeline must succeed; master and release/* as target need approval.
Milestone / ReleaseOne vX.Y.Z per version; MRs are attached to the Milestone; Releases are created by the Release Manager, with notes generated automatically.
RegistryImmutable tags; the retention policy only cleans commit tags, never version tags.
RunnerJobs that call the GitLab API use the company's group runners (tags:); pure-git checks use GIT_FETCH_EXTRA_FLAGS.

Environment repo (SRE — one for the whole company) contract

SettingValue
Protected branchmain: push = No one; merge = Maintainers + each department's bot; Code owner approval required.
CODEOWNERS/workloads/*/production*/, /workloads/*/gray/ → the sre group. CODEOWNERS only; no extra approval rule over all of main.
Merge methodMerge commit (the Revert button rolls back a whole deployment).
BotOne service account per department (Developer + allowed to merge main), token in the department's group CI variable; changes are submitted with git commit + push, committer <svc>-gitops-bot@fmt.com.tw.
Deploy freezeGitLab deploy freeze; during a freeze requests can still be opened but are not approved.

7. FAQ

We do not use the template pipeline. What do we need?

Only the delivery: one tag + full commit hash per repo by D-1, an artifact built from that commit with settings injected at deploy time, and the deployment request (section 2). How you produce it is up to you.

A job calling the GitLab API gets 403 / 404?

The top-level group has an IP allowlist; gitlab.com SaaS runners are not on it. Point the job at the company's group runners (tags:). Pure-git checks need no API; use GIT_FETCH_EXTRA_FLAGS so the runner fetches the refs itself.

The bot gets 400 writing the environment repo through the commits API?

The push rule requires an @fmt.com.tw author email and the bot account's email is not. Use git commit + push with committer <svc>-gitops-bot@fmt.com.tw.

Run pipeline with variables is rejected?

New projects default to ci_pipeline_variables_minimum_override_role = no_one_allowed. Set it to maintainer when needed, or use pipeline inputs.

Why can a Maintainer push directly to master in a new repo?

When GitLab auto-protects the default branch it gives push to Maintainers. Change it to No one after creating the repo.

Is a protected tag really undeletable?

It blocks git push --delete; a Maintainer can still delete it from the UI / API. "Tags are immutable" is enforced by process, not by this setting.

The rollback revert returns 400 through the API?

The commits /revert API does not accept merge commits. Use the Revert button on the MR page, or git revert -m 1.

A rollback MR opened by SRE needs no second approver?

An MR opened by a code owner counts as satisfying the code owner rule, which keeps rollbacks fast. If two people must be required, add a separate approval rule.