Skaffold/Kustomize Automated Deployments in Production

This document outlines the requirements and procedures for executing automated deployments to the production environment using Skaffold and Kustomize in conjuction with the Jenkins Shared Library.

1. Requirements

For automated deployments to execute successfully, the following configuration requirements must be met.

1.1. Jenkins Configuration

The jenkins.yaml file must be updated with the following values to enable production deployments.

enableTagRelease: true

prod: # Production build environment overrides
  skip: true
  deployStageOnly: true # Disables all but the deploy stage
  deployNamespace: prod
  enableAutoDeploy: true
  enableManualDeploy: true

1.2. Project Manifest

Your project must contain a manifest.json file in the root directory with the following structure.

{
  "version": "vX.Y.Z-conf##",
  "NS": "prod"
}

Where vX.Y.Z is the semantic version and conf## represents the configuration iteration (e.g., conf01, conf02).

During the deployment, the semantic version (X.Y.Z) in manifest.json is strictly compared to the version in the parent pom.xml. The deployment will only proceed if these versions match.

2. Automated Deployment Trigger

An automated deployment is triggered under specific conditions:

  • The Jenkins job must be configured to scan Release/* branches.

  • A deployment is attempted only when a commit at HEAD of a scanned release branch includes a change to the manifest.json file.

There should only be a single Jenkins job for production, configured for this auto-deployment process.

3. SRE: Creating the Production Jenkins Job

This section is for SRE staff responsible for provisioning the production Jenkins job that performs the automated deployments described above. Each service gets exactly one production auto-deploy job, created as a Multibranch Pipeline in the production Jenkins instance (https://utility.apps.va.gov/prodjenkins/).

3.1. Create the Multibranch Pipeline Job

  1. In production Jenkins, select New Item.

  2. Enter the job name using the convention <service-name>-auto-deploy-mb (e.g., mobile-ad-service-auto-deploy-mb).

  3. Select Multibranch Pipeline and click OK.

3.2. Configure the Branch Source

Under Branch Sources, add a Bitbucket source and configure the following:

Field Value

Server

https://coderepo.mobilehealth.va.gov

Credentials

The shared Bitbucket service account credential for the Jenkins instance

Owner

The Bitbucket project key containing the repository (e.g., CKM). This varies by project.

Repository Name

The service repository (e.g., mobile-ad-service)

Under Behaviors, ensure the following are present:

  • Discover branches — strategy: Exclude branches that are also filed as PRs

  • Discover pull requests from origin — strategy: Merging the pull request with the current target branch revision

  • Discover pull requests from forks — strategy: Merging the pull request with the current target branch revision, trust: Forks in the same account

  • Filter by name (with regular expression) — see Branch Filter Regex below.

Under Build Configuration, leave the mode as by Jenkinsfile with script path Jenkinsfile.

3.3. Branch Filter Regex (Point-Forward View)

The name filter regex restricts the job to release branches from the current release forward. This prevents Jenkins from scanning (and potentially deploying) old release branches that predate the auto-deployment configuration, while automatically picking up all future release branches with no further job changes.

For example, if the current release version on main is 1.14, the regex is:

^[Rr]elease/1\.(?:1[4-9]|[2-9]\d|[1-9]\d{2,})(?:\.\d+)*$

How it reads:

  • ^[Rr]elease/ — matches Release/ or release/ branch prefixes.

  • 1\. — the major version (pinned to 1; update the pattern if the service ever bumps its major version).

  • (?:1[4-9]|[2-9]\d|[1-9]\d{2,}) — the point-forward minor version: 14–19, 20–99, or 100+.

  • (?:\.\d+)* — allows optional patch segments such as Release/1.14.2.

This matches Release/1.14, Release/1.15, Release/1.20, Release/1.100, Release/1.14.1, etc., and excludes Release/1.13 and anything older.

3.3.1. Using AI to Generate the Regex

The minor-version alternation is fiddly to write by hand, so use an AI assistant (e.g., Claude Code) to generate and verify it. A prompt like the following works well:

I am setting up a Jenkins multibranch pipeline job for autodeployments. I need a
regex for the "Filter by name (with regular expression)" field that gives a
point-forward view of release branches: only the current release branch and all
future ones. Branches are named like Release/1.14 (sometimes with a patch
segment, e.g. Release/1.14.2, and sometimes lowercase "release"). The current
release version is 1.14. Please verify the regex matches 1.14 and later
(including 1.20, 1.100, 1.14.2) and rejects 1.13 and earlier.

Substitute the service’s actual current release version. Ask the assistant to test the regex against real and hypothetical branch names before using it — a wrong filter either deploys old branches or silently matches nothing.

After creation, run Scan Multibranch Pipeline Now and confirm that only the intended release branches (current version and forward) appear in the job.

3.4. Enable the Bitbucket Webhook

Once the job has been confirmed working by triggering it manually, enable the webhook in Bitbucket so repository events are sent to Jenkins automatically. This allows branch scans and builds to trigger on push events instead of relying on manual scans or polling.

4. Production Release Process

Follow these steps to cut a new release for an automated production deployment.

  1. Update manifest.json

    Modify the version field in manifest.json for the new tag you intend to create. The configuration version should be higher than conf01, as conf01 is reserved for initial SQA deployments.

    {
  "version": "v1.0.0-conf02",
  "NS": "prod"
}

  2. Create a Git Tag

    Add an annotated tag pointing to the commit where manifest.json was updated.

    git tag -a v1.0.0-conf02 -m "Tag for production release of v1.0.0 using configuration v1.0.0-conf02"

  3. Push Changes

    Push both the commit and the new tag to the remote repository.

    git push --follow-tags

5. Deployment Safeguards

The JSL autodeployment script ensures all of the following conditions are true before proceeding:

  • The environment it is running in is production.

  • The branch being executed against matches the Release/* pattern.

  • The commit at HEAD contains a change to the manifest.json file.

  • The semantic version in manifest.json matches the version in pom.xml.

If any of these conditions are not met, the production deployment will be aborted.

6. Rollback Instructions

If a rollback is necessary, you have two options.

6.1. Option 1: Git-based Rollback

This is the preferred method for rolling back.

  1. Update the manifest.json file to the version of the git tag you want to restore.

  2. Commit and push this change to the Release/* branch to trigger a new deployment with the old version.

6.2. Option 2: Manual Jenkins Rollback

This option can be used as an alternative.

  1. Navigate to the Jenkins job and select "Build with Parameters".

  2. Enter the specific git tag you wish to roll back to in the appropriate parameter field and trigger the build manually.