Canary Deployment Guide (Deprecated)

Canary deployments are utilized as a way to stage a release. Using a weighting system, user load is gradually moved from an existing deployment to a canary deployment. This methodology is used to prove out new features and functionality found in the canary deployment, but still have the safety of a proven existing release to fall back on. The idea is to increase the canary weight and decrease the existing deployment’s weight as features are vetted. Once the canary deployment is weighted at 100%, it effectively becomes the new release, and the existing deployment can be removed.

Prerequisites

  • Canary deployments are currently only supported for the SQA/staging environment. Creating a canary deployment will generate the required artifacts configured for a deployment to the SQA/staging environment.

  • Canary deployment artifacts are currently only supported via Kustomize. In order to begin creating canary deployments, a service must first transition to using Kustomize generated K8s artifacts. This is normally accomplished by setting the kustomize.enable parameter on build.
    However, since canary deployments are only supported for the SQA/staging environment, you must build the application with the appropriate overlay applied using the kustomize.overlay parameter, which also enables kustomize.
    So to create a kustomize-enabled build for SQA, you would execute

    mvn clean install -Dkustomize.overlay=kubernetes/overlays/deploy/staging

  • Prior to performing a kustomize enabled build, the kustomize artifacts must be initialized using the kustomize.init flag. This will create the service local directories and artifacts used by Kustomize.

    mvn package -Dkustomize.init

  • Kustomize generates all-kustomized.yaml, which contains all of the K8s artifact definitions, including deployment.yaml, service.yaml and horizontalpodupscaler.yaml artifacts. In order to deploy correctly, these artifacts should be removed and replaced with all-kustomized.yaml. There must also not be any other files in the kubernetes directory due to the way the Jenkins deployment container entrypoint’s command is executed. It will iterate over every file in the directory (including subdirectories) and attempt to create Kubernetes objects from each file, which is not valid and will cause a deployment failure.

Enablement

The following steps should be followed in order to deploy a canary deployment:

  1. Build and deploy the service with Kustomize enabled (SQA/staging):

    • Initialize the service to work with kustomize and create the necessary templates and directories using

      mvn package -Dkustomize.init

    • Enable Kustomize and build for SQA/staging using

      mvn clean install -Dkustomize.overlay=kubernetes/overlays/deploy/staging

      • Ensure that any existing deployment.yaml, service.yaml and horizontalpodupscaler.yaml files are removed and replaced with all-kustomized.yaml, which contains the kustomize-generated service, deployment and horizontal pod upscaler configuration.

  2. Build and deploy the service with canary deployments enabled

    • To support canary deployments, the canaryDeploy build flag was added that enables Kustomize and sets the kustomize.overlay directory to kubernetes/overlays/deploy/canary. Execute using

      mvn clean install -DcanaryDeploy

    • The following parameters will also be added to the Jenkinsfile when using the canaryDeploy option:

      • CANARY_DEPLOY: flag to enable/disable canary deployment

      • PRIOR_V: prior deployment version; defaults to the canary version, but this should be manually changed to the version number of the deployment that the canary deployment is to run next to. If left as the canary deployment’s version number, then all traffic will default to the canary deployment on first deploy.

      • PRIOR_WT: prior deployment weight, expressed as a numeric value 0-100 representing the percentage of traffic that should route to the prior deployment; defaults to 100

      • CANARY_WT: canary deployment weight, expressed as a numeric value 0-100 representing the percentage of traffic that should route to the canary deployment; defaults to 0

      • REMOVE_PRIOR: flag to set to remove the prior deployment if PRIOR_WT is set to 0

    • Destination Rule and Virtual Service configuration is added to all-kustomized.yaml. These are used for traffic routing based on the weight values set during deployment.

Troubleshooting

  • No traffic is routed to the canary deployment

    • Ensure deployment has CANARY_DEPLOY set to true.

    • Ensure CANARY_WT has a non-zero value and PRIOR_WT is not set to 100.

  • All traffic is routed to the canary deployment on first deploy

    • Ensure PRIOR_V is not set to the same version as the canary deployment (DEPLOY_VERSION) value.

  • Canary parameters do not show in Jenkins

    • When first switching to a canary deployment from a standard deployment, the new canary-specific parameters will not be loaded into Jenkins until after the first deployment. When this occurs, execute another deployment, and the parameters will be present.

  • Canary deployment fails on startup

    • Ensure that only all-kustomized.yaml is present in the service’s kubernetes directory