Kustomize (Experimental Preview)

Kustomize can now be used as an optional Kubernetes manifest generation mechanism. It will eventually replace the current proprietary Kubernetes generation method as well as the seperate test-only Helm-based Kubernetes configuration to better align with industry-standard practices.

The Kustomize patches and overlays in the kubernetes directory are applied only if the kustomize.enable flag is present or the kustomize.overlay parameter is set, which overrides the default overlay location (kubernetes/overlays/dynamic).

Usage

To create the required directory structure and download the initial Kustomize files from the ckm-deployment-templates repository, build with the kustomize.init flag:

# initialize Kustomize directories and files
mvn package -Dkustomize.init

Once any desired changes have been made, build the application with the kustomize.enable flag:

# render Kubernetes manifests to 'all-kustomized.yaml' using Kustomize (with the default Kustomize overlay at 'kubernetes/overlays/deploy/dynamic')
mvn clean install -Dkustomize.enable

Or, if a different overlay is to be rendered to the all-kustomized.yaml file, specify the overlay path (relative to the project root) with the kustomize.overlay parameter:

# render Kubernetes manifests to 'all-kustomized.yaml' using Kustomize with the specified overlay location
mvn clean install -Dkustomize.overlay=kubernetes/overlays/deploy/staging-with-canary

Processing Model

Initialization

The plugin’s KubernetesMojo will pull down template files from the remote kustomize directory (or from the local templates/kubernetes directory), replace project-specific placeholder variables within each file, and copy the resulting files into the local kubernetes directory.

For instance, the ckm-deployment-templates kustomize/base/deployment.yaml contains the following Deployment image spec:

template:
  spec:
    containers:
    - image: dtr.mapsandbox.net/${dtrOrg}/${artifactId}:${version}

The base DTR is set to dtr.mapsandbox.net because this is used both locally and in the first test environment. Other environments override the image value, setting the DTR as needed. When the -Dkustomize.init option is included, the properties surrounded by ${...} are replaced with the values from the Maven project pom.xml as well as from the new local template properties file located at templates/kubernetes/kubernetes.properties (if it exists).

Unlike before, these template files are only used to initialize the local files when building with the kustomize.init flag, and will not overwrite the project-local Kustomize files automatically for each build. Eventually all deployment-related elements currently specified in the project metadata.yaml file will be removed in favor of specifying these values directly in the Kubernetes and Kustomize files themselves.

Build

Once the "hard part" of the developer creating and updating Kustomizations for the project have been completed, the ci-file-generator executes a simple command during the Maven build to render the desired Kustomization (default kubernetes/deploy/dynamic) to the file system (default kubernetes/all-kustomized.yaml):

# ci-file-generator executes this command "behind the scenes" during its build when Kustomize is enabled
kubectl kustomize kubernetes/deploy/dynamic -o kubernetes/all-kustomized.yaml

This command is invaluable for development and debugging of project Kustomizations, and does not require a full Maven rebuild of the project to test each change.

The output file includes all Kubernetes resources generated during the Kustomization process, unlike previously where each Kubernetes resource is specified in a separate file (i.e., deployment.yaml, service.yaml, etc.).

Overlays and Components

Central to Kustomize is the concept of the "overlay" which is just a collection of resources and transformations on those resources (including "patches") that, when rendered and applied to a Kubernetes cluster, have the necessary configurations set to operate successfully within a specific environment or for a particular usage. The project’s Kustomize overlays are located in the kubernetes/overlays directory.

A Kustomize component is a relatively new construct that allows a set of resources and patches to be re-used among multiple overlays so that common configurations and transformations do not need to duplicated. These are located in the kubernetes/components directory.

The base directory contains the common Kubernetes resource definitions that applies to all usages and environments and can be referenced by overlays to change aspects of these common resources.

The deploy and jenkins directories are parent directories grouping overlays that relate to deployment and CI test build configurations, respectively. The directory names and contents are modeled after the recommended directory layout defined in https://github.com/kubernetes-sigs/kustomize#2-create-variants-using-overlays.

<project-root>/
├─ kubernetes/
│  ├─ base/                         <- Base k8s manifest templates
│  ├─ components/                   <- Re-usable configurations that can be applied to any overlay
│  │  ├─ canary/                    <- Istio Canary-specific resources and configurations
│  │  ├─ configmap-env-vars/        <- Adds ConfigMap-based environment variable management
│  │  ├─ consul-env-vars/           <- Adds Consul-based environment variable management
│  │  ├─ hpa/                       <- Adds horizontalpodautoscaler.yaml and related configurations
│  │  └─ vault-env-vars/            <- Adds Vault-based environment variable management
│  ├─ overlays/                     <- Standard directory for environment- and usage-specific kustomizations
|  │  ├─ deploy/                    <- Static environment deployment-related Kustomizations
|  │  │  ├─ dynamic/                <- Kustomization that renders standard k8s manifests (output to single 'all-kustomized.yaml' file)
|  │  │  ├─ production/             <- Production-specific Kustomization
|  │  │  ├─ sandbox/                <- Sandbox-specific Kustomization
|  │  │  ├─ staging/                <- SQA/Staging-specific Kustomization
|  │  │  └─ staging-with-canary/    <- SQA/Staging-specific Kustomization for Istio canary deployment configuration
│  │  ├─ dev/                       <- Local environment Kustomization for single service
│  │  ├─ dev-with-deps/             <- Local environment Kustomization for service and its dependencies (ideally replacing Helm-based test deployments)
│  │  └─ jenkins/                   <- CI build Kustomizations
|  │     ├─ sandbox/                <- Jenkins Sandbox Kustomization
|  │     └─ staging/                <- Jenkins SQA/Staging Kustomization
│  ├─ patches-optional/             <- Collection of useful Kustomize patches that can be copied and updated where needed
│  ├─ all-kustomized.yaml           <- The rendered output of the Kustomization specified (defaults to 'overlays/dynamic')
│  ├─ deployment.yaml               <- "Legacy" k8s deployment resource (to be deprecated)
│  ├─ horizontalpodautoscaler.yaml  <- "Legacy" k8s HPA resource (to be deprecated)
│  └─ service.yaml                  <- "Legacy" k8s service resource (to be deprecated)