Kustomize/Kubernetes

From the Kustomize introduction page:

Kustomize provides a solution for customizing Kubernetes resource configuration free from templates and DSLs.

Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

Kustomize targets kubernetes; it understands and can patch kubernetes style API objects. It’s like make, in that what it does is declared in a file, and it’s like sed, in that it emits edited text.

Kustomize allows us to define Kubernetes configurations for each service in a DRY ("Don’t Repeat Yourself") manner across different deployment environments, including local development and testing. This approach enhances Dev/Prod parity, a key principle of the Twelve-Factor App methodology, by enabling us to specify only the configuration differences between environments rather than duplicating entire configurations.

Overlays and Components

Central to Kustomize is the concept of the "overlay" which is just a collection of resources and transformations on those resources (including "patches") that, when rendered and applied to a Kubernetes cluster, have the necessary configurations set to operate successfully within a specific environment or for a particular usage.

A Kustomize component is an alpha-version construct that allows a set of resources and patches to be re-used among multiple overlays so that common configurations and transformations do not need to be duplicated. Unlike the Kustomize resource, components do not validate each referenced resource before applying the specified transformation(s), and so are much more flexible at applying ad-hoc changes to configurations, albeit at the expense of making order matter, in terms of where in the components list each component is specified.

NGSS Kustomize Conventions

Since Kustomize is extremely flexible (by design), there are many ways to define a project’s Kustomize "architecture" in terms of file/directory structure, naming, and configuration specification. For consistency and comprehensibility across projects, as well as being able to easily integrate with how Skaffold works, we have defined a set of conventions and guidelines for defining and managing Kustomize/Kubernetes configurations.

Directory/File Layout

The directory names and contents are modeled loosely after the recommended directory layout defined in https://github.com/kubernetes-sigs/kustomize#2-create-variants-using-overlays.

📁 my-service
  📁 kustomize
  📁 base                                     <- Base k8s manifest templates containing the minimal Kubernetes configs
    📄 application.env
    📄 deployment.yaml
    📄 kustomization.yaml
    📄 service.yaml
  📁 components                               <- Re-usable configurations that can be applied to any overlay
    📁 dev                                  <- The main service local overlay, used for testing and including as a dependency
      📄 application.env
      📁 db                               <- Optional database-specific patches for service DB configs
        📄 README.md
        📄 add-db-config.yaml
      📁 wiremock                         <- Optional wiremock-specific patches for service wiremock configs
        📄 README.md
        📄 add-wiremock-config.yaml
      📄 deployment.yaml
      📄 kustomization.yaml
      📄 service.yaml
    📁 remove-resource-constraints          <- Removes all deployment resource CPU and memory configs
      📄 kustomization.yaml
    📁 secrets                              <- Adds ConfigMap entries designated as secrets for special handling
      📄 application-secrets.env
      📄 kustomization.yaml
    📁 vault                                <- Sets up Vault Kubernetes configs for testing Vault-based deployment
      📄 kustomization.yaml
      📄 service.yaml
      📄 statefulset.yaml
    📁 vault-populator                      <- Sets up the deployment to populate designated secret entries into Vault
      📄 application.env
      📄 deployment.yaml
      📄 kustomization.yaml
      📄 populate.sh
  📁 debug-all                                <- Overlay that adds JVM-based configs to enable all services deployed to be remotely debugged
    📄 kustomization.yaml
  📁 dev-with-dependencies                    <- Overlay that includes dependent service overlays (usually '/components/dev') with deployment
    📄 kustomization.yaml
  📁 dev-with-dependencies-vault              <- Overlay that inherits from 'dev-with-dependencies' but with Vault enabled
    📄 application-vault.env
    📄 kustomization.yaml
  📁 localized                                <- local "cache" of the overlays which avoids re-fetching remote git resources for every build
  📁 prod                                     <- Overlay used in the deployment to Production
    📄 application.env
    📄 deployment.yaml
    📄 dtr.properties
    📄 horizontalpodautoscaler.yaml
    📄 kustomization.yaml
  📁 sandbox                                  <- Overlay used in the deployment to Sandbox
    📄 application.env
    📄 deployment.yaml
    📄 kustomization.yaml
  📁 sqa                                      <- Overlay used in the deployment to Staging
    📄 application.env
    📄 deployment.yaml
    📄 dtr.properties
    📄 horizontalpodautoscaler.yaml
    📄 kustomization.yaml
  📁 sqa-build-test                           <- Overlay used in build and integration tests in SQA
    📄 dtr.properties
    📄 kustomization.yaml

The base directory contains the common Kubernetes resource definitions that applies to all usages and environments and can be referenced by overlays to change aspects of these common resources.

The deploy and jenkins directories are parent directories grouping overlays that relate to deployment and CI test build configurations, respectively.

As a convention, the dev-with-dependencies overlay is used since its kustomization.yaml is specifically constructed for local development, and so includes the deployment of runtime service dependencies necessary for the correct operation of the service under development.

Kustomize-based Service Dependencies

Each dependency that has been updated to define Kustomize overlays can be included as kustomization.yaml resources in the dev-with-dependencies overlay using their Git repository and branch/tag info:

dev-with-dependencies kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Include simple Kubernetes resources (each resource's references are validated pre-render)
resources:
- https://coderepo.mobilehealth.va.gov/scm/ckm/common-app-config.git//dev?ref=main

# Include components that may have interdependencies (each component's references are NOT validated pre-render)
components:
- https://coderepo.mobilehealth.va.gov/scm/iums/mobile-mvi-service.git//kubernetes/overlays/dev?ref=Release/1.19.0 (1)
- ../components/dev (2)
# ...
1 Each service dependency can be added as a remote Kustomization using the Git URL, the directory path to the Kustomize overlay, and the ref, which can be a branch or tag. This overlay does not get referenced by other service kustomizations, so it would include any "test/debug" changes applied for this project (see below)
2 This is the "base" dev Kustomization which includes just the changes needed to deploy to a local k8s cluster. It contains the changes necessary to make the service operational for local development and testing. This usually includes necessary configuration overrides for the service and its dependencies

Temporary vs Permanent Development Changes

When making changes to deployment configurations during local development, consider whether the change should be temporary or permanent:

Temporary changes for testing/debugging should be made in dev-with-dependencies/kustomization.yaml. For example, to use a locally-built image of a dependency instead of the DTR-hosted one (or vice-versa), add an entry to the images stanza:

dev-with-dependencies/kustomization.yaml addition
images:
  - name: dtr.mapsandbox.net/ckm/jwt-signing-service
    newTag: '1.0.0-local'

Permanent changes that are necessary for the service to operate successfully in local development (such as mock data, local database configurations, or environment-specific settings) should be made in components/dev/kustomization.yaml instead:

components/dev/kustomization.yaml addition
configMapGenerator:
- behavior: merge
  envs:
  - overrides.env
  name: my-service-app-config
- files:
  - db/sql/001_myservice_ddl.sql
  - db/sql/002_myservice_metadata-table.sql
  name: my-service-db-config
For DB and/or mock data configurations, include the provided scripts/add-configmap-files.sh script as a skaffold.yaml pre-render hook. An example is commented-out in the skaffold.yaml template.

You can add, update, and remove any Kubernetes configuration using Kustomize transformations in a similar manner - however, most of the time these changes will be defined as a patch.

Overriding (Patching) Resources

Upon building and deploying the service and its dependencies, there may be issues that require aspects of their configuration to be overridden in the local environment. When working with Helm, these overrides were handled by setting values via the command-line or YAML files, generated from the original metadata.yaml. However, Kustomize works in a completely different, and much more flexible way (see https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/ for reference).

Any "overrides" needed are to be made with a patch, and then included in the kustomizaton.yaml. Patches can be one of two types, strategic merge or JSON 6902.

It is more natural to define a strategic merge patch since it mirrors the format of the Kubernetes resource to be patched, and so a strategic merge patch is preferred if feasible. For instance, here is a patch that updates the name of the imagePullSecret for my-service:

add-image-secrets.yaml (Strategic merge patch)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service # provide the resource name to be patched, and namespace is not needed
spec:
  template:
    spec:
      imagePullSecrets:
        - name: new-secret-name
kustomization.yaml to include the patch
patches:
- patches/add-image-secrets.yaml

In comparison, if we wanted to update a Helm chart’s imagePullSecrets, we would have to first define a template variable if it wasn’t already defined, and then override this value in an override YAML file.

If it is overly verbose or simply impossible to define a strategic merge patch given a requirement, specify a JSON 6902 patch instead (in YAML or JSON format):

add-image-secrets.yaml (JSON 6902 patch)
- op: replace
  path: /spec/template/spec/imagePullSecrets/0/name
  value: new-secret-name
kustomization.yaml to include patch
patches:
- target:
    version: v1
    kind: Deployment
    name: my-service
  path: add-image-secrets.yaml

Note that if defining a JSON 6902 patch that the resource name, kind, and version are required in the target element.

In the course of building, deploying, and testing, there will inevitably be issues that require patching to either the service itself or its dependencies. Understanding how Kustomize works is necessary to being able to fix these issues, especially when you don’t have control over how the dependencies' Kustomizations are configured by default.