Build Tools Revamp Roadmap

Background

Currently, the main source of NGSS project configuration has been mostly consolidated into the metadata.yaml, which configures aspects of:

  1. Continuous Integration (CI): build/publish - controls the static generation of Jenkinsfiles (one for Sandbox and one for SQA/Production), which builds and tests project artifacts, then publishes them to the Maven repository and/or Docker registry

  2. Deployment (CD): deploy - controls the static generation of Kubernetes manifests used for application deployment to various environments including Production

  3. Development: helm - controls the static generation of Helm chart resources and resolution and retrieval of service dependency resources solely to support verification via local and Jenkins-based build and integration testing

  4. General project requirements/info: documentation - controls the static generation of data needed for the application to operate as well as human-readable documentation

The common theme with these categories is "static generation" via custom, internal Maven plugins.

When the application complexity was relatively low and composition was simple of the projects, and when there were fewer projects, static generation fit well, since there wasn’t much "branching" logic needed, and for the most part, it worked consistently. But, for each change made to the build-tool stack to fix or enhance its operation, applying these upgrades to each and every project proves to be time-consuming and error-prone, especially when an unplanned/untested configuration caused a particular project build or deployment to fail. There seems to always be at least one if not more issues that arose during the upgrade process, and so the tools need to therefore be patched and re-released, causing a seemingly endless churn of upgrade/build/test/fix.

At first, consolidating the configuration (as much as possible) to one project file - metadata.yaml, instead of a set of Maven properties and plugin configurations, seemed to be a step forward for ease-of-specification and readability compared with using Maven pom.xml properties, and the YAML format enabled a built-in mechanism for grouping the different categories of configuration in a much more comprehensive way.

However, defining a "single source of truth" for these various tools requires a considerable amount of abstraction between the vocabulary of the metadata.yaml and the resulting configuration (post-transformation) of each internal tool. Developers, DevOps engineers, and other stakeholders not only needed to learn the syntax and usage of the metadata.yaml, but also learn how each element maps to each configuration of the custom tools, and also what the tool does when configured with those values. The complexity is multiplied in each of these steps, decreasing the ability of the people using the tools to understand how to effect the change(s) needed using the custom file format and Maven configurations provided. Frameworks like jersey-service-parent attempted to decrease the apparent complexity by providing another abstraction via a pre-configured parent POM, and ci-file-generator and helm-maven-plugin attempted to assist the developer in providing "starter" templates for their respective tools: Jenkins and Helm. This helps, but having to manage and understand how to use these templates adds yet another factor to the complexity of the system.

Functional Decomposition

Since there are multiple aspects that contribute to complexity and non-comprehensibility of the build tool stack’s definition and operation, each aspect should be dealt with individually. These components and their corresponding decomposition actions should consist of the following:

  1. Move from bespoke, custom tools to industry-standard tools, where feasible

    1. helm-maven-plugin → Skaffold + Kustomize + Service Dependency Resolver Script (optional)

    2. ci-file-generator Jenkins Pipeline generation → Jenkins Shared Library

    3. ci-file-generator Kubernetes manifest generation → Kustomize

  2. Move from the "single source of truth" abstracted configuration in metadata.yaml to specifying each configuration according to the tool being used.

    1. metadata.yaml/helm → skaffold.yaml

    2. metadata.yaml/helm/dependencies → service-dependencies.yaml

    3. metadata.yaml/build → jenkins.yaml

    4. metadata.yaml/deploy → kustomization.yaml

  3. Handle common organizational project needs by moving from embedded template processing to more visible transformation via "one-off" external scripting and/or documented conventions

    1. Helm chart interpolation and values population from metadata.yaml during build → Kustomize + Skaffold starter templates

    2. ci-file-generator Jenkinsfile generation from metadata.yaml during build → jenkins.yaml starter templates

    3. ci-file-generator Kubernetes manifest generation from metadata.yaml during build → Kustomize starter templates

Separating Responsibilities

As described above, two custom tools (helm-maven-plugin and ci-file-generator's KubernetesMojo) can be migrated to two standard open-source, well-maintained tools (Skaffold and Kustomize), in addition to the custom transitive dependency logic, which requires a new script or component temporarily referenced here as Service Dependency Resolver Script. In doing so, metadata.yaml can be completely removed as a necessary configuration resource, in favor of standard skaffold.yaml and kustomization.yaml files. The Jenkins configuration would be moved to a distinct, project-specific jenkins.yaml and the custom dependency specification would be moved to service-dependencies.yaml.

This improves the overall system by applying the Single Responsibility Principle to the build tool stack.

Current dev tool stack
Figure 1. Current dev tool stack

The one metadata.yaml contains the configurations for multiple tools in one place, which makes it convenient but also hard to debug and maintain due to its highly properietary nature, and forces each distinct tool to be highly-coupled to a single file.

New dev tool stack
Figure 2. New dev tool stack

Skaffold and Kustomize handle largely the same functions as helm-maven-plugin and ci-file-generator KubernetesMojo, and in fact, provide even more easy-to-use development and build support. The separate configuration files are much easier to understand and maintain because they each have their distinct uses, and are controlled by distinct sets of configuration values.

The dependencyResolverScript contains logic that recreates the helm-maven-plugin Maven artifact dependency resolution logic by retrieving each dependency’s skaffold.yaml or kustomization.yaml to find their specified service dependencies, and then selecting the newest version of each, outputting the result to a list, which can be used in the project’s skaffold.yaml or kustomization.yaml.

The Jenkins aspect can be similarly improved by dedicating its configuration to a single file, jenkins.yaml, which just provides default variables to the Jenkins shared library operations.