Dev and Build Tools Re-architecture

The current set of development and build tools, including primarily helm-maven-plugin, are:

These tools should be as simple as possible without trying to do too much at once. If more functionality is desired, the developer should have the option to include it or not. If it is included, its configuration should be explicitly defined within the scope of the build itself, not buried inside logic within a plugin. Debugging build and/or test problems should not necessitate investigating or stepping through the plugin code itself.

The general solution to these problems is to simplify the set of plugins to both decrease the complexity of each plugin and decrease the complexity of each plugin’s goal, so that any specific build action is simple to understand, accomplishes a single task, and which can be composed explicitly to execute more complex tasks.

The "mental model" of the project build should not require multiple levels of translation and conversion, such as the build-time associations between the pom.xml, metadata.yaml and Helm-specific values files used in the actual test deployment. Any conversions or migrations necessary should be executed directly by the developer (for instance during project initialization), or defined in an explicit plugin configuration referencing dynamic values.

Convention over Configuration should be the norm, and any organizational or technical conventions should be implemented as initialization tasks (i.e., a collection of "starter" templates/code). Templates for initialization and migration should be managed separately from the plugin implementations themselves.

Plugins which adapt native tools to Maven usage should produce artifacts which are processable by the native tools themselves. In the case of helm-maven-plugin, the Helm resources provided in a project should be able to be referenced via native helm commands and work "out of the box".

Instead of defining a monolithic build system where multiple actions happen “behind the scenes” within the internals of a single Maven plugin goal or tightly-coupled multiple goals, the aim is to define a more explicit approach that allows fine-grained tasks to be executed by the developer, but which also allows easy-to-comprehend composition of these simpler commands to provide the same level of functionality as before if desired.

The current system provides an easy method to handle most build/test scenarios with a simple mvn clean install command when the project contains the plugin configurations present in either a parent POM (like jersey-service-parent) or a set of Maven tiles (like helm-tile).

Ideally this works because the Helm templates stored in the helm-maven-plugin JAR are automatically copied to the project directory, and the defaults stored in ci-file-generator are merged at build-time so that the developer does not need to provide default values and write their own Helm templates for their project. When the developer decides to upgrade to newer versions of the parent POM or the set of plugins, the version(s) are incremented and the developer rebuilds the project resulting in new and/or updated resources and configurations.

This can be problematic if the upgrade changes the resulting deployable, which could impact projects that depend on this artifact downstream. The developer seemingly does not have the option of changing how the project is built, since these changes are all "automatic" with the upgrade to the parent POM or version of these plugins (it seems to be an "all or nothing" choice). Helm chart updates are well-documented to cause difficulties in the development process.

There are ways to customize these aspects, of course, but it takes more configuration to reach a more minimally invasive state, when it should be the other way around.

The "Principle of least astonishment" states that

Currently, the system offers a "one-button" solution for building a metadata.yaml-based service, but at the expense of complexity and unexpected (and many times, unused) functionalities.

For instance, helm-maven-plugin build tasks are expected to generally mirror the functions of the helm executable, with some additional options to conceptually bridge Maven to Helm. However, currently the helm-maven-plugin contains features that are completely outside the scope of Helm and are specific to the organizational needs and standards of NGSS.

Since the vast majority of incompatibility issues among service charts have been caused by built-in automatic environment value management in each version’s template code, moving to Kustomize-based post-processing would for all intents and purposes fix this issue. Therefore, version forward and backward dependency compatibility tests (related to helm-maven-plugin) should not be required moving forward.

Simplifying the plugin implementations and reconstructing the build process to be composed of smaller tasks has the potentially beneficial side effect of becoming more architecturally similar to that of a Gradle-based build definition. This could make it more straightforward to port to Gradle, if or when it’s decided to move to that platform.

Regardless, breaking down the development, build, and test tasks from a monolithic "one button" proprietary solution to a (still relatively small) series of manual steps using ubiquitous third-party tools: