Skaffold and Kustomize Known Issues

swagger.json doesn’t get generated for our skaffold

A mvn clean only cleans out the openapi.json but not the hash file (swagger.json-generate-full-swagger.sha256). A fix will be put into mobile-framework soon to avoid this as there isn’t a known scenarios where we don’t want the openapi.json file regenerated.

[INFO] --- openapi-generator:7.1.0:generate (generate-full-swagger) @ jwt-signing-service ---
[INFO] Code generation is skipped because input was unchanged

openapi-generator is looking at the src/main/resources/.openapi-generator/swagger.json-generate-full-swagger.sha256 and seeing the swagger.json matches the hash so it isn’t regenerating the openapi.json even thought openapi.json has been removed.

solution

To work around this issue until mobile-framework is updated to prevent it delete the swagger.json-generate-full-swagger.sha256

"No resources found"

Skaffold command output
my-system:h2-oracle-test$ skaffold dev --port-forward
Cleaning up...
- No resources found
image "ckm/h2-oracle-test" context "h2-oracle-test-v1" does not exist

solution

In the skaffold.yaml ensure that the context and jib project are set.

Skaffold.yaml:

    context: .
    sync:
      auto: true
    jib:
      project: mobile-webauthn-sts

Context should always be . which is telling skaffold where to look for files it should be managing. jib:project should be the name of the project.

Failed to get Jib dependencies

getting hash for artifact "ckm/mockserver": getting dependencies for "ckm/mockserver": could not fetch dependencies for workspace mockserver: initial Jib dependency refresh failed: failed to get Jib dependencies: running [mvn jib:_skaffold-fail-if-jib-out-of-date -Djib.requiredVersion=1.4.0 -Pbuild-only,skaffold-dev --non-recursive jib:_skaffold-files-v2 --quiet --batch-mode]
 - stdout: "[ERROR] Missing:\n[ERROR] ----------\n[ERROR] 1) gov.va.mobile.tools.maven:jacoco-integration-tests-tile:xml:${ngss-maven-tiles.version}\n[ERROR]----

solution

  • update the ngss-maven-tiles.version to latest

  • check tile name is correct.

  • check that the approptiate three tiles are the only ones present:

    <tile>gov.va.mobile.tools.maven:jacoco-integration-tests-tile:${ngss-tiles.version}</tile>
<tile>gov.va.mobile.tools.maven:jib-maven-plugin-tile:${ngss-tiles.version}</tile>
<tile>gov.va.mobile.tools.maven:filegen-tile:${ngss-tiles.version}</tile>

Missing jar

This occurs typically when the cient doesn’t build.

solution

mvn clean install -pl cdw-service-client

JWT not being found

> [logger-service-v1-7c684c9df8-phnd9 logger-service] Caused by: java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'actuatorSecurityFilter' defined in class path resource [gov/va/mobile/service/autoconfigure/ActuatorSecurityAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.web.servlet.FilterRegistrationBean]: Factory method 'actuatorSecurityFilter' threw exception; nested exception is java.lang.IllegalStateException: A RSA 'JWT_PUBLIC_KEY' environmental variable must be set.

solution

The JWT is being pulled in from the common-app-configmap. This resource is added in the kustomization.yaml found in the dev-with-dependencies directory. Ensure it is present.

Failed to find unique target for patch deployment.

    - No resources found
    running [kustomize build /Users/josephdamuth/IdeaProjects/sms-service/kubernetes/localized/dev-with-dependencies/dev-with-dependencies]
    - stdout: ""
    - stderr: "Error: no matches for Id Deployment.v1.apps/oracle-test-v3.[noNs]; failed to find unique target for patch Deployment.v1.apps/oracle-test-v3.[noNs]\n"
    - cause: exit status 1

solution

A required resource is missing. In the case of above adding:

as a resource in the dev-with-dep kustomization resolves the issue.

When Jenkins fails and you see IOExceptions similar to below

IOExceptions are usually permissions or existence of files.

Caused: java.io.IOException: Failed to extract /home/jenkins/agent/workspace/cle_kustomize-profiles/transfer of 234 files

java.io.IOException: Entry 'target/jib-cache/selectors/d024becb0497bd56b98eafee9ae0d6f3e5ad16f9ff6f8e5e66154be68ffa1a73' closed at '0' before the '64' bytes specified in the header were written

solution

The archiveArtifacts call included the target/jib-cache where it was failing during the copy of its contents (assuming this gets deleted by jib so maybe by the time it tries to copy the directory, those files are gone). Adding /target/jib-cache/ to the excludes seemed to fix it:

Skaffold deploy command failing in sandbox

    + skaffold deploy --build-artifacts=build.json --profile=sandbox-build-test
    Starting pre-render hooks...
    Completed pre-render hooks
    time="2023-06-20T18:53:38Z" level=warning msg="kubernetes/localized/sandbox-build-test/sandbox/build-test did not match any file" subtask=-1 task=DevLoop
    Tags used in deployment:
    - dtr.mapsandbox.net/ckm/twilio-mock -> dtr.mapsandbox.net/ckm/twilio-mock:1.1.0-1687287178@sha256:9ad0804960279f87c915d1e15e46ddce025dee6464382cdde5de3e62a5ff3831
    Starting deploy...
    Starting pre-deploy hooks...
    namespace/twilio-mock-v1-test-ku-f0c5fc1 created
    secret/docker-config created
    serviceaccount/default patched
    Completed pre-deploy hooks
    nothing to deploy

solution

The localized folder being ignored by git. Ensure it was pushing it up where sandbox could build it. (i.e above: kubernetes/localized/sandbox-build-test/sandbox/build-test)

Tests fail at times when you just want to skip tests

This is just a matter of changing the skaffold.yaml and adding - -DskipTests to the jib arguments for the main build (i.e not a profile).