Add PathPattern (directory discovery) to ArtifactGenerator

[vocarista]

Part of controlplaneio-fluxcd/flux-operator#421

This PR introduces the pathPattern feature to the ArtifactGenerator CRD to support dynamic, directory-based ExternalArtifact generation. This resolves the need for users to manually define an OutputArtifact for every single directory (e.g., every app/environment in a monorepo).

Example

Given a monorepo with this directory structure:

apps/
├── auth/
│   └── envs/
│       ├── dev/
│       │   └── app.yaml
│       └── prod/
│           └── app.yaml
└── payments/
    └── envs/
        ├── dev/
        │   └── app.yaml
        └── staging/
            └── app.yaml

Before (static)

Each artifact must be explicitly listed. Adding a new app or environment requires updating this spec:

apiVersion: source.extensions.fluxcd.io/v1beta1
kind: ArtifactGenerator
metadata:
  name: platform-apps
  namespace: flux-system
spec:
  sources:
    - alias: monorepo
      kind: GitRepository
      name: my-monorepo
  artifacts:
    - name: auth-dev
      copy:
        - from: "@monorepo/apps/auth/envs/dev/**"
          to: "@artifact/"
    - name: auth-prod
      copy:
        - from: "@monorepo/apps/auth/envs/prod/**"
          to: "@artifact/"
    - name: payments-dev
      copy:
        - from: "@monorepo/apps/payments/envs/dev/**"
          to: "@artifact/"
    - name: payments-staging
      copy:
        - from: "@monorepo/apps/payments/envs/staging/**"
          to: "@artifact/"

After (with pathPattern)

A single pathPattern replaces all explicit entries. The artifacts block becomes a template that is expanded once per matched directory:

apiVersion: source.extensions.fluxcd.io/v1beta1
kind: ArtifactGenerator
metadata:
  name: platform-apps
  namespace: flux-system
spec:
  sources:
    - alias: monorepo
      kind: GitRepository
      name: my-monorepo
  pathPattern: "@monorepo/apps/{app}/envs/{env}"
  artifacts:
    - name: "{{ .app }}-{{ .env }}"
      copy:
        - from: "@monorepo/apps/{{ .app }}/envs/{{ .env }}/**"
          to: "@artifact/"

Both specs produce the same 4 ExternalArtifact objects in the cluster. With pathPattern, the captured variables are additionally injected as labels on each generated ExternalArtifact (e.g., app: auth, env: dev).

When a new directory (e.g., apps/payments/envs/prod/) is added to the repo, the controller automatically creates the corresponding ExternalArtifact — no spec update required. Conversely, when a directory is removed, its ExternalArtifact is garbage collected.

Key Features & Implementation Details:

1. API Extension: Added spec.pathPattern to ArtifactGeneratorSpec and enabled Go template syntax validation for OutputArtifact.Name.
2. Dynamic Discovery: Implemented directory traversal that dynamically converts patterns like @monorepo/apps/{app}/envs/{env} into regex with named capture groups to match target directories.
3. Template Rendering: When pathPattern is defined, spec.outputArtifacts acts as a template. The Name, Copy.From, and Copy.To fields are rendered using standard text/template (e.g., {{ .app }}).
4. Label Injection: Captured path variables are automatically injected as standard Kubernetes labels on the generated ExternalArtifact objects, facilitating easy discovery by downstream controllers (like Kustomize or Helm).
5. Selective Reconciliation: Ensured the pipeline isolates revisions correctly, meaning if only apps/auth/envs/dev changes, only the auth-dev artifact gets a new revision hash, leaving other untouched artifacts intact.
6. Documentation: Added the "Path Pattern (Directory Discovery)" section to the V1Beta1 spec docs.

Testing:

1. Unit Tests (artifactgenerator_pathpattern_test.go): Covered static paths, single capture patterns, invalid formats, exact-depth boundary files, and Exclude/Strategy field preservation.
2. Integration Tests (artifactgenerator_controller_test.go): Covered end-to-end artifact generation, strict label assertions, unpacking and asserting against physical copy paths, garbage collection on directory deletion, and selective revision updates.

[matheuscscp]

@vocarista Please edit the PR description with YAML examples explaining the new feature's behavior.

[stefanprodan]

Dir names are not compatible with Kubernetes labels which are restricted to DNS. Also dir names do not qualify for Kubernetes object names. What if the dirs are in Chinese or they are named like .bar.foo?

[vocarista]

@stefanprodan, thanks for catching that, I hadn't taken that into consideration. I see two quick options, we either skip dirs that do not match the regex or fail the reconciliation and ask the user to use compatible dir names. Thoughts?

[stefanprodan]

I expect most users to not be able to use this feature, the label value restrictions are very tight. What's the reason to use the dirs as labels?

[stefanprodan]

I guess whoever whats to use this must have a dir structure that complies with Kubernetes label value restrictions which we should document.

After extraction, the controller should validate the values using https://github.com/fluxcd/flux-schema/blob/aa86d0ac25d21f7312366272ea79e50d475be9e8/internal/validator/metadata.go#L79

If validation passes, the controller should apply to lower on the values before setting the labels and artifact name. If the validation fails, we can mark the reconciliation as stalled and return a terminal error. If paths gets fixed, this will trigger a new reconciliation.

[matheuscscp]

Specifically content.IsLabelValue?

[vocarista]

@stefanprodan the intent behind label propagation was to align with your original comment about letting RSIP select ExternalArtifacts using label. I do agree that the directory name compliance with label values has to be documented. I think another thing that also needs to be documented is what if two directory names result in the same value (MY-APP and My-App would both be sanitised to my-app.) In this case too, we'll stall the reconciliation and throw an error along the lines of duplicate directories found right?

[stefanprodan]

Yes good catch, we should also validate for duplicates and error out.

Validation workflow could be:

• run ToLower on all extracted values
• validate values with content.IsLabelValue
• validate all EA names with https://github.com/fluxcd/flux-schema/blob/aa86d0ac25d21f7312366272ea79e50d475be9e8/internal/validator/metadata.go#L55
• validate for duplicates

Any validation error should contain the pathPattern verbatim as in the spec, so that people can figure out from where values come from.