Skip to content

Pod groups when Single CQ Pod should be admitted after the group labels are added #11491

@mimowo

Description

@mimowo

Which test is flaking?:

Pod groups when Single CQ Pod should be admitted after the group labels are added
Link to failed CI job or steps to reproduce locally:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/11474/pull-kueue-test-e2e-extended-main-1-35/2057761190188158976

Failure message or logs:

End To End Extended Suite: kindest/node:v1.35.0: [It] Pod groups when Single CQ Pod should be admitted after the group labels are added [area:singlecluster, feature:pod] expand_less	1m32s
{ failed [FAILED] Timed out after 90.000s.
Workload did not finish
metadata:
  annotations:
    kueue.x-k8s.io/is-group-workload: "true"
  creationTimestamp: "2026-05-22T10:12:24Z"
  finalizers:
  - kueue.x-k8s.io/resource-in-use
  generation: 1
  managedFields:
  - apiVersion: kueue.x-k8s.io/v1beta2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kueue.x-k8s.io/is-group-workload: {}
      f:status:
        f:admission:
          f:clusterQueue: {}
          f:podSetAssignments:
            k:{"name":"4b0469f7"}:
              .: {}
              f:count: {}
              f:flavors:
                f:cpu: {}
              f:name: {}
              f:resourceUsage:
                f:cpu: {}
        f:conditions:
          k:{"type":"Admitted"}:
            .: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:observedGeneration: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"QuotaReserved"}:
            .: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:observedGeneration: {}
            f:reason: {}
            f:status: {}
            f:type: {}
    manager: kueue-admission
    operation: Apply
    subresource: status
    time: "2026-05-22T10:12:24Z"
  - apiVersion: kueue.x-k8s.io/v1beta2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kueue.x-k8s.io/is-group-workload: {}
        f:finalizers:
          .: {}
          v:"kueue.x-k8s.io/resource-in-use": {}
        f:ownerReferences:
          .: {}
          k:{"uid":"0585215d-b95d-4254-a9a5-1539a71f8c0a"}: {}
      f:spec:
        .: {}
        f:active: {}
        f:podSets:
          .: {}
          k:{"name":"4b0469f7"}:
            .: {}
            f:count: {}
            f:name: {}
            f:template:
              .: {}
              f:metadata: {}
              f:spec:
                .: {}
                f:containers:
                  .: {}
                  k:{"name":"c"}:
                    .: {}
                    f:args: {}
                    f:image: {}
                    f:imagePullPolicy: {}
                    f:name: {}
                    f:resources:
                      .: {}
                      f:limits:
                        .: {}
                        f:cpu: {}
                      f:requests:
                        .: {}
                        f:cpu: {}
                    f:terminationMessagePath: {}
                    f:terminationMessagePolicy: {}
                    f:volumeMounts:
                      .: {}
                      k:{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}:
                        .: {}
                        f:mountPath: {}
                        f:name: {}
                        f:readOnly: {}
                f:dnsPolicy: {}
                f:enableServiceLinks: {}
                f:preemptionPolicy: {}
                f:priority: {}
                f:restartPolicy: {}
                f:schedulerName: {}
                f:schedulingGates:
                  .: {}
                  k:{"name":"kueue.x-k8s.io/admission"}:
                    .: {}
                    f:name: {}
                  k:{"name":"kueue.x-k8s.io/topology"}:
                    .: {}
                    f:name: {}
                f:securityContext: {}
                f:serviceAccount: {}
                f:serviceAccountName: {}
                f:terminationGracePeriodSeconds: {}
                f:tolerations: {}
                f:volumes:
                  .: {}
                  k:{"name":"kube-api-access-pngn5"}:
                    .: {}
                    f:name: {}
                    f:projected:
                      .: {}
                      f:defaultMode: {}
                      f:sources: {}
            f:topologyRequest:
              .: {}
              f:podIndexLabel: {}
        f:priority: {}
        f:queueName: {}
    manager: kueue
    operation: Update
    time: "2026-05-22T10:12:24Z"
  name: test-group
  namespace: pod-e2e-xqs6t
  ownerReferences:
  - apiVersion: v1
    kind: Pod
    name: pod-0
    uid: 0585215d-b95d-4254-a9a5-1539a71f8c0a
  resourceVersion: "13291"
  uid: 46fd67c2-a14a-4c00-acec-96acaad4bb9b
spec:
  active: true
  podSets:
  - count: 1
    name: 4b0469f7
    template:
      metadata: {}
      spec:
        containers:
        - args:
          - entrypoint-tester
          image: registry.k8s.io/e2e-test-images/agnhost:2.63.0
          imagePullPolicy: IfNotPresent
          name: c
          resources:
            limits:
              cpu: "1"
            requests:
              cpu: "1"
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
            name: kube-api-access-pngn5
            readOnly: true
        dnsPolicy: ClusterFirst
        enableServiceLinks: true
        preemptionPolicy: PreemptLowerPriority
        priority: 0
        restartPolicy: Never
        schedulerName: default-scheduler
        schedulingGates:
        - name: kueue.x-k8s.io/admission
        - name: kueue.x-k8s.io/topology
        securityContext: {}
        serviceAccount: default
        serviceAccountName: default
        terminationGracePeriodSeconds: 30
        tolerations:
        - effect: NoExecute
          key: node.kubernetes.io/not-ready
          operator: Exists
          tolerationSeconds: 300
        - effect: NoExecute
          key: node.kubernetes.io/unreachable
          operator: Exists
          tolerationSeconds: 300
        volumes:
        - name: kube-api-access-pngn5
          projected:
            defaultMode: 420
            sources:
            - serviceAccountToken:
                expirationSeconds: 3607
                path: token
            - configMap:
                items:
                - key: ca.crt
                  path: ca.crt
                name: kube-root-ca.crt
            - downwardAPI:
                items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace
    topologyRequest:
      podIndexLabel: kueue.x-k8s.io/pod-group-pod-index
  priority: 0
  queueName: queue
status:
  admission:
    clusterQueue: cq-pod-e2e-xqs6t
    podSetAssignments:
    - count: 1
      flavors:
        cpu: on-demand-pod-e2e-xqs6t
      name: 4b0469f7
      resourceUsage:
        cpu: "1"
  conditions:
  - lastTransitionTime: "2026-05-22T10:12:24Z"
    message: Quota reserved in ClusterQueue cq-pod-e2e-xqs6t
    observedGeneration: 1
    reason: QuotaReserved
    status: "True"
    type: QuotaReserved
  - lastTransitionTime: "2026-05-22T10:12:24Z"
    message: The workload is admitted
    observedGeneration: 1
    reason: Admitted
    status: "True"
    type: Admitted


The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/util/util.go:600 with:
Expected success, but got an error:
    <*errors.StatusError | 0x875816583c0>: 
    workloads.kueue.x-k8s.io "test-group" not found
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "workloads.kueue.x-k8s.io \"test-group\" not found",
            Reason: "NotFound",
            Details: {
                Name: "test-group",
                Group: "kueue.x-k8s.io",
                Kind: "workloads",
                UID: "",
                Causes: nil,
                RetryAfterSeconds: 0,
            },
            Code: 404,
        },
    }
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/extended/pod_test.go:630 @ 05/22/26 10:13:55.204
}

Anything else we need to know?:

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.kind/flakeCategorizes issue or PR as related to a flaky test.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions