Changes since v0.19.0:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.18.0, v0.19.0.

• LeaderWorkerSet: Fixed a quota bypass where raising spec.leaderWorkerTemplate.size on an already-admitted, Kueue-managed LeaderWorkerSet ran more pods per group than the reserved quota covered. spec.leaderWorkerTemplate.size is now immutable while the LeaderWorkerSet is managed by Kueue, behind the new LWSImmutableGroupSize feature gate (Beta, enabled by default). spec.replicas stays mutable.

  If you change spec.leaderWorkerTemplate.size on a Kueue-managed LeaderWorkerSet, recreate it at the new size instead, or disable the LWSImmutableGroupSize feature gate to keep the previous behavior, which also restores the quota bypass. (#13809, @ivnovakov)

• TAS: Enforce stricter slice-size validation for Workloads. When podSetSliceRequiredTopology is specified, podSetSliceSize must also be specified and must be greater than 0. Non-positive slice sizes in topology constraints are also rejected.

If you create Workload objects directly (or via custom controllers), update manifests before upgrade so that:

• podSetSliceRequiredTopology is never set without podSetSliceSize
• podSetSliceSize is always greater than 0
• podSetSliceSize is not set when podSetSliceRequiredTopology is absent
• every podsetSliceRequiredTopologyConstraints entry has size greater than 0

If you need a phased rollout, temporarily disable TASValidateWorkloadSliceSize, clean up invalid Workloads, then re-enable it. (#13737, @mszadkow)

• TAS: Fix a bug where TASRecomputeAssignmentWithinSchedulingCycle can be enabled even if TopologyAwareScheduling is disabled.

If you disable TopologyAwareScheduling, also set TASRecomputeAssignmentWithinSchedulingCycle=false before upgrading. (#14257, @tenzen-y)

Changes by Kind

Feature

• Helm: the controller-manager Deployment now supports optional controllerManager.strategy, controllerManager.hostNetwork, and controllerManager.dnsPolicy values. (#13825, @dinhxuanvu)
• MultiKueue: Forwarded in-place serveConfigV2 (Ray Serve application config) updates on a RayService from the manager to the worker cluster, so editing the Serve config on the manager now takes effect on the worker promptly. Changes to rayClusterConfig/upgradeStrategy (zero-downtime upgrade) are not yet propagated. (#14036, @kevin85421)

Bug or Regression

• AFS: Fixed a bug that could modify cached Workload data while calculating LocalQueue fair-sharing usage, potentially producing inconsistent scheduling snapshots. (#13568, @aburan28)
• AFS: Fixed a bug where a LocalQueue with fairSharing.weight: 0 could be prioritized for admission instead of deprioritized when AdmissionFairSharing is enabled. (#13559, @sumanthd032)
• AFS: Fixed pending Workload snapshot ordering when a referenced LocalQueue is missing. (#13515, @YQ-Wang)
• AdmissionFairSharing: Fixed a bug where resource usage smaller than one milli-unit was truncated to zero before it could accumulate, so with a long usageHalfLifeTime the consumedResources for CPU and extended resources such as GPUs stayed at 0 permanently and were ignored by fair sharing. (#13761, @Shreesha001)
• AdmissionFairSharing: Fixed a bug where workloads admitted via AdmissionChecks could keep their entry penalty permanently, inflating LocalQueue fair-sharing usage and deprioritizing later workloads. (#13795, @apullo777)
• AdmissionFairSharing: Fixed stale fair-sharing usage caused by entry penalties being applied to non-usage-based ClusterQueues or reapplied during second scheduling passes. (#13851, @apullo777)
• AdmissionFairSharing: Fixed transient LocalQueue lookup errors causing a pending-Workload snapshot to mix
  fair-sharing comparisons with standard queue ordering, resulting in a non-transitive comparator and inconsistent
  admission order. When a lookup fails, the entire snapshot now falls back to standard queue ordering. (#13546, @YQ-Wang)
• ClusterQueue: Fixed a bug where a terminating ClusterQueue (one with a deletion timestamp still retained by the resource-in-use finalizer because workloads are reserving quota) stopped updating status.pendingWorkloads, status.admittedWorkloads, and status.reservingWorkloads and never set its Active condition to Terminating, leaving stale status. Kueue now keeps the status of a terminating ClusterQueue accurate. (#13757, @kaushik229)
• ConcurrentAdmission: Fix preemption ordering by waiting for more-preferred variants to be evaluated for admission before opening the preemption gate for a less-preferred variant. (#14281, @yuluo-yx)
• Corrected invalid PodSet info errors to report the expected and actual PodSet counts in the correct order. (#13672, @cryo-zd)
• DRA: Fixed DeviceClass validation errors reporting a duplicated request field path with an incorrect request index in counter-based and capacity-based quota paths. (#13899, @cryo-zd)
• DRA: Fixed a bug where extended resource quota could be charged against a DeviceClass the scheduler would not allocate from when multiple DeviceClasses share the same extendedResourceName. (#14124, @thc1006)
• DRA: Fixed a startup crash when KueueDRAIntegrationPartitionableDevices or KueueDRAIntegrationConsumableCapacity feature gates are enabled but the ResourceSlice API (resource.k8s.io/v1) is not available on the cluster. (#13720, @MaysaMacedo)
• ElasticJobsViaWorkloadSlices & ProvisioningRequest: Fixed scale-from-zero admission for elastic jobs. Kueue now
  omits zero-count PodSets, which are invalid in a ProvisioningRequest. If there are no other PodSets requiring ProvisioningRequests the AdmissionCheck is marked Ready. (#14210, @neilb-dotcom)
• ElasticJobsViaWorkloadSlices: Fixed elastic jobs (e.g. autoscaling RayClusters via ElasticJobsViaWorkloadSlices) leaving scaled-up pods stuck SchedulingGated after the origin workload slice was deleted. (#14139, @dinhxuanvu)
• ElasticJobsViaWorkloadSlices: Fixed the bug that changes to the kueue.x-k8s.io/priority-class label were not
  reflected on the live Workload slices. (#13871, @thc1006)
• Fixed a bug where a ClusterQueue with flavorFungibility.preference: PreemptionOverBorrowing could leave workloads pending indefinitely. A flavor that required preemption but had no preemption candidates could outrank a later flavor that fits, purely because its quota was sourceable at a shallower borrowing level in the cohort tree. (#13896, @YQ-Wang)
• Fixed a bug where a Workload could be re-nominated to the same ResourceFlavor indefinitely and never reach the remaining flavors of its ResourceGroup. The flavor scan progress recorded for a Workload was discarded whenever the ClusterQueue's allocatable resource generation advanced, whenever the Workload was skipped due to in-cycle contention, or whenever the Workload was updated, all of which happen continuously on a busy Cohort. This most visibly affected Topology-Aware Scheduling, where a Workload whose topology cannot be placed on the flavor selected by quota needs to fall through to the next flavor. Controlled by the new PreserveFlavorScanProgress feature gate, enabled by default. (#13956, @varunsyal)
• Fixed a bug where a transient ProvisioningRequest or PodTemplate creation error could remain in Workload status and later be reported as the cause of an unrelated deactivation. (#13874, @apullo777)
• Fixed a bug where deleting a child object whose owner was already deleted (e.g. mixed foreground/background propagation during namespace teardown) could leave the child stuck in Terminating, because Kueue webhooks denied the garbage collector's finalizer-removal request with "workload owner not found". The tolerance applies only to objects already being deleted, and is gated by the new SkipAncestorCheckForDeletedWorkloads feature gate (Beta, enabled by default). (#13857, @tomsen02)
• Fixed a bug where elastic-job worker pods could remain SchedulingGated for up to ~90s after a scale rollover when the ungater requeued a slice that had already finished. (#14277, @dinhxuanvu)
• Fixed a bug where, with TASFailedNodeReplacementFailFast disabled, replacement pods for a workload whose node became unhealthy were ungated onto that same unhealthy node and immediately terminated, exhausting the pod recreation budget instead of waiting for a replacement domain. (#14119, @varunsyal)
• Fixed a quantity larger than int64 on a resource other than cpu being converted to a number of another magnitude, or of another sign, when Kueue computes a Workload's requests. A large enough resource transformation product could arrive negative and then be floored to zero, so the Workload was admitted against no quota at all. (#14112, @thc1006)
• Fixed elastic job pods being ungated against a workload slice that was already being evicted, which allowed more pods to start than the slice still holding the reservation granted. (#13923, @thc1006)
• Fixed missing UpdatedWorkload event when the AdmissionGatedBy annotation is propagated from a StatefulSet to its Workload. (#14120, @Shreesha001)
• Fixed overly broad ClusterRole permissions by scoping webhook configuration and CRD access to only Kueue's own resources using resourceNames (#13610, @prash2512)
• Fixed resource totals wrapping to a negative number when two contributions to the same resource sum past the int64 range. Both Requests implementations now saturate in Add and Sub, as they already did in Mul, so an unrepresentable total is no longer read as an empty request. (#14108, @thc1006)
• HA: Fix a data race between concurrent reconciles in non-leading replicas, where the leader-aware decorator ...

Changes since v0.18.4:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.17.0, v0.18.0.

• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.18.1, v0.18.2, v0.18.3, v0.18.4.

• LeaderWorkerSet: Fixed a quota bypass where raising spec.leaderWorkerTemplate.size on an already-admitted, Kueue-managed LeaderWorkerSet ran more pods per group than the reserved quota covered. spec.leaderWorkerTemplate.size is now immutable while the LeaderWorkerSet is managed by Kueue, behind the new LWSImmutableGroupSize feature gate (Beta, enabled by default). spec.replicas stays mutable.

  If you change spec.leaderWorkerTemplate.size on a Kueue-managed LeaderWorkerSet, recreate it at the new size instead, or disable the LWSImmutableGroupSize feature gate to keep the previous behavior, which also restores the quota bypass. (#13807, @ivnovakov)

• TAS: Enforce stricter slice-size validation for Workloads. When podSetSliceRequiredTopology is specified, podSetSliceSize must also be specified and must be greater than 0. Non-positive slice sizes in topology constraints are also rejected.

If you create Workload objects directly (or via custom controllers), update manifests before upgrade so that:

• podSetSliceRequiredTopology is never set without podSetSliceSize
• podSetSliceSize is always greater than 0
• podSetSliceSize is not set when podSetSliceRequiredTopology is absent
• every podsetSliceRequiredTopologyConstraints entry has size greater than 0

If you need a phased rollout, temporarily disable TASValidateWorkloadSliceSize, clean up invalid Workloads, then re-enable it. (#13741, @mszadkow)

• TAS: Fix a bug where TASRecomputeAssignmentWithinSchedulingCycle can be enabled even if TopologyAwareScheduling is disabled.

If you disable TopologyAwareScheduling, also set TASRecomputeAssignmentWithinSchedulingCycle=false before upgrading. (#14256, @tenzen-y)

Changes by Kind

Feature

• Helm: the controller-manager Deployment now supports optional controllerManager.strategy, controllerManager.hostNetwork, and controllerManager.dnsPolicy values. (#13824, @dinhxuanvu)
• MultiKueue: Forwarded in-place serveConfigV2 (Ray Serve application config) updates on a RayService from the manager to the worker cluster, so editing the Serve config on the manager now takes effect on the worker promptly. Changes to rayClusterConfig/upgradeStrategy (zero-downtime upgrade) are not yet propagated. (#14036, @kevin85421)
• Observability: Added kueue_unadmitted_workloads and kueue_local_queue_unadmitted_workloads metrics (gated by UnadmittedWorkloadsObservability) to track the count of unadmitted workloads by ClusterQueue/LocalQueue and the underlying blockage cause (e.g., WaitingForQuota, ChecksNotReady). (#13617, @j-skiba)
• Observability: Added granular Kubernetes warning event reasons for unadmitted
  Workloads, such as WaitingForQuota, NoMatchingFlavor,
  ExceedsMaxQuota, and TopologyPlacementFailed, matching the reason reported
  in the QuotaReserved condition. The feature is disabled by default,
  and guarded by UnadmittedWorkloadsObservability. The related proactive
  initialization of explicit unadmitted status conditions is separately guarded
  by UnadmittedWorkloadsExplicitStatus, which is also disabled by
  default, and requires UnadmittedWorkloadsObservability. (#13659, @j-skiba)
• Observability: Added support for the UnadmittedWorkloadsObservability feature gate in the workload controller. When enabled, Kueue populates the QuotaReserved workload condition with granular reasons (such as Misconfigured, Suspended, or AdmissionGated) and detailed messages when a workload cannot be admitted, making it easier for operators to diagnose admission issues. (#13584, @j-skiba)
• Scheduling: Added the UnadmittedWorkloadsExplicitStatus feature gate. When enabled, newly created workloads immediately receive explicit unadmitted status conditions (QuotaReserved=False and Admitted=False) during initial reconciliation to improve queue state observability. (#13590, @j-skiba)
• Scheduling: Workloads bypassed by the scheduling equivalence cache now receive the granular failure reason (e.g., WaitingForQuota) and a bypass message in their QuotaReserved condition, improving visibility into why the workload was unadmitted. (#13593, @j-skiba)
• When the UnadmittedWorkloadsObservability feature gate is enabled, workloads that fail to obtain a quota reservation now receive detailed diagnostic reasons in their QuotaReserved status condition (such as WaitingForQuota, ExceedsMaxQuota, TopologyPlacementFailed, or NoMatchingFlavor) along with an explicit Admitted: False condition. (#13532, @j-skiba)
• Workload: When UnadmittedWorkloadsObservability is enabled, clearing workload quota reservation in the ConcurrentAdmission controller reports QuotaReserved: False with the PendingEvaluation reason instead of Pending. (#13624, @j-skiba)
• Workload: When UnadmittedWorkloadsObservability is enabled, releasing quota reservation in JobFramework and StatefulSet controllers reports QuotaReserved: False with the PendingEvaluation reason instead of Pending. (#13620, @j-skiba)

Bug or Regression

• AFS: Fixed a bug that could modify cached Workload data while calculating LocalQueue fair-sharing usage, potentially producing inconsistent scheduling snapshots. (#13569, @aburan28)
• AFS: Fixed a bug where a LocalQueue with fairSharing.weight: 0 could be prioritized for admission instead of deprioritized when AdmissionFairSharing is enabled. (#13481, @sumanthd032)
• AFS: Fixed pending Workload snapshot ordering when a referenced LocalQueue is missing. (#13517, @YQ-Wang)
• AdmissionFairSharing: Fixed a bug where resource usage smaller than one milli-unit was truncated to zero before it could accumulate, so with a long usageHalfLifeTime the consumedResources for CPU and extended resources such as GPUs stayed at 0 permanently and were ignored by fair sharing. (#13763, @Shreesha001)
• AdmissionFairSharing: Fixed a bug where workloads admitted via AdmissionChecks could keep their entry penalty permanently, inflating LocalQueue fair-sharing usage and deprioritizing later workloads. (#12786, @apullo777)
• AdmissionFairSharing: Fixed stale fair-sharing usage caused by entry penalties being applied to non-usage-based ClusterQueues or reapplied during second scheduling passes. (#13853, @apullo777)
• AdmissionFairSharing: Fixed transient LocalQueue lookup errors causing a pending-Workload snapshot to mix
  fair-sharing comparisons with standard queue ordering, resulting in a non-transitive comparator and inconsistent
  admission order. When a lookup fails, the entire snapshot now falls back to standard queue ordering. (#13525, @YQ-Wang)
• ClusterQueue: Fixed a bug where a terminating ClusterQueue (one with a deletion timestamp still retained by the resource-in-use finalizer because workloads are reserving quota) stopped updating status.pendingWorkloads, status.admittedWorkloads, and status.reservingWorkloads and never set its Active condition to Terminating, leaving stale status. Kueue now keeps the status of a terminating ClusterQueue accurate. (#13759, @kaushik229)
• Corrected invalid PodSet info errors to report the expected and actual PodSet counts in the correct order. (#13673, @cryo-zd)
• DRA: Fix workloads retaining DRA-mapped resource names after their DeviceClass is deleted. (#14085, @tenzen-y)
• DRA: Fixed DeviceClass validation errors reporting a duplicated request field path with an incorrect request index in counter-based quota paths. (#13924, @cryo-zd)
• DRA: Fixed a bug where extended resource quota could be charged against a DeviceClass the scheduler would not allocate from when multiple DeviceClasses share the same extendedResourceName. (#14044, @thc1006)
• DRA: Fixed a startup crash when KueueDRAIntegrationPartitionableDevices feature gate is enabled but the ResourceSlice API (resource.k8s.io/v1) is not available on the cluster. (#13721, @MaysaMacedo)
• ElasticJobsViaWorkloadSlices & ProvisioningRequest: Fixed scale-from-zero admission for elastic jobs. Kueue now
  omits zero-count PodSets, which are invalid in a ProvisioningRequest. If there are no other PodSets requiring ProvisioningRequests the AdmissionCheck is marked Ready. (#14240, @neilb-dotcom)
• ElasticJobsViaWorkloadSlices: Fixed elastic jobs (e.g. autoscaling RayClusters via ElasticJobsViaWorkloadSlices) leaving scaled-up pods stuck SchedulingGated after the origin workload slice was deleted. (#14139, @dinhxuanvu)
• ElasticJobsViaWorkloadSlices: Fixed the bug that changes to the kueue.x-k8s.io/priority-class label were not
  reflected on the live Workload slices. (#13780, @thc1006)
• Fixed a bug where a ClusterQueue with flavorFungibility.preference: PreemptionOverBorrowing could leave workloads pending indefinitely. A flavor that required preemption but had no preemption candidates could outrank a later flavor that fits, purely because its quota was sourceable at a shallower borrowing level in the cohort tree. (#13903, @YQ-Wang)
• Fixed a bug where a Workload could be re-nominated to the same ResourceFlavor indefinitely and never reach the remaining flavors of its ResourceGroup. The flavor scan progress recorded for a Workload was discarded whenever the Clu...

Changes since v0.18.0:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.17.0, v0.18.0.

• If you maintain an in-house integration you will need to modify the code
  to pass the k8s context when calling the RestorePodSetsInfo function. (#13114, @ivnovakov)

• KueuePopulator Helm: helm uninstall removes the ClusterQueue, ResourceFlavor, Topology, ConfigMap, and RBAC created by the chart, which previously leaked after uninstall.

If you installed a previous version of the kueue-populator chart, its ConfigMap and RBAC (*-kueue-hook-* ServiceAccount/ClusterRole/ClusterRoleBinding and the *-kueue-resources ConfigMap) were created as Helm hooks and are not adopted by the new release. Delete them manually before upgrading to avoid helm upgrade/install ownership conflicts. (#12402, @kevin85421)

• MultiKueue: Fixed a security vulnerability in locationType=Path kubeconfig handling
  that could allow users with MultiKueueCluster create or update access to make the
  controller read arbitrary files. Kueue now validates path-based kubeconfigs to stay under
  /etc/multikueue/kubeconfigs.

If you use locationType=Path, plan to move kubeconfig files under
/etc/multikueue/kubeconfigs, or switch to locationType=Secret or ClusterProfile.
This prepares your setup for future releases where MultiKueueKubeConfigPathValidation
is expected to be enabled by default. (#12223, @kannon92)

• RayCluster: Fixed a bug where the Ray autoscaler sidecar container's resources were not counted against quota when in-tree autoscaling was enabled, causing the head PodSet to be under-counted. The head PodSet now includes the autoscaler sidecar (KubeRay's default 500m CPU / 512Mi memory, or spec.autoscalerOptions.resources when set).

users with autoscaling-enabled RayClusters may need to increase their ClusterQueue CPU quota by 500m and memory quota by 512Mi per head pod to avoid admission failures after upgrading. (#12405, @kevin85421)

• RayJob: Fixed a bug where the Ray job submitter container's resources were not counted against quota when submissionMode: SidecarMode was used, causing the head PodSet to be under-counted. The head PodSet now includes the submitter sidecar (KubeRay's default 500m CPU / 200Mi memory).

After upgrading, RayJobs using submissionMode: SidecarMode reserve the submitter sidecar's resources (default 500m CPU / 200Mi memory) on the head. ClusterQueues sized without this headroom may fail to admit such RayJobs; increase the affected ClusterQueue's CPU/memory quota accordingly. (#12454, @kevin85421)

• TAS: A negative subGroupCount on a Workload now produces an admission warning.

Starting with the 0.20 release, a negative subGroupCount will be rejected at the API level. (#13101, @reruno)

• WaitForPodsReady is now enabled by default. New Kueue installations and existing installations that do not explicitly configure waitForPodsReady will use the default WaitForPodsReady configuration (30 minute timeout, 30 minute recovery timeout). (#11855, @amirialy)

Changes by Kind

Deprecation

• DRA: Remove the deprecated DynamicResourceAllocation feature gate. Use KueueDRAIntegration instead. (#12258, @kshalot)
• MultiKueue: Added accessProviders as the preferred ClusterProfile field for
  configuring cluster access providers. The existing credentialsProviders field remains
  supported but is deprecated and cannot be used together with accessProviders. (#12011, @kahirokunn)

API Change

• Use SchemeGroupVersion instead of GroupVersion in the API.

  If your code references the GroupVersion variable from the API, update it to use SchemeGroupVersion instead. (#12738, @mbobrovskyi)

Feature

• AFS Observability: Added kueue_local_queue_admission_fair_sharing_usage Prometheus metric to report AFS usage per LocalQueue, calculated from the resource-weighted sum of consumed resources and pending admission penalties, and divided by the LocalQueue's fair sharing weight. (#12326, @ShaanveerS)
• ConcurrentAdmission: Fixed Variants not being created or deleted when a ClusterQueue's resource flavors change. (#12501, @ivnovakov)
• ConcurrentAdmission: make sure there is at most one preemption variant issuing preemptions at any given time.
  This is achieved using the "preemption gates" mechanism. (#11872, @reruno)
• DRA Partitionable Devices: support multi-counter tracking by allowing the same DeviceClass in multiple deviceClassMappings with different counter sources. Add ResourceSliceCache for consolidated ResourceSlice listing. (#13018, @PannagaRao)
• DRA: Adds capacity-based quota for DRA devices with multiple allocations. (#13152, @sohankunkerkar)
• Graduate KueueDRAIntegrationExtendedResource to Beta (enabled by default) (#13102, @PannagaRao)
• Graduate KueueDRAIntegrationPartitionableDevices to Beta (enabled by default) (#13167, @PannagaRao)
• Graduate ManagedJobsNamespaceSelectorAlwaysRespected to GA (#13021, @PannagaRao)
• Graduate the AdmissionGatedBy feature gate to Beta, enabled by default. Users who previously had to manually enable this gate no longer need to. Users who do not use the kueue.x-k8s.io/admission-gated-by annotation are unaffected. (#12110, @carterpewpew)
• Helm: Added enableVisibilityAuthReaderRoleBinding Helm value (default: true) to make the visibility server's auth-reader RoleBinding in kube-system optional. Set to false when deploying under a GitOps project that cannot manage resources in kube-system, and create the RoleBinding out-of-band instead. (#12699, @amy)
• Increase OOTB QPS and concurrency for Kueue: QPS: 300, Burst=500, Workload concurrency: 10, LQ and CQ: 5. (#12440, @yuluo-yx)
• KueueViz: Added a global rate limiter to the KueueViz backend to protect against distributed Denial of Service (DoS) attacks and TokenReview amplification. (#13173, @Vaishnav88sk)
• MultiKueue Observability: Added a new metric multikueue_workloads_dispatched_total to count remote workloads successfully created by the MultiKueue manager per worker cluster. (#12782, @Mostafahassen1)
• MultiKueue: Added a new metric multikueue_workloads_admitted_total that counts remote workloads admitted by a worker cluster, labeled by cluster_queue, cluster, and replica_role. (#13050, @Mostafahassen1)
• MultiKueue: Elastic RayCluster worker-group replica changes made on the management cluster (via the ElasticJobsViaWorkloadSlices feature gate) now propagate to the RayCluster on the admitting worker cluster. Previously the remote RayCluster was created once and never resized. (#12885, @jiaoew1991)
• MultiKueue: The incremental dispatcher now nominates worker clusters in the order defined in MultiKueueConfig.spec.clusters instead of alphabetically, enabling priority-based spillover (for example, trying cheaper on-premises clusters before public-cloud clusters). (#13041, @andrewseif)
• MultiKueue: provide stepSize configuration for the Incremental Dispatcher. (#11208, @Mostafahassen1)
• Observability: Added kueue_unadmitted_workloads and kueue_local_queue_unadmitted_workloads metrics (gated by UnadmittedWorkloadsObservability) to track the count of unadmitted workloads by ClusterQueue/LocalQueue and the underlying blockage cause (e.g., WaitingForQuota, ChecksNotReady). (#12759, @j-skiba)
• Observability: Added granular Kubernetes warning event reasons for unadmitted
  Workloads, such as WaitingForQuota, NoMatchingFlavor,
  ExceedsMaxQuota, and TopologyPlacementFailed, matching the reason reported
  in the QuotaReserved condition. The feature is disabled by default,
  and guarded by UnadmittedWorkloadsObservability. The related proactive
  initialization of explicit unadmitted status conditions is separately guarded
  by UnadmittedWorkloadsExplicitStatus, which is also disabled by
  default, and requires UnadmittedWorkloadsObservability. (#13022, @j-skiba)
• Observability: Added support for the UnadmittedWorkloadsObservability feature gate in the workload controller. When enabled, Kueue populates the QuotaReserved workload condition with granular reasons (such as Misconfigured, Suspended, or AdmissionGated) and detailed messages when a workload cannot be admitted, making it easier for operators to diagnose admission issues. (#12510, @j-skiba)
• Observability: Added the kueue_pod_scheduling_gate_removal_seconds histogram metric to
  measure the time from Workload admission to Pod scheduling-gate removal, helping operators
  track delays before admitted Pods can be scheduled. (#12137, @mbobrovskyi)
• Observability: Updated the alpha custom metric labels API with support for source-specific labels, including labels sourced from Workloads, and allowlisting of tracked label values. The feature remains disabled by default and can be enabled using the CustomMetricLabels feature gate. (#12713, @Singularity23x0)
• Observability: When enabling custom metric labels, workloads will automatically copy appropriate labels
  and annotations from underlying jobs. PodSets must match annotation and label values defined as custom
  metric label value sources across component Pods if feature enabled. (#13146, @Singularity23x0)
• Promoted QuotaCheckStrategy to Beta and enabled by default. (#13075, @MaysaMacedo)
• Scheduling: Added the UnadmittedWorkloadsExplicitStatus feature gate. When enabled, newly created workloads immediately receive explicit unadmitted status conditions (QuotaReserved=False and Admitted=False) during initial reconciliation to improve queue state observability. (#12719, @j-skiba)
• Scheduling: Workloads bypassed by the scheduling equivalence cache now receive the granular failure reason (e.g., `WaitingForQ...

Changes since v0.18.3:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.17.0, v0.18.0.
• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.18.1, v0.18.2, v0.18.3.

Changes by Kind

Feature

• Helm: Added enableVisibilityAuthReaderRoleBinding Helm value (default: true) to make the visibility server's auth-reader RoleBinding in kube-system optional. Set to false when deploying under a GitOps project that cannot manage resources in kube-system, and create the RoleBinding out-of-band instead. (#13048, @amy)

Bug or Regression

• AFS: Fixed a Denial of Service (DoS) vulnerability where deleting a LocalQueue could cause the Kueue scheduler to hang during AdmissionFairSharing calculations. (#13214, @Vaishnav88sk)
• AFS: Fixed a race in Admission Fair Sharing penalty updates where concurrent workload operations could lose penalty changes, causing LocalQueues to receive incorrect priority. (#13285, @MaysaMacedo)
• DRA: Fixed a bug where byte-valued Partitionable Devices (counter-based) resources were displayed as raw byte integers in Workload and ClusterQueue status.
  These resources are formatted using human-readable BinarySI units, such as Mi and Gi. (#13038, @amarkdotdev)
• DRA: introduce a safeguard for invalid parameter combinations to prevent nil dereference crashes (#12979, @mykysha)
• ElasticJobsViaWorkloadSlices: Fixed a bug that allowed a replacement Workload slice to reference a Workload from another namespace when both used the same ClusterQueue, potentially causing the unrelated Workload to be treated and finished as the replaced slice. Workload slice replacements are now restricted to Workloads in the same namespace. (#13081, @mykysha)
• ElasticJobsViaWorkloadSlices: Fixed a bug that could cause elastic Jobs to
  stall after Pods succeeded or failed, because terminal Pods continued to count
  against the active Workload slice's admitted PodSet count and prevented
  replacement Pods from being ungated. (#13204, @garg02)
• ElasticJobsViaWorkloadSlices: Fixed a bug where an elastic job could permanently fail to start (FailedToStart) due to stale Kueue-owned annotations on the pod template, e.g. after its workload was deleted, or after eviction of a previously scaled-up job. (#13131, @mcochner)
• ElasticJobsViaWorkloadSlices: Fixed a bug where reclaimable Pod accounting after scaling down an elastic Job could reserve quota for Pods that were no longer running. Reserved quota now tracks the remaining running Pods for indexed and non-indexed Jobs. (#13263, @Shreesha001)
• ElasticJobsViaWorkloadSlices: Fixed a bug where scaling down an elastic Job could leave a stale reclaimablePods count, causing Kueue to account for less quota than the Job's remaining Pods were using. (#13060, @Shreesha001)
• KueueViz: Fixed a security issue in kueueviz where WebSocket connections continued streaming cluster data after a bearer token expired or was revoked.
  Connections are now closed within 30 seconds of token invalidation. (#13130, @Vaishnav88sk)
• KueueViz: Navigating to an invalid cohort now displays a graceful error message instead of crashing the UI. (#13246, @Vaishnav88sk)
• KueueViz: Prevent workload detail pages from crashing when Kubernetes Events have missing or invalid timestamps. (#13207, @YQ-Wang)
• MultiKueue: Fixed a bug where a remote Workload finishing with reason OutOfSync was mirrored as a terminal finish, leaving the manager Job stranded. Kueue now resets the MultiKueue AdmissionCheck to Retry, retries the Workload, and emits a warning event identifying the worker cluster. (#13086, @Smuger)
• MultiKueue: Fixed a bug where a transient watch reconnect to a worker cluster could evict a running admitted workload. Kueue now measures the worker-lost grace from when the worker cluster's connection first dropped, rather than from the admission check's transition time, and retries immediately only when the reserving worker is reachable but its remote workload is gone. (#12999, @kevin85421)
• MultiKueue: Fixed a bug where admitted workloads could remain stuck instead of being evicted and retried after workerLostTimeout when reconnecting to a worker cluster failed after its connection configuration changed. (#13188, @kevin85421)
• MultiKueue: Fixes an observability bug where Pods scheduled in a worker cluster could still appear unscheduled
  in the manager cluster (as PodScheduled=False would be preserved). The PodScheduled condition is now
  synchronized from the worker cluster, while preserving SchedulingGated for unschedulable Pods to avoid spurious
  scale-ups. (#13197, @fg91)
• Observability: Fix verbose DRS logs failing to report DRS values due to JSON parsing error when handling fair sharing weight set to 0. (#13157, @kshalot)
• RayJob, RayCluster, RayService, JobSet, MPIJob, and Kubeflow Trainer jobs: Fixed a bug where changing a running job's pod set count, for example adding a worker group to a running RayCluster, could crash the Kueue controller during reconciliation. (#13104, @ivnovakov)
• TAS & Scheduling: Fixed a bug where Workloads owned by a single Pod could be reassigned after eviction or during TAS node hot swap, even though the existing Pod could not consume the new assignment. The fix applies when the SkipReassignmentForPodOwnedWorkloads feature gate is enabled. The gate is Beta and enabled by default in 0.19+, and Alpha and disabled by default in the 0.17 and 0.18 release branches. (#12980, @yakticus)
• TAS: Added a fix for premature node replacement when a node remains NotReady while the workload's Pods are still running, which could cause the topology assignment to diverge from the actual Pod placement and corrupt per-node capacity accounting. The termination-driven behavior applies when TASReplaceNodeDueToNotReadyOverFixedTime is disabled. The gate is deprecated and disabled by default in 0.19+, and Beta and enabled by default in the 0.17 and 0.18 release branches. (#13096, @yakticus)
• TAS: Fix a performance bug where repeatedly checking the enablement of the TASRespectNodeAffinityPreferred feature gate inside a hot sorting loop could significantly increase the scheduling time (14% by the attached benchmark). (#13145, @j-skiba)
• TAS: Fixed a bug where a PodSet slice size that did not evenly divide its count could make the topology ungater panic repeatedly, so the workload's Pods stayed stuck gated. The ungater no longer panics and ungates the Pods that fit the topology assignment. (#13268, @ivnovakov)
• TAS: Fixed a bug where a PodSet with subGroupIndexLabel set but a missing or zero subGroupCount could crash the tas-ungater controller. Kueue now falls back to greedy domain assignment for these pods instead of panicking. (#13065, @reruno)
• TAS: Fixed a performance bug that caused remaining capacity to be repeatedly recalculated and resource maps to be unnecessarily copied during workload evaluation, particularly when evaluating multiple preemption candidate sets in
  large clusters. The fix is guarded by the Beta TASCachingRemainingResources feature gate, which is enabled by default. (#13235, @j-skiba)
• TAS: domain selection is now deterministic when multiple domains tie on score; ties are broken by the domains' levelValues ordering. (#13031, @mvanhorn)
• TAS: fixed excessive scheduling latency for workloads requiring preemption caused by repeatedly evaluating node selectors, tolerations, and affinity for each preemption simulation. The optimization is controlled by the beta TASCacheNodeMatchResults feature gate, enabled by default. (#13206, @j-skiba)
• VisibilityOnDemand: Fixed a bug where a large or negative limit query parameter on the pending-workloads endpoints could crash the Kueue controller manager via memory exhaustion or a panic. The limit is now capped at 100000. (#13053, @reruno)

Other (Cleanup or Flake)

• Observability: Introduced logging of node replacements by NodeHotSwap. (#13215, @dkaluza)
• Observability: Introduced logging of unhealthy nodes on workload updates. (#13273, @dkaluza)
• TAS: Improved scheduling evaluation performance and reduced memory allocations for Topology-Aware Scheduling (TAS). (#13327, @j-skiba)
• TAS: improved workload evaluation performance by optimizing domain-ordering tie-breaks for sibling node domains with equal available capacity. (#13331, @j-skiba)

Changes since v0.17.7:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.16.0, v0.17.0.
• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.17.1, v0.17.2, v0.17.3, v0.17.4, v0.17.5, v0.17.6, v0.17.7.

Changes by Kind

Feature

• Helm: Added enableVisibilityAuthReaderRoleBinding Helm value (default: true) to make the visibility server's auth-reader RoleBinding in kube-system optional. Set to false when deploying under a GitOps project that cannot manage resources in kube-system, and create the RoleBinding out-of-band instead. (#13049, @amy)

Bug or Regression

• AFS: Fixed a Denial of Service (DoS) vulnerability where deleting a LocalQueue could cause the Kueue scheduler to hang during AdmissionFairSharing calculations. (#13264, @Vaishnav88sk)
• AFS: Fixed a race in Admission Fair Sharing penalty updates where concurrent workload operations could lose penalty changes, causing LocalQueues to receive incorrect priority. (#13287, @MaysaMacedo)
• ElasticJobsViaWorkloadSlices: Fixed a bug that allowed a replacement Workload slice to reference a Workload from another namespace when both used the same ClusterQueue, potentially causing the unrelated Workload to be treated and finished as the replaced slice. Workload slice replacements are now restricted to Workloads in the same namespace. (#13082, @mykysha)
• ElasticJobsViaWorkloadSlices: Fixed a bug that could cause elastic Jobs to
  stall after Pods succeeded or failed, because terminal Pods continued to count
  against the active Workload slice's admitted PodSet count and prevented
  replacement Pods from being ungated. (#13203, @garg02)
• ElasticJobsViaWorkloadSlices: Fixed a bug where an elastic job could permanently fail to start (FailedToStart) due to stale Kueue-owned annotations on the pod template, e.g. after its workload was deleted, or after eviction of a previously scaled-up job. (#13132, @mcochner)
• ElasticJobsViaWorkloadSlices: Fixed a bug where reclaimable Pod accounting after scaling down an elastic Job could reserve quota for Pods that were no longer running. Reserved quota now tracks the remaining running Pods for indexed and non-indexed Jobs. (#13178, @Shreesha001)
• ElasticJobsViaWorkloadSlices: Fixed a bug where scaling down an elastic Job could leave a stale reclaimablePods count, causing Kueue to account for less quota than the Job's remaining Pods were using. (#13044, @Shreesha001)
• KueueViz: Fixed a security issue in kueueviz where WebSocket connections continued streaming cluster data after a bearer token expired or was revoked.
  Connections are now closed within 30 seconds of token invalidation. (#13164, @Vaishnav88sk)
• KueueViz: Navigating to an invalid cohort now displays a graceful error message instead of crashing the UI. (#13247, @Vaishnav88sk)
• KueueViz: Prevent workload detail pages from crashing when Kubernetes Events have missing or invalid timestamps. (#13208, @YQ-Wang)
• MultiKueue: Fixed a bug where a remote Workload finishing with reason OutOfSync was mirrored as a terminal finish, leaving the manager Job stranded. Kueue now resets the MultiKueue AdmissionCheck to Retry, retries the Workload, and emits a warning event identifying the worker cluster. (#13087, @Smuger)
• MultiKueue: Fixed a bug where a transient watch reconnect to a worker cluster could evict a running admitted workload. Kueue now measures the worker-lost grace from when the worker cluster's connection first dropped, rather than from the admission check's transition time, and retries immediately only when the reserving worker is reachable but its remote workload is gone. (#12999, @kevin85421)
• MultiKueue: Fixed a bug where admitted workloads could remain stuck instead of being evicted and retried after workerLostTimeout when reconnecting to a worker cluster failed after its connection configuration changed. (#13188, @kevin85421)
• MultiKueue: Fixes an observability bug where Pods scheduled in a worker cluster could still appear unscheduled
  in the manager cluster (as PodScheduled=False would be preserved). The PodScheduled condition is now
  synchronized from the worker cluster, while preserving SchedulingGated for unschedulable Pods to avoid spurious
  scale-ups. (#13198, @fg91)
• Observability: Fix verbose DRS logs failing to report DRS values due to JSON parsing error when handling fair sharing weight set to 0. (#13156, @kshalot)
• RayJob, RayCluster, RayService, JobSet, MPIJob, and Kubeflow Trainer jobs: Fixed a bug where changing a running job's pod set count, for example adding a worker group to a running RayCluster, could crash the Kueue controller during reconciliation. (#13111, @ivnovakov)
• TAS & Scheduling: Fixed a bug where Workloads owned by a single Pod could be reassigned after eviction or during TAS node hot swap, even though the existing Pod could not consume the new assignment. The fix applies when the SkipReassignmentForPodOwnedWorkloads feature gate is enabled. The gate is Beta and enabled by default in 0.19+, and Alpha and disabled by default in the 0.17 and 0.18 release branches. (#12980, @yakticus)
• TAS: Added a fix for premature node replacement when a node remains NotReady while the workload's Pods are still running, which could cause the topology assignment to diverge from the actual Pod placement and corrupt per-node capacity accounting. The termination-driven behavior applies when TASReplaceNodeDueToNotReadyOverFixedTime is disabled. The gate is deprecated and disabled by default in 0.19+, and Beta and enabled by default in the 0.17 and 0.18 release branches. (#13097, @yakticus)
• TAS: Fixed a bug where a PodSet slice size that did not evenly divide its count could make the topology ungater panic repeatedly, so the workload's Pods stayed stuck gated. The ungater no longer panics and ungates the Pods that fit the topology assignment. (#13266, @ivnovakov)
• TAS: Fixed a bug where a PodSet with subGroupIndexLabel set but a missing or zero subGroupCount could crash the tas-ungater controller. Kueue now falls back to greedy domain assignment for these pods instead of panicking. (#13072, @reruno)
• TAS: Fixed a performance bug that caused remaining capacity to be repeatedly recalculated and resource maps to be unnecessarily copied during workload evaluation, particularly when evaluating multiple preemption candidate sets in
  large clusters. The fix is guarded by the Beta TASCachingRemainingResources feature gate, which is enabled by default. (#13234, @j-skiba)
• TAS: domain selection is now deterministic when multiple domains tie on score; ties are broken by the domains' levelValues ordering. (#12052, @mvanhorn)
• TAS: fixed excessive scheduling latency for workloads requiring preemption caused by repeatedly evaluating node selectors, tolerations, and affinity for each preemption simulation. The optimization is controlled by the beta TASCacheNodeMatchResults feature gate, enabled by default. (#13205, @j-skiba)
• VisibilityOnDemand: Fixed a bug where a large or negative limit query parameter on the pending-workloads endpoints could crash the Kueue controller manager via memory exhaustion or a panic. The limit is now capped at 100000. (#13052, @reruno)

Other (Cleanup or Flake)

• Observability: Introduced logging of node replacements by NodeHotSwap. (#13215, @dkaluza)
• Observability: Introduced logging of unhealthy nodes on workload updates. (#13272, @dkaluza)
• TAS: Improved scheduling evaluation performance and reduced memory allocations for Topology-Aware Scheduling (TAS). (#13326, @j-skiba)
• TAS: improved workload evaluation performance by optimizing domain-ordering tie-breaks for sibling node domains with equal available capacity. (#13332, @j-skiba)

Changes since v0.18.0:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.17.0, v0.18.0.

• If you maintain an in-house integration you will need to modify the code
  to pass the k8s context when calling the RestorePodSetsInfo function. (#13114, @ivnovakov)

• KueuePopulator Helm: helm uninstall removes the ClusterQueue, ResourceFlavor, Topology, ConfigMap, and RBAC created by the chart, which previously leaked after uninstall.

If you installed a previous version of the kueue-populator chart, its ConfigMap and RBAC (*-kueue-hook-* ServiceAccount/ClusterRole/ClusterRoleBinding and the *-kueue-resources ConfigMap) were created as Helm hooks and are not adopted by the new release. Delete them manually before upgrading to avoid helm upgrade/install ownership conflicts. (#12402, @kevin85421)

• MultiKueue: Fixed a security vulnerability in locationType=Path kubeconfig handling
  that could allow users with MultiKueueCluster create or update access to make the
  controller read arbitrary files. Kueue now validates path-based kubeconfigs to stay under
  /etc/multikueue/kubeconfigs.

If you use locationType=Path, plan to move kubeconfig files under
/etc/multikueue/kubeconfigs, or switch to locationType=Secret or ClusterProfile.
This prepares your setup for future releases where MultiKueueKubeConfigPathValidation
is expected to be enabled by default. (#12223, @kannon92)

• RayCluster: Fixed a bug where the Ray autoscaler sidecar container's resources were not counted against quota when in-tree autoscaling was enabled, causing the head PodSet to be under-counted. The head PodSet now includes the autoscaler sidecar (KubeRay's default 500m CPU / 512Mi memory, or spec.autoscalerOptions.resources when set).

users with autoscaling-enabled RayClusters may need to increase their ClusterQueue CPU quota by 500m and memory quota by 512Mi per head pod to avoid admission failures after upgrading. (#12405, @kevin85421)

• RayJob: Fixed a bug where the Ray job submitter container's resources were not counted against quota when submissionMode: SidecarMode was used, causing the head PodSet to be under-counted. The head PodSet now includes the submitter sidecar (KubeRay's default 500m CPU / 200Mi memory).

After upgrading, RayJobs using submissionMode: SidecarMode reserve the submitter sidecar's resources (default 500m CPU / 200Mi memory) on the head. ClusterQueues sized without this headroom may fail to admit such RayJobs; increase the affected ClusterQueue's CPU/memory quota accordingly. (#12454, @kevin85421)

• TAS: A negative subGroupCount on a Workload now produces an admission warning.

Starting with the 0.20 release, a negative subGroupCount will be rejected at the API level. (#13101, @reruno)

• WaitForPodsReady is now enabled by default. New Kueue installations and existing installations that do not explicitly configure waitForPodsReady will use the default WaitForPodsReady configuration (30 minute timeout, 30 minute recovery timeout). (#11855, @amirialy)

Changes by Kind

Deprecation

• DRA: Remove the deprecated DynamicResourceAllocation feature gate. Use KueueDRAIntegration instead. (#12258, @kshalot)
• MultiKueue: Added accessProviders as the preferred ClusterProfile field for
  configuring cluster access providers. The existing credentialsProviders field remains
  supported but is deprecated and cannot be used together with accessProviders. (#12011, @kahirokunn)
• TAS: node replacement is now triggered only when the workload's Pods on a NotReady node are terminating or terminated. The previous behavior of additionally marking a node as failed after it has been NotReady for a fixed 30s is deprecated and retained behind the TASReplaceNodeDueToNotReadyOverFixedTime feature gate (enabled by default in 0.17/0.18 backports, disabled by default and deprecated since 0.19, planned for removal). When that gate is disabled, TASReplaceNodeOnPodTermination has no effect. Nodes that are deleted or lack a Ready condition are still marked as failed immediately. (#13043, @yakticus)

API Change

• Use SchemeGroupVersion instead of GroupVersion in the API.

  If your code references the GroupVersion variable from the API, update it to use SchemeGroupVersion instead. (#12738, @mbobrovskyi)

Feature

• Add curvePreferences support to TLSOptions (#11832, @kannon92)
• Added kueue_local_queue_admission_fair_sharing_usage Prometheus metric to report AFS usage per LocalQueue, calculated from the resource-weighted sum of consumed resources and pending admission penalties, and divided by the LocalQueue's fair sharing weight. (#12326, @ShaanveerS)
• Added a new metric multikueue_workloads_dispatched_total to count remote workloads successfully created by the MultiKueue manager per worker cluster. (#12782, @Mostafahassen1)
• Added the TASSkipReassignmentForPodOwnedWorkloads feature gate (Alpha in 0.18, Beta/on-by-default in 0.19). When enabled, Kueue no longer recomputes the topology assignment of a Workload owned by a single Pod: on node failure the existing assignment is kept, and on eviction (e.g. preemption) the Workload is parked with Requeued=False instead of being re-admitted with a new assignment. Such Workloads cannot receive replacement pods, so a recomputed assignment can never be consumed and only makes the stored placement diverge from the node the pod actually runs on. Recovery happens through the owning controller's replacement pod, which arrives as a new Workload. (#12980, @yakticus)
• ConcurrentAdmission: Fixed Variants not being created or deleted when a ClusterQueue's resource flavors change. (#12501, @ivnovakov)
• ConcurrentAdmission: make sure there is at most one preemption variant issuing preemptions at any given time.
  This is achieved using the "preemption gates" mechanism. (#11872, @reruno)
• Custom Metric Labels (Alpha): Admitted Active Workloads metric support for Workload labels/annotations. (#13210, @Singularity23x0)
• Custom metric labels update:
  • Adds SourceKind and TrackedValues fields support.
  • Adds Workload labels support.
  • Enforces new custom label config validation.
    This is a breaking change for the custom metric labels alpha feature. (#12713, @Singularity23x0)
• DRA Partitionable Devices: support multi-counter tracking by allowing the same DeviceClass in multiple deviceClassMappings with different counter sources. Add ResourceSliceCache for consolidated ResourceSlice listing. (#13018, @PannagaRao)
• DRA: Adds capacity-based quota for DRA devices with multiple allocations. (#13152, @sohankunkerkar)
• Graduate KueueDRAIntegrationExtendedResource to Beta (enabled by default) (#13102, @PannagaRao)
• Graduate ManagedJobsNamespaceSelectorAlwaysRespected to GA (#13021, @PannagaRao)
• Graduate the AdmissionGatedBy feature gate to Beta, enabled by default. Users who previously had to manually enable this gate no longer need to. Users who do not use the kueue.x-k8s.io/admission-gated-by annotation are unaffected. (#12110, @carterpewpew)
• Helm: Added enableVisibilityAuthReaderRoleBinding Helm value (default: true) to make the visibility server's auth-reader RoleBinding in kube-system optional. Set to false when deploying under a GitOps project that cannot manage resources in kube-system, and create the RoleBinding out-of-band instead. (#12699, @amy)
• Increase OOTB QPS and concurrency for Kueue: QPS: 300, Burst=500, Workload concurrency: 10, LQ and CQ: 5. (#12440, @yuluo-yx)
• Increase the maximum number of PodSets per Workload from 10 to 18. (#12819, @mcochner)
• KueueViz: Added a global rate limiter to the KueueViz backend to protect against distributed Denial of Service (DoS) attacks and TokenReview amplification. (#13173, @Vaishnav88sk)
• MultiKueue: Added a new metric multikueue_workloads_admitted_total that counts remote workloads admitted by a worker cluster, labeled by cluster_queue, cluster, and replica_role. (#13050, @Mostafahassen1)
• MultiKueue: Elastic RayCluster worker-group replica changes made on the management cluster (via the ElasticJobsViaWorkloadSlices feature gate) now propagate to the RayCluster on the admitting worker cluster. Previously the remote RayCluster was created once and never resized. (#12885, @jiaoew1991)
• MultiKueue: Fixed a bug where admitted workloads could remain stuck instead of being evicted and retried after workerLostTimeout when reconnecting to a worker cluster failed after its connection configuration changed. (#13188, @kevin85421)
• MultiKueue: The incremental dispatcher now nominates worker clusters in the order defined in MultiKueueConfig.spec.clusters instead of alphabetically, enabling priority-based spillover (for example, trying cheaper on-premises clusters before public-cloud clusters). (#13041, @andrewseif)
• MultiKueue: provide stepSize configuration for the Incremental Dispatcher. (#11208, @Mostafahassen1)
• Observability: Added kueue_unadmitted_workloads and kueue_local_queue_unadmitted_workloads metrics (gated by UnadmittedWorkloadsObservability) to track the count of unadmitted workloads by ClusterQueue/LocalQueue and the underlying blockage cause (e.g., WaitingForQuota, ChecksNotReady). (#12759, @j-skiba)
• Observability: Added support for the UnadmittedWorkloadsObservability feature gate in the workload controller. When enabled, Kueue populates the QuotaReserved workload condition with granular reasons (such as Misconfigured, Suspended, or AdmissionGated) and detailed messages when a workload cannot be admitted, making it easier for operators to diagno...

Changes since v0.18.2:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.17.0, v0.18.0.

• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.18.1, v0.18.2.

• RayJob: Fixed a bug where the Ray job submitter container's resources were not counted against quota when submissionMode: SidecarMode was used, causing the head PodSet to be under-counted. The head PodSet now includes the submitter sidecar (KubeRay's default 500m CPU / 200Mi memory).

  After upgrading, RayJobs using submissionMode: SidecarMode reserve the submitter sidecar's resources (default 500m CPU / 200Mi memory) on the head. ClusterQueues sized without this headroom may fail to admit such RayJobs; increase the affected ClusterQueue's CPU/memory quota accordingly. (#12726, @kevin85421)

Changes by Kind

Bug or Regression

• AFS: Fixed ConsumedResources CPU truncating to zero when the sampling interval guard was bypassed by informer cache lag during initialization. (#12691, @sohankunkerkar)
• AFS: Fixed a race where a sampling tick running concurrently with workload settlement could persist a skewed ConsumedResources value in LocalQueue fair-sharing status. (#12940, @apullo777)
• AFS: Fixed consumed-resources cache initialization and warm-start recovery so LocalQueue usage is not over-counted during cache seeding, and persisted historical usage is preserved after manager restarts when workload settlement runs before LocalQueue reconciliation. (#12891, @apullo777)
• CLI: Fix --dry-run flag being silently ignored in kueuectl resume/stop localqueue and clusterqueue subcommands. (#12624, @carterpewpew)
• DRA: Fix an integer overflow in device-count quota accounting where a ResourceClaimTemplate with very large device counts could be admitted over quota and leave a negative used-quota in the ClusterQueue status. (#12901, @thc1006)
• DRA: Fixed incorrect quota charging for invalid driver-published device counters by clamping them to the non-negative
  int64 range before computing quota charges. (#12947, @thc1006)
• DRA: fixed a potential int64 overflow in the counter-based device quota charge computation that could under-count quota when a driver publishes very large counter values. (#12928, @thc1006)
• ElasticJobsViaWorkloadSlices: Fix workload slice misordering that could finish a correctly-admitted elastic workload slice when 3+ slices were created within the same second. (#12964, @mimowo)
• ElasticJobsViaWorkloadSlices: Fixed a bug where scaling a Job below its accumulated succeeded count could permanently wedge the Workload reconciler and leak quota. (#12959, @Shreesha001)
• ElasticJobsViaWorkloadSlices: Fixed a bug where worker pods of an elastic job could be ungated after scale up,
  past the ClusterQueue quota; ungating is now capped to the replicas granted quota across the workload-slice chain. (#12045, @mcochner)
• Helm: Fix helm chart failing to install with a manager CrashLoopBackoff when cert-manager integration is enabled. (#12878, @meln5674)
• Kueue-populator: Fixed a bug where an error creating a LocalQueue was logged but not returned from Reconcile,
  preventing controller-runtime from retrying. LocalQueue creation failures are now aggregated and returned so the request is requeued. (#12905, @NasitSony)
• KueueViz: Fixed a Cross-Site WebSocket Hijacking (CSWSH) vulnerability in the KueueViz Backend by strictly validating WebSocket Origin headers to prevent unauthorized cross-origin data extraction. (#12875, @Vaishnav88sk)
• KueueViz: Fixed a Denial of Service vulnerability where an oversized WebSocket frame could exhaust backend memory (OOM). Connections now enforce an 8 KiB read limit. (#12704, @ABHIGYAN-MOHANTA)
• KueueViz: Fixed dashboard crash caused by missing optional chaining on flavor.resources (#12668, @ABHIGYAN-MOHANTA)
• KueueViz: Improved workloads dashboard performance by avoiding repeated Pod list operations per Workload (#12857, @cryo-zd)
• KueueViz: backend includes HTTP server timeouts (ReadHeaderTimeout, ReadTimeout, WriteTimeout, IdleTimeout) to prevent connection resource exhaustion. (#12866, @ABHIGYAN-MOHANTA)
• KueueViz: frontend container image now runs as a non-root user (node) to adhere to the principle of least privilege. (#12702, @ABHIGYAN-MOHANTA)
• LeaderWorkerSet: Fixed a bug where a LeaderWorkerSet with a negative or excessively large spec.replicas could crash the Kueue controller during reconciliation and MultiKueue workload processing. Kueue now rejects spec.replicas values that are negative or greater than 1000000 (#12755, @reruno)
• MultiKueue: Fixed a bug where obsolete remote Workloads could remain on temporarily unavailable worker clusters when the manager Workload lost its reservation or was deleted. Kueue now retries cleanup after worker clusters reconnect. (#11515, @vamsikrishna-siddu)
• MultiKueue: Fixed a data race where reconnecting a remote cluster could swap the remote client while other goroutines were reading it, which could crash-loop the controller manager. (#12612, @apullo777)
• MultiKueue: Fixed custom jobs using external-framework adapters being repeatedly created and deleted on worker clusters when source-cluster metadata was copied to the remote object. (#12677, @apullo777)
• Observability: Fixed LocalQueue gauge metrics not being reported after a LocalQueue starts matching the configured metrics selector. (#12903, @ikchifo)
• PodGroup integration: Fixed a bug that allowed Workloads corresponding to PodGroups with the WaitingForReplacementPods=True condition to be re-admitted immediately. (#12872, @mbobrovskyi)
• ProvisioningRequest: Fix a bug where ProvisioningRequest owned by finished or evicted Workloads are not cleaned up. The CleanupProvisioningRequestsOnEviction feature gate allows cleanup on eviction to be enabled by default. (#12632, @MatteoFari)
• RayJob: Fix the integration controller dropping Kueue admission placement constraints (nodeSelector, tolerations, nodeAffinity) for the submitter pod when submitterPodTemplate is not explicitly set and submissionMode is K8sJobMode. (#12696, @carterpewpew)
• RayService: Fixed a bug where deleting a Kueue-managed RayService with GCS fault tolerance enabled left KubeRay's Redis cleanup Job suspended forever, leaking the RayCluster's Redis metadata namespace. Kueue now defers finalizing the RayService's Workload until the cleanup Job completes. (#12778, @kevin85421)
• ResourceTransformations: Fixed a bug where milli-valued quantities were rounded before
  resource transformation multiplication. For example, multiplying 300m CPU by 1000
  now correctly produces 300 instead of 3000. (#12961, @mimowo)
• Scheduling: Fixed resource accounting and validation for Pods using Kubernetes pod-level
  resources (pod.spec.resources), including LimitRange defaulting and request/limit
  validation. (#12731, @anuragdalvi)
• Scheduling: Fixed stale scheduling queue entries for pending Workloads that transition
  to WorkloadOnHold. (#12942, @anuragdalvi)
• SparkApplication: Fixed a bug where the global spec.nodeSelector could overwrite driver or executor node selectors when they were admitted to different ResourceFlavors. (#12687, @carterpewpew)
• StatefulSet: Fixed a bug where scaling a StatefulSet to zero caused its Workload to be incorrectly requeued for scheduling during the terminating-pod window, competing for quota it should no longer hold. (#12650, @gola)
• TAS: Fixed a bug that permanently leaked Topology-Aware Scheduling (TAS) resources if a workload was deleted while its ClusterQueue was temporarily missing a required Topology. (#12751, @Vaishnav88sk)
• VisibilityOnDemand: Fixed a data race between the Visibility API pending-workloads endpoint and preemption requeuing that could crash the queue manager for BestEffortFIFO ClusterQueues. (#12754, @somaz94)

Other (Cleanup or Flake)

• TAS: Reduced the CPU and memory overhead of building the topology snapshot on large clusters by no longer cloning per-node usage maps on every scheduling cycle. (#12705, @akshay-pm)

Changes since v0.17.6:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.16.0, v0.17.0.

• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.17.1, v0.17.2, v0.17.3, v0.17.4, v0.17.5, v0.17.6.

• RayJob: Fixed a bug where the Ray job submitter container's resources were not counted against quota when submissionMode: SidecarMode was used, causing the head PodSet to be under-counted. The head PodSet now includes the submitter sidecar (KubeRay's default 500m CPU / 200Mi memory).

  After upgrading, RayJobs using submissionMode: SidecarMode reserve the submitter sidecar's resources (default 500m CPU / 200Mi memory) on the head. ClusterQueues sized without this headroom may fail to admit such RayJobs; increase the affected ClusterQueue's CPU/memory quota accordingly. (#12725, @kevin85421)

Changes by Kind

Bug or Regression

• AFS: Fixed ConsumedResources CPU truncating to zero when the sampling interval guard was bypassed by informer cache lag during initialization. (#12694, @sohankunkerkar)
• AFS: Fixed a race where a sampling tick running concurrently with workload settlement could persist a skewed ConsumedResources value in LocalQueue fair-sharing status. (#12946, @mimowo)
• AFS: Fixed consumed-resources cache initialization and warm-start recovery so LocalQueue usage is not over-counted during cache seeding, and persisted historical usage is preserved after manager restarts when workload settlement runs before LocalQueue reconciliation. (#12891, @apullo777)
• CLI: Fix --dry-run flag being silently ignored in kueuectl resume/stop localqueue and clusterqueue subcommands. (#12627, @carterpewpew)
• ElasticJobsViaWorkloadSlices: Fix workload slice misordering that could finish a correctly-admitted elastic workload slice when 3+ slices were created within the same second. (#12965, @mimowo)
• ElasticJobsViaWorkloadSlices: Fixed a bug where scaling a Job below its accumulated succeeded count could permanently wedge the Workload reconciler and leak quota. (#12963, @mimowo)
• ElasticJobsViaWorkloadSlices: Fixed a bug where worker pods of an elastic job could be ungated after scale up,
  past the ClusterQueue quota; ungating is now capped to the replicas granted quota across the workload-slice chain. (#12045, @mcochner)
• Helm: Fix helm chart failing to install with a manager CrashLoopBackoff when cert-manager integration is enabled. (#12877, @meln5674)
• Kueue-populator: Fixed a bug where an error creating a LocalQueue was logged but not returned from Reconcile,
  preventing controller-runtime from retrying. LocalQueue creation failures are now aggregated and returned so the request is requeued. (#12930, @NasitSony)
• KueueViz: Fixed a Cross-Site WebSocket Hijacking (CSWSH) vulnerability in the KueueViz Backend by strictly validating WebSocket Origin headers to prevent unauthorized cross-origin data extraction. (#12876, @Vaishnav88sk)
• KueueViz: Fixed a Denial of Service vulnerability where an oversized WebSocket frame could exhaust backend memory (OOM). Connections now enforce an 8 KiB read limit. (#12703, @ABHIGYAN-MOHANTA)
• KueueViz: Fixed dashboard crash caused by missing optional chaining on flavor.resources (#12667, @ABHIGYAN-MOHANTA)
• KueueViz: Improved workloads dashboard performance by avoiding repeated Pod list operations per Workload (#12858, @cryo-zd)
• KueueViz: backend includes HTTP server timeouts (ReadHeaderTimeout, ReadTimeout, WriteTimeout, IdleTimeout) to prevent connection resource exhaustion. (#12867, @ABHIGYAN-MOHANTA)
• KueueViz: frontend container image now runs as a non-root user (node) to adhere to the principle of least privilege. (#12586, @ABHIGYAN-MOHANTA)
• LeaderWorkerSet: Fixed a bug where a LeaderWorkerSet with a negative or excessively large spec.replicas could crash the Kueue controller during reconciliation and MultiKueue workload processing. Kueue now rejects spec.replicas values that are negative or greater than 1000000 (#12756, @reruno)
• MultiKueue: Fixed a bug where obsolete remote Workloads could remain on temporarily unavailable worker clusters when the manager Workload lost its reservation or was deleted. Kueue now retries cleanup after worker clusters reconnect. (#11515, @vamsikrishna-siddu)
• MultiKueue: Fixed a data race where reconnecting a remote cluster could swap the remote client while other goroutines were reading it, which could crash-loop the controller manager. (#12612, @apullo777)
• MultiKueue: Fixed custom jobs using external-framework adapters being repeatedly created and deleted on worker clusters when source-cluster metadata was copied to the remote object. (#12643, @apullo777)
• Observability: Fixed LocalQueue gauge metrics not being reported after a LocalQueue starts matching the configured metrics selector. (#12912, @ikchifo)
• PodGroup integration: Fixed a bug that allowed Workloads corresponding to PodGroups with the WaitingForReplacementPods=True condition to be re-admitted immediately. (#12873, @mbobrovskyi)
• ProvisioningRequest: Fix a bug where ProvisioningRequest owned by finished or evicted Workloads are not cleaned up. The CleanupProvisioningRequestsOnEviction feature gate allows cleanup on eviction to be enabled by default. (#12654, @MatteoFari)
• RayJob: Fix the integration controller dropping Kueue admission placement constraints (nodeSelector, tolerations, nodeAffinity) for the submitter pod when submitterPodTemplate is not explicitly set and submissionMode is K8sJobMode. (#12695, @carterpewpew)
• RayService: Fixed a bug where deleting a Kueue-managed RayService with GCS fault tolerance enabled left KubeRay's Redis cleanup Job suspended forever, leaking the RayCluster's Redis metadata namespace. Kueue now defers finalizing the RayService's Workload until the cleanup Job completes. (#12778, @kevin85421)
• ResourceTransformations: Fixed a bug where milli-valued quantities were rounded before
  resource transformation multiplication. For example, multiplying 300m CPU by 1000
  now correctly produces 300 instead of 3000. (#12962, @mimowo)
• Scheduling: Fixed resource accounting and validation for Pods using Kubernetes pod-level
  resources (pod.spec.resources), including LimitRange defaulting and request/limit
  validation. (#12780, @anuragdalvi)
• Scheduling: Fixed stale scheduling queue entries for pending Workloads that transition
  to WorkloadOnHold. (#12948, @mimowo)
• SparkApplication: Fixed a bug where the global spec.nodeSelector could overwrite driver or executor node selectors when they were admitted to different ResourceFlavors. (#12688, @carterpewpew)
• StatefulSet: Fixed a bug where scaling a StatefulSet to zero caused its Workload to be incorrectly requeued for scheduling during the terminating-pod window, competing for quota it should no longer hold. (#12657, @gola)
• TAS: Fixed a bug that permanently leaked Topology-Aware Scheduling (TAS) resources if a workload was deleted while its ClusterQueue was temporarily missing a required Topology. (#12752, @Vaishnav88sk)
• VisibilityOnDemand: Fixed a data race between the Visibility API pending-workloads endpoint and preemption requeuing that could crash the queue manager for BestEffortFIFO ClusterQueues. (#12736, @somaz94)

Other (Cleanup or Flake)

• TAS: Reduced the CPU and memory overhead of building the topology snapshot on large clusters by no longer cloning per-node usage maps on every scheduling cycle. (#12706, @akshay-pm)

Changes since v0.18.1:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.17.0, v0.18.0.

• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.18.1.

• KueuePopulator Helm: helm uninstall removes the ClusterQueue, ResourceFlavor, Topology, ConfigMap, and RBAC created by the chart, which previously leaked after uninstall.

  If you installed a previous version of the kueue-populator chart, its ConfigMap and RBAC (*-kueue-hook-* ServiceAccount/ClusterRole/ClusterRoleBinding and the *-kueue-resources ConfigMap) were created as Helm hooks and are not adopted by the new release. Delete them manually before upgrading to avoid helm upgrade/install ownership conflicts. (#12432, @kevin85421)

Changes by Kind

Bug or Regression

• DRA: Fixed a bug where workloads with device constraints (matchAttribute) or device config were incorrectly rejected as unsupported instead of being admitted for quota. (#12471, @sohankunkerkar)
• Importer: Fixed LocalQueue namespace isolation to prevent information leakage between
  namespaces when multiple LocalQueues with the same name exist in different namespaces. (#12349, @Singularity23x0)
• KueueViz: Fixed WebSocket backend handlers to report errors while fetching dashboard data
  instead of silently ignoring them. (#12346, @yuluo-yx)
• MultiKueue: Creating a Job on the manager cluster deletes any pre-existing remote worker Job that happens to share the same NamespacedName. (#12380, @mszadkow)
• MultiKueue: Fixed a bug that could leave stale status for Kubernetes Jobs in the manager
  cluster when the worker-cluster Job reached steady state quickly and stopped getting
  updates while the manager-cluster Job was still suspended. (#12297, @andrewseif)
• MultiKueue: Fixed a bug where admitted Pod workloads could trigger unnecessary Cluster Autoscaler scale-ups
  in the manager cluster. Kueue now preserves the scheduling-gated PodScheduled condition for manager-cluster
  Pods, since they are intended to run only in worker clusters. (#12272, @fg91)
• Observability: Fixed a race condition that could leave stale LocalQueue metrics after a label change caused the LocalQueue to stop matching the metrics selector. (#12291, @andrewseif)
• RayJob, RayCluster, and RayServe integrations: Fixed missing quota accounting for Redis cleanup resources when GCS fault tolerance is enabled. Kueue accounts for the Redis cleanup Job resources for workloads by folding the cleanup Job requests into the Ray head PodSet. (#12395, @nerdeveloper)
• Scheduling: Fixed a bug where a workload could be stuck pending when its node selector referenced a label key declared by a different flavor in the same resource group. (#12449, @carterpewpew)
• TAS: Fixed a bug that could cause workloads from ClusterQueues considered later in a scheduling cycle to remain pending for prolonged periods. This could happen because TAS assignments computed independently during nomination were likely to conflict on some topology domains. Kueue now re-evaluates TAS assignments during scheduling when needed. (#12521, @mimowo)

Changes since v0.17.5:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.16.0, v0.17.0.

• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.17.1, v0.17.2, v0.17.3, v0.17.4, v0.17.5.

• KueuePopulator Helm: helm uninstall removes the ClusterQueue, ResourceFlavor, Topology, ConfigMap, and RBAC created by the chart, which previously leaked after uninstall.

  If you installed a previous version of the kueue-populator chart, its ConfigMap and RBAC (*-kueue-hook-* ServiceAccount/ClusterRole/ClusterRoleBinding and the *-kueue-resources ConfigMap) were created as Helm hooks and are not adopted by the new release. Delete them manually before upgrading to avoid helm upgrade/install ownership conflicts. (#12450, @kevin85421)

Changes by Kind

Bug or Regression

• Importer: Fixed LocalQueue namespace isolation to prevent information leakage between
  namespaces when multiple LocalQueues with the same name exist in different namespaces. (#12348, @Singularity23x0)
• KueueViz: Fixed WebSocket backend handlers to report errors while fetching dashboard data
  instead of silently ignoring them. (#12347, @yuluo-yx)
• MultiKueue: Creating a Job on the manager cluster deletes any pre-existing remote worker Job that happens to share the same NamespacedName. (#12383, @mszadkow)
• MultiKueue: Fixed a bug where admitted Pod workloads could trigger unnecessary Cluster Autoscaler scale-ups
  in the manager cluster. Kueue now preserves the scheduling-gated PodScheduled condition for manager-cluster
  Pods, since they are intended to run only in worker clusters. (#12273, @fg91)
• RayJob, RayCluster, and RayServe integrations: Fixed missing quota accounting for Redis cleanup resources when GCS fault tolerance is enabled. Kueue accounts for the Redis cleanup Job resources for workloads by folding the cleanup Job requests into the Ray head PodSet. (#11260, @nerdeveloper)
• Scheduling: Fixed a bug where a workload could be stuck pending when its node selector referenced a label key declared by a different flavor in the same resource group. (#12449, @carterpewpew)
• TAS: Fixed a bug that could cause workloads from ClusterQueues considered later in a scheduling cycle to remain pending for prolonged periods. This could happen because TAS assignments computed independently during nomination were likely to conflict on some topology domains. Kueue now re-evaluates TAS assignments during scheduling when needed. (#12523, @mimowo)