🧩 CompositePodGroup (CPG) Scheduling Lifecycle k8s.io/kubernetes/pkg/scheduler

🚀 Open Live Go Simulator (/builder)
Phase 1 · Scheduling Queue Gatekeeper
How GangScheduling.PreEnqueue & HierarchyTracker Gate Pod Activation
Before any Pod enters activeQ, GangScheduling.PreEnqueue(ctx, pod) queries HierarchyTracker.FindRootGroupReadiness(pgKey). The entire root tree must satisfy its bottom-up readiness quorum without exceeding WorkloadMaxTreeDepth (4).

Bottom-Up Readiness PropagationO(Depth) Upward Update

1. Leaf PodGroup (PG) Readiness (minCount)

A leaf PodGroup counts both unscheduled pods in the queue AND already-scheduled pods in the cache: totalPods = unscheduledPods + scheduledPods. When totalPods >= spec.schedulingPolicy.gang.minCount (or >= 1 for Basic), the leaf PG marks itself READY and increments its parent CPG's readyChildren counter.

2. CompositePodGroup (CPG) Readiness (minGroupCount)

An intermediate or root CompositePodGroup is READY when readyChildren >= spec.schedulingPolicy.gang.minGroupCount (or >= 1 for Basic). Readiness propagates up to the top-most Root CPG.

3. Fail-Closed Validation (WorkloadMaxTreeDepth = 4)

If climbing parent links takes more than 4 hops (or hits a cycle), FindRootGroupReadiness returns an error and PreEnqueue rejects the pod with UnschedulableAndUnresolvable.

Example: Partial Quorum Allows Entry!Live in /builder

Suppose cpg-root has minGroupCount: 1 with two child subtrees (cpg-sub1 and cpg-sub2). Even if cpg-sub2 is NOT ready yet, as soon as cpg-sub1 becomes ready, cpg-root has 1/1 ready children — so all queued pods in the hierarchy pass PreEnqueue!

cpg-root (Gang: minGroupCount = 1) --> [READY: 1/1 child groups ready] ├── cpg-sub1 (Gang: minGroupCount = 1) --> [READY: 1/1 child PGs ready] │ └── pg1 (Gang: minCount = 2) --> 2 queued pods (READY ✅) └── cpg-sub2 (Gang: minGroupCount = 2) --> [NOT READY: 1/2 child PGs ready] ├── pg2 (Gang: minCount = 2) --> 2 queued pods (READY ✅) └── pg3 (Gang: minCount = 2) --> 1 queued pod (WAIT ⏳) Verdict for pg1, pg2, AND pg3: GangScheduling.PreEnqueue(pod) => fwk.Success! Why? Root group (cpg-root) is READY! All 5 queued pods enter activeQ together.
Phase 2 · ActiveQueue.Pop() Payload Construction
What Enters the Scheduling Cycle? PodGroupInfo vs. QueuedPodGroupInfo
When PriorityQueue.Pop() pops a ready hierarchy, it returns a *framework.QueuedPodGroupInfo wrapping a structural *framework.PodGroupInfo tree. Understanding what is inside each structure is essential.

1. PodGroupInfo (Structural Hierarchy Tree)All Groups in Tree

PodGroupInfo contains the entire known hierarchy tree rooted at the top-most CompositePodGroup, with Children deterministically sorted by name during structural mutations (AddSubtree / UpdateGenericPodGroup).

type PodGroupInfo struct { *fwk.GenericPodGroup // Root CPG (or standalone PG) Children []*PodGroupInfo // Sorted child CPGs / leaf PGs UnscheduledPods []*v1.Pod // Unscheduled pods belonging to this leaf } // Reconciled against snapshot at start of scheduleOnePodGroup: // sched.reconcilePodGroupWithSnapshot(podGroupInfo) // Ensures every CPG and PG in the tree matches nodeInfoSnapshot!

2. QueuedPodGroupInfo (Unscheduled Pods for This Cycle)Only Unscheduled Queued Pods

QueuedPodGroupInfo embeds *PodGroupInfo and maps each leaf EntityKey to the []*QueuedPodInfo that were waiting in the queue when popped:

type QueuedPodGroupInfo struct { *PodGroupInfo // Maps each leaf PodGroup key -> []*QueuedPodInfo to schedule NOW queuedPodInfos map[fwk.EntityKey][]*QueuedPodInfo } // What about pods that were ALREADY scheduled in an earlier cycle? // - They are NOT in QueuedPodGroupInfo.queuedPodInfos! // - Instead, they live in Cache / Snapshot: // podGroupState.ScheduledPodsCount() (used by PlacementProgress.Scheduled!)
Phase 3 · Dynamic Evolution Across Scheduling Cycles
What If a CPG Is Already Scheduled, and Later New Pods Arrive?
A CompositePodGroup does not have to schedule all of its subtrees in a single cycle. Click through the 3 stages below to see how an already-scheduled CPG wakes up when new pods turn on additional subgroups!
Phase 4 · Inside runRootSchedulingAlgorithm
PlacementFeasible, Node CPU Contention & S → F Subtree Rollback
During runRootSchedulingAlgorithm, the scheduler traverses the CPG tree depth-first, tentatively assuming pods on nodes via assumeAndReserveWithRevert and checking PlacementFeasible after every pod and child subgroup.

How Node CPU Contention Triggers S → F RollbackTestCPGHierarchicalScheduling_Internal

Step 1: cpg-sub1 (pg1: 2 pods × 2 CPU) → Fits on node-1 & node-2

pg1-pod-0 takes 2 CPU on node-1; pg1-pod-1 takes 2 CPU on node-2. cpg-sub1 meets minGroupCount: 1. Remaining cluster CPU: 2 CPU on node-1, 2 CPU on node-2.

Step 2: cpg-sub2 requires minGroupCount: 2 (needs BOTH pg2 AND pg3)

First, pg2 (2 pods × 2 CPU) runs: pg2-pod-0 fits on node-1 (2/4 CPU left -> 4/4 CPU used) and pg2-pod-1 fits on node-2 (4/4 CPU used). pg2 raw status is Success!

Step 3: pg3-pod-0 Fails NodeResourcesFit → cpg-sub2 Fails → pg2 Rolled Back!

Next, pg3-pod-0 tries to schedule, but both nodes are at 4/4 CPU! NodeResourcesFit returns 0/2 nodes available: 2 Insufficient cpu. pg3 fails PlacementFeasible (pg3-pod-1 is skipped immediately). Now cpg-sub2 only has 1/2 scheduled children (`pg2`), so cpg-sub2 fails PlacementFeasible! Its revertFn rolls back pg2's reservations on node-1 and node-2, and completeCompositePodGroupAlgorithmResult overwrites pg2 from Success → Unschedulable (S → F)!

State Trace inside completeCompositePodGroupAlgorithmResultS → F Propagation

Group Raw Status (Before Complete) Final Status (After Complete) Action -------------------------------------------------------------------------------------------------- cpg-root Success (1/1 subtrees ok) Success Root succeeds! cpg-sub1 Success (1/1 PGs ok) Success Committed! └─ pg1 Success (node-1, node-2) Success 🟢 Bound to nodes! cpg-sub2 Unschedulable (1/2 PGs ok) Unschedulable Failed minGroupCount=2 ├─ pg2 Success (node-1, node-2) ==> Unschedulable (S -> F!) 🟠 Reverted & Sent to FailureHandler! └─ pg3 Unschedulable (No CPU) Unschedulable 🔴 pg3-pod-0 FailedFilter, ⚪ pg3-pod-1 SkippedInfeasible!
🔥 Run This Exact Scenario in /builder
Phase 5 · Post-Cycle Queue Reconciler
Where Do Failed, Rolled-Back & In-Flight Pods Go? AddAttemptedPodGroupIfNeeded
At the end of submitPodGroupAlgorithmResult, every pod that was not bound (failed filter, skipped, or rolled back S → F) plus any new pods that arrived while the CPG was popped are reconciled by PriorityQueue.AddAttemptedPodGroupIfNeeded.

1. Parking in pendingPodGroupPodsDuring Cycle

While cpg-root is popped from activeQ, activeQ.isLastPoppedEntity(cpg-root) is true.

• Every failed/rolled-back pod passed to FailureHandler -> AddUnschedulablePodIfNotPresent is intercepted and placed into pendingPodGroupPods[rootKey].
• Any brand-new pod arriving via PriorityQueue.Add() during the cycle ALSO goes into pendingPodGroupPods[rootKey]!

2. Rebuilding QueuedPodGroupInfoEnd of Cycle

AddAttemptedPodGroupIfNeeded(logger, podGroupInfo, cycle, rootStatus) drains pendingPodGroupPods[rootKey] and constructs a fresh QueuedPodGroupInfo containing:

[pg2-pod-0, pg2-pod-1, pg3-pod-0, pg3-pod-1, + any in-flight pods]

Notice that pg1-pod-0 and pg1-pod-1 are already bound in Cache, so they are NOT in the new QueuedPodGroupInfo!

3. Target Queue DecisionrootStatus Check

Where does the rebuilt QueuedPodGroupInfo go?

• If PreEnqueue fails on the remaining pods: parked in unschedulableEntities until more pods/events arrive.
• If rootStatus == Success (e.g. cpg-sub1 bound!): requeueWithoutBackoff = true & preserveTimestamp = true → goes straight into activeQ with zero backoff!
• If rootStatus == Unschedulable: increments UnschedulableCount and goes to backoffQ.