fix(e2e): improve e2e test reliability (16 fixes)

[timmy-wright]

E2E Reliability Improvements

This PR implements 16 targeted reliability fixes to the AgentBaker e2e test suite identified through a comprehensive code review.

Critical fixes

• Do not memoize transient errors in cachedFunc (prevents suite-wide poisoning)
• Avoid mutating global DefaultVMSKU under parallel tests (fixes data race)
• Wait for VMSS deletion and register cleanup immediately after create (prevents orphans)
• Propagate context cancellation through SSH exec and bastion paths

Systematic flakiness fixes

• Retry transient API errors in wait/poll loops instead of failing immediately
• Replace fixed sleeps with polling loops in validators
• Add jitter to retry delays to prevent synchronized storms
• Fix WaitUntilPodRunning to not accept PodSucceeded; add WaitUntilPodCompleted
• Treat ssh.ExitMissingError as failure, not success

Correctness & debuggability

• Remove nested t.Parallel() to reduce isolation pressure
• Implement real deep copy in copyScenario
• Generate unique pod names per retry attempt
• Fix sysctl arg slice init (leading empty strings)
• Fail Windows version validator when settings unreadable
• Require full type+reason+message match in validateNPDCondition
• Add cycle detection to DAG

Closes AB#38423358

[Copilot]

Pull request overview

This PR targets reliability and debuggability improvements across the AgentBaker e2e test suite, focusing on reducing flakiness (transient Azure/K8s errors, fixed sleeps) and improving cleanup/cancellation behavior.

Changes:

• Improve polling/retry behavior (transient error handling, jittered backoffs, replace some fixed sleeps with polling).
• Harden VMSS lifecycle handling (earlier cleanup registration, wait for VMSS deletion completion, avoid mutating global VM SKU under parallelism via per-scenario VMSize).
• Improve correctness primitives (pod readiness semantics, NPD condition matching/debug output, DAG cycle detection, SSH exec context cancellation propagation).

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
e2e/vmss.go	Registers cleanup earlier, waits for VMSS deletion to complete, and uses per-scenario VM size in the VMSS model.
e2e/validators.go	Fixes sysctl key slice init, replaces restart sleep with polling, strengthens NPD validation logging, hardens Windows settings read, and improves transient retry logging.
e2e/validation.go	Adds per-retry unique pod naming and improves resource polling cancellation diagnostics.
e2e/types.go	Adds ScenarioRuntime.VMSize to avoid global DefaultVMSKU mutation.
e2e/test_helpers.go	Removes top-level parallelization, implements a deeper scenario copy for parallel subtests, and computes/stores an effective VM SKU per scenario.
e2e/kube.go	Splits pod wait into running vs completed modes, adds transient error handling, and uses container readiness checks.
e2e/exec.go	Propagates context into SCP timeout, supports SSH exec cancellation, treats ExitMissingError as failure, and adds jitter to pod-exec retries.
e2e/dag/dag.go	Adds runtime cycle detection for explicitly declared dependencies.
e2e/cluster.go	Adds transient error handling in cluster readiness polling and jitter to cluster creation retries.
e2e/cache.go	Avoids memoizing transient errors by caching only successful results.
e2e/bastionssh.go	Propagates context through token/session creation and adds jitter to bastion SSH dial retries.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

e2e/cluster.go:497

• math/rand is never seeded in this repo, so the jitter here is deterministic across processes (default seed = 1). That reduces the benefit of jitter for de-synchronizing retries in CI runs. Consider using a per-call RNG seeded with time, or a package-level RNG seeded once at startup.

	// Note, it seems like the operation still can start a trigger a new operation even if nothing has changes
	pollerResp, err := config.Azure.AKS.BeginCreateOrUpdate(
		ctx,
		config.ResourceGroupName(*cluster.Location),
		*cluster.Name,
		*cluster,

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.