fix(azure): set ProvisionVMAgent=false for peer pod VMs#55
Closed
alhassankhedr-cohere wants to merge 1 commit into
Closed
fix(azure): set ProvisionVMAgent=false for peer pod VMs#55alhassankhedr-cohere wants to merge 1 commit into
alhassankhedr-cohere wants to merge 1 commit into
Conversation
Peer pod VMs are ephemeral and never run the Azure Linux guest agent (waagent). With the default ProvisionVMAgent=true, Azure keeps the VM in ProvisioningState=Creating until an in-guest agent/afterburn check-in reports ready. The Ubuntu podvm image enables afterburn-checkin.service but never installs the afterburn binary (only fedora.conf does), so the check-in never fires and the VM create LRO never completes. CAA's CreateInstance blocks in PollUntilDone until the CRI context is canceled, abandoning the VM and triggering an endless retry loop that leaks VMs. Setting ProvisionVMAgent=false tells Azure not to wait for a guest agent, so provisioning completes once the VM boots. This is the correct model for agentless peer pod VMs and is distro-agnostic. Signed-off-by: Alhassan Khedr <alhassan.khedr@cohere.com>
Author
|
Closing in favor of the proper fix: installing afterburn on the Ubuntu podvm image (PR #53, commit f648055). afterburn's Validated end-to-end: an H100 peer pod on the afterburn image reaches provisioningState=Succeeded (~120s) and the pod goes 1/1 Running. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ProvisionVMAgent=falsein the Azure provider's VM OS profile so peer pod VM creation completes without waiting on an in-guest agent/afterburn check-in.Why
Peer pod VMs are ephemeral and never run the Azure Linux guest agent (waagent). With the default
ProvisionVMAgent=true, Azure keeps the VM inProvisioningState=Creatinguntil an in-guest check-in reports ready. The Ubuntu podvm image enablesafterburn-checkin.servicefor all distros but only fedora.conf installs theafterburnbinary — so on the Ubuntutdx-snp-nvidiaimage the check-in never fires. Result: the VM boots (PowerState=running) butProvisioningStatestaysCreating, CAA'sCreateInstanceblocks inPollUntilDoneuntil the CRI context is canceled, the VM is abandoned, and kubelet retries — leaking a VM per attempt.Observed directly while bringing up an
Standard_NCC40ads_H100_v5confidential GPU peer pod in eastus2.Fix
ProvisionVMAgent=falsetells Azure not to expect a guest agent; provisioning completes once the VM boots. Distro-agnostic and the correct model for agentless peer pod VMs.Test plan
kata-remoteH100 pod; confirm the peer pod VM reachesSucceededand the pod goesRunning.Note
Medium Risk
Changes Azure VM provisioning behavior on the CreateInstance hot path; low blast radius but affects all new peer pod VMs on Azure.
Overview
Sets
ProvisionVMAgent: falseon the Linux OS profile ingetVMParameterswhen Azure peer pod VMs are created.Azure no longer waits for waagent or afterburn guest check-in before the create LRO succeeds, so
CreateInstancecan finishPollUntilDoneafter the VM boots instead of staying in Creating until the CRI context is canceled (e.g. Ubuntu podvm images without afterburn).This matches ephemeral, agentless peer pod VMs and avoids leaked VMs from kubelet retries.
Reviewed by Cursor Bugbot for commit fd14358. Bugbot is set up for automated code reviews on this repo. Configure here.