Skip to content

fix(azure): set ProvisionVMAgent=false for peer pod VMs#55

Closed
alhassankhedr-cohere wants to merge 1 commit into
coherefrom
alhassankhedr/azure-provisionvmagent-false
Closed

fix(azure): set ProvisionVMAgent=false for peer pod VMs#55
alhassankhedr-cohere wants to merge 1 commit into
coherefrom
alhassankhedr/azure-provisionvmagent-false

Conversation

@alhassankhedr-cohere

@alhassankhedr-cohere alhassankhedr-cohere commented Jul 3, 2026

Copy link
Copy Markdown

Summary

  • Set ProvisionVMAgent=false in the Azure provider's VM OS profile so peer pod VM creation completes without waiting on an in-guest agent/afterburn check-in.

Why

Peer pod VMs are ephemeral and never run the Azure Linux guest agent (waagent). With the default ProvisionVMAgent=true, Azure keeps the VM in ProvisioningState=Creating until an in-guest check-in reports ready. The Ubuntu podvm image enables afterburn-checkin.service for all distros but only fedora.conf installs the afterburn binary — so on the Ubuntu tdx-snp-nvidia image the check-in never fires. Result: the VM boots (PowerState=running) but ProvisioningState stays Creating, CAA's CreateInstance blocks in PollUntilDone until the CRI context is canceled, the VM is abandoned, and kubelet retries — leaking a VM per attempt.

Observed directly while bringing up an Standard_NCC40ads_H100_v5 confidential GPU peer pod in eastus2.

Fix

ProvisionVMAgent=false tells Azure not to expect a guest agent; provisioning completes once the VM boots. Distro-agnostic and the correct model for agentless peer pod VMs.

Test plan

  • Rebuild CAA image, redeploy the daemonset.
  • Launch a kata-remote H100 pod; confirm the peer pod VM reaches Succeeded and the pod goes Running.

Note

Medium Risk
Changes Azure VM provisioning behavior on the CreateInstance hot path; low blast radius but affects all new peer pod VMs on Azure.

Overview
Sets ProvisionVMAgent: false on the Linux OS profile in getVMParameters when Azure peer pod VMs are created.

Azure no longer waits for waagent or afterburn guest check-in before the create LRO succeeds, so CreateInstance can finish PollUntilDone after the VM boots instead of staying in Creating until the CRI context is canceled (e.g. Ubuntu podvm images without afterburn).

This matches ephemeral, agentless peer pod VMs and avoids leaked VMs from kubelet retries.

Reviewed by Cursor Bugbot for commit fd14358. Bugbot is set up for automated code reviews on this repo. Configure here.

Peer pod VMs are ephemeral and never run the Azure Linux guest agent
(waagent). With the default ProvisionVMAgent=true, Azure keeps the VM
in ProvisioningState=Creating until an in-guest agent/afterburn check-in
reports ready. The Ubuntu podvm image enables afterburn-checkin.service
but never installs the afterburn binary (only fedora.conf does), so the
check-in never fires and the VM create LRO never completes. CAA's
CreateInstance blocks in PollUntilDone until the CRI context is canceled,
abandoning the VM and triggering an endless retry loop that leaks VMs.

Setting ProvisionVMAgent=false tells Azure not to wait for a guest agent,
so provisioning completes once the VM boots. This is the correct model
for agentless peer pod VMs and is distro-agnostic.

Signed-off-by: Alhassan Khedr <alhassan.khedr@cohere.com>
@alhassankhedr-cohere

Copy link
Copy Markdown
Author

Closing in favor of the proper fix: installing afterburn on the Ubuntu podvm image (PR #53, commit f648055). afterburn's --check-in reports provisioning completion to the Azure fabric, which flips the VM's provisioningState Creating -> Succeeded on its own — so forcing ProvisionVMAgent=false is no longer necessary.

Validated end-to-end: an H100 peer pod on the afterburn image reaches provisioningState=Succeeded (~120s) and the pod goes 1/1 Running.

@alhassankhedr-cohere alhassankhedr-cohere deleted the alhassankhedr/azure-provisionvmagent-false branch July 3, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant