Skip to main content

Azure how-to guides

These guides cover the tasks of running Polaris on Azure, from a first deployment of your own workload through monitoring, upgrades and teardown. The Azure reference has the sizes, permissions and config keys they mention.

Run every command from the top folder of the Polaris tooling, and always from the same copy of it. That copy holds each deployment's Terraform state, tokens and pinned host keys, which status, promote, watch, destroy and the SSH commands below rely on.

Some tasks that exist on AWS are not available on Azure yet. Each section says so where that is the case.

Deploy your own workload​

Your workload fits Polaris when it follows the workload contract: one container built for linux/amd64, serving plain HTTP on one port and keeping its state outside the VM.

1. Sign in and choose a subscription​

Sign in with the Azure CLI. deploy stops with Azure CLI not authenticated. Run: az login otherwise, and it switches the CLI to cloud.azure.subscription_id for the rest of the run.

az login

The identity needs the rights in The identity that deploys, including creating role assignments.

The machine that runs deploy also needs Docker running, Terraform 1.6 or newer, openssl, and an SSH key pair. The public key defaults to ~/.ssh/id_rsa.pub.

2. Choose a region and sizes​

Pick a region that offers the size you want. For SEV-SNP, the defaults are Standard_DC2as_v6 for both VMs, and Standard_EC2as_v6 gives the workload more memory. For Intel TDX, set tee.cpu: tdx and the defaults become Standard_DC2es_v6.

cloud:
provider: azure
azure:
subscription_id: "<subscription id>"
location: eastus

tee:
cpu: sev-snp

compute:
cvm_machine_type: Standard_DC4as_v6

Keep metadata.name to 12 characters or fewer, since the Key Vault name is built from it.

3. Point the config at your image​

The VMs log in to the deployment's own container registry, so the simplest choice is to let the tooling build your image and push it there:

workload:
dir: my-workload
port: 8080

workload.dir is a folder with a Dockerfile, relative to the top folder of the tooling. To use an image you built yourself, put it in the deployment's registry and name it with workload.image, pinned by digest. An image in any other registry has to be public, since the VMs log in to no other.

Polaris checks the running container against the exact image it deployed, so ship each change under a new tag or digest.

4. Allow inbound clients and outbound destinations​

By default the endpoint on port 443 admits only the public address of the machine that ran deploy. Name your clients:

network:
workload_ingress_cidrs:
- "203.0.113.0/24"

The module refuses 0.0.0.0/0 and any other /0 or /1 range, so name your clients or the range of a CDN in front of the endpoint.

Outbound, both VMs can reach any internet address on ports 443 and 80. For other ports, list the ranges, which then open on every port:

network:
allowed_egress_cidrs:
- "198.51.100.10/32"

The runtime layer also learns where the workload connects, and under enforcement a connection to an address it has not seen counts as unknown behavior.

5. Turn on runtime and kernel evidence​

The Sentinel agent collects the runtime and kernel evidence, and on Azure it runs only when features.ebpf is true. The tooling's default is false.

features:
ebpf: true

6. Set configuration​

Pass settings as environment variables:

workload:
env:
LOG_LEVEL: info

The values are written into the VM's custom data, which anyone who can read the VM through the Azure API can read. The module keeps its own tokens out of custom data for that reason. So a secret goes in only as ciphertext: encrypt it with the deployment's gated key, keep the ciphertext in workload.env or your own storage, and have your workload decrypt it at run time, as Use the gated key from your workload shows. It can do so only while the deployment passes.

From Polaris Identity 1.0 an entry can also be a reference, polaris://secret/<name>, which the proxy fetches from your secret store only while the node passes. The workload contract's section on Identity, keys and secrets describes both, and what a revoke ends.

Set workload.resources.memory_mb and workload.resources.cpus as well, so the workload runs within limits and leaves room for Polaris's own components.

7. Open SSH for yourself​

cloud:
azure:
operator_ssh_cidrs:
- "203.0.113.4/32"

With it set, deploy reads each VM's SSH host keys from its boot diagnostics console and pins them, then captures the Policy Manager's identity record through the workload VM. Without it, deploy skips both and warns that Polaris Desktop has no record to check the Policy Manager against. status and promote reach the Policy Manager through the same SSH route, so they need it too.

8. Deploy and verify​

./scripts/deploy.sh deploy --config <file>

The VMs read their tokens from Key Vault at boot and retry for up to 10 minutes while their new role assignments take effect, so a first deploy can take several minutes after Terraform finishes.

On tee.cpu: sev-snp, deploy then checks the AMD firmware of the host each new VM landed on. Azure's SEV-SNP hosts do not all run the same firmware, and the Policy Manager refuses a workload VM, and the workload VM refuses a Policy Manager, on a host whose SNP firmware is below the level AMD bulletin AMD-SB-3020 requires: SPL 27 on Milan and Genoa, 4 on Turin. deploy reads each VM's level through the Azure agent, and deallocates and starts a VM below it, which usually moves it to another host. It tries this three times per VM before it stops. A VM is checked once, so a later deploy does not read or move a VM that is already serving.

Then check the deployment as in Verifying a Deployment, and watch learning with watch.

Use the gated key from your workload​

Your workload never holds an Azure credential, and its container cannot reach the instance metadata service. It asks the Polaris proxy in its own VM, on the proxy's internal listener, which only the workload's network reaches. The proxy calls Key Vault as the workload VM's managed identity, which holds Key Vault Crypto User on the key while the VM passes. The Policy Manager removes that role when a layer you enforce fails.

curl -s -X POST http://polaris-proxy:3000/polaris-container/key/encrypt \
-H 'Content-Type: application/json' -d '{"plaintext":"aGVsbG8="}'
# {"ciphertext":"..."}

curl -s -X POST http://polaris-proxy:3000/polaris-container/key/decrypt \
-H 'Content-Type: application/json' -d '{"ciphertext":"..."}'
# {"plaintext":"aGVsbG8="}
  • The routes are on in every deployment with a Policy Manager whose key is RSA or RSA-HSM, which includes the key the deployment creates. The proxy wraps with RSA-OAEP-256, which an EC key cannot do.
  • plaintext is base64, up to a little under 768 KiB, since the body is capped at 1 MiB. Treat the ciphertext as opaque. It names the key version that wrapped it, so old ciphertext still opens after the key rotates.
  • A 400 means the input was malformed or Key Vault could not use it. Anyone who can read the key's public half can make a ciphertext that opens, so a 400 does not tell you who made it. A 503 means there is no passing verdict yet, or Key Vault refused the call.
  • Nothing is cached, so every decrypt is a fresh Key Vault call. A plaintext your workload keeps in memory outlives a revoke, so decrypt per operation if a revoke has to stop it.
  • On a fail the proxy refuses at once. Key Vault stops the VM's identity once the Policy Manager deletes the role assignment, after a delay Azure does not bound and Polaris has not measured. A new assignment takes up to about five minutes, so plan for at least that. A Policy Manager that stops or hangs leaves the assignment in place.
  • The ciphertext decrypts only with this Key Vault key. Moving the deployment to another platform means re-encrypting your data there.

Move a layer to enforce​

A layer in advise reports findings and leaves key access alone, while a layer in enforce removes the key's role assignment when it fails. The baseline the tooling writes for Azure sets l2, l3 and l31 to advise, and the platform check always enforces.

promote reaches the Policy Manager over SSH through the workload VM, with the host keys deploy pinned for both VMs. Before you start, check that cloud.azure.operator_ssh_cidrs includes your address and that terraform/deploy/azure/terraform-<name>.known_hosts exists. A deploy with operator_ssh_cidrs set writes it. Without the pinned keys, promote stops rather than trust whatever host answers.

  1. Wait until the deployment has learned the workload:

    ./scripts/deploy.sh status --config <file>

    Overall and every layer should show pass, with Alerts at 0, and the autolearn section should show no learner still active. status prints one snapshot, so run it again until they do.

    The Policy Manager also refuses to enforce a layer until it has been clean under advise for the promotion window, a day by default, so that the VM's hourly and daily jobs have run at least once. To change the window, pass --promotion-window, such as 12h or 0, to deploy.

  2. Move one or more layers:

    ./scripts/deploy.sh promote --config <file> --layers l2,ima_boot,l3

    The layers are l2 (image), l3 (runtime), l31 (kernel) and ima_boot (boot-time measurements). The runtime layer also has three parts that you can set on their own, l3_syscalls, l3_binaries and l3_network. promote prints the settings the deployment has before it changes anything, refuses to enforce a layer whose allowlist is still empty, and reads the settings back to confirm they took.

    The Azure deployment sets ima_boot to advise on its own, so enforcing l2 leaves it advisory. Name it to enforce the boot-time measurements too, and promote checks its allowlist.

To keep network checks advisory while the rest of the runtime layer enforces:

./scripts/deploy.sh promote --config <file> --layers l2,ima_boot,l3_syscalls,l3_binaries
./scripts/deploy.sh promote --config <file> --layers l3_network --mode advise

To move a layer back, run promote with --mode advise.

While l2 or l31 enforces, the Policy Manager also raises l3, its three parts and ima_boot from advise to enforce on its own, once their learners have applied and each has been clean under advise for the promotion window. promote warns when a change turns this on. A layer promote sets to advise is not held there, since the only way to make the Policy Manager leave a layer alone is its PUT /api/v1/enforcement call, which deploy.sh has no command for yet. To be sure a layer stays advisory, such as l3_network above, send PUT /api/v1/enforcement with {"l3_network":"advise"} to the Policy Manager with the management token. A layer you move back after the Policy Manager raised it stays where you put it, until a deploy changes the baseline file and the Policy Manager starts watching every layer again.

The Policy Manager keeps these settings, and everything it learned, on its state disk, so they survive a restart and a replacement of the Policy Manager VM. A deploy that changes the deployment's baseline, such as a new workload image digest or a new VM image, returns the settings to advise and starts learning again, and you run promote again once the layers are ready.

Under l3 enforce, commands run in an SSH session on a VM can count as unknown behavior and remove key access, so plan hands-on maintenance for a time when a short withdrawal is acceptable.

The Marketplace template asks for an enforcement mode at deploy time instead. enforce there makes a failed image check or a live kernel change remove key access, while the runtime layer stays in advise.

Monitor a deployment​

The proxy serves three status routes on the deployment's endpoint. They stay reachable while the proxy is holding workload traffic.

RouteReturns
/polaris-container/health{"status":"ok"} while the proxy runs
/polaris-container/readyready and gate, with 503 when not ready
/polaris-container/attestation-statusThe latest verdict, with status, a status for each layer and checked_at. With the read-only token it adds the failure reason and alerts

Read them together like this:

  • The proxy has just started. On Azure the proxy has no start-up grace period, so it holds workload traffic from the moment it starts until the first passing verdict. attestation-status returns 503 with no appraisal yet until then.
  • A layer in advise found something. attestation-status shows alert, and traffic still flows.
  • The Policy Manager has stopped answering. attestation-status keeps showing the last verdict, with a checked_at that keeps getting older. Alert when it is more than a few report intervals old.
  • Your workload is down. The status routes look healthy, and requests to your workload fail.

status lists the VMs and the endpoint, and reads each workload VM's verdict, the license state and how far learning has got from the Policy Manager itself, over the same SSH route promote uses:

./scripts/deploy.sh status --config <file>

Without operator_ssh_cidrs and the pinned host keys, status lists the VMs and says why it did not read the Policy Manager.

For learning progress through the endpoint, with no SSH, run watch with the read-only token:

./scripts/deploy.sh watch --endpoint https://<address> \
--token-file terraform/modules/azure/terraform-<name>.readonly.token

Azure has no equivalent of the AWS grant retirer. If the Policy Manager stops or hangs, the workload VM's role assignment on the key stays until the Policy Manager runs again. Watch checked_at for that reason.

To send the Policy Manager's health metrics to your own collector, pass --otel-endpoint to deploy, and --otel-egress-cidrs when the collector is outside the virtual network.

Diagnose a withdrawal of access​

When requests start returning 503 with attestation pending, the Policy Manager's records say why, and you can read them through the endpoint with the read-only token.

  1. Keep the token at hand:

    TOKEN="$(cat terraform/modules/azure/terraform-<name>.readonly.token)"
  2. Find the workload VM's ID in the cvm_id field:

    curl -sk -H "Authorization: Bearer $TOKEN" https://<address>/management/status
  3. Read the latest incident as a plain-text report:

    curl -sk -H "Authorization: Bearer $TOKEN" \
    https://<address>/management/incidents/<cvm_id>/report
  4. For more context, /management/checks/<cvm_id> lists the latest checks, and /management/history/<cvm_id> returns recent verdicts.

On the Azure side, the Policy Manager's grant is a role assignment of Key Vault Crypto User on the workload key, described as polaris:cvm=<cvm_id>. Listing the role assignments on the key shows whether it is in place.

Read the workload's logs​

Container logs stay on the workload VM, and you read them over SSH. The pinned host keys are filed under the VM's vmUniqueId, so pass it as the host key alias:

cd terraform/deploy/azure
terraform output -state=terraform-<name>.tfstate -json cvm_vm_unique_ids
cd ../../..

ssh -i <private key> \
-o StrictHostKeyChecking=yes \
-o UserKnownHostsFile=terraform/deploy/azure/terraform-<name>.known_hosts \
-o HostKeyAlias=<vmUniqueId> \
azureuser@<address> 'sudo docker logs --tail 200 workload'

The workload container is named workload. The proxy is polaris-proxy and the Sentinel polaris-sentinel. SSH needs your address in cloud.azure.operator_ssh_cidrs. Commands you run over SSH can show up as runtime findings.

Change a deployment​

Edit the config and run deploy again.

  • A change to the workload VM's configuration, such as a new workload image or new environment variables, replaces it. The new VM has a new managed identity, which the Policy Manager's own configuration names, so the Policy Manager is replaced too.
  • The workload VM keeps its static public address across a replacement, since the address is a separate resource.
  • The replacement Policy Manager attaches the same state disk. It keeps what it learned and your promote settings unless the run changed the deployment's baseline, which names the workload image's digest, the workload port and the VM image. A new digest or a new VM image therefore returns every layer to advise and starts learning again.
  • Unless you set compute.cvm_image_id, the workload VM boots the version of Canonical's Ubuntu confidential VM image that latest names when deploy runs. When Canonical publishes a new version, the next deploy replaces the workload VM, and with it the Policy Manager, even if you changed nothing, and the new image in the baseline starts learning again.
  • A run that changes nothing, at a time Canonical has published nothing new, replaces nothing. --force-replace replaces the workload VM anyway, and with it the Policy Manager.
  • On tee.cpu: sev-snp, and on tee.cpu: tdx with a proxy that has the Azure TDX check and cloud.azure.pm_tdx_mrtds set, the workload VM checks the Policy Manager's attestation and pins the Policy Manager's VM ID, which it reads from Key Vault. A change to the Policy Manager's configuration alone, such as its baseline, its license or cloud.azure.id_key_digests, replaces only the Policy Manager, and the new one has a new VM ID, which the run writes to Key Vault. The workload VM reads it again every five minutes, so for up to five minutes after the new Policy Manager is up it refuses it, gets no verdicts and holds traffic. Then it restarts its proxy on the new VM ID. Run deploy --force-replace instead if you'd rather replace the workload VM too.
  • On tee.cpu: tdx without that check, the workload VM trusts the certificate the Policy Manager presented when the workload VM first booted, and never reads it again. The Policy Manager's proxy keeps its TLS key in memory, as it must on a confidential VM, so it presents a new certificate after every restart of its proxy, every reboot of the Policy Manager and every replacement of it. From then on the workload VM refuses the Policy Manager, gets no verdicts, and after about 150 seconds holds traffic, until the workload VM is replaced. deploy --force-replace replaces both. deploy warns about this on every TDX run without the check, and after a plan that would replace only the Policy Manager. Setting cloud.azure.pm_tdx_mrtds, as in Pin the Policy Manager's MRTD on TDX, moves the workload VM to the check, which follows a new certificate on its own.

set-workload changes the workload image without a deploy. It needs the endpoint, the management token or the workload-swap token from Files the tooling keeps, and an image the workload VM can pull:

./scripts/deploy.sh set-workload --endpoint https://<address> \
--token-file terraform/modules/azure/terraform-<name>.token \
--image <registry>/<repository>@sha256:<digest>

The Policy Manager replaces the image allowlist and resets its learner, and the workload VM pulls the image and restarts the container within about 30 seconds. Update workload.image in the config as well, so the next deploy names the same image.

To move a deployment to another region, tear it down and deploy it again with the new cloud.azure.location.

Pin the Policy Manager's MRTD on TDX​

On tee.cpu: tdx the workload VM checks the Policy Manager's TDX quote only when cloud.azure.pm_tdx_mrtds names the paravisor's MRTD, the measurement of Microsoft's paravisor that every TDX VM on Azure boots. Without it the check would accept a quote from any cloud TDX VM that names the Policy Manager's VM ID, which is weaker than trusting the certificate the Policy Manager first presents, so deploy keeps the certificate and warns:

The proxy this deploy runs has the Azure TDX PM verifier, but cloud.azure.pm_tdx_mrtds is not set, so it stays off ...

The check also needs a proxy that has it. With a release, that's one whose manifest says so. With --local-build, it's a proxy checkout that has pkg/pmverify/azure_tdx_verifier.go. When neither is true, deploy says so instead. The workload VMs then don't use the key, but deploy still stops when the Policy Manager's MRTD isn't in it, and Desktop's record holds the Policy Manager to it.

To set it:

  1. Deploy once without the key, with your address in cloud.azure.operator_ssh_cidrs. After the apply, deploy captures the Policy Manager's identity over SSH and prints its MRTD and the line to add:

    ▸ The PM's paravisor MRTD is <96 hex characters>
    ▸ To have the TDX workload VMs verify the PM, check it and set it in the config, then deploy again:
    ▸ cloud.azure.pm_tdx_mrtds: ["<96 hex characters>"]

    The same value is under mr_tds in terraform/deploy/azure/pm-identity-<name>.json.

  2. Check it. deploy read it from the Policy Manager's own evidence over the pinned SSH route, and does not verify the quote itself. The Policy Manager logs the MRTD of every TDX workload VM it appraises, and on one paravisor build in one region the two are the same, so compare them:

    ssh -i <private key> \
    -o StrictHostKeyChecking=yes \
    -o UserKnownHostsFile=terraform/deploy/azure/terraform-<name>.known_hosts \
    -o HostKeyAlias=<Policy Manager vmUniqueId> \
    -o ProxyCommand="ssh -i <private key> -o StrictHostKeyChecking=yes -o UserKnownHostsFile=terraform/deploy/azure/terraform-<name>.known_hosts -o HostKeyAlias=<workload vmUniqueId> -W <Policy Manager private address>:22 azureuser@<workload address>" \
    azureuser@<Policy Manager private address> \
    'sudo docker logs polaris-pm 2>&1 | grep "paravisor MRTD" | tail -n 1'

    terraform output -state=terraform-<name>.tfstate in terraform/deploy/azure gives pm_vm_unique_id, pm_private_ip, cvm_vm_unique_ids and cvm_public_ips. A value from another deployment in the same region on the same paravisor build should match too. Add the printed value only when it matches the one the Policy Manager logged. The printed value comes from the Policy Manager's own evidence, so on its own it shows nothing.

  3. Add it to the config and run deploy again:

    cloud:
    azure:
    pm_tdx_mrtds:
    - "<96 hex characters>"

    Setting the list for the first time turns the check on, which changes the workload VM's boot configuration, so the run replaces the workload VM, and with it the Policy Manager. deploy logs that the TDX workload VMs verify the Policy Manager's TDX quote, and the workload VM's proxy logs Azure TDX RA-TLS verification of PM succeeded with the MRTD it accepted.

When Microsoft updates the paravisor, a new VM can boot with a new MRTD. If the Policy Manager's is no longer in the list, its workload VMs refuse it with an error that its MRTD is not in the allowed Azure TDX paravisor measurements, and the next deploy stops after the apply, printing the new value, when it can reach the Policy Manager over SSH (cloud.azure.operator_ssh_cidrs). Without that route it reports success, and only the workload VMs' proxy logs show the refusal. Check it as above, against the MRTD the Policy Manager logs for its workload VMs, append it to cloud.azure.pm_tdx_mrtds rather than replacing the list, and deploy again. The list is kept in Key Vault as polaris-pm-tdx-mrtds, not in the workload VM's boot configuration, so a change to it alone replaces no VM. The workload VM reads the list again every five minutes and restarts its proxy on the new one, which then accepts the Policy Manager. A run that also changes something else in the workload VM's boot configuration, such as a new workload image, still replaces it.

This can also happen with no deploy at all. Host maintenance or a redeploy of the Policy Manager's VM can move it to a new paravisor build. Its workload VMs then refuse the Policy Manager, and nothing warns before they do. That fails closed, and the workload is down until the new MRTD is in the list. No deploy prints it then, so read it from the workload VM's proxy error, which names the MRTD it refused, and match it against the MRTD the Policy Manager logs for its workload VMs, as in step 2. The value in the proxy error comes from the Policy Manager's quote, so on its own it shows nothing. If the two differ, the workload VMs may still be on the older build. Then compare it with a TDX VM in the same region that booted on the new build, such as the Policy Manager logs of another deployment made since, before you add it. Then append it and deploy again. That run only rewrites the Key Vault list, and the workload VMs pick it up within about five minutes without being replaced. For now, watch the workload VMs' proxy logs for that error.

Take a security fix​

The VMs boot Canonical's Ubuntu 22.04 confidential VM image, and install Docker and its dependencies from the Ubuntu archive at first boot. A VM picks up newer packages when it is recreated. The workload VM also switches off automatic package updates at boot, since an update would change the files Polaris measures.

  • The workload VM boots the image version latest names when deploy runs, unless you set compute.cvm_image_id. When Canonical publishes a new version, a plain deploy replaces the workload VM, and the Policy Manager with it.
  • The Policy Manager VM boots whatever latest names when it is created, and a new version alone does not replace it.

To recreate both, whether or not a new image is out:

./scripts/deploy.sh deploy --config <file> --force-replace

This is a short outage. A new VM image changes the baseline, so Polaris learns the new VMs from the start, and you run promote again once learning has finished.

Upgrade Polaris​

A Polaris release brings a new version of the tooling, whose manifest names the exact Polaris images to run. Switch to the new tooling and run deploy. The new image references change both VMs' boot configuration, so both are replaced and both move to the new versions together. The Policy Manager's state disk carries over, so it keeps what it learned and your promote settings unless the run also changed the baseline.

To roll back, deploy again with the previous tooling.

Prepare for production​

  • Give the deployment a subscription of its own, and keep the list of people who can create role assignments in it short. Anyone who can change role assignments on the key, the vault or anything above them can grant access to the key, or widen the Policy Manager's role. The Policy Manager itself can assign Key Vault Crypto User on the key only to this deployment's workload VM identities.
  • Count write access to the deployment Key Vault's secrets as trust in the Policy Manager. Whoever can set polaris-pm-vm-unique-id, or on TDX polaris-pm-tdx-mrtds, decides which Policy Manager the workload VMs accept, within five minutes and with no VM replaced. The Policy Manager's own identity can only read them.
  • Know that the deploying identity keeps Key Vault Crypto Officer on the vault after deploy, which lets it use the key.
  • Set features.ebpf: true for runtime and kernel evidence.
  • Pin your workload image by digest.
  • Set network.workload_ingress_cidrs to your clients, and keep allowed_egress_cidrs to what your workload needs.
  • Keep operator_ssh_cidrs as narrow as you can, and set it before you rely on the Policy Manager identity record, status or promote, since all three go over SSH.
  • Enforce at least l2 and l3 with promote once learning has finished, with l3_network advisory if your workload's destinations change often. Name ima_boot along with l2.
  • Treat the tooling folder as a secret. The state, the tfvars file and the token files carry the deployment's tokens, and the tfvars file the registry password.
  • Monitor checked_at on attestation-status, since a stopped Policy Manager leaves the key's role assignment in place.
  • Plan for the key's lifetime. The deployment creates the key, and destroy deletes it along with the vault.
  • When Microsoft rolls a new paravisor signing authority, deploy stops and prints the new digest, when it can reach the Policy Manager over SSH. Append it to cloud.azure.id_key_digests once you have checked the rollout.
  • On tee.cpu: tdx, set cloud.azure.pm_tdx_mrtds, as in Pin the Policy Manager's MRTD on TDX, so the workload VM checks the Policy Manager's attestation instead of trusting its first certificate.

Troubleshoot​

deploy says Azure requires metadata.name to be at most 12 characters. The Key Vault name is built from it. Shorten the name.

deploy says Azure CLI not authenticated or Cannot access subscription. Run az login with an identity that can see cloud.azure.subscription_id.

deploy says the SSH public key was not found. Create a key pair, or point cloud.azure.ssh_public_key_file at an existing .pub file.

deploy reports unknown config keys. A key is misspelled or does not exist for this tooling. Check it against the Azure reference.

Terraform says the size does not match the TEE. tee.cpu: sev-snp needs an AMD size, such as Standard_DC2as_v6, and tee.cpu: tdx an Intel one, such as Standard_DC2es_v6, for both VMs.

deploy says tee.gpu nvidia-h100 requires --local-build. The release manifest pins no GPU build of the proxy. A GPU deployment also needs compute.cvm_image_id.

Terraform cannot write the key or the secrets. The vault's firewall admits the public address api.ipify.org reports for the machine that runs deploy. Run from a machine whose outbound address is stable and is the one Key Vault sees.

You cannot reach the endpoint. With network.workload_ingress_cidrs empty, the endpoint admits only the address that ran deploy. Set your clients' ranges and run deploy again.

The VMs take a long time to start. Each VM retries reading its tokens from Key Vault for up to 10 minutes while its new role assignment takes effect.

deploy says the PM's paravisor is signed by an ID key deploy does not accept. Microsoft may have rolled a new paravisor signing authority. Append the printed digest to cloud.azure.id_key_digests once you have checked the rollout, and run deploy again.

deploy warns there is no SSH route to the Policy Manager. cloud.azure.operator_ssh_cidrs is empty or the private key file is missing. The deployment works, while the host keys and the Policy Manager identity record are skipped.

deploy says the console never printed an SSH host key block. Cloud-init had not finished on that VM within 5 minutes. Run deploy again, or read the VM's boot log in the portal.

deploy says a VM is still on a host below the TCB floor. Azure placed that VM on a host whose AMD SEV-SNP firmware is older than AMD-SB-3020 requires, three times in a row. The Policy Manager and the workload VM would refuse it, so deploy stops rather than leave a deployment that cannot pass. The message names the VM, the host's SPL and the floor. This is the host's firmware, and nothing in the deployment changes it. Run deploy again later, which checks that VM again and tries other hosts, or deploy to another region.

deploy warns that it could not read a VM's SEV-SNP firmware level. The run-command through the Azure agent failed or printed no report, so that VM's host was not checked. The deployment goes on, and a later deploy doesn't read it again, since by then the VM may be serving. If the Policy Manager then refuses the VM with TCB snp below minimum, the host is below the floor, as above.

promote or status says there are no pinned host keys. deploy pins both VMs' SSH host keys only when cloud.azure.operator_ssh_cidrs is set. Add your address and run deploy again.

promote cannot reach the Policy Manager. cloud.azure.operator_ssh_cidrs no longer includes your address, or the private key is missing. Update it and run deploy, and run promote from the copy of the tooling that deployed, since it needs that copy's state, token and pinned host keys.

promote refuses a layer. Either that layer's allowlist is still empty, so learning is still running, or the layer has not yet been clean under advise for the promotion window, a day by default. promote prints the reasons. Wait and try again.

Requests return 503 with attestation pending. The proxy is waiting for a verdict or holding traffic after a failure. Read the status routes and the latest incident, as in Diagnose a withdrawal of access.

watch cannot reach the Policy Manager. It may still be starting. Otherwise check that you are using the workload VM's endpoint and a token for this deployment.

Learned allowlists are gone and layers are back in advise. A deploy changed the deployment's baseline, for example with a new workload image digest or a new VM image from Canonical. A restart or replacement of the Policy Manager alone keeps them. Run promote again once learning has finished.

deploy stops after the apply, saying the Policy Manager's paravisor MRTD is not in cloud.azure.pm_tdx_mrtds. The Policy Manager booted on a paravisor build the list doesn't name, and the workload VMs refuse it until it does. Check the new value and append it, as in Pin the Policy Manager's MRTD on TDX.

The workload VM stops reaching the Policy Manager after a deploy, on tee.cpu: sev-snp, or on tdx when the workload VM checks the Policy Manager. The run replaced only the Policy Manager, and the workload VM still holds the old one's VM ID, so its proxy logs a vmUniqueId mismatch. It reads the new VM ID from Key Vault every five minutes and restarts its proxy on it, which journalctl -t polaris-pm-pin-refresh on the workload VM shows. If it hasn't after ten minutes, check that the workload VM can read the polaris-pm-vm-unique-id secret, or run deploy --force-replace, as in Change a deployment. deploy warns about this after the plan, naming the workload VMs it would leave in place.

The workload VM stops reaching the Policy Manager on tdx when it does not check the Policy Manager, after the Policy Manager or its proxy restarted. The workload VM pins the certificate the Policy Manager presented at the workload VM's first boot, and the Policy Manager's proxy makes a new one every time it starts, so the workload VM's proxy fails TLS to the Policy Manager from then on. It does not recover on its own. Run deploy --force-replace, as in Change a deployment, and set cloud.azure.pm_tdx_mrtds so it doesn't happen again.

The Policy Manager does not start after you changed attestation.trust_domain or attestation.deployment_id. Its state disk holds a record written under the old values, and it refuses to start against a record with another deployment ID. Set them back.

destroy says there is no state file. Run it from the copy of the tooling that deployed.

Tear down a deployment​

./scripts/deploy.sh destroy --config <file>

destroy runs terraform destroy, which removes the resource group and everything in it: the VMs and their disks, the Policy Manager's state disk with what it learned, the network, the custom role, the role assignments Terraform made, and the Key Vault with the workload key and the secrets. It then deletes the token files. The role assignment the Policy Manager made on the key is not in Terraform's state.

The key goes with the deployment

On Azure the deployment creates the workload key, and destroy deletes it. Azure keeps the deleted vault and key in a soft-deleted state for the retention window, 90 days by default, and purge protection stops anyone purging them sooner. Data encrypted under the key can be decrypted again only if the vault is recovered within that window.

The container registry and its resource group remain, since other deployments in the subscription may share them. So do the tooling folder's state, tfvars, known hosts and identity record files, listed in Files the tooling keeps. Delete them once you no longer need them, since the state and tfvars carry the deployment's tokens.

Tear down with destroy rather than by deleting the resource group in the portal, so the tooling's state stays in step.