Skip to main content

Deploying on your own hardware

Polaris on hardware you own needs no cloud attestation service, no cloud identity, and no outbound internet once the artifacts are in place. Read Deploying Polaris first for the choices common to every environment. This page covers what is specific to bare metal, including the air-gapped case.

On-premises deployments are delivered as engagements with our team, so you will not be doing this alone. This page is what we work through together, and it is written so your own operators can run it afterwards.

What this gives you, and what it does not​

Worth being plain about the boundary before anything else, because it differs from the cloud deployments.

On your own hardware you get memory confidentiality for the workload against host software, since SEV-SNP encrypts guest memory with a key the hypervisor cannot read. You get integrity monitoring, because the sentinel watches syscalls, network destinations, kernel modules and the IMA log while the Policy Manager appraises what it sees against a baseline. And you get key gating, because once a layer is promoted from advisory to enforcing, a failed appraisal stops the workload being served.

What it does not give you is a defence against whoever owns the machine. For this release the host is inside the trust boundary. The person with root controls the key manager, the baseline and the artifacts the guests are launched from, so an operator who wants the keys can have them. Polaris on-premises is there to detect drift in your own estate and to gate keys on that, not to protect a workload from its own infrastructure owner. If your case has the workload owner and the host owner as different parties, tell us, because that needs a different design.

Before you start​

You need an AMD EPYC server with SEV-SNP, Milan generation or later, and SEV-SNP enabled in firmware. On most vendors that is more than one setting, typically SEV, SEV-SNP and an SNP memory reservation, so consult your vendor's documentation rather than assuming a single toggle. Two guests is the minimum, one for the workload and one for the Policy Manager, so you need the ASIDs for both.

Confirm the host agrees before going further:

cat /sys/module/kvm_amd/parameters/sev_snp # expect Y
ls -l /dev/sev # must exist
dmesg | grep -i 'SEV-SNP' # the firmware version the PSP reports

If sev_snp reports N or the parameter is absent, the fault is in firmware or the kernel and nothing below will work.

Record three numbers from the machine that will run the guests:

grep -E '^cpu family|^model[[:space:]]|^stepping' /proc/cpuinfo | head -3

These go into the configuration and they are required. CPUID 1 EAX appears in every VMSA and every VMSA is covered by the launch measurement, so a set of artifacts is valid for one CPU model. Launching them on a host whose family, model or stepping differs is refused rather than silently accepted.

Polaris supports the RHEL 10 family, such as Rocky Linux 10.2, with libvirt-daemon, qemu-kvm, virt-install, guestfs-tools and python3 installed. The Policy Manager's measured boot needs a QEMU and libvirt that support kernel hashes for SEV-SNP guests, which Rocky Linux 10.2's do. The machine that builds the artifacts needs Docker: the guest images are baked in containers and the firmware is built in one.

SELinux​

SELinux Enforcing is the RHEL default and is supported. The deployment tool labels what it stages, because the artifact directory is not a path the libvirt policy knows and the defaults there are not usable by the processes that need them:

  • the guest firmware as virt_content_t, which is the type qemu may read. Without it a guest fails with qemu: could not load PC BIOS and no mention of SELinux.
  • a serial/ directory for the guests' console logs as virt_log_t, root-owned and mode 0700. virtlogd has to create and rotate files there, which needs permission on the directory and not just the files.

Labels are applied with semanage fcontext where policycoreutils-python-utils is installed, so they survive a filesystem relabel, and with chcon otherwise. On an Enforcing host the tool stops if it cannot label at all, rather than leaving you a deployment that will not start.

If something still does not start, the host has already recorded why. ausearch -m AVC -ts recent, or grep 'avc: *denied' /var/log/audit/audit.log, names the process and the path. On a Permissive host the same lines appear with permissive=1, meaning the access was allowed but would have been blocked, so a Permissive host will tell you what Enforcing is going to break before you switch.

TEE options​

AMD SEV-SNP, verified directly against AMD's published keys. There is no cloud attestation service in the path and no external attestation authority, which is the property that makes this configuration suitable for air-gapped and sovereign environments.

Confidential GPUs are supported on-premises as they are in the clouds, with the caveat that NVIDIA's attestation service is the one outbound dependency Polaris cannot remove when they are in play.

No vTPM on bare metal​

An on-premises guest has no virtual TPM. A vTPM inside the trust boundary needs something beneath the guest to host it, an SVSM or a cloud provider's paravisor, and the on-premises deploy runs neither. A host-run emulator such as swtpm was considered and rejected: the host would own it, so its quotes would prove nothing about the guest. So the five checks that need a TPM quote do not run here. They are the IMA log replay into PCR-10, the PCR-10 quote, the behavioral PCR chain replay, the attestation key's binding to an endorsement key and the behavioral PCR quote. The check list shows them as n/a with the reason "no vTPM on this platform", and an n/a check does not change the verdict. The IMA log and the behavioral digest are still appraised, but nothing beneath the guest kernel vouches that the guest reported them in full. The hardware report, the launch measurement and the kernel checks are unaffected. The way to close this is an SVSM running a vTPM at VMPL0 beneath the guest, which is on the roadmap and not yet supported on-premises.

How you deploy​

Start from the shipped on-premises example configuration and read its comments, which are the reference for what each key changes. Five values have no default and the tool stops without them: an NTP server, the Policy Manager's address, and the three host CPU numbers above. There is no metadata service on bare metal, which is why the time source and the Policy Manager address have to be stated rather than discovered.

To see what a configuration produces without building anything:

./scripts/deploy.sh deploy --dry-run --cloud onprem --config onprem.yaml

A dry run measures nothing, so its artifact set has no Policy Manager pin. When the artifact directory already holds a set from an earlier deploy, the dry run writes to dry-run/ inside it and leaves that set alone.

Then write the deployment's access policy (next section), bake the artifacts and launch:

./scripts/deploy.sh deploy --cloud onprem --config onprem.yaml \
--image-tag <tag> --workload-dir ./workload \
--access-policy-v1 access-policy-v1.signed.json
./scripts/deploy.sh launch --cloud onprem --config onprem.yaml

On this path the tool acquires no container images. It expects your Docker daemon to already hold them and exports exactly those references into the guest image, so if you build elsewhere then save, copy and load them on this host first.

With cloud.onprem.vault set, bake Policy Manager v1.1.0-rc.49 or later and proxy v1.1.0-rc.34 or later. The Vault policy lets the Policy Manager change only the certificate field of its certificate entries, and an older Policy Manager writes the whole entry, gets refused, and never publishes its CAs, so no node can log in. The bake refuses a Policy Manager image whose release label is older, and warns when the image has no label, as a local build doesn't. An older proxy gets no workload certificate from rc.49, so it can't read secrets.

The bake prints two launch measurements, one per guest. Both are computed from the same staged firmware, and they cover different things.

The workload guest boots from its disk, so its measurement covers the firmware and its vCPUs, not the disk. Container images are not part of it, so rebuilding an image leaves it unchanged. Catching a changed container is the job of the runtime layers and the IMA allowlist.

The Policy Manager boots measured. Its kernel, initrd and kernel command line are handed to the guest directly, with kernel hashes on, so the launch measurement covers all three. The command line carries the dm-verity root hash, salt and size of its root filesystem, so the measurement covers every block of that too, and the guest checks each block as it reads it. It runs its two containers only from the images on that root filesystem, checked against the digests the bake recorded, and never pulls one. The root filesystem is read-only with a RAM overlay on top, and the Policy Manager does not run cloud-init: it takes a fixed set of files and settings from its launch seed, and the settings that decide how its issuer keys are kept are part of the measured root filesystem. So the Policy Manager measurement pin the workload checks, and the identity record the owner checks in Polaris Desktop, accept only the Policy Manager this bake built. A guest booted with another kernel, command line or root filesystem reports a different measurement and is refused. Its state disk and launch seed are outside the measurement. The seed settings that decide verdicts are bound through HOST_DATA instead, as What the launch seed can change describes.

What the launch seed can change​

The launch host writes the Policy Manager's seed, so anything on it is the host's to choose unless something checks it. Deploy hashes the settings that decide verdicts and identity, with access policy version 1, into one launch manifest, and launches the Policy Manager with the manifest's SHA-256 as its HOST_DATA. HOST_DATA is in the Policy Manager's SNP report. The CVM's Policy Manager pin names it, and the identity record Desktop checks carries the manifest as launch_manifest. At every start the Policy Manager rebuilds the manifest from its own seed and refuses to start when it does not hash to its HOST_DATA. The refusal lists what the seed gave, so you can compare it with the record.

So a host that hands the Policy Manager another baseline gets a Policy Manager that does not start. If the host also changes HOST_DATA to match, the workload's pin and Desktop refuse that Policy Manager.

Seed settingBoundWhy
baseline.yamlyesIt is what the Policy Manager appraises CVMs against.
access-policy-v1.jsonyesIt names the owners.
PM_DEPLOYMENT_ID, PM_TRUST_DOMAINyesThey name the SPIFFE IDs the Policy Manager signs.
PM_AUTOLEARN_ENABLED, PM_AUTOLEARN_MIN_OBSERVATION_SECONDS, PM_PROMOTION_WINDOW_SECONDSyesWith learning on and no window, whatever a CVM runs is learned and promoted.
PM_LOG_LEVEL (info, warn or error), PM_YANG_PUSH_ENABLEDnoThey decide no verdict.
license.jws, snp-cert-chain.pemnoFr0ntierX and AMD sign them, and the Policy Manager checks those signatures.
vault-token, vault-ca.pemnoThey decide which Vault the Policy Manager logs in to. The token is single use (see How the Policy Manager's token reaches it).
PM_MANAGEMENT_TOKEN, PM_READONLY_TOKEN, PM_WORKLOAD_TOKENnoSee below.

The management token is not bound, and binding its hash would not help, because the host writes it and so knows it. Every bearer write to the Policy Manager still takes it, including a baseline upload, the IMA allowlist, the kernel golden values and the profiles. So whoever controls the host can still change the baseline at runtime through the Policy Manager's API, even though it cannot change the one the Policy Manager starts with. That stays true until the on-premises Policy Manager stops taking the management token for writes.

The binding covers the seed, not the state disk. The state disk (pm-state.qcow2) is a plain filesystem the host attaches and can write. The Policy Manager restores its applied baseline and enforcement dials from it at start, over the seed's baseline, and accepts them when they are signed with keys from its issuer key set. Under the default cloud.onprem.pm_issuer_key_protection: file, that set sits unwrapped on the same disk. So a host can read the keys, write a signed applied baseline for the bound template, and the Policy Manager starts on it. The state disk holds against the host only under owner-unseal, where the issuer key set is never on a disk. Closing this for the other protections is a separate follow-up.

Changing a bound setting changes HOST_DATA, so it takes a new bake and launch, like a new Policy Manager image. Relearning does not: learned allowlists and promoted dials live on the state disk, with the limit above, and the seed's baseline is only the template they started from. A Policy Manager from this release refuses the HOST_DATA an earlier deploy wrote. That was the hash of version 1, which bound version 1 alone, or, for a deployment without version 1, the deployment and node ids, which bound nothing. Upgrade the Policy Manager and redeploy together, with a Polaris Desktop that reads launch_manifest.

The tool still refuses a configuration that gives both guests the same vCPU count, a bake whose two measurements come out equal, and a launch from an artifact set that gives both guests the same count.

To take the guests down without discarding the artifacts, use destroy with the same configuration.

Who manages the deployment​

The Policy Manager takes management requests signed by device keys that are listed in the deployment's access policy. The deployment does not issue tokens for people. The policy's first version comes with the install, and nothing in the system creates it. Each later version has to be signed by the keys the version before it names.

Version 1 lists:

  • the installer's device key as the one owner. It is created in Polaris Desktop's Secure Enclave and never leaves the Mac, so Desktop has to sign version 1. Deploy cannot sign it.
  • a break-glass key. You create it offline, for example on a PIV token that stays in a safe, and only its public key goes into the policy. It acts as an owner, and the Policy Manager logs a warning each time it is used. It exists so that losing the installer's Mac does not lose the deployment.
  • optionally, a read-only device and a workload-swap device. These replace the read-only and workload-swap tokens.
  • no Policy Manager keys yet. The Policy Manager creates them on its first boot, and the owner adds them in version 2 the first time Desktop connects.

Write version 1, have the owner device sign it, then pass the signed file to deploy:

./scripts/deploy.sh policy-v1 --cloud onprem --config onprem.yaml \
--owner-device-jwk installer.jwk --owner-label "installer mac" \
--break-glass-jwk break-glass.jwk \
--reader-device-jwk ci-reader.jwk \
--out access-policy-v1.json
# Sign access-policy-v1.json in Polaris Desktop with the installer's device key.
./scripts/deploy.sh deploy --cloud onprem --config onprem.yaml ... \
--access-policy-v1 access-policy-v1.signed.json

Each key file is a P-256 public key, as a JWK or a PEM. The deployment id and trust domain come from the configuration, so run policy-v1 with the same configuration you deploy with. Deploy checks the signed file the same way the Policy Manager will, and stops if the Policy Manager would refuse it. For a lab or CI that has no device key, --owner-signing-key <pem> --break-glass-jwk <jwk> writes and signs version 1 during the deploy using a software key. That makes whoever holds the PEM the owner.

Deploy keeps the file in the artifact set as access-policy-v1.json, and records its hash in three places:

  • it is in the Policy Manager's launch manifest, whose hash is the Policy Manager guest's HOST_DATA, and the CVM's Policy Manager pin names the same value. The Policy Manager refuses to start with a version 1 that is not the one its HOST_DATA binds (What the launch seed can change).
  • /api/v1/status shows it as access_policy.genesis_hash.
  • the Policy Manager identity record that Desktop checks the Policy Manager against carries it as access_policy_genesis_hash, and so does the deploy's complete event. The record also carries the whole launch manifest as launch_manifest.

Version 1 is fixed for the life of the deployment, like the deployment id. Once the Policy Manager has stored a policy chain, it keeps that chain and only compares it against the file. So a redeploy reuses the kept file, and refuses a different version 1. Make access changes as new versions from Desktop.

Without --legacy-tokens, an on-premises deployment does not create the read-only or workload-swap token and does not give anyone the management token. The management token still exists between the workload guest's proxy and the Policy Manager, because the 1.0 Policy Manager will not start without one and the proxy uses it to fetch its nonce and the access policy. It stays in secrets/ on the launch host. Treat that directory as you treat the host, which is inside the 1.0 trust boundary.

Without the tokens, deploy.sh watch and deploy.sh set-workload have no credential to use on-premises: both take a bearer token, and the proxy's /management/ path does not take a device's signed request yet. Use --legacy-tokens if you need them before Desktop signs with the enrolled devices. A launch follows what the deploy recorded, so launch --legacy-tokens does not bring the tokens back.

An existing deployment moves over by having an admin write version 1 with the management token (POST /api/v1/identity/policy), then redeploying with that file as --access-policy-v1. --legacy-tokens keeps the old behaviour with all three tokens and no policy. Use it only until the move is done.

Verification path​

Read the verdict from the Policy Manager through its own attested TLS front. Polaris Desktop does this with the device key it enrolled. Without Desktop, a device in the policy signs each request with DPoP. On a deployment made with --legacy-tokens, the read-only token the deployment generated also works:

curl -sk -H "Authorization: Bearer $READONLY_TOKEN" https://<pm_ip>:443/api/v1/status
curl -sk -H "Authorization: Bearer $READONLY_TOKEN" https://<pm_ip>:443/api/v1/checks

See Verifying a deployment for what each layer means.

Two things to expect. The first cycles will raise findings, because the learners converge over roughly a day and a converged deployment reports a passing status with no alerts. And the Policy Manager's alert list is the observer, not the guest's console log. A console log shows only what the containers chose to write, so when what you are checking is that something did not happen, read the alert list.

Baselines ship with every layer advisory, which means findings are raised and no key is ever withheld. Promoting a layer to enforcing is a deliberate step, because enforcing a layer whose allowlist is still empty is either inert or immediately fatal depending on the layer. A layer reading clean on one status check is not enough to go on: advise and enforce make the same call each cycle, so a layer that alerts now and then reads clean on most checks and then fails the first cycle after promotion. So the Policy Manager decides. It refuses to raise a dial until it has been clean under advise for a day, with at least 90% of the report slots in that day covered, and promote prints its reasons when it refuses. The status response's promotion_readiness shows each dial's standing. attestation.promotion_window (or --promotion-window) sets the day, and 0 turns the check off for a lab. On-premises:

./scripts/deploy.sh promote --cloud onprem --config onprem.yaml \
--layers l31 --mode enforce

--layers takes any of l2, l3, l31, ima_boot, l3_syscalls, l3_binaries and l3_network, comma-separated. The three l3_ dials govern one L3 category each and follow l3 while unset, so syscalls can enforce while network still follows an advising l3. --mode is enforce, advise or off. While l2 or l31 enforces, the Policy Manager raises l3 and its three parts from advise to enforce on its own once their learners have applied, and promote warns when a change turns that on. The on-prem baseline leaves ima_boot unset, so it follows l2 and enforces as soon as l2 does. A promote --mode advise does not hold a layer against it, only PUT /api/v1/enforcement with the management token does, for example {"l3":"advise"}. Add --dry-run to see the current dials and what the change would be without sending it, which also prints what each layer has actually learned so far.

Once a layer is enforcing, a failed appraisal stops the workload being served. There is no cloud key service to revoke on your own hardware, so what you observe instead is the proxy refusing traffic and its readiness route reporting unavailable, which takes the workload out of service behind any load balancer in front of it. The timing is a budget of failed reports rather than a single one, so expect a couple of minutes between attestation stopping and traffic being refused, and recovery within seconds of the cause being resolved.

Key management​

Key release integrates with your own key manager rather than a cloud service. For HashiCorp Vault, Polaris uses the transit engine for keys and reads secrets from a KV v2 engine you own, and needs no enterprise features. No namespaces, no Vault Agent, and nothing installed: built-in auth and secret engines only, because a plugin is the thing a security review of an air-gapped site tends to stop on.

The key never leaves Vault. Polaris moves permission rather than key material, so a node that fails appraisal loses the ability to ask Vault to decrypt, rather than losing a copy it could have kept. The Policy Manager never holds the key either.

What Polaris writes into your Vault​

At deploy time, with the provisioning token described below, all of it namespaced to the deployment:

  • a certificate auth mount at auth/polaris-<deployment id>, with two trusted-certificate entries: nodes, which your nodes log in with, and workloads, which a node's proxy logs in with using the workload certificate to read secrets. Each starts on a placeholder CA, which the Policy Manager replaces with its own
  • a transit key, polaris-workload unless you name another
  • a policy for your nodes, polaris-<deployment id>-transit, allowing encrypt and decrypt on that one key and nothing else, not even reading the key
  • a policy for the workloads entry, polaris-<deployment id>-secrets, allowing read on <kv_mount>/data/<kv_prefix>/* and revoking its own token. No list and no metadata. kv_mount and kv_prefix come from cloud.onprem.vault and default to secret and polaris-<deployment id>. Polaris does not create that KV engine: you do, and you write the secrets into it. The grant covers every workload of the deployment, and KV v2 serves old versions to the same read, so set max_versions=1 on the prefix or destroy old versions when you rotate a secret
  • a policy for the Policy Manager covering seven paths: update on the nodes and workloads entries limited to the certificate field, plus reading them, update on the mount's CRL, looking up a token by accessor, revoking one, revoking leases under its own mount, and renewing its own token. The lookup is there so the Policy Manager can check a token belongs to this deployment before revoking it, since Vault offers no way to scope a revoke by accessor to one mount
  • one periodic token for the Policy Manager, carrying that policy

scripts/vault-provision.sh is the whole of it and is readable on its own, so you can review exactly what will be written before you run a deploy, or undo it afterwards by hand. Re-running a deploy re-applies the same objects rather than creating more.

The provisioning token comes from VAULT_TOKEN in your environment. It needs to create mounts and policies, which is far more than Polaris ever holds afterwards, and it is used for that one step and never stored. What Polaris keeps is the periodic token above, which carries only those seven paths.

How the Policy Manager's token reaches it​

The deploy never holds that token. It is minted response-wrapped, so what the deploy writes, and what travels in the launch seed and sits under /etc/polaris/vault-token in the guest, is a single-use wrapping token. On its first start the Policy Manager unwraps it and seals the token on its state disk the same way it seals its issuer keys, to the vTPM or an SNP derived key. After that every copy outside the Policy Manager is spent.

That also makes a copied seed visible. Vault destroys a wrapping token the first time anyone uses it, so if something unwraps the seed before the Policy Manager does, the Policy Manager refuses to start and says so. Treat the token as exposed: revoke it and re-run the deploy.

The wrapping token is good for 24 hours, which has to cover the bake and the first launch. Set VAULT_PM_TOKEN_WRAP_TTL (a Vault duration such as 72h) if yours take longer. The seed records when it expires, so a Policy Manager launched too late says the seed expired rather than that it was stolen. What the deploy keeps is the token's accessor, in vault-token.accessor beside the seed, so the next deploy can revoke it.

The token is not a child of the VAULT_TOKEN you deploy with. Vault revokes a token's children with it, so a child would die when your login expires or when you revoke the root token after setup. Creating it this way needs root or sudo, which the deploy needs anyway to create a periodic token, and you are free to revoke the provisioning token as soon as the deploy finishes.

The token is periodic with a thirty day period, and the Policy Manager renews it at a third of each lease, so it does not run out while the Policy Manager runs. Its status shows vault_pm_token: how the token is kept, when it expires, and the last renewal error. If renewals fail for thirty days, for example because Vault was unreachable that long, the token lapses and the Policy Manager can no longer revoke a failed node's token or publish the CRL.

A deployment made before the token was wrapped has a plain token in all those places, and a policy without renew-self. It keeps working, with a warning, until the token's period runs out. Redeploying fixes both.

Getting a fresh token takes all three commands, in this order:

./scripts/deploy.sh destroy --cloud onprem --config onprem.yaml
./scripts/deploy.sh deploy --cloud onprem --config onprem.yaml \
--image-tag <tag> --workload-dir ./workload
./scripts/deploy.sh launch --cloud onprem --config onprem.yaml

deploy mints the new token and revokes the previous one the moment it runs, and the guests only pick the new one up when they are relaunched. launch refuses to start while the old guests are up. So the destroy has to come first, and the deployment is down from there until the launch finishes. Worth planning for before the thirty days rather than after.

Set server_ca to the PEM for your Vault's own serving certificate. On a cloud you could lean on the public trust store, and here you usually cannot, because the certificate is issued by your own CA. Leaving it out means the provisioning token goes to whatever answers on that address.

Give address as an IPv4 address, not a hostname, for example https://10.20.0.5:8200. The workload CVM connects to Vault itself for every key operation, so the generated L3 allowlist names Vault, and that rule has to be an address. A deploy with a hostname there stops and says so. Vault's serving certificate then needs an IP SAN for that address, or the TLS check fails at the first login.

Both guests need a route to that address on that port: the Policy Manager, which revokes and publishes the CRL, and the workload CVM, whose proxy logs in and calls Transit. Opening it for the Policy Manager alone leaves the workload unable to decrypt, with every key operation answering 503.

How a node proves it should get the key​

The Policy Manager issues a node a certificate each time it passes appraisal, signed by its own node-ca, over the key the node already attested with. The node presents that certificate to Vault over mTLS, and Vault accepts it because the certificate entry pins node-ca. There are no shared secrets, no AppRole, and no bearer token for a node to leak.

The entry starts with a placeholder CA that trusts nothing, because node-ca does not exist until the Policy Manager first starts. It replaces the placeholder itself on that first start, and again whenever its issuer keys rotate. If you inspect the entry before the Policy Manager has ever run, a certificate you do not recognise is expected.

A re-deploy that adds an entry to a running deployment, such as workloads when you upgrade to secret references, restarts nothing. The Policy Manager re-checks its entries every ten minutes and publishes into the new one then. Until it does, a proxy cannot log in there and secret reads fail. Restart the Policy Manager to publish at once.

How a workload uses the key​

The workload never talks to Vault and never holds a Vault credential. It asks the Polaris proxy in its own CVM, and the proxy calls Transit under the node's login. When the Policy Manager stops passing the node, the next call fails.

Two routes, on the proxy's internal listener:

POST http://polaris-proxy:3000/polaris-container/key/encrypt
POST http://polaris-proxy:3000/polaris-container/key/decrypt

The bodies are Transit's. Plaintext goes in as standard base64, and the ciphertext comes back exactly as Vault writes it:

curl -s -X POST http://polaris-proxy:3000/polaris-container/key/encrypt \
-H 'Content-Type: application/json' -d '{"plaintext":"aGVsbG8="}'
# {"ciphertext":"vault:v1:..."}

curl -s -X POST http://polaris-proxy:3000/polaris-container/key/decrypt \
-H 'Content-Type: application/json' -d '{"ciphertext":"vault:v1:..."}'
# {"plaintext":"aGVsbG8="}

Only the workload's own network reaches them. A request from the guest itself, through the proxy's published port or from anywhere else, gets 403. They are never on 443.

What each answer means:

StatusMeaningWhat to do
200Done.
400The request was malformed, or Vault could not use the input, such as a ciphertext that is not one of its own.Fix the input. Retrying will not help.
403The request did not come from the workload network.Call from the workload container.
413The body is over 1 MiB.Split it, or encrypt a data key and use that for the bulk.
503No key operation right now: the node has no passing verdict yet, it has no node certificate yet, or Vault refused or could not be reached. Comes with Retry-After: 30.Wait and retry. At startup this is normal until the first appraisal passes. Later it means the node lost its grant or Vault is down.

The error body says which case it is, and never repeats what you sent.

Call per operation if a revoke has to end decryption. A data key the workload decrypts once and keeps in memory outlives the revoke, as it would with envelope encryption on any key service. The proxy caches no results, so every call is a fresh Transit request under a token that is at most a minute old. Transit ciphertext opens only in this Vault, so moving the deployment to another platform means re-encrypting. The workload contract covers the rest of what a workload gets, including secrets from Vault KV by reference from Polaris Identity 1.0.

How long a withdrawal takes​

When the Policy Manager decides a node should lose access, it revokes that node's Vault token straight away and publishes a CRL so the node cannot log in again. That is immediate.

The number to plan around is the case where the Policy Manager cannot reach Vault at all, so it can revoke nothing. Then access ends when the node's current certificate and token expire on their own, which is six minutes by default: a five minute node certificate plus a sixty second token. Both are deliberately short for this reason, and both are written by the deploy rather than left to Vault's defaults, which are far longer.

Two behaviours sit behind that number and are worth understanding rather than memorising. A token issued from a certificate login outlives the certificate that bought it, so the token lifetime is the window a withdrawn credential keeps working. And revoking a certificate stops the next login without touching tokens already issued. Polaris does both actions for that reason, and the six minutes is what remains when it can do neither.

Unsealing, and what happens after a power cut​

Unsealing your Vault is yours and never ours. Use Shamir shares distributed to holders you choose, or PKCS#11 auto-unseal where you already run an HSM. Polaris never holds an unseal share, because a Polaris component that could unseal the key manager that gates Polaris keys would make the gate decorative.

The consequence is worth planning for before you meet it. After a site-wide restart, Vault comes back sealed unless you have configured auto-unseal, and nothing Polaris runs can change that. Until enough keyholders are available, your guests attest normally and produce verdicts normally, and no workload can decrypt anything. If that is not an acceptable recovery story for the site, PKCS#11 auto-unseal against an HSM you already run is the way out, and it is a Vault configuration rather than a Polaris one.

The Policy Manager's learned allowlists and enforcement dials do come back after a restart, because it keeps them on its state disk (pm-state.qcow2 in the artifact directory), which a relaunch reuses. They return to the baseline file's advisory values, with the learners starting over, only if that disk was lost or replaced, or a deploy changed the baseline file. Check the dials with promote --dry-run --mode advise, which prints them and sends nothing, before assuming a recovered site is back in the posture it was in.

Air-gapped deployments​

Everything Polaris would otherwise fetch has a local alternative: the guest firmware package, the guest base image, the guest's own packages, the container images, and AMD's certificate and revocation endpoints, which are replaced by certificates supplied locally. Your own NTP server covers time. The generated on-premises baseline contains no cloud metadata destinations and no cloud identity fields.

Keep local copies of both pinned upstream files, the firmware source package (cloud.onprem.ovmf_srpm) and the guest base image. Each is verified by digest, so a substituted file is refused rather than used, but a file that has been removed upstream cannot be fetched at all, and distributions do prune superseded versions. If you may need to reproduce a build months later, copy both into your own storage now and point the configuration at them. A build host with no egress needs this regardless.

Also keep a local cache of the guest's packages. Without one the build resolves them against the distribution archive, which needs egress and is not reproducible, so two builds weeks apart will produce different guest images.

Keeping the firmware current​

Polaris builds the guest firmware from your distribution's EDK2 source package with one patch, scripts/onprem-ovmf/0001-measured-direct-boot-no-unverified-blobs.patch. Upstream EDK2 checks the measured kernel, initrd and command line, but it also runs other boot files the host adds without checking them, and falls back to the host's disks when the kernel does not start. Either would let the host run its own code under the Policy Manager's measurement. The patch makes the firmware stop instead, and only when the launch measured a kernel, so the workload guest's disk boot is unchanged.

The build runs in a Rocky Linux container pinned by digest, with its compiler and assembler pinned by version, from the source package pinned by digest, and the result is checked against a pinned firmware digest. Anyone can repeat it with scripts/onprem-ovmf/build-in-container.sh and get the same bytes. The digest is the contract, because both launch measurements are computed for those bytes, so a mismatch is refused rather than shipped. It needs Docker and an x86_64 container. A build host that cannot run it can use a firmware built elsewhere with cloud.onprem.ovmf_fd, which is accepted only if it has the pinned digest.

When EDK2 has a vulnerability, the path is that your distribution ships a patched source package, Polaris updates the pinned version and digests, and you rebuild. The firmware bytes change, so both launch measurements change and so does the Policy Manager's measurement pin, which means a new set of artifacts rather than a package update inside a running guest.

Notes​

There is no SSH into the guests. They are measured and carry no authorized keys, so the serial consoles are the way in, written to serial/ inside the artifact directory and readable as root. virsh console <domain> works too.

If the guests boot, load their images, start every container and then fail attestation, check the platform setting. Bare-metal SEV-SNP has its own value, and a cloud platform value on bare metal produces evidence claiming to come from that cloud.

If every guest fails the launch measurement check, the artifacts were built for different firmware, a different vCPU count or a different host CPU. All three are covered by the measurement.

If the workload's proxy refuses the Policy Manager over a measurement mismatch, the Policy Manager measurement pin in the artifacts is from an earlier build. A pin is only valid for the firmware, guest sizing, kernel, initrd, command line and root filesystem it was computed from. The launch checks the kernel, initrd, command line and root hash tree against what the pin was computed for and refuses a mismatch.

If the Policy Manager stops early in boot with a dm-verity or polaris: message on its serial console, its root filesystem or hash tree does not match the root hash on its measured command line. It halts rather than opening a shell, because the serial console belongs to the host. Rebake the artifact set.

If the Policy Manager guest boots but the policy manager does not start, and its console says an image "is not loaded" or "is not the image the bake staged", its images did not load from the root filesystem. They load into memory, so the usual cause is a guest with too little of it.