Kubernetes how-to guides
These guides cover running Polaris in a GKE cluster with confidential nodes, from installing the chart through onboarding workloads, enforcing, monitoring, upgrades and removal. The Kubernetes reference has the values keys, objects and permissions they mention, and Polaris on Kubernetes explains why it works this way.
The commands assume the release is called polaris, installed into polaris-system, from a checkout of the Polaris deployment tooling, with the chart at charts/polaris.
Install on GKE with confidential nodes
Your workload fits when it follows the workload contract: a container that serves plain HTTP on one port. The proxy in front of it handles TLS.
1. Create the cluster
Create a GKE Standard cluster whose Polaris pool has SEV-SNP confidential nodes on Container-Optimized OS, Workload Identity, Dataplane V2, and the pool label:
gcloud container clusters create <cluster> \
--project <project> --zone <zone> \
--release-channel regular \
--num-nodes 1 --machine-type n2d-standard-4 \
--enable-confidential-nodes --confidential-node-type sev_snp \
--image-type COS_CONTAINERD \
--workload-pool <project>.svc.id.goog \
--workload-metadata GKE_METADATA \
--enable-ip-alias \
--enable-dataplane-v2 \
--node-labels polaris.fr0ntierx.com/pool=<pool name>
gcloud container clusters get-credentials <cluster> --project <project> --zone <zone>
Keep one machine type and one node image across the Polaris pool. Plan one node for each protected pod, plus room for the Policy Manager. If you taint the pool, use the key polaris.fr0ntierx.com/pool with the effect NoSchedule, which the GKE values file already tolerates.
On a private cluster with the webhook on, add a firewall rule that lets the control plane reach the nodes on port 8443.
2. Create the key and the identities
Create the Cloud KMS key the Policy Manager gates, a Google service account for the Policy Manager and one for each workload:
gcloud kms keyrings create <ring> --project <project> --location <region>
gcloud kms keys create <key> --project <project> --location <region> \
--keyring <ring> --purpose encryption
gcloud iam service-accounts create <pm-gsa> --project <project>
gcloud iam service-accounts create <workload-gsa> --project <project>
Give the Policy Manager's account a role on the key that carries cloudkms.cryptoKeys.getIamPolicy and cloudkms.cryptoKeys.setIamPolicy. The VM deployments use a custom role with exactly those two permissions. roles/cloudkms.admin carries them too, along with much more.
gcloud kms keys add-iam-policy-binding <key> --project <project> --location <region> \
--keyring <ring> --role <role> \
--member serviceAccount:<pm-gsa>@<project>.iam.gserviceaccount.com
Give the workload's account no role on the key. The Policy Manager adds and removes its roles.
Bind each Kubernetes service account to its Google service account. The chart names the Policy Manager's polaris-policy-manager, and each workload's after its workloads entry:
gcloud iam service-accounts add-iam-policy-binding <pm-gsa>@<project>.iam.gserviceaccount.com \
--project <project> --role roles/iam.workloadIdentityUser \
--member "serviceAccount:<project>.svc.id.goog[polaris-system/polaris-policy-manager]"
gcloud iam service-accounts add-iam-policy-binding <workload-gsa>@<project>.iam.gserviceaccount.com \
--project <project> --role roles/iam.workloadIdentityUser \
--member "serviceAccount:<project>.svc.id.goog[polaris-system/<workload name>]"
For the Policy Manager's own keys, see Protect the Policy Manager's issuer keys with Cloud KMS. Do that before the first install if you can, since moving later takes two upgrades.
3. Make the images pullable
The Polaris images on ghcr.io are private, and GKE nodes pull from Artifact Registry with the node's own service account. Copy each Polaris image into an Artifact Registry repository in your project, by digest:
docker buildx imagetools create \
--tag <region>-docker.pkg.dev/<project>/<repository>/polaris-proxy:<version> \
ghcr.io/fr0ntierx/polaris-proxy@<digest>
Do the same for polaris-policy-manager, polaris-sentinel and, if you use the webhook, polaris-webhook. The digests are the defaults under images in the chart's values.yaml, and the versions are in manifest/known-good-images.yaml. Point the chart at your copies, keeping the digests:
images:
proxy:
repository: <region>-docker.pkg.dev/<project>/<repository>/polaris-proxy
digest: <same digest as the chart's default>
Instead of copying, you can give the nodes a pull Secret for ghcr.io in imagePullSecrets.
helm uninstall runs a Job that removes the Policy Manager's bindings on your key, as in Uninstall. It runs Google's public google-cloud-cli image from gcr.io/google.com/cloudsdktool, pinned by digest in policyManager.releaseGrantsOnUninstall.image. If your nodes cannot pull from gcr.io, copy that image as well and point the key at your copy.
Your workload image can sit in Artifact Registry in the same way. Pin it by digest. The chart refuses a tag.
4. Create the namespace and the tokens
kubectl create namespace polaris-system
kubectl -n polaris-system create secret generic polaris-tokens \
--from-literal=management=$(openssl rand -hex 32) \
--from-literal=readonly=$(openssl rand -hex 32) \
--from-literal=workload=$(openssl rand -hex 32) \
--from-literal=sentinel=$(openssl rand -hex 32)
GKE sets no default Pod Security level, so the namespace needs no label. If your cluster enforces a stricter default, label it privileged, as in the chart README.
5. Read the cluster's ranges
The chart's baseline names your cluster's address ranges:
gcloud container clusters describe <cluster> --location <location> \
--format='value(clusterIpv4Cidr,servicesIpv4Cidr,privateClusterConfig.privateEndpoint)'
gcloud compute networks subnets describe <subnet> --region <region> \
--format='value(ipCidrRange)'
The first prints the pod range, the service range and the control plane's address, and the second the node range. Then pick a free address in the service range for the Policy Manager's Service. It cannot change once the Service exists.
6. Choose a trust domain and deployment ID
The trust domain and deployment ID name the Policy Manager's keys and node registry, and neither may change later. Derive the ID from the trust domain, which gives the same ID wherever you run it:
python3 -c 'import uuid; print(uuid.uuid5(uuid.NAMESPACE_DNS, "polaris.acme.prod-eu"))'
Then make the Policy Manager's identity key, which every sidecar pins, with that ID and the Policy Manager's Google service account from step 2:
./scripts/gke-pm-identity-key.sh -p <project> -l <region> -r <identity ring> \
-s <pm-gsa>@<project>.iam.gserviceaccount.com \
-d <the UUID> > pm-identity.yaml
It creates a key ring and a Cloud KMS key for this alone, wraps a new key that only <pm-gsa> can open, sets that key's IAM policy to decrypt for <pm-gsa> and nothing else, and prints the chart values. You need encrypt on the key from the key ring or the project, and permission to set its IAM policy. Keep pm-identity.yaml: running the script again makes a new key, and every sidecar then has to be recreated. See How the sidecars recognise the Policy Manager.
7. Write your values file
tokens:
existingSecret: polaris-tokens
policyManager:
trustDomain: polaris.acme.prod-eu
deploymentId: <the UUID from step 6>
baseline:
cluster:
podRange: <pod range>
serviceRange: <service range>
nodeRange: <node range>
controlPlaneAddress: <control plane address>
service:
clusterIP: <free address in the service range>
serviceAccount:
annotations:
iam.gke.io/gcp-service-account: <pm-gsa>@<project>.iam.gserviceaccount.com
kms:
PM_KMS_PROJECT_ID: <project>
PM_KMS_LOCATION: <region>
PM_KMS_KEY_RING: <ring>
PM_KMS_KEY_ID: <key>
workloads:
- name: <workload name>
image:
repository: <region>-docker.pkg.dev/<project>/<repository>/<image>
digest: sha256:<digest>
port: 8080
writableDirs:
- path: /tmp
sizeLimit: 64Mi
serviceAccount:
annotations:
iam.gke.io/gcp-service-account: <workload-gsa>@<project>.iam.gserviceaccount.com
cloudIdentity: <workload-gsa>@<project>.iam.gserviceaccount.com
networkPolicy:
# Cloud KMS for the gated key. See the note below.
egressTo:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, <control plane endpoint>/32]
ports:
- protocol: TCP
port: 443
Add the images block from step 3. A few points about the workload entry:
- The root filesystem is read-only, so list every path your image writes to under
writableDirs. An entry replaces the default list. - All capabilities are dropped. List any your container needs under
capabilities, without theCAP_prefix. - Port 3000 is the proxy's own. Move your workload off it.
- Pass settings as a Secret named in
envFromSecret. Anyone who can read Secrets in the namespace can read them. - Protected pods can reach only DNS, the metadata server, the Policy Manager and the Sentinel, unless
networkPolicy.egressToadds more. Your workload calls Cloud KMS to use the gated key, and on the cluster from step 1cloudkms.googleapis.comresolves to public Google addresses that change, so the rule above allows 443 to any public address.<control plane endpoint>is the addressgcloud container clusters describe <cluster> --location <location> --format='value(endpoint)'prints. Leave it out ofexceptand the pod, which mounts a service account token, can connect to the Kubernetes API on 443; on GKE Dataplane V2 the entry is what blocks it. After a control plane IP rotation, update the entry. With GKE's DNS-based control plane endpoint on, the API is also reachable on public Google addresses that this rule allows. Add a rule for anything else your workload calls. To narrow the KMS rule to199.36.153.4/30, add a private DNS zone that resolves*.googleapis.comtorestricted.googleapis.com, as Google describes in Configure DNS for Private Google Access; see also Egress from protected pods.
8. Install
helm install polaris charts/polaris -n polaris-system \
-f charts/polaris/values/gke-sev-snp.yaml -f my-values.yaml -f pm-identity.yaml --wait
Without pm-identity.yaml the render stops and names the script. To install without the identity key, set policyManager.identityKey.enabled: false, and the sidecars then recognise the Policy Manager by its measurement alone.
Before anything else, a hook Job checks that the cluster enforces NetworkPolicy, and the install stops if it does not.
The workload pod becomes Ready only once its proxy has its first passing verdict, so --wait returns after that.
kubectl -n polaris-system get pods -o wide
Expect the Policy Manager, one Sentinel and one AppArmor loader per Polaris node, a device plugin per node, and your workload with 2/2 containers Ready.
9. Pin the Policy Manager and let it learn
Pin the Policy Manager's measurement, then drive traffic while Polaris learns. Once learning has finished, move a layer to enforce.
Pin the Policy Manager's measurement
Without a pin, each sidecar trusts the first SEV-SNP Policy Manager that answers. With one, it refuses a Policy Manager with any other launch measurement or VMPL. The measurement covers only the node's firmware and vCPU count, so the pin matches any SEV-SNP guest of that shape on that firmware, not only your Policy Manager. See How the sidecars recognise the Policy Manager.
-
Install without
proxy.policyManagerPin, as above. -
Read the running Policy Manager's measurement as values:
./scripts/pm-snp-pin.sh > pm-pin.yamlIt takes
-n <namespace>and-s <service>for another release. It reads the report through a port-forward and does not check AMD's signature itself. The sidecars do that, and a wrong pin makes them refuse the Policy Manager rather than accept another. -
Upgrade with it:
helm upgrade polaris charts/polaris -n polaris-system --reuse-values -f pm-pin.yaml
The chart's workloads roll with the upgrade. Pods the webhook injected keep the sidecar they were created with, so restart them as well.
The measurement covers the node's firmware and its vCPU count. If Google changes the node firmware, or the Policy Manager lands on a node of another machine type, the sidecars refuse it with PM measurement mismatch. Find out why it moved before you run the script again.
Drive traffic while Polaris learns
Polaris learns what it sees, so an idle workload teaches an idle profile. Turn on the soak CronJob while it learns:
soak:
enabled: true
It calls every workloads entry through its Service once a minute. A run fails when a workload does not answer, and the last one stays behind:
kubectl -n polaris-system logs -l app.kubernetes.io/component=soak
While it is on, any pod in the release namespace with the soak pod's labels gets through to protected pods on 443, whatever networkPolicy.ingressFrom says. Turn it off when learning has finished. Pods the webhook protects get no soak traffic, so drive them yourself.
Protect a pod with the webhook
Use the webhook when the workload comes from your own manifests or chart.
-
Turn it on:
webhook:enabled: truenetworkPolicy:protectedNamespaces:- appsThe namespace must exist before you install or upgrade, and it cannot be the release namespace. The webhook needs its image too, as in step 3 of the install.
-
Label the namespace:
kubectl label namespace apps polaris.fr0ntierx.com/webhook=enabledThe webhook acts on the label alone, and the NetworkPolicy goes only to namespaces in
protectedNamespaces. A labelled namespace you did not list has protected pods with no policy, open on every port they listen on, so keep the two in step. The chart also puts the Policy Manager identity key's public half, the ConfigMappolaris-pm-identity, into each listed namespace, and a protected pod there cannot start without it. -
Copy the tokens Secret into the namespace under the same name. Secrets do not cross namespaces, and the sidecar reads its tokens from it:
kubectl -n polaris-system get secret polaris-tokens -o json \| jq 'del(.metadata.namespace,.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.managedFields)' \| kubectl -n apps apply -f - -
Give the pod its label, its identity and one declared port:
apiVersion: apps/v1kind: Deploymentmetadata:name: my-appnamespace: appsspec:strategy:rollingUpdate:maxSurge: 0maxUnavailable: 1template:metadata:labels:polaris.fr0ntierx.com/protected: "true"annotations:polaris.fr0ntierx.com/cloud-identity: <workload-gsa>@<project>.iam.gserviceaccount.comspec:serviceAccountName: my-appcontainers:- name: client-workloadimage: <image>@sha256:<digest>ports:- containerPort: 8080- The pod must declare exactly one
containerPort, across all its containers. A second listener, such as a metrics exporter, goes in a pod of its own. - For a Deployment, set
strategy.rollingUpdatetomaxSurge: 0andmaxUnavailable: 1, as above, or useRecreate. For a DaemonSet, leaveupdateStrategy.rollingUpdate.maxSurgeat 0. Each Polaris node takes one protected pod, so a surge pod staysPendingwhen every node is taken. The chart's admission policy refuses a Deployment or DaemonSet with protected pods that surges. - Bind the pod's service account to its Google service account through Workload Identity, as in step 2 of the install, with the namespace and name of your service account.
- Add that service account to
admissionPolicy.gatedServiceAccountsas<namespace>/<name>, for exampleapps/my-app, and upgrade the release. The chart's admission policy then refuses an unlabelled pod that runs as it, which could otherwise use the key. Run nothing else as that account. - Name the container
client-workload, or whatever your baseline names.
- The pod must declare exactly one
-
Give the Policy Manager a baseline that names the pod. The chart's
gke-cosbaseline lists only theworkloadsentries, so a pod the webhook protects needs a baseline you write, with its image digest under its container name and its Google service account among the allowed ones. Start fromtemplates/policy-manager/_baseline-gke-cos.tpland pass yours withpolicyManager.baseline.contentorpolicyManager.baseline.existingConfigMap.
The webhook adds the proxy sidecar, the protected PriorityClass, the device volumes, the pool's node affinity and the pool toleration. It does not make your container's root filesystem read-only or drop its capabilities, so set those yourself.
If the webhook refuses a pod, the kubectl error names the pod and the reason. With failurePolicy: Fail, a labelled pod is refused while no webhook pod is running, rather than started without its proxy.
Use the gated key from your workload
On GKE your workload calls Cloud KMS itself, as its Google service account. Under Workload Identity, GKE's metadata server gives the pod credentials for that account, so any Google Cloud client library works without a key file. The call succeeds while the Policy Manager holds the account's roles on the key, and fails once it removes them.
As a plain REST call:
TOKEN=$(curl -s -H 'Metadata-Flavor: Google' \
http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token \
| python3 -c 'import sys, json; print(json.load(sys.stdin)["access_token"])')
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"plaintext":"aGVsbG8="}' \
https://cloudkms.googleapis.com/v1/projects/<project>/locations/<region>/keyRings/<ring>/cryptoKeys/<key>:encrypt
- Decrypt each time you need the plaintext. A plaintext your workload keeps in memory outlives a revoke. The workload contract says what else a revoke does not end.
- Every pod on the same Google service account shares one binding. A failure in any of them removes the key from all.
- The pod reaches Cloud KMS only through a
networkPolicy.egressTorule, as in step 7 of the install. Without one the call times out, which looks like a revoke but is not one. - The runtime layer learns the workload's connections to the metadata server and to Cloud KMS. Make those calls while it learns, so they are in the profile when you enforce.
Move a layer to enforce
A layer at alert or advise reports findings and leaves your key alone. A layer at enforce removes the key when it fails.
-
Reach the Policy Manager and read a token:
kubectl -n polaris-system port-forward deploy/polaris-policy-manager 3000:3000 &TOKEN=$(kubectl -n polaris-system get secret polaris-tokens -o jsonpath='{.data.management}' | base64 -d) -
Check that learning has finished. Each learner lists its sessions, and a session reads
learninguntil it converges:curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/v1/profiles/learncurl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/v1/kernel/learncurl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/v1/statusThe status should show
passfor every layer with no alerts. -
Restart each protected pod once, then wait for the new pods' sessions to apply. The runtime layer learns a container's start (your image's entrypoint scripts, and the
/runcthat starts it) only from a start it sees. A profile learned from pods that were already running passes step 2, and then the first restart under enforce fails the runtime layer for about 30 seconds and removes the key. A restarted pod reports as a new CVM with sessions of its own, so it shows up again in the lists from step 2:kubectl -n <namespace> rollout restart deploy/<name>kubectl -n <namespace> rollout status deploy/<name>curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/v1/profiles/learnGo on once those sessions have converged and the status shows
passagain. Do this before you set any layer to enforce, not onlyl3: withl2orl31at enforce, the Policy Manager raisesl3itself once its learners apply and it has been clean forpolicyManager.promotionWindowSeconds.The new pods' alerts during those 30 seconds start
l3's clean time again, so step 4 refusesl3for anotherpromotionWindowSeconds, a day by default. Restart early, while the Policy Manager is still learning, rather than after step 2. -
Read the current settings, then change them:
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/v1/enforcementcurl -s -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \-d '{"l2":"enforce","l3":"enforce"}' http://127.0.0.1:3000/api/v1/enforcementThe Policy Manager answers
{"status":"updated",...}. It answers409with"status":"refused"and a reason for each layer when a layer has not yet been clean under advise forpolicyManager.promotionWindowSeconds, a day by default. Wait and try again.
The layers are l2 (image), l3 (runtime), l31 (kernel) and ima_boot, and the runtime layer has three parts you can set on their own, l3_syscalls, l3_binaries and l3_network. On GKE ima_boot is off, and the kernel layer starts empty, so l31 changes nothing until the kernel learner has converged. To keep network checks advisory while the rest of the runtime layer enforces:
curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"l3":"enforce","l3_network":"advise"}' http://127.0.0.1:3000/api/v1/enforcement
To move a layer back, set it to advise.
The setting survives a Policy Manager restart. It does not survive a change to the baseline the chart renders, which returns every layer to the baseline's setting and starts learning again. See Upgrade Polaris.
Under l3 enforce, kubectl exec or kubectl debug into a protected pod counts as unknown behavior and removes the key.
The shipped host profile allows the connection a kubectl port-forward makes only into pods that run no Polaris proxy. The Policy Manager's pod runs one, so a port-forward to it can raise a runtime alert on that node. Keep port-forwards short once layers enforce.
Monitor a deployment
From Kubernetes. A protected pod is Ready only while its proxy's gate is open, so kubectl get pods shows which pods are serving, and a Service routes only to those.
From the proxy. Each protected pod's proxy serves status routes on 443 that stay reachable while it holds traffic:
| Route | Returns |
|---|---|
/polaris-container/health | The proxy is running |
/polaris-container/ready | 200 while the gate is open, 503 while it is closed |
/polaris-container/attestation-status | The latest verdict. With the management token it adds the reason and the alerts |
From the Policy Manager. Through a port-forward as above, with the read-only token:
TOKEN=$(kubectl -n polaris-system get secret polaris-tokens -o jsonpath='{.data.readonly}' | base64 -d)
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/v1/status
Each node appears with its cvm_id, its verdict and each layer's status. Alert when a node's last_seen is more than a few report intervals old, since a quiet node keeps reading live and pass until the dead man's switch fires at 300 seconds, although its grant lapses sooner.
Logs.
kubectl -n polaris-system logs deploy/polaris-policy-manager -c polaris-policy-manager
kubectl -n polaris-system logs deploy/<workload name> -c polaris-proxy
kubectl -n polaris-system logs ds/polaris-sentinel
Nothing revokes while the Policy Manager is down, so alert on its pod not being Ready.
Diagnose a withdrawal of access
When a protected pod drops out of its Service, the Policy Manager's records say why.
-
Port-forward to the Policy Manager and read the read-only token, as in Monitor a deployment.
-
Find the node's
cvm_idin/api/v1/status. The proxy makes a new key each time it starts, so look the ID up fresh. -
Read the latest incident:
curl -s -H "Authorization: Bearer $TOKEN" \http://127.0.0.1:3000/api/v1/incidents/<cvm_id>/reportIt records what opened it, each layer's status, the reason, each alert and the grant and revoke events.
-
For more,
/api/v1/checks/<cvm_id>lists the latest checks and/api/v1/history/<cvm_id>the past verdicts. -
Check whether the failure locked the Google service account:
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/v1/kms/locksA failure locks the account for every pod that runs as it, and the lock lifts when the pod that failed passes again. If that pod is gone, clear the lock with the management token once you know the cause:
curl -s -X DELETE -H "Authorization: Bearer <management token>" \http://127.0.0.1:3000/api/v1/kms/locks/<account>
A node that can no longer attest
On AMD nodes the guest kernel turns off its attestation key, VMPCK0, for the rest of the boot if a request to the security processor times out. From then on every SEV-SNP report on that node fails, each protected pod there closes its gate after five reports, and a Policy Manager there cannot start its attested front.
The Sentinel checks for this once at start and once a minute. Its pod on that node stays 0/1, and its log and /ready say why:
kubectl -n polaris-system get pods -l app.kubernetes.io/component=sentinel -o wide
kubectl -n polaris-system exec <sentinel-pod> -- wget -qO- http://127.0.0.1:54323/ready
The proxies on the node log the guest kernel disabled its attestation key (VMPCK0); SNP reports fail until the node boots again, so recreate the node. Confirm it in the kernel log before you touch the node:
kubectl debug node/<node> -n default --profile=sysadmin --image=busybox -- dmesg
kubectl -n default logs <node-debugger-pod> | grep VMPCK
kubectl -n default delete pod <node-debugger-pod>
A node in this state shows Disabling VMPCK0 communication key to prevent IV reuse. Only a new boot brings the key back. Cordon and drain the node, then delete its VM, and the node pool creates a new one. Nothing cordons the node for you.
Protect the Policy Manager's issuer keys with Cloud KMS
By default the Policy Manager keeps its issuer keys on its volume, guarded by the disk's encryption and by who can read the claim, and logs that this is not for production. Under kms, a Cloud KMS key of its own wraps them, so they open on any node.
-
Create a second key for this, not the gated key, and give the Policy Manager's Google service account
roles/cloudkms.cryptoKeyEncrypterDecrypteron it:gcloud kms keys create <wrap-key> --project <project> --location <region> \--keyring <ring> --purpose encryptiongcloud kms keys add-iam-policy-binding <wrap-key> --project <project> --location <region> \--keyring <ring> --role roles/cloudkms.cryptoKeyEncrypterDecrypter \--member serviceAccount:<pm-gsa>@<project>.iam.gserviceaccount.com -
Set it in your values:
policyManager:issuerKeyProtection: kmsissuerKeyKmsKey: projects/<project>/locations/<region>/keyRings/<ring>/cryptoKeys/<wrap-key>On a Policy Manager that already runs with
file, also setissuerKeyRewrapFrom: filefor this one upgrade, which keeps the same keys. Upgrade again without it afterwards. While it stays set, the Policy Manager warns on each start. -
Once the Policy Manager logs
Issuer key set loaded, reduce its role on the wrap key toroles/cloudkms.cryptoKeyDecrypter. It needs encrypt again only to make a new key set.
Without decrypt, the Policy Manager refuses to start with issuer_key_unwrap_failed: kms_error.
Upgrade Polaris
A new version of the deployment tooling brings a new chart, whose image digests match its manifest/known-good-images.yaml. Copy the new images as in step 3 of the install, then:
helm upgrade polaris charts/polaris -n polaris-system \
-f charts/polaris/values/gke-sev-snp.yaml -f my-values.yaml -f pm-identity.yaml
Plan for these:
-
The Policy Manager identity key is on by default. An install from before it has no
pm-identity.yaml, and the upgrade stops and names the script. Make the key as in step 6 of the install, or setpolicyManager.identityKey.enabled: false. With the key, the Policy Manager and the chart's workloads roll. Restart the pods the webhook injected, and list their namespaces innetworkPolicy.protectedNamespacesso each gets the public half. -
Learning starts again when the baseline changes. If the new chart's
gke-cosbaseline differs in any byte, or you changed a workload's image, port or identity, the cluster ranges or the Policy Manager's address, the Policy Manager drops what it learned and every layer goes back toalert. Let it learn and move layers to enforce again. -
The Policy Manager is down while it restarts. Its Deployment uses
Recreate, so the old pod lets go of the volume before the new one starts. If it is gone for more than five report intervals, about two and a half minutes, every protected pod closes its gate until it is back. A slow volume detach, or a disk in another zone than the free nodes, can take that long. Use a regional storage class if the Policy Manager's pool spans zones. -
After a restart the Policy Manager holds no grants. Each workload gets its key back on its next passing report.
-
Protected pods lose all egress the chart does not list. From the version that adds the egress policy, a protected pod reaches only DNS, the metadata server, the Policy Manager and the Sentinel. Set
networkPolicy.egressTofor Cloud KMS and anything else your workloads call before you upgrade, as in step 7 of the install. -
Pods the webhook injected keep their old sidecar. Restart them after an upgrade that changes the proxy.
-
The pin may move. A new node firmware changes the Policy Manager's measurement. See Pin the Policy Manager's measurement.
-
Never change
trustDomainordeploymentId. The Policy Manager refuses to start on a registry stored under another ID.
When you change a ConfigMap baseline you made yourself, restart the Policy Manager, which reads its baseline only at start:
kubectl -n polaris-system rollout restart deploy/polaris-policy-manager
Uninstall
helm uninstall polaris -n polaris-system
Once the release's other objects are gone, a Job, polaris-release-grants, removes roles/cloudkms.cryptoKeyEncrypterDecrypter and roles/cloudkms.publicKeyViewer on your key from the Google service account of every workloads entry. It runs after the Policy Manager has stopped, so nothing can put a binding back, and it runs as the Policy Manager's Google service account, which may change the key's IAM policy. A binding that is already gone counts as removed. Any other error fails the uninstall's hook, and the Job's log names each binding it could not remove. policyManager.releaseGrantsOnUninstall.enabled turns it on, which is the default, and it runs only on GKE, with policyManager.cloudProvider: gcp and policyManager.kms naming the key.
The Job does not cover every case. Check the key's IAM policy yourself, and remove what is left, after an uninstall where:
- pods were protected by the webhook. The Job knows only the Google service accounts of
workloadsentries. - you ran
helm uninstall --no-hooks, which skips the Job. namespace.createrendered the namespace the release itself runs in. The uninstall deletes that namespace, and the Job cannot start in a namespace being deleted.- the Job failed. A failed Job stays behind, so read why with
kubectl -n polaris-system logs job/polaris-release-grants.
gcloud kms keys get-iam-policy <key> --project <project> --location <region> --keyring <ring>
gcloud kms keys remove-iam-policy-binding <key> --project <project> --location <region> \
--keyring <ring> --role roles/cloudkms.cryptoKeyEncrypterDecrypter \
--member serviceAccount:<workload-gsa>@<project>.iam.gserviceaccount.com
gcloud kms keys remove-iam-policy-binding <key> --project <project> --location <region> \
--keyring <ring> --role roles/cloudkms.publicKeyViewer \
--member serviceAccount:<workload-gsa>@<project>.iam.gserviceaccount.com
What stays behind:
- The Policy Manager's state claim,
polaris-policy-manager-state, which the chart marks to keep. It holds the issuer keys. Deleting it gives the next Policy Manager new keys, which voids every certificate the old one issued. - The tokens Secret you made, and its copies in namespaces you labelled for the webhook. Remove the namespace label as well.
- Your Cloud KMS keys and Google service accounts. Keep the gated key for as long as your data lives.
Troubleshoot
The render fails naming a missing value. The message names the key, such as policyManager.trustDomain is required while the PM is enabled or policyManager.baseline.cluster.podRange is required with policyManager.baseline.nodeImage gke-cos. Set it. See Values you must set.
The render fails saying no workload sets a cloud identity. The gke-cos baseline takes its allowed service accounts from workloads[].serviceAccount.cloudIdentity. Set it on at least one entry, or pass your own baseline.
helm install fails on an unknown key. The values schema refuses keys it does not know. Check the spelling against the reference.
The install stops at the NetworkPolicy check. The cluster does not enforce NetworkPolicy. Use Dataplane V2 on GKE. The check's resources stay behind:
kubectl -n polaris-system logs job/polaris-network-policy-check
Sentinel or device plugin pods are refused with insufficient quota. GKE admits system-node-critical pods outside kube-system only under a quota. Keep nodeCriticalQuota.create on, raise nodeCriticalQuota.pods for a larger cluster, and recreate the DaemonSet, whose controller backs off.
A Sentinel pod stays 0/1. Read its log and /ready. If it says the attestation key is off, see A node that can no longer attest. Otherwise check that the ClusterRoleBinding polaris-sentinel-pod-reader exists, since the Sentinel is not ready until it has read its node's pods.
A protected pod stays Pending with didn't match pod anti-affinity rules. Every Polaris node already runs a protected pod, and each node takes one. Add a node to the pool, or let the cluster autoscaler add one. If it is the new pod of a rollout, the Deployment surges; set maxSurge: 0 as in Protect a pod with the webhook. See One protected pod per node.
A protected pod is refused with polaris: a pod labelled polaris.fr0ntierx.com/protected=true must carry the polaris-proxy native sidecar. The admission policy refused a protected pod the webhook did not reach. Label its namespace polaris.fr0ntierx.com/webhook=enabled, or drop the protected label from the pod. The policy's other refusals name the missing piece the same way.
A webhook-protected pod sits in CreateContainerConfigError. Its namespace has no copy of the tokens Secret. Copy it, as in Protect a pod with the webhook.
The webhook refuses a pod. The error names the reason: no declared containerPort, more than one, or a port that is the proxy's own.
Labelled pods are refused after a GitOps sync. Without cert-manager or webhook.certificate.existingSecret, a render with no cluster, as Argo CD and Flux do, makes a new webhook certificate every time. Use one of those two.
The proxy's first start fails with PM nonce fetch failed. The Policy Manager was not up yet. The sidecar restarts on its own and passes once it is.
The Policy Manager crash-loops with The policy-manager identity key is configured and could not be opened. Its Google service account cannot decrypt with policyManager.identityKey.kmsKey, or the key was wrapped for another deployment ID. Check the key's IAM policy names roles/cloudkms.cryptoKeyDecrypter for it, and that -d was policyManager.deploymentId. Otherwise run the script again and upgrade with its output.
Sidecars refuse the Policy Manager with the PM identity signature does not verify against the pinned key or presented no identity statement. The Policy Manager that answers does not hold the identity key the sidecar pins: another Policy Manager, an older public half in the sidecar, or one that started without the key. Pods the webhook injected keep the pin they started with, so recreate them after a new key.
A webhook-protected pod sits in ContainerCreating with configmap "polaris-pm-identity" not found. Its namespace is not in networkPolicy.protectedNamespaces, so the chart made no copy of the public half there. Add it and upgrade. The render cannot see which namespaces carry the webhook label, so this shows up only when the next pod there is created, not at the upgrade.
Pods the webhook injected refuse the Policy Manager after helm rollback or policyManager.identityKey.enabled: false. They started with the pin, and the Policy Manager no longer sends the statement. Recreate them. The chart's workloads roll on their own.
The render stops with policyManager.serviceAccount.annotations has no iam.gke.io/gcp-service-account. The Policy Manager opens the identity key as its Google service account through Workload Identity, so its service account needs that annotation, set to the account scripts/gke-pm-identity-key.sh was given with -s.
Sidecars refuse the Policy Manager with PM measurement mismatch. The Policy Manager's measurement moved. See Pin the Policy Manager's measurement.
The Policy Manager crash-loops with issuer_key_unwrap_failed: kms_error. Its Google service account has no decrypt on the wrap key, or no encrypt on a first start. See Protect the Policy Manager's issuer keys with Cloud KMS.
The Policy Manager refuses to start over its deployment ID. deploymentId differs from the one its state was written under. Set it back.
The key works for one workload and not another on the same account. A failure locked the shared Google service account. See step 5 of Diagnose a withdrawal of access.
Layers are back at alert. The baseline the chart renders changed. Let Polaris learn and move them to enforce again.
Every protected pod drops out of its Service at once. The proxies could not reach the Policy Manager for five reports. Check its pod, its volume and its node.