Skip to main content

Kubernetes reference

Facts to look up while installing and running Polaris in a cluster. For why it works the way it does, see Polaris on Kubernetes.

Object names on this page assume the release is called polaris and installed into polaris-system. With another release name, polaris becomes <release>, or <release>-polaris when the release name does not contain polaris.

What Polaris offers on Kubernetes today​

CapabilityIn a cluster
GKE Standard with AMD SEV-SNP confidential nodes on Container-Optimized OSAvailable
A baseline shipped with the chart for those nodesAvailable, as gke-cos
Attestation of image, runtime and kernel, with key access withdrawn when a layer you enforce failsAvailable
Protecting pods from your own manifests through the webhookAvailable
Pinning the Policy Manager's launch measurementAvailable
Wrapping the Policy Manager's issuer keys with Cloud KMSAvailable on GCP
Ending key access while the Policy Manager is downNot available. Bindings stay until it is back
Removing the Policy Manager's bindings on your key at helm uninstallAvailable on GKE, for workloads entries. Pods the webhook protects are not covered
Encrypt and decrypt routes on the proxy for the workloadNot available on GKE. The workload calls Cloud KMS itself, through a networkPolicy.egressTo rule
IMA measurements of what runs on the nodeNot available on GKE's node images
A baseline for GKE Ubuntu nodesNot available. Bring your own
GKE with Intel TDX, AKS, EKS, OKENot yet measured
GKE AutopilotNot supported
A load balancer or Ingress in front of workloads or the Policy ManagerNot created by the chart
Enforcing one protected pod per nodeAvailable. The scheduler keeps a second protected pod off a node through a required anti-affinity, and an admission policy refuses a protected pod without it. Measured on kind, not yet on GKE. See One protected pod per node
RBAC that keeps exec and port-forward out of protected namespacesNot shipped. Write your own
Egress NetworkPoliciesAvailable for protected pods. Measured on GKE Dataplane V2 only; the EKS, AKS and OKE values files leave egress open
Pod Security baseline for protected namespacesNot available. They run at privileged

Cluster requirements​

RequirementWhy
Kubernetes 1.30 or newerThe sidecar's AppArmor field is pruned without an error on 1.29, so the chart refuses older versions
GKE Standard, not AutopilotThe Sentinel needs host PID, host paths and capabilities
A node pool with confidential nodes of type SEV-SNPThe proxy reads its report from the node
Container-Optimized OS with containerd on the Polaris poolThe gke-cos baseline was learned on it
One node image and one machine type across the Polaris poolThe Policy Manager holds one baseline, and a pinned measurement covers firmware and vCPU count
The node label key polaris.fr0ntierx.com/pool on Polaris nodesEvery Polaris component requires it. The value is free
Workload Identity, with the node metadata server set to GKE_METADATAPods act as their Google service accounts
A CNI that enforces NetworkPolicy, such as GKE Dataplane V2The chart refuses to install without it
On a private cluster with the webhook on, a firewall rule from the control plane to the nodes on port 8443The API server calls the webhook there

The chart's GKE values were measured on GKE Standard 1.35.8-gke.1225000, regular channel, with n2d-standard-4 nodes. Secure Boot is off on GKE node pools unless you create the pool with --shielded-secure-boot.

Values you must set​

The render fails without these:

  • The tokens: tokens.existingSecret, or all four of tokens.management, tokens.readonly, tokens.workload and tokens.sentinel.
  • A baseline: policyManager.baseline.content, policyManager.baseline.existingConfigMap or policyManager.baseline.nodeImage.
  • policyManager.trustDomain and policyManager.deploymentId.
  • With policyManager.baseline.nodeImage: gke-cos: the four policyManager.baseline.cluster ranges, policyManager.service.clusterIP, and at least one workloads entry with serviceAccount.cloudIdentity.
  • name, image and port on each workloads entry.
  • policyManager.issuerKeyKmsKey when policyManager.issuerKeyProtection is kms.
  • policyManager.identityKey.kmsKey, wrappedKey and publicKey, which scripts/gke-pm-identity-key.sh prints, unless policyManager.identityKey.enabled is false. With policyManager.enabled: false, only publicKey.
  • With policyManager.enabled: false, proxy.policyManagerUrl.

These render without an error, but the stack does not work without them on GKE:

  • values/gke-sev-snp.yaml, passed before your own values file.
  • policyManager.kms, the key the Policy Manager gates.
  • The Workload Identity annotations on the Policy Manager's and each workload's service account, and serviceAccount.cloudIdentity on each workload.

values.schema.json refuses unknown keys at every level, so a misspelt key fails at helm install rather than being ignored.

Values keys​

Tokens and images​

KeyDefaultWhat it does
tokens.existingSecretemptyA Secret in the release namespace with the keys management, readonly, workload and sentinel. Use this outside throwaway installs
tokens.management, tokens.readonly, tokens.workload, tokens.sentinelemptyThe four tokens as values. They end up in the Helm release record
images.proxy, images.policyManager, images.sentinel, images.sentinelInvary, images.webhookReleased digestsEach is repository and digest. The digest must be sha256: and 64 hex characters. The defaults follow manifest/known-good-images.yaml
imagePullSecrets[]Secrets in the release namespace for registries the node cannot pull from. GKE pulls from Artifact Registry with the node's service account and needs none

Namespace, scheduling and devices​

KeyDefaultWhat it does
namespace.createfalseRenders a Namespace with Pod Security labels. Helm will not adopt a namespace it made with --create-namespace, so use this only for a release in another namespace
namespace.namepolaris-systemThe namespace namespace.create renders
namespace.podSecurity.enforce, .audit, .warnprivilegedThe Pod Security labels on that Namespace
nodeCriticalQuota.createtrueA ResourceQuota that lets system-node-critical pods run outside kube-system, which GKE requires
nodeCriticalQuota.pods30Pods the quota admits, sized for three per node
priorityClasses.createtrueRenders the two PriorityClasses
priorityClasses.policyManager.value900000Priority of the Policy Manager pod
priorityClasses.protected.value800000Priority of protected pods
nodePool.labelKeypolaris.fr0ntierx.com/poolThe node label key every Polaris pod requires
nodePool.taint.key, .effectpolaris.fr0ntierx.com/pool, NoScheduleThe pool taint every protected pod tolerates. An empty key adds no toleration and drops the admission policy's check for one
tee.devicesamd.com/sev-snp and google.com/cc, 1 eachExtended resources every proxy and Sentinel container requests. A values file replaces the whole list
tee.configfstrueMounts /sys/kernel/config from the node, where the proxy asks for SEV-SNP reports
tee.securityfstrueMounts /sys/kernel/security read-only, for the IMA log
devicePlugin.gke.installfalse, true in the GKE fileInstalls google/cc-device-plugin
devicePlugin.gke.image, .resourcesPinned digestThe plugin's image and resources
devicePlugin.gke.nodeSelector{}The plugin runs on every node unless you set this

AppArmor​

KeyDefaultWhat it does
apparmor.profileempty, polaris-proxy in the GKE fileEmpty runs the proxy sidecar Unconfined. A name runs it under that Localhost profile
apparmor.loader.enabledfalse, true in the GKE fileA DaemonSet that loads the chart's profile into each Polaris node with the node's own apparmor_parser
apparmor.loader.externalfalseSet when the profile reaches the nodes some other way. A named profile with neither this nor the loader fails the render
apparmor.loader.image, .resources, .nodeSelector, .tolerationsbusybox, small, empty, tolerate allThe loader's own settings

Proxy sidecar​

KeyDefaultWhat it does
proxy.ratlsPlatformgcp-sev-directWhat the workload sidecars attest as
proxy.pmRatlsPlatformemptyWhat the Policy Manager's proxy attests as and the sidecars pin it by. Empty means ratlsPlatform. With ratls trust it must be gcp-sev, gcp-sev-direct, aws-sev, svsm or snp-direct
proxy.policyManagerTrustratlsratls pins the Policy Manager's TEE identity. caCert is for Azure and Oracle
proxy.policyManagerPin.measurementHexemptyThe Policy Manager's launch measurement, 96 hex characters. Set, every sidecar refuses a Policy Manager with any other. It covers the node's firmware and vCPU count, so it matches any SEV-SNP guest of that shape, not only your Policy Manager
proxy.policyManagerPin.requiredVmpl0The VMPL the Policy Manager must report, 0 to 3
proxy.policyManagerCaCertSecretemptyOnly with caCert trust
proxy.policyManagerUrlemptyWhere sidecars post evidence. Empty means the chart's Policy Manager Service
proxy.reportIntervalSeconds30How often each sidecar reports. Must equal sentinel.reportIntervalSeconds. Under 30 a sidecar restart can revoke the grant for up to about 90 s, whatever policyManager.grantTtlSeconds is. The render does not check this
proxy.enforcementGracePeriodSeconds0Wait before the gate closes on a missing first verdict. The readiness probe already keeps the pod out of its Service until then
proxy.websockettruePasses WebSocket traffic through
proxy.loggingtrueRequest logging
proxy.managementPassthroughfalseServes the Policy Manager's API as /management/* on every workload sidecar and gives each sidecar the read-only and workload tokens. Off by default, since it puts two Policy Manager credentials into every protected pod
proxy.resources.cpu, .memory500m, 512MiApplied as both request and limit
proxy.readinessProbe.periodSeconds, .failureThreshold5, 3The probe on /polaris-container/ready
proxy.extraEnv[]Extra environment entries for every sidecar

Workloads​

workloadDefaults holds the defaults, and each workloads entry may set any of the same keys.

KeyDefaultWhat it does
workloads[].namerequiredName of the Deployment, Service and ServiceAccount. Lowercase letters, digits and hyphens
workloads[].image.repository, .digestrequiredYour image, pinned by digest. A tag is refused
workloads[].portrequiredThe port your container listens on. It cannot be the proxy's internal port
replicas1Replicas of the Deployment
containerNameclient-workloadYour container's name. The baseline's image check is keyed on it
proxyInternalPort3000The sidecar's own loopback port
resources100m and 64Mi requested, 500m and 128Mi limitYour container's resources
capabilities[]Capabilities added back after all are dropped, without the CAP_ prefix
writableDirs/tmp, 64MiPaths your container may write to, each an emptyDir with a sizeLimit. The root filesystem is read-only. An entry replaces the whole list
envFromSecretemptyA Secret in the release namespace loaded as your container's environment
serviceAccount.annotations{}Annotations on the ServiceAccount, such as iam.gke.io/gcp-service-account
serviceAccount.cloudIdentityemptyThe Google service account the Policy Manager grants to. The gke-cos baseline lists it as allowed
nodeSelector, tolerationsempty, a toleration for the pool taint in the GKE fileScheduling on top of the pool label

Soak traffic​

KeyDefaultWhat it does
soak.enabledfalseA CronJob that calls every workloads entry through its Service, so learning sees a working system
soak.scheduleevery minuteCron schedule
soak.image, soak.resourcescurlimages/curl by digest, smallThe CronJob's settings

Webhook​

KeyDefaultWhat it does
webhook.enabledfalseRenders the sidecar webhook
webhook.namespaceLabel.key, .valuepolaris.fr0ntierx.com/webhook, enabledThe label that opts a namespace in
webhook.replicas2Replicas, with a PodDisruptionBudget of one when more than one
webhook.port8443The port the webhook pods listen on
webhook.failurePolicyFailFail refuses a labelled pod while the webhook is down. Ignore admits it without a proxy
webhook.timeoutSeconds101 to 30
webhook.certManager.enabledfalsecert-manager issues and renews the serving certificate
webhook.certManager.issuerRef{}name, kind (Issuer or ClusterIssuer) and group. Empty uses a self-signed Issuer the chart makes
webhook.certificate.validityDays3650Validity of a certificate the chart or cert-manager makes
webhook.certificate.existingSecretemptyA kubernetes.io/tls Secret you made, for GitOps without cert-manager
webhook.certificate.caCertemptyThe PEM CA that signed it. Required with existingSecret
webhook.resources, .nodeSelector, .tolerationssmall, empty, emptyThe webhook's own scheduling. It needs no confidential node

Admission policy​

KeyDefaultWhat it does
admissionPolicy.enabledtrueRenders the ValidatingAdmissionPolicy and binding that check every protected pod
admissionPolicy.validationActions[Deny]Audit or Warn report without refusing, for a cluster with protected pods from before the policy. Deny and Warn together are refused
admissionPolicy.gatedServiceAccounts[]Kubernetes service accounts, as namespace/name, that a pod may run as only when labelled protected, on top of each workloads entry's account. List the accounts of pods the webhook protects. The render fails on another format and on the chart's own accounts in the release namespace, default included

Policy Manager​

KeyDefaultWhat it does
policyManager.enabledtrueRuns the Policy Manager in the cluster
policyManager.logLevelinfotrace to panic
policyManager.cloudProvidergcpThe cloud whose attestation and KMS it uses
policyManager.yangPushtrueThe RESTCONF event streams
policyManager.autolearn.enabledtrueLearning
policyManager.autolearn.minObservationSeconds600The shortest a learning session watches
policyManager.promotionWindowSeconds86400How long a layer must be clean under advise before it may enforce. 0 turns the check off
policyManager.grantTtlSeconds180How long a grant lasts without a passing report. Keep it at least 3 x max(proxy.reportIntervalSeconds, 30) + proxy.reportIntervalSeconds, so a sidecar restart keeps the key while the Policy Manager holds its reports. The render fails when it is not over proxy.reportIntervalSeconds
policyManager.deadmanIntervalSeconds300How long a node may go silent before it counts as gone, and how long the Policy Manager waits after a restart before that check runs
policyManager.oidcIssuerGoogle's confidential computing issuerSet on GCP
policyManager.kms{}The gated key as environment variables. On GKE: PM_KMS_PROJECT_ID, PM_KMS_LOCATION, PM_KMS_KEY_RING and PM_KMS_KEY_ID
policyManager.baseline.contentemptyA baseline file as text, usually with --set-file
policyManager.baseline.existingConfigMapemptyA ConfigMap you made with the key polaris-baseline.yaml
policyManager.baseline.nodeImageempty, gke-cos in the GKE fileThe baseline the chart ships for that node image, used when the two above are empty
policyManager.baseline.cluster.podRange, .serviceRange, .nodeRangeemptyThe cluster's IPv4 ranges, required with gke-cos
policyManager.baseline.cluster.controlPlaneAddressemptyThe control plane's private address, required with gke-cos
policyManager.baseline.restrictedGoogleApisfalseFor a VPC where *.googleapis.com resolves to restricted.googleapis.com. Limits the node's 443 to that range and GKE's. Not yet run on a node
policyManager.trustDomainemptyNames the issuer keys and node registry. Lowercase letters, digits, ., - and _. Never change it
policyManager.deploymentIdemptyA UUID, chosen once. A Policy Manager that finds a registry under another ID refuses to start
policyManager.service.clusterIPemptyA fixed address in the service range for the Policy Manager's Service. Required with gke-cos. It cannot change once the Service exists
policyManager.persistence.size1GiThe state volume
policyManager.persistence.storageClasscluster defaultUse a regional class when the Policy Manager's pool spans zones
policyManager.persistence.existingClaimemptyA claim you made. The chart then makes none
policyManager.issuerKeyProtectionfilekms wraps the issuer keys with Cloud KMS. file keeps them on the volume and logs that it is not for production. vtpm and snp-derived tie the keys to one node, which a pod that moves cannot use
policyManager.issuerKeyKmsKeyemptyThe wrap key, as projects/<p>/locations/<l>/keyRings/<r>/cryptoKeys/<k>. Not the gated key
policyManager.issuerKeyRewrapFromemptySet to file for one upgrade to move existing keys to kms
policyManager.identityKey.enabledtrueThe Policy Manager identity key every sidecar pins. With the Policy Manager in the chart, the render fails until the three keys below are set and policyManager.serviceAccount.annotations has iam.gke.io/gcp-service-account, the account that opens it. false turns it off, whatever else is set. Refused under caCert trust and for a gcp-sev Policy Manager
policyManager.identityKey.kmsKeyemptyThe Cloud KMS key that wrapped it, as projects/<p>/locations/<l>/keyRings/<r>/cryptoKeys/<k>. Its own key, not the gated key or the wrap key
policyManager.identityKey.wrappedKeyemptyBase64 of the Cloud KMS ciphertext, wrapped for policyManager.deploymentId. The chart puts it in a Secret for the Policy Manager
policyManager.identityKey.publicKeyemptyBase64 of the public half, DER. The chart puts it in a ConfigMap in the release namespace and in each of networkPolicy.protectedNamespaces
policyManager.signingKey.existingSecretemptyA Secret with signing-key.pem, a P-256 key that seeds the first appraisal key of a new key set
policyManager.license.existingSecretemptyA Secret with license.jws. A renewal written into it reaches the Policy Manager within a minute
policyManager.serviceAccount.annotations{}On GKE, iam.gke.io/gcp-service-account with the Policy Manager's Google service account
policyManager.resources250m and 256Mi requested, 1 CPU and 512Mi limitThe Policy Manager container
policyManager.proxy.resources250m, 256MiIts proxy, as request and limit
policyManager.nodeSelector, .tolerationsempty, a toleration for the pool taint in the GKE fileScheduling on top of the pool label
policyManager.extraEnv[]Extra environment entries
policyManager.releaseGrantsOnUninstall.enabledtrueOn helm uninstall, a Job removes the two roles the Policy Manager binds on your key from each workloads[].serviceAccount.cloudIdentity. Renders only with cloudProvider: gcp, kms.PM_KMS_KEY_ID set and at least one cloudIdentity, and then needs PM_KMS_PROJECT_ID, PM_KMS_LOCATION and PM_KMS_KEY_RING too. Pods the webhook protects are not covered, and --no-hooks skips it
policyManager.releaseGrantsOnUninstall.imageGoogle's google-cloud-cli slim image from gcr.io/google.com/cloudsdktool, by digestThe Job's image, as repository and digest
policyManager.releaseGrantsOnUninstall.resources50m and 128Mi requested, 500m and 256Mi limitThe Job's resources

Network policy​

KeyDefaultWhat it does
networkPolicy.enabledtrueRenders the ingress policies and the egress policy for protected pods
networkPolicy.defaultDenytrueDenies ingress to every pod in the release namespace that no other policy admits
networkPolicy.ingressFrom[]NetworkPolicy peers allowed to reach 443 on protected pods and the Policy Manager. Empty means anywhere
networkPolicy.protectedNamespaces[]The namespaces you labelled for the webhook. Each gets the 443-only policy for protected pods. They must exist before the install
networkPolicy.egressTo[]Egress rules for protected pods on top of DNS, the metadata server, the Policy Manager and the Sentinel. Whatever else the pod calls goes here: Cloud KMS for your workload, and the Policy Manager proxy.policyManagerUrl names when that is set. An entry of {} allows all egress. See Egress from protected pods
networkPolicy.enforcementCheck.enabledtrueThe install-time check that the cluster enforces NetworkPolicy. Turning it off is not supported
networkPolicy.enforcementCheck.image, .timeoutSecondsbusybox, 120The check's settings

Sentinel​

KeyDefaultWhat it does
sentinel.enabledtrueRuns the Sentinel DaemonSet
sentinel.port54323Its port, also published on the node
sentinel.integritytrueKernel integrity checks
sentinel.imaDisabledfalse, true in the GKE fileTurns off the IMA reader. GKE's images measure nothing in user space
sentinel.accumulatortrueThe evidence accumulator
sentinel.persistKeyEpochtrueKeeps the accumulator's epoch on the node, so a Sentinel rollout does not fail the runtime layer
sentinel.pcrIndex15The TPM register the accumulator extends
sentinel.reportIntervalSeconds30Must equal proxy.reportIntervalSeconds
sentinel.resources.cpu, .memory1, 256MiAs request and limit
sentinel.readinessProbe5 s, 3 failuresThe probe on /ready
sentinel.nodeSelector, .tolerationsempty, tolerate allScheduling on top of the pool label
sentinel.invary.*offThe Invary kernel integrity sensor. values/invary.yaml turns it on

Values files​

FileForStatus
values/gke-sev-snp.yamlGKE, SEV-SNP nodes on Container-Optimized OSMeasured
values/gke-tdx.yamlGKE, TDX nodesNot measured
values/aks-sev-snp.yamlAKS, SEV-SNPNot measured
values/eks.yamlEKSNot measured
values/oke.yamlOKENot measured
values/invary.yamlThe Invary Sentinel, on top of any of them

values/gke-sev-snp.yaml sets the gcp-sev-direct platform, the two device plugin resources, the device plugin, the AppArmor profile and its loader, the Sentinel's IMA reader off, the gke-cos baseline, and a toleration for a polaris.fr0ntierx.com/pool taint on the Policy Manager and workloads. Pass it before your own file.

Labels and annotations​

NameOnSet byMeaning
polaris.fr0ntierx.com/poolNodesYouPolaris pods run only on nodes with this key
polaris.fr0ntierx.com/webhook=enabledNamespacesYouOpts the namespace in to the webhook. The release namespace is never selected
polaris.fr0ntierx.com/protected: "true"PodsThe chart, or youA protected pod. The webhook acts only on pods with it, and the Sentinel treats the pod's other containers as the workload
polaris.fr0ntierx.com/protected: "false"PodsThe chart, on the Policy ManagerA pod that runs the proxy with nothing behind it. The Sentinel reports every such pod, and the runtime layer fails on any the baseline does not expect. The Policy Manager expects its own pod
polaris.fr0ntierx.com/workload-portPodsYouOptional. A protected pod declares exactly one port, and a value other than that port is refused
polaris.fr0ntierx.com/cloud-identityPodsYouThe Google service account the sidecar reports for a pod the webhook protects

The protected label is set by whoever writes the pod and is not attested. The NetworkPolicies use it to narrow who can reach the Sentinel and the Policy Manager, and the tokens are what guard them.

What the chart creates​

Always

  • ResourceQuota polaris-node-critical, for system-node-critical pods.
  • PriorityClasses polaris-policy-manager and polaris-protected. They are cluster-scoped and named after the release.
  • ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding polaris-protected-pod, cluster-scoped, unless admissionPolicy.enabled is off.
  • ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding polaris-gated-service-account, cluster-scoped, when admissionPolicy.enabled is on and at least one account is gated (a workloads entry or admissionPolicy.gatedServiceAccounts). It refuses an unlabelled pod running as a gated account.
  • Secret polaris-tokens, unless you set tokens.existingSecret.

Policy Manager

  • ServiceAccount, Deployment and Service polaris-policy-manager. The Deployment has one replica and the Recreate strategy, and the Service exposes 443 only.
  • ConfigMap polaris-policy-manager-baseline, unless you set baseline.existingConfigMap.
  • PersistentVolumeClaim polaris-policy-manager-state, ReadWriteOnce, unless you set persistence.existingClaim. It is kept on helm uninstall.
  • Secret polaris-policy-manager-identity, the wrapped identity key, while identityKey is on.
  • ConfigMap polaris-pm-identity, the identity key's public half, in the release namespace and in each of networkPolicy.protectedNamespaces, while identityKey is on.

Sentinel

  • ServiceAccount and DaemonSet polaris-sentinel.
  • ClusterRole and ClusterRoleBinding polaris-sentinel-pod-reader.

For each workloads entry

  • ServiceAccount, Deployment and Service named after the entry. The Service exposes 443 only.

With the GKE values file

  • DaemonSet polaris-cc-device-plugin.
  • ConfigMap and DaemonSet polaris-apparmor-loader.

With the webhook

  • ServiceAccount, Deployment, Service and PodDisruptionBudget polaris-webhook.
  • ConfigMap polaris-webhook-sidecar, the sidecar the webhook injects.
  • MutatingWebhookConfiguration polaris-webhook, cluster-scoped.
  • Secret polaris-webhook-tls, or with cert-manager a Certificate polaris-webhook and, without an issuerRef, an Issuer polaris-webhook-selfsigned.

NetworkPolicies

  • polaris-default-deny-ingress in the release namespace.
  • polaris-protected-https-only in the release namespace and in each of networkPolicy.protectedNamespaces.
  • polaris-protected-egress in the release namespace and in each of networkPolicy.protectedNamespaces.
  • polaris-policy-manager-https-only, polaris-sentinel-from-protected and, with the webhook, polaris-webhook-from-apiserver.

At install and upgrade, then removed

  • NetworkPolicy, Service and Job polaris-network-policy-check, and Deployment polaris-network-policy-check-target. They stay behind when the check fails.

At uninstall, then removed

  • Job polaris-release-grants, with policyManager.releaseGrantsOnUninstall.enabled on GKE. It runs after the release's other objects are deleted, under a new ServiceAccount polaris-policy-manager that carries the Policy Manager's annotations, so Workload Identity gives it the Policy Manager's Google service account. Both are deleted when the Job succeeds, and stay behind when it fails.

Optional

  • CronJob polaris-soak, with soak.enabled.
  • Namespace namespace.name, with namespace.create.

What runs where​

PodRuns asNotable settings
Policy ManagerIts own ServiceAccount, on a Polaris nodeProxy sidecar on 443 with its internal port moved to 3001, Policy Manager on 3000, state volume at /var/lib/polaris-pm, labelled protected: "false"
Protected podIts own ServiceAccount, on a Polaris nodeProxy sidecar on 443, internal port on 127.0.0.1:3000. Your container with a read-only root filesystem, all capabilities dropped and no privilege escalation, when it comes from a workloads entry
Sentinelpolaris-sentinel, on every Polaris nodeHost PID, the capabilities BPF, SYS_ADMIN, PERFMON, SYS_PTRACE and SYSLOG, AppArmor Unconfined, read-only root filesystem, host port 54323
AppArmor loaderDefault, on every Polaris nodePrivileged, host PID
Device pluginDefault, on every node unless you set a selectorPrivileged
Webhookpolaris-webhook, any nodeNon-root, no service account token, read-only root filesystem

The proxy sidecar runs under the polaris-proxy AppArmor profile in the GKE file, or Unconfined without it. Its requests equal its limits, so the kubelet kills your container before the proxy under memory pressure.

Kubernetes RBAC​

The chart grants one permission in the cluster: get and list on pods, cluster-wide, to the Sentinel's ServiceAccount. The Sentinel uses it to tell the proxy, the workload and the node's own processes apart. Kubernetes cannot scope it to one node, so the Sentinel can read every pod spec in the cluster, including literal environment values. It cannot watch pods or read Secrets.

The webhook's ServiceAccount mounts no token and calls no Kubernetes API.

The person who installs the chart needs rights to create every object above, including the cluster-scoped PriorityClasses, ClusterRole, ClusterRoleBinding and MutatingWebhookConfiguration.

Google Cloud IAM​

IdentityNeedsOn
The Policy Manager's Google service accountcloudkms.cryptoKeys.getIamPolicy and cloudkms.cryptoKeys.setIamPolicy, which the uninstall Job uses tooThe gated key
roles/cloudkms.cryptoKeyEncrypterDecrypter for its first start or a move to kms, then roles/cloudkms.cryptoKeyDecrypterThe wrap key, with issuerKeyProtection: kms
roles/cloudkms.cryptoKeyDecrypter, the only binding on that key, set by scripts/gke-pm-identity-key.shThe identity key, identityKey.kmsKey
The Kubernetes ServiceAccount polaris-system/polaris-policy-managerroles/iam.workloadIdentityUserThe Policy Manager's Google service account
Whoever runs scripts/gke-pm-identity-key.shcloudkms.keyRings.create and cloudkms.cryptoKeys.create (or an existing ring and key), encrypt from the key ring or the project, and cloudkms.cryptoKeys.setIamPolicyThe identity key's key ring
Each workload's Kubernetes ServiceAccountroles/iam.workloadIdentityUserThat workload's Google service account
Each workload's Google service accountroles/cloudkms.cryptoKeyEncrypterDecrypter and roles/cloudkms.publicKeyViewer, granted and removed by the Policy ManagerThe gated key

The member for a Workload Identity binding is serviceAccount:<project>.svc.id.goog[<namespace>/<ksa>]. Give the workload's Google service account no standing role on the gated key, or it keeps access when the Policy Manager removes its binding.

The Policy Manager removes an IAM binding per Google service account, not per pod. Every pod that runs as one account shares its binding, and a failure in one of them locks the account for all of them.

Anyone who can act as the Policy Manager's Google service account can unwrap its issuer keys under kms and its identity key, and can change the IAM policy on your key.

Where a deployment connects​

FromToFor
Each proxy sidecarThe Policy Manager's Service on 443Evidence reports and verdicts, over attested TLS
Each proxy sidecarThe Sentinel on its node, port 54323 on the node's addressThe node's runtime and kernel evidence, with the Sentinel token
Each proxy sidecarYour container on 127.0.0.1:<port>Forwarded requests
Each proxy sidecarThe GKE metadata serverIts identity
Your containerThe GKE metadata server and Cloud KMSUsing the gated key
The Policy ManagerCloud KMSReading and changing the IAM policy on your key, and unwrapping its issuer keys under kms
The SentinelThe Kubernetes API serverThe pod list for its node
The API serverThe webhook on 8443Pod admission
ClientsProtected pods on 443Your workload, through the proxy

NetworkPolicies​

The chart writes ingress rules for every pod it runs and an egress rule for protected pods. Egress from the Policy Manager, the Sentinel and the webhook is not limited.

PolicySelectsAdmits
Default denyEvery pod in the release namespaceNothing. Kubelet probes come from the node and still arrive
Protected, 443 onlyPods labelled protected: "true", in the release namespace and each listed namespace443 from networkPolicy.ingressFrom, or from anywhere when it is empty. In the release namespace the soak pod too, when soak.enabled is on
Policy Manager, 443 onlyThe Policy Manager pod443 from networkPolicy.ingressFrom plus protected pods in any namespace, or from anywhere when it is empty
SentinelSentinel podsPort 54323 from protected pods in any namespace
WebhookWebhook pods8443 from anywhere, since the API server is not a pod
Protected, egressPods labelled protected: "true", in the release namespace and each listed namespaceOutbound only to DNS (kube-dns and the node's DNS cache, on 53), the metadata server on 80, the Policy Manager pod on 443, the Sentinel pods on 54323, and networkPolicy.egressTo

In a namespace you listed for the webhook, the chart adds only the protected-pod policy. Another policy there that selects protected pods adds the ports it admits, since NetworkPolicies add up. A labelled namespace you did not list gets no policy, and its protected pods answer on every port they listen on.

The Policy Manager listens on 3000 on every interface. The NetworkPolicy is what keeps other pods off that port.

Egress from protected pods​

A NetworkPolicy covers the whole pod, so your container gets the same egress list as the proxy. Anything the pod calls beyond DNS, the metadata server, the Policy Manager and the Sentinel needs a rule in networkPolicy.egressTo, or the connection is dropped:

  • Cloud KMS, when your workload uses the gated key.
  • Any other service your workload calls, such as a database.
  • Cloud KMS from the proxy, when you set POLARIS_GCP_KMS_KEY in proxy.extraEnv, and the Confidential Computing API, when proxy.ratlsPlatform is gcp-sev.
  • The Policy Manager that proxy.policyManagerUrl names, whenever it is set. The chart then leaves out its own Policy Manager rule and refuses to render while egressTo is empty.

Google APIs need care. On a VPC with a private DNS zone that resolves *.googleapis.com to restricted.googleapis.com, they answer on 199.36.153.4/30:

networkPolicy:
egressTo:
- to:
- ipBlock:
cidr: 199.36.153.4/30
ports:
- protocol: TCP
port: 443

Use 199.36.153.8/30 with a zone that points at private.googleapis.com instead. Without either zone, as on the cluster the how-to creates, the names resolve to public Google addresses that change. Either add the zone, as Google describes in Configure DNS for Private Google Access, or allow 443 to addresses outside your private ranges:

networkPolicy:
egressTo:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, <control plane endpoint>/32]
ports:
- protocol: TCP
port: 443

The second form lets the pod reach any public address on 443. The node's own egress rules and L3 are then the limit. If your pod or node ranges lie outside the private ranges listed, add them to except, or the pod can reach other pods on 443. Put the control plane's public endpoint in except too (gcloud container clusters describe <cluster> --location <location> --format='value(endpoint)'). Pods still mount a service account token, and without that entry the pod can connect to the Kubernetes API on its public endpoint: measured on the spike (GKE Dataplane V2) as a TCP connect on 443, which went through without the entry and was blocked with it. The entry covers one address. A control plane IP rotation moves the public endpoint, so update except after one. A cluster with GKE's DNS-based control plane endpoint (gke-*.gke.goog) also serves the API on public Google addresses that this rule allows; turn that endpoint off, or narrow the KMS rule with the restricted zone. The in-cluster kubernetes Service goes to the private endpoint, which the private ranges already cover.

The egress policy has been measured on GKE Dataplane V2 only. On another CNI a rule that does not match closes every gate: the proxy cannot reach DNS or the Policy Manager, a new pod never turns Ready and running pods lose their key. Upstream NodeLocal DNSCache, for one, runs on the host network, so the chart's DNS rule matches nothing. The EKS, AKS and OKE values files set egressTo: [{}] for that reason, which leaves egress open until each is measured.

One protected pod per node​

The Policy Manager keeps its state per confidential VM, named by its GCE instance, and that state holds one service account: the Google service account the pod reports through Workload Identity, which gets the key grant. A second protected pod on the node would share that state with the first, so each Polaris node runs one protected pod.

WhatEnforced byCovers
A second protected pod on a nodeThe scheduler, through a required pod anti-affinity on polaris.fr0ntierx.com/protected: "true" with topologyKey: kubernetes.io/hostname and an empty namespaceSelector, so it counts protected pods in every namespacePods placed by default-scheduler. The second pod stays Pending
A protected pod without the anti-affinity, the sidecar, the pool affinity or the pool tolerationThe admission policyEvery pod labelled protected, on create. Adding or removing the label on a running pod is refused, through pods or pods/status
A protected pod with more than one containerPortThe webhook and the admission policyInit containers and native sidecars count
A protected pod that sets spec.nodeName or another schedulerNameThe admission policyBoth would skip the anti-affinity
Unrelated pods on the poolThe nodePool.taint taint, if you set it on the poolNot DaemonSets. GKE's own tolerate every NoSchedule taint
DaemonSets on the poolThe Sentinel's host profile. Each one on a GKE COS node is seeded in the gke-cos baselineA DaemonSet you add is an L3 alert until it is seeded

The admission policy cannot see other pods, so it does not count protected pods on a node. That is the anti-affinity's job. Pods that ran before the policy was installed are not checked until they are recreated, and two that already share a node keep sharing it. Anyone allowed to create pods/binding can bind a pending protected pod to a taken node, since a binding skips the scheduler and the policy does not see it; that is a scheduler's privilege. The sidecar check looks only at the container's name and restartPolicy, so a pod that brings its own polaris-proxy with every other piece in place is admitted. The anti-affinity counts only pods with the protected label, so a pod that runs the proxy without it is not counted.

Rollouts take a pod down before starting its replacement. The chart's workloads use maxSurge: 0 and maxUnavailable: 1. The default is 25% each, surge rounded up and unavailable rounded down, so below four replicas the new pod finds every pool node taken and stays Pending, no old pod may go first, and helm upgrade --wait times out. From four replicas on the rollout moves, one pod at a time. A Deployment made protected by the webhook needs the same strategy, or Recreate, and a DaemonSet maxSurge: 0. The chart's <fullname>-protected-rollout admission policy refuses one that surges. <fullname> is <release>-polaris, or the release name alone when it contains polaris, or fullnameOverride when set. On an update it checks only a change to the pod template or the strategy, so an older Deployment that surges keeps its other updates until its next rollout. It covers apps/v1 Deployments and DaemonSets only. An Argo Rollout or another controller with a protected pod template is not checked and can still get stuck the same way, so give it a strategy that takes the old pod down first.

A helm rollback to a chart from before the one-protected-pod-per-node rule needs the old pods gone first. That chart's pods have no anti-affinity and its workloads surge, but the running pods carry the anti-affinity, which the scheduler applies in both directions, so the rollback's new pods stay Pending. With the rollout policy installed, the rollback is refused instead. With admissionPolicy.enabled on, delete the binding with kubectl delete validatingadmissionpolicybinding <fullname>-protected-rollout, and stop if that fails, since with the binding in place the rollback is refused. With it off there is no binding, so skip the delete and still scale to 0. kubectl get validatingadmissionpolicybinding -l app.kubernetes.io/instance=<release> lists the name. Scale the release's workloads to 0 with kubectl -n <namespace> scale deploy -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=workload --replicas=0, and your own protected Deployments too, then run helm rollback. It removes the policy and brings back the replica count. Between two charts that both have the rule, a rollback needs no step.

Scaling is by node count: three replicas need three Polaris nodes. With the cluster autoscaler on the pool, a protected pod left Pending by the anti-affinity is what makes it add a node, so the pool's maximum is the number of protected pods plus one for the Policy Manager. This was not measured on GKE.

Measured on kind with two pool nodes: the webhook adds the anti-affinity, the policy refuses each missing piece and admits a complete pod, and of three protected pods in two namespaces two are placed on two nodes and the third stays Pending. With both nodes taken, a restart of the chart's workload places its new pod. Not yet measured on GKE confidential nodes, with the autoscaler, or as three PASS verdicts from three replicas. The chart README lists the DaemonSets on the pool and the binary each runs.

The gke-cos baseline​

policyManager.baseline.nodeImage: gke-cos renders the baseline from the chart's template templates/policy-manager/_baseline-gke-cos.tpl.

PartWhat it holds
Enforcementl2, l3 and l31 at alert, which the Policy Manager treats as advise. ima_boot off
PlatformThe cloudIdentity of each workloads entry as an allowed service account
ContainersEach workloads entry's digest under its container name
IMAExpected only when the Sentinel's IMA reader is on. The allowlist starts empty
Proxy profileDNS to kube-dns, at the service range's base address plus 10, 443 to the Policy Manager's Service address, the Sentinel's port in the node range, and loopback to each workload port. Syscalls start at accept, accept4, connect, execve and openat, and the rest is learned
Workload profileInbound from loopback, the node range and 169.254.4.6, where kubelet's probes come from on Dataplane V2 from GKE 1.35.1-gke.1516000. Outbound is learned
Host profileThe binaries and connections of a GKE COS node, learned on a test node and pruned by hand, with the cluster's ranges filled in, and the entrypoint of every DaemonSet on the pool. 443 to any address, or only to the restricted Google ranges with restrictedGoogleApis
KernelEmpty hashes and allow_unknown modules, so the kernel learner fills it

It fails the render when a cluster range, the Policy Manager's Service address or every workload's cloudIdentity is missing. An empty list of allowed service accounts would skip that check.

Pods the webhook protects are not in this baseline. Their image digests and Google service accounts are not listed, so a cluster that runs them needs a baseline you write, passed with baseline.content or baseline.existingConfigMap.

Upgrade behaviour​

ChangeWhat happens
The rendered baseline changes in any byteThe Policy Manager pod restarts, since it reads the baseline only at start. The new file wins over what it saved: learned allowlists and profiles go, layers return to the baseline's settings and learning starts again
A workloads entry's image, port or cloudIdentity, a cluster range, service.clusterIP, sentinel.port or sentinel.imaDisabledAll of these are in the gke-cos baseline, so the same as above
A chart version whose gke-cos template differsThe same as above
A ConfigMap you made yourselfNothing until you restart the Policy Manager
Proxy settings, such as proxy.policyManagerPinThe chart's workloads and the Policy Manager roll. Pods the webhook injected keep their old sidecar until you recreate them
policyManager.identityKeyThe same: the Policy Manager and the chart's workloads roll, and pods the webhook injected keep the pin they started with until you recreate them
The webhook's sidecar specThe webhook pods roll and inject the new sidecar into pods created after that
The Policy Manager restartsIt is down while the old pod releases the volume. It then holds no grants and waits deadmanIntervalSeconds before its silence check runs. Each workload gets its key back on its next passing report
trustDomain or deploymentIdThe Policy Manager refuses to start on a registry stored under another ID

Promotions you make through the API, and everything the Policy Manager learned, survive a restart as long as the baseline file is unchanged.

Policy Manager API routes​

The Policy Manager answers on plain HTTP on port 3000 inside its pod, which you reach with kubectl port-forward. Its proxy also forwards /api/v1/* on 443.

RouteTokenReturns or does
GET /api/v1/statusRead-only or managementEvery node's latest verdict, with its cvm_id
GET /api/v1/status/<cvm_id>Read-only or managementOne node
GET /api/v1/incidents/<cvm_id>/reportRead-only or managementThe latest incident as plain text
GET /api/v1/checks/<cvm_id>, GET /api/v1/history/<cvm_id>Read-only or managementThe latest checks, and past verdicts
GET /api/v1/profiles/learn, /api/v1/ima/learn, /api/v1/kernel/learnRead-only or managementLearning sessions and their state
GET /api/v1/enforcementRead-only or managementThe current setting of each layer
PUT /api/v1/enforcementManagementSets layers, for example {"l3": "enforce"}. Learned state is kept
GET /api/v1/kms/locksRead-only or managementGoogle service accounts locked by a failure
DELETE /api/v1/kms/locks/<account>ManagementLifts a lock
GET /.well-known/jwks.jsonNoneThe Policy Manager's signing keys. Its readiness probe

The layers are l2, l3, l31, ima_boot, l3_syscalls, l3_binaries and l3_network, and each takes enforce, advise or off. An unset ima_boot follows l2, and unset l3_* parts follow l3.

Proxy routes​

RouteWhereReturns
/polaris-container/health443The proxy is running
/polaris-container/ready443200 while the gate is open, 503 otherwise. The readiness probe
/polaris-container/attestation-status443The latest verdict. With the management token it adds the reason and alerts. The read-only token works too only with proxy.managementPassthrough