Authors:
Andy Lu, Strategic Cloud Engineer
Liga Viskinte, Security Cloud Consultant
Implementing mutual TLS (mTLS) in Kubernetes historically introduced significant overhead in certificate management and application reload orchestration. Google Kubernetes Engine (GKE) Managed Workload Identity (MWLID) simplifies this by automatically issuing short-lived, auto-rotated SPIFFE certificates to GKE pods. Under the hood, MWLID uses Google Cloud Certificate Authority Service (CAS) as its cryptographic anchor. You generally have two paths: use the Google-managed default Certificate Authority (CA) for a fully automated, zero-management experience, or deploy a Custom CA to meet strict compliance requirements and maintain an isolated root of trust.
If compliance dictates the Custom CA route, platform teams inherit the responsibility of managing the underlying root and subordinate CA lifecycles. Rotating active CAs carries inherent operational risks: prematurely deleting an old CA before its issued leaf certificates expire immediately invalidates active mTLS sessions, leading to service disruption.
To eliminate these manual risks, we designed an automated, zero-downtime rotation architecture using a "trust, then swap" model. Here is a look at the underlying architecture.
Building a Hands-Off CA Lifecycle Manager
The automated CA lifecycle management tool is implemented as an event-driven architecture using Google Cloud native services. The core orchestration and rotation logic is handled by an agent written in Go, deployed on Cloud Run, and scheduled via Cloud Scheduler. Long-running state delays are using Cloud Tasks.

Here is how the automated rotation ensures zero downtime for GKE workloads:
1. The Graceful Flip (Cloud Run + Cloud Scheduler)
Routine rotations are triggered automatically by Cloud Scheduler. The Go agent spins up, queries the regional CA pool, and executes a delicate "flip" rather than a destructive replacement:
-
Discovers the current active Root CA and active Subordinated CAs in the pool.
-
Creates a new Subordinate CA, generates a Certificate Signing Request (CSR), and signs it against the Root CA.
-
Enables the new CA so it can start issuing new pod certificates.
-
Disables the old CA. The old CA stops issuing new certificates but crucially remains in the pool's trust bundle. This allows existing short-lived pod certificates to safely validate until they naturally expire.
2. The 48-Hour (Configurable) Waiting Room (Cloud Tasks)
You cannot delete the old Subordinate CA while pods are still using certificates signed by it. Pausing a Cloud Run Job for two days to wait for expirations is inefficient and costly. To solve this, the Rotator agent generates a delayed webhook payload, drops it into a Cloud Tasks queue to be executed after a configurable delay (48 hours by default), and exits successfully.
3. The Sweeper (Cloud Run)
Two days later, Cloud Tasks dispatches the webhook back to a secondary Cloud Run "Sweeper" job. The Sweeper performs the final cleanup by executing the permanent DELETE API call on the old CA, and spins back down, completing the rotation lifecycle.
Designing for Resilience
Automating cryptographic operations requires accounting for disruptions from API timeouts and partial state failures. To ensure high availability and prevent accidental key destruction, the lifecycle manager incorporates four core resilience patterns:
Dynamic Auto-Discovery
To avoid hardcoding CA resource identifiers in Terraform state, the agent queries Google Cloud CA APIs at runtime to discover the active Root and Subordinate CAs. In newly provisioned Google Cloud projects with empty CA pools, the agent automatically bootstraps the foundational Root CA using a predefined baseline configuration.
The "Paused Job" Failsafe
Permanent deletion of a cryptographic key is irreversible. The Root CA Scheduler is deployed in PAUSED state to provide a human-in-the-loop safety net. The scheduled job will not execute until an operator explicitly triggers the job. This gives teams a chance to verify cluster health before any keys are permanently destroyed.
Staggered Regional Rollouts:
To limit the blast radius, the system never rotates CAs in all regions simultaneously. When a global Root CA rotation is triggered, the Go agent uses Cloud Tasks to schedule a staggered regional flip. A configurable 1-hour delay is introduced for each region (e.g., US-East at T+3h, Europe-West at T+4h). This ensures that if a trust propagation issue occurs, it is localized to a single region, giving SREs time to pause the execution before the next region begins.
Two-Layer Idempotency
Network blips and API rate limits can interrupt execution mid-rotation. The automation enforces idempotency across two layers. If a Cloud Run job crashes mid-rotation, its inner loop simply retries. If the agent wakes up and finds a CA in a half-rotated state, it knows how to read the resource labels, heal the state, and pick up exactly where it left off. For broader outages, Cloud Tasks acts as the outer loop, handling exponential backoffs automatically.
Managing the Root of Trust
While regional Subordinate CAs rotate on a short-term cadence, rotating the Root CA is a high-risk operational event with a fleet-wide blast radius. To provide a secure transition, this process uses the same Cloud Run and Cloud Tasks architecture to coordinate a multi-day lifecycle. The system divides the rotation into three phases: staging the new Root, enabling it, and executing regional flips. State payloads are passed through delayed Cloud Tasks to complete the global transition without disrupting active workloads.
Real-Time Alerting: Knowing When Things Break
To monitor rotation health, the Cloud Run jobs integrate directly with Google Cloud Monitoring.
If an API times out, a step fails, or a critical error occurs during the rotation, the system automatically triggers a log-based metric alert and sends out an email notification to the SRE team. While the process remains automated, operators are brought into the loop if an issue arises.
Conclusion
Automating management of custom CA lifecycles greatly reduces operational risk. Combining the stateless execution of Cloud Run with the delayed scheduling capabilities of Cloud Tasks creates a reliable, automated pipeline that handles rotation seamlessly in the background. Get started today with the complete Go codebase, Terraform models, and setup runbooks available in the Github Repository.
