Authors:
Marina Kaganovich, Office of the CISO.
Anton Chuvakin, Office of the CISO
Introduction
“AI governance” is a term that is often too narrowly focused on the technological shift genAI has spurred. While its transformative nature has been significant, what’s equally important is that genAI enables an opportunity for an end to end enterprise-wide transformation that impacts technology, products and services, and operations, providing an opportunity to reimagine and redefine comprehensive governance at scale.
As strategic advisors in the Office of the CISO at Google Cloud, we are often asked for guidance on how organizations should think about structuring their AI governance programs. Unsurprisingly, they’re eager to implement AI, but need to ensure they do it with the proper guardrails. Likewise, while the majority of organizations’ leadership is keen to weave AI into all aspects of their operations, the reality of accomplishing this task is frequently throttled by fear and uncertainty resulting in analysis paralysis, with a lack of consensus preventing the desired forward motion to implementation.
So what does good AI governance look like?
While the nuanced response varies by industry sector and geography, most organizations we work with use a mix of the NIST AI Risk Management Framework (RMF), ISO/IEC 42001, and the EU AI Act to inform their approach to building a comprehensive, structured and repeatable mechanism for governing AI use cases and defining the associated controls. As an aside, for security controls in particular, robust governance programs also overlay NIST’s security requirements, Google’s SAIF, and refer to the OWASP Top 10 for LLM threats. We’ll focus primarily on the risk management frameworks for purposes of this blog, but if you’re interested in further reading on our AI security governance guidance, see our recent blog here.
The Shared DNA of Modern AI Frameworks
While ISO 42001, the NIST AI RMF, and the EU AI Act differ in their nature - one is a certifiable management standard, one is a voluntary framework, and one is a binding regulation - they share a common operational philosophy. They all converge on five core operational pillars. As recently noted, “The most significant trend is the transition from AI as a passive assistant to AI as an active part of the team, where specialized agents can orchestrate entire workflows.” Let’s unpack what this means in practice for security and risk management.
1. Discovery & AI Inventory
You can’t manage what you can’t see. This was true for servers in 1996, it was true for cloud and SaaS in 2016, and it is true for LLMs and agents today. All three frameworks essentially mandate an AI registry to maintain a formal inventory of AI systems in use, their intended purpose, risk classification, and the identification of a responsible party if things go sideways.
Data quality is paramount. Before you can build a comprehensive registry, you have to get a full view of the AI in your environment, which includes bringing shadow AI out of the shadows. This goes beyond implementing an Acceptable Use Policy toward actively identifying the rogue tools that may be operating in your environment, and onboarding the valid use cases through the governance process.
2. Risk-Tiering
The frameworks universally reject a rigid, one-size-fits-all approach. Rather, they encourage a risk-based approach that informs the types of technical and operational controls that are appropriate relative to the potential harm that may result. For instance:
-
EU AI Act formalizes this into a four-tier risk based system (Unacceptable, High, Limited, and Minimal).
-
NIST AI RMF focuses heavily on the "Map" and "Measure" functions to quantify negative impacts.
-
ISO 42001 requires a documented risk assessment process to determine which technical controls are legally and operationally in scope.
3. Technical Guardrails
Likewise, from a security governance perspective, the following best practices have surfaced:
-
Pre-Deployment: Vetting the Supply Chain
Verify the security posture of third-party model providers and audit the Bill of Materials for open-source models. Evaluate the tech stack holistically, ensuring security controls are built in throughout the infrastructure, application, data and model layers. -
Architecture: Identity & Access Management
Assign unique service accounts and credentials to AI agents. Grant only the permissions absolutely necessary for their tasks and for the shortest required time as determined by the desired outcome. Every action performed by an agent must be unequivocally attributable to that agent. There should be no anonymous agents running amok; endpoint and network security monitoring should have detections. -
Deployment: Content Filtering & Guardrails
Today’s AI is prone to hallucinations. Accuracy and output quality are attributes that can be addressed by grounding in authoritative data sources.
Deploy a firewall for AI that inspects every prompt for injection attacks and every response for PII leakage or untrustworthy content as defined by your internal policy. For instance, Google’s Model Armor provides comprehensive protections against prompt injection, sensitive data leaks, and harmful content by detecting malicious files, malware, and unsafe URLs within AI prompts and responses. -
Monitoring: Continuous Red Teaming
Schedule recurring adversarial tests. The roles are clear: humans plan the scope (govern), and AI doing the testing. Unlike traditional software, AI vulnerabilities (like jailbreaks) evolve at unprecedented rates, requiring a continuous testing loop rather than a point in time audit.
4. Human Oversight
A recurring, non-negotiable theme across the board is that AI cannot be a black box left entirely to its own devices; it must have well-defined human controllers to ensure accountability and prevent unchecked autonomy in critical situations. There is a shared requirement for a human “kill switch" or intervention mechanism. Whether this means Human-in-the-Loop (HITL) for high-stakes decisions, or Human-on-the-Loop (HOTL) acting as an auditor, depends entirely on the velocity and risk of the use case.
The frameworks demand clearly defined roles and technical interfaces that allow a human to override or abruptly terminate a system when it deviates from expected behavior. Many organizations look at a high-risk AI agent and say, "It’s fine, we will just design it to escalate to a human if it’s unsure." Resist this temptation and be mindful of the realities of machine speed and scale, and ensure that the risk-tiering informs escalation paths. The human just clicking "OK" isn't enough; the human must have the competence and time to actually understand and assess the output. If they don't, the system is effectively autonomous, regardless of the UI.
5. Lifecycle-Wide, Continuous Monitoring
None of these bodies treat risk as a rubber-stamp exercise done at deployment. You are expected to monitor the AI's performance, drift, and security posture continuously post-deployment. This translates to a heavy emphasis on event logging. If an AI system makes a catastrophic decision, you must possess the immutable logs required to reconstruct the entire algorithmic decision-making process.
The Matrix: A 5-Step Practical Guide to Risk-Ranking AI Use Cases
Transitioning from high-level principles to daily operations requires a consistent, repeatable way to evaluate the potential impact of every AI implementation. Once leadership agrees on the theory, the conversation often hits a wall on execution.
So, how can you actually rank AI use cases without drowning in paperwork or grinding innovation to a halt?
Based on what we have seen work in real-world environments, you need a pragmatic, matrixed view of risk, evaluating not just the model, but its sphere of influence across your customer base, financial health, and legal liability. Here is the 5-step guide you can leverage to risk-rank your AI use cases.
Step 1: Quantify Autonomy & Control
How much power are you actually handing over to the machine? Effective risk management requires carefully limited powers that align with a system’s intended purpose and the organization’s risk tolerance. This means moving beyond static permissions toward dynamic constraints that can evaluate the risk of an action in real-time, ensuring a system cannot inappropriately escalate its own authority. We categorize the execution authority into three distinct buckets, risk-ranking them by their potential impact on the organization’s customers, financial health, reputation, and regulatory requirements.
-
Autonomous (Critical): The AI has the authority to independently execute transactions, terminate contracts, or modify production environments without human sign-off. The risk profile requires well-defined use cases and decision trees, alongside real-time automated guardrails to minimize risk, and the “big red button” for a graceful shutdown should one be needed.
-
Augmented (Moderate): The AI generates a decision, recommendation, or analysis, but a human must actively review and click "approve" before any action is taken. In such instances, the risk shifts to automation bias where humans rubber-stamp outputs, particularly when they’re presented in a convincing manner. Some ways to mitigate this risk is through prompt engineering, explicitly requiring citations for output to help with the validation, and instructing the model to only answer using approved sources., If the answer isn't there, the AI should say it doesn't know. Further controls to consider include providing an underlying justification for its output to support the human reviewer in making their assessment, and to consider implementing a confidence scoring threshold, so if the AI doesn’t have sufficient justification for a response, it flags that too.
-
Supportive (Low): The AI assists purely with drafting, brainstorming, or summarizing internal text without directly impacting final operational outcomes. Standard data handling policies typically suffice, though a reminder to critically assess the AI’s output is always warranted.
Step 2: Evaluate the Degree of Agency
We have seen this before with software development: we underestimate the complexity of architectural shifts. Moving from standard LLM prompts to agentic architectures drastically shifts the failure modes. You need to classify your systems by their level of architectural agency:
-
Tools (Bounded Systems): These are reactive, deterministic systems operating within tight, predefined boundaries. They don't look for new ways to solve a problem. However, they are highly vulnerable to prompt injections that can force them to leak PII or unauthorized data.
-
Agents (Adaptive Systems): These are goal-seeking systems. You give them a mission, and they determine the steps, adjust their strategies, and interact with other APIs to achieve it. The risk here is goal misalignment. An agent might optimize too rigidly for a metric and cause massive operational damage to achieve it.
-
Collectives (Emergent Systems): These are interconnected networks of multiple autonomous agents interacting with one another with zero human oversight. The unpredictable, emergent behavior of these systems represents a potentially catastrophic blast radius if left unmonitored.
Step 3: Audit Data Access, Volume, and IAM Boundaries
The risk of an AI system is fundamentally limited by the data it can access and manipulate. Evaluate the degree to which sensitive data is exposed, for how long, and enforce strict Identity and Access Management (IAM) boundaries around the AI. If an agent is compromised via a prompt injection attack, its blast radius is identical to the data volume it can access. Treat the AI agent like a highly privileged, potentially untrustworthy service account.
Step 4: Focus on the Blast Radius
Where do the ripples of an AI failure stop? If the system goes completely off the rails, who is impacted?
-
Customer-Facing (Higher Risk): Direct, unmediated interactions with the public (e.g., AI wealth advisors, patient-facing diagnostic assistants). A failure here results in immediate reputational damage, financial lawsuits, or regulatory penalties. Worst case it causes a life-safety issue.
-
Employee-Facing (Moderate Risk): Internal productivity tools (e.g., automated HR screening tools, internal coding assistants). The risk is largely contained within your perimeter, but a failure can still corrupt operational integrity or introduce systemic software bugs.
-
Back-Office/System-Facing (Lower Risk): Backend process optimization (e.g., automated log analysis or data cleansing). Failures here are usually caught by downstream systems, making this the ideal testing ground for new models.
Importantly, these categories are for illustrative purposes. For instance, there can be a customer-facing impact that is medium risk, and a back office process impact that is high risk. So these use case considerations are highly dependent on the business mix and risk appetite of the organization.
Step 5: Evade the “Set It & Forget It” Trap
Invest in threat modelling to systematically analyze system architecture and data flows, evaluate what could go wrong, and how these risks can be mitigated. While guidance applied long before AI came on the scene, the non-deterministic nature of AI makes observability even more critical today. Beyond a simple log of the final answer; organizations can now leverage emerging tools to gain visibility into the reasoning steps and tool choices a system makes along the way. This transparency into the planning process is designed to help organizations ensure accountability or troubleshoot instances where there may be a divergence from intended behavior.
Identify specific preventative and detective controls that could mitigate the risk of each type of impact - remember, these must be actual operational controls, not just documented policies. Solely implementing a policy that prohibits a particular behavior, (e.g., "The AI shall not access unapproved data") is governance theater. It does absolutely nothing to prevent the activity being warned against.
Verify that the controls actually apply to the use case and validate that the controls aren’t present in only one or some systems, but not in others, and would therefore only partially work. The controls need to be universally applied. For example, if you have robust prompt-filtering and input sanitization on your web interface, but your backend API endpoints are exposed directly to the model without those same filters, the robust control posture is an illusion. If a control can’t be programmatically verified, it shouldn't be considered as a mitigating control in the risk matrix.
Conclusion
AI governance is frequently framed as a series of hurdles designed to slow things down. In reality, it is the fundamental architecture that allows an enterprise to move up from the speed of a human-led process and go up to machine-speed without losing control. By aligning with the shared attributes of the NIST RMF, ISO 42001, and the EU AI Act, you aren't just checking a compliance box—you are building a repeatable engine for innovation.
As we transition into an era of autonomous agents and emergent collectives, the margin for error narrows even further. By leveraging the 5-step risk-ranking matrix, organizations can stop treating all AI use cases as equal threats and start applying precision security where it actually matters. Governance is a living process, not a one-time audit. As the LLMs and AI applications evolve, so must our governance and oversight mechanisms.
