Generative AI (gen AI) is rapidly moving into production environments and while many organizations are still in the testing phase, a growing number have already deployed gen AI solutions. For early adopters, we are seeing clear evidence of security missteps and some common patterns have emerged. In this post we expand on the themes we started here, and include additional details for these increasingly common security mistakes. We are highlighting those security mistakes that appear to be most prevalent to help you identify and avoid them in your gen AI implementations.
Mistake 1: Weak AI Governance
The mistakes here often stem from a lack of, or poorly defined, AI governance. Without proper AI governance, organizations will inadvertently perpetuate inconsistent practices and likely induce security gaps which increase the likelihood of both data breaches or misuse. Weak AI governance hinders transparency and accountability, making it difficult to track how AI models are used, the data they are accessing, and the decisions they inform. This lack of visibility creates risks for the business and its customers since their data may be mishandled or used in ways they did not consent to.
Some organizations take an open approach, even going so far as to use consumer-grade gen AI for business purposes. Others impose excessively restrictive governance, leading to shadow AI as individuals find workarounds to utilize gen AI outside of what they see as stifling rules. Irrespective of where your organization is on this spectrum, what we’re frequently seeing is an oversimplified approach to governance that’s focused on form over substance.
We find that outright bans on gen AI use, regardless of whether they pertain to consumer or enterprise tools, are not an effective practice and will drive those who are keen to accomplish their business goals with the aid of gen AI to circumvent controls. This is where good governance and employee upskilling can help - it’s an opportunity to ensure that empowered SMEs from across the organization are gathered to review gen AI initiatives in a holistic, programmatic and repeatable format that has a line of sight into the entire end to end process rather than only to certain component parts.
Structurally, an AI Governance program should also plug into the organization’s Data Governance program because AI models typically require high-quality data which should be appropriately sourced, cleansed, and normalized. It should also form connections with the Procurement and Third Party Risk Management programs that have a line of sight into the use of third party tools. And, of course, have input from Security, Legal, Risk, Compliance, Privacy, and other teams so that risks are comprehensively considered.
We previously published some tips for establishing a robust governance structure for gen AI. Organizations should look to create a well-lit-path approach that’s clearly communicated and informed by a feedback loop for enhancements when needed. By taking this approach, the governance committee is positioned as an enabler rather than a blocker. In addition to being a decision-making body, it can also be an input mechanism for creative ideas, such as by using this process to drive use case ideation, not just a review of particular gen AI technologies’ applicability to specific use cases.
Mistake 2: Bad Data
Organizations often underestimate the effort needed to collect, clean, and label data, and the iterative steps required throughout the data lifecycle. Training AI models requires extensive, high-quality data. If that data is inaccurate or incomplete, the AI output will also be flawed.
Poor data governance practices severely inhibit AI development and lead to a range of negative consequences that erode trust in AI outputs. Additionally, flawed data sets compound these negative outcomes by making it harder to identify and correct errors. Likewise, when data lineage and transformations are not properly tracked, it becomes challenging to identify the root cause of problems in AI models, hindering debugging and improvement efforts. Lastly, absent clear guidelines on data usage, there is a risk of data being used for purposes beyond its intended scope, potentially harming individuals and exposing the organization to legal and reputational risk.
This is why it’s crucial to implement robust data governance practices. Since AI models typically require high-quality data which should be appropriately sourced, cleansed, and normalized, the AI governance program should include a cross functional team that includes data SMEs and should also closely align to your organization’s existing data governance program. In addition, business leaders should implement strong security measures to protect data confidentiality and integrity, including using encryption, deploying access controls, and applying anonymization techniques.
Mistake 3: Excessive, Overprovisioned Access
A frequent oversight in gen AI security is granting excessive access to corporate data without implementing sufficiently granular access controls. This can lead to unintended and severe consequences, such as the inadvertent exposure of sensitive information. In the security context, holistically evaluating and tailoring role-based access rights can help guard against the risk of having security bolted on rather than built in which can occur when gen AI implementation is retrofit into existing workflows and development pipelines that don’t adequately assess security controls coverage.
Over-provisioned access can generate the same types of risk, with the gen AI using and exposing potentially sensitive information that is assumed to be inaccessible because the human testers don’t have access to it. For example, a gen AI chat bot intended for internal use by IT support may have access to sensitive vulnerability management data that it then exposes when queried, even though the individual interacting with it isn’t provisioned to see such data.
By ensuring there’s a focus on implementing strict, role-based access controls, organizations can significantly reduce the risk of data leakage and maintain the confidentiality of sensitive information. In addition, access rights granted to non-humans through APIs should also be considered as the review of these rights isn’t always apparent or done with the same consistency or depth as rights granted to humans. This applies to access to control APIs, as well as structured and unstructured data repositories.
Mistake 4: Neglecting Inherited Vulnerabilities
Up to this point, we’ve discussed the gen AI-powered systems used by the corporate enterprises and focused on their access controls. However, it’s important to be mindful of the inverse possibility where the model and access controls are appropriate, but the platform on which the model is trained and operated is not secure. This situation carries the risks of exposing sensitive data leading to data leakage and potential model tampering. Organizations using third-party models often make incorrect assumptions about model ownership and security.
It's important to remember that models can inherit vulnerabilities from their foundational models. There is also often an assumption that organizations fully control their AI models. However, with fine-tuned gen AI models or third-party APIs, much of the model's behavior might be opaque, passing hidden vulnerabilities if unchecked. It’s thus critical to ensure that the gen AI being used is fit for purpose.
AI models often interact with other systems in production, and each interaction can introduce security risks which can be difficult to trace and uncover. Continuous testing of the entire system is necessary to understand the interconnectedness between AI models and other applications and systems, and how these interactions might be exploited. Consider, for example, a company which deploys multiple AI models that interact with one another and other systems, and a vulnerability in one model allows attackers to compromise the entire system through their interconnectedness.
Gen AI system security should be approached holistically, evaluating infrastructure, application, model and data security. Take the overall infrastructure into account when building AI models or determining which third party models to use. Make sure not to propagate misconfigurations, such as by replicating a testing environment in production and furthering insecure configurations in deployment.
In addition, systems should be kept up to date and updates well-documented. Without rigorous version control, security patches applied to one model version may not carry over during updates, resulting in inconsistent security across different versions. For instance, a development team updating a gen AI model but neglecting to reapply a crucial security patch from a previous version, can leave the system open to attack. A more fulsome discussion on best practices for securely deploying AI can be found here.
Lastly, resilience is a point that is frequently overlooked in the context of gen AI as it has not historically been considered as part of an organization’s BC/DR. However, as AI use becomes more pervasive, this needs to be re-evaluated and included in your planning exercises. AI systems, particularly those targeted by poisoning attacks, can become corrupt or ineffective, and recovery may be difficult. Without strong business continuity and recovery strategies, organizations may struggle to restore operations after an attack.
Mistake 5: Assuming Risks Only Apply to Public-Facing AI
A critical oversight in gen AI security is the false assumption that risks solely or primarily pertain to customer-facing and public-facing AI models. Organizations often underestimate the potential vulnerabilities within their internal chatbots and AI-powered internal tools. Organizations often prioritize security for public-facing applications because of their direct exposure to external threats and higher potential for reputational damage. This prioritization can create a false sense of security for internal AI tools, which are presumed to be less vulnerable due to their limited accessibility. This is happening to companies today!
The perception that internal AI tools are primarily used by trusted employees can create a false sense of security, and encourage a more relaxed approach to security. It’s important to fight this urge. After all, organizations have been compromised via weaknesses in internal tools, even before gen AI. Actions taken based on this perception often overlook the possibilities of insider threats, accidental data exposure, and vulnerabilities in the internal network, all of which can be exploited to access the sensitive information processed by these AI tools.
As an example, a seemingly harmless internal chatbot could inadvertently leak confidential information if queried cleverly, or worse, if compromised. Imagine the impact of an internal chatbot with unrestricted access to employee records revealing confidential details such as performance reviews or compensation if queried in a specific way.
These examples further highlight the critical need for organizations to carefully manage data access for their AI systems to ensure that they only have access to the information necessary to perform their designated tasks. Be sure to apply consistent security measures to both public-facing and internal AI tools. This involves implementing robust access controls, data encryption, and regular security assessments for all AI implementations, regardless of their intended audience.
Bonus Mistake 6: Granting Excessive Exceptions
Since we noted AI governance as foundational mistake #1, we thought it made sense to close this blog with a bonus mistake on this topic. We often see confidence eroding in the governance process when there are excessive decision-making delays on whether a gen AI use case can proceed, often because its functionality and controls are poorly understood by reviewers. These cause well-meaning personnel to start seeking workarounds as they start to view the process as being too onerous and ineffective.
On the flip side, in an attempt to manage the resolution of outstanding requests, exceptions are granted too freely as a risk-acceptance when the committee bends to the will of the business and doesn’t enforce controls that are sufficiently robust to effectively guard against risk.
We recognize that for certain time-sensitive circumstances, an exception process may be needed, but in such cases it should be structured narrowly to provide decisive but temporary exceptions, ensuring that the request is re-evaluated within short order to ensure enterprise controls compliance is still achieved. If granted, exceptions should be friction-heavy and difficult to obtain, and approved only in truly exceptional circumstances. Exception rationale should be clearly documented, articulating the basis for the decision, and providing a path for exception reporting to inform visibility and oversight.
Key Takeaways
The early deployment of gen AI highlights the importance of a holistic security approach that encompasses planning, development, and operational stages. To be successful and avoid these missteps, establish holistic governance, carefully manage data access, and pay close attention to infrastructure, application, model and data security. By learning from these early mistakes, you’ll be able to anticipate and avoid them so that your gen AI is deployed securely and responsibly.