What is generative AI security?
By Identra · Updated
Generative AI security is the practice of protecting the data, identities and systems involved when AI generates text, code, images or other content. It covers what people submit, what AI applications can reach, and what happens when generated output drives a decision or an action.
What does generative AI security cover?
Start with how people actually use it. Someone pastes a contract into ChatGPT in a browser tab. Microsoft 365 Copilot pulls a SharePoint file into an answer. Claude Code edits a repo and runs the tests. A custom app built on a model updates CRM records. Each one touches different data with different authority.
Approved tools are in scope, and so is shadow AI. For each, know the app, its owner, the accounts in use, its data sources and what it connects to. Watch for AI features switched on inside SaaS you approved long ago. The original vendor review never looked at a feature that reads every customer email.
Why does generative AI create security risks?
It makes new content out of prompts and retrieved material, and that content can look authoritative and be wrong. Feed it into a web page, a SQL query or a shell and a mistake becomes an incident.
Prompt injection plants instructions in the input or in something the AI reads. One PDF can carry useful facts and hostile directions side by side. Telling the model to ignore the hostile part won't protect anything sensitive.
AI data leakage runs through prompts, uploads, retrieved files, shared chat links and tool results. Retention, training terms, sharing settings and who owns the account all change the exposure. Opting out of training still leaves open questions about storage, access and deletion.
How do controls differ across AI use cases?
Hosted services mostly need account, configuration and data controls. Apps you build need those plus secure development around retrieval, output and tool execution.
ChatGPT, Gemini and other chat apps
- Typical exposure
- Company data pasted into personal accounts
- Control priority
- Approved workspaces, account restrictions, prompt and upload policy
Microsoft 365 Copilot and SaaS AI features
- Typical exposure
- Overshared files and broad connectors
- Control priority
- Fix document permissions, connector scopes and feature settings
Coding assistants and agents
- Typical exposure
- Source code, .env secrets, local files, shell commands
- Control priority
- Limit execution, protect secrets, review generated changes
Custom model apps
- Typical exposure
- Untrusted input steering retrieval or actions
- Control priority
- Authorize outside the model and validate tool arguments and output
What does a generative AI security incident look like?
Say a support assistant reads tickets and drafts replies. An attacker files a ticket asking it to paste an internal troubleshooting doc into the reply. Retrieval runs on an overprivileged service account and replies go out automatically. The doc leaves the company.
Several boundaries should have caught it. Retrieval limited to what this workflow may use. Ticket text with no say over permissions. Outbound mail governed by a policy enforced outside the model, plus human review when anything sensitive is attached.
Test it with a fake internal doc and a mailbox you control. A pass means the doc is out of reach and the reply can't be sent. One run where the model happens to refuse proves nothing.
Why do accounts and permissions matter?
Same service, different account, different risk. ChatGPT on a personal Gmail login sits outside company retention and offboarding. ChatGPT Enterprise behind SSO is inside them, and can still reach too much through its connectors. Approving the product and approving the account are two decisions.
Apply least privilege to users, integrations and agent identities. For retrieval, check the user's access before content reaches the model. Background jobs get a narrow service identity. Every connection gets an owner, and grants nobody uses get revoked.
How do you secure generative AI in practice?
Start with workflows that touch sensitive data or can change systems. AI governance sets ownership and acceptable use. Security's job is turning that into controls that actually block something.
- Inventory AI tools, embedded features, accounts, agents and connectors. Give each an owner.
- Offer an approved workspace and a clear exception process. Say which accounts are allowed for company work.
- Read the provider's terms and admin settings for retention, training, deletion and sharing.
- Apply data policy to prompts and uploads. Keep API keys and passwords out of model context.
- Treat generated output as untrusted. Parameterize queries and encode anything rendered in a page.
- Require approval for external sharing and deletion, with the exact target on screen.
- Have a playbook for pausing a workflow, revoking access and rotating exposed secrets.
How do you verify that defenses work?
Point AI red teaming at the app and everything it connects to. Unauthorized retrieval, malicious documents, unsafe generated commands, surprise tool calls. A prohibited action should fail at the authorization layer, every time. Rerun after any change to models, permissions, tools or data sources.
Log the acting identity, the request, the policy decision and the outcome. Those logs hold sensitive content too, so lock them down.
How Identra thinks about it
Identra shows security teams how AI is used across the browser, endpoints, and identity, SaaS and cloud services. Teams can steer supported browser AI apps to company accounts, check prompts for sensitive data on the device before they're sent, and restrict endpoint agent actions by policy. Analysts can revoke risky OAuth grants, and the result is recorded.
Go deeper: AI security, built on identity
Frequently asked questions
How is generative AI security different from AI governance?
Governance decides what's acceptable and who's accountable. Security builds the access controls, app design, testing, monitoring and response that enforce it.
Is an enterprise AI subscription enough to make usage secure?
No. Enterprise terms and admin settings help. Accounts, permissions, submitted data, connected services and generated output still need managing.
Does generative AI security apply if we do not build models?
Yes. Hosted chat apps, copilots and AI features inside SaaS all handle company data and need access and configuration controls.
Can prompt filtering prevent every generative AI incident?
No. Filtering catches some sensitive content in prompts. It doesn't replace authorization, output validation or limits on tool execution, and exposure can start in a retrieved document.
How does generative AI security differ from agentic AI security?
Generative AI security covers content generation and the workflows around it. Agentic AI security focuses on systems that choose and run actions, which adds delegated authority, tool permissions and control over long-running tasks.