What is AI agent observability?
By Identra · Updated
AI agent observability is the ability to reconstruct what an AI agent did from recorded inputs, tool calls, data access and outcomes. For security teams, it also ties each action to the identity, permissions and approvals behind it.
What does an AI agent trace record?
One agent run touches a lot. Say Claude Code is asked to fix a failing test. It reads files, calls the model several times, runs the test suite, edits a config and retries. Observability lets you line those steps up afterward. Usually that means a shared run ID, timestamps and parent-child links between calls.
A single event says little on its own. Reading customer_export.csv is normal inside a reporting task. Inside a task to fix a CSS bug it is strange. The AI audit trail keeps the evidence. Observability is how you query it while agents are still running. It shows what was captured and can't prove nothing was missed.
- Run: task, start and end time, parent run, final status.
- Who: agent identity, owner, the user who started it, device or runtime.
- Action: tool, operation, target and arguments, with secrets stripped.
- Result: the authorization decision, any approval, the tool response and what actually changed.
How is developer tracing different from security monitoring?
Developer tracing in tools like LangSmith answers engineering questions. Why was that response slow? Why did the tool call retry? Security asks whether the action was allowed and what it touched.
Both can read the same records and reach different verdicts. A tool call that returned success can still have mailed a confidential file. A run marked failed can still have pushed a commit before it crashed.
Task
- Developer asks
- Did it finish correctly?
- Security asks
- Was it authorized?
Tool call
- Developer asks
- Why did it fail or retry?
- Security asks
- Which identity and permissions did it use?
Data
- Developer asks
- Did it get useful context?
- Security asks
- Should this recipient have seen it?
Outcome
- Developer asks
- Was the answer good?
- Security asks
- What changed, and did it need approval?
Which identity should a trace record?
The agent's name isn't enough. Record three identities separately: the accountable owner, the user who started the task and the identity the agent ran as. That last one might be an Entra ID service principal or a GitHub personal access token. AI agent identity is how you tell a user's request from an unattended 2 a.m. scheduled run.
Handoffs break traces. When one agent passes work to another, log the identity the second agent used and what it was allowed to do. Otherwise AI agent delegation leaves you with two clean traces and nothing joining them.
Capture the permission decision at execution time. Roles get edited, and the access in place during an incident may be gone by the time anyone looks. Store credential references. Never the secret.
What does an investigation look like?
Say a support agent is asked to summarize a customer's open cases and email the account team. One case has an attachment with hidden text telling the agent to send the summary to an outside Gmail address. That is a classic indirect prompt injection setup.
The investigator pulls the original request, the attachment retrieval, the recipient the agent proposed, the mail tool call, the identity used and any approval. Then they check the mail system itself, for example the Exchange message trace in Microsoft 365, to see whether anything left. The agent replying "Done, summary sent" proves nothing.
A blocked send and a delivered send are different incidents. If the trace kept the recipient and message ID but not the body, you may know where it went and not what it said. Write that gap into the case.
Where should a security team start?
With actions that hurt when they go wrong. Sending email, exporting records, changing permissions, running shell commands. For each one, decide what evidence you need and check that the agent runtime and the target system produce it. Wire it into your AI incident response process with a named owner.
Logging a bad action does not stop it. Prevention takes a separate control, such as scoped permissions or an approval step before the call runs.
- Inventory agents, owners, tools and where each one runs.
- Pass a run ID through tool requests where the framework allows it.
- Log the request, the policy decision and the observed effect as separate records. A timeout can leave the outcome unknown.
- Pull in destination audit logs from Microsoft 365 or Google Workspace where you can. A permission grant is not evidence of access.
- Test an allowed action, a denied one and a partial failure. A reviewer should be able to rebuild each from the records.
Should traces store full prompts?
Not by default. Prompts, retrieved documents and tool output carry API keys, customer data and source code. Capture all of it and the trace store becomes one of the most sensitive systems you run. Apply least privilege to who can read traces and set a retention period.
Resource IDs, action metadata and outcomes answer most questions. Capture content for sensitive actions only and redact secrets before storage. Treat logged instructions as untrusted text, including when an AI tool summarizes the logs for you.
Less content means less exposure. Sometimes it also means you can't say what the agent read.
How Identra thinks about it
Identra records every AI agent run on macOS and Windows endpoints with the user, device, AI client and whether it was allowed or blocked. Browser, endpoint and provider activity lands on one timeline per person, where identity resolves, so an agent run can be investigated next to that person's other activity.
Go deeper: AI security, built on identity
Frequently asked questions
Is AI agent observability the same as logging prompts?
No. Prompt logs capture inputs. Agent observability links inputs to tool calls, identities, decisions and outcomes across a whole task.
Does observability show an agent's reasoning?
It shows recorded inputs, outputs and actions. An agent's own explanation of why it acted is not reliable evidence.
Can observability detect prompt injection?
It can surface suspicious instructions and the actions that followed. A trace alone rarely proves injection caused a specific action.
Do you need to store every prompt and tool result?
No. Match capture to what investigations need and what your data policy allows. Action metadata covers most cases.
Does observability prevent unsafe actions?
No. It supports detection and investigation. Prevention needs tool authorization, restricted access or a required approval.