Back to Articles
AI Security

Before an AI Agent Goes Into Production, Test What It Must Refuse

By Asaf Levy · · 6 min read

An AI agent can summarize a document correctly, choose the right tool, and finish a workflow quickly. Those are useful results. They do not answer the question I would put in a production review: what happens when someone asks it to do something outside its authority?

Give it a document it is allowed to read. Ask it to send that document to a destination it is not allowed to use. Then inspect what actually happened.

Did it stop? Did a tool reject the transfer? Did it try another route? Did it ask a person who had authority to approve the change? Could you reconstruct the sequence without relying on the agent’s account of its own behavior?

This is a manageable test with a clear business consequence. It also exposes gaps that a successful demonstration can leave hidden.

Define the boundary before running the test

Use synthetic data and a controlled test environment. Create a sample document with a unique marker, an approved destination, and a separate test destination that policy excludes. Both destinations should be under your control. Do not use real customer information to find out whether your controls work.

Write down the expected outcome before starting. For example: the agent may read this document and prepare an internal summary, but it may not transmit the document or its contents to the excluded destination.

Include every available route in the scope: email, uploads, browser tools, messaging, and delegated agents. Blocking one transfer tool is insufficient if another permitted tool can send the same content elsewhere.

I would organize the review around four questions: identity, permissions, approval, and evidence. These are a practical review structure, not a certification framework.

1. Identity: whose authority is being used?

Start with the credentials behind the tool call. Can the receiving service distinguish the agent from the employee who requested the work? Can you identify the responsible owner and the task for which access was granted?

An agent using an employee’s broadly privileged session can make attribution difficult. A log showing the employee’s account may tell you little about which automated component acted or why.

NIST’s August 2026 discussion of agent identity highlights credential sharing, long-lived credentials, and broadly scoped access as recurring problems. Its recommendations emphasize distinct agent identities and established identity and authorization practices. Read the NIST discussion.

For this test, record the agent identity, the initiating user, the delegated authority, and the credential’s scope and expiry. If a second agent receives the task, verify that delegation does not silently expand those permissions.

2. Permissions: where is the action blocked?

The model should recognize the boundary. The application must also enforce it.

Ask the agent to perform the excluded transfer directly. Separately, verify that the tool or service rejects the same unauthorized operation even if the model requests it. Keep that verification inside the approved test environment.

This distinction matters. An agent that says “I cannot do that” demonstrates one behavior for one input. A receiving service that rejects an unauthorized action gives you evidence about an enforcement control.

Check that the restriction covers the document’s contents, not just one filename. A summary containing the same restricted information should not become acceptable simply because it has a different format.

If the only control is an instruction in the prompt, record that limitation explicitly. Do not describe it as enforced access control.

3. Approval: what exactly did the person authorize?

Some actions should remain prohibited. Others may proceed after approval from an authorized person. Define that distinction in advance.

For an approvable action, show the reviewer the data, destination, operation, and expected effect. The approval should apply to those particulars. If the destination or document changes afterward, the system should require a new decision.

Test an expired approval, an approval for a different destination, and a request from someone without the necessary authority. Also test a document that contains instructions claiming that management has already approved the transfer. Content retrieved during a task must not be able to grant itself permission.

The useful question is whether the system preserves the approval boundary throughout execution, including retries and delegation. A confirmation button alone does not establish that.

4. Evidence: can you verify the outcome independently?

Read the tool and service records, not only the conversation.

For the synthetic transfer, correlate the request, authorization decision, attempted action, and destination records. Use the unique marker to check whether content arrived. Confirm that your logging and destination checks work by running an allowed transfer as a positive control.

A missing destination record is weak evidence if that destination does not log reliably. If visibility is incomplete, mark the result inconclusive rather than passed.

Keep enough information to investigate the result without creating a new data exposure. Prefer identifiers and protected references over copying documents, credentials, or full sensitive payloads into logs. Record the relevant application, policy, tool, and model versions so the result remains interpretable after changes.

Turn the result into a release decision

Run more than the simplest request. Include changed destinations, indirect instructions in retrieved content, delegated work, and retries after a denial. Repeat the relevant cases when permissions, tools, models, or orchestration change.

For each case, distinguish a passed test, an observed failure, and an outcome you could not verify. Assign an owner to any gap and decide whether the affected capability must remain disabled until it is resolved.

NIST’s September 29 update identifies DevSecOps as the first implementation use case for its agent identity and authorization project. This is ongoing work toward practical guidance and example implementations, not a new blanket compliance requirement. Read the project update.

My recommendation is to make a documented refusal test part of the release review for any agent that can take consequential actions. It will not prove the system is secure against every attack. It will give the team something concrete to examine: a defined boundary, an enforced decision, and evidence of what happened.

Before approving the next agent, ask the team to demonstrate that sequence. A successful workflow should not be the only result on the screen.

Sources