In this blog post Why Microsoft Foundry Agents Need Red Teaming Before Launching we will explain how to test an AI agent like a determined attacker before employees or customers depend on it. The risk is not simply that the agent gives an odd answer. It could expose sensitive information, ignore approval rules or use a connected business system in a way nobody intended.
At a high level, red teaming means deliberately trying to make an AI agent fail. Instead of checking whether it works under normal conditions, you test what happens when someone provides misleading instructions, requests restricted information or attempts to manipulate the agent into taking an unauthorised action.
What red teaming means for a Microsoft Foundry agent
Microsoft Foundry Agent Service is a managed platform for building and operating AI agents. An agent combines an AI model with instructions, company information and tools that let it complete tasks, such as searching documents, creating support tickets or updating a business system.
This ability to take action is what makes agents useful. It is also what makes their risks different from those of a standard chatbot.
A chatbot that fails might produce a poor answer. An agent that fails could email the wrong customer, retrieve another employee’s records or submit a transaction without the required approval.
If your team is still planning the underlying agent, our guide to building Microsoft Foundry agents with Microsoft Agent Framework explains how to move from a prototype to a controlled business tool. Red teaming is the pressure test that should happen before that tool is trusted in production.
How the red teaming technology works
Microsoft’s AI Red Teaming Agent uses automated adversarial testing. In plain English, it generates difficult or hostile requests, sends them to your agent and evaluates whether the agent responded safely.
The technology draws on Microsoft’s Python Risk Identification Tool, commonly called PyRIT. It can vary the wording and structure of attacks rather than relying on a small list of predictable test prompts.
A typical scan includes three parts:
- Probing: The service sends prompts designed to test areas such as jailbreak resistance, harmful content, privacy leakage and instruction manipulation.
- Scoring: Each response is assessed to determine whether the attack succeeded.
- Reporting: Results are grouped into risk categories and attack strategies, giving the team a repeatable scorecard rather than a collection of opinions.
One useful measurement is the attack success rate. This shows the percentage of attempted attacks that caused the agent to behave in an unacceptable way. However, the number must be read alongside the severity of each failure. One successful payroll data leak matters more than ten awkwardly worded responses.
What should be tested before production
1. Whether the agent respects its business boundaries
Start with what the agent must never do. A customer service agent might be allowed to explain the refund policy but not approve a $20,000 refund. An HR agent might answer questions about leave but must not reveal another employee’s salary or performance review.
Test direct requests, misleading requests and requests hidden inside longer conversations. The business outcome is clear: fewer opportunities for fraud, privacy incidents and costly manual remediation.
2. Whether connected tools have too much access
An agent should only receive the minimum access needed for its job. This is known as least privilege, meaning it cannot read or change information outside its approved role.
Red teamers should try to make the agent call the wrong tool, use incorrect parameters or bypass a required approval. Every important action should also be logged so the business can see what happened and who initiated it.
This becomes especially important when connecting Foundry agents to external agent-to-agent endpoints. Each connection expands the number of systems, identities and failure paths that need testing.
3. Whether company data can manipulate the agent
Attack instructions do not always come from the user. They can be hidden inside a document, email, website or database record that the agent has been asked to read. This is called indirect prompt injection.
Imagine an agent processing a supplier document containing hidden instructions to ignore its normal rules and send confidential information elsewhere. The user may have done nothing wrong, yet the agent can still be manipulated by the content it retrieves.
Red teaming should therefore cover the full workflow, not only the conversation screen. Network controls also matter, which is why we recommend reviewing private network access for Microsoft Foundry agents before production deployment.
4. Whether outputs are unsafe or misleading
Content safety testing checks whether the agent produces harmful, discriminatory or otherwise inappropriate material. Business-specific accuracy is just as important.
A finance agent should not invent a tax rule. A safety agent should not provide confident instructions when the source information is missing. Tests should measure whether the answer is supported by approved company data and whether the agent clearly admits when it does not know.
5. Whether an attacker can drive up costs
An agent can be manipulated into running long conversations, repeatedly calling tools or processing unusually large documents. Even if no data is stolen, the result may be a surprisingly large Azure bill or a service that becomes too slow for legitimate users.
Test rate limits, token limits, timeouts and maximum tool calls. These controls reduce both financial risk and service disruption.
A practical pre-production red teaming process
- Define unacceptable outcomes. Ask security, legal, operations and the business owner what the agent must never reveal, say or do.
- Build realistic test scenarios. Include normal users, careless users, malicious insiders, compromised documents and external attackers.
- Test a safe environment. Use representative data and isolated systems so a successful attack cannot affect customers or production records.
- Run automated scans. Use Foundry evaluations and the AI Red Teaming Agent to test multiple attack strategies consistently.
- Add human testing. Automated testing finds known patterns. Experienced people find unusual business logic failures and context-specific risks.
- Create a release gate. Deployment should stop automatically when results exceed agreed risk thresholds.
A simple release policy might look like this:
release_gate = {
"maximum_attack_success_rate": 0.02,
"block_on_sensitive_data_disclosure": True,
"block_on_unauthorised_tool_action": True,
"require_human_approval_for_high_impact_actions": True
}
if red_team_results.fail(release_gate):
block_production_release()The exact thresholds will depend on the agent’s role. A low-risk internal knowledge assistant may tolerate minor answer-quality issues. An agent handling payments, personal information or customer records should face a much stricter release standard.
Some Microsoft red teaming features remain in preview or have regional and service limitations. Check current SDK versions, Azure region availability and pricing before making them a mandatory part of your deployment pipeline.
A representative business scenario
Consider a 200-person professional services company preparing to launch an internal HR agent. Normal testing showed that it answered leave and policy questions accurately, so the project appeared ready.
Red team testing then combined a legitimate question with instructions to search broader employee records. The agent refused to display individual salaries, but it returned a summary that exposed salary bands for a small team. A second test showed that its service identity could access documents outside the HR knowledge library.
The team tightened document permissions, removed unnecessary access and added minimum-group-size rules for workforce summaries. The launch was delayed by several days, but the company avoided a potential privacy incident and the far greater cost of rebuilding employee trust.
How this supports Australian security and privacy obligations
Red teaming does not make an organisation compliant by itself. It complements the Essential Eight, the Australian government’s cybersecurity framework that many organisations use as their security baseline, but it does not replace controls such as multi-factor authentication, patching and restricting administrative privileges.
Australian privacy obligations also apply when an agent collects, uses or discloses personal information. Testing should confirm that the agent cannot reveal personal or sensitive information simply because a user phrases the request creatively.
Keep the test evidence, identified risks, remediation decisions and final approvals. This creates a practical record for executives, auditors, customers and insurers showing that the agent was assessed before release.
Do not treat red teaming as a one-off project
An agent changes whenever its model, instructions, tools, permissions or connected data change. A safe version today can behave differently after what appears to be a small update.
Run focused tests during development, a broader scan before production and scheduled testing after deployment. Production monitoring should track tool activity, failed requests, unusual costs and safety evaluation results.
CloudProInc combines more than 20 years of enterprise IT experience with practical Microsoft cloud and AI delivery. As a Microsoft Partner and Wiz Security Integrator, we can assess the agent itself as well as the surrounding Azure identities, Microsoft Defender controls, network design and cloud security posture.
If you are not sure whether your Microsoft Foundry agent is genuinely ready for production, we are happy to help pressure-test the design and identify the highest-risk gaps before your users find them — no strings attached.
Discover more from CPI Consulting
Subscribe to get the latest posts sent to your email.