How to choose an AI agent security testing provider

Start with the decision the test needs to support

A release review, a customer request and an internal investigation need different evidence. Write down who will read the report and which decision they need to make. That gives a provider a concrete outcome to scope against.

List the agent’s intended actions, the systems it can reach and the identities it uses. Include the boundaries you rely on: approval prompts, tenant separation, network restrictions and tool permissions. A provider cannot meaningfully quote “test our agent” without these details.

Separate the service from the delivery platform

An assessment service may use automation, a platform, manual investigation or a combination. Ask who confirms the findings, who owns the scope and how you can discuss a disputed result. A dashboard alone does not tell you the depth of the engagement.

Our directory labels assessment services and hybrid offerings. These labels describe the public engagement model; they are not scores. Ask a hybrid provider which human review is included in the proposed price.

Request one example of the evidence

A useful finding explains the test identity, the input that triggered it, the tool or data path involved, the observed effect and the conditions required to reproduce it. Ask for a redacted sample report before you sign.

Ask how the provider distinguishes a strange answer from a security impact. For an agent that can change records or send messages, the meaningful result may be an unauthorized action rather than an offensive response.

Agree how a finding closes

Write retest terms into the proposal: timing, number of rounds, scenario repetitions and what happens when the model or tool implementation changes during the engagement. Reproducing an original scenario is different from running a generic scan again.

Compare proposals on the same test environment and authorized actions. If one proposal excludes authenticated workflows or connected tools, a lower quote may reflect a different engagement.

Use the shortlist as a conversation starter

Open the relevant provider profiles, read the sources and take the unanswered questions into your scoping call. Confirm availability, data handling, commercial terms and technical compatibility directly. Public positioning is the start of due diligence, not its conclusion.