How to evaluate an AI agent platform.

19 questions to ask every vendor, why each one matters, and a scorecard to compare the answers side by side.

Deployment and data residency

Runs in your environment
Can the whole platform run in our cloud account or data center, not only the model?
Why it matters: Regulators and security teams ask where prompts, documents and traces live. A managed-only platform makes that answer someone else's.
Isolated operation
Can it run with no outside access, and who sets that up?
Why it matters: Some workloads can never reach the internet. Find out early whether that is a product capability or a custom project.
Data location
Where are prompts, retrieved documents, traces and credentials stored, and for how long?
Why it matters: Retention and residency rules apply to agent traces as much as to source data.

Certifications you can verify
Which attestations do you hold, and will you share the reports under NDA?
Why it matters: A logo on a web page is not evidence. Ask for the report and the scope it covers.
Identity and access
Do you support SSO with our identity provider, projects and role-based access?
Why it matters: Agents act on real systems. Access to build and deploy them should follow your identity controls.
Secrets handling
How are credentials for connected systems stored, rotated and revoked?
Why it matters: An agent platform holds keys to your CRM, core systems and data stores.

Human approval
Can any step wait for a person to approve, and can the reviewer edit the values first?
Why it matters: Most regulated workflows need a person to sign off on the steps that move money, data or decisions.
Input screening
Are PII and prompt injection detected before data reaches a model?
Why it matters: Screening at the platform level beats relying on every agent builder to remember.
Traceability
Is every run recorded step by step, with inputs, outputs, cost and latency, and can we replay it?
Why it matters: When an auditor or customer asks why an agent did something, you need the full record.

Evals before launch
Can we score agents against our own datasets and metrics before they go live?
Why it matters: Success criteria agreed up front are the difference between a demo and a production agent.
Evals after launch
Are live deployments evaluated continuously, and can production traces become test cases?
Why it matters: Models, data and prompts drift. Quality has to be measured every day, not once.

Model choice
Which model providers are supported, can we run open models ourselves, and can we switch without rewriting agents?
Why it matters: The best model for a task changes every few months. Lock-in at the model layer is expensive.
Connections to your systems
How does an agent reach our systems: prebuilt connectors, APIs, databases, MCP servers?
Why it matters: An agent that cannot act in your systems is a chatbot.
Per-user access
Can a deployed agent act with each end user's own permissions?
Why it matters: Retrieval and actions should respect the permissions people already have.

Two ways to build
Can business teams build visually and engineers build in code, on the same runtime?
Why it matters: Separate tools for each audience mean two platforms to govern.
Openness
Is the core open source, and under which license?
Why it matters: An open core lowers switching costs and lets your engineers read what runs in production.
Delivery help
Will your engineers work with ours to reach production, and who owns what they build?
Why it matters: The first agent is the hardest. Make sure the result, including the code, is yours.

How you buy
Can we buy through our cloud marketplace, and how is usage priced?
Why it matters: Marketplace procurement can shorten purchasing and count toward existing cloud commitments.
Exit
If we leave, what do we take with us: agents, prompts, data, traces?
Why it matters: A clean exit path is the best test of how much you depend on a vendor.

Book a working session and bring your checklist. Our engineers will answer on your use case.