Direct answer
Test the complete workflow with representative normal, edge, misuse, and adversarial cases. Define acceptance thresholds before reviewing results, verify data and permissions, exercise human escalation and rollback, inspect logs, and release gradually only when remaining risk has an accountable owner.
Pre-launch gate
Require evidence before the system reaches customers
Launch status
0 / 6Evidence is still missing
A demo is not a release decision. Close every gate or document who accepts the remaining risk.
Before testing
Write the acceptance standard before seeing the results
- Intended users, context, and business outcome
- Included and prohibited tasks
- Required quality by failure category
- Information and privacy boundaries
- Allowed actions and approval requirements
- Latency, cost, and availability expectations
- Escalation, fallback, and recovery time
- Who can approve, limit, or stop release
Representative evidence
Build a test set that resembles deployment
| Case group | Purpose | Examples |
|---|---|---|
| Normal | Measure expected daily performance | Common inputs across users, products, and time periods |
| Edge | Find valid but unusual conditions | Incomplete records, conflicting sources, rare categories |
| Known failure | Verify detection and recovery | Past errors, ambiguous requests, stale information |
| Misuse | Check prohibited or unintended behavior | Requests outside scope, attempts to bypass rules |
| Adversarial | Probe security and manipulation | Prompt injection, malicious documents, data extraction attempts |
| Operational | Test the surrounding system | Timeouts, duplicates, unavailable integrations, permission failure |
Evaluation layers
Test more than the model response
Output quality
Accuracy, completeness, grounding, consistency, tone, and usefulness for the job.
Workflow quality
Correct trigger, retrieval, rules, routing, approval, write-back, and exception handling.
Security and privacy
Access, secrets, data minimization, leakage, injection resistance, logs, and retention.
Human oversight
Reviewer context, skill, workload, authority, and ability to detect meaningful errors.
Operations
Latency, cost, availability, monitoring, fallback, rollback, incident response, and change ownership.
Thresholds
Measure errors by consequence, not only one average score
A single accuracy percentage can hide a rare but unacceptable failure. Separate failure types and weight them by impact. A system may tolerate a formatting error while requiring zero release of restricted information.
| Metric | Useful question | Control |
|---|---|---|
| Task success | Did the output complete the intended job? | Minimum pass rate on representative cases |
| Critical failure | Did any prohibited event occur? | Zero-tolerance gate or executive risk acceptance |
| Grounding | Can claims be traced to approved sources? | Citation or source verification |
| Escalation | Were uncertain cases routed correctly? | Recall for high-risk exceptions |
| Review effectiveness | Did people catch meaningful errors? | Blind comparison and override analysis |
| Operational reliability | Did the full workflow complete safely? | Retries, duplicate rate, latency, and fallback |
Controlled release
Increase exposure only when evidence stays stable
Internal sandbox
Use protected test data and no production actions.
Shadow mode
Run on real work without affecting the official outcome.
Limited users
Release to trained users with prior approval and close support.
Bounded production
Allow only approved cases, monitor continuously, and sample outputs.
Expand or reverse
Increase scope only after thresholds hold, or roll back when they do not.
The value point
After this page, you should be able to decide:
Whether evidence supports a controlled release, what must be fixed, and who accepts any residual risk.Your working output should be a launch gate, representative test set, failure taxonomy, threshold table, and rollback drill.
Questions business leaders ask
Frequently asked questions
How many test cases does an AI system need?+
There is no universal number. Use enough representative cases to cover normal variation and each material failure category, then continue monitoring because pre-launch tests cannot capture every production condition.
What is red teaming for AI?+
Red teaming intentionally probes how an AI system can be manipulated, misused, or caused to fail. It complements normal quality and workflow testing.
Can vendor benchmarks replace our own evaluation?+
No. Vendor results may describe general capability, but your workflow, source data, users, integrations, risks, and acceptance thresholds require context-specific testing.
What should trigger a rollback?+
Define conditions before release, including critical data exposure, harmful output, broken approvals, repeated integration failure, unacceptable error rates, or loss of monitoring.
Research anchors
Primary and authoritative sources
Examples and planning ranges are clearly labeled. Source terms, provider behavior, and regulations can change; verify current requirements for your organization and jurisdiction.
Prepared and reviewed by the Future Made Useful systems editorial team. Material guidance reviewed July 17, 2026.