AI Fusion Evaluation lets you test an agent the way a customer uses it: you replay realistic conversations against the agent, and an LLM judge checks the result — answer by answer against the answers you expect, over the whole conversation against criteria you write, or both. The results are checked against pass/fail rules, and optional Broadway flows can run before and after each test to prepare data, verify what the agent did and clean up. Run the same tests after every prompt, model, tool or data change and you know immediately whether quality went up or down.
Note: The examples and screenshots in this section use the banking demo (
customer_bank›banking). See About the Examples.
In AI Fusion 3, evaluation is part of AI Fusion Studio — there is no separate evaluation application. Everything happens in Studio, next to the agent you are building:
The suites are files in the agent's folder inside the Data Product (LU), so they are versioned and deployed with the agent.
Traditional tests assume that the same input always produces the same output. Agents do not work that way:
Two common approaches fail here:
AI Fusion Evaluation addresses this by judging meaning and quality, not wording, by judging whole conversations against goals as well as single answers, and by pinning the data each test runs on.

| Component | Role |
| Eval suite files | <LU>/AI/Agents/<agent>/Evals/<suite>.csv — one row per test case, deployed with the LU. |
| AI Fusion Studio | The AI Explorer tree lists suites and cases with their last run status; the Eval Editor is where you author cases, run them and read results. See AI Fusion Studio Overview. |
| Pipeline extension | The engine behind Run Evaluation (the aifusion.AgentEvaluator pipeline). You do not open it; it only needs to be installed and deployed. See Installation and Setup. |
| Evaluation web services | Optional: start runs and read results from CI/CD or any job, without opening Studio. |
| Insights DB | Stores the evaluation conversations, the per-answer ratings, the conversation-level verdicts and the hook results. |
| Evaluation storage | Stores the data snapshots that test cases replay on. |
Evaluation was redesigned in AI Fusion 3. The main differences:
Evals folder, not MTables. A case stores the agent's initialization parameters instead of a customer ID and application.aifusion.AgentEvaluator pipeline and started from Studio or the evaluation web service. The 2.x aifusion.evaluatorPipeline is not used.The 2.x articles remain available in the legacy documentation.
Next article: Authoring Test Cases
AI Fusion Evaluation lets you test an agent the way a customer uses it: you replay realistic conversations against the agent, and an LLM judge checks the result — answer by answer against the answers you expect, over the whole conversation against criteria you write, or both. The results are checked against pass/fail rules, and optional Broadway flows can run before and after each test to prepare data, verify what the agent did and clean up. Run the same tests after every prompt, model, tool or data change and you know immediately whether quality went up or down.
Note: The examples and screenshots in this section use the banking demo (
customer_bank›banking). See About the Examples.
In AI Fusion 3, evaluation is part of AI Fusion Studio — there is no separate evaluation application. Everything happens in Studio, next to the agent you are building:
The suites are files in the agent's folder inside the Data Product (LU), so they are versioned and deployed with the agent.
Traditional tests assume that the same input always produces the same output. Agents do not work that way:
Two common approaches fail here:
AI Fusion Evaluation addresses this by judging meaning and quality, not wording, by judging whole conversations against goals as well as single answers, and by pinning the data each test runs on.

| Component | Role |
| Eval suite files | <LU>/AI/Agents/<agent>/Evals/<suite>.csv — one row per test case, deployed with the LU. |
| AI Fusion Studio | The AI Explorer tree lists suites and cases with their last run status; the Eval Editor is where you author cases, run them and read results. See AI Fusion Studio Overview. |
| Pipeline extension | The engine behind Run Evaluation (the aifusion.AgentEvaluator pipeline). You do not open it; it only needs to be installed and deployed. See Installation and Setup. |
| Evaluation web services | Optional: start runs and read results from CI/CD or any job, without opening Studio. |
| Insights DB | Stores the evaluation conversations, the per-answer ratings, the conversation-level verdicts and the hook results. |
| Evaluation storage | Stores the data snapshots that test cases replay on. |
Evaluation was redesigned in AI Fusion 3. The main differences:
Evals folder, not MTables. A case stores the agent's initialization parameters instead of a customer ID and application.aifusion.AgentEvaluator pipeline and started from Studio or the evaluation web service. The 2.x aifusion.evaluatorPipeline is not used.The 2.x articles remain available in the legacy documentation.
Next article: Authoring Test Cases