Data Snapshots and Production Import

An agent's answers depend on the data it reads. A test written for a customer's data today may fail next month — not because the agent got worse, but because the customer paid a bill or closed an account. Or it may pass for the wrong reason. Snapshots solve this by freezing the data a test case runs on. Production import brings real users' conversations, with their data, into your test suites.

What Is Captured with a Test Case

When a test case is saved (with Upload Artifacts in the Eval Editor, or Save as test case in Preview), AI Fusion stores two kinds of artifacts under the case's session:

The business snapshot is a point-in-time copy of the entity's micro-database — the unified data the LU assembled from the source systems — not of the source systems themselves.

Save with Snap can only be chosen when a case is first saved. Cases generated by Studio AI are always saved with a snapshot.

Why Use Snapshots

• Repeatable results. Every run uses the same data, so a change in score means a change in the agent, not in the data.
• Portable tests. A case runs the same way in development, QA and production environments.
• Independence from source systems. A case runs even if the source systems are unavailable or the customer no longer exists.

Making an Agent Snapshot-Ready

Snapshots are taken where the agent loads its entity. For an entity-scoped agent, that is the step_start hook flow, which brings the customer's instance into scope (see Hooks and Inputs).

Load the instance in that flow with the GetWithSnap actor instead of a plain GET. GetWithSnap works like a GET and adds the snapshot behavior:

• In Studio, it captures a snapshot of the instance on the first turn of the conversation, so any conversation can later be saved as a test case with its data.
• During an evaluation run, it does not read live data. It imports the snapshot recorded for that instance into an isolated copy and reads the copy, so the replay cannot change the baseline or the real instance.
• It records which LU instances the session touched, which is what Observation export uses to include the data.

Set the actor's snapStepsMode input explicitly to none (Studio still captures the first turn). Leaving it empty makes the hook fail on the first turn of every session.

GetInstance:
  parent: GetWithSnap
  in:
    luType:
      const: Customer
    syncMode:
      const: 'ON'
    snapStepsMode:
      const: none
    iid:
      link: RootIid/value

An agent that loads its data with a plain GET still runs in evaluations, but on live data — its cases are not repeatable.

Where Snapshots Are Stored

Both are generic interfaces of the AI Fusion project, and the retention periods are AI Fusion globals. By default evaluation_storage is a local folder on the Fabric server. If suites are run in more than one environment, point evaluation_storage to storage that all of them can read.

Snapshots referenced by a suite are never removed by the cleanup, so deleting a case or suite is what eventually frees its snapshots.

Importing Production Conversations

The conversations real users have with your deployed agent show the questions, phrasings and edge cases you did not think of. Turn the problematic ones into test cases so the same problem cannot come back unnoticed.

1. Find the conversation. In Observation, open the conversation in the conversation review — for example, one with negative feedback or a low signal score. See Conversation Review.
2. Export it. Click the export button, select Export with snap to include snapshots of the LU instances the conversation used, and click Export. A ZIP file is downloaded.
3. Import it in Studio. Open the agent's Preview chat and click Import. Select the ZIP file. The conversation opens as a new chat titled "Imported chat", and its snapshots are stored under a new session.
4. Review it. Read the conversation. Check that the initialization parameters of the imported chat are the ones the agent expects.
5. Save it as a test case. Click Save as test case, choose the suite, and save.
6. Correct the expected answers. Open the case in the Eval Editor, click Edit, and change any answer that was wrong in production to the answer the agent should have given. Click Upload Artifacts and save the suite.

The case now fails until the agent is fixed — and protects the fix afterwards.

Sharing Conversations Between Environments

The same ZIP format is used inside Studio. In Preview, click Share (select Save with Snap to include the business data) to download a conversation, and Import to load one. Use it to hand a scenario to a colleague in another environment, or to reproduce a reported issue in development on exactly the same data.

Next article: Evaluation Best Practices