Introduction to Observation
Observation gives you production-level visibility into how your agents behave. Evaluation checks quality before you deploy. Observation shows what happens after the agents go live: it records every conversation, has an LLM classify it, and shows the results on an analytics dashboard that you can view for all agents or for one agent.
Why Observation
Design-time tests cover the scenarios you thought of. Production brings the ones you did not: real users ask unexpected questions, data changes, and edge cases appear that no test suite anticipated. Without structured monitoring you cannot reliably detect quality regressions, spot systemic issues or measure improvement over time.
Reading conversations by hand does not scale. An agent that handles thousands of sessions a day cannot be monitored by sampling alone. Observation uses an LLM-as-a-judge approach: an analysis process reads each conversation, writes a summary and assigns structured tags (signals), so you can filter, chart and prioritize sessions without reading every one.
What Observation Collects
Observation captures each production session from three perspectives.
Operational
Every step of the agent's execution is recorded in the session's aifusion Data Product (LU) instance:
- LLM calls — the interface used, the messages sent, the responses and the latency of each call
- Agent executions — the agent and sub-agent executions in each turn, with their status and duration
- Tool calls — the tools invoked, their arguments, responses and duration
- Turns — each user question and assistant answer, with the turn duration and time to first token
You use this data for root-cause analysis when a conversation goes wrong, and to learn from conversations that went well.
Cost
Token usage is recorded for each LLM call — input, output, cache-read and cache-write tokens — together with the LLM interface that produced it. The analysis process sums it per session, so you can compare token consumption across agents and over time.
Behavior
The full conversation is stored and enriched with quality data:
- The complete message history (user and assistant turns)
- User feedback — thumbs up, thumbs down and free-text notes (see Collecting Feedback)
- Session metadata — session ID, agent name and its Data Product, the LU instances the session ran against, who submitted it, and timestamps
Signals: Automatic Conversation Tagging
When the analysis process runs, an LLM reads each conversation and picks a value for each signal from a fixed list of allowed values. The answers are stored as tags and drive the dashboard widgets, the conversation table and its filters.
Every agent is analyzed against three core signals:
On top of the core signals, each agent can define its own signals in its Observation/signals.yaml file — for example a customer-service agent can track Follow-up, Customer Lifetime Value Risk or Resolution Pathway. The analysis also counts the positive feedback, negative feedback and notes a session received and stores those counts as tags.
See Configuring Observation for how signals are defined.
Agent-Aware Observation
In AI Fusion 3, Observation is organized around agents:
- Every session records the agent that handled it and the agent's Data Product (LU), so the same agent name in two LUs is kept apart.
- The dashboard has an agent selector. All Agents shows the overall widgets and cross-agent comparisons; selecting one agent shows only that agent's own widgets and conversations.
- Each agent carries its Observation configuration next to its
agent.yaml, in<LU>/AI/Agents/<agent>/Observation/:signals.yaml(what the analyzer tags) andwidgets.yaml(what the dashboard shows for that agent).
Session recording is controlled by the agent's observability setting in agent.yaml (default true). See agent.yaml Reference.
How Analysis Runs
Analysis is not triggered automatically at the end of a conversation, and AI Fusion does not schedule it for you. You run it on demand or schedule it yourself:
RunAnalyze— a Broadway flow that analyzes the open sessions of the last N days, one at a time. Run it withbroadway aifusion.RunAnalyze day=7, or call therun-analyticsweb service (GET /api/run-analytics?day=7).ObservationAnalyzerandBatchObservationAnalyzer— pipelines that analyze sessions in bulk. The batch pipeline submits the work through the LLM provider's batch API, which is cheaper for large volumes.
Until a session is analyzed it appears on the dashboard without a summary, topics or resolution status. When a user continues an analyzed conversation, the session is re-opened and is analyzed again on the next run.
See Configuring Observation for the details, and for the cleanup process that removes old sessions.
Where the Data Lives
See Insights Database Schema Reference for the tables and columns.
Opening Observation
Observation is the Observation tab of the AI Data Fusion web application, registered in apps.json of the Web Services LU (k2_ws/web/apps.json) and enabled by default. Open it at:
https://<server>[/<tenant>]/static/aifusion/observation
The tenant segment is present only on tenant-based deployments. Add ?agent=<lu>/<agent> to open the dashboard scoped to one agent, for example .../static/aifusion/observation?agent=customer_bank/banking.
Next article: Observation Dashboard
Introduction to Observation
Observation gives you production-level visibility into how your agents behave. Evaluation checks quality before you deploy. Observation shows what happens after the agents go live: it records every conversation, has an LLM classify it, and shows the results on an analytics dashboard that you can view for all agents or for one agent.
Why Observation
Design-time tests cover the scenarios you thought of. Production brings the ones you did not: real users ask unexpected questions, data changes, and edge cases appear that no test suite anticipated. Without structured monitoring you cannot reliably detect quality regressions, spot systemic issues or measure improvement over time.
Reading conversations by hand does not scale. An agent that handles thousands of sessions a day cannot be monitored by sampling alone. Observation uses an LLM-as-a-judge approach: an analysis process reads each conversation, writes a summary and assigns structured tags (signals), so you can filter, chart and prioritize sessions without reading every one.
What Observation Collects
Observation captures each production session from three perspectives.
Operational
Every step of the agent's execution is recorded in the session's aifusion Data Product (LU) instance:
- LLM calls — the interface used, the messages sent, the responses and the latency of each call
- Agent executions — the agent and sub-agent executions in each turn, with their status and duration
- Tool calls — the tools invoked, their arguments, responses and duration
- Turns — each user question and assistant answer, with the turn duration and time to first token
You use this data for root-cause analysis when a conversation goes wrong, and to learn from conversations that went well.
Cost
Token usage is recorded for each LLM call — input, output, cache-read and cache-write tokens — together with the LLM interface that produced it. The analysis process sums it per session, so you can compare token consumption across agents and over time.
Behavior
The full conversation is stored and enriched with quality data:
- The complete message history (user and assistant turns)
- User feedback — thumbs up, thumbs down and free-text notes (see Collecting Feedback)
- Session metadata — session ID, agent name and its Data Product, the LU instances the session ran against, who submitted it, and timestamps
Signals: Automatic Conversation Tagging
When the analysis process runs, an LLM reads each conversation and picks a value for each signal from a fixed list of allowed values. The answers are stored as tags and drive the dashboard widgets, the conversation table and its filters.
Every agent is analyzed against three core signals:
On top of the core signals, each agent can define its own signals in its Observation/signals.yaml file — for example a customer-service agent can track Follow-up, Customer Lifetime Value Risk or Resolution Pathway. The analysis also counts the positive feedback, negative feedback and notes a session received and stores those counts as tags.
See Configuring Observation for how signals are defined.
Agent-Aware Observation
In AI Fusion 3, Observation is organized around agents:
- Every session records the agent that handled it and the agent's Data Product (LU), so the same agent name in two LUs is kept apart.
- The dashboard has an agent selector. All Agents shows the overall widgets and cross-agent comparisons; selecting one agent shows only that agent's own widgets and conversations.
- Each agent carries its Observation configuration next to its
agent.yaml, in<LU>/AI/Agents/<agent>/Observation/:signals.yaml(what the analyzer tags) andwidgets.yaml(what the dashboard shows for that agent).
Session recording is controlled by the agent's observability setting in agent.yaml (default true). See agent.yaml Reference.
How Analysis Runs
Analysis is not triggered automatically at the end of a conversation, and AI Fusion does not schedule it for you. You run it on demand or schedule it yourself:
RunAnalyze— a Broadway flow that analyzes the open sessions of the last N days, one at a time. Run it withbroadway aifusion.RunAnalyze day=7, or call therun-analyticsweb service (GET /api/run-analytics?day=7).ObservationAnalyzerandBatchObservationAnalyzer— pipelines that analyze sessions in bulk. The batch pipeline submits the work through the LLM provider's batch API, which is cheaper for large volumes.
Until a session is analyzed it appears on the dashboard without a summary, topics or resolution status. When a user continues an analyzed conversation, the session is re-opened and is analyzed again on the next run.
See Configuring Observation for the details, and for the cleanup process that removes old sessions.
Where the Data Lives
See Insights Database Schema Reference for the tables and columns.
Opening Observation
Observation is the Observation tab of the AI Data Fusion web application, registered in apps.json of the Web Services LU (k2_ws/web/apps.json) and enabled by default. Open it at:
https://<server>[/<tenant>]/static/aifusion/observation
The tenant segment is present only on tenant-based deployments. Add ?agent=<lu>/<agent> to open the dashboard scoped to one agent, for example .../static/aifusion/observation?agent=customer_bank/banking.
Next article: Observation Dashboard
