Skip to content

FDA Observability

Production observability for a Microsoft Fabric Data Agent (FDA) that runs NL2DAX over a Power BI semantic model and is surfaced to end users through Microsoft 365 Copilot.

It captures, per interaction, as much of the troubleshooting/tuning triple as the platform exposes:

The triple

question → (rephrase / grounding) → generated DAX → executed DAX (+ perf) → answer → user → timestamp

and lands it in a Fabric Eventhouse (KQL), with a C# WinForms review/search/config app and a Real-Time Dashboard on top.

Why this exists

No single Microsoft API returns the whole triple for live M365-originated calls. The prompt/answer text, the audit metadata, and the executed DAX each live on a different surface — and the model's internal reasoning is not exposed at all for live traffic. This project captures from three surfaces in parallel and correlates them by user + time-window + workspace/model into one curated row per interaction.

See the capture matrix for the honest, surface-by-surface breakdown of what is and is not programmatically available.

How it fits together

flowchart TD
    user([End user]) --> copilot[Microsoft 365 Copilot]
    copilot --> fda[Fabric Data Agent<br/>NL2DAX tool]
    fda --> model[(Power BI<br/>semantic model)]

    subgraph capture [Capture surfaces]
        A[A · MS Graph<br/>aiInteractionHistory<br/>prompt + response text]
        B[B · Office 365 Mgmt API<br/>CopilotInteraction<br/>metadata / index]
        C[C · Workspace monitoring<br/>SemanticModelLogs<br/>executed DAX + perf]
    end

    copilot -.-> A
    copilot -.-> B
    model -.-> C

    A --> collector[FDA_Collector<br/>scheduled Fabric notebook]
    B --> collector
    C --> collector

    collector --> eh[(Eventhouse / KQL<br/>Raw_* → FdaInteractions)]
    eh --> app[WinForms review app<br/>per-interaction drill-down]
    eh --> dash[Real-Time Dashboard<br/>fleet health & trends]

The full data flow, including correlation, is described in Architecture.

Components at a glance

Path What it is Docs
fabric/kql/ Eventhouse schema: raw landing tables, curated FdaInteractions, update policies, app functions, analyst queries Eventhouse · KQL functions
fabric/notebooks/FDA_Collector.py Scheduled Fabric notebook (service principal): pulls all sources, correlates, ingests, advances watermarks Collector
fabric/notebooks/FDA_SDK_Replay.py Optional offline harness for full reasoning/step capture on sampled questions SDK replay
fabric/dashboards/ FDA_Observability_RTI_Dashboard.json — Real-Time Dashboard (fleet health & tuning trends) + query pack Dashboard
deploy/ Deploy-FdaObservability.ps1 — provisions the Eventhouse schema + schedules the collector Deploy
src/FdaObservability.App .NET 8 WinForms review + search + configuration app (interactive AAD) Review app
src/FdaObservability.{Core,Api,Mcp,Cli} Shared query core + REST API (OpenAPI), MCP server, and fda-obs CLI — surface the data to any tool/interface/harness Programmatic access
docs/ This documentation site —

The WinForms app is for per-interaction drill-down; the Real-Time Dashboard is for fleet-level trends. Both read the same curated FdaInteractions table.

Quick start

  1. Tenant + identity — enable settings, register the collector app, grant permissions, provision the Eventhouse.
  2. Eventhouse schema — run 01_tables.kql then 02_policies.kql, or use Deploy-FdaObservability.ps1 -ProvisionSchema.
  3. Collector — import FDA_Collector.py, set parameters, run once, then schedule (every 15–60 min).
  4. Review app — open src/FdaObservability.sln, build, run, complete the in-app Configuration dialog.
  5. Dashboard — import the Real-Time Dashboard JSON and point it at the Eventhouse.

See the full deployment guide for the end-to-end order and verification.

Prerequisites (summary)

  • Fabric capacity F2+ (or P1+ with Fabric enabled); the FDA published over a Power BI semantic model.
  • Workspace monitoring enabled on the workspace hosting the semantic model (creates the monitoring Eventhouse).
  • An Entra collector app registration with the required Graph / Management API permissions and Eventhouse ingest rights (see Tenant & identity).
  • M365 Copilot licensing for the users whose interactions you export via Graph.
  • .NET 8 SDK + Windows for the WinForms app.

Preview surfaces degrade gracefully

Several capture paths are in preview (Purview DSPM for AI audit for FDA, FDA SDK). The collector degrades gracefully: any source that is unavailable is skipped and logged, not fatal. The GA spine is Office 365 Management API + workspace monitoring.

License

Released under the MIT License.