Skip to content

Eventhouse (KQL)

The Eventhouse is the storage and query spine. Three .kql files in fabric/kql/ define it, run in order in the KQL database query editor (or via Deploy-FdaObservability.ps1 -ProvisionSchema).

File Run order Contents
01_tables.kql first Landing tables (Raw_*), curated FdaInteractions, Watermarks
02_policies.kql second Retention + ingestion-batching policies, app-facing stored functions, KQL-native correlation
03_queries.kql ad-hoc Analyst / debugging queries — paste as needed, not part of provisioning

01_tables.kql — schema

Creates all tables with .create-merge (idempotent: re-running adds any missing columns without dropping data). Full column-by-column documentation is in the data model. In brief:

  • Raw_GraphInteractions — source A, prompt/response text per message.
  • Raw_AuditInteractions — source B, CopilotInteraction metadata (RecordType 261).
  • Raw_ExecutedDax — source C, executed DAX + perf from workspace monitoring.
  • FdaInteractions — the curated triple, one row per interaction.
  • Raw_SdkRuns — optional source D, SDK replay reasoning/steps.
  • Watermarks — per-source incremental state.

02_policies.kql — policies & functions

Retention (PII-aware)

Raw payloads carry sensitive text, so they expire sooner than the curated triple:

.alter-merge table Raw_GraphInteractions policy retention softdelete = 90d
.alter-merge table Raw_AuditInteractions  policy retention softdelete = 90d
.alter-merge table Raw_ExecutedDax        policy retention softdelete = 90d
.alter-merge table Raw_SdkRuns            policy retention softdelete = 365d
.alter-merge table FdaInteractions        policy retention softdelete = 730d

Ingestion batching (low latency)

Raw_GraphInteractions, Raw_AuditInteractions, Raw_ExecutedDax, and FdaInteractions use a 30-second / 500-item batching policy so review is near-real-time:

.alter table FdaInteractions policy ingestionbatching
    '{"MaximumBatchingTimeSpan":"00:00:30","MaximumNumberOfItems":500}'

Stored functions

The app-facing functions (folder FDA) and the internal correlation function (folder FDA/internal) are defined here and documented in detail on the KQL functions page:

Function Used by Purpose
SearchFdaInteractions(...) review app Filtered search/list of curated interactions
GetFdaInteraction(id) review app Single interaction detail
FdaTuningSummary(from,to) dashboards Volume, failure rate, latency, unmatched rate
FdaFailures(from,to) review/ops Failures & low-confidence rows worth attention
FdaTopQuestions(from,to,topN) dashboard Most frequent question patterns (tuning candidates)
BuildFdaInteractionsKql(lookback,windowSec) validation KQL-native reproduction of the collector's correlation

Why functions, not raw queries

The review app calls these stored functions rather than embedding KQL. That keeps query logic server-side and versioned with the schema, and lets you change correlation/filter behaviour without redeploying the app.

03_queries.kql — analyst queries

A pack of ready-to-paste queries for pipeline health, source coverage, slow-DAX tuning, failing questions, raw DAX inspection, SDK-vs-production comparison, and per-user correlation spot-checks. These are reproduced and explained on the Analyst queries page.

Hosting choice

You may host the observability tables in the same Eventhouse as workspace monitoring or in a dedicated one. A dedicated observability Eventhouse is recommended so retention and permissions are independent of the monitoring store. See Tenant & identity → Provision the Eventhouse.