AI Diagnostics

How AI Analysis Works

When you open an Automated Troubleshooting session and describe what's wrong, ET Ducky runs a multi-step investigation rather than a single AI call. The agent gathers evidence from the host in rounds, the AI decides what to look at next based on what each round surfaces, and the conclusion (along with any proposed remediation) lands back in the session for your review.

You describe the issue in natural language (or attach a ticket).
The cloud orchestrator starts an AI-driven dynamic ETW investigation against the target agent (see below).
The AI picks ETW providers from the host's actual catalog, enables them on a separate diagnostic session, captures briefly, and reads the results.
If the first round isn't conclusive, the AI adjusts — enables more providers, drops noisy ones, narrows keyword masks — and runs another round.
Once the AI has enough evidence, it produces a root cause + recommendations writeup. In live sessions where remediation is well-defined, it also proposes a remediation script that goes through the org's existing automations approval gate before any execution.

Example Queries

“Why is the SQL Server service failing to start?”
“What processes are accessing C:\ProgramData?”
“Show me all failed authentication attempts in the last hour”
“Why is the system slow right now?”
“What changed before the application started crashing?”
“DNS resolution is intermittent on this box, find out why”
“Logon stalls at ‘preparing windows’ for 90 seconds, what's responsible?”

Dynamic ETW Diagnostic Sessions

Every Windows host registers between roughly 1,500 and 2,500 ETW providers. The traditional approach (collect everything, analyse after) is expensive; the manual approach (operator picks providers) is impractical for non-experts. ET Ducky's AI orchestrates the session layer instead: it picks providers for each round based on what the captured evidence suggests is worth looking at next.

How it works on the host

The agent enumerates its full ETW provider catalog at startup and uploads a tier-filtered inventory to the cloud: ~50 well-known "Tier 1" providers (kernel, networking, .NET, security baseline) plus any providers marked recently-active become the AI's working set. The long tail is reachable but not shown by default.
When an investigation begins, the agent spins up a separate diagnostic ETW session with a unique name — never the always-on baseline session. The two sessions share no state.
The AI's per-round plan (which providers to enable, which to disable, how long to capture) is applied to the live diagnostic session. The agent counts events into a per-(provider, event id, task, opcode) tally; only the counts cross back to the cloud.
Before tear-down, a local deterministic rule engine runs over a bounded sample of the round's events. When it recognises a known failure pattern (Azure AD Conditional Access denials, TLS untrusted-root chains, Kerberos SPN-unknown errors, etc.) it emits a verdict — severity plus pre-authored explanation and suggestion text, no raw event payloads — that ships alongside the counts summary. The AI planner sees these verdicts as authoritative findings; when one explains the symptom the investigation can conclude immediately instead of running additional rounds.
When the investigation finishes, the diagnostic session is torn down. Raw event buffers are purged. The conclusion travels; the evidence stays on the endpoint.

What the AI sees per round

The user's reported issue
What's currently enabled on the diagnostic session
The previous round's counts-only snapshot (top buckets by event count)
Any verdicts the agent's local rule engine emitted over the previous round's events
Remaining budget headroom (how many more providers the session can take)
The host's tier-filtered provider catalog

The AI returns a structured plan: enables, disables, capture seconds, and a "should we conclude?" flag with confidence. The loop applies, captures, snapshots, re-plans — up to a default 5 rounds (hard ceiling 10) with each round capturing for 10–120 seconds.

Deterministic Verdicts & Connection Stories

ET Ducky's AI doesn't read a wall of raw events from scratch for every question. Two pre-processing layers run between the captured events and the model so the AI sees structured, semantically dense input — and so the answer is deterministic for the patterns we can already recognise.

Pre-classified verdicts

A local rule engine scans the captured events against a curated set of known failure patterns: Azure AD Conditional Access denials (AADSTS error codes), TLS handshake failures (untrusted root CAs, unknown_ca alerts, protocol-version mismatches), Kerberos errors (SPN unknown, pre-authentication required), DNS failures (NXDOMAIN, SERVFAIL), HTTP-layer failures (401, 403 with service-specific diagnostics), and more.

When a rule matches, the engine emits a verdict: a severity, a short title, a 1–3 sentence explanation, and an actionable next step. Verdicts appear at the top of the AI's prompt as "pre-classified findings"; the AI treats them as authoritative ground truth and writes a confident, specific answer instead of inferring from raw events and hedging.

The rule engine is deterministic and cheap. For the common patterns it covers, an answer that previously required several AI inference passes now lands with a single tight synthesis call.

Connection stories

For network-shaped questions, captured events are also grouped into per-destination connection stories: one row per (process, hostname) pair, with the chronological chain of DNS → TCP → TLS → HTTP → auth events for that one logical operation. A story collapses what used to be six to eight raw events spread across the sample into a single dense narrative the AI can read in one pass. Network destinations with mixed outcomes (some steps OK, some failed) sort first so the AI's attention lands on what's interesting.

Where this applies

Both layers run inside the unified query engine that powers Ducky Chat (the desktop "ask the AI about my system" path) and the cloud orchestrator's hypothesis loop (the dashboard's Automated Troubleshooting investigation). The same rule registry serves both surfaces; a verdict that fires for a Ducky Chat question fires the same way for a live investigation, so the diagnoses are consistent across single-machine and fleet contexts.

Budget & Safety Caps

The AI is constrained by the engine, not trusted to behave. Caps live inside the dynamic capture engine; the AI can't bypass them. A request that would exceed a cap is refused with a structured error and the AI re-plans.

Cap	Default
Concurrent providers per investigation	16
Aggregate events/sec per session	25,000/s
Aggregate events/sec across all diagnostic sessions on the host	50,000/s
In-flight buffer memory per session	64 MiB
Hypothesis-loop rounds per investigation	5 default / 10 hard ceiling
Per-round capture window	10–120 seconds
Concurrent investigations per host	4

If observed throughput exceeds a cap mid-capture (the per-provider estimate was wrong, the host is busier than expected), the engine triggers emergency scale-back: the most-recently-enabled provider is automatically disabled and the AI is told so it can re-plan rather than retry. Investigations that go 15 minutes without a command from the cloud are reaped by an on-agent sweeper.

Remediation Script Handoff

When an investigation concludes with a well-defined fix, the AI generates a remediation script (PowerShell on Windows, Bash on Linux) and submits it to the org's Process Automations approval flow. Mode-dependent behavior:

Live Automated Troubleshooting session, operator attending: the script is created with auto_run_eligible = true. After org-admin approval, the operator can run it immediately in-session through the existing automations execution path (which itself enforces command security policy).
Background / scheduled / dashboard-initiated investigation, no operator: the script is created with auto_run_eligible = false and stays as analysis-only until a human picks it up.

The product invariant — ET Ducky does not take unauthorised actions on customer hosts — holds for AI-generated scripts the same way it holds for human-written ones. The approval gate, per-command security policy, and org-admin role check apply equally. AI-generated remediations are tagged with Origin = "process_automation" and a CreatedBy = "dynamic-etw:<jobId>" marker so the audit trail makes the source unambiguous.

Query Pool System

Tier	Monthly Queries	Cost
BYOK	Unlimited	Free (your provider key: Anthropic, OpenAI, or Azure OpenAI)
Professional	1,000	$39/mo
Business	5,000	$99/mo
Enterprise	50,000	$249/mo

What Consumes Queries

Live Session Question: 1 query, plus one additional query per round of the dynamic ETW investigation that fires for that question (default cap 5 rounds, hard ceiling 10).
Multi-Agent Question: 1 per agent per question
Cross-Correlation: 1 additional per device
Alert AI Analysis: 1 per alert (cached 24 hours)
Remediation Script Generation: 1 query when the investigation concludes with a proposed fix

Shared across all org members. Unused queries do not roll over.

Inflection Root-Cause Analysis (RCA)

Every agent’s heartbeat is scanned for inflections — sudden metric movements like CPU spikes, memory drops, or disk-time surges. The dashboard surfaces unacknowledged inflections as Anomalies on each agent row. Clicking View anomalies opens a per-agent drawer where you can select one or more anomalies and click Analyze (RCA) to launch a focused AI investigation.

Baselines

From agent 2.8.5.2, each agent keeps a running average and spread of CPU, memory use, disk queue, disk write rate, network receive and send rates and active connections. When a sample is at least three standard deviations from that agent's own average, and at least a set distance from it (15 points of CPU, 10 points of memory, a queue of 2, 5 MB per second, 200 connections), the agent reports a *_baseline_deviation inflection, such as cpu_baseline_deviation. Six standard deviations or more is critical. The fixed-threshold inflections keep firing as before; when a baseline exists, they carry its average, spread and the sample's z-score as context.

A baseline needs 30 samples before it reports, which is 30 minutes at the default one-minute metrics interval. The agent keeps its baselines across restarts in its private state folder and drops any older than 7 days. Turn the rules off with AnomalyDetection.BaselineEnabled set to false.

What RCA does

Brackets the timeline. The cloud takes the min/max timestamps of the selected inflections and pads ±60 seconds, then queries every captured event in that window from the agent’s event store.
Builds a focused prompt. The model receives the inflection list (metric, direction, before/after values, percentage change) plus events grouped by ProviderName, severity-tagged, and message-truncated. Up to 500 events are included to stay within token budgets.
Returns structured output. The response is split into three sections — Root Cause, Evidence, Recommendations — rendered as collapsible panels in a modal overlay. OS-tagged commands are suggested where evidence supports them (PowerShell on Windows, bash/systemctl/journalctl on Linux).

When evidence is thin

If the bracket contains zero captured events — for example, the inflection happened before the agent started shipping events, or capture was disabled during the spike — the model is told to reason from the inflection metric alone and to be explicit about the uncertainty. You’ll see a “Zero events in bracket” note in the Evidence section.

Cost & provider

Each RCA consumes 1 AI query from your workspace pool, just like a live-session question. The call routes through the same BYOK / platform-key resolver as the rest of the AI surface — if you’ve set a custom provider key in Team → AI, RCA uses it; otherwise the platform default is used. Token cost and elapsed time are surfaced in the modal footer.

Rating AI Results

An Automated Troubleshooting result asks Was this run useful? with Helpful and Not helpful. Any member of the organization can rate a run. The rating is stored against the run with the member who gave it, without the run's text. Ratings are used to measure AI result quality for your organization, and a Helpful rating lets later runs in your organization use the rated run as a hint. Ratings from one organization are never used for another.

Troubleshooting runs are the only results with a rating button in the dashboard.

AI Result Quality summary

Team → AI shows an AI Result Quality card covering the last 30 days. For each source it lists how many troubleshooting commands the AI proposed and how many were approved, edited or skipped, how many fixes verified or did not verify, and how many Helpful and Not helpful ratings were given.

Only members who can manage privileges see the numbers. Other members see a notice instead.
The card is built from a summary written once a day for the previous day (UTC), so a run appears on the card the day after it happens. Until the first summary exists, it says that there are no summary rows for the period.

Bring Your Own AI Key (BYOK)

Team → AI. By default, AI features (live-session questions, RCA, smart reports, alert enrichment, script generation) use ET Ducky’s platform key and draw from your workspace query pool. With BYOK you point ET Ducky at your own provider account instead — useful for data residency, cost control, or higher limits.

Choosing a provider

Open Team → AI and pick a provider:

Anthropic Claude — key, plus an optional model override.
OpenAI — key, plus an optional model override.
Microsoft Copilot (Azure OpenAI) — key, plus the Azure Endpoint URL and the deployment name in the Model field (both required for Azure).

Paste your provider key and save. The key is never shown again (only a hash is stored); once saved, the tab shows a green “A key is saved” badge and the field placeholder reads “leave blank to keep it.” To rotate the key, type a new one; to update only the model or endpoint, leave the key field blank.

What BYOK changes

AI calls from your workspace route through your provider key — their cost, your bill.
BYOK calls do not deduct from your platform query quota.
If a BYOK call fails (bad key, provider outage), ET Ducky falls back to the platform key so troubleshooting isn’t blocked, and records the failure on the AI tab.

Usage

The AI usage card on the same tab breaks down token use by feature (live queries, RCA, smart reports, and so on) so you can see where your AI spend is going — whether a call ran on BYOK or the platform key.

Chat Evidence in the Desktop App

Each Ducky Chat answer in the desktop app that came from your subscription or your own API key keeps a record of what it was built from: the question, the events sent with it, the rule verdicts, the answer and the prompt. The record stays on your computer in %LOCALAPPDATA%\ETWMonitor\evidence\. Before it is written, user names, user profile paths, addresses and the computer name are removed as they are for the prompt, and tokens, password values and long hex or base64 strings are masked.

Export and tickets

Open the menu under an answer:

Export evidence saves a zip to your Documents folder with the record as evidence.json and as evidence.docx. Times in it are UTC. If your Documents folder is synced (for example to OneDrive), the zip is synced with it.
Attach to ticket adds the record to an ET Ducky ticket in your organization as a note. No file is uploaded; to give the reviewer the zip, export it and attach it to the ticket yourself. External tickets from Jira or ServiceNow are not listed.

Settings

Keep chat evidence on this computer (on by default). Turn it off and no record is written.
Keep memory evidence (off by default). A memory investigation summary can contain text read from other programs, including secrets that no pattern recognizes, so it is left out of the record and of the saved chat history unless this is on. The answer to a memory investigation, and every later answer in the same conversation, can be exported but not attached to a ticket, because the AI's view of the conversation includes that analysis. Clear the chat to start a conversation that can be attached again.
Clear evidence deletes every record.

Records are kept for 30 days, at most 200 records and 100 MB in total; the oldest go first. A record larger than 2 MB keeps the first 500 events and says it was cut.