Alert System

Root cause analysis on every alert

ET Ducky is an RMM that runs an automatic root-cause analysis on every alert from Windows and Linux endpoints. The analysis is built from the kernel events that triggered the alert (ETW on Windows, eBPF on Linux).

Each analysis gives the root cause, an impact assessment, immediate actions, long-term recommendations, and a confidence level. Open it with "View Details" on the alert.

Alerts Dashboard

The Alerts page has three tabs for managing monitoring rules, notifications, and alert history. Auto-refreshes every 30 seconds.

Tab 1: Alert History

Stat Cards: Counts for Critical, Warning, Info, Active Total, and Resolved Today
Filters: By status (All/Active/Acknowledged/Resolved) and severity (All/Critical/Warning/Info)
Alert Cards: Severity badge, status badge, time since triggered, rule name, agent name, description, AI Analysis badge

Alert Status	Available Actions
Active	Acknowledge, Resolve, View Details
Acknowledged	Resolve, View Details
Resolved	View Details

View Details opens a modal with full alert info, triggering metrics, and AI root cause analysis with recommendations.

Tab 2: Alert Rules

Define conditions that trigger alerts. Click Create Rule to open the Rule Builder.

Rule Builder Fields

Rule Name & Description
Severity: Critical, Warning, or Info (radio buttons). A rule saved through the API with another severity the server accepts, such as error, shows that severity as a fourth, selected option marked as the stored value.
Conditions: One or more conditions with AND/OR logic
Notification Channels: Select channels to notify
Enabled Toggle: On/off without deleting

Available Metrics

The condition list in the Rule Builder comes from the server's metric catalog, which is the same list the server evaluates rules against. Each metric is shown with its unit and grouped by category. A rule names a metric by its ID.

Category	Metric	ID	Unit
Performance	CPU usage	`cpu_usage_percent`	%
Memory	Memory usage	`memory_usage_percent`	%
	Memory used	`memory_used_mb`	MB
	Memory available	`memory_available_mb`	MB
Disk	C: drive free	`disk_c_free_percent`	%
	C: drive free	`disk_c_free_gb`	GB
	Lowest drive free	`min_disk_free_percent`	%
	Disk read rate	`disk_read_bytes_per_sec`	bytes/s
	Disk write rate	`disk_write_bytes_per_sec`	bytes/s
	Disk queue length	`disk_queue_length`	count
	File size (needs a target: the file; agent 2.8.5.1 or later)	`file_size_mb`	MB
	Folder size (needs a target: the folder; agent 2.8.5.1 or later)	`folder_size_mb`	MB
Network	Network send rate	`network_send_bytes_per_sec`	bytes/s
	Network receive rate	`network_receive_bytes_per_sec`	bytes/s
	Active connections	`network_active_connections`	count
	Ping loss (needs a target: the host; agent 2.8.5.1 or later)	`ping_loss_percent`	%
	Ping round trip (needs a target: the host; agent 2.8.5.1 or later)	`ping_rtt_ms`	ms
System	Uptime	`uptime_days`	days
	Pending reboot	`pending_reboot`	0 or 1
	Recently failed services	`services_recently_failed_count`	count
	Automatic services stopped	`stopped_automatic_services_count`	count
	Event log errors, last hour	`event_log_errors_last_1h`	count
	Service running (needs a target: the service)	`service_running`	0 or 1
Event log entries (needs a target: log, event ids and source; a window; Windows agent 2.8.5.1 or later)	`event_count`	count
Security	Antivirus definition age	`definition_age_days`	days
	Antivirus enabled	`antivirus_enabled`	0 or 1
	Antivirus definitions up to date	`antivirus_up_to_date`	0 or 1
	Firewall enabled	`firewall_enabled`	0 or 1
	System drive encrypted	`system_drive_bitlocker_enabled`	0 or 1
Patching	Windows Update enabled	`windows_update_enabled`	0 or 1
	Pending updates (last patch scan)	`pending_updates_count`	count
	Days since an update was installed	`days_since_last_patch`	days
	Failed patch jobs, last 24 hours	`failed_patch_jobs_24h`	count
	Pending security and critical updates	`pending_security_updates_count`	count
Availability	Seconds since last contact	`last_heartbeat_seconds`	seconds
	Days since last sample	`agent_silent_days`	days
	Expected telemetry blocks missing	`telemetry_block_missing`	count
Hardware	CPU temperature	`hw_cpu_temperature_c`	°C
	Disk wear	`hw_disk_wear_percent`	%
	Battery health	`hw_battery_health_percent`	%
	Battery charge	`hw_battery_charge_percent`	%
	System thermal state	`hw_system_thermal_state`	code: 1 Other, 2 Unknown, 3 Safe, 4 Warning, 5 Critical, 6 Non-recoverable
Agent Properties	Agent Name (Server Name)	`agent_name`	text
	Agent Hostname	`agent_hostname`	text
	Agent Tags (comma-separated)	`agent_tags`	text

Lowest drive free is the lowest free-space percentage across the agent's fixed drives (Windows) or mounted filesystems (Linux), from the disk space report agents send by default. Drive sizes are reported in whole gigabytes, so drives under 1 GB are left out.
Recently failed services counts services that failed. On Windows these are services set to start automatically that are stopped and have a crash or start-failure event from the Service Control Manager in the last 24 hours; delayed-start and trigger-start services are not counted. On Linux these are systemd services in the failed state. Agents before 2.8.4.0 report something narrower: on Windows, automatic services that are stopped (up to 5, with or without a crash), and on Linux nothing. Agents report it when service metrics are on, which is the default; the Minimal Monitoring and Standard Monitoring presets turn them off. Upgrading to 2.8.4.0 turns service metrics on once for an agent that had them off; apply the preset again to turn them off.
Event log errors, last hour counts errors in the last 60 minutes: on Windows, Error and Critical entries in the Application and System logs; on Linux, system journal entries at error priority or more severe. From agent 2.8.4.0 event log metrics are on by default, and upgrading turns them on once for agents that had them off. Earlier Windows agents report it only when event log metrics were turned on in the agent's configuration, and earlier Linux agents do not report it.
A condition on a metric the agent did not report is not met, so a rule on one of these three does not fire for an agent that does not send it.
Failed patch jobs, last 24 hours counts the agent's ET Ducky patch jobs that failed, regressed or were rolled back and finished in the last 24 hours. An agent with no failed job reads 0. Pending security and critical updates counts the updates classified security or critical in the agent's last patch scan; an agent that has not reported a patch scan has no value, so a rule on it does not fire there. Both are read from the organization's patch data, which the server refreshes for alerting every 2 minutes, so a new failure or scan reaches a rule within 2 minutes of being recorded. Neither keeps a history, so a sustained-for window does not apply to them.
Hardware readings: the five hw_ metrics are read from the agent's most recent hardware reading, and only when that reading is less than 30 minutes old.
Availability: Seconds since last contact is the time since the agent last sent a heartbeat or a health sample. Days since last sample is the age of the agent's latest health sample, so it rises for an agent that keeps sending heartbeats without health data. Rules on these two are checked whenever an agent reports, and every 2 minutes for agents that have stopped reporting. For an agent that has gone silent, a rule fires no sooner than 5 minutes after its last contact, when the Agents page shows it offline. A rule that combines these with any other metric, such as CPU usage, is not checked for silent agents. An alert raised for a silent agent notifies once and is not repeated while the agent stays silent. When the agent reports again, the alert resolves on its next evaluation. After the server restarts, the check for silent agents waits 10 minutes so agents have time to reconnect, and an agent that registered but never sent a heartbeat is not checked.
Expected telemetry blocks missing counts the heartbeat details that the organization's Heartbeat Detail setting keeps on but that a heartbeat did not carry. It is 0 for an organization that has not changed Heartbeat Detail.
Older IDs: rules and templates created before the catalog may name an older ID, such as disk_usage_percent, disk_free_percent, network_bytes_sent, network_bytes_received, active_connections or error_events_per_minute. These keep working. The Rule Builder offers an older ID only on a condition that already uses it.
IDs not in the catalog: earlier versions of the Rule Builder offered IDs that the server never evaluated, such as Agent ID, Agent Type and OS. A rule that names one keeps it, and the Rule Builder shows it as “(not in the metric catalog)”. A condition on such an ID is never met, so the rule does not fire. Edit the condition to use a catalog metric.

Conditions with a target

Some metrics need to know what to look at. When you pick one, the Rule Builder shows a target box beside the metric, with a hint below it, and the rule is not saved until the box is filled in. A target is checked when the rule is saved: a service name may hold letters, digits, spaces and the characters . _ - @ $ ( ) & ' , + # :, may not start with -, and is at most 256 characters.

Service running (service_running) takes a service name or display name, for example Spooler or Print Spooler; on Linux, a systemd unit such as sshd. Case does not matter. It is 1 when the service runs and 0 when it is stopped or failed, so a rule written as = 0 fires while the service is stopped. Agents from 2.8.5.1 check every service a rule names, every 2 minutes (see Agent watch list). Older agents report only the services in their health data: the services on the agent's critical services list (by default W3SVC, WAS, MSSQLSERVER, SQLSERVERAGENT and the ET Ducky agent on Windows; sshd, cron or crond, systemd-journald and the ET Ducky agent on Linux) and any service that failed recently; for a service the health data does not list, an inventory the agent sent in the last 30 minutes is used (the agent sends one at start, once a day and when you refresh it). A service the agent does not have, or does not report, is unknown: the condition is not met, and an alert already open for it stays open until the service is reported again.
Event log entries (count) (event_count) takes log|event ids|source, for example System|7031,7034|Service Control Manager (a service crashed) or System|41|Microsoft-Windows-Kernel-Power (an unexpected restart). The log is System, Application or another Windows Event Log channel such as Microsoft-Windows-TaskScheduler/Operational; give 1 to 10 event ids separated by commas; the source is optional. Beside the target, the window box sets how far back to count, 5 to 1440 minutes (60 when empty). The value is the number of matching entries in that window, counted every 2 minutes and at most 10,000, so > 0 fires while a matching entry is in the window and resolves once it ages out. Windows only. Needs agent 2.8.5.1 or later.
File size (file_size_mb) and Folder size (folder_size_mb) take an absolute local path, for example C:\inetpub\logs\LogFiles or /var/log. Network (UNC) paths, a drive root or /, wildcards and . or .. segments are refused. File size is checked every 2 minutes; folder size adds up every file below the folder every 15 minutes, and a folder with more than 200,000 files, or one that takes more than 20 seconds to measure, has no value. A path that does not exist has no value. Sizes are in MB. Needs agent 2.8.5.1 or later.
Ping loss (ping_loss_percent) and Ping round trip (ping_rtt_ms) take a host name or IP address. Every 2 minutes the agent sends 4 pings. Ping loss is the share that went unanswered, 0 to 100, so = 100 fires while the host does not answer at all. Ping round trip is the average of the answered pings in milliseconds, with no value while the host does not answer. Needs agent 2.8.5.1 or later.
A target is part of the condition. Two conditions on the same metric with different targets are evaluated separately, and an export and import keeps the target.

Agent watch list

The metrics above that need the agent to look at something are collected from a watch list. ET Ducky builds each agent's list from the enabled rules that apply to it and sends it with the agent's next heartbeat when it changes; the agent reports its readings with every heartbeat. A rule whose conditions are joined by AND and include Agent Tags or Agent Name conditions applies only to the agents those conditions match; any other rule applies to every agent in the organization. One agent's list holds at most 50 services, 50 event log filters, 20 files and folders together and 10 ping hosts; items past those limits are left out in a fixed order and the API logs it. A reading older than 30 minutes is not used. The watch list needs agent 2.8.5.1 or later; older agents never receive it, so rules on Event log entries, File size, Folder size, Ping loss and Ping round trip do not fire on them, and Service running falls back to the health data described above. The agent calls the operating system's service, event log, file system and ping functions directly and never passes a target to a shell.

What leaves the device. For each item on its watch list the agent sends back only a reading: whether the service runs, the number of matching event log entries, a size in MB, or the ping loss and round trip, with the time it was taken and a fixed status word (ok, not found, access denied, timed out, error or not supported). It never sends event log message text, the names of files inside a watched folder, file contents or ping replies. The watch list itself holds only the targets your rules name.

Condition Operators

Numeric	String
> ≥ < ≤ = !=	= != contains not_contains starts_with ends_with

Sustained conditions

Each numeric condition has an optional sustained for requirement — tick the box beside the threshold and give it a number of minutes. With it off (the default, and how every rule behaves unless you change it) the condition fires the moment a single sample crosses the threshold. With it on, the condition only fires if every sample across that window was over the line.

Use it on rules that flap: a host that touches 95% CPU for four seconds while an installer runs will trip “CPU above 90%” and clear again on the next sample, and the alert history fills with noise nobody reads. “CPU above 90% sustained for 5 minutes” describes a machine actually in trouble.

A sustained condition needs the history to prove it. An agent that has only just come online, or one that stopped reporting for part of the window, will not fire a sustained rule until it has been reporting across the whole window — deliberately, so a normal boot-time spike on a freshly installed agent cannot look like five minutes of sustained load. If you need to know about a host the instant it crosses a line, leave the box unticked.

Saved rules display as cards with name, severity, evaluation interval, channel count, and edit/delete/toggle controls.

Organization admins and owners can export every alert rule to a file and import rules from one. Imported rules start disabled. To enable one, click Edit, select at least one notification channel (the rule builder does not save a rule without one), tick Enable this rule and click Update Rule. See Configuration Export and Import. To bring rules over from another RMM tool, follow Moving a setup from another tool, step by step.

How Rules Are Evaluated

A rule is evaluated each time an agent reports, against the agent's latest health sample.

Missing metrics: a condition is met only when the agent reported the metric it names. A rule on a metric an agent does not send, for example a Windows-only reading on a Linux agent, does not fire for that agent. A missing value is not treated as 0.
Posture checks: Antivirus enabled, Firewall enabled and System drive encrypted are the exception. When the agent reports nothing for one of them, the value counts as 0, a failed check, so a rule written as < 1 or == 0 fires.
Closing alerts: an open alert is resolved automatically only when a sample shows its condition is no longer met. While any metric the rule names is missing from a sample, the alert stays open.
Sustained conditions apply only to metrics with stored history. Hardware readings, Seconds since last contact, Days since last sample and Expected telemetry blocks missing are checked on the current sample alone, whether or not the box is ticked.

Built-in Rule Templates

ET Ducky provides built-in rule templates through the API. GET /api/alerts/templates lists them, and POST /api/alerts/rules/from-template/{templateId} creates a rule from one. A rule created from a template can be edited in the Rule Builder like any other rule.

These templates evaluate today:

Template	Condition
High CPU Usage	CPU usage above 80% sustained for 5 minutes
Critical CPU Usage	CPU usage above 95% sustained for 1 minute
High Memory Usage	Memory usage above 85% sustained for 5 minutes
High Disk I/O	Disk read rate above 100,000,000 bytes/s sustained for 5 minutes
High Network Traffic	Network send or receive rate above 100,000,000 bytes/s sustained for 5 minutes
System Overload	CPU and memory usage both above 90% sustained for 3 minutes
Resource Exhaustion Warning	CPU above 75% or memory above 80%, each sustained for 30 minutes, or lowest drive free below 20%.
Low Disk Space	Lowest drive free below 15% sustained for 10 minutes
High Error Event Rate	More than 10 event log errors in the last hour, sustained for 5 minutes. Agents that report event log errors only.
Agent Offline	Seconds since last contact above 300
No Health Data in 7 Days	Days since last sample above 7
Expected Telemetry Missing	A heartbeat left out a detail that the organization's Heartbeat Detail setting keeps on

These templates name metrics that no agent reports yet. A rule created from one does not fire until agents report the metric:

Template	Waits on
Multiple Failed Login Attempts	A failed sign-in count
Unauthorized Access Attempt	An unauthorized file or registry access count
Network Connectivity Loss	A network interface state

All three name IDs that have no source yet, so the Rule Builder shows their condition as “(not in the metric catalog)”. Critical Service Stopped uses Service running, which names no service in the template: a rule created from it does not fire until you edit its condition and fill in the service as the target (see Conditions with a target).

Tab 3: Notification Channels

Configure where alerts are delivered. Channels are reusable across multiple rules.

Type	Configuration	Use Case
Email	Recipient addresses, subject template	Individual notifications, reports
Slack	Webhook URL	Team chat, DevOps channels
Microsoft Teams	Webhook URL	Enterprise collaboration
Webhook	Custom URL, HTTP method, headers	ITSM, automation, custom integrations
PagerDuty	Integration key	Incident management, escalation policies
Push notification	Audience (everyone / admins / specific members / on-call)	Alerts on your phone or desktop — see Mobile Notifications & On-Call

Test: Send a test notification to verify configuration
Enable/Disable: Toggle without deleting
Each channel card shows type icon, name, type label, and toggle switch
Audiences: Email and Push notification channels can target everyone, workspace admins, specific members, or the on-call person — resolved when the alert fires. See Mobile Notifications & On-Call.

AI-Powered Alert Analysis

Every alert receives automatic Claude AI analysis including root cause, impact assessment, immediate actions, long-term recommendations, and confidence level. Analysis is cached for 24 hours — similar alerts reuse cached results.