Building Custom Roslyn Analyzers From Production Bug Post-Mortems

The ET Ducky cloud API ships with eleven custom Roslyn analyzers. Each one exists because a specific bug reached production or an audit found a class of latent ones. This post documents the method we settled on. It covers how a post-mortem becomes a compile-time rule, why the rules are deliberately conservative, how structured justification comments work as the escape hatch, and the furthest we have taken the approach, which is an analyzer whose annotations are the documentation and are verified against runtime behavior by an end-to-end check.

Fixing the class of bug rather than the single site

A bug fix repairs one site. The post-mortem question that produces an analyzer asks which decision the author failed to make, and whether the compiler can force the next author to make it. Three production examples from this codebase:

• An unconfigured cookie container. A shared HttpClient kept its default cookie container. The first response from an integrated service (Harbor) set a session cookie. Every later mutating call replayed that cookie, and the service's CSRF middleware returned a 403 even though the credentials were valid. The integration worked once per process lifetime. The fix was one line, UseCookies = false on a dedicated named client. The class of bug is "nobody decided what the cookie container does for this integration."
• Background services that returned no rows. EF Core global query filters scope every query to the current tenant. Background services run under a root context with no tenant. When a tenant-filter fix shipped, five hosted services began returning zero rows for every tenant, because none of their queries called IgnoreQueryFilters(). There was no exception, no log entry, and no data. The class of bug is "code that runs outside a tenant context must explicitly declare how it crosses the tenant boundary."
• An entitlement check on the wrong field. A checkout endpoint gated repurchase on whether a non-revoked license existed. Churn deliberately leaves licenses non-revoked and lets them expire through the renewal gate. A churned workspace was therefore told it was already subscribed and could not repurchase. The class of bug is "a license's revocation state is not an entitlement signal, and any boolean built on !Revoked diverges from the truth on churned customers."

Each fix shipped and each one produced a rule. ETD0011 requires every AddHttpClient registration to make its cookie policy explicit. ETD0002 requires tenant-filter handling in background-service query chains. ETD0013 bans !Revoked-shaped entitlement checks outside the license-domain services.

Why each rule has a narrow scope

Every analyzer in the set is a syntax-level heuristic with a deliberately narrow scope, and the limitations are documented in the analyzer's own header. ETD0002 inspects direct same-class helper calls and does not follow cross-method flow. ETD0010 covers controllers only, because controllers are where the property it checks is syntactically decidable. Extending it to services is tracked as a ticket. A Roslyn analyzer is held to the standard of a code review comment rather than that of a theorem prover. A rule that fires false positives gets suppressed project-wide within a month and then catches nothing. A rule that catches eighty percent of a bug class with zero false positives keeps running. When a rule cannot decide, it asks the author to decide, which is the next mechanism.

Justification comments as structured escape hatches

Most of the rules do not ban a construct. They require a decision about it. The mechanism is a structured comment with a rule-specific marker and a mandatory reason:

// cookie-policy: bearer-token API, server sets no session cookies
// tenant-opt-out: deliberate cross-tenant sweep, admin metrics endpoint
// cloud-egress: license renewal POST to api.etducky.com
// sql-reviewed: executes vetted migration files only
// local-safe: fails loudly via LocalClerkService; SPA hides this surface in local mode

The analyzer accepts the marker on the statement, the enclosing member, or, for boundary classes, the type. This does three things a plain suppression attribute does not do. The reason is mandatory and adjacent to the code, so a reviewer sees it. The markers are uniform and greppable, so every exception to a rule can be listed with one command. Writing a marker takes deliberate effort, so fixing the code is often less work than annotating it.

Annotations as verified documentation

The pattern we have found most useful is ETD0014, the egress rule. The product's self-hosted tier documents a small, enumerable set of outbound network destinations. That list previously lived in a whitepaper and went out of date whenever a service added a cloud call, because nothing checked it. ETD0014 requires every outbound HTTP dispatch in code reachable in local mode to carry a // cloud-egress: <reason> marker. Running grep -rn "cloud-egress:" --include=*.cs now produces the documented egress list.

The second half of the mechanism runs at runtime. A delegating handler on every factory-built HTTP client logs each dispatch host in local composition, and an end-to-end check asserts that the observed host set matches the annotated set. The analyzer fixes the authoring-time contract and the handler records the runtime behavior. Any difference between the two is a build failure. The egress list is generated from enforced annotations and checked against observed behavior, so it stays current with the code.

One implementation detail is worth noting. The rule detects HTTP dispatches semantically, by checking that the receiver derives from HttpMessageInvoker rather than by matching method names. SendAsync and GetAsync appear on SignalR clients, distributed caches, and third-party SDKs, so name matching would produce false positives immediately.

Smaller source hygiene rules in the set

Not every analyzer comes from a production incident. Three in the set are source-hygiene rules with codebase-specific rationale. Config keys for the deployment mode may only be read inside the one profile class that owns the composition decision, because a stray Configuration["Deployment:Mode"] would split that decision across two places. Non-constant SQL passed to raw-SQL EF APIs requires a review marker, because row-level security constrains what a query can see and does not constrain whether it is injectable. Invisible or bidirectional Unicode characters in source are flagged, which covers the Trojan-source class, with a single allow-listed NUL that a composite-key separator uses. Alongside the custom rules, the standard BannedApiAnalyzers package with a BannedSymbols.txt covers the cases where the fix is "never call this API here."

Build cost and what the rules cover

The eleven analyzers are a single netstandard2.0 project referenced by the API project as an analyzer. Build-time cost is negligible and there is no runtime component except the egress-observation handler. Writing one takes a day the first time and hours after that, because the statement-scope walking and comment-detection helpers are shared across rules. The tenant-filter rule covers the regression that once caused every background service in production to return zero rows, and it re-checks that condition on every build. A test would have covered the one call site it was written against. The analyzer covers every call site that matches the pattern.

The method is as follows. When a bug's post-mortem names a decision that was never made, write a conservative syntax-level rule that forces the decision. Provide a greppable justification marker as the escape hatch. Document the rule's limitations in its own header. Where the rule protects an external claim, add a runtime check that the annotations still match observed behavior.