Concepts

A few small mechanisms compose into the whole product: deterministic flag bucketing, a canary state machine, load-aware balancing, layered constraint enforcement with a verifiable audit trail, and leases-by-stream. Each is deliberately small enough to reason about.

Feature flags & sticky bucketing

A FlagRule routes individual requests to a variant of the target service, evaluated in-process on every call:

flag: "use-v2"
service: "demo.greeter"
targeting:                 # evaluated in order, first hit wins
  - { attribute: "group", value: "beta", variant: "v2" }
rollout_percent: 20        # then: sticky 20% of remaining traffic
rollout_variant: "v2"

• Targeting rules first — exact attribute matches (group=beta → v2), evaluated in order, first hit wins.
• Percentage rollout second — a sticky cohort of the remaining traffic, bucketed by user.
• Flags pick among variants that survived the constraint filter; a flag can never override residency.

The cross-language bucketing contract

Bucketing is deterministic and specified, not incidental:

/// The cross-language contract: FNV-64a(unit ‖ 0x00 ‖ flag) mod 100.
/// Sticky per user, independent across flags.
pub fn bucket(unit: &str, flag: &str) -> u32 {
    (fnv64a(&[unit.as_bytes(), &[0], flag.as_bytes()]) % 100) as u32
}

FNV-64a(unit ‖ 0x00 ‖ flag) mod 100 maps a user to a bucket in 0..100 — sticky per user (the same user always lands in the same cohort, every call), independent across flags (being in flag A's 10% says nothing about flag B). The Go and Rust SDKs implement it byte-for-byte identically and pin the same fixture values in their test suites: a user lands in the same cohort regardless of which language called. Every future SDK must pass the shared conformance fixtures before release.

Every flag decision emits an exposure event (who saw which variant, decided by which flag) back to the control plane — the raw material for experiment analysis.

The canary state machine

A rollout is started with an intent, not a script:

service: "demo.greeter"
canary_variant: "v2"
stable_variant: "v1"
steps: [10, 40, 100]       # percent of traffic per step
step_seconds: 8            # dwell time per step
max_error_percent: 2       # rollback threshold
min_requests: 20           # min sample size before judging a step

The controller drives a deliberately small state machine:

          ┌──────────── healthy at 100% ───────────┐
          │                                         ▼
none ─► stepping (10 → 40 → 100, dwell + judge)   promoted
          │
          └─── error rate > max_error_percent ──► rolled_back
               (with ≥ min_requests samples)        (traffic back on stable, instantly)

• At each step the canary variant's rolling health window (built from SDK-reported call outcomes) is judged: enough samples and an error rate under the threshold → advance; a breach → weights snap back to stable and the rollout ends in rolled_back.
• min_requests guards against deciding on noise; step_seconds guarantees dwell time at each weight.
• Blue/green is a degenerate rollout — two steps, 0 → 100 with a hold.
• Canary weights are per-request (a traffic split); flag rollouts are per-user (sticky). Both are soft selection, both stay under the constraint layer.

Prometheus-compatible analysis queries are in the API and tested: breaches roll back, missing data holds, and the controller never promotes blind. The current dona rollout command does not expose analysis flags; deployment strategies can carry analysis through the API.

Load-aware balancing

Selection picks a variant; balancing picks the endpoint within it. Round-robin spreads requests evenly, which is the wrong answer when endpoints are not equal — a cold start, a noisy neighbour, or an instance mid-GC should receive less.

• Load reported on the lease stream. Each workload already holds an open lease stream to the brain (see endpoint leases); it piggybacks a live load signal on it, so there is no separate health-probe path.
• The picker weights toward the least loaded. Among the ready endpoints of the chosen variant, the picker biases new requests to the least-loaded one rather than blindly rotating — so a struggling instance sheds work instead of being fed at the same rate as a healthy one.
• Balancing stays strictly under selection, which stays under the constraint filter: load can only ever break ties inside the set a flag or canary already chose, which a residency constraint already allowed.

Residency constraints & audit

Two different kinds of decision stay strictly layered — folding them together is the classic trap:

Enforcement is defense in depth — both points exist in the code today:

• Client-side filter (efficiency): the SDK filters the eligible endpoint set before selection; an unsatisfiable constraint fails the call with PermissionDenied without ever sending it anywhere.
• Server-side interceptor (the guarantee): the serving process itself refuses any call whose residency constraint it cannot satisfy. A client that bypasses Dodona routing entirely still cannot extract the data — this is the sidecar-bypass gap, removed, and it is what makes the property provable rather than configured.

Every constrained decision — allow or deny, at either point — emits an audit event:

message AuditEvent {
  string service = 1;
  string unit = 2;        // who: request unit or authenticated SPIFFE peer
  string constraint = 3;  // e.g. "residency=eu-west"
  bool denied = 4;
  string detail = 5;      // which enforcement point decided
}

The unit field now carries the authenticated mTLS/SPIFFE peer identity of the caller, not just request metadata — the same API, with the identity it always wanted. Every event is appended to a hash-chained log with signed anchors, so tampering with any past event breaks the chain.

Attestation & the offline verifier

A signed audit log is only useful if a third party can check it without trusting the system that produced it. So verification is a separate, self-contained step:

• Hash chaining. Each audit event commits to the hash of the previous one; the running anchor is signed by the control plane's key. Reorder, drop, or edit an event and the recomputed chain no longer matches the signature.
• Offline verifier. A standalone verifier recomputes the chain over an exported log and checks the anchor signature against the public key — no access to the live system required. An auditor runs it on their own machine.
• Honest limit. Without external anchor publication the verifier cannot detect whole-log tail truncation; that publication step is roadmap R2, and the gap is documented and tested, not hidden.

Per-period attestation reports are built now: the brain returns report JSON, a signature, and the signer public key for offline verification. External publication of chain heads is still roadmap work.

Endpoint leases

There is no DNS polling and no health-check sweeper. A workload registers by opening a stream and heartbeating on it:

message EndpointHeartbeat {
  string service = 1;  // e.g. "demo.greeter"
  string variant = 2;  // e.g. "v1", "v2"
  string address = 3;  // host:port the workload serves on
  string region = 4;   // residency label, e.g. "eu-west"
}

• The stream is the liveness signal. Stream death — process crash, network partition, deliberate shutdown — is deregistration. Endpoints leave the routing table the moment they stop being reachable, and every subscribed client gets the new snapshot pushed.
• Endpoint/health state is the churn plane: never durably written, rebuilt from leases on control-plane restart. Durable things (flags, ingress, artifacts, releases, deployments, cells, shared services, and audit records) live in the store when donad --data-dir is set. Multi-replica HA wiring is the later store integration.
• If the control plane is unreachable, the data plane keeps serving the last config it received — level-triggered, stale config beats no config.