⚠ STATE-ROOT PATHS: this volume said
/etc/helixand/var/lib/helixuntil2026-09-13h, three days after they moved. The roots are/etc/eventlake,/var/lib/eventlakeand/var/lib/eventlake-sensorfor the Sensor (/var/lib/eventlake-moteuntil2026-09-12m, which the agent now only reads as the LEGACY root it migrates from; code, packaging and the live node's ceremony,2026-09-10). ⛔ A host that has NOT been cut over is still correct on the old paths, andeventlake-ctl doctorstill probes them — if a command here does not find what it expects, check which generation the host is on before changing anything. Runbook:docs/reference/state-root-cutover-runbook.md.How the platform is deployed, how big it has to be, how the network estate it monitors feeds it, and how its own recovery is drilled rather than asserted. The operator's half of the four-plane model.
This volume owns everything between "we have binaries" and "it is running, correctly sized, and proven to recover":
This volume does NOT own:
| Not owned here | Lives in |
|---|---|
| The charter, the four-plane doctrine, the roadmap and the strategic review | 00-charter-and-strategy.md |
| The event model, the lake and its storage engine, ingestion semantics, the node-OS decision, distributed-detection design, composition, vector recall | 01-substrate-plane.md |
Detection semantics and coverage — including the flow arm's detector logic whose sources are configured here |
02-analytics-plane.md |
| The conductor's guardrail chain, SOAR, Firewall's compiler, Validation, CTEM, EASM, cases, hardening baselines | 03-execution-plane.md |
| Identity, MFA and the F5 RADIUS bridge, Messages, EventLake Authenticator, zero-trust doctrine, the policy plane, crypto-agility | 04-identity-and-policy-plane.md |
| Compliance regimes, AI governance, the trust center, lawful tiers, the platform threat model, supply-chain verification | 06-governance-and-trust.md |
| Licensing, service tiers, the customer console, hosting cost, company formation, GTM | 07-commercial-and-gtm.md |
| The REST and gRPC surfaces themselves, the query surface, plugin authoring, GenAI, protocol strategy, the integration map | 08-interfaces-and-extensibility.md |
Four boundaries worth stating twice:
snmp, decode/netflow and decode/pcap
are configured, collected and sized in this volume. What flow's exfil/beaconing/port-scan/host-sweep
detectors mean is Volume 02's.Exempt living documents this volume defers to (never restated here, always cited):
docs/reference/build-status.md — authoritative for the D4/D5c performance baseline and for what
is proven end to end. Every throughput figure below is quoted from it with its date and box.docs/reference/package-map.md — what each conduit package does.docs/production-readiness.md — the consolidated remaining-work-to-market scoreboard; the
deferrals in §6 are pointers into it, not a second scoreboard.docs/coverage-gap-analysis.md — authoritative for any ATT&CK figure.docs/design-handoff-tracker.md — owns the sequencing of any operator surface named here.Infrastructure is the plane the other three stand on, and the only one whose failures are measured in minutes of blindness rather than missed detections.
┌──────────────────────────────────────────────────────────┐
substrate │ event · lake · ingestion — the single-writer durable │
(Volume 01) │ lake this volume gives a disk, a horizon and a backup │
└───────────────────────────┬──────────────────────────────┘
│ runs on
┌───────────────────────────▼──────────────────────────────┐
analytics │ detection arms · the schedulers §2.1.7 turns on │
(Volume 02) └───────────────────────────┬──────────────────────────────┘
│ acts through
┌───────────────────────────▼──────────────────────────────┐
execution │ conductor → Sensor — the agent this volume INSTALLS, │
(Volume 03) │ flags, packages, updates and drills │
└───────────────────────────┬──────────────────────────────┘
│ deployed / sized / proven by
┌─────────────────────────────────────────▼──────────────────────────────┐
│ INFRASTRUCTURE & OPERATIONS (THIS volume) │
│ topology · durable mode · backup · HA · island mode · fleet updates · │
│ key custody · cells & tiers · EPS and byte math · SNMP/flow/pcap · │
│ the drill suite │
│ "how do I run it, how big is it, and how do I know it recovers?" │
└─────────────────────────────────────────────────────────────────────────┘
Two properties make this plane unusual, and both come from the substrate:
replicas: 1 is a load-bearing line in a manifest.| Responsibility | Owner artifact(s) | Boundary |
|---|---|---|
| Ship verifiable binaries | release pipeline (v* tags) + deploy/Dockerfile |
Signing and attestation semantics are 06-governance-and-trust.md#supply-chain's; what to run is here |
| Terminate TLS and route | your proxy (Caddy/nginx/Envoy/Ingress) | TLS + routing ONLY — auth is in-process and never delegated |
| Make the lake durable and bounded | EVENTLAKE_SEGMENT_DIR · _GROUP_COMMIT · EVENTLAKE_MEMORY_HORIZON · EVENTLAKE_ROLL_EVERY |
The store's format and semantics are 01-substrate-plane.md#lake-storage-engine's; the operational guidance is here |
| Gate traffic on readiness | /v1/readyz (503 until boot replay completes) |
Liveness (/v1/healthz) is separate and always on |
| Hold the roots of trust | platform-keys.json · the root KEK · EVENTLAKE_SHRED_DIR · EVENTLAKE_BYOK_DIR+_KEK |
The custody rules (in the backup set vs excluded) are here; what the keys protect is Volume 04's |
| Turn autonomy on | the EVENTLAKE_*_SCHEDULE set (§2.1.7) |
Every loop is off by default; the dev server stays inert |
| Install, flag and update the edge | mote + deploy/systemd, deploy/launchd, deploy/windows |
The agent's enforcement legs are Volume 03's; their install, packaging, spool and rollout are here |
| Answer "how big" | §2.3 constants + ui_kits/public-site/sizing.html |
The licensed dimension (ingested TB/day) is 07-commercial-and-gtm.md#licensing-model's |
| Feed the NOC | snmp · decode/snmp · decode/netflow · decode/pcap |
The detectors over those streams are Volume 02's |
| Prove recovery | conduit/resil + the §2.5.4 ops calendar |
Not a load harness (cmd/eventlake-bench owns that), not customer-estate chaos (Validation owns that) |
Three flows define this plane: standing a node up, sizing it, and proving it recovers.
Flow A — from binaries to a node serving traffic:
a tagged release (or a source build)
│ verify FIRST: cosign signature · CycloneDX SBOM · build provenance [Volume 06]
▼ CONFIGURE /etc/eventlake/eventlake-api.env — private listener, segment dir,
│ memory horizon, root KEK, node id, the autonomy set
▼ START systemd unit (StateDirectory, TimeoutStopSec > drain)
│ └─ BOOT REPLAY: sealed segments → control plane rebuilt
│ "replayed N events (… 0 unverified) in T"
▼ READY /v1/readyz flips 503 → 200; only now does the proxy route
▼ FRONT TLS terminates at the proxy on the private/mgmt interface
▼ OBSERVE /metrics + structured stderr logs + out-of-band paging
▼ DRAIN SIGTERM → stop accepting → drain 10 s → close spine →
close the segment store LAST (flushes deferred group-commit)
Flow B — from an estate inventory to a bill of materials:
the log-producing estate
▼ INVENTORY §2.3.3 catalog → Σ average EPS
▼ PEAK × PEAK_FACTOR 2.0 (sustained busy hour, not the worst second)
▼ POSTURE raw ON (forensic/compliance) or raw OFF (budget) — the single
│ biggest lever; per-source mixing is the norm
▼ VOLUME EPS × wire bytes → GB/day ingested = the LICENSE dimension
│ EPS × store bytes → GB/day stored
▼ RETENTION hot RAM window + on-node days + compliance archive
▼ COMPUTE vCPU = EPS_peak/1,500 + 2 (min 4) · RAM = 8 GB + horizon + 2/10k
▼ MAP onto a tier (§2.2.6) and a cell count — or onto a SaaS plan
▼ PILOT 7–14 days of real sources; the platform METERS ITSELF and the
measured numbers override every planning default
Flow C — the recovery claim, made executable:
a durability claim ("a restart forgets nothing")
▼ INJECT six faults, one per hermetic drill instance [conduit/resil]
│ kill-replay · torn-tail · disk-full · clock-skew ·
│ partition-heal · restore-rehearsal
▼ ASSERT the exact invariant the claim rests on — replay equality,
│ zero unverified seals, EXACTLY the torn record lost,
│ the measured RPO gap, one owner post-heal
▼ REPORT it/resilience/report on the lake — trendable, queryable
│ a FAILED drill is SevHigh with a failure outcome
▼ REHEARSE what in-process injection cannot reach — a REAL backup on a
clean host, kill −9 under load, true ENOSPC, a firewall-level
partition, an NTP step — on the ops calendar (§2.5.4)
| # | Invariant | Why it is load-bearing |
|---|---|---|
| I1 | The footprint SLO: steady-state < 1 % CPU / < 150 MB RAM per host. | It is a hard constraint, not a target. It is why the collector spool refuses a per-record fsync'd cursor, and why Sensors are "free" relative to the brain when sizing |
| I2 | One brain owns one segment directory. Never point two brains at one lake; never scale a StatefulSet past replicas: 1 per lake |
The lake is single-writer by design. Two writers is not slow — it is corrupt |
| I3 | Readiness gates traffic; liveness never does. /v1/readyz stays 503 until boot replay completes |
An orchestrator that routes to a half-assembled brain gets answers computed over a partial lake |
| I4 | unverified > 0 is an integrity signal, not noise. A record that fails seal verification is kept and flagged, never silently dropped |
The record plane may never say "nothing" (Volume 04 I5). Acceptance for every restore is … 0 unverified |
| I5 | platform-keys.json travels WITH the segments |
Segments restored without it replay as unverified — a backup that looks complete and proves nothing |
| I6 | The root KEK is IN the backup set; the shred keystore is EXCLUDED. Do not generalize either rule to the other | They are opposites: shred keys protect a promise to FORGET (restoring them undoes an erasure); the KEK protects the ability to REMEMBER (losing it makes every subject re-enroll) |
| I7 | A malformed key file fails the boot LOUDLY — never mint a replacement, never start empty | Silently re-keying orphans every sealed seed; an empty keystore orphans every existing ciphertext. A visible failure is recoverable; an invisible one is not |
| I8 | The proxy is TLS + routing; the process is the security boundary | A bypassed proxy exposes cleartext, never an unauthenticated surface — so "the proxy failed open" is a confidentiality incident, not an authorization one |
| I9 | Every scheduler is off by default | The dev server stays inert; production turns on exactly what it needs. An always-on loop is a footprint tax on every lab instance |
| I10 | Enforcement at the edge is opt-in, per leg, and reversible. Dry-run is the default; -enforce, -enforce-exec, -self-update and -watchdog are separate switches |
Each carries a distinct risk with its own prerequisite. A firewall element add/delete is naturally reversible; an eBPF LSM deny entry is reversible; a kill-process is not — and the difference is what the conductor gates on |
| I11 | Nothing ships to the whole fleet at once, and no ring widens without measured health | The negative benchmark is the 2024 channel-file incident: a global same-instant content push with no rings, no health gate, no halt |
| I12 | No evidence ⇒ no promotion. Silence holds a rollout; it never advances one | Adopted-then-dark is the crash signal a broken update leaves. Halting needs no gate — stopping distribution is always safe |
| I13 | Absent ≠ zero, and an offline agent's figure is a FLOOR, not a depth | A missing spool key means unknown, not "no backlog"; an agent offline for an hour has been spooling unobserved that whole time — which is exactly when understating it would be worst |
| I14 | Loss is measured and reported, never hidden | The spool counts what its byte budget dropped; the torn-tail drill asserts exactly one record lost; the restore rehearsal reports the RPO gap in events |
| I15 | A drill that cannot run skips VISIBLY with its reason | A vacuous pass is worse than a failure — it is a claim with nothing behind it. "Replay equality over nothing proves nothing" |
| I16 | Drills are hermetic. The live lake, live detection state and live control plane are never touched; the only live write is the report snapshot | Self-validation must never become the incident |
| I17 | A world-changing device write goes through the conductor. An automated SNMP Set is an actuation, never a bare client call |
snmp.Client.Set is a low-level primitive; the gate lives at the spine/actuation layer |
| I18 | Every capacity number is dated and labelled M or P | An undated throughput figure becomes a promise the next box cannot keep. The pilot overrides every P |
The operator's answer to "how do I actually run it". Everything in this section describes what is built and verified today; pieces that are designed but blocked or deliberately deferred are collected in §6 — nothing is silently pretended.
Two binaries, both stdlib-only Go (CGO off, static), built from conduit/:
| Binary | Role | Listens |
|---|---|---|
eventlake-api |
The brain: assembled spine (lake + detection + conductor + every arm) + the management-plane REST API + optional web console, gRPC, cluster mesh, and subject-facing public plane | :8080 REST (default) · optional gRPC / public / mesh listeners |
eventlake-sensor |
The edge agent (the Sensor; the binary was mote until 2026-09-12m): polls the brain for conductor-approved commands and applies them (dry-run by default; -enforce = nftables/iptables on Linux, netsh on Windows, a pf anchor+table on macOS), runs optional inline enforcement (DNS sinkhole resolver, inline IPS / virtual patching, inline WAF, inline-inspect verdicts), receives compiled edge detections, verifies + acks fleet updates (§2.1.11), records brokered PAM sessions — and IS the PTY wrapper that writes them (-wrap-session, unix) — forwards raw host telemetry (-collect-auditd Linux · -collect-winlog Windows · -collect-eslogger macOS), can swap its own binary (-self-update, boot-safe A/B + crash-loop revert), and can supervise itself (-watchdog) |
outbound to the brain; optional local -dns-addr / -ips-addr / -inspect-addr |
Get them. A tagged release (v*) publishes per-platform binaries — eventlake-api for linux/amd64,
linux/arm64, windows/amd64; eventlake-sensor for every edge OS (those three plus windows/arm64, darwin/amd64,
darwin/arm64, and — since 2026-09-19ai — the 32-bit estate: linux/arm at GOARM=5, linux/mips + mipsle at
GOMIPS=softfloat, linux/386, windows/386; one artifact per arch, at the floor that runs on every core of it,
with main.version stamped to the tag; per-target table in deploy/README.md) — with a CycloneDX SBOM, a cosign keyless
signature, and GitHub build provenance per artifact. Verify before running
(06-governance-and-trust.md#supply-chain carries the exact cosign verify-blob / gh attestation
verify commands). Or build from source, reproducibly flagged:
cd conduit
CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o eventlake-api ./cmd/eventlake-api
CGO_ENABLED=0 go build -trimpath -ldflags='-s -w -X main.version=v1.2.3 -X main.channel=stable' -o eventlake-sensor ./cmd/eventlake-sensor
The -X main.version stamp is what the Sensor reports to the fleet board and to the rollout engine's
adoption evidence — stamp it with the release tag. -X main.channel must be stamped in the SAME
build, because development and production versions are drawn from disjoint spaces and the agent
checks the two agree before it starts: a production lane (stable, lts) requires a plain
vMAJOR.MINOR.PATCH, and every pre-production lane (dev, edge, beta) requires that lane's own
prerelease identifier (v1.3.0-beta.2). A mismatch — the common one being a version stamped without
its channel — refuses to start, by design: the alternative is a half-stamped build that enrolls,
reports adoption and lands on a support-matrix row. An un-stamped build needs neither flag; it is a
development build and says so (v0.0.0-dev.0, channel dev). Omitting -X main.version while
stamping a production channel fails the same way. The release workflow derives the channel from the
tag rather than accepting it twice (.github/workflows/release.yml). Cross-compile with GOOS=linux GOARCH=amd64|arm64;
the static Linux agent is ~7 MB.
Container images are built from the repo's first-party deploy/Dockerfile (multi-stage; build with
--target api or --target sensor). There is no published registry image yet (§6) — build and push to
your registry:
# one brain image per posture; POSTURE DEFAULTS TO self-managed (owner directive 2026-10-05) — a customer's
# image carries no OEM code at all, and our own cells pass POSTURE=hosted explicitly
docker build --target api --build-arg POSTURE=self-managed -t <your-registry>/eventlake-api:v0-self-managed -f deploy/Dockerfile .
docker build --target api --build-arg POSTURE=hosted -t <your-registry>/eventlake-api:v0-hosted -f deploy/Dockerfile .
docker build --target sensor -t <your-registry>/eventlake-sensor:v0 -f deploy/Dockerfile .
EventLake separates its network surfaces into planes with distinct trust domains. The rule of thumb: the process serves plain HTTP on private interfaces; TLS terminates at a fronting proxy you already trust — except gRPC (native TLS, h2 requires it) and the edge↔brain ingest path (native mTLS).
INTERNET
│
TLS-terminating proxy (Notary/CLM cert)
│ subject-only surface
┌──────────────────────▼──────────────────────────┐
│ eventlake-api │
│ :8080 management REST + console ◀── TLS proxy on the PRIVATE/mgmt net
│ :8443 public subject plane (EVENTLAKE_PUBLIC_ADDR) (MFA·wallet·agora·enroll)
│ :8843 gRPC (EVENTLAKE_GRPC_ADDR — NATIVE TLS)
│ :9443 cluster mesh (EVENTLAKE_CLUSTER_ADDR — NATIVE mTLS, cluster CA)
│ EVENTLAKE_SEGMENT_DIR ──▶ durable lake (PVC / local disk)
└───────▲─────────────────────────────────────────┘
│ transport mTLS (HTTP/2, optional PQC-hybrid KEM) [package seam — §6]
Sensor agents / conflux Tier-2 aggregators (edge)
| Plane | Listener | TLS story (today) | Who reaches it |
|---|---|---|---|
| Management REST + console | EVENTLAKE_API_ADDR (default :8080) |
Plain HTTP from the process (api.HardenedServer — real timeouts, D21). Terminate TLS at a fronting proxy and keep the listener on a private/mgmt interface. |
Operators, the portal, connectors, POST /v1/ingest |
| Public subject plane | EVENTLAKE_PUBLIC_ADDR (optional) |
Same hardened plain-HTTP listener serving PublicHandler() only — MFA, wallet, Messages, enrollment, push, identity-me; management/cluster/metrics/console are refused on it. Front with a TLS proxy carrying your public (Certificates/CLM-issued) cert. |
EventLake Authenticator apps over the internet |
| gRPC | EVENTLAKE_GRPC_ADDR (optional) |
Native TLS on its own listener (gRPC needs h2): EVENTLAKE_GRPC_TLS_CERT + EVENTLAKE_GRPC_TLS_KEY; without them a self-signed dev cert is minted (fine for a lab, not production). Min TLS 1.2. |
gRPC clients (08-interfaces-and-extensibility.md#grpc-surface) |
| Edge↔brain ingest | transport.NewServer / conflux aggregator |
Native mTLS (HTTP/2, client-cert-bound observer identity via CertBoundRegistry); transport.RequireHybridKEM pins the X25519MLKEM768 post-quantum hybrid KEM (FIPS 203) and refuses classical-only peers. Library/package seam today — no shipped gateway binary (§6); the wired ingest path is REST POST /v1/ingest behind the proxy. |
Sensors / site aggregators |
| Cluster mesh | EVENTLAKE_CLUSTER_ADDR (optional) + EVENTLAKE_CLUSTER_TLS_{CERT,KEY,CA} |
Gossip + scatter-query listener, a distinct trust domain from the management API, serving native mTLS (2026-09-16xd, F9): a client certificate chaining to the cluster CA is required of every peer, every peer URL must be https, and an answer counts only if its certificate names the member it was addressed to. ⛔ A mesh bound beyond loopback without the three TLS keys refuses to boot. Network isolation is now defence in depth, not the control. |
Peer eventlake-api nodes only |
The proxy in front of the REST/public planes is genuinely just TLS + routing: authentication (bearer keys, deny-by-default RBAC, tenant forcing, the D2 cookie-session plane) is all enforced inside the process — a bypassed proxy exposes no unauthenticated surface, only cleartext (I8).
First time? The step-by-step first run — brain → enrol a Sensor → telemetry → first detection → case — is
docs/reference/quick-start-self-managed.md. This section is the production shape.
Two equivalent paths; both give you durable lake + control-plane replay on boot, TLS in front, graceful drain on restart, and readiness-gated traffic.
Path A — systemd (bare metal / VM — the COTS shape).
/usr/local/bin/eventlake-api (§2.1.1).deploy/eventlake-api.env.example → /etc/eventlake/eventlake-api.env. The minimum
production set:EVENTLAKE_API_ADDR=127.0.0.1:8080 # private; the proxy is the only public face
EVENTLAKE_TRUSTED_PROXIES=127.0.0.1,::1 # the address that proxy connects FROM. REQUIRED here:
# without it every client is attributed to the proxy, so
# the per-IP rate limiters share ONE bucket (§2.1.8d).
EVENTLAKE_SEGMENT_DIR=/var/lib/eventlake # durable mode ON (systemd StateDirectory)
EVENTLAKE_MEMORY_HORIZON=48h # bounded RAM (D17) — older records disk-resident
EVENTLAKE_KMS_KEK=<64 hex chars> # the root KEK — sealed state (MFA enrollment seeds) only
# reopens under the SAME key, AND it is the HKDF root of the
# enrolment, OIDC and wallet-issuer signing keys: a copy MINTS
# CREDENTIALS FOR ANY SUBJECT IN ANY TENANT. Hold it here, from a
# secrets manager. Omitted ⇒ minted at <segment dir>/root-kek
# (0600), which a production build REFUSES unless
# EVENTLAKE_KMS_KEK_ACKNOWLEDGE_ON_DISK=1; that file must be
# custodied separately, never ride a plain state backup (§2.1.8).
EVENTLAKE_ORG=<your-organisation> # the tenant every record is filed under — set BEFORE first ingest (no move)
EVENTLAKE_POSTURE=self-managed # a customer-run node: no demo estate
EVENTLAKE_SENSOR=1 # accept Sensor registration (unset ⇒ register answers 503)
EVENTLAKE_UI_ROOT=/opt/eventlake/ui # serve the portal (stage it: deploy/stage-ui.sh --posture self-managed <dir>)
EVENTLAKE_API_KEY=<from your secrets manager> # else a random admin key is minted per boot (EPHEMERAL)
EVENTLAKE_PLATFORM_KEY=<from your secrets manager> # node-admin key (connector enable); else minted per boot
EVENTLAKE_NODE_ID=site-a-brain-1 # PIN IT. Control-plane replay folds only records THIS
# observer id emitted, so changing it later makes the
# node a different brain. The boot now REFUSES rather
# than serving an empty control plane — see §2.1.8a.
EVENTLAKE_OBS=1 # structured logs + /metrics
# autonomy set — see §2.1.7 for the full recommended table
EVENTLAKE_ANALYTICS_SCHEDULE=5m
EVENTLAKE_SOURCE_HEALTH=5m
EVENTLAKE_GOVERNANCE_SWEEP=1h
EVENTLAKE_METER_SCHEDULE=24h
deploy/systemd/eventlake-api.service (hardened unit: StateDirectory=aether,
TimeoutStopSec > the 10 s drain, ProtectSystem=strict), then:systemctl enable --now eventlake-api
curl -fsS http://127.0.0.1:8080/v1/readyz # 503 until boot replay completes, then 200
deploy/Caddyfile):api.eventlakes.com {
reverse_proxy 127.0.0.1:8080
}
Caddy provisions the certificate itself (ACME), or use tls /path/cert.pem /path/key.pem with your
Certificates/CLM-issued pair; for nginx the equivalent is a standard proxy_pass vhost. Done: durable,
TLS-terminated, restart-safe — systemctl restart eventlake-api drains in-flight requests, flushes the
durable tail, and the next boot replays cases/holds/connector state from the segment store.
Path B — Docker Compose (lab → small production). deploy/docker-compose.yml stands up eventlake-api
(built from deploy/Dockerfile, durable volume, readiness healthcheck) behind a Caddy TLS front
(self-signed tls internal out of the box — swap in your domain/cert for real use):
docker compose -f deploy/docker-compose.yml up -d
docker compose -f deploy/docker-compose.yml logs eventlake-api # the minted admin key prints ONCE here
curl -k https://localhost/v1/healthz
The compose file is the readable reference for the container contract: /data is the segment dir (named
volume — the lake survives down/up), the healthcheck polls /v1/readyz, and stop_grace_period
respects the drain.
Durable mode is one env var: EVENTLAKE_SEGMENT_DIR=<dir>. Everything below is its operational
behaviour — verified by the platform's own tests and the D4/D17 measurements in
docs/reference/build-status.md. The store's format and semantics belong to
01-substrate-plane.md#lake-storage-engine; what follows is how to operate it.
Write path. Every accepted event is written through to the hot-segment store before the call returns
(strict durable-on-return). EVENTLAKE_SEGMENT_GROUP_COMMIT=1 opts into the licensed group-commit shape —
fsync deferred over a 256-event / 5 ms window (measured −3.7 % durable tax at baseline; the trade is an
≤ 5 ms / 256-event loss window on power failure). Choose it deliberately.
Boot. The process replays the segment store before serving: the control plane (cases, legal holds, connector state, plan assignments, honeytokens, revocations, shred obligations — D22) is rebuilt from sealed events, and the boot line reports exactly what happened:
eventlake-api: durable lake at /var/lib/eventlake — replayed 113100 events
(86000 disk-resident, 42 cases, 3 holds, 12 connectors, 0 unverified) in 1.9s
unverified > 0 is an integrity signal, not noise (I4): records that failed seal verification are
kept + flagged, never silently dropped. Check platform-keys.json before trusting reads./v1/readyz stays 503 until replay completes (I3). Liveness (/v1/healthz) is separate and always on.Key custody. platform-keys.json in the segment dir persists every boot's seal-verify public key,
so records sealed by prior process incarnations verify forever. It must live and travel with the
segments — a segment backup without it replays as unverified (I5).
The root KEK (root-kek, or EVENTLAKE_KMS_KEK). Durable mode also needs a durable root of trust, not
just a durable lake. Some control-plane state is a secret the platform must hold to do its job — above
all MFA enrollment seeds (a TOTP seed is a SYMMETRIC credential: the server must keep it to
verify a code). Those events ride the lake sealed under the KMS root KEK, so:
A node that seals durable state must reload the SAME KEK on every boot. Ephemeral nodes mint a fresh KEK per process; a node that did that would find every enrollment it had sealed unopenable after the very restart durability exists to survive — and every MFA/VPN user locked out.
So durable mode persists it: EVENTLAKE_KMS_KEK (64 hex chars) if you supply one from your secrets manager —
the recommended shape, since it keeps the KEK out of the lake directory entirely — else a root-kek
file (32 bytes, 0600) minted beside the segments on first boot. A malformed/short keyfile fails the
boot loudly rather than minting a replacement (I7), because silently re-keying would orphan every sealed
seed on disk. Losing or rotating the KEK is not recoverable: subjects re-enroll. Production points the
KMS at an HSM / cloud KMS / Vault instead (assembly.WithKMSBackend). The identity half of this story is
04-identity-and-policy-plane.md#verichain-platform.
Bounded RAM (D17). EVENTLAKE_MEMORY_HORIZON=<duration> (e.g. 48h) keeps only that much history
RAM-resident; older records stay disk-only and reads union them in transparently. Boot replay honours
the horizon (the disk-resident count). This is the footprint-SLO lever for long-retention nodes — the D17
soak shows bounded mode oscillating instead of growing monotonically.
The runtime roll janitor (EVENTLAKE_ROLL_EVERY, added 2026-07-17 — newer than the original runbook). The
horizon alone bounds RAM at boot. Keeping it bounded while the process runs is the always-on roll
janitor: default 5 m, 0 disables, and without it nothing rolls at runtime → unbounded RSS. It
costs one lock-and-walk per tick. Treat the pair as one control: EVENTLAKE_MEMORY_HORIZON states the window,
EVENTLAKE_ROLL_EVERY is what keeps the process inside it.
Shutdown. SIGTERM/SIGINT triggers the D21 graceful sequence: stop accepting → drain in-flight
requests (10 s budget) → close the spine (releases Messages WAL handles) → close the segment store last
(flushes any deferred group-commit sync). Give the process ≥ 15–20 s before SIGKILL (TimeoutStopSec=20 in
the unit; terminationGracePeriodSeconds: 30 in k8s).
Crypto-shred keystore (EVENTLAKE_SHRED_DIR). When crypto-shredding (N13) is on, the per-subject DEK
keystore (shred-keys.json, 0600, atomic rewrites) is the destruction point. Operational rules: point
it at protected storage, and keep it OUT of the ordinary backup set (§2.1.8) — a keystore backup is a
shred-undo vector. A malformed keystore fails the boot loudly (starting empty would orphan every
existing ciphertext) — intended fail-closed behaviour, not a bug (I7).
Lead-plane erasure keys (lead-keys.json). The lead plane (POST /v1/leads/demo → the §11-6/§11-7
CRM boards) stores a prospect's email, phone and name as per-lead shred tokens, so the same destruction
point exists for it — and it is always on, not opt-in like EVENTLAKE_SHRED_DIR: a demo request is
personal data held on a stated consent basis and must be erasable in every deployment. In durable mode
eventlake-api opens lead-keys.json in the segment dir automatically (EVENTLAKE_LEAD_KEYS_DIR moves it to
different or more protected storage); an ephemeral node uses an in-memory keystore, whose lifetime then
matches the lake it protects. Same two rules as the shred keystore: out of the ordinary backup set
(restoring it undoes an erasure — boot replay re-destroys the key generation, and the discipline still
belongs here), and a malformed file fails the boot loudly rather than silently orphaning every lead
ever received.
| Endpoint | What | Notes |
|---|---|---|
GET /v1/healthz |
Liveness | Always on, unauthenticated (probes carry no secrets) |
GET /v1/readyz |
Readiness | 503 until boot (incl. durable replay) completes; then boot facts an orchestrator can sanity-check |
GET /metrics |
Platform self-observability | With EVENTLAKE_OBS=1: assembly_ingest_events_total, detect_alerts_total, actions_decisions_*_total, store_integrity_failures_total, segstore_persist_errors_total, store_persist_errors_total, assembly_analytics_round_*, … (each family named for a package §13 Kind D renamed is ALSO published under its retired spine_/lake_/conductor_/crucible_ spelling, # HELP … DEPRECATED, same value, until an owner date — 2026-10-02ah; write new rules on the current names) structured logs to stderr (EVENTLAKE_OBS_LEVEL=debug\|info\|warn\|error) |
The two instruments that alarm on a SILENT failure (2026-08-22). Everything else on
/metrics reports work that happened; these two report work that quietly stopped happening,
which is the only kind of failure this platform cannot otherwise surface.
| Instrument | Reads | Alarm on |
|---|---|---|
store_persist_errors_total |
failed segment-store write-throughs, counted at the lake's SegmentStore seam (segstore_persist_errors_total counts the same failures inside the default store; this one also covers a non-segstore store, and is the scrapable twin of Lake.PersistErrors() that the resilience drill reads) |
any non-zero. ⛔ The record is kept in RAM and POST /v1/ingest still answers 202 {"accepted": n} — the ingest ack is not a durability ack, so a producer that trusts it has already advanced its cursor over records with no durable copy. |
assembly_analytics_round_seconds (histogram) · assembly_analytics_round_millis_last (gauge) |
how long the last continuous-detection round took. The gauge exists because the shared bucket ladder tops out at 10 s and a real multi-tenant round can sit past it | graph it; it is the input to the two below |
assembly_analytics_round_overrun_total |
rounds that took ≥ the interval they were scheduled on | capacity warning. StartAnalytics runs on a time.Ticker, which DROPS a missed tick rather than queueing it, so an overrunning round does not fall behind visibly — it just runs less often. Arms run serially within a tenant AND the tenants run serially, so this grows with both the arm count and the estate. |
spine_arm_<name>_millis_last |
how long each of the 43 correlation arms took in the last round it ran | the follow-up question. The round-duration gauge says a round is slow; this says WHICH arm. It exists because the arms became a registry (assembly/armregistry.go) rather than 43 hard-coded dispatch lines — per-arm timing was not previously expressible. An arm absent from the scrape did not run: either its tenant's monitoring profile does not cover its purpose, or the round never reached it. ⛔⛔ Above one worker this is a WEAKER reading than it looks — with EVENTLAKE_ANALYTICS_PARALLELISM raised it is wall time under contention, written by whichever TENANT finished that arm last, so read it beside assembly_analytics_workers or not at all. |
assembly_analytics_workers |
the number of tenants whose rounds ran CONCURRENTLY in the last round (1 = sequential, the default) | not an alarm — a decoder for the two rows above. An operator who does not know the degree cannot interpret the per-arm gauge. It also confirms a raised EVENTLAKE_ANALYTICS_PARALLELISM actually took effect: it is capped by the tenant count, so a one-tenant estate reports 1 no matter what was configured. |
assembly_analytics_workers_clamped_total |
the configured parallelism exceeded 2×GOMAXPROCS and was reduced | configuration warning. The setting is not doing what it says. The round is allocation-bound (docs/reference/analytics-round-parallelism.md), so the clamp costs nothing in throughput — but a deployment reading round cost as evidence about the number it set is reading the wrong experiment. |
assembly_analytics_window_widened_total · assembly_analytics_window_millis_last |
rounds whose lookback had to grow past -analyze-window to reach back to the previous round's start, and how wide it grew |
watch. ⭐ The gap is REPAIRED, not merely reported: -analyze-window is a floor, and a tick that starts late widens its lookback to the spacing that actually occurred, so no slice of the stream can go uncorrelated. What this counts is the cost — a widened round scans more, which makes the next one slower still. Nothing is clamped; a cap would restore the silent gap. |
assembly_analytics_window_runaway_total |
the lookback passed 8× the configured window | ⛔⛔ page. The widening has stopped being slack and become a spiral. Remedy: on a MULTI-TENANT estate raise EVENTLAKE_ANALYTICS_PARALLELISM first (a bounded fan-out over tenants — docs/reference/analytics-round-parallelism.md; ⛔ it buys nothing on a single-tenant estate, and near-nothing on one dominated by a single tenant, because a round is only as fast as its largest tenant); otherwise lengthen -analyze-every, shard the estate, or reduce the armed arm set. |
assembly_analytics_round_uncovered_total |
the lookback did not reach back to the previous round's start | ⛔⛔ page — and suspect the platform, not the estate. Since the lookback widens automatically this is structurally unreachable, so it is an assertion on the wire: if it ever ticks, the repair itself is broken and a slice of the stream really was correlated by no round. An open-ended round (window ≤ 0) re-reads the whole lake and can never raise it. |
access_denials_suppressed_total · access_denial_rollups_total |
denials this node COUNTED but did not write to the lake individually, and the rollup events that stand for them (conduit/assembly/denialburst.go). ⭐ The per-PEP access_denials_<pep>_total counters remain EXACT regardless — they increment before admission, so the metric plane is the lossless channel and the event stream is the sampled one. A surface that needs the true refusal rate must read the counter, not count events. |
not a fault — a probe indicator. Non-zero means refusals are arriving faster than the record plane writes them (60 per offender per minute, 600 node-wide), which is the bound working; the node read raises denials_folded and says so. Sustained non-zero on a node nobody is testing = someone is scanning the endpoint. ⛔ Do NOT tune this down to silence it: the allowance is pinned ABOVE the three threshold rules that read the denial stream, and a cap below one of them makes that rule unreachable with nothing reporting the loss. |
Out-of-band paging: EVENTLAKE_ALERT_NOTIFY=<severity floor> pages detections at/above the floor through the
configured notifier (webhook and/or SMTP and/or FCM — they compose), throttled per rule+entity
(EVENTLAKE_ALERT_NOTIFY_THROTTLE, default 5 m). The SOC hears about a critical in seconds, not from the case
queue.
Demand paging is the same seam pointed at the other kind of urgency. EVENTLAKE_LEAD_NOTIFY_TO=<recipient>
routes a NEW public-site lead through the same composed notifier, so the site's "a solutions engineer will
reach out within one business day" is kept by the platform rather than by an operator's habit of opening the
control-plane demand inbox — a board behind mTLS and an IP allowlist. Presence is the opt-in; there is no floor
to set, because a deployment running a public form wants every lead.
⛔ The message carries NO contact detail, and that is a correctness property. The email, phone and name are
sealed under the lead's own DEK so an erasure destroys them everywhere they exist; a mailbox is the one copy that
key cannot reach, so mailing the address out would silently turn a DPDP §12 / GDPR Art.17 guarantee into a false
one. The notification carries the reference, company and source — the fields that survive an erasure anyway —
and points at the board. ⛔ Outbound volume is bounded (20 per hour, then ONE "intake is flooding" notice
naming the board, then silence for the rest of the window): POST /v1/leads/demo is open and unauthenticated,
so one-mail-per-lead is an amplifier a stranger holds the throttle on. Nothing is lost — every submission is
still stored and on the board. conduit/assembly/leadnotify.go.
The health probes in §2.1.5 say the process is up. They do not say the node will take a record and
hand it back. This is the smallest proof that it does: one event in through POST /v1/ingest, the same
event out through GET /v1/events, and — on a durable node — the same event again after a restart.
Every output below is a real capture (2026-10-15, macOS arm64, a go build of cmd/eventlake-api
from main, durable mode on a scratch segment directory, a free loopback port). Only the minted key is
redacted; the scratch path is written <dir>.
1 · Start the node. Durable mode is one variable; leave it out and the same steps work in RAM only (a restart is then a clean slate, so step 5 does not apply).
EVENTLAKE_API_ADDR=127.0.0.1:18414 EVENTLAKE_SEGMENT_DIR=<dir> ./eventlake-api
# eventlake-api: durable lake at <dir> — replayed 0 events (0 disk-resident, 0 cases, 0 holds, 0 connectors, 0 unverified; graph 0 events + 0 dep edges) in 0s
# eventlake-api: generated admin API key (default tenant) → <dir>/boot-credentials
# eventlake-api: listening on 127.0.0.1:18414 (try: curl -H 'Authorization: Bearer <key>' http://127.0.0.1:18414/v1/healthz)
⚠ The minted key is not on stdout. With EVENTLAKE_API_KEY unset the node writes the generated
bootstrap-admin key (admin, tenant default) to <dir>/boot-credentials (mode 0600) and prints only
the path; it is different on every boot until you pin it (§4.3 lists which keys to pin from custody).
Read it from the file for this walk-through:
K=$(grep -A1 'bootstrap-admin' <dir>/boot-credentials | tail -1 | tr -d ' ')
curl -s http://127.0.0.1:18414/v1/readyz
# {"cluster_alive": 1, "connectors_enabled": 0, "status": "ready"} HTTP 200
2 · Ingest one event. The body is a batch — an object with an events array — and the event
classification nests under class:
curl -s -X POST http://127.0.0.1:18414/v1/ingest -H "Authorization: Bearer $K" \
-H 'Content-Type: application/json' \
--data '{"events":[{"time":"2026-10-15T15:50:00Z","class":{"category":"authentication","type":"logon","activity":"failure","domain":"it"},"subject":{"kind":"user","ref":"first-event-user"},"src":{"ip":"203.0.113.7"},"attributes":{"note":"first event back"}}]}'
# {"accepted": 1, "durable": true} HTTP 202
Assert accepted equals the number of events you sent — an empty batch is legal and answers
accepted: 0. A bare event without the wrapper is refused, not silently dropped:
{"error": "the body must be a batch object with an \"events\" array; a bare event or any other object is not a batch (nothing was ingested)",
"message_id": "ingest.events_wrapper_required"} HTTP 400
3 · Read it back.
curl -s "http://127.0.0.1:18414/v1/events?category=authentication&limit=5" -H "Authorization: Bearer $K"
{
"count": 1,
"events": [
{
"id": "01a14040-28f2-7824-a748-ccd12eaffac7",
"schema": "",
"time": "2026-10-15T15:50:00Z",
"ingest_time": "2026-10-15T15:48:29.042532Z",
"observer": { "id": "bootstrap-admin", "kind": "gateway" },
"class": { "domain": "it", "category": "authentication", "type": "logon", "activity": "failure" },
"severity": "info",
"subject": { "kind": "user", "ref": "first-event-user" },
"src": { "ip": "203.0.113.7" },
"attributes": { "ingress_bytes": 64, "note": "first event back" },
"labels": { "masked": "none", "tenant": "default" }
}
]
}
(HTTP 200; whitespace compacted, fields as served.) What the node added is the part worth reading:
an id, an ingest_time, the observer stamped from the principal that sent it (not from the
body), ingress_bytes, and labels.tenant — tenancy is forced from the key, so the same query with a
key from another tenant cannot see this record. The same read with no key answers HTTP 401.
4 · No alert is expected. A single failed logon fires nothing: the windowed correlation arms run
on POST /v1/analyze/*, on EVENTLAKE_ANALYTICS_SCHEDULE, or with EVENTLAKE_STREAMING=1 (§2.1.7).
"I ingested and see no alert" is that, not a fault.
5 · Restart, read it again (durable mode only). Stop the process and start it with the same
EVENTLAKE_SEGMENT_DIR. The boot line now counts the replay, and the read returns the same id
(a new key is minted on this boot, so re-read boot-credentials first):
eventlake-api: durable lake at <dir> — replayed 8 events (0 disk-resident, 0 cases, 0 holds, 0 connectors, 0 unverified; graph 8 events + 0 dep edges) in 2ms
"count": 1, "id": "01a14040-28f2-7824-a748-ccd12eaffac7", "ref": "first-event-user", "note": "first event back" HTTP 200
Eight, not one: the lake also holds the node's own records from the first boot (the keyless read in
step 3, for one, is written back as it/security/access-denied telemetry). 0 unverified is the
line to watch — a non-zero count means a prior boot's verify key is missing from
platform-keys.json (§2.1.4). ⛔ 202 accepted alone is not a durability ack (§2.1.5, store_persist_errors_total). The
durable field is the one to read: true only when the node has a segment store AND holds no
RAM-only record (conduit/api/ingestdurability.go); this restart is the end-to-end proof.
Every listener is an api.HardenedServer (D21): ReadHeaderTimeout 5s (slow-loris cut), ReadTimeout
60s, WriteTimeout 120s, IdleTimeout 120s, MaxHeaderBytes 1 MiB. Auth is deny-by-default RBAC over
bearer keys; tenancy is forced server-side; the open enrollment plane is rate-limited (D3). None of this
needs configuration — it is the floor, not an option.
Everything below is off by default (I9); production turns on what it needs. All intervals are Go durations; every loop stops cleanly on the signal context.
| Env | Recommended | What it runs |
|---|---|---|
EVENTLAKE_ANALYTICS_SCHEDULE (+_WINDOW) |
5m |
The windowed correlation arms (UEBA·flow·fraud·ICS·DNS·CIEM·CloudSec) per tenant — detections fire continuously |
EVENTLAKE_BAS_SCHEDULE (+_MAX_AGE) |
24h |
The NATIVE breach-and-attack-simulation plane: pull adversary content, run it through the live arms, adjudicate every published coverage claim. Off by default. Inert without at least one source below |
EVENTLAKE_BAS_TELEMETRY_DIR |
— | The strongest source: OTRF Security-Datasets — RECORDED intrusions decoded by this platform's own production decoders, so a hit proves decoder → arm → detection rather than a string match |
EVENTLAKE_BAS_ATOMIC_DIR |
— | Atomic Red Team — 1,683 command records over 323 techniques; the corpus declares its own ATT&CK id |
EVENTLAKE_BAS_TELEMETRY_DIR |
— | The strongest source: OTRF Security-Datasets — RECORDED intrusions decoded by this platform's own production decoders, so a hit proves decoder → arm → detection rather than a string match |
EVENTLAKE_BAS_ATOMIC_DIR |
— | Atomic Red Team — 1,683 command records over 323 techniques; the corpus declares its own ATT&CK id, so nothing here translates a taxonomy |
EVENTLAKE_BAS_CRS_DIR · EVENTLAKE_BAS_NUCLEI_DIR · EVENTLAKE_BAS_EXPLOITDB_DIR |
— | Local clones the operator vetted. Nothing is vendored; nothing is executed |
EVENTLAKE_BAS_EXPLOITDB_LIVE (+_URL) |
1 |
Pull the live Exploit-DB index over HTTPS each cycle — the "newly released exploits" leg. Opt-in a second time on purpose: the directory form reads a tree the operator reviewed, this one trusts a mirror at run time |
EVENTLAKE_EXPLOIT_AVAILABILITY_DIR |
— | Arms vuln's THIRD exploitability signal from an Exploit-DB index — "a working exploit is published" (CVSS E:P / SSVC poc), distinct from KEV (observed) and EPSS (predicted). 25,069 CVEs as measured 2026-08-22 |
EVENTLAKE_EDGE_BENIGN_CORPUS |
— | An access-log directory: the benign HTTP corpus every BLOCKING edge rule is replayed against, through the edge's own evaluator, before a staged rollout widens it; one blocked line refuses the promotion (rollout.replay.false_positive, naming the rule and file:line). Unset refuses every widening (rollout.replay.no_corpus) — fail-closed, 2026-10-22g row 473 |
EVENTLAKE_STREAMING=1 (+_WINDOW, _LATENESS) |
on | N5 streaming twins — completing-hop firing; batch stays the backfill net |
EVENTLAKE_SOURCE_HEALTH |
5m |
Log-source-silence monitor (source-silent → alert → case; recovery event on return) |
EVENTLAKE_GOVERNANCE_SWEEP (+_GRACE) |
1h |
DPDP §12 storage-limitation + §13 grievance-overdue duties (+ per-purpose retention, N13) |
EVENTLAKE_EXPOSURE_WATCH |
unset (OFF) | Breach-exposure notification pass (row P family B, 2026-08-20n). ⛔ The one scheduler here that is off by default on purpose, and not out of conservatism: a pass SENDS MAIL TO REAL PEOPLE. Every other entry in this table moves platform state; this one puts a message in a private individual's inbox. A subject's first pass seeds silently, so switching it on does not mail the estate about breaches from years ago. Manual form: POST /v1/platform/exposure/watch |
EVENTLAKE_METER_SCHEDULE (+_WINDOW) |
24h |
Sealed it/platform/usage license rollup per tenant (07-commercial-and-gtm.md#licensing-model) |
EVENTLAKE_VULN_FEED (+_KEV_URL, _EPSS_URL) |
24h |
CISA KEV + FIRST EPSS pulled and hot-swapped into VA prioritization (needs outbound HTTPS) |
EVENTLAKE_VULN_FEED_HEALTH (+_STALE_AFTER) |
1h |
Edge-triggered feed-stale → alert (a silently stale exploit feed is surfaced) |
EVENTLAKE_REPORT_SCHEDULE |
1h |
Renders due report schedules (registered via POST /v1/reports/schedules) |
EVENTLAKE_DECEPTION_ROTATE (+_MAX_AGE) |
1h / 168h |
Honeytoken freshness rotation (durable, D22) |
EVENTLAKE_MYTHOS_SCHEDULE (+_WINDOW) |
1h |
Attack-path hypothesis refresh — prediction only, never auto-runs Validation |
EVENTLAKE_VULN_POSTURE_SNAPSHOT / EVENTLAKE_HARDENING_POSTURE_SNAPSHOT / EVENTLAKE_GOVERNANCE_POSTURE_SNAPSHOT (+lookbacks) |
24h |
Sealed trendable posture snapshots (exposure / CIS-STIG drift / DPO picture) |
EVENTLAKE_AGENTLESS_VA |
6h |
Agentless vulnerability assessment (P5, D-VA5): the brain evaluates the loaded definitions against every CMDB host no agent covers, from that host's reported software. Findings ride the same it/vulnerability/finding contract; each swept host gets an assessment record marked agentless, because a pass that abstains on every non-software predicate makes a weaker claim than a Sensor sweep. Unset = on-demand POST /v1/vuln/agentless only. Do not set it faster than the inventory refreshes |
EVENTLAKE_INTEL_TAXII_POLL (+_URL, _COLLECTION, _APIROOT, _TOKEN, _SOURCE) |
15m |
Incremental TAXII 2.x pull into the intel store (more feeds via POST /v1/intel/feeds) |
EVENTLAKE_FORWARD_TICK |
2s (default — ON) |
Continuous-egress drain (W40): registered downstream forwarders (POST /v1/forwarders — webhook/syslog pipes to a coexistence SIEM or MSSP uplink, durable D22) deliver every tick; 0 disables (queues drop oldest, audited). Registering a forwarder is the opt-in, so this one runs by default |
Enrichment & delivery seams (set when used): EVENTLAKE_GEOIP=<csv> (operator-loaded IP→geo table — EventLake
bundles no geolocation dataset; a bad table fails boot loudly) · EVENTLAKE_WEBHOOK_URL/_SECRET ·
EVENTLAKE_SMTP_ADDR/_FROM/_USER/_PASS/_TLS/_AUTH/_TENANT_FROM · EVENTLAKE_FCM_CREDENTIALS
(content-less mobile wake-ups) · EVENTLAKE_SHRED_DIR (N13 crypto-shred) · EVENTLAKE_MESSAGING_DIR (Messages E2E
commit log; defaults under the segment dir) · EVENTLAKE_SENSOR=1 (real actuation via the Sensor, §2.1.10).
Tooled since
2026-09-16sq:eventlake-ctl node backup | restore | backup-keyproduce and restore an encrypted, integrity-checked archive of the brain's state root (segments,platform-keys.json,root-kek), refuse a live store, restore all-or-nothing into an empty root, and EXCLUDE the shred/lead/BYOK keystores so a restore cannot resurrect an erased key. Procedure, custody and the measured restore drill:docs/reference/backup-and-restore.md. The manual procedure below remains valid for volume snapshots.
The backup set (all of it, atomically consistent as a unit):
EVENTLAKE_SEGMENT_DIR — the lake segments plus platform-keys.json (they are one unit; segments
restored without the key file replay as unverified).EVENTLAKE_KMS_KEK in your secrets manager, or the root-kek file in the segment dir.
Segments restored without it still replay, but every sealed record stays shut: all MFA
enrollments are lost and every subject must re-enroll (MFA/VPN lockout). Back it up.EVENTLAKE_MESSAGING_DIR if you pointed it outside the segment dir (default is nested → covered).Explicitly excluded: the EVENTLAKE_SHRED_DIR keystore, the lead-plane lead-keys.json (§2.1.4 —
same rule, and it exists on every durable node whether or not crypto-shredding is on), and, by the same
rule, EVENTLAKE_BYOK_DIR (§2.1.12). A backed-up keystore is an undo vector for provable erasure. The platform defends in depth — boot
replay verifies sealed shred events and re-destroys keys that a stale keystore restore would resurrect
(trust-gated, D22) — but the discipline is: erasure keys don't go in backups.
Do not generalize that exclusion to all key material — the two keys are opposites (I6). The shred keystore is excluded because restoring it undoes an erasure; the root KEK must be INCLUDED because losing it erases every enrollment. Shred keys protect a promise to FORGET; the KEK protects the ability to REMEMBER. Treat them separately (different stores, different policies).
A THIRD policy: the agent trust root is not in this backup set at all. It is neither included like the
KEK nor excluded like the shred keystore — it is custodied somewhere else entirely, because a backup this
host can reach is a backup an attacker who owns this host can reach, and the whole point of the root is that
owning the brain does not yield it (§2.1.8b). The brain-held half (EVENTLAKE_BRAIN_SIGNING_KEY +
EVENTLAKE_BRAIN_SIGNING_GRANT) may ride the ordinary backup, but does not need to: losing it costs one
signing ceremony, not a fleet.
What it protects. A Sensor agent takes its feed-verify key from the brain at register time, and four
signed channels hang off that pin (VerifyHardenBundle, VerifyHardenLease, VerifyFeed,
VerifyDLPFeed). While an operator typed the brain URL on every install, the install itself rooted that
pin. Once agents carry a compiled public endpoint, whoever wins that name — DNS or BGP hijack, one
mis-issuing CA, a lapsed renewal — could otherwise answer register, define the pin, and sign hardening
bundles that verify on every host in the fleet. The Sensor carries -enforce-kill, -enforce-isolate,
-enforce-exec and -enforce-harden, so the blast radius is fleet-wide destructive action. The hostname
and the root ship together, or neither ships.
Three keys, and only one of them is precious:
| Key | Lives | If lost | If stolen |
|---|---|---|---|
| Root (private) | OFFLINE machine, never on a brain host | Every agent needs a re-signed build | The fleet's trust anchor is gone — treat as a full re-key |
| Brain signing (private) | The brain (EVENTLAKE_BRAIN_SIGNING_KEY) |
One signing ceremony | Forged feed keys until the grant expires — recovery is a new ceremony, not new agent binaries |
| Signing grant | The brain (EVENTLAKE_BRAIN_SIGNING_GRANT), served to every agent |
Re-sign it | Public material; it authorizes nothing on its own |
That middle row is why the grant is bounded and an unbounded one is refused outright
(mote.ErrSigningGrantUnbounded): the expiry is the only thing that makes a brain-host compromise
recoverable rather than permanent.
The ceremony. eventlake-trustroot (conduit/cmd/eventlake-trustroot). Steps 1 and 3 run on the offline
machine; step 2 can run anywhere.
# 1. ONCE, OFFLINE. root.key never leaves this machine.
eventlake-trustroot gen-root -out root.key
# → prints the PUBLIC half and the exact -ldflags line to stamp into agents.
# 2. Mint the keypair the brain will hold (brain host, or anywhere).
eventlake-trustroot gen-signing -out brain-signing.key -pub-out brain-signing.pub
# 3. OFFLINE again — carry brain-signing.pub to the root and authorize it.
eventlake-trustroot sign -root root.key -pub brain-signing.pub \
-id brain-signing-1 -days 365 -out brain-signing.grant.json
Then on the brain (/etc/eventlake/eventlake-api.env):
EVENTLAKE_BRAIN_SIGNING_KEY=/etc/eventlake/brain-signing.key # 0600
EVENTLAKE_BRAIN_SIGNING_GRANT=/etc/eventlake/brain-signing.grant.json
Startup prints agent trust root ACTIVE — signing key "…" valid until …. A brain that is
half-configured, holds a mismatched pair, or holds an expired grant refuses to start — every one of those
otherwise surfaces only as "agents mysteriously will not register", on someone else's machine.
Then, and only then, stamp agents — in the SAME build as the endpoint:
go build -ldflags "-X main.defaultBrain=https://api.eventlakes.com \
-X main.rootKeyID=root-1 -X main.rootKeyB64=<root public key>" ./cmd/mote
Ordering is not advisory. A rooted agent pointed at a brain that does not serve a grant fails
register() and exits; under Restart=always it crash-loops. Brain first, agents second.
Renewal. Re-run steps 2–3 before NotAfter and redeploy both files. Nothing about the compiled root
changes, so no agent needs a new binary — that is the entire reason the root is not the direct signer.
An expired grant stops NEW registrations; agents already holding a pin keep verifying their feeds.
The ceremony above answers what an agent trusts. This one answers where it dials, and it is optional:
an on-prem agent is given -brain and skips the layer entirely. It exists for a fleet whose binaries compile
in a public NAME (-X main.defaultBrain) rather than a brain URL — the name answers /v1/discover, and the
answer says which brain that fleet actually uses.
Why the root signs this one directly, when it could not sign feed keys. A feed key is reminted every boot, so a root signing it would have to be permanently online — hence the intermediate. A brain URL changes almost never, so the root signs it once and the host that serves it holds no key at all. That is the point: in a SaaS deployment the discovery host answers for brains that are not itself, and a signing key there would let one compromised host re-pin every other brain's fleet. Serving a blob it could not have minted, it is worth nothing to an attacker — who can therefore serve stale bytes or none, but can never rewrite where a fleet dials.
# OFFLINE, at the same machine as steps 1 and 3 above.
eventlake-trustroot discover -root root.key \
-url https://api.eventlakes.com -days 365 -out discovery.json
Then on whichever host answers the public name:
EVENTLAKE_DISCOVERY_DOC=/etc/eventlake/discovery.json # 0644 — public material, served to every agent
EVENTLAKE_TRUST_ROOT_PUB=<root public key, base64> # OPTIONAL; see below
Startup prints endpoint discovery ACTIVE — serving <url> until <date>. Unset EVENTLAKE_DISCOVERY_DOC means
this deployment does not do discovery, and /v1/discover answers 503 — deliberately distinct from an
empty document, because a rooted agent must be able to tell "no discovery here, keep your compiled endpoint"
apart from "something answered with an unsigned document", which it refuses loudly.
EVENTLAKE_TRUST_ROOT_PUB is optional because this host is not the party whose verification is a security
control — the agent is. Supplying it only moves the discovery of a mismatch (wrong file, or a document
minted under a different root than the fleet carries) from a fleet-wide enrollment stall to this host's
startup log. Set it. The same three refusals — unsigned, unbounded, cleartext-to-a-routable-host — are made
at mint time, at brain boot, and at the agent, on purpose: each earlier one saves an air-gapped session.
Artifact-origin hosts (since 2026-10-25e). A brain that accepts release-trust.json / time-anchor.json
uploads on the artifact trust door now FAILS CLOSED: with no EVENTLAKE_TRUST_ROOT_PUB it refuses a release-trust
upload (409 artifacts.trust.root_missing), and with no EVENTLAKE_ARTIFACT_PUBLISHER_PUB (comma-separated base64
release-publisher keys, the same keys agents verify anchors with) it refuses a time anchor
(409 artifacts.trust.anchor_keys_missing). An anchor dated more than 10 minutes ahead of the brain clock is refused,
and only a file that verifies counts toward the never-go-backwards check, so a planted file cannot block later uploads.
What an agent does with it (conduit/cmd/eventlake-sensor/discover.go; full table in the spec §3.4). In order:
an explicit -brain wins outright, then EVENTLAKE_BRAIN, then a fresh cached <state>/discovery.json, then
discovery against the compiled name, then the dev default. Two operational consequences:
Renewal. Re-run the command before NotAfter and replace the file. Agents renew at half the
document's validity, so a changed endpoint propagates within one period and the rest is outage headroom. An
expired document stops agents that have neither a cached endpoint nor -brain; agents already holding one
keep dialling it.
The trust root (§2.1.8b) and the routing statement (§2.1.8c) decide which brain an agent trusts. This
decides what answers when it dials. Four hostnames share one IP; configs in deploy/nginx/
(eventlake.conf + eventlake-tls.inc + eventlake-core.inc + eventlake-ops-allow.inc.example), install/cutover/DNS
in that directory's README.md.
| Hostname | Audience | Surface |
|---|---|---|
eventlakes.com, www. |
anyone | the static marketing site, served by nginx — plus exactly one open brain path, GET /v1/metering/rate-card |
api. |
the Sensor + EventLake Authenticator + EventLake Command | the dataplane allowlist only: the agent routes cmd/mote dials + the subject routes conduit/api/public.go names. The management plane is not reachable here |
console. |
customer (tenant) admin | full proxy minus the three OEM-only paths; the superadmin kits 404. PUBLIC since 2026-07-29 (§2.1.8e) |
ops. |
superadmin (SOC + OEM) | full proxy behind two legs — a required IP allowlist and a client certificate (§2.1.8e) |
This replaced the :443 customer / :8443 superadmin port split. 8443 is blocked on exactly the
networks the Sensor runs inside (corporate egress allowlists, guest wifi, mobile carriers), cookies are not
port-scoped so the split isolated nothing, and the marketing site and ops. move to another hosting
provider later while keeping their URLs — which only a hostname can promise. api. is the one value
compiled into every agent and app binary (D8), so it is the most exposed name and carries the least
surface. Tenancy still derives from the key, never the Host (D11): console. is one hostname for
every customer.
Two rejecting defaults, because nginx makes two independent choices. It picks the server block
from the Host header and the certificate from SNI. The old config's three server_name _; blocks
meant any hostname aimed at the address landed on whichever nginx defaulted to — the customer Console.
Closing that takes both halves: ssl_reject_handshake on on the :443 default_server refuses an
unmatched or absent SNI before any certificate is presented (the one legal listen … ssl block with no
certificate), and return 444 closes the other door — SNI api. with Host: something-else is a
real handshake routed to the default server by name, and a default server with no return falls through
to nginx's built-in root and answers. A caller sending no SNI is refused too; that is intended.
The ops. allowlist is a required include, not a comment. eventlake.conf ends with
include conf.d/eventlake-ops-allow.inc; and that file is not in git — a missing include fails nginx -t
loudly, whereas a wildcard include matching nothing succeeds silently and would serve the OEM control
plane to the internet. Hostnames are not a control: every certificate is published to Certificate
Transparency, so ops. is enumerable minutes after issuance. The control is the allowlist, plus
platform-role RBAC and step-up.
Four certificate paths, currently one 5-SAN leaf (updated 2026-09-01). The per-name
ssl_certificate indirection is kept — /etc/eventlake/tls/<name>/{fullchain,privkey}.pem,
issuer-agnostic — but all four point at one certificate. It is no longer a wildcard: the zone
moved from the hand-operated RailTel servers to Oracle Cloud DNS, :80 opened, and issuance
moved to certbot over HTTP-01 — a leaf named eventlake carrying apex · www. · api. ·
console. · ops.. The wildcard existed to dodge a five-name manual DNS ceremony that no longer
exists, so the reason for it is gone.
The four front directories now reach that one lineage by symlink into
/etc/letsencrypt/live/eventlake/, which is what keeps renewal an nginx no-op: certbot renews in
place and a deploy hook reloads. The property given up is unchanged — moving ops. to another
provider would carry only that name's key — and the property kept is now stronger: reversing it is a
second certbot lineage plus re-pointing one symlink, not an install-script rewrite and not an nginx
change. ⚠ Issuance is NOT deploy/acme/ any more — those scripts target the retired
nameservers and refuse to run; see deploy/acme/README.md.
Client attribution — set EVENTLAKE_TRUSTED_PROXIES on any proxied node (closed 2026-07-28). The
brain honours X-Forwarded-For only from a peer the deployment has declared to be one of its
proxies: put EVENTLAKE_TRUSTED_PROXIES=127.0.0.1,::1 — the address nginx connects from — in
/etc/eventlake/eventlake-api.env. Unset is the safe default (the header is ignored and the transport peer
is used, exactly as before the option existed), which is correct for a directly-dialled node and
wrong for this one.
The gate is the security property. X-Forwarded-For is a request header, so without an independent
statement of which peers are ours it is the caller naming its own source; with the declaration, the
chain is walked right-to-left, because nginx's $proxy_add_x_forwarded_for appends the address
it saw — a client that pre-seeds the header only pushes its lie further left, and the first hop from
the right that is not a declared proxy is the client. Any anomaly (undeclared, untrusted peer,
unparseable hop, an over-long chain, or no untrusted hop at all) falls back to the TCP peer, the one
value nobody can forge. Declaring a range that is not a proxy is the way to reintroduce spoofing.
Leaving it unset behind the proxy is silent rather than broken-looking, which is why it matters:
clientIP() is the key of the per-origin token buckets (enrollment, session mint, VDP intake)
as well as the Source on denial telemetry, so every client collapses onto 127.0.0.1 — the per-IP
limiter becomes a second global one that a single flooder can hold shut (defaultSessionMintPerIP
is burst 10 / 2 per second for the whole internet), and the it/security/access-denied stream
feeding the burst/spray detectors pools every source on loopback. The brain emits one WARN naming
the variable the first time it sees an unhonoured header, so a missed cutover step shows up in
journalctl -u eventlake-api instead of never.
bash ui_kits/check.sh guard 9 asserts these properties offline — no stray catch-all, both defaults
reject, per-host certs, the required allowlist include, the api. path list still agreeing with
conduit/api/public.go in both directions, and (2026-07-29) the ops. block still demanding a
client certificate and a CRL plus console. still 404-ing the three OEM-only paths. The
certificate directives are matched anchored, because this config explains them in its own
comments and a substring match would be satisfied by the prose — or by a commented-out directive,
which is exactly how a leg gets disabled "temporarily". It does not replace nginx -t on the box.
console. and the second leg on ops. (2026-07-29)Owner decision, applied to the live node as one change because each half is the reason for the
other: console.eventlakes.com is public (allow all;, A record 2026-07-30) and
ops.eventlakes.com now requires a client certificate in addition to its IP allowlist.
Why they are one change. Until now console. and ops. were both VPN-only, so the OEM
control plane was not internet-reachable by any door. Opening console. leaves the ops.
allowlist as the only thing between the superadmin plane and the open internet — on a node whose
perimeter admits any source in India, i.e. a very large set of hosts rather than a restriction.
An allowlist answers which network; it cannot tell two hosts on the admin LAN apart, and behind
this node's 1:1 NAT (or any L7 balancer) every client collapses to one address, at which point it
admits or refuses everyone. A certificate answers which key, travels with the operator, and is
revocable on its own. Two legs mean an attacker needs a position on an admin network and a
private key that never left an operator's machine.
The certificate is not the principal. It authorizes a channel. The operator is still the platform-role API key + session that in-process RBAC checks on every request, and nothing downstream reads the certificate. A cert on the wire is never an argument for a weaker gate behind it — the same rule as "the name split is not the security boundary".
Why the ops CA is not Certificates (pki). The certificate that lets an operator reach the plane must not
depend on the plane being reachable; bootstrapping through the brain's own PKI is a circular
dependency whose failure mode is being locked out of the surface you need to fix it. Same argument
as §2.1.8b's offline trust root. Tooling is a 300-line deploy/ops-mtls/ops-ca.sh
(init/issue/revoke/crl/list/verify); material lives in /etc/eventlake/tls/ops-ca/, and
ca.key is a fourth key-custody item alongside root-kek, the shred keystore and the trust
root — but unlike those, losing it costs only a re-issue ceremony for a handful of operators.
What opening console. deliberately did not open. eventlake.conf 404s /v1/platform/,
/v1/metering/fleet and /v1/metering/plan on the customer name — the surfaces that are
unambiguously OEM-only by URL. The rest of the Platform-gated capabilities share URLs with
per-tenant admin ones and stay behind their in-process gate. Paired controls shipped with it:
/v1/session + /v1/auth/ rate-limited on $binary_remote_addr (a floor that holds even when
EVENTLAKE_TRUSTED_PROXIES is wrong and the in-process per-origin limiter has degraded into a global
one), both layers answering 429; and nosniff + SAMEORIGIN + Referrer-Policy beside HSTS.
Three failure modes to know, each of which locks out every operator at once.
nextUpdate passes the whole plane is unreachable at the TLS layer with no UI to
explain it. Hence a 10-year nextUpdate and revocation-by-regeneration; recovery is
ops-ca.sh crl on the box.revoke returned
302; a second later, 403. For an urgent revocation prefer systemctl restart nginx.ops. request over a connection opened for console. that never carried a certificate
request. Chrome and Firefox retry on a fresh connection; curl does not. Unchanged by the move
off the wildcard (2026-09-01) — a 5-SAN leaf coalesces exactly as a wildcard did; what changed
is the fix's price. A leaf covering only ops. is now a second certbot lineage and one
re-pointed symlink, not "one more hand-driven DNS-01 ceremony".Proven on the node the day it shipped: ops. without a certificate 403s with a message naming the
fix, with one 302s; a revoked certificate 403s; console. serves and 404s all three OEM paths;
api. and the marketing face are unchanged; 10 logins pass then 429. Full recipes:
deploy/ops-mtls/README.md and deploy/nginx/README.md.
Taking the backup. Segments are append-only with atomic control-file writes, but the clean shapes are: (a) a filesystem/volume snapshot of the segment dir while running, or (b) stop-then-copy. In group-commit mode remember the ≤ 5 ms / 256-event deferred window — a snapshot is crash-consistent, i.e. as good as a power cut at that instant, which replay handles.
Restore & prove it:
systemctl stop eventlake-api
rsync -a backup/eventlake/ /var/lib/eventlake/
systemctl start eventlake-api
# boot line: replayed N events (… X cases, Y holds, Z connectors, 0 unverified)
Acceptance is the boot line: unverified must be 0 and the case/hold/connector counts must match
expectations. The whole control plane rebuilds from the sealed log — there is no second database to restore
or reconcile. RTO ≈ replay time (seconds per ~100 k events on baseline hardware; measure yours from the boot
line).
Scope note. This section is the platform's own state. Restoring customer estate (hosts, databases)
after e.g. ransomware is the N12 recovery orchestration feature (GET /v1/recovery/plan,
conductor-held restore waves) — a product capability owned by Volume 03, not this runbook.
Drill it (O3). This restore path is exercised in code on every CI run and on demand:
POST /v1/resilience/drill runs the hermetic fault-injection suite and lands a trendable
it/resilience/report (§2.5). What in-process injection cannot cover — a REAL backup restored onto a clean
host, kill −9 under load, true ENOSPC, a firewall-level gossip partition — is the scheduled-rehearsal table
in §2.5.4; put those on the ops calendar.
EVENTLAKE_NODE_ID is not a label. Control-plane replay folds only the records this observer id
emitted (the D22 provenance gate: a valid seal proves storage, not origin, so observer identity is
the only thing that distinguishes a record the brain decided from one a client asserted). A successor
booted under a different id is therefore a different brain: it restores no tenants, no credentials,
no cases, no suppressions and no kill-switch scopes — the estate emergency stop comes back
DISENGAGED — while every record still sits on disk and every seal still verifies.
The ways this happens are ordinary: an edited unit file, a lake directory moved to a differently-named host, a StatefulSet whose stable identity changed, a container orchestrator defaulting the variable.
Since 2026-07-28 the boot REFUSES rather than serving an empty platform as a healthy one. The trigger is deliberately narrow — the node discarded control-plane records and rebuilt none of its own — so a first boot, an ordinary restart, and a healthy node carrying a few forged records all come up normally. (That last exclusion is load-bearing twice over: without it any ingestor key could stop the next boot by injecting one control-plane-shaped event, and the refusal message would name the forger's id in its remediation advice.)
To rename a node deliberately, declare the predecessor:
EVENTLAKE_NODE_ID=new-name
EVENTLAKE_ADOPT_NODE_ID=old-name
The predecessor's control plane then folds as this node's own, and the takeover lands in the record
plane as it/security/node-adoption (how many records, from which node, to which). Keep the adopt
variable set permanently. It is not a migration window: the old records are sealed under the old id
and cannot be re-stamped, so removing the variable later discards exactly what it restored — the same
reason shred accepts its retired token framing forever. Unset it only once those records have aged
past retention. The two variables must differ; adopting your own id is refused at parse time as a
half-applied rename.
Be precise about what exists.
Within one node — durability and fast recovery are the story: durable-on-return writes, sealed replay, readiness gating, graceful drain. A brain restart loses nothing and self-verifies.
Across nodes (cluster mesh, built) — EVENTLAKE_CLUSTER_ADDR (+_ADVERTISE, _PEERS) runs the gossip
mesh on its own listener: membership convergence + scatter queries (a query fans out to peers and unions
results). This is a read/visibility fabric — each node's durable lake stays single-writer. Do not
point two brains at one segment dir, and do not scale a k8s StatefulSet past replicas: 1 per lake (I2).
Brain failover (today) — a scripted, fenced warm standby: eventlake-ctl node ship copies the
primary's sealed segments to a standby root while it runs, node promote verifies the copy end to end, claims
a new epoch on a shared fence witness and waits out the lease, and the old primary refuses writes (readyz 503)
once superseded. RTO = promote (verify + 13 s lease) + replay; RPO = the unsealed active segment + the ship
cadence (bound it in time with EVENTLAKE_SEGMENT_MAX_AGE). Procedure, fencing limits and the measured
drill: docs/reference/failover-runbook.md. Volume replication (DRBD, cloud snapshots) remains a valid
alternative transport, but carries no fence. Active-active / raft-replicated lake is deferred —
go.etcd.io/raft is a named D5 perf-tier candidate, adopted only with a measured justification (§6).
Multi-site / multi-region (the designed shape — 01-substrate-plane.md#distributed-detection) — deploy
a brain per site/region (data residency: raw stays local), Sensors and Tier-2 aggregators feed the site
brain, and the site forwards alerts + reduced metadata up to a global brain. Island mode: when the
uplink is down, the site keeps detecting and responding autonomously (its own lake, detection, conductor);
edge telemetry buffers and replays on reconnect — cursor-backed collectors resume from the OS's own log
position, stream-backed ones from the agent's bounded on-disk spool (-spool-bytes, §2.1.10, with the
drill that proves it) — so detection does not go blind during a partition. The site→global uplink is the
transport mTLS path; the global-brain federation tier itself is design-complete but not yet a shipped
binary (§6).
cmd/eventlake-sensor) at the edgeTwo form factors, one binary. The agent's enforcement semantics (what a block-indicator means, how the
conductor gates it) are Volume 03's; what follows is the install, the flags, and the operational contracts.
Host micro-agent (per host / per k8s node):
# enrol once (preferred): on any shell holding an admin key, mint a single-use code + the install line
eventlake-ctl fleet enroll-token -api https://api.eventlakes.com -brain https://api.eventlakes.com
# then, on the host (the code is traded for this host's own rotating credential):
eventlake-sensor -brain https://api.eventlakes.com -enroll-code <code> -profile standard
# or with a static agent key (it must be SUPPLIED as EVENTLAKE_AGENT_KEY on the brain, or it changes every boot):
eventlake-sensor -brain https://api.eventlakes.com -key $EVENTLAKE_AGENT_KEY -id $(hostname)
-profile standard arms the native process / logon-session / persistence legs (no third-party software,
no privilege, observation only, measured shares inside the footprint budget). With no profile and no leg
flag the Sensor registers and ships no telemetry — the default stays empty so an upgrade never re-arms a
host. Their connectors (procwatch, logonwatch, persistwatch) are enabled once with the node-admin
(platform) key. First-run walkthrough: docs/reference/quick-start-self-managed.md.
Brain side: EVENTLAKE_SENSOR=1 switches the conductor from NoopActuator to real Sensor delivery and mints
an ingestor-class agent key. Default is observable dry-run — every command lands as recorded state,
nothing is enforced (I10).
-enforce applies real, reversible OS firewall rules for block-indicator commands (needs
root/CAP_NET_ADMIN). On Linux the backend is nftables-first — a dedicated inet eventlake-sensor table
with blocked4/blocked6 sets (element add/delete = naturally reversible, one look audits the agent's
whole footprint) — falling back to a dedicated HELIX-MOTE iptables/ip6tables chain when nft is
absent; Windows uses named netsh advfirewall rules (HELIX-mote-block-<ip>); macOS uses pf — the
agent's own eventlake-sensor anchor holding one family-agnostic eventlake-sensor-blocked persist table plus one
quick outbound drop rule (pfctl -a eventlake-sensor -t eventlake-sensor-blocked -T show audits the whole
footprint). pf only evaluates anchors the main ruleset references, so the first block appends a marked
2-line anchor hook to /etc/pf.conf and reloads it (append-only, self-healing; remove those lines +
pfctl -f /etc/pf.conf to retire the agent's enforcement). Plans are idempotent (re-apply /
rollback-of-absent both succeed) and self-heal a flushed table. The conductor's canary→verify→rollback
chain and kill switch stay upstream of every applied command; a real actuator can never bypass the gate,
and a malformed target is refused before any exec.docs/owner-open-items.md §6cap; queue row 451, 2026-10-25s). Two
legs see more with a capability and degrade on their own without it, logging which:
CAP_NET_ADMIN lets the exec leg subscribe to the kernel's proc connector (procnotify_linux.go: every
execve as it happens, short-lived processes included; without it the /proc sweep misses a process that
starts and exits between two sweeps). CAP_SYS_ADMIN lets the canary leg attribute a decoy READ through
fanotify (canary_fanotify_linux.go: the record carries the reader's pid · uid · exe, detect:"fanotify",
and the brain's it/file/access fact and it/deception/triggered finding carry the same three attributes;
without it inotify still reports every read, unattributed). No lesser capability attributes: Linux ≥ 5.13
admits an unprivileged fanotify listener only in FID mode, and the kernel zeroes the pid of every event
another process caused for such a listener. The deployed unit grants NEITHER by default: CAP_NET_ADMIN is
the -enforce capability itself (firewall, routes, interfaces), so granting it to a dry-run agent removes the
dry-run's one promise — a compromised observer cannot change the host's network; CAP_SYS_ADMIN is
near-root. An -enforce host already holds CAP_NET_ADMIN, so its exec leg is event-driven with no further
change. Whether the gain is worth the grant fleet-wide was put to the owner (docs/owner-open-items.md §6cap)
and answered "Keep off" (owner call 2026-10-25): neither is granted by default, and a host that wants either
leg opts in per host through the commented block in deploy/systemd/eventlake-sensor.service. ⚠ The rate limit folds a window's later reads into its first
record, so a folded record names the window's FIRST reader; ⚠ attribution is wired on the 64-bit ABIs
(amd64 · arm64 · riscv64 · loong64 · ppc64le · s390x) — a 32-bit agent keeps inotify reads.-enforce-exec arms the in-kernel exec-block for block-exec commands (target
host:<id>|path:<abs>) — the PREVENTIVE, in-kernel leg (blueprint P1 / E1, "block a malicious exec"):
the agent resolves the named binary to its inode and adds/removes it in an eBPF LSM deny map that
a pre-loaded aether_exec_block program consults on every execve, so a hit is refused with -EPERM
before the image runs. Linux only, and the LSM program must be loaded first —
deploy/linux/ebpf/load-lsm.sh load (kernel ≥ 5.7 with CONFIG_BPF_LSM + lsm=…,bpf). It is a
separate opt-in from -enforce because it is a distinct, in-kernel risk with its own prerequisite;
the agent stays stdlib-only by driving the map through the kernel's own bpftool (as the firewall legs
drive nft/iptables). Unlike a kill-process this is reversible — the inverse command removes the
deny entry (the agent remembers path→inode in <state>/exec-deny.json so a rollback resolves the key
even if the file was deleted). Fail-closed: an unloaded driver or a bad target is reported NOT applied,
never a fake block. The aether_exec_block.bpf.c source is the reviewed source of record (authored,
not built/loaded in CI — like the WFP driver); the Go leg that drives it IS built + tested. See
deploy/linux/ebpf/README.md.-planes edge-waf,edge-dns,… scopes which enforcement planes this agent owns
(§C13 routing); -owns scopes targets (default host:<id>); -interval is the poll cadence (default
2 s); -state is the safe-applier state dir. -plane/-version are what the fleet board and rollout
engine see (version defaults to the -ldflags "-X main.version=…" build stamp).-inspect-addr 127.0.0.1:9090 exposes the inline-inspect endpoint: a data plane POSTs an event and
acts on the {block,decisions} verdict.-ips-addr 0.0.0.0:8080 -ips-upstream 10.0.0.9:80 runs the inline IPS / virtual-patching proxy
(W32) in FRONT of a protected origin: a transparent forwarder that BLOCKS a matched exploit inline
(403, the origin never sees it) and forwards benign traffic untouched. Deny-by-default — nothing
blocks until a conductor-approved virtual-patch command (target cve:<id>) arms a shield from the
clean-room catalog (Log4Shell/Shellshock/Struts-OGNL/Spring4Shell/Apache-httpd/Pulse/FortiOS/F5); the
agent owns cve:* so those commands route here, and every shield hit is reported to the brain (D3-ITF,
rollback = the inverse command disarms it).-ips-waf block|shadow (with -ips-addr) arms the inline WAF (charter §6c / Phase 1) on the same
proxy: the brain-compiled waf catalog — pulled and digest-verified over the §C13 edge-detection channel
like any other compiled detection — decides on the LIVE request path instead of only answering
/inspect. It shares the rule engine with that endpoint, so one pulled artifact enforces on both and
there is no second copy to drift. Nothing is enforced until the brain deploys a program, which
requires the rule to have earned forge's trust bar first; an agent armed with no program forwards
everything. shadow records matches and forwards them — the honest rollout: watch what a freshly
compiled catalog would have blocked before it blocks anything. The two switches compose as an AND:
shadow can downgrade a block program, but arming the leg can never promote an alert program into a
block. Blocks return a bare 403 that names no rule and echoes no payload (a block page that names the
signature is a bypass-tuning oracle). Every hit is reported as it/enforcement/block with activity
inline-waf. One deployment contract, and it is load-bearing: the WAF inspects the event field
http.path, which MUST carry the full request target — path AND query, because that is exactly what
the brain's arm inspects and what the patterns were compiled against. the Sensor's own proxy satisfies this by
construction; an external data plane calling /inspect must too, or the query string — where most
injection payloads ride — goes uninspected.-ips-redirect <service-port> (with -ips-addr) makes the proxy TRANSPARENTLY inline via an
in-kernel packet redirect so no origin reconfiguration is needed; the rule is HELIX-owned, idempotent,
and reversed on shutdown (details below).GET /v1/sensor/update, re-derives the staged payload's
sha256 under -artifact-dir (default <state>/artifacts, shape <name>-<version>) against the
directive's digest, stages the verified payload under <state>/updates for the supervisor swap, and acks
honestly — mismatch ⇒ ok:false (halts the rollout), missing payload ⇒ silence (the ring holds).
-self-update opts the agent into performing the swap ITSELF (all three OSes): after the staged ack
lands, a boot-safe A/B rename dance installs the digest-re-verified payload (<exe>.new → rename
<exe>→<exe>.old → rename into place; a failed final rename rolls back, so a bootable binary always
sits at <exe>), writes the <state>/updates/self-swap.json custody marker, and exits code 3
("relaunch me" — non-zero because the Windows Scheduled Task restarts only on failure; systemd
Restart=always / launchd KeepAlive relaunch regardless). Each boot of the not-yet-healthy new build
increments the marker; a successful register RESOLVES it (health) and removes .old, while 3 unhealthy
boots make the new build REVERT itself (<exe>→.bad evidence, .old restored) and exit for a
relaunch of the last-known-good build. -watchdog closes the can't-exec payload tier: run the
service entry point as mote -watchdog <agent flags…> and the binary is its own RESIDENT supervisor — it
spawns the agent as a child and relaunches it on exit (exit 3 = immediate, the swap contract; otherwise
capped backoff), and because a running process's code is memory-resident and immune to the file swap, it
survives a payload so broken it cannot exec: repeated start failures while a swap marker is pending
revert <exe> to the parked .old build and relaunch (the agent child still counts its own crash-loop
boots — the two revert tiers never race). Narrowed residual: a broken payload plus a host REBOOT
inside the seconds-wide revert window still needs a manual restore (on that boot the service manager
execs the broken <exe> itself and no resident code exists yet). Default remains stage-only.record-session command tails the session transcript <id>.cast
under -session-dir (default <state>/sessions) and streams strictly-sequenced chunks to
POST /v1/sensor/session/chunk; a <id>.done sentinel reports the normal close. The agent binary IS
that PTY wrapper first-party: mote -wrap-session <id|env> (unix-only) drives the OS's own script(1)
as the PTY engine — util-linux script -q -e -f -c <cmd> <cast> on Linux, BSD script -q -t 0 <cast>
<cmd…> on macOS (both flush per write so the tail streams live; no -e on macOS, so there the wrapper's
exit code is script's own) — writes <id>.pid (its OWN pid, before the engine starts: under an sshd
ForceCommand the wrapper is the session leader, so the terminate group-kill takes wrapper, engine, and
— via the collapsing PTY — the shell inside) and the <id>.done sentinel on every exit path, then exits
with the session's code. env reads the broker-minted id from $EVENTLAKE_SESSION_ID (the
static-ForceCommand shape: sshd AcceptEnv + client SetEnv); the wrapped command is the -- cmd…
tail, else $SSH_ORIGINAL_COMMAND, else the user's shell. A hand-rolled script(1)/ForceCommand wrapper
honouring the same <id>.cast/.pid/.done contract remains a valid substitute. terminate-session stops
the recording and force-kills the pid in <id>.pid — the whole process TREE on Windows (taskkill /T
/F, TerminateProcess fallback), the process GROUP on Linux/macOS when the pid leads one (a PTY session
shell does; SIGKILL alone never cascades), the portable single-process kill as the fallback either way.
No -session-dir ⇒ the record command is REFUSED (no recording ⇒ no session, honestly failed).Raw telemetry forwarding (all three OSes — the agent parses NOTHING, so detection content evolves without an agent release):
Linux: -collect-auditd /var/log/audit/audit.log tails the audit log and ships NEW records as raw
frames to POST /v1/ingest/raw (connector auditd, enable it for the tenant) — the brain decodes them
into it/process/exec, it/authentication and it/network/flow (the socket leg, 2026-07-15). The
agent parses nothing, so what the tail carries is decided by the audit RULES on the host, not by the
agent — an unloaded rule is simply a stream that never arrives. The socket leg needs its rule loaded
explicitly (/etc/audit/rules.d/eventlake.rules, then augenrules --load):
```
-a always,exit -F arch=b64 -S bind -F success=1 -F saddr_fam=2 -k eventlake_socket -a always,exit -F arch=b64 -S bind -F success=1 -F saddr_fam=10 -k eventlake_socket
```
The key is yours to choose. The decoder classifies by the syscall and never by the key, and it
copies whatever key the rule carries into the event's key attribute. A host still running the rules
this page recommended before 2026-10-01 (aether.rules, -k aether_socket) decodes identically
(decode/auditd.TestSocketDecodeIsKeyAgnostic), so upgrading is optional. If you do upgrade, rename the
file and the keys in place, then reload. Do not leave both files loaded: they differ only in their key,
so auditctl does not reject the second set as a duplicate, and the key your analysts pivot on then
depends on which file loaded first.
success=1 and saddr_fam mirror drops the decoder makes anyway (a refused connect opened no channel;
AF_UNIX is not a network flow) — pushing them into the kernel means dbus/journald/docker.sock chatter
is never tailed or parsed at all. saddr_fam needs Linux 5.3+ (RHEL 9 is fine); drop the
-F saddr_fam clauses on an older kernel and the decoder still discards AF_UNIX, just later and at the
cost of the tail. Use arch=b32 lines too only on a host that actually runs 32-bit binaries — the
socket decoder deliberately refuses the i386 ABI (it multiplexes every socket call through
socketcall(2)), so those records are shipped and then dropped. On a fleet whose hosts all tail into
ONE auditd connector, set name_format=hostname in auditd.conf — a raw audit record does not name
its own host, and a backdoor-bind finding proposes a kill naming that host.
- Linux, the stock-host leg: -collect-journald _COMM=sshd,_COMM=sudo,_COMM=su polls the systemd
journal and ships journalctl -o json NDJSON verbatim (connector journald). This is the leg that needs
nothing configured on the host: auditd above is richer, but it carries only what the audit RULES were told
to carry, and a large share of real estates run the stock ruleset or none — while the journal is on by default
and is already where sshd, sudo, su, PAM, the kernel and every unit write. The brain decodes the well-known
message forms into it/authentication (feeding the UEBA spray / brute-force detectors, with the peer address
in Src) and everything else into it/log/syslog, carrying the journal's trusted _-prefixed provenance
(_SYSTEMD_UNIT/_COMM/_EXE/_CMDLINE/_PID/_UID), which journald stamps from the sender's credentials
rather than from the sender. It also stamps _HOSTNAME on every entry — so unlike auditd, a fleet tailing
into one connector attributes correctly by construction, with no name_format to set.
- The match IS the budget. Every entry the frame carries becomes an event, deliberately — a decoder that
silently dropped "uninteresting" entries would be a filter nobody can see or tune. So the volume gate is the
journalctl match you give the flag, the same doctrine that pushes success=1/saddr_fam into the auditd
rule. _COMM=sshd,_COMM=sudo,_COMM=su is the recommended default and is near-free on a normal host;
-collect-journald all is the whole journal and is an explicit firehose opt-in, to be sized against
§3.1a's volume math before it goes fleet-wide. Only FIELD=VALUE matches and the OR separator + are
accepted — a token that could read as a journalctl OPTION (-u sshd.service, --file=…) is refused before
any exec, so the selector can never re-point the reader at another journal. A unit filter is written
_SYSTEMD_UNIT=sshd.service.
- Cursor-backed, and it holds the cursor. journalctl --show-cursor/--after-cursor keep the position
under <state>/collect/journald.cursor (deliberately not --cursor-file, which arrived in systemd 246 and
would silently fail on RHEL 8 and Amazon Linux 2). A fresh agent seeds at the newest entry — ship from
now on, never history. The property that matters on an outage: the cursor advances only over records the
brain accepted, so a poll whose frames did not land leaves the cursor where it was and the next poll
re-reads that window. The journal is the write-ahead log; this leg needs no -spool-bytes. Two honest
bounds: the rewind is only as deep as the journal's own retention (SystemMaxUse), and a host configured
Storage=volatile loses it on reboot. Delivery is at-least-once — a poll that ships some batches and
then loses the uplink re-delivers those on recovery, the same trade the spool documents.
- Windows: -collect-winlog Security polls the channel via the OS's own wevtutil qe … /f:xml, ships
the native XML rendering verbatim (connector winevtx — its decoder accepts the XML form alongside the
three JSON shapes), and lets wevtutil's own /bm//sbm bookmark file (under <state>/collect) keep
the cursor — even the position is the OS's, not a parse. A fresh agent seeds the bookmark at the NEWEST
record (ship from now on, never history); a failed poll (rotated log, stale bookmark) reseeds with capped
backoff — an honest gap, never a re-ship. Channel→connector routing: a Sysmon channel
(-collect-winlog Microsoft-Windows-Sysmon/Operational) ships to connector sysmon instead — its
decoder owns the Sysmon EID set (exec/flow/access/file/registry/driver/module/DNS) and accepts the same
native XML rendering; every other channel stays winevtx. A name check, not a parse — enable the
matching connector for the tenant.
- macOS: -collect-eslogger
exec,openssh_login,openssh_logout,lw_session_login,lw_session_logout,create,rename,unlink spawns the
built-in eslogger (macOS 13+; stdout is JSON Lines by default) and forwards its NDJSON lines (connector
esf) — this feeds all six macthreat LOOBins detectors + UEBA, and the create/rename/unlink file
events feed the fim arm (it/file/change, the macOS twin of Sysmon EID 11; a rename ships as the
osquery-style MOVED_FROM+MOVED_TO twin records so a watched file renamed away AND an atomic-replace
rename onto a watched path both fire). eslogger needs root + Full Disk Access (TCC); a refusal surfaces
in the agent log and the child is restarted with capped backoff, never a silent no-op.
Island mode for the collector legs — the on-disk spool (SR2). The three legs do NOT have the same outage behaviour, and the difference is the OS's, not ours:
| Leg | Rewind after an outage | Why |
|---|---|---|
-collect-winlog (Windows) |
Inherent — nothing to add | The /bm//sbm bookmark resumes from the channel; the OS event log IS the write-ahead log |
-collect-journald (Linux) |
Inherent — nothing to add | The stored cursor advances ONLY over records the brain accepted, so a held cursor re-reads the window; the journal IS the write-ahead log (bounded by its own retention — and Storage=volatile loses it on reboot) |
-collect-auditd (Linux) |
-spool-bytes |
A file tail is a stream: records seen while the brain is unreachable have no second source |
-collect-eslogger (macOS) |
-spool-bytes |
Same — an ES stream is not replayable |
-spool-bytes (default 32 MiB, per stream-backed leg) parks failed frames under
<state>/spool/<connector>/ and replays them on reconnect:
-spool-total-bytes, default 256 MiB, 0 = off;
queue row 454, 2026-10-20w). An agent runs a spool per stream-backed or sweep leg — around twenty — so
-spool-bytes alone let the real disk ceiling reach ~20× the flag. Over the agent-wide cap, whole segments are
evicted lowest-signal first: bulk (OTLP metrics/traces, host metrics, NetFlow, SNMP traps, inventory),
then ordinary logs (syslog, OTLP logs, any leg not ranked), and security evidence (auditd, eslogger, the
host-watch sweeps, canary/decoy tripwires, passive DNS) only when nothing lower holds an evictable segment;
within a tier the largest spool pays first. Every evicted record is counted against the leg that lost it, in
the same per-leg dropped the fleet row shows. The table is cmd/eventlake-sensor/spoolbudget.go.sensor: auditd uplink RECOVERED — spool drained (N
records dropped…)). It never silently discards (I14).sensor: ISLAND BOOT — …, starts the collection and inline legs at once, and keeps registering behind them
(… island boot over when it lands). What waits is what needs the registration's answer — the feed pin and
the declared tenant: every signed feed sync, the command poll, detection and update pulls. Kernel-held
enforcement (nftables / pf / WFP objects) persists on its own. A permanent refusal (a 4xx) stays fatal
and never islands. Proven end to end by TestIslandBootCollectsWhileTheBrainIsDown.<state>/island-detections.json: the signed feed exactly as received, the feed key
it verified under, the compiled root's vouch for that key (grant or chain), and the declared tenant. An
island boot re-checks the vouch against the root compiled into the binary (never against anything in
the state root), then the feed's signature, the persisted replay floor (feed-watermarks.json) and its
addressing, and re-arms the inline rules: sensor: ISLAND — N edge detection(s) re-armed from the verified
cache. Anything that fails is refused with its reason — … cached edge detections REFUSED (<why>) — this
host enforces NO edge detections until the brain answers — and arms nothing. The first live sync after
registration re-acks the island-armed rules, so the fleet board is right again. ⚠ An un-rooted build
(dev, CI, test-sensor.ps1) neither writes nor believes the cache: nothing there can re-vouch a pin read
from disk. ⚠ An expired intermediate (chain not_after) refuses the cache — keep the brain's signing
grant inside its window. Proven with the real stamped binary against a brain answering 503 by
TestAnIslandBootEnforcesTheLastVerifiedDetectionsWithTheRealBinary.-vuln (2026-09-24am). A rooted
registration writes <state>/island-vouch.json (the root's vouch for the feed key and the declared tenant —
no content). An island boot re-checks it against the compiled root exactly as above, then verifies the
existing <state>/vulndefs/bundle.json under that key through the same adopt path the live sync uses (host
OS, agent, tenant, strict definition parse): sensor: ISLAND — vulnerability definitions "<rev>" (N) adopted
from the verified cache. So the sweep assesses during the outage instead of reporting no definitions. A
refusal logs … cached vulnerability definitions REFUSED (<why>) and adopts nothing, and — unlike the
post-registration loader — never deletes the file: the failure may be the vouch rather than the bundle,
and the load under the live key is the one entitled to judge it. Proven by
TestAnIslandBootAssessesAgainstTheCachedDefinitionsWithTheRealBinary.-spool-bytes ÷ raw-bytes-per-record = the outage you can absorb. The spool holds the RAW
frame, so the figure to divide by is the agent→brain tail bandwidth — a measured 303 B per auditd
record (§2.3.4; not the 520 B stored-event cost, which is a post-decode brain-side number). 32 MiB ≈
110 k records — about 20 hours at 1.5 EPS, or ~1 hour at 30 EPS (30 EPS being the socket leg's
entire 90-day retention budget on the reference node, so an hour there is the realistic worst case). Set
it from your worst credible outage; the budget is per leg, so a host running two stream legs can hold 2×.
0 disables it: a failed forward is dropped, loudly.Drill — total uplink failure (run this before you rely on island mode). Proves the three things that matter: enforcement keeps working, telemetry is not lost, and the depth is visible.
# 0. On the agent host, with a stream-backed leg armed:
sensor -brain https://api.eventlakes.com -key $EVENTLAKE_AGENT_KEY -id $(hostname) \
-collect-auditd /var/log/audit/audit.log -spool-bytes 33554432 -enforce &
# 1. Baseline: confirm records are arriving (brain side).
curl -sH "Authorization: Bearer $ADMIN_KEY" \
"https://api.eventlakes.com/v1/events?type=it/process/exec&limit=1" | head
# 2. CUT the uplink (host-local, reversible — do NOT use the isolate leg, that is a different test).
sudo nft add table inet drill \; add chain inet drill out '{type filter hook output priority 0;}' \
\; add rule inet drill out ip daddr <BRAIN_IP> drop
# 3. Generate telemetry during the outage, then watch the agent log:
sudo useradd drilluser && sudo userdel drilluser # audit records
# expect: "sensor: auditd uplink DOWN — entering island mode, spooling to disk: …" (ONCE)
du -sh /var/lib/eventlake-sensor/spool/auditd/ # the depth, growing on disk
# 4. ENFORCEMENT CONTINUITY — the point of an edge plane. With the brain unreachable:
sudo nft list table inet eventlake-sensor # previously-applied blocks STILL loaded (eventlake-mote on a pre-2026-09-23 agent)
# Rules already applied stay applied; NEW conductor commands cannot arrive (agent-initiated
# transport, by design) — the edge holds the line it was given, it does not invent one.
# 4b. THE OTHER DEPTH — brain side, while the agent is unreachable. Issue a containment for
# this host and watch it QUEUE rather than vanish (the fleet read has always shown this):
curl -sH "Authorization: Bearer $ADMIN_KEY" https://api.eventlakes.com/v1/sensor/agents
# expect the host's row with a rising "queued" and state "offline" once it misses its poll.
# Two directions, two buffers, both bounded and both visible:
# brain→agent = queued COMMANDS (mote.Dispatcher, drained on the agent's next poll)
# agent→brain = spooled TELEMETRY (this section, drained on the next flush tick)
# 5. RESTART the agent mid-outage (the second loss path):
sudo systemctl restart sensor
# expect: "sensor: auditd adopted N spooled frames (B bytes) from a previous run"
# 6. RESTORE the uplink:
sudo nft delete table inet drill
# expect, within one flush tick (2s):
# "sensor: auditd spool replayed N frames (0 still queued)"
# "sensor: auditd uplink RECOVERED — spool drained (0 records dropped to the byte budget…)"
# 7. Verify the brain HAS the outage window — the records from step 3, in order:
curl -sH "Authorization: Bearer $ADMIN_KEY" \
"https://api.eventlakes.com/v1/events?type=it/authentication&limit=50" | grep drilluser
Pass criteria: step 4 shows enforcement intact · step 4b shows the command queue rising and the agent
going offline · step 5 adopts a non-zero backlog · step 6 reports 0 still queued and a drop count of
0 · step 7 finds the outage-window records. A non-zero drop count is not a failure of the mechanism —
it is the budget telling you truthfully that the outage outran -spool-bytes. Raise it, or accept the
documented loss.
Fleet-wide spool depth (the former known limit — CLOSED). The agent reports its spool totals on the poll
it already makes, so GET /v1/sensor/agents carries a spool:{records,bytes,dropped} object beside queued
and an operator reads outage backlog fleet-wide, not host-by-host. Two properties to read it correctly
(I13):
spool key means the agent never reported one — spooling is off
(-spool-bytes 0), no stream-backed leg is armed, or it is a pre-SR2 build. That is unknown, not "no
backlog", and must not render as 0.last_seen. An agent offline for an hour has been spooling unobserved
that whole time and its true backlog is almost certainly larger — which is exactly when understating it
would be worst.Per-leg attribution (the recorded follow-on — now CLOSED). The spool object stays the agent-wide SUM
("is this edge backlogged?"), and the row also carries spool_legs[{leg, records, bytes, dropped}]
answering the question an operator asks next: which source. That distinction is operationally real,
because the legs fail independently — each stream-backed leg has its own spool directory, budget and drain
goroutine, so a single wedged auditd tail can hold a growing backlog while every sibling leg drains
normally. The sum cannot separate that from a whole-host uplink outage, and the two call for opposite
responses: restart one collector, versus go find the network. The agent piggybacks the breakdown on the same
poll (&spool_legs=auditd:40:8192:3,winlog:2:512:0), so it still costs zero extra round trips. Reading it
correctly:
records:0 with a nonzero dropped is the worst row on the host, not the calmest: it is
empty because its byte budget already threw records away. That loss is unrecoverable.spool_legs is the footnote, spool is the answer. The breakdown degrades independently — if it is
malformed, or disagrees with the totals by claiming more than they do, the brain drops the attribution and
keeps the totals. An empty spool_legs beside a present spool means the depth is known but its source
is not; it never means the legs are idle.Clustered COTS server (the ISP resolver shape, N4). -dns-addr :53 runs the clean-room inline DNS
sinkhole resolver (forward-upstream except conductor-sinkholed domains; -dns-upstream,
-dns-sinkhole-ip). Run N identical Sensors behind your VIP/anycast — each polls the brain and flips its live
sinkhole set on conductor-approved sinkhole-domain commands (D3-DNSDL, rollback = the inverse command).
Binding :53 needs CAP_NET_BIND_SERVICE. Fail-open vs fail-closed is a per-deployment policy (an ISP
resolver likely fail-open with logging; an enterprise egress resolver fail-closed) — the resolver itself
answers SERVFAIL when the upstream is unreachable and drops malformed queries.
Inline IPS / virtual patching (the reverse-proxy shield shape, W32). -ips-addr :8080 -ips-upstream
<origin> runs the clean-room inline IPS in front of an UNPATCHABLE asset (a legacy OT HMI, a network
appliance, a frozen web app): it inspects the FULL request bytes (request line + every header value + the
body prefix — vectors the brain's metadata-only WAF cannot see) and drops a matched exploit before it
reaches the origin, forwarding everything else transparently. Run N identical Sensors behind your VIP for
horizontal scale, each arming shields on conductor-approved virtual-patch commands. Fail-open by
design — an unparseable / non-HTTP / oversized request is forwarded (a virtual patch must never become a
self-inflicted outage; it shields SPECIFIC known exploits, the deliberate inverse of the sinkhole's
drop-the-malformed posture), and the shields are precision-first (near-zero-FP, the distinctive form of
each CVE). eBPF/AF_XDP line-rate capture plugs in later behind the same Capture seam — the userspace proxy
IS the real inline enforcement.
Transparent inline (the kernel-redirect Capture seam). Add -ips-redirect <service-port> to put the
proxy IN the flow without reconfiguring the origin: the agent installs an in-kernel packet redirect steering
the protected service port into the proxy's listen port — nftables (fallback iptables nat
REDIRECT) on Linux, a pf rdr anchor on macOS, and — when the EventLake WFP callout driver (eventlakemote_wfp) is
installed — an in-kernel WFP ALE_CONNECT_REDIRECT on Windows, all three redirecting in the kernel.
Each rule/object lives in an agent-owned container (inet eventlake-sensor-rdr table / eventlake-sensor-rdr pf
anchor / the aethermote_wfp WFP provider+sublayer), is idempotent, and is torn down on shutdown; a
redirect the host refuses is logged and skipped (the proxy still works as a manual front-door — never fail
the plane over it). Windows two-tier: the agent PREFERS the WFP driver (the true kernel intercept, which
also preserves the original destination) and falls back to the built-in netsh interface portproxy IP
Helper relay (userland) when the driver is not installed. The driver is a separate, WDK-built, signed
kernel artifact (source + build/sign/deploy runbook in deploy/windows/wfp/); shipping it unsigned or
unbuilt is the one remaining caveat — WinDivert is deliberately NOT used (LGPL, off the SPDX allowlist).
Requires the elevated capability the -enforce firewall path needs (CAP_NET_ADMIN / root / admin).
Packaging — one binary, three service shapes. deploy/systemd/mote.service is the Linux reference unit
(hardened, Restart=always, capability notes inline). deploy/launchd/com.helix.mote.plist is the macOS
LaunchDaemon (KeepAlive + ThrottleInterval 5 mirror Restart=always/RestartSec=5; the key rides
root-only /etc/eventlake/mote.env — the EnvironmentFile idiom — never the world-readable plist).
deploy/windows/install-mote.ps1 registers a SYSTEM Scheduled Task (at-startup + restart-on-failure — the
lightweight service shape for a plain console binary; the key lives in an ACL-locked %ProgramData% file a
generated launcher loads into EVENTLAKE_AGENT_KEY, never on a command line). On Kubernetes a DaemonSet with
the same flags works; the host form is primary.
Trying the agent on a laptop — deploy/windows/test-mote.ps1. The dev shape beside that production
installer: it builds the brain and the agent, runs both locally, drives real actions through the conductor
gate and then asks the HOST what happened. It installs nothing — no service, no Scheduled Task, no
%ProgramFiles% — and stops both processes on exit. Default is a non-admin dry run, and the assertion
that matters there is counter-intuitive: the end-to-end action must come back failed, with a reason
naming the unarmed leg. That is edgelink/actuator.go refusing to let a Simulated result be banked as a
verified remediation, and the harness then proves the same claim against the OS — in a dry run
netsh advfirewall must carry no rule for the test IP. -Enforce (elevated) arms the real reversible
firewall leg, expects executed + a live rule, and lifts it again through the conductor's own undo door
(a compensating action), never by calling netsh behind the platform's back. The destructive legs
(-enforce-kill, -enforce-isolate) are deliberately never armed by it: a real isolate-host cuts the
network of the machine you are sitting at — exercise those on a throwaway VM, or through their fail-closed
refusal path (give the agent -owns host:<id>,host:other-box and target host:other-box: ownership routes
the command, the self-guard refuses it, and the whole conductor→agent→report loop is still exercised with
zero OS effect). Two environment facts it encodes so nobody re-learns them: the work dir lives inside the
repo because Defender's ASR blocks running a freshly-linked unsigned binary out of %LOCALAPPDATA%, and
the fleet read needs the operator key (tenant acme, the agent's tenant) — the bootstrap admin key is
tenant default and correctly sees an empty fleet.
The doctrine: nothing ships to the whole fleet at once, and no ring widens without measured health
(I11). The negative benchmark is the 2024 channel-file incident — a global same-instant content push with no
rings, no health gate, and no halt. EventLake's update orchestrator (conduit/rollout + the spine registry,
review §7a.7 O1) is the opposite by construction:
agent | content, version, and a REQUIRED
sha256 digest — an unverifiable artifact never ships) and an ordered ring plan. The default doctrine
plan: canary 1 % (1 h soak) → early 10 % (4 h) → broad 50 % (12 h) → fleet 100 % (24 h),
zero-tolerance health gates throughout. Ring membership is deterministic (an FNV bucket over artifactpromote-rollout with BlastRadius = the next ring's true measured size — a canary→early hop
auto-executes on clean health; a fleet-wide hop on a big estate is HELD for a human (POST
/v1/rollouts/{id}/promote, step-up, releases it). Halting needs no gate — stopping distribution is
always safe, and the auto-halt is loud (SevHigh audit event).GET /v1/sensor/update). Hosts that already updated stay put until
rollback or a fixed successor rollout — the orchestrator governs how far a version spreads; it never
pushes bytes.GET /v1/rollouts/{id}/bundle exports the platform-signed
(ed25519) rollout MANIFEST; an isolated site imports it fail-closed (POST /v1/rollouts/import —
unsigned/unknown-key/tampered all refused) and verifies the separately-carried payload hashes to the
manifest's digest. The bundle carries trust, not bytes.it/platform/rollout full-state snapshot and every apply result an
it/platform/rollout-ack (D22 replay) — a brain restart never forgets how far an update had spread.The agent contract is pull-based like edge detections: poll GET /v1/sensor/update?agent_id=…, verify the
payload against the directive's digest, apply, then POST /v1/sensor/update/ack (ok:false halts the rollout
in the same call). cmd/mote implements this leg (§2.1.10) and, opted in via -self-update, the swap
executor itself. The default remains stage-only — the host supervisor performs the swap from the staged,
verified payload — and the can't-even-exec payload tier is covered by -watchdog.
The identity plane's principle — keys live in the tenant's own root of trust — extended to the data plane
as a documented, attestable contract (conduit/byok; benchmark: Splunk/Sentinel customer-managed keys).
The crypto-shred plane already encrypts every classified personal-data span at ingest under a per-(tenant,
subject) DEK; the key-root contract puts a per-tenant root above those DEKs:
platform (default — the root is minted in-process and rests sealed under the
deployment KEK), byok (the customer brings the 32-byte root material through the enrollment ceremony;
its sha256 fingerprint is computable customer-side, so the tenant can verify the platform holds their
key), hyok (the root never enters the platform — wrap/unwrap delegate to a customer-held KMS behind the
byok.Custodian seam; the seam ships, cloud-KMS drivers are deployment integrations)./v1/keyroots (every write admin + step-up): enroll → rotate (the next generation
rewraps every live DEK, then destroys the old root — also the platform→byok custody-upgrade path)
→ revoke. A tenant with no ceremony still gets the contract: the first ingested byte auto-enrolls a
platform-custody root.[erased], in the live store, segments, and
every backup of the lake — while the sealed events keep verifying. It is doubly gated: step-up to
propose, then the conductor HOLDS it (the revoke-keyroot kind is inherently irreversible; BlastRadius =
the live DEK count), then step-up again on the dedicated approve route. Fail direction is always "the
root survives." After a revoke, new classified data for that tenant stores the destructive mask (never
plaintext, never a token under a dead root) until a fresh root is deliberately enrolled — a new
relationship with fresh keys.GET /v1/keyroots/attestation returns the platform-signed custody statement (root
generation, custody mode, fingerprint, live-DEK count) verifiable offline against the platform public key
— hand it to the customer's auditor.EVENTLAKE_BYOK_DIR/keyroots.json (0600, atomic+fsync rewrites) holds
DEKs wrapped under their root and roots sealed under EVENTLAKE_BYOK_KEK — the deployment KEK from your
secret store (base64, 32 bytes, stable across restarts; a changed KEK refuses to unseal and the boot
fails loud). An offline copy of the file is useless without the KEK. Like the shred keystore it is
excluded from backups (§2.1.8); the boot replay heals a stale restore by re-destroying revoked root
generations against the sealed it/platform/keyroot events (trust-gated, D22).Honest scope. The per-tenant unit of encryption is the classified-span token plane (the data that matters: personal/Confidential+ spans, which is what rides inside segments, replicas and backups). Whole-file segment encryption stays a deployment-layer control (encrypted volume / LUKS / BitLocker) because one segment interleaves many tenants — a per-tenant root cannot cover a shared file. Single-tenant dedicated deployments get full-store coverage by combining both layers.
deploy/k8s/eventlake-api.yaml is the annotated manifest set: Namespace, Secret (EVENTLAKE_API_KEY),
StatefulSet (stable identity → EVENTLAKE_NODE_ID — a StatefulSet rather than a Deployment precisely
because that identity must survive rescheduling; see §2.1.8a for what a changed id costs and how to
rename on purpose — volumeClaimTemplates → the segment dir), Service, and
a TLS Ingress. The load-bearing choices, so you can adapt rather than copy:
replicas: 1 per brain. The durable lake is single-writer (I2); horizontal scale is *more site brains/v1/readyz with a generous failureThreshold — boot replay on a large lake takes real
time and the probe is exactly what gates traffic until it is done. Liveness = /v1/healthz, less
aggressive than readiness.terminationGracePeriodSeconds: 30 ≥ drain (10 s) + WAL/segment close.EVENTLAKE_GRPC_TLS_CERT/KEY).deploy/Dockerfile (§2.1.1) — no public image exists to pull.§2.1 answers how to run one node well. This section answers the questions above it: which deployment
model fits a given customer, what the reference architecture looks like in each, and how to scale
horizontally and vertically without fighting the platform's design. Everything here composes what is
built and verified today (the deploy/ artifacts, the durable single-writer lake, the cluster mesh, WAL
store-and-forward, per-site brains); designed-but-deferred pieces are called out inline and collected in §6.
Whether EventLake should ever own the node itself — a sealed OS image under the brain/probe/collector, and
why the answer is assemble, gated on a box-shaped SKU, never a kernel — is decided in
01-substrate-plane.md#aether-os (D27 PROPOSED).
EventLake is a small number of tightly-integrated binaries around a single-writer durable lake (the D5
doctrine): one eventlake-api brain owns one segment directory; durability, sealed replay and bounded-RAM
reads are per-node properties; the cluster mesh is a membership + scatter-read fabric (union queries
across brains), not storage replication. Consequences you build every model on:
replicas: 1 per lake (I2).If you remember one thing: cells scale out, nodes scale up, edges buffer, the mesh reads across.
And its corollary: there is no platform ceiling. Every "limit" in this volume is a measured per-cell
envelope on named hardware, never an architectural wall — estate capacity is envelope × cells, and cells
are added, moved and rebalanced at runtime (§2.2.3). Growth is a topology decision, not a migration (the
blueprint's scale doctrine, 00-charter-and-strategy.md#aether-blueprint §7b).
| Model | Who it is for | Telemetry location | Ops burden on customer | Typical size |
|---|---|---|---|---|
| SaaS · pooled cell | Small/mid teams who want zero platform ops ("just send logs") | Provider cloud, chosen region | ~None (agents + a plan) | ≤ ~1 TB/day |
| SaaS · dedicated cell | Larger/regulated customers on SaaS terms but with isolation, residency pinning, private connectivity | Provider cloud, single-tenant cell (own VPC option) | ~None | 1–15 TB/day |
| Self-managed · VM cluster (systemd) | Sovereignty-first buyers (govt/PSU/telco/defence/banks), simple estates, appliance-like ops | Customer premises only | Moderate (the §2.1 runbook) | any |
| Self-managed · Kubernetes | Customers who already operate K8s and want EventLake inside their platform tooling | Customer premises / customer cloud account | Moderate (their existing K8s muscle) | any |
| Self-managed · air-gapped | Classified / isolated networks | Inside the gap | Highest (offline everything) | any |
| Hybrid · per-site brains | Multi-site/multi-country estates: local sovereignty + central visibility | Raw stays per site; alerts/metadata roll up | Shared | large |
Decision drivers, in the order they usually decide it:
What scales vertically (the brain node). The single-writer brain converts cores, RAM and NVMe directly into capacity:
| Resource | What it buys | The equation |
|---|---|---|
| vCPU | Sustained EPS through the full pipeline (ingest→seal→durable→detect) | §2.3.8 — plan 1,500 EPS/vCPU |
| RAM | The hot RAM window (EVENTLAKE_MEMORY_HORIZON) → fast recent queries + shorter replay |
§2.3.7 — GB ≈ EPS × hours × 0.0126 |
| NVMe | On-node retention days + scan speed for hunts | §2.3.7 — stored GB/day × days × 1.44 |
| Disk sync policy | Group-commit (256/5 ms) vs write-through: throughput vs a ≤ 5 ms loss window | §2.1.4 |
Vertical ceilings to respect: the validated basis is 10 k EPS sustained with 82 % headroom (≈18 k ceiling) on a 4-core ultrabook; treat ~10–20 k EPS_avg per brain as the conservative planning envelope until the server-class baseline is ratified. Past it, don't buy a bigger box on faith — shard.
What scales horizontally.
| Layer | How it scales | Mechanism (built today unless noted) |
|---|---|---|
| Sensors / edge agents | Linearly with hosts — each is < 1 % CPU / < 150 MB | Independent agents polling the brain; DaemonSet on K8s |
| Inline enforcement pools (the N4 DNS-resolver shape) | N identical Sensors behind a VIP/anycast | Each polls the brain; conductor commands flip all |
| Ingest/TLS front tier | Stateless proxies (Caddy/nginx/Envoy/ALB) — add instances behind the LB | TLS terminates at the proxy; auth is in-process (§2.1.2) |
| Brains (cells) | The primary scale-out axis: one brain per site / per tenant-cell / per domain | Each with its own lake + standby; mesh joins reads |
| Cross-brain queries | Scatter queries union results across peers | Cluster mesh (EVENTLAKE_CLUSTER_ADDR, §2.1.9) |
| Site → global | Alerts + reduced metadata roll up; raw stays local | Designed shape (01-substrate-plane.md#distributed-detection); the global federation binary is the deferred F-tier (§6) |
Sharding axes, in order of preference:
Rebalancing is operational, not magic: move a source/tenant by repointing its forwarders/Sensors at the new brain — the WAL absorbs the cutover; history stays queryable on the old cell until retention ages it out (or migrate segments by restore, §2.1.8).
Symptom → lever:
| Symptom | Lever |
|---|---|
| Ingest p99 rising, CPU > 60 % sustained | Vertical: +vCPU; or shed a domain to a new brain |
| Queries on recent data slow | +RAM → widen EVENTLAKE_MEMORY_HORIZON |
| Hunts over weeks slow | NVMe read throughput (provisioned MB/s), not cores |
| Disk projection > box | Shorten on-node retention + archive tier; or split by site |
| Boot/failover replay too long | Shrink the horizon (RTO ≈ window ÷ 50 k/s) — RAM is a knob, not only a boon |
| One tenant/source floods a shared node | Meter + ingest boundary gate hold the line; move the offender to its own cell |
| > ~10–20 k EPS_avg on one brain | Stop scaling up; shard by site/tenant/domain |
The SaaS offer is cell-based: small customers share pooled cells, big customers get dedicated ones, and every cell is the same well-understood unit — which is exactly what makes the fleet operable.
Route 53 (latency/geo) ─── per-region ingest edge
│
NLB/ALB (TLS, ACM cert) ┌─ CELL (one tenant, or N small tenants) ─┐
│ │ eventlake-api brain (EC2/EKS, AZ-a) │
stateless proxy tier (Caddy/Envoy pods) ──────▶│ EBS gp3/io2 = durable lake (segment │
(routing + WAF; auth stays IN-process) │ dir + platform-keys.json) │
│ warm standby (AZ-b, segment replica) │
Sensors / forwarders / connectors (customer side) │ S3 archive bucket (Object Lock/WORM, │
— WAL store-and-forward buffers uplink loss │ lifecycle → Glacier tiers) │
│ KMS (per-cell CMK; SHRED keystore │
│ NEVER in the backup set) │
└─────────────────────────────────────────┘
Control plane (fleet-wide, separate account): provisioning · entitlement · rate-card/metering
rollups · billing · fleet observability (the OEM Control-Plane program — tracker §11)
| Component | AWS shape | Notes |
|---|---|---|
| Brain | EC2 (m7i/r7i by tier, §2.2.6) or an EKS StatefulSet replicas: 1 |
The same binary/manifests as self-managed (deploy/) — one artifact set for both models |
| Store volume | EBS gp3 (provision 500 MB/s / 6 k IOPS) or io2 at L-tier | Snapshots = crash-consistent backups (replay handles the tail — §2.1.8) |
| Warm standby | Second AZ, EBS replication/snapshot shipping (or FSx/DRBD-style volume replication) | RTO = horizon replay (minutes); RPO = replication lag |
| TLS/ingest edge | NLB/ALB + ACM, proxy tier on Fargate/EKS | Per §2.1.2 — the proxy is TLS+routing only; a bypassed proxy exposes cleartext, never an unauthenticated surface |
| Archive | S3 + Object Lock (compliance mode) + lifecycle | WORM outside + tamper-evident seals inside; restore self-proves (0 unverified) |
| Keys | KMS CMK per cell; EVENTLAKE_SHRED_DIR on the node, excluded from backups |
Erasure keys never enter the backup set — the crypto-shred guarantee |
| Private connectivity (dedicated cells) | PrivateLink / VPN / Direct Connect to the customer | The "SaaS but our traffic never crosses the internet" option |
Pooled vs dedicated:
| Pooled cell (small customers) | Dedicated cell (big/regulated) | |
|---|---|---|
| Tenancy | N tenants on one brain — isolation is the platform's own tenancy (server-side tenant forcing, per-tenant metering/keys/RBAC; the same enforcement self-managed customers rely on) | One tenant per cell (+ optional own VPC/account) |
| Blast radius | Cell-level (a pooled cell is deliberately kept small: ≤ ~5 k EPS aggregate, tens of tenants) | Tenant-level everything |
| Noisy neighbour | Meter + per-tenant ingest gating hold the line; chronic offenders get promoted to a dedicated cell | N/A |
| Residency | Region choice | Region pinning + PrivateLink + optional customer-held CMK |
| Price shape | Plan (Starter/Growth) | Plan (Growth/Enterprise) + dedicated-cell add-on |
Small-customer path: sign up → tenant provisioned onto a pooled cell in the chosen region → install Sensors / point syslog & cloud connectors → done; the plan (TB/day) is the only sizing decision, and §2.3.10 gives them the two numbers they still own: agent count and egress bandwidth. Big-customer path: dedicated cell(s), sized like self-managed nodes (§2.2.6) but by us; multi-site customers get per-region cells with the same hybrid roll-up posture as the hybrid shape below.
Fleet operations (what "we host it" means). Upgrades are per-cell rolling (drain → binary swap →
readiness-gated replay → standby next); standby failover is drilled per cell; archive lifecycle, restore
tests (0 unverified acceptance), capacity watch (the symptom table as fleet alerts on /metrics), and the
24×7 pager are provider-side. Cell autoscaling is procedural, not automatic (§6): assignment and
promotion of tenants between cells is an operator action on the control plane.
Multi-region & DR. A cell lives in one region (its standby in a second AZ). Cross-region DR = archive replication (S3 CRR) + a cold cell template; RTO is then infrastructure-provisioning time + replay, honest and documented per plan. Multi-region active customers are really the hybrid/per-site shape with cells in each region — not one stretched cell.
The sovereignty model: telemetry never leaves the building. Everything below uses the same artifacts SaaS
runs (deploy/ — Dockerfile, compose, k8s manifest, systemd units), so there is no second-class on-prem
build.
VM cluster (systemd) — the recommended default on-prem shape. The COTS shape from §2.1.3, in three sizes:
/v1/readyz until replay completes). This is the production norm.EVENTLAKE_CLUSTER_ADDR, native mTLS against
a cluster CA — §2.1.2) for scatter-read visibility; site autonomy is island mode (§2.1.9).VM spec per tier: §2.2.6. Keep the lake on local NVMe (or the SAN's NVMe tier); the proxy can share the node at small tiers.
Kubernetes. Choose K8s when the customer already operates K8s — the platform gains nothing from K8s it does not already do itself (durability, replay, readiness, drain), so the cluster is for their operational uniformity, not ours.
deploy/k8s/eventlake-api.yaml (§2.1.13): Namespace, Secret, StatefulSet replicas: 1
per brain (the load-bearing line), PVC via volumeClaimTemplates on an NVMe-backed StorageClass
(local-path / OpenEBS-LVM / Portworx / Ceph-NVMe — provision for the §2.3.7 read throughput), readiness
/v1/readyz with a generous failureThreshold, terminationGracePeriodSeconds ≥ 30.Air-gapped / classified. Everything runs without internet by design (stdlib-only core, no phone-home, feeds off by default):
06-governance-and-trust.md#supply-chain); container images
from deploy/Dockerfile into the enclave registry.EVENTLAKE_VULN_FEED, EVENTLAKE_INTEL_TAXII_POLL) stay off or point at
an inside mirror fed by your one-way transfer process; GeoIP is operator-loaded (EVENTLAKE_GEOIP csv) —
EventLake bundles no external dataset.platform-keys.json
custody per §2.1.4; the shred keystore never crosses the gap.docs/build_docs.py) — ship it in.Hybrid (per-site brains + a global view). The multi-country/multi-site posture: self-managed brains per
site (raw telemetry never leaves the site/country), alerts + reduced metadata roll up — today that is
scatter-read visibility via the mesh + the site→global uplink design in
01-substrate-plane.md#distributed-detection; the packaged global-brain federation tier is design-complete
but not a shipped binary (§6) — a hybrid sale scopes the global pane accordingly (mesh queries +
per-site consoles today; the federation content loop when it ships).
Quick recommendations:
Sized with the §2.3 constants (raw ON, mixed estate; adjust per §2.3.2/§2.3.8):
| Tier | EPS_avg (peak) | Ingested | vCPU | RAM | Store NVMe (90 d) | AWS example (brain) | On-prem VM example |
|---|---|---|---|---|---|---|---|
| XS | ≤ 500 (1 k) | ≤ 64 GB/day | 4 | 16 GB | 6 TB | m7i.xlarge + gp3 | 4 vCPU / 16 GB / 8 TB NVMe |
| S | ≤ 2,500 (5 k) | ≤ 320 GB/day | 8 | 32 GB | 30 TB | m7i.2xlarge + gp3 | 8 vCPU / 32 GB / 32 TB NVMe |
| M | ≤ 10,000 (20 k) | ≤ 1.3 TB/day | 16 | 64–128 GB | 120 TB (or 30 d + archive) | r7i.4xlarge + gp3→io2 | 16 vCPU / 128 GB / 2×NVMe RAID |
| L | ≤ 20,000 (40 k) | ≤ 2.6 TB/day | 32 | 256 GB | 30–60 d + archive | r7i.8xlarge / i7ie (local NVMe) | 32 vCPU / 256 GB / NVMe shelf |
| XL | > 20 k | > 2.6 TB/day | — | — | — | N × M/L cells (sharding) — not a bigger box | same |
Every tier doubles as a cell size on SaaS. Standby doubles the lake storage line. The L-tier sits at the edge of the validated envelope — deploy it with the pilot's measured numbers, not the catalog defaults.
The bridge to the arithmetic. The full method, formulas, ready-reckoner tables and three fully-worked examples are §2.3; the worksheet is §2.3.9. Compact recap — 1,000 EPS (raw ON) ≈ 127 GB/day ingested ≈ 90 GB/day stored; a 90-day self-managed M-tier at 10 k EPS is 16 vCPU / 64–128 GB / ~117 TB NVMe (×2 with standby) + ~83 TB/yr archive; the same estate on SaaS is agents + ~120/240 Mbps egress + the Growth plan.
| Model | Node loss | Site loss | RTO / RPO |
|---|---|---|---|
| SaaS pooled/dedicated | Warm standby in 2nd AZ, provider-drilled | Archive CRR + cold-cell template (documented per plan) | RTO ≈ horizon replay (minutes) + cutover; RPO ≈ replication lag |
| Self-managed VM pair | Standby boots, readiness-gated | Per-site pairs + island mode; other sites unaffected | Same expressions — you pick the replication cadence |
| Self-managed K8s | StatefulSet reschedule (same PV) or standby StatefulSet | Multi-cluster = per-site cells | PV reattach + replay |
| Any | — | — | Acceptance is always the boot line: … 0 unverified |
And in every model: a restart loses nothing (durable-on-return writes, sealed replay), and the edge keeps buffering through it.
The plane/TLS story is identical everywhere (§2.1.2 — proxy-terminated REST/public planes, native TLS gRPC, mTLS+PQC-hybrid edge transport seam, isolated mesh segment); what differs:
| Concern | SaaS | Self-managed |
|---|---|---|
| Certificates | ACM/provider PKI at the edge | Your proxy + Certificates/CLM-issued certs |
| Tenant isolation | In-process tenancy (both models) + cell isolation + optional VPC/account | In-process tenancy; physical isolation is the premises |
| Archive immutability | S3 Object Lock compliance mode | WORM NAS / object store / tape discipline |
| Erasure guarantee | Per-cell shred keystore, excluded from provider backups | Same rule on your storage (§2.1.8) |
| Residency | Region choice / pinning / PrivateLink | Absolute by construction |
| Supply-chain proof | We verify releases into the fleet | You verify (cosign/SLSA) |
Either way the security boundary is the process, not the perimeter helper: RBAC, tenancy, entitlement, conductor gating and seals are in-binary and travel with every model unchanged.
| Ops item (self-managed owner: you) | On SaaS |
|---|---|
| Hardware/cloud procurement per tier + refresh | In the plan |
| Durable-mode runbook: sync policy, horizon tuning, disk watch | Ours |
Backup set + restore drills (0 unverified acceptance) |
Ours (drilled per cell) |
| Standby replication + failover exercises | Ours |
| Upgrades (drain → swap → replay-gated) | Rolling, per cell |
| Archive lifecycle + compliance-period storage | Plan add-on |
| 24×7 platform pager (not the SOC pager — that is yours in both models) | Ours |
| Capacity re-sizing as EPS grows | Cell promotion, ours |
What SaaS does not change: your SOC owns detections/cases/response in both models; your data plane
(agents/forwarders) is your install in both; and the license dimension (ingested TB/day) is the same number —
which keeps a later SaaS↔self-managed migration a replatform, not a re-license (segments restore
verifiably across models — same store, same seals). The dollar figures behind the SaaS side are
07-commercial-and-gtm.md#hosting-cost-plan.
The crossover intuition: below ~1 TB/day, SaaS is almost always cheaper than the ops attention a self-managed HA pair deserves; above that, the driver is sovereignty and control, not cost — which is why both models are first-class.
The customer-facing "how big is it" answer. Everything a buyer or operator needs to size a deployment
for a given estate: which devices produce how many events per second and per day, what a canonical event
costs in bytes (with and without raw-record retention), and the formulas that turn an inventory into CPU /
RAM / storage / bandwidth — for a self-managed deployment (you host it) or for SaaS (we host it; you
size only agents, egress, and your ingestion plan). The same math ships as an interactive calculator on the
public site (ui_kits/public-site/sizing.html, §2.3.12).
Constants are versioned planning defaults, not a ratified SLO (I18). Every number in §2.3.12 is either measured on this codebase (M, with the method) or an industry-consistent planning default (P — validate with a pilot). The formal per-node capacity target is an open ratification decision (
docs/reference/build-status.md, D4 DECISION block); when the server-class baseline lands, re-issue these constants as v2. Constants version: v1 (2026-07-03).Every number here is a per-cell envelope, not a platform bound. Estate capacity = this envelope × the number of cells (§2.2.1): size one cell here, then count cells.
Original) or normalized-only
(§2.3.2) — it roughly halves or doubles volume, so it is the single biggest sizing lever.The canonical event, and what "raw data as a field" means. Every source converges on one canonical
envelope (aether.event/1 — conduit/event): id, time/ingest-time, observer,
class:{domain,category,type,activity}, severity, outcome, subject/object entities, src/dst endpoints, a
typed attribute map, causal parents, labels, and the tamper-evident seal (Ed25519 over a hash chain;
~230 bytes of the envelope). Two fields control the raw-data posture:
original — the verbatim source record (the syslog line, the CEF string, the rendered Windows event,
the CloudTrail JSON), carried inline as a field of the event. With raw retention ON, every event keeps
its original alongside the normalized fields; the seal covers both, so the raw evidence is as
tamper-evident as the normalization.raw_ref — a pointer variant for oversized payloads (an event references an artifact instead of
inlining it); used by the forensics/pcap paths, not by ordinary log flow.Raw ON is the forensic/compliance-grade posture (the original record is evidence; regulators and DFIR want it). Raw OFF (normalized-only) is the budget posture — full detection fidelity (the detectors read normalized fields, never the raw string), roughly 35–45 % of the stored bytes, but you cannot re-parse history or produce the verbatim original later. Per-source mixing is the pragmatic norm: raw ON for auth/endpoint/cloud/audit sources, raw OFF for flow-family sources (NetFlow/IPFIX is a binary wire format — there is no meaningful raw line to keep).
Measured sizes (M — this codebase, real seal + real store). Measured 2026-07-03 by a probe that
builds fully-populated representative events per source family, seals them with the real event.NewSealer,
marshals the canonical JSON, and persists 2,000 varied copies per family through the real segstore v2
binary segment store, then gzips the resulting segments:
| Source family (representative event) | Raw record | Wire JSON, raw OFF | Wire JSON, raw ON | Stored (v2), raw OFF | Stored (v2), raw ON | Segment gzip |
|---|---|---|---|---|---|---|
| Linux auth — sshd accepted login | 150 B | 886 B | 1,061 B | 496 B | 647 B | 3.0× |
| Windows Security 4624 (rendered JSON) | 1,269 B | 918 B | 2,372 B | 546 B | 1,817 B | 6.9× |
| NGFW deny (CEF) | 336 B | 867 B | 1,218 B | 485 B | 822 B | 3.6× |
| NetFlow/IPFIX flow (binary source — no raw) | — | 817 B | 817 B | 471 B | 471 B | 2.4× |
| DNS resolver query log | 174 B | 830 B | 1,018 B | 462 B | 637 B | 2.8× |
| AWS CloudTrail (AssumeRole JSON) | 1,348 B | 949 B | 2,462 B | 597 B | 1,947 B | 5.9× |
| EDR/endpoint process-start | 518 B | 1,025 B | 1,625 B | 677 B | 1,197 B | 4.8× |
| Web/WAF HTTP request (access-log line) | 271 B | 935 B | 1,226 B | 569 B | 841 B | 3.6× |
| Mixed-estate mean | 903 B | 1,475 B | 538 B | 1,047 B | 4.3× |
Added 2026-07-15 — measured by the same method, and deliberately NOT folded into the mean above (the mean is the standard estate mix; the auditd socket leg is an opt-in add-on, so averaging it in would silently move a constant the rest of this guide is built on):
| Source family (representative event) | Raw record | Wire JSON, raw OFF | Wire JSON, raw ON | Stored (v2), raw OFF | Stored (v2), raw ON | Segment gzip |
|---|---|---|---|---|---|---|
socket — auditd socket leg, connect() (decode/auditd → it/network/flow) |
303 B | 841 B | 841 B | 520 B | 520 B | 2.8× |
The socket family is structurally raw-OFF, like NetFlow but for a different reason: decode/auditd never
sets original, because the "raw record" is a 2-line SYSCALL+SOCKADDR group whose only security-bearing
content (the peer, the owning process) is already normalized into the envelope — a verbatim copy would be
303 B of hex sockaddr no analyst reads and no detector queries. So there is no raw-ON posture to choose, and
the two columns are equal by construction. The practical consequence for §2.3.4: a socket event costs 520 B
stored — 50 % of the 1,047 B raw-ON estate mean — so the lake affords almost exactly twice the socket
EPS it affords mixed-estate EPS. This is the envelope floor; it cannot get cheaper without the perf tier's
zstd.
Reading the table:
POST /v1/ingest carries it and as a REST read returns it — the
basis for ingest bandwidth and for the licensed ingestion volume.compress/flate) and OFF by default for self-managed nodes
and development builds (EVENTLAKE_SEGMENT_COMPRESS=1 opts in); a release-stamped hosted cell writes v3 by
default (owner §6du, answered 2026-10-04; =0 opts out). These constants still assume uncompressed v2 — when the
first cell's measured ratio (segstore_seg3_{logical,physical}_bytes_total) is read, the stored-bytes constants and
ARCHIVE_COMPRESSION move together in ONE commit, so the two can never be quoted at a double count (owner §6dw (b)).The standard planning numbers. Unless you have a measured mix, use the mixed-estate mean:
| Posture | Wire (ingest/license) | Stored (lake, v2) | RAM-resident |
|---|---|---|---|
| Raw ON (default quote) | 1,475 B ≈ 1.5 KB/event | 1,047 B ≈ 1.05 KB/event | ~3.5 KB/event |
| Raw OFF (normalized-only) | 903 B ≈ 0.9 KB/event | 538 B ≈ 0.55 KB/event | ~3.5 KB/event |
(RAM-resident is the RSS cost per hot event on a node that is ingesting — 3.4–3.8 KB/event measured 2026-09-01; plan at 3.5 KB. It does not change with raw posture in the hot window, because the hot window keeps the full event either way. It is ~3× the stored cost, not a fraction of it — the resident form is a graph of live Go objects, the stored form is a positional binary frame. §2.3.7a derives it, and records why the withdrawn 0.7 KB was never a provisioning number.)
Windows-heavy and cloud-audit-heavy estates run bigger (their raw records are 1.3–2.5 KB): if > 40 % of your EPS is Windows Security / cloud audit with raw ON, plan at 1.8–2.0 KB wire and 1.4–1.6 KB stored instead of the mean. Flow-heavy carrier estates run smaller (~0.85 KB wire).
How to read this. Avg EPS is the sustained 24 h average per unit for a typically-tuned source (security-relevant subset, not debug-everything); busy is the sustained busy-hour rate to expect while sizing compute; events/day = avg × 86,400. Size family keys into §2.3.2 for bytes/event. These are planning defaults (P) — real estates vary by a factor of 2–5× either way with audit-policy tuning, which is why the pilot (§2.3.5) is the final word. Where a per-device number is meaningless, the row is normalized per 1,000 users or per traffic unit.
Endpoints & servers:
| Source | Unit | Avg EPS | Busy EPS | Events/day | Size family | Notes |
|---|---|---|---|---|---|---|
| Windows workstation (Security/System, tuned audit policy) | host | 0.5–2 (plan 1) | 5 | ~86 k | windows | Logon/logoff, process audit OFF |
| Windows workstation + Sysmon/EDR-grade telemetry | host | 2–10 (plan 5) | 20 | ~430 k | edr | Process/net/file telemetry dominates endpoints |
| Windows member server (file/print/app) | host | 5–20 (plan 10) | 40 | ~860 k | windows | Shares & service accounts drive it |
| Windows domain controller | host | 25–100 (plan 50) | 300 | ~4.3 M | windows | Kerberos/NTLM; large forests hit 500+ |
| Linux server (syslog + auth + tuned auditd) | host | 2–10 (plan 5) | 25 | ~430 k | syslog | Verbose auditd rules can 10× this — §2.3.4 prices the worst offender |
⤷ + auditd socket leg, -S bind only (backdoor-bind sensor) |
host | ~0 (plan 0.1) | 2 | ~9 k | socket | Binds are a startup burst, then near-silence — §2.3.4 |
⤷ + auditd socket leg, -S connect (real-time outbound flow) |
host | 1–50+ (measure it) | 10× avg | ~86 k–4 M | socket | Unbounded by workload — §2.3.4 before enabling |
| macOS endpoint (unified log, security subset) | host | 1–5 (plan 2) | 10 | ~170 k | edr | Endpoint Security framework events |
| Hypervisor host (ESXi/Proxmox/Hyper-V) | host | 3–15 (plan 8) | 30 | ~690 k | syslog | vCenter/task+auth events |
| Kubernetes node (runtime + kubelet share) | node | 10–50 (plan 20) | 100 | ~1.7 M | edr | Container churn drives bursts |
| Kubernetes API audit (cluster) | cluster | 50–500 (plan 150) | 800 | ~13 M | cloudjson | Scale with controllers/operators |
| Database server (DAM policy-scoped audit) | instance | 10–200 (plan 50) | 500 | ~4.3 M | syslog | Full query audit is 10–100× — scope it |
| Mail server (transactions) | per 1k users | 5–20 (plan 10) | 50 | ~860 k | syslog |
Identity & access:
| Source | Unit | Avg EPS | Busy EPS | Events/day | Size family | Notes |
|---|---|---|---|---|---|---|
| Cloud IdP (Entra ID / Okta — sign-in + audit) | per 1k users | 1–5 (plan 2) | 15 | ~170 k | cloudjson | 9 am Monday is the peak that matters |
| ADFS / federation broker | host | 5–25 (plan 10) | 60 | ~860 k | windows | |
| RADIUS / NPS / TACACS+ | server | 5–50 (plan 15) | 100 | ~1.3 M | syslog | Wi-Fi/802.1X estates at the high end |
| VPN concentrator | per 1k concurrent | 20–100 (plan 40) | 200 | ~3.5 M | syslog | Session + tunnel events |
| PAM / bastion (session-scoped) | server | 1–10 (plan 3) | 20 | ~260 k | syslog | |
| MFA ceremonies | per 1k users | 0.2–1 (plan 0.5) | 5 | ~43 k | cloudjson |
Network & edge:
| Source | Unit | Avg EPS | Busy EPS | Events/day | Size family | Notes |
|---|---|---|---|---|---|---|
| Branch firewall / UTM (deny + session-end) | device | 20–100 (plan 50) | 300 | ~4.3 M | cef | Allow-logging OFF |
| Enterprise NGFW (campus/DC edge) | device | 500–3,000 (plan 1,000) | 6,000 | ~86 M | cef | With allowed-session logging; THE volume driver in most estates |
| Carrier / DC-core firewall | device | 5,000–25,000 (plan 10,000) | 40,000 | ~860 M | cef | Log-sample or flow-offload above this |
| Router / switch (control-plane syslog) | device | 0.2–2 (plan 0.5) | 5 | ~43 k | syslog | Per device — fleets add up |
| NetFlow/IPFIX/sFlow | per sustained 1 Gbps | 1,000–4,000 (plan 2,000) | 8,000 | ~170 M | flow | Biflow dedup halves it; carrier links scale linearly |
| IDS/IPS sensor (alerts + protocol metadata) | per 1 Gbps | 300–1,500 (plan 500) | 3,000 | ~43 M | syslog | Zeek-style full metadata = 1.5–5 k/Gbps |
| Web proxy / SWG | per 1k users | 100–400 (plan 200) | 900 | ~17 M | http | Browsing hours only |
| DNS resolver | per 1k users | 50–200 (plan 100) | 500 | ~8.6 M | dns | Enterprise appliance 0.5–5 k; carrier anycast node 10 k+ (the Sensor resolver logs this) |
| DHCP | server | 2–20 (plan 5) | 40 | ~430 k | syslog | |
| Load balancer / ADC (per-request logs) | per 1k req/s | ≈1,000 | — | ~86 M | http | Sample or log errors-only to control it |
| Email security gateway | per 1k users | 5–20 (plan 8) | 40 | ~690 k | syslog | Verdicts + transactions |
| WAF / WAAP | per 1k HTTP req/s | ≈1,000 | — | ~86 M | http | Full-request logging; verdict-only is 10–50× less |
| Wireless controller | controller | 10–100 (plan 25) | 200 | ~2.2 M | syslog | Assoc/roam/auth storms |
Cloud & SaaS:
| Source | Unit | Avg EPS | Busy EPS | Events/day | Size family | Notes |
|---|---|---|---|---|---|---|
| AWS CloudTrail (management events) | active account | 5–50 (plan 15) | 150 | ~1.3 M | cloudjson | Data events (S3/Lambda) can add 10–100× — enable selectively |
| VPC / VNet flow logs | busy VPC | 500–5,000 (plan 1,000) | 10,000 | ~86 M | flow | The cloud NetFlow analogue |
| Azure activity + Entra sign-in | subscription | 5–50 (plan 15) | 100 | ~1.3 M | cloudjson | |
| GCP Cloud Audit | project | 5–50 (plan 10) | 100 | ~860 k | cloudjson | |
| Cloud security findings (GuardDuty/Defender/SCC) | account | <1 (plan 0.2) | 5 | ~17 k | cloudjson | Low rate, high value |
| M365 / Google Workspace unified audit | per 1k users | 2–10 (plan 5) | 30 | ~430 k | cloudjson | |
| SaaS app audit (Salesforce/GitHub/Slack/…) | per 1k users per app | 0.5–5 (plan 1) | 10 | ~86 k | cloudjson | Multiply by connected apps |
| CI/CD (pipelines + runners + registry) | org | 1–10 (plan 3) | 30 | ~260 k | cloudjson | |
LLM / AI gateway (it/ai/* — prompts, tool calls) |
per 1k users | 0.5–5 (plan 1) | 15 | ~86 k | cloudjson | The AI-SPM stream (N11) |
OT / IoT / physical:
| Source | Unit | Avg EPS | Busy EPS | Events/day | Size family | Notes |
|---|---|---|---|---|---|---|
| OT gateway / historian | gateway | 1–20 (plan 5) | 50 | ~430 k | syslog | |
| OT protocol tap (DPI metadata: Modbus/DNP3/IEC-104…) | monitored link | 10–200 (plan 50) | 500 | ~4.3 M | flow | Poll-cycle dependent |
| PACS badge/door controllers | per 100 doors | 1–5 (plan 2) | 20 | ~170 k | syslog | Shift changes are the bursts |
| CCTV / VMS analytics events | per 100 cameras | 5–50 (plan 10) | 100 | ~860 k | cloudjson | Analytics events, not video |
| IoT fleet (sensor status/telemetry subset) | per 1k devices | 5–100 (plan 20) | 200 | ~1.7 M | flow | Wildly deployment-specific |
The platform's own events. EventLake writes its outputs back to the lake as events (alerts, findings, risk
elevations, posture snapshots, conductor decisions, meter rollups). At healthy alert rates this derived
stream is < 2–5 % of telemetry EPS — include it as the PLATFORM_OVERHEAD factor (§2.3.12), not as catalog
rows. A detection storm is rate-limited by design (alert cooldowns, dedup keys) rather than sized for.
Overlap warnings (don't double-count):
Every other row in §2.3.3 is bounded by security policy: you choose the audit policy, and the rate
follows. The auditd socket leg (decode/auditd's bind/connect path — the edge agent's real-time network sensor, the
one that sees a listener that opens and closes inside a -net-conns poll interval) is different: once the
rule is loaded, the rate is set by application workload. A quiet appliance emits a handful of events an
hour; a busy proxy or a chatty microservice emits thousands a second. Nothing in the platform bounds it, so
it is priced here before it is recommended anywhere.
The two syscalls are not the same decision, and that is the whole point:
| Rule | What it buys | Rate | Verdict |
|---|---|---|---|
-S bind |
The backdoor-bind detector's real-time sensor (flow.BackdoorBind — a command shell owning a new reachable listening socket; the one flow finding that proposes an irreversible kill) |
A startup burst, then near-silence. A server binds its listening sockets once at boot and then essentially never again; loopback binds are dropped at the decoder, and short-lived client binds are suppressed as source-port binds | Enable unconditionally. The cost rounds to zero and it closes a real blind spot |
-S connect |
Real-time beaconing / port-scan / host-sweep / lateral-movement — every outbound connection, at syscall fidelity rather than 60 s poll fidelity | One event per routable outbound connection. Workload-bound and unbounded | Budgeted opt-in. Measure on a canary first |
What a connect event costs (M, 2026-07-15, §2.3.2 method): 520 B stored · 841 B wire · 303 B of agent→brain tail bandwidth per record · 2.8× archive gzip · decode 55,018 events/sec/core (decode CPU is not the constraint — storage is).
The break-even, worked against the reference node (the deployed RHEL 9.8 brain: 157 GB free, ~122 GB usable after headroom). Because a socket event is 520 B against the 1,047 B estate mean, the same disk affords ~2.0× the EPS:
| Retention | Mixed-estate EPS | Socket-leg EPS (520 B) |
|---|---|---|
| 90 days | ~15 | ~30 |
| 30 days | ~46 | ~93 |
| 7 days | ~197 | ~396 |
Read that honestly: ~30 connect-EPS is the entire 90-day budget of that node — every auth event, every exec, every RADIUS accept competing for the same disk. One moderately busy server can produce that alone. The RAM ceiling is the looser constraint (~66 EPS at the node's 3.5 KB/event resident over a 24 h horizon — re-measured platform-wide 2026-09-01 and now the normative constant, §2.3.7a/§2.3.12, though still not re-measured for this event shape: the resident cost is dominated by Go object overhead, not payload, so do not assume it halves like the stored cost does; and it is real only with the §2.1.4 roll janitor running).
So: measure the rate on a canary before the fleet. Load the rule on ONE representative host for a busy hour, read the actual events/sec off that host's ingress (the §2.3.5 pilot method), and multiply. The per-event cost above is a constant you can trust; the rate is yours and nobody else's.
Three of the decoder's four drop rules push down into the kernel — use them. decode/auditd already
refuses to emit an event for AF_UNIX/AF_NETLINK peers, failed syscalls, and loopback peers. Those drops
protect the lake but not the tail: the record is still generated, shipped and parsed. auditctl's filter
fields let you move the first two into the kernel, where they cost nothing at all:
-F saddr_fam=2 (AF_INET) / -F saddr_fam=10 (AF_INET6) — the big one on a systemd box, where a large
share of connect() calls are AF_UNIX (dbus, journald, docker.sock, nscd) and would otherwise be tailed
and parsed only to be discarded. Linux 5.3+ (RHEL 9's 5.14 has it).-F success=1 — a refused connect opened no channel and the decoder drops it anyway.-F auid>=1000 -F auid!=4294967295 — narrows to processes traceable to a logged-in human. This is a
security tradeoff, not a free win: it is the rule that makes the volume tractable on a busy server, but
an attacker who exploits a service inherits that service's unset auid and becomes invisible to it.
Prefer saddr_fam+success first; reach for auid only when the measured rate leaves no choice, and
record that you accepted the gap.What you cannot do: auditctl has no destination IP or port filter — the sockaddr is not a filterable
field (only its family is, via saddr_fam). So "audit connects except those to my DNS resolver" is not
expressible in a rule, and the loopback drop cannot be pushed down. Plan the volume; you cannot carve the
boring destinations out of it.
Peak factor. Size compute for the sustained busy-hour peak: peak_EPS = avg_EPS × PEAK_FACTOR,
default 2.0 (office-hours estates run 1.5–2.5×; retail/media spikes and incident storms hit 3–5×). Short
bursts above that are absorbed by design — the ingest boundary gate (AdmitTelemetry) backpressures instead
of collapsing, and edge Sensors/aggregators buffer in the WAL store-and-forward — so you size for the
sustained peak, not the worst second. Size storage with the 24 h average (bursts wash out of a daily
volume).
Growth. Security telemetry grows faster than the estate: new sources come online, audit policies deepen. Default +25 %/year compounded; size initial hardware for the end of the refresh horizon (3-year hardware → ×1.95 the day-one volume).
The two totals that drive everything:
EPS_avg = Σ (unit_count × per-unit avg EPS) → storage & license volume
EPS_peak = EPS_avg × PEAK_FACTOR → compute (vCPU) & bandwidth peaks
Measure, then trust: the pilot. The catalog gets you a quote; a 7–14-day pilot gets you the truth.
Point the real sources at a single-node EventLake (§2.1.3 quick start — one binary, durable mode): the platform
meters itself — per-source running rates and byte counts accumulate continuously
(schemaread.SourceStat) and the daily sealed usage rollup (EVENTLAKE_METER_SCHEDULE, it/platform/usage) is
the same auditable measurement the license reads. Re-run this math with your measured EPS and bytes/event;
they override every P default.
GB/day (wire, ingested = LICENSE volume) = EPS_avg × wire_bytes × 86,400 / 10⁹
GB/day (stored, lake) = EPS_avg × store_bytes × 86,400 / 10⁹
Sustained ingest bandwidth (Mbps) = GB/day(wire) × 0.0926 [× PEAK_FACTOR for the peak]
Ready reckoner (mixed-estate constants, §2.3.2):
| EPS_avg | Events/day | Wire GB/day raw ON | Wire GB/day raw OFF | Stored GB/day raw ON | Stored GB/day raw OFF | Avg / peak bandwidth (raw ON) |
|---|---|---|---|---|---|---|
| 100 | 8.6 M | 12.7 | 7.8 | 9.0 | 4.6 | 1.2 / 2.4 Mbps |
| 500 | 43 M | 64 | 39 | 45 | 23 | 5.9 / 12 Mbps |
| 1,000 | 86 M | 127 | 78 | 90 | 46 | 12 / 24 Mbps |
| 2,500 | 216 M | 319 | 195 | 226 | 116 | 30 / 59 Mbps |
| 5,000 | 432 M | 637 | 390 | 452 | 232 | 59 / 118 Mbps |
| 10,000 | 864 M | 1,274 (≈1.3 TB) | 780 | 905 | 465 | 118 / 236 Mbps |
| 25,000 | 2.16 B | 3,186 (≈3.2 TB) | 1,950 | 2,262 | 1,162 | 295 / 590 Mbps |
| 50,000 | 4.32 B | 6,372 (≈6.4 TB) | 3,901 | 4,523 | 2,324 | 590 / 1,180 Mbps |
| 100,000 | 8.64 B | 12,744 (≈12.7 TB) | 7,802 | 9,046 | 4,648 | 1.2 / 2.4 Gbps |
Two conversion constants worth memorizing: 1,000 EPS ≈ 127 GB/day ingested (raw ON), and 1 TB/day
ingested ≈ 7,850 EPS (raw ON) / ≈ 12,800 EPS (raw OFF). The public plan ceilings (Starter 1 · Growth 5 ·
Enterprise 15 TB/day — /v1/metering/rate-card) therefore correspond to roughly 7.8 k / 39 k / 118 k EPS
at the raw-ON mix.
The lake is one store with three temperatures, and each has its own equation.
Hot (RAM) — the memory-horizon window. EVENTLAKE_MEMORY_HORIZON keeps the most recent window RAM-resident;
everything older stays disk-resident and reads union it in transparently (D17). Sizing rule:
RAM_horizon (GB) ≈ EPS_avg × horizon_hours × 0.0126 [3.5 KB/event × 3,600 s]
| EPS | 1 h | 4 h | 24 h | 48 h |
|---|---|---|---|---|
| 500 | 6.3 GB | 25 GB | 151 GB | 302 GB |
| 2,500 | 32 GB | 126 GB | 756 GB | — |
| 10,000 | 126 GB | 504 GB | — | — |
| 50,000 | 630 GB | — | — | — |
So: even a small node holds hours, not days — that is fine, because the disk tier serves older queries
(indexed segments: blooms, sparse time index, rollup-served aggregates). Pick the horizon to fit the RAM
tier you buy, not vice versa — at 3.5 KB/event a 64 GB node affords roughly 5.1 M events of hot window
however you spend them (≈ 2.8 h at 500 EPS, ≈ 8.5 min at 10 k EPS).
✅ The calculator now does exactly this (owner §6dv, built 2026-10-11gk): its RAM per node control
takes the tier you buy and DERIVES the hot window — hours = (tier − 8 − 2 × EPS_peak/10k) ÷ (EPS_avg × 3,600 ×
3.5 KB), capped at 24 h, and it says so in words when the base + buffers already exhaust a tier. Its auto
setting keeps the old horizonH() bands as the interim (§2.3.7a). The horizon also bounds failover/boot replay time
(§2.3.8): a smaller horizon = a faster standby. And the horizon only holds at runtime if the roll janitor
is running (§2.1.4).
This section exists because the document carried two irreconcilable figures — a 700 B constant here and a 3.5 KB figure in §2.3.4 — and the small one drove the public calculator. It was re-measured 2026-09-01 rather than adjudicated on paper. The 3.5 KB figure was right; 700 B is withdrawn.
Four regimes, one event. "RAM per event" is not one quantity, and the two figures were measuring different ones. Measured on the same box (macOS arm64, Go 1.26.5) over the same platform:
| Regime | What it is | Measured |
|---|---|---|
| Settled floor | RSS of an idle node, whole lake resident, after the Go scavenger has returned everything it will | 1.26 KB/event |
| Live heap | HeapAlloc after a forced GC — the bytes actually retained |
1.83 KB/event |
| Steady state under ingest | RSS of a node that is ingesting, which is every production hour | 3.4–3.8 KB/event |
| Burst | heap-in-use sampled during a query/analytics phase |
5.7 KB/event |
The provisioning number is the third row, because a brain never sits in the idle regime: you buy RAM for the node you run, not for the node at rest. That is 3.5 KB. No regime produces 0.7 KB — it is below even the fully-scavenged idle floor, so it could not have been a measurement of resident cost at any load.
Why the gap between rows 2 and 3 is structural, not noise. eventlake-api sets neither GOGC nor
debug.SetMemoryLimit (only the Sensor agent does, cmd/eventlake-sensor/footprint.go), so it runs at the Go default
GOGC=100 — the heap is allowed to grow to ~2× live before a collection. The control: per-event HeapAlloc
was identical to the byte (2,023 B) at GOGC = 50, 100, 400 and 800, while RSS/event over the same run went
3.5 → 4.5 → 8.2 → 11.5 KB. A retained byte does not care what GOGC is; headroom is linear in it. So the ~1.9×
between the floor and the planning constant is the collector's working set, and it is not recoverable by
sizing against the floor — sizing against row 2 provisions a node that OOMs in row 3.
The marginal cost is flat; an apparent growth curve was amplification. Per-ingested-event cost appeared to climb 2.0 → 4.3 KB as the lake grew, which would have made the whole hot-window formula wrong in shape, not just in constant. It does not. With the analytics arms off, the lake held 400,152 records for 400,000 ingested (1.00×) and the retained cost was flat at ~1.8 KB across the whole range. With the arms on it held 623,755 records for the same 400,000 (1.56×) — and the cost per lake record was again flat (~2.3 KB). The climb was the platform writing derived events (findings → alerts → cases → risk elevations) back into the same lake, not per-record cost growing. Two consequences:
RAM ∝ EPS × horizon, no super-linear term.EPS_avg in that formula means ingested EPS, and the constant must absorb the derived-event
multiplier. 1.56× is a bench yield (the synthetic workload is built to fire arms) and is deliberately
NOT baked in; a real estate's amplification is far lower, and RAM_BASE/RAM_PER10K carry platform
overhead. A detection-heavy estate should size the hot window on records, not ingested events.Method, so this is re-runnable (the in-process legs ran 2026-08-31, the live legs 09-01). Live leg: eventlake-api with EVENTLAKE_SEGMENT_DIR +
EVENTLAKE_SEGMENT_GROUP_COMMIT=1 + EVENTLAKE_MEMORY_HORIZON=48h + EVENTLAKE_OBS=1 and the full autonomy set,
driven by eventlake-loadgen in 60 s rounds at 200 EPS and 2,000 EPS — the marginal RSS/event agreed across
both (3,622/3,398/3,716/2,931 B and 3,442/3,561/3,442/3,802 B), so the constant is not EPS-dependent over
that decade. Residency was confirmed from the node's own gauge (aether_lake_events = 720,016 for 720,000
ingested), never assumed — RSS over an assumed denominator is how 700 B survives. In-process leg: the D4
harness (cmd/eventlake-bench) spine, ingesting in chunks with a settling GC and a runtime.MemStats +
RSS sample between chunks. ⚠ Read RSS deltas cumulatively, never per round: the scavenger returns memory
in bursts (one round read 7,895 B/event and the next 514 B/event on a node whose behaviour never changed),
and the same node's idle RSS fell 1,702 → 1,234 → 897 MB over four minutes with every event still resident.
Corroboration already in the repo. docs/reference/build-status.md recorded "heap ≈2.6 KB/event" as a
D4 finding and a soak line implying 4.2 KB/event, both against this section's withdrawn 0.7 KB. The
document disagreed with its own reference file for as long as the constant stood.
✅ DECIDED + BUILT — the horizon is an output of the RAM tier (owner §6dv, decided 2026-10-05, built
2026-10-11gk). The calculator's horizonH() band table (24 h ≤ 1 k EPS · 8 h ≤ 2.5 k · 2 h ≤ 10 k · 1 h ≤ 25 k ·
30 min above) was tuned when a hot window cost 700 B/event, and at that constant every band landed on a 64 GB
node. At the true 3.5 KB it lands on 256–384 GB — including 256 GB for the 300-person / ~1.9 k EPS worked
example in §2.3.11. Of the three fixes — (a) shorten each band ~5× (silently cutting every customer's hot
window), (b) keep the bands and quote 256–384 GB nodes, (c) make the horizon a derived output of a chosen
RAM tier — the owner took (c), the only shape that cannot rot against the constant, with (b) kept as the
interim. The public calculator now carries a RAM per node control: a tier (16 – 384 GB) derives the hours of hot
window it affords, capped at 24 h (RAM beyond that buys nothing the disk tier does not serve) and refused in words
when the base + buffers already exhaust it; auto keeps the bands and their honest (b) quote.
Warm (on-node disk) — the durable lake. The v2 segment store is WORM append-only, 64 MiB segments, whole-segment retention pruning:
Lake disk (GB) = stored_GB/day × on-node retention days × 1.15 (indexes/footers/slack) ÷ 0.80 (keep 20 % free)
≈ stored_GB/day × retention_days × 1.44
| Stored GB/day | 30 d | 90 d | 180 d | 365 d |
|---|---|---|---|---|
| 45 (≈500 EPS raw ON) | 1.9 TB | 5.8 TB | 11.7 TB | 23.7 TB |
| 226 (≈2.5 k EPS) | 9.8 TB | 29 TB | 59 TB | 119 TB |
| 905 (≈10 k EPS) | 39 TB | 117 TB | 235 TB | 476 TB |
| 4,523 (≈50 k EPS) | 195 TB | 587 TB | — | — |
Above ~10 k EPS, long on-node retention gets expensive fast — the standard shape is 30–90 days on NVMe + the archive tier for the compliance balance.
Cold (archive) — the compliance tier. Sealed segments are immutable files — archiving is copying
segment files out (filesystem/object snapshot, §2.1.8) before retention pruning deletes them locally. They
compress well (measured 4.3× mixed; plan 3×) and restore verifiably (platform-keys.json travels with
them; seals re-verify on replay — an archive restore proves its own integrity):
Archive (GB) = stored_GB/day × archive_days ÷ 3 [on S3/object storage, use lifecycle tiering]
| Stored GB/day | 180 d | 1 y | 3 y | 7 y |
|---|---|---|---|---|
| 45 | 2.7 TB | 5.5 TB | 16 TB | 38 TB |
| 226 | 13.6 TB | 27 TB | 82 TB | 192 TB |
| 905 | 54 TB | 110 TB | 330 TB | 770 TB |
On AWS: S3 with Object Lock (compliance mode) gives the WORM guarantee end to end (tamper-evident seals
inside immutable objects), and lifecycle rules walk it down Standard → Glacier tiers as it ages. On-prem: any
object store / WORM NAS / tape with the same discipline. Never archive the EVENTLAKE_SHRED_DIR keystore
(crypto-shred erasure would become undoable — §2.1.8).
Replication & backup overhead. Warm standby (the built HA shape): ×2 the lake disk (full segment replica on the standby). Backups: snapshot-based, incremental-friendly (segments are append-only files) — budget +30–50 % of lake size for a rolling snapshot set if you keep separate backups in addition to the standby + archive.
Compliance/retention presets:
| Regime / driver | Log retention expectation | Suggested split (on-node + archive) |
|---|---|---|
| PCI-DSS 4.0 (Req 10.5.1) | 12 months, 3 months immediately available | 90 d on-node + 9 mo archive |
| CERT-In directions 2022 (India) | 180 days security logs, within India | 90 d on-node + 90 d in-country archive (or 180 d on-node) |
| HIPAA | 6 years for audit/documentation records (common interpretation for security logs) | 90 d on-node + 6 y archive |
| SOX / financial | Commonly 7 years for audit-relevant records | 90 d on-node + 7 y archive |
| GDPR / DPDP | No fixed period — purpose & storage limitation (keep only as long as justified) | Per-purpose retention policies (/v1/governance/retention, N13) + the governance sweep enforces expiry |
| ISO 27001 / SOC 2 | No fixed statutory period; 12 months is common practice | 30–90 d on-node + to 12 mo archive |
| Telecom license conditions (e.g. India DoT CDR/IPDR) | Commonly ~2 years — confirm current circulars for your license | 30–90 d on-node + 2 y archive |
| NIS2 / sectoral CSIRT expectations | Not fixed in the directive; 12–24 months is the common bar | 90 d on-node + 12–24 mo archive |
(Regulatory rows are planning aids, not legal advice — your compliance team owns the number; the math
above turns whatever number they give into terabytes. The obligations themselves are
06-governance-and-trust.md#compliance-as-code.)
What is measured (M). The D4 benchmark (full pipeline: ingest → seal → durable lake → detection baseline;
docs/reference/build-status.md) on the reference dev machine — an i7-10510U ultrabook, 4 cores / 8
threads, 20 GB RAM — measured fresh in July 2026 across the full detector/decoder estate:
eventlake-api boot on the RHEL server measured ≈ 5.35 k events/s (2,556,632 events in 7 m 58 s incl. graph + projection rebuild, docs/reference/deployment-e2e-test-2026-09-13.md drill 2); plan RTO on the real figure (2026-09-16tb, audit B4-D1); bounded-horizon soak: flat oscillation, no monotonic growth.A server-class Xeon/EPYC core outruns this ultrabook core; treating laptop-derived numbers as the per-vCPU planning floor bakes in a safety margin. The formal per-node capacity target is a pending ratification (the "≥ 50 k EPS/node durable" ambition vs the measured basis — build-status D4 DECISION block); until a server-class baseline is published, plan with the derated constant below.
The planning constant (P, derived from M):
vCPU = ceil( EPS_peak / 1,500 ) + 2 (platform floor) [minimum 4 vCPU]
1,500 sustained EPS per vCPU ≈ the measured 18.2 k ceiling ÷ 8 hardware threads (≈2.3 k), derated ~35 % for: the full analytics fan-out enabled (UEBA/flow/correlation rounds), streaming twins on top of batch, durable write-through, query/console load sharing the node, and headroom to absorb detection storms. Derate further for unusual mixes:
| Workload modifier | Adjustment |
|---|---|
| Heavy DPI / pcap / OT protocol decode inline | −25 % (≈1,100 EPS/vCPU) |
| WAF/HTTP-heavy with per-request inline inspect verdicts | −20 % |
| Very high-cardinality UEBA (> 250 k active principals) | −15 % |
| Raw OFF (normalized-only, lighter marshal/IO) | +10–15 % |
| Detection arms scoped down (log-management-first posture) | +25 % |
RAM:
RAM (GB) = 8 GB platform base
+ RAM_horizon (§2.3.7 — the sizing lever; pick horizon to fit)
+ 2 GB per 10k EPS_peak (buffers, projections, query working set)
Practical tiers — the horizon each tier AFFORDS at 3.5 KB/event (peak ×2; the calculator's RAM per node control derives the same figure for any load, owner §6dv): 16 GB ≈ 1.2 h at 500 EPS · 17 min at 2 k · 32 GB ≈ 3.7 h at 500 EPS · 54 min at 2 k · 64 GB ≈ 8.6 h at 500 EPS · 2.1 h at 2 k · 50 min at 10 k · 256 GB ≈ 9.5 h at 2 k · 3.8 h at 10 k. Buy RAM to buy horizon; the disk tier serves everything older.
Disk performance & failover math:
RTO ≈ RAM-window events ÷ 5k/s (the MEASURED real-boot rate, derated from 5.35 k/s; the old "÷ 50k/s" was the bench replay and ~10× optimistic — B4-D1) — e.g. 10 k EPS × 1 h horizon = 36 M
events ≈ 2 h until the boot-replay profile (B4 P0) removes the gap; at a 24 h horizon it would be hours (another reason high-EPS nodes keep short
horizons). Readiness (/v1/readyz) gates traffic for exactly this window.The Sensor edge (per monitored host). The edge agent is bounded by the footprint SLO: < 1 % of one CPU and < 150 MB RAM per host steady-state, network = its own telemetry share (I1). For sizing purposes Sensors are free relative to the brain; the clustered COTS resolver shape (N4) sizes like any DNS VIP pool — N small nodes (2 vCPU / 4 GB each) behind the VIP, per-node QPS bounded by the upstream resolver, not by the Sensor.
1. EPS_avg = Σ inventory × catalog (§2.3.3)
2. EPS_peak = EPS_avg × 2.0 (§2.3.5)
3. EPS_sized = EPS_peak × growth (×1.95 @3y) (§2.3.5)
4. Wire GB/day = EPS_avg × 1.475 KB × 86,400 → license plan (§2.3.6)
5. Stored GB/day = EPS_avg × 1.047 KB × 86,400 (§2.3.6)
6. Lake TB = stored × on-node days × 1.44 (§2.3.7) [× 2 with standby]
7. Archive TB= stored × archive days ÷ 3 (§2.3.7)
8. vCPU = EPS_sized / 1,500 + 2, min 4 (§2.3.8)
9. RAM = 8 + horizon eq. + 2/10k (§2.3.8)
10. Map to a tier / cell count (§2.2.6)
| Dimension | SaaS (we host) | Self-managed (you host) |
|---|---|---|
| What you deploy | Sensors / forwarders / connectors only | Everything: brain node(s), standby, storage, proxy, archive |
| Your compute | ~0 (agents are inside the per-host SLO) | §2.3.8 vCPU/RAM per node × nodes × standby |
| Your storage | 0 (retention is in the plan) | §2.3.7 lake + archive + backups |
| Your network | Egress = wire GB/day (§2.3.6 bandwidth column) to the SaaS ingest edge; WAL store-and-forward rides out uplink blips | Internal ingest paths + optional site→global uplinks |
| The commercial number | Ingested TB/day → plan (Starter 1 / Growth 5 / Enterprise 15 TB/day) + retention add-on beyond the plan's included window | License (same volume dimension) + your hardware/cloud spend |
| Ops you own | Pointing sources; everything else is ours (upgrades, HA, replay, archive) | The §2.1 runbook: durable mode, backups, standby drills, upgrades |
| Data location | Our region/cell (pick region for residency; dedicated cell option) | Never leaves your premises — the sovereignty posture |
Small enterprise — ~300 employees, one site + M365. Inventory: 300 workstations (EDR-grade, plan 5) = 1,500 · 20 Linux/Windows servers ×10 = 200 · 2 DCs ×50 = 100 · 2 branch NGFW ×50 = 100 · VPN (300 concurrent ≈ 0.3 k) ×40 = 12 · DNS (0.3 k users) = 30 · email gateway = 3 · M365 audit (0.3 k users) = 2 · IdP = 1. EPS_avg ≈ 1,950 (endpoint telemetry dominates — typical). EPS_peak ≈ 3,900; 3-y sized ≈ 7,600.
S-tier
node + warm standby.Mid-size bank — 5,000 users, two DCs + cloud, PCI + CERT-In. Inventory: 5,000 endpoints ×5 = 25,000 · 400 servers ×10 = 4,000 · 12 DCs ×75 = 900 · 4 campus NGFW ×1,000 = 4,000 · 2 DC-core FW ×2,500 = 5,000 · NetFlow 2×1 Gbps ×2,000 = 4,000 · proxy (5 k users) = 1,000 · DNS = 500 · VPN (2 k conc) = 80 · 30 CloudTrail accounts ×15 = 450 · 2 busy VPC flow ×1,000 = 2,000 · M365 (5 k) = 25 · DAM on 40 instances ×50 = 2,000. EPS_avg ≈ 49,000 — endpoint fleets at plan-5 dominate; a tuned-down endpoint policy (plan 2) lands ≈ 34 k. Take EPS_avg = 40,000 after the overlap dedupe; peak 80 k.
Carrier / ISP-scale — RailTel-class NOC+SOC, multi-site. Drivers: 6 anycast DNS resolver sites (the Sensor resolver logs, plan 10 k EPS/site busy) · NetFlow on 20 × 10 Gbps backbone links (plan 2 k/Gbps → 400 k flows/s raw — sample 1:10 → 40 k EPS) · carrier core FW 4 × 10 k = 40 k · plus enterprise IT (~10 k). EPS_avg ≈ 150 k, flow-heavy mix (~0.85 KB wire / ~0.6 KB stored, raw OFF on flow).
01-substrate-plane.md#distributed-detection): a brain per site (6–10 cells at 10–25 k EPS each;
L-tier: 32 vCPU / 256 GB / 60–200 TB NVMe by site retention), site-local raw retention (sovereignty +
backhaul economics — you do NOT haul 12 TB/day to one point), alerts + reduced metadata up to a global
view; island mode keeps every site detecting through uplink loss. Archive per site to the 2-year telecom
bar: ≈ 1.3 TB/day/site × 730 ÷ 3 ≈ 316 TB/site.ui_kits/public-site/sizing.html — linked from the public site nav ("Sizing") — implements exactly this
section: the §2.3.3 catalog as editable inventory rows (or a direct EPS / GB-per-day entry), the §2.3.2 size
constants with the raw ON/OFF toggle, §2.3.5 peak/growth, §2.3.7 retention + compliance presets, §2.3.8
compute math, and the §2.3.10 SaaS-vs-self-managed split as its two result panes. When opened against a live
deployment it silently folds the published /v1/metering/rate-card ceilings into the plan suggestion (same
additive, numeric-only pattern as the pricing page); offline it uses the same defaults. The table below is
its constants file in prose — change a constant here, change it there, in the same commit.
| Constant | Value | Basis |
|---|---|---|
WIRE_BYTES_RAW_ON |
1,475 B | M §2.3.2 probe, mixed mean |
WIRE_BYTES_RAW_OFF |
903 B | M §2.3.2 |
STORE_BYTES_RAW_ON |
1,047 B | M §2.3.2 (segstore v2) |
STORE_BYTES_RAW_OFF |
538 B | M §2.3.2 |
RAM_BYTES_PER_EVENT |
3,500 B | M 2026-09-01 — RSS/event under sustained ingest, live eventlake-api at 200 and 2,000 EPS (§2.3.7a). This is the planning constant; the calculator uses this one. |
RAM_BYTES_PER_EVENT_FLOOR |
1,800 B | M 2026-09-01 — settled live heap (HeapAlloc after a forced GC), GOGC-invariant. The bench floor, NOT a provisioning number (§2.3.7a) |
ARCHIVE_COMPRESSION |
3× | M 4.3× gzip measured; planned conservatively |
EPS_PER_VCPU |
1,500 | P derived from M 18.2 k ceiling ÷ 8 threads, −35 % derate (§2.3.8) |
PLATFORM_VCPU_FLOOR |
+2 vCPU (min 4 total) | P |
RAM_BASE |
8 GB (+2 GB/10 k peak EPS) | P |
REPLAY_EPS |
5,000/s | M real eventlake-api boot 5.35 k/s measured (09-13 server drill), derated — the bench's 58 k/s omits graph + projection rebuild (B4-D1, 2026-09-16tb) |
PEAK_FACTOR |
2.0 | P (§2.3.5) |
GROWTH |
+25 %/yr | P (§2.3.5) |
DISK_OVERHEAD |
×1.44 (1.15 index/slack ÷ 0.80 free) | P over the v2 format |
STANDBY_FACTOR |
×2 lake disk when HA | Built HA shape (§2.1.9) |
PLATFORM_OVERHEAD |
+3 % EPS | P §2.3.3 |
| Device catalog EPS | §2.3.3 tables | P — field-planning norms; the pilot overrides |
| Sensor footprint | < 1 % CPU · < 150 MB/host | Hard SLO (I1) |
| Per-node validated basis | 10 k EPS licensed target holds with 82 % headroom (18.2 k ceiling) on a 4-core ultrabook; durable tax −3.7 % | M D4 2026-07-02/03; server-class target = open ratification |
Honesty box. (1) The capacity-target ratification is open — these constants deliberately derate laptop
measurements rather than promise unratified server numbers. (2) Block compression is built in the core store
(segstore v3, stdlib flate) and OFF by default except on a release-stamped hosted cell (owner §6du); these constants
assume it OFF, and they re-issue as v2 — together with ARCHIVE_COMPRESSION, in one commit — once a cell's measured
ratio is read (owner §6dw (b)), because an archive tier that gzips already-compressed segments would otherwise be
counted twice. (3) Catalog EPS values are planning defaults, not guarantees — the pilot (the
platform metering itself) is the authoritative number for a real estate. (4) Storage math assumes the WORM v2
store's whole-segment pruning; sub-day retention granularity is a segment boundary, not a per-event delete
(by design — erasure is crypto-shred, not deletion).
The ITOps/observability arm on the one lake: device health and working/non-working status become canonical events, so the NOC and the SOC share one source of truth and one alert→case path. Clean-room throughout: the entire SNMP stack (BER codec, PDUs, USM crypto) and every flow/pcap decoder is implemented from the public specs — no third-party SNMP, flow, capture or DPI library.
Status: full SNMP, NetFlow/IPFIX/sFlow and pcap/DPI — built + green + tested.
Get, GetNext, GetBulk, Set, Walk, BulkWalk (bulk transport auto-selected for
v2c/v3), plus interface auto-discovery via a walk of ifOperStatus.TrapServer (UDP receiver) that decodes v1 Trap-PDUs, v2c SNMPv2-Traps, and
InformRequests (which it acknowledges), each handed to a handler / decode/snmp.mib.go) names MIB-II / IF-MIB / HOST-RESOURCES objects so
events carry ifOperStatus.2 etc. instead of raw OIDs; Register(name, oid) adds vendor objects.
Setsafety (I17): a SET changes device configuration — a world-changing action. An automated SET must be routed through the conductor's guardrails (theActuatorseam), never issued blindly.snmp.Client.Setis the low-level primitive; the gating lives at the spine/actuation layer (Volume 03).
Honest deferrals: a full ASN.1 .mib file compiler (this is a curated catalog, not a compiler), the
AgentX sub-agent protocol, SNMP-over-(D)TLS (RFC 6353), AES-192/256 (their key extension is
non-standardized), and authenticated v3 inform acknowledgement.
v3 usage:
client := snmp.NewClientV3(&snmp.UDPTransport{Addr: "10.0.0.5"}, snmp.USMUser{
Name: "monitor",
AuthProto: snmp.AuthSHA256, AuthPassword: "….",
PrivProto: snmp.PrivAES128, PrivPassword: "….",
})
vars, err := client.Get([]string{snmp.OIDsysName}) // discovery + auth + priv handled automatically
| Direction | Package | What it does |
|---|---|---|
| Active poll | conduit/snmp |
GET device OIDs on a schedule; reachability + metrics → Sample → events |
| Passive trap | conduit/decode/snmp |
Decode an SNMPv2-Trap/Inform a device pushed → it/device/trap event |
Active polling:
// 1. a transport to the device (real UDP, or any Transport for tests/edge)
tr := &snmp.UDPTransport{Addr: "10.0.0.5", Timeout: 2 * time.Second}
client := snmp.NewClient(tr, "public") // community string
// 2. a monitoring profile — which OIDs, and how to read them
profile := snmp.MIB2(). // sysName, sysDescr, sysUpTime
WithInterface(1). // ifOperStatus/in/out for ifIndex 1
WithInterface(2)
// 3. poll → Sample → events into the lake
sample := snmp.NewPoller(client, profile).Poll("device:core-sw-1")
assembly.Ingest(sample.Events(observer)...)
A transport error (timeout / no route) means the device is DOWN — Sample.Reachable == false. That is the
core "working/non-working status" signal. Profiles are NMS templates; add device-specific OIDs with
Metric{Name, OID, Kind} where Kind is text|uptime|gauge|counter|status (a status metric uses
ifOperStatus semantics: 2 = down). Poller.Poll does a single GET of the profile's OIDs; to discover
interface indices automatically rather than hard-coding them, walk the ifTable first:
p := snmp.NewPoller(client, snmp.MIB2())
idxs, _ := p.DiscoverInterfaces("device:core-sw-1") // Walk(ifOperStatus) → [1 2 3 …]
p.Profile = snmp.MIB2().WithInterfaces(idxs...)
Consecutive polls compute per-second rates for counter metrics (e.g. if1_in_per_sec), with Counter32
wrap handling — so the lake gets interface throughput, not just raw counters.
Inbound traps: run the receiver and route each datagram to the decoder:
srv := &snmp.TrapServer{Addr: ":162", Handler: func(src net.Addr, m *snmp.Message) {
if e, err := snmpdecode.DecodeTrap(rawOf(m), observer, src.String()); err == nil {
assembly.Ingest(e)
}
}}
go srv.ListenAndServe()
It decodes v1 Trap-PDUs (enterprise + generic/specific codes), v2c SNMPv2-Traps, and informs
(acknowledged automatically). A linkDown (v2c) or generic-trap=2 (v1) becomes a high-severity device/trap
failure event so it alerts like a polled outage; coldStart/auth-failure and others are informational.
All varbinds ride as attributes, named via the MIB catalog.
| Event class | When | Severity / outcome |
|---|---|---|
it/device/health activity up |
reachable poll | Info / success — metrics (uptime, gauges, counters) in attributes |
it/device/health activity down |
unreachable poll | High / failure + error |
it/device/interface activity down |
an interface's ifOperStatus = 2 | Medium / failure |
it/device/trap activity link-down |
inbound linkDown trap | High / failure |
it/device/trap (other) |
other traps | Info/Low |
Category is "device" throughout; the subject is device:<name|ip>.
Detection reuses the existing path — no new detector engine. detect.DeviceDownRule() (in the spine
default rule set) matches Category:"device" + Outcome:failure and inherits the event's severity — so
device-unreachable, interface-down and linkDown traps all raise an alert → open a case through the path
every other arm uses. Thresholds (CPU/mem high, error-rate, flap detection) are further detect.Rules or
baseline detectors over the metric attributes; their semantics belong to Volume 02.
Dashboards & SLA over the lake (no new engine). The lake already turns one saved query into a dashboard
panel / report / alert (store/query.go):
Aggregate(Query{Category:"device", Type:"health"}, "activity") → up vs down
counts.Query{Category:"device", Activity:"down"}.Query{Category:"device", Type:"interface"}.up/down health event stream over a period (uptime % =
time reachable ÷ window); utilization trends from the counter/gauge attributes via reduce percentiles.decode/netflow (BUILT)The second NMS sense after device health is traffic: who talked to whom, how much, over what. EventLake ingests the four dominant export formats and converges them on one canonical record so the NOC and SOC read the same flow stream.
decode/snmp's relation to the snmp BER codec):unix_secs + sys_uptime.Decoder is stateful: it caches templates keyed by exporter +
observation domain + template id (exactly as a real collector does).flowStart/EndMilliseconds / …Seconds).Flow.Event(obs) → it/network/flow: Subject = source host, Object = dst host (so
timeline.Endpoints yields the src→dst "who-talked-to-whom" edge for the graph and blast-radius),
Src/Dst endpoints carry the ports, and bytes/packets/protocol/TCP-flags/ToS/AS/ifIndex/sampling ride as
attributes. A flow is observational telemetry (SevInfo/success); it is not itself an alert —
detection is a downstream rule over the flow stream.netflow.Collector (mirrors snmp.TrapServer) binds the port, decodes each datagram with one
stateful Decoder (so v9/IPFIX templates persist per exporter), stamps the UDP source as the exporter, and
hands the it/network/flow events to a handler (→ assembly.Ingest). The Decoder also works standalone via
Decode(conduit.Raw{Bytes, Meta:{peer, observer}}) for file/replay ingestion.Flow analytics — the flow arm (BUILT; semantics owned by Volume 02). assembly.AnalyzeFlows(since, until)
runs conduit/flow over the lake's it/network/flow events in the window and emits it/flow/<kind> findings
— which ride detect.FlowRule → alert → risk → case (any containment through the conductor gate). It is a
windowed correlation run on demand / on a schedule (flows are high-volume; the signals are windowed). The
clean-room detectors, listed here because they are what the sources above are for: exfil (a src→dst pair
moving more than a byte budget over the window, High — with internal CIDRs configured, only
internal→external counts, so LAN backups don't false-fire) · beaconing (a src→dst:port series of many
small flows at a regular cadence — low inter-arrival coefficient-of-variation, the C2 callback signature,
High) · port-scan (one src probing many distinct ports on one dst; vertical scan, Medium) · host-sweep
(one src touching many distinct dst hosts; horizontal scan / worm spread, High).
Dashboards (no alerts): top-talkers — assembly.TopTalkers(window, n) (or flow.TopTalkers) ranks source
hosts by total bytes; reduce Count-Min keyed by src/dst/AS gives the same with bounded memory;
store.Aggregate(Query{Category:"network", Type:"flow"}, "src_as") for conversation/AS breakdowns.
Volume/capacity — the bytes_per_sec/duration_sec attributes feed reduce percentiles and interface
utilization panels (joined to SNMP ifIndex).
Queued augmentation: a streaming flow-volume EWMA baseline (baseline) for anomaly scoring alongside
the windowed thresholds; lateral-movement east-west graphing via timeline over the src→dst edges.
decode/pcap (BUILT)Flow records say who talked to whom, how much; packet capture + DPI says what they said — the L7
metadata SecOps hunts on (DNS lookups, TLS SNI, HTTP virtual hosts). decode/pcap reads an uploaded capture
file and lifts that metadata onto the one lake, clean-room from the public specs:
.pcap: 24-byte global header, both byte orders, µs + ns timestamp
magics) and pcapng (.pcapng, Wireshark's default: Section Header Block fixes the byte order, Interface
Description Blocks carry linktype + if_tsresol, Enhanced/Simple Packet Blocks carry the frames). Other
pcapng blocks (ISB/NRB/secrets) are skipped by declared length.Decode(conduit.Raw{Bytes:<file>}) → one it/network/l7 event per dissected message
(Subject=src, Object=dst, the L7 fields in attributes — so a DNS qname or TLS SNI flows straight into the
intel matcher and any saved-query hunt). WithSessionEvents() additionally emits a per-packet
it/network/session 5-tuple record for full forensics — off by default: the brain wants metadata.Edge doctrine (01-substrate-plane.md#distributed-detection): heavy DPI is a Sensor (edge) job; the brain
receives metadata + reductions, not raw packets. This package is that metadata extractor + the file/replay
path.
Honest deferrals: live AF_PACKET/AF_XDP capture (plugs in behind a Capture seam without touching the
decode/DPI core), TCP stream reassembly (so far per-packet, which covers DNS, small HTTP requests, and the
TLS ClientHello — the common cases), more L7 dissectors (SMB/SMTP/…), and the raw-pcap never-drop store
via schemaread for retro re-parse.
TrapServer on UDP/162 and route datagrams to decode/snmp.DecodeTrap.decode/netflow.Decoder.decode/pcap.Decoder (a live tap follows behind the
Capture seam).asset) so crown-jewel weighting and blast-radius
apply to network gear too.Status: BUILT (v1, 2026-07-07 — Review-#2 O3). The fault-injection drill suite is conduit/resil
(kill-replay · torn-tail · disk-full · clock-skew · partition-heal · restore-rehearsal), run hermetically by
assembly.ResilienceDrill() and served at GET /v1/resilience / POST /v1/resilience/drill; every run lands as
a trendable it/resilience/report event.
The 2026-07-05 docs-audit verdict (O3): the blueprint promises "proven at every rung," and the platform sells recovery — N12 orchestrates customer restore waves, D5/D22 promise a restart forgets nothing — yet no chaos/fault-injection harness, DR drill, or consistency validation existed anywhere. The industry lesson is the same one the O1 rollout doctrine drew from the 2024 channel-file incident: the failure mode that hurts is the one you never rehearsed. Jepsen's discipline is the benchmark — do not trust the design argument; inject the fault and check the invariant.
So O3 is executable: a drill suite that builds hermetic platform instances, injects one fault each, and
asserts the exact invariants the platform's durability story rests on. It runs in CI on every change (go
test — the drills ARE tests), on demand over the wire, and on the operator's schedule; the report is an
event on the lake, so "when did we last drill, and did it pass" is a category query. A failed drill is
SevHigh with a failure outcome — the platform failing its own resilience validation is alert-worthy, not
informational.
Honesty rules (the same bar as backtest/comply):
| Drill | Fault injected | Invariants asserted |
|---|---|---|
kill-replay |
process dies; store survives (disk-backed when the opener is wired, else the in-memory serialization boundary) | successor replay count == predecessor lake count · zero unverified seals across the round-trip · case projection rebuilt EQUAL (D22) |
torn-tail |
the active segment is truncated mid-frame — the artifact a kill mid-write legitimately leaves | reopen never fails (fail-safe, not fail-corrupt) · exactly the torn record lost · surviving prefix verifies clean |
disk-full |
the store refuses writes mid-cascade (a faultStore wrapper; batch semantics mirror the real store) |
the record plane never drops an in-RAM record (lake doctrine) · the cascade still completes (a case opens on a dying disk) · PersistErrors counted · a successor boots the honest durable prefix |
clock-skew |
events authored 26 h in the past and 2 h in the future; the successor boots with its clock 3 h ahead | replay equality and seal verification are time-independent · projections fold identically under skew |
partition-heal |
a 5-node fabric membership splits 2/3; each side's failure detector declares the other dead | each side places only on its own alive set · the split-brain window is measured (not denied) · after heal, every view converges to the IDENTICAL state under three different merge orders (the CRDT order-independence claim) · post-heal there is exactly one owner for the single-writer lake key · the SWIM refute-above-the-suspicion recovery path works |
restore-rehearsal |
back up the closed store · keep operating · destroy the live copy · restore from backup | the backup boots clean (count + seals + cases) · the RPO gap (events written after the backup) is measured and reported |
Suite mechanics: every drill runs even if an earlier one failed (the report is the whole picture); a
panic inside a drill is recovered into a failure; results carry per-drill metrics (events, lost,
at_risk, rpo_gap_events, split_brain_keys, heal_rounds, …) so trend queries see the numbers, not just
pass/fail.
Seams (D5 discipline). resil imports only event/store/cluster — never assembly (the assembly
imports the harness and adapts itself to the Target seam) and never a store implementation (the disk
drills ride Config.OpenStore; cmd/eventlake-api wires segstore.Open). The in-memory memStore is a real
serialization boundary — events round-trip through canonical JSON, so replay hands back fresh objects whose
seals must re-verify, not the same pointers.
go test suites: the resil unit tests (fake target + real
segstore torn-tail/rehearsal + the vacuous-pass guard), assembly.TestPlatformResilienceDrill (the REAL
platform passes all six — real ed25519 seals, the real device-down cascade, live-lake isolation),
api.TestResilienceAPI (the wire surface + RBAC). A regression that breaks a durability invariant fails
the build, not the customer.POST /v1/resilience/drill (admin; no step-up — hermetic and reversible by
construction). Returns the fresh report and appends the snapshot.06-governance-and-trust.md#compliance-as-code).GET /v1/resilience serves the newest snapshot (canRead — the report carries drill names and measured counts
only, no tenant data) and answers {"ran": false} honestly when the platform has never drilled.
In-process fault injection approximates faults at the seam. These remain operator rehearsals on the real deployment — scheduled, logged (the drill snapshot proves the in-process half; the ops calendar owns this half), and named here so the gap is explicit:
| Rehearsal | Cadence (recommended) | What it covers that the suite cannot | Runbook |
|---|---|---|---|
| Restore from a REAL backup onto a clean host | quarterly, and after any storage change | OS-level file semantics, permissions, platform-keys.json custody, the keystore-exclusion discipline (EVENTLAKE_SHRED_DIR stays OUT of ordinary backups) |
§2.1.8 — restore to a scratch EVENTLAKE_SEGMENT_DIR, boot, check the replay line + unverified == 0, then GET /v1/healthz and a spot query |
| Kill −9 / power-cycle the API node under load | quarterly | true OS crash semantics below the process (fsync behaviour, torn writes at the device layer) | §2.1.4 — restart, read the boot replay stats, alarm on unverified > 0 |
| Fill the segment volume (real ENOSPC) | semi-annual | the syscall-level failure path + the segstore_persist_errors_total alarm the obs seam exposes |
watch PersistErrors climb, free space, verify write-through resumes with no RAM loss |
| Multi-host gossip partition (drop node↔node traffic at the firewall) | semi-annual, mesh deployments only | the real wire (HTTP anti-entropy, Lifeguard timeouts) above the CRDT the suite validates in-process | §2.1.9: islands keep serving reads; the lake stays single-writer — failover is a health-gated HUMAN promotion, never automatic split-brain |
| Clock step on a live node (NTP jump) | annual | kernel/NTP interactions above the event-time logic the suite covers | verify ingest continues, watermarks recover within the configured lateness |
The single-writer doctrine, restated for the failover rehearsal: the lake has ONE writer by design. A
partition therefore creates islands that keep serving reads; promotion of a standby writer is a
deliberate, health-gated act. The partition-heal drill's post-heal uniqueness check is the in-code half of
that guarantee; the rehearsal's job is to confirm the humans and the runbook do their half.
it/resilience/report (Category resilience, Type report, Activity drill; platform-scoped, free of
charge in metering — platform self-validation, not customer ingress). Attributes (canonical maps, D22
seal-determinism): healthy, passed, failed, skipped, and drills: [{drill, status, duration_ms,
detail?, metrics?}]. GET /v1/resilience returns exactly this plus at/event_id; Reports/maturity read
the same stream.
Deliberate non-goals:
cmd/eventlake-bench owns throughput/latency (N3/D4); the drills assert
correctness under faults, at fixture scale, in milliseconds.resil points the same discipline at EventLake itself.Status: DOCUMENTED, not held. Every credential below is a prerequisite EventLake does not
currently possess. This section is the acquisition runbook, not a record of things done. Written
2026-08-02; external facts (prices, policies, wait times) are dated where they are volatile — re-verify
before spending. Windows prices and policies RE-VERIFIED 2026-10-12 (§2.6.3, §2.6.5), and every artifact
name brought to the post-cutover names (mote → eventlake-sensor, 2026-09-12m; deploy/README.md).
The four in-kernel enforcement planes each carry a build→sign stanza in their own README
(deploy/linux/ebpf/, deploy/windows/execblock/, deploy/windows/wfp/, deploy/macos/execblock/).
What no README carries is the external half: which vendor gate applies, whether that gate blocks
testing as well as deployment, what it costs, and how long approval takes. Three separate things get
called "signing" and they do not substitute for each other:
| Layer | Question it answers | Who grants it | Our status |
|---|---|---|---|
Publisher signature — cmd/eventlake-releasesign (§2.1, D05-14) |
"Did EventLake build this binary?" — verified by our own agent against a key compiled into it | Us, from an offline ceremony | BUILT |
| OS code signing — Developer ID / Authenticode | "Is a legally-identified, revocable entity behind this code?" — verified by Gatekeeper / SmartScreen / the NT kernel | A commercial CA, or Apple | NOT HELD |
| Restricted entitlement — ES client, driver attestation | "Did the OS vendor approve this company to use a privileged API?" | Apple / Microsoft, by review | NOT HELD |
The load-bearing consequence, and the reason this is a section rather than a footnote: eventlake-releasesign
satisfies our trust chain and satisfies zero OS requirements, and on macOS signing is not the
entitlement — a perfectly signed, notarized ES daemon without the granted entitlement still dies at
es_new_client (deploy/macos/execblock/eventlake_execblock.c:194 already logs exactly this). Neither
layer can be substituted for the other, and neither can be worked around.
Why the vendor gate exists at all (worth stating, because it shapes what a good application looks like): these APIs veto process execution and observe all system activity — precisely what malware wants. The approval is an accountability mechanism, binding privileged code to a revocable real-world entity. Post-CrowdStrike (2024-07), Microsoft's kernel scrutiny tightened materially, and the April 2026 policy in §2.6.3.4 is a direct continuation.
| Artifact | Kind | Gate | Approval? | Blocks testing? |
|---|---|---|---|---|
eventlake-sensor (darwin/amd64, darwin/arm64) |
Go userspace agent | Developer ID Application + notarization | No — automated | No |
deploy/macos/execblock/eventlake_execblock |
ES client daemon | Developer ID + com.apple.developer.endpoint-security.client |
Yes — Apple review | No (SIP path, §2.6.2.4) |
eventlake-sensor.exe (windows/amd64, arm64, 386) |
Go userspace agent | Authenticode | No | No |
sensor-desktop.exe (windows) |
the person's desktop app over the Sensor (conduit/cmd/sensor-desktop, installed by install-sensor.ps1 -DesktopBinPath) |
Authenticode — the same certificate as the agent | No | No |
deploy/windows/execblock/eventlake_execblock.sys |
WDM kernel driver (service EventLakeExecBlock) |
EV cert + Partner Center attestation | Yes — MS submission | No (test-signing, §2.6.3.6) |
deploy/windows/wfp/eventlakemote_wfp.sys |
WFP callout driver | same as above (one EV cert covers both) | Yes | No |
eventlake-sensor (linux/*) |
Go userspace agent | Distro package signing (our own GPG key) | No | No |
deploy/linux/ebpf/eventlake_exec_block.bpf.c → .bpf.o |
eBPF LSM object | No vendor gate — technical prerequisites only | No | No |
⚠ eventlakemote_wfp keeps its retired mote token on purpose — it is a signed driver's service name and
stays unmoved until a re-signed build ships (CLAUDE.md's carrier list). Do not rename it in passing while
preparing the first attestation submission: the .inf is part of what gets signed, so any rename is a
separate, deliberate change made before submitting, never during. The exec-block driver already moved
(HelixExecBlock → EventLakeExecBlock, owner call 2026-10-04); its .inf display string still reads
"EventLake Mote exec-block" and is worth correcting in that same pre-submission pass.
The pattern that should drive scheduling: nothing is blocked for testing, everything is blocked for deployment onto a machine that is not ours.
eventlake-sensor and the eventlake_execblock daemon..pkg. Both are needed if we
ship an installer rather than a tarball; this catches people out.xcrun notarytool submit → xcrun stapler staple. Free, automated, minutes, no
review. Required from macOS 10.15 for anything downloaded and run outside the App Store; without it
Gatekeeper quarantines the download.com.apple.developer.endpoint-security.client is restricted: Apple grants it by hand, per team.
Request via the system-extension request form.
Three traps, each of which has cost real projects real time:
| Flavor | Consequence |
|---|---|
| Apple Silicon (arm64) vs Intel (x86_64) | Ship a universal binary (lipo -create) or per-arch artifacts. eventlake-sensor already cross-compiles for both (§2.1). Signing/notarization is per-artifact — a universal binary is signed once. |
| macOS 10.15 Catalina | Floor for the Endpoint Security framework and for mandatory notarization. Below it, the exec-block plane does not exist at all. |
| macOS 11 → 26 | No further ES-entitlement differences we rely on. Approval UX moved to System Settings → Privacy & Security. |
| MDM-managed fleets | The customer's MDM can pre-approve the daemon (system-extension / PPPC payload), so users never see a prompt. This is the only sane enterprise path — an unattended daemon awaiting a click is not a deployment. |
| Unmanaged Macs | Requires a human to approve in System Settings after install, then launchctl kickstart -k. Already in the install script's flow. |
Full Disk Access is TCC, not an entitlement — a distinct axis. The FIM/collection legs need it, it carries no Apple review, and it is granted by an MDM PPPC profile at deploy time. Budget it as a deployment step, not an approval.
Disabling SIP on a dedicated test Mac permits an unentitled ES client to run. The widely-cited caveat
that "disabling SIP is insufficient for most restricted entitlements" applies specifically to system
extensions — and eventlake_execblock is a plain launchd system daemon
(com.eventlake.sensor.execblock.plist), not a sysex. That choice, made for other reasons, is what keeps the
free test path open for us where it would be closed for a sysex-based design.
csrutil disable.csrutil disable.make smoke (2026-08-02, see the daemon's README) already compiles and links the source of record on any
Mac with Xcode with no identity and no entitlement. What remains unproven without the SIP path is
runtime behaviour — the daemon is still un-load-tested.
eventlake-sensor.exe and sensor-desktop.exeBoth need Authenticode. Neither requires it to execute, but SmartScreen warns on an unsigned security
agent, which is commercially fatal, and install-sensor.ps1 already signature-checks the desktop app
beside the agent. Two routes (re-verified 2026-10-12):
.exes too, so no second purchase. Since
Microsoft's SmartScreen change, EV no longer buys an instant pass — OV and EV both build reputation by
download history — so EV is chosen here for the drivers, not for SmartScreen.⚠ A future constraint, recorded so it is not re-discovered: if the Sensor ever registers with Windows Security Center as an anti-malware product (an ELAM driver + a protected service), Microsoft requires the user-mode anti-malware binaries to be signed by Artifact Signing, not by a CA certificate. That would turn the geography line above from a cost into a blocker. Nothing ships that path today.
For eventlake_execblock.sys and eventlakemote_wfp.sys, Microsoft's requirement (Learn, updated
2026-08-11) is explicit: registering for the Windows Hardware Developer Program needs an extended
validation (EV) code-signing certificate, and "you only need the certificate to register" — the
attestation signature itself is Microsoft's.
.exes.Only the legal entity can complete this: the CA verifies the company, a named officer answers the verification call, and Partner Center needs a legal contact who can accept agreements. No step can be delegated to a session or an agent.
.cab (.sys + .inf + .pdb), upload per
architecture (x64 and ARM64 are separate submissions), download the Microsoft-signed result.eventlake-sensor.exe and sensor-desktop.exe with signtool sign /fd sha256 /tr <CA RFC 3161
timestamp URL> /td sha256 using the EV key. Always timestamp, so signatures outlive the certificate.Attestation — automated checks, Microsoft countersigns, free, no lab. Sufficient for a driver we distribute ourselves with our own installer. WHQL/HLK — full compatibility certification, needed only for Windows Update distribution, and materially more expensive (lab hardware plus test time). Attestation is the correct target; WHQL is out of scope until Windows Update distribution is a goal.
Post-dates the driver READMEs and must be read before planning. With the April 2026 Windows update, Microsoft removes kernel trust for drivers signed via the deprecated cross-signed root program; drivers must be WHCP-certified and signed through the Microsoft HDC portal. Rollout is evaluation mode first (audit-only) on Windows 11 24H2/25H2/26H1 and Server 2025, with enforcement later on a date Microsoft has not fixed publicly.
For us this is not a blocker and arguably favourable — we are a new driver going through Partner Center attestation, which is the surviving path. What it kills permanently is the old "buy a cross-signed certificate and self-sign" shortcut, which some older guidance still describes. An administrator override via Application Control for Business policy exists for internal use cases, but it must be signed by an authority in the device's Secure Boot Platform Key — an enterprise-fleet mechanism, not a distribution strategy.
| Flavor | Consequence |
|---|---|
| Windows 10 / 11, x64 | Baseline. Win8/Server 2012 is the floor for the PsSetCreateProcessNotifyRoutineEx veto the exec-block driver depends on. |
| ARM64 | A separate driver build, and a separate attestation submission. Do not assume the x64 signature carries. |
| Server 2016/2019/2022/2025 | Same signing path. 2025 is in the April 2026 evaluation-mode set. |
| Secure Boot enabled (default on modern hardware) | Test-signed drivers will not load. This is what forces the EV path for any real host. |
| HVCI / Memory Integrity | Imposes additional driver-compatibility constraints; verify both drivers under HVCI before shipping. Recorded as unverified — see §6. |
| Windows 10/11 in S mode | No third-party kernel drivers at all. The exec-block plane cannot exist there; the userspace agent legs still can. |
Fully supported and free, and already documented in the driver READMEs: self-signed test certificate,
bcdedit /set testsigning on, reboot, Secure Boot off. This exercises the driver, the deny-set, the
registry channel and the agent's sc query probe end to end. Use a disposable VM or a dedicated lab host
— test-signing plus Secure Boot off is not a configuration to leave on anything else.
There is no OS-vendor gate, no approval, and no cost. No entity grants permission to load an eBPF program. This asymmetry is worth stating plainly because it inverts the usual assumption: the Linux plane is the cheapest to ship and the only one with no external dependency on a third party's review queue.
The gates that do exist are technical, not commercial:
| Prerequisite | Detail |
|---|---|
| Kernel BPF LSM enabled | Booted with lsm=...,bpf. Not the default on every distro — a deployment check, and the most common reason the plane is silently absent. |
| BTF | CONFIG_DEBUG_INFO_BTF for CO-RE. Standard on modern distro kernels. |
| Capabilities | CAP_BPF + CAP_SYS_ADMIN (agent already runs as root). |
| fs-verity | Optional; enables the content-identity deny map (bpf_get_fsverity_digest kfunc, kernel ≥ 6.7). |
| Secure Boot + kernel lockdown | The real constraint — see below. |
Lockdown. With Secure Boot on, distros commonly enable kernel lockdown. In integrity mode BPF
generally continues to work; in confidentiality mode the kernel-memory-reading helpers
(bpf_probe_read and friends) are disabled, and some distro kernels have historically restricted bpf()
much more aggressively under lockdown. Verify the exec-block plane on each target distro with Secure
Boot ON — this is a per-distro empirical question, not something to reason about from the doctrine.
Signed BPF programs are a genuine future option: support was merged for Linux 6.18, keyring-based, carrying signature metadata in a light skeleton for the trusted loader to present at load. Not required today and not a gate we must pass; recorded so it is not re-discovered later (see §6).
Note we ship no kernel module. The eBPF object is not a .ko, so the whole MOK-enrolment /
kmodsign path that Secure Boot imposes on out-of-tree modules does not apply to us. That is a
consequence of the eBPF choice worth keeping visible, since it is a recurring assumption in Linux agent
deployment.
What Linux costs us is distribution signing, with our own GPG key. No CA, no fee.
| Family | Package | Signing |
|---|---|---|
| Debian / Ubuntu | .deb |
Repo metadata signed — InRelease / Release.gpg; optionally dpkg-sig per package |
| RHEL / Fedora / Rocky / Alma / CentOS Stream | .rpm |
rpm --addsign with a GPG key; publish the public key for rpm --import |
| SUSE / openSUSE | .rpm |
As above |
| Arch | .pkg.tar.zst |
GPG-signed, pacman verifies against its keyring |
| Alpine | .apk |
abuild-sign with an RSA key |
| Container images | OCI | Sigstore/cosign or Notation — the natural fit for the k8s reference in §2.2 |
One GPG key covers every distro family. The custody rule is the one already established for the offline roots in §2.1: the package-signing key is offline and is not in any backup a brain host can reach.
| Item | Cost | Note |
|---|---|---|
| Apple Developer Program (Organization) | $99/yr | D-U-N-S free |
| Apple notarization | $0 | Automated |
| ES entitlement | $0 | Free — the cost is time, 4–6+ weeks |
| EV code-signing certificate | $249–$411/yr | The dominant cost. Covers both drivers and both .exes. Re-verified 2026-10-12 at each CA's own store — SSL.com $349 (1 yr) · $299/yr (2) · $249/yr (3); Sectigo $410.85 (1 yr) · $349.80/yr (2) · $288.20/yr (3). DigiCert is materially dearer (reseller-listed ~$515–$685/yr). Certum, GlobalSign, IdenTrust not priced here |
| EV key custody | $0 – $379+ | A shipped FIPS token is bundled or charged at checkout (Sectigo shows it only in the cart). SSL.com's eSigner cloud signing is a separate subscription; a YubiKey from SSL.com is $379 and may not satisfy EV for kernel-driver use — buy the CA's own EV token or cloud HSM, not a generic key |
| Windows attestation signing | $0 | Free once the EV cert exists |
| Partner Center hardware account | $0 | Microsoft's registration page (updated 2026-08-11) lists no fee; the cost is the EV certificate it requires |
| Microsoft Artifact Signing (optional) | $9.99/mo Basic · $99.99/mo Premium | User-mode .exes only — not the drivers. ⛔ Public Trust not issued to Indian entities (§2.6.3.1); redundant once the EV cert exists |
| Windows, all-in | ≈ $250–$411/yr | One EV cert: ≈$349–$411 for a single year, or ≈$747–$865 up front for three; everything else $0 |
| Linux (all flavors) | $0 | Own GPG key |
| WHQL / HLK | Out of scope | Only for Windows Update distribution |
Testing, all three OSes: $0. Cost is a spare Mac and a disposable Windows VM.
Why Windows costs 2.5–4× Apple, and why its price varies when Apple's is fixed (owner question 2026-10-12). The two platforms put the identity check in different hands:
Production year one: ~$400–$700, dominated by the EV certificate.
lsm=bpf. All three enforcement planes can be fully exercised while the approvals sit in queues.The scheduling insight this section exists to record: the two approval gates are on different clocks and neither is on the critical path of development. Apple's is a queue we join and wait out; Microsoft's is a purchase plus a days-long vetting, made once funded. Starting Apple's early costs nothing and removes the only multi-week external dependency in the programme.
The publisher-signature ceremony (§2.1, D05-14) is untouched and remains the mechanism by which an agent
verifies its own updates. OS signing is additive: it satisfies the platform, while eventlake-releasesign
satisfies us. A signed-and-notarized binary that fails publisher verification must still be rejected by
the agent — the two checks are independent by design, and neither is permitted to stand in for the other.
Volume-local decision ids are D05-nn so they cannot collide with the platform-wide D-numbers (D3 rate
limiting, D4 the perf baseline, D5/D5b/D5c the storage doctrine, D17 bounded RAM, D21 the hardened server +
graceful drain, D22 durable replay, D27 the node-OS gate, N4/N12/N13/N14, O1/O3, W32/W40/W42/W44, SR2), which
are referenced by their own names.
| ID | Decision | Date | Alternatives REJECTED | Why | Status |
|---|---|---|---|---|---|
| D05-01 | Two binaries, both stdlib-only Go, CGO off, static — the brain and the edge agent, nothing else to install | standing | a multi-service deployment (broker + workers + a separate store) — REJECTED: it multiplies the operational surface for a platform whose whole durability story fits in one directory | A 7 MB static agent and one unit file is what makes the "boring on purpose" on-prem shape true | BUILT |
| D05-02 | TLS terminates at a fronting proxy; the process serves plain HTTP on private interfaces | standing | native TLS on the management listener — REJECTED as needless certificate plumbing when every deployment already trusts a proxy. Two carve-outs stand: gRPC (h2 REQUIRES TLS → native, own listener, own cert) and edge↔brain (native mTLS, identity-bearing) | Auth is in-process, so a bypassed proxy exposes cleartext, never an unauthenticated surface (I8) | BUILT |
| D05-03 | The public subject plane is a SEPARATE listener serving PublicHandler() only |
standing | one listener with path-based filtering — REJECTED: a routing mistake would expose the management plane to the internet | Management/cluster/metrics/console are refused on the public listener by construction, not by config | BUILT |
| D05-04 | The cluster mesh is a distinct trust domain on an isolated segment/overlay; native mesh mTLS is deferred | standing | running gossip on the management listener — REJECTED: different peers, different trust, different blast radius | Certificate plumbing (Certificates-issued node certs) is designed, not wired — so the honest control today is network isolation | ⛔ SUPERSEDED 2026-09-16xd (owner-decided): native mesh mTLS SHIPPED, fail closed. The "distinct trust domain" half STANDS. The deferral fell to F9 of the 09-05 security scan: isolation was a deployment convention nothing enforced, and the listener served cross-tenant lake aggregates and an SSRF to anyone who reached it. Node certificates are operator-issued PEM files (EVENTLAKE_CLUSTER_TLS_{CERT,KEY,CA}) — the Certificates-plane issuance this row designed is still not wired to the mesh. |
| D05-05 | Durable mode is ONE env var (EVENTLAKE_SEGMENT_DIR), and write-through durable-on-return is the DEFAULT |
standing | making group-commit the default for throughput — REJECTED: the fast shape trades an ≤ 5 ms/256-event loss window, and a default must never silently trade durability | EVENTLAKE_SEGMENT_GROUP_COMMIT=1 is an opt-in you choose deliberately, measured at −3.7 % durable tax |
BUILT |
| D05-06 | Readiness gates traffic; liveness never does | standing | one health endpoint — REJECTED: an orchestrator would either route to a half-replayed brain or kill a healthy one mid-replay | /v1/readyz 503 until replay completes; /v1/healthz always on, unauthenticated (I3) |
BUILT |
| D05-07 | unverified > 0 is kept and flagged, never dropped |
standing | discarding records that fail seal verification on replay — REJECTED: it destroys exactly the evidence an integrity failure exists to surface | Acceptance for every restore is the boot line reading 0 unverified (I4) |
BUILT |
| D05-08 | platform-keys.json persists every boot's verify key and travels WITH the segments |
standing | keeping only the current key — REJECTED: records sealed by prior incarnations would stop verifying, turning a normal restart into a fleet of unverified rows |
It is part of the segment dir as a unit, not a separate artifact (I5) | BUILT |
| D05-09 | Durable mode loads a PERSISTED root KEK, and a malformed key file fails the boot loudly | 2026-07-14 | (a) the per-boot minted KEK — REJECTED: sealed enrollments would be unopenable after exactly the restart durability exists to survive; (b) minting a replacement on a bad file — REJECTED: silently re-keying orphans every sealed seed on disk | EVENTLAKE_KMS_KEK from a secrets manager is preferred because it keeps the KEK out of the lake directory entirely; the 0600 root-kek file is the fallback (I7) |
BUILT |
| D05-10 | The root KEK is IN the backup set; the shred keystore (and EVENTLAKE_BYOK_DIR) is EXCLUDED — and the rule does not generalize |
standing | one uniform policy for "key material" — REJECTED in both directions: backing up erasure keys makes a provable erasure undoable; excluding the KEK makes every subject re-enroll | Shred keys protect a promise to FORGET; the KEK protects the ability to REMEMBER. Different stores, different policies (I6) | BUILT |
| D05-11 | Shutdown closes the segment store LAST, after draining requests and closing the spine | standing | closing the store with the process — REJECTED: a deferred group-commit window would be lost at exactly the moment an operator believes they performed a clean stop | TimeoutStopSec=20 / terminationGracePeriodSeconds: 30 exist to make room for this sequence (D21) |
BUILT |
| D05-12 | EVENTLAKE_MEMORY_HORIZON bounds RAM, and the ALWAYS-ON roll janitor keeps it bounded at runtime |
2026-07-17 | horizon-at-boot only (the prior state) — REJECTED once measured: nothing rolled at runtime, so RSS grew unbounded and the horizon was a boot-time fiction | EVENTLAKE_ROLL_EVERY defaults to 5 m and 0 disables; the pair is one control, not two. This supersedes the original runbook's env list |
BUILT |
| D05-13 | Every autonomy loop is off by default | standing | shipping a "sensible" always-on scheduler set — REJECTED: it taxes every lab and dev instance, and an unexpected outbound feed poll is a surprise in an air-gapped enclave | The one exception is EVENTLAKE_FORWARD_TICK, and only because registering a forwarder is itself the opt-in (I9) |
BUILT |
| D05-14 | Enforcement at the edge is dry-run by default, opt-in per leg | standing | one -enforce switch arming every leg — REJECTED: the firewall leg, the in-kernel eBPF LSM exec-block, the self-update swap and the resident watchdog carry different risks and different prerequisites |
Four switches, four decisions; the conductor gate sits upstream of all of them (I10) | BUILT |
| D05-15 | nftables-first on Linux with an iptables fallback; a pf anchor on macOS; named netsh rules on Windows | standing | a single lowest-common-denominator backend — REJECTED: element add/delete in a dedicated set is naturally reversible and auditable in one look, which iptables chains are not | On macOS the first block appends a marked, append-only 2-line hook to /etc/pf.conf because pf only evaluates referenced anchors — self-healing, and removable in two lines |
BUILT |
| D05-16 | The exec-block leg is REVERSIBLE by construction — the inverse command removes the deny entry, resolved from a remembered path→inode map | standing | treating in-kernel exec denial like a kill-process (irreversible) — REJECTED: reversibility is conductor guardrail #9, and an irreversible leg would be held for a human every time |
<state>/exec-deny.json means a rollback resolves the key even if the file was deleted. Fail-closed: an unloaded driver or bad target reports NOT applied, never a fake block |
BUILT |
| D05-17 | WinDivert is deliberately NOT used for the Windows transparent-redirect leg | standing | WinDivert — REJECTED: LGPL, off the SPDX allowlist (the hard licence constraint) | Instead: a WDK-built, signed first-party WFP callout driver preferred, with the built-in netsh interface portproxy IP-Helper relay as the userland fallback. Shipping the driver unsigned/unbuilt is the one remaining caveat |
BUILT (two-tier), driver signing open |
| D05-18 | A WAF block returns a bare 403 that names no rule and echoes no payload | 2026-07 | a descriptive block page — REJECTED: a block page that names the signature is a bypass-tuning oracle | The hit is reported to the brain as it/enforcement/block; the attacker learns nothing |
BUILT |
| D05-19 | shadow and block compose as an AND — shadow may downgrade a block program; arming a leg may never promote an alert program into a block |
2026-07 | letting the agent flag override the compiled program's action in both directions — REJECTED: the brain's trust bar would become advisory | The honest rollout: watch what a freshly compiled catalog would have blocked before it blocks anything | BUILT |
| D05-20 | http.path MUST carry the full request target — path AND query |
2026-07 | inspecting the path alone — REJECTED once stated: most injection payloads ride in the query string, and the brain's arm inspects the full target, so a half-target edge would silently under-inspect | the Sensor's own proxy satisfies this by construction; an external data plane calling /inspect must too. This is a deployment contract, not an implementation detail |
BUILT |
| D05-21 | The inline IPS fails OPEN; the DNS sinkhole drops the malformed — the inverse postures are deliberate | standing | one uniform posture — REJECTED: a virtual patch shields SPECIFIC known exploits in front of an unpatchable asset and must never become a self-inflicted outage; a sinkhole resolver's job is to refuse | Fail-open/fail-closed for the resolver itself stays a per-deployment policy (ISP resolver vs enterprise egress) | BUILT |
| D05-22 | The collector spool is at-least-once; a per-record fsync'd cursor is REFUSED | 2026-07 | an fsync'd per-record cursor that would close the duplicate window — REJECTED against the footprint SLO (a synchronous write per record on the collector path) | Duplicate raw frames decode to duplicate events; the loss this prevents is unrecoverable (I1) | BUILT |
| D05-23 | A permanently-refused (4xx) frame is dropped and counted, never wedged | 2026-07 | retrying every failure identically — REJECTED: the brain will refuse THIS frame on every replay, so it would sit at the FIFO head blocking every good frame behind it | The spool exists for an OUTAGE (transport failure, 5xx, 408/429) where the uplink, not the frame, is at fault. A fresh 4xx never enters island mode at all | BUILT |
| D05-24 | Ordering is strict: a fresh batch queues BEHIND an existing backlog | 2026-07 | draining live traffic first for freshness — REJECTED: the brain would receive interleaved streams and replayed telemetry could be overtaken by live telemetry | One ordered stream per leg, at the cost of recency during recovery | BUILT |
| D05-25 | Absent ≠ zero; an offline agent's spool figure is a FLOOR; spool_legs is the footnote and spool is the answer |
2026-07-21 | rendering a missing spool key as 0 — REJECTED: that is unknown presented as healthy, and it is worst precisely when an agent has been spooling unobserved |
The breakdown degrades independently: malformed or over-claiming spool_legs is dropped and the totals kept (I13) |
BUILT (SR2, end to end) |
| D05-26 | Rings, not broadcasts — canary 1 % (1 h) → early 10 % (4 h) → broad 50 % (12 h) → fleet 100 % (24 h), with a REQUIRED sha256 digest | standing | a global same-instant content push — REJECTED with a named negative benchmark: the 2024 channel-file incident (no rings, no health gate, no halt) | An unverifiable artifact never ships; ring membership is a deterministic FNV bucket, so which host canaries is reproducible and auditable (I11) | BUILT (O1) |
| D05-27 | Promotion is earned, never scheduled — and the gate is fail-closed in order: failed apply ⇒ halt · adopted-then-dark ⇒ halt · no evidence ⇒ no promotion | standing | time-based promotion — REJECTED: soak time without adoption evidence measures nothing. Silence must HOLD, never advance | Adopted-then-silent is the crash signal a broken update leaves. Halting needs no gate — stopping distribution is always safe (I12) | BUILT |
| D05-28 | A cleared gate still passes the conductor, with BlastRadius = the next ring's true measured size | standing | auto-promoting once health is clean — REJECTED for the fleet-wide hop: a 100 % ring on a large estate deserves a human | Canary→early auto-executes; the fleet hop is HELD for step-up release | BUILT |
| D05-29 | Small fleets must PIN canary hosts | standing | trusting the percentage on any fleet size — REJECTED once stated: a 1 % bucket of 20 agents selects nobody, and an empty ring holds forever | Recorded as operator guidance rather than a silent auto-widen, because auto-widening would defeat the ring | BUILT (documented) |
| D05-30 | The air-gap rollout bundle carries TRUST, not BYTES | standing | shipping payloads inside the signed bundle — REJECTED: it conflates distribution with authorization and bloats the transfer | The platform-signed (ed25519) MANIFEST imports fail-closed; the separately carried payload is verified against its digest | BUILT |
| D05-31 | BYOK/HYOK revocation is crypto-shred at tenant scale, and it is DOUBLY gated | 2026-07 | a single-approval revoke — REJECTED: destroying a root is inherently irreversible and renders every subject's spans [erased] in the store, segments and every backup |
Step-up to propose → conductor HOLD (BlastRadius = live DEK count) → step-up again to approve. The fail direction is always "the root survives" | BUILT (W42) |
| D05-32 | Whole-file segment encryption stays a deployment-layer control (LUKS/BitLocker/encrypted volume) | standing | per-tenant encryption of whole segments — REJECTED as structurally impossible: one segment interleaves many tenants, so a per-tenant root cannot cover a shared file | The per-tenant unit is the classified-span token plane — the data that matters. Single-tenant deployments combine both layers for full coverage | BUILT, scope stated |
| D05-33 | replicas: 1 per brain; horizontal scale is more brains, not more replicas |
standing | scaling a StatefulSet on one PVC — REJECTED: the lake is single-writer, so a second writer is not slow, it is corrupt | Sharding axes in preference order: site/region → tenant cell → telemetry domain (I2) | BUILT |
| D05-34 | HA today is warm standby; active-active / raft-replicated lake is deferred by doctrine | standing | adopting go.etcd.io/raft now — REJECTED under D5: a perf-tier dependency enters only with a measured justification |
RTO = replay time, RPO = replication lag. Named candidate, not a silent absence | DEFERRED, recorded |
| D05-35 | One artifact set for SaaS and self-managed — there is no second-class on-prem build | standing | a cut-down on-prem distribution — REJECTED: the sovereignty buyer is a first-class customer, and divergence would make on-prem the untested path | deploy/ (Dockerfile, compose, k8s, systemd) is what both models run |
BUILT |
| D05-36 | Choose Kubernetes only when the customer already operates Kubernetes | standing | K8s as the recommended default — REJECTED: the platform gains nothing from K8s it does not already do itself (durability, replay, readiness, drain) | The cluster is for their operational uniformity, not ours; the boring systemd shape stays the on-prem default | STANDING |
| D05-37 | A pooled cell is deliberately kept small (≤ ~5 k EPS aggregate, tens of tenants) | standing | packing pooled cells to the node ceiling — REJECTED: the cell IS the blast radius, and tenancy isolation is in-process either way | Chronic noisy neighbours get promoted to a dedicated cell rather than throttled forever | DESIGNED |
| D05-38 | Cell autoscaling and tenant rebalancing are PROCEDURAL, not automatic | standing | automating cell assignment now — REJECTED as unproven: moving a tenant is a data-locality decision with residency consequences | Operator action on the control plane; the OEM control-plane program owns the automation depth | PROCEDURAL (honest) |
| D05-39 | Sizing constants are versioned, and every one is labelled M or P | 2026-07-03 | quoting a single "capacity" figure — REJECTED: an unlabelled number becomes a promise the next box cannot keep | Constants version v1; the pilot's measured numbers override every P (I18) | BUILT |
| D05-40 | EPS_PER_VCPU = 1,500 deliberately DERATES a laptop measurement rather than promising an unratified server number |
2026-07-03 | publishing the ≥ 50 k EPS/node ambition as a planning constant — REJECTED: the measured basis is a 4-core ultrabook, and the server-class target is an open ratification | 18.2 k ceiling ÷ 8 threads, −35 % for analytics fan-out, streaming twins, durable write-through and query load. A server core outruns this — the margin is deliberate | P, derived from M |
| D05-41 | The socket family is deliberately NOT folded into the mixed-estate mean | 2026-07-15 | averaging it in — REJECTED: the auditd socket leg is an opt-in add-on, and folding it in would silently move a constant the whole guide is built on | Measured separately (520 B stored), so the socket-vs-estate ratio can be reasoned about explicitly | M |
| D05-42 | -S bind is enabled unconditionally; -S connect is a budgeted opt-in |
2026-07-15 | enabling both by default — REJECTED after measurement: ~30 connect-EPS is the ENTIRE 90-day budget of the reference node, and the rate is set by application workload, not by audit policy | Binds are a startup burst then near-silence, so the backdoor-bind sensor is effectively free. Measure a canary for a busy hour before any fleet rollout | BUILT + PRICED |
| D05-43 | Push the decoder's drop rules into the kernel where possible — but auid filtering is a SECURITY TRADEOFF, not a free win |
2026-07-15 | recommending -F auid>=1000 as a standard volume control — REJECTED as a default: an attacker who exploits a service inherits its unset auid and becomes invisible to it |
Prefer saddr_fam + success first; reach for auid only when the measured rate leaves no choice, and record that you accepted the gap. auditctl has no destination IP/port filter at all — plan the volume, you cannot carve it |
DOCUMENTED |
| D05-44 | Archive compression is planned at 3× against a measured 4.3× | 2026-07-03 | planning at the measured mean — REJECTED: flow-heavy estates compress worst (~2.4×) and an under-provisioned archive is discovered at the compliance audit | Conservative by intent, like every other P here | P |
| D05-45 | Retention granularity is a segment boundary, not a per-event delete | standing | per-event deletion for sub-day retention — REJECTED by the WORM design: erasure is crypto-shred, not deletion | Whole-segment pruning is what keeps the store append-only and the seals verifiable | BUILT |
| D05-46 | The public-site calculator and §2.3.12 are one artifact in two forms — change a constant in one, change it in the other, in the same commit | standing | letting the calculator drift as "marketing" — REJECTED: a prospect's self-serve estimate and the operator's plan must be the same arithmetic | The calculator additively folds the live /v1/metering/rate-card ceilings when opened against a deployment; offline it uses these defaults |
BUILT |
| D05-47 | The entire SNMP/flow/pcap stack is clean-room from the public specs | standing | a third-party SNMP/flow/capture/DPI library — REJECTED under the clean-room + permissive-licence constraints | It also means the BER codec, the stateful v9/IPFIX template cache and the DPI dissectors are ours to extend for OT protocols | BUILT |
| D05-48 | An automated SNMP Set routes through the conductor's Actuator seam |
standing | exposing Client.Set to automation directly — REJECTED: a SET changes device configuration, which is a world-changing action like any other |
The client is the low-level primitive; the gate is at the spine/actuation layer (I17) | BUILT (primitive) + gated |
| D05-49 | Device detection reuses the existing rule path — no new detector engine | standing | a bespoke NMS alerting engine — REJECTED: Category:"device" + Outcome:failure already carries severity, so device-down, interface-down and linkDown traps ride alert→case unchanged |
The NOC and SOC therefore share one alert path, which was the point of putting NMS on the lake | BUILT |
| D05-50 | Heavy DPI is an EDGE job; the brain receives metadata and reductions, not raw packets | standing | shipping full captures to the brain — REJECTED on both volume and privacy grounds | WithSessionEvents() (per-packet 5-tuple records) is off by default for the same reason: the brain wants metadata |
BUILT |
| D05-51 | Recovery that has not been drilled is a claim, not a capability — so the drills ARE tests | 2026-07-07 | asserting durability from the design argument — REJECTED against Jepsen's discipline: inject the fault and check the invariant | They run in CI on every change, so a regression that breaks a durability invariant fails the build, not the customer (O3) | BUILT |
| D05-52 | A drill that cannot run skips VISIBLY with its reason; a fixture that fails to drive the real cascade fails LOUDLY | 2026-07-07 | letting an unwired drill pass quietly — REJECTED: replay equality over nothing proves nothing, and a vacuous pass is worse than a failure | There is an explicit vacuous-pass guard in the unit tests (I15) | BUILT |
| D05-53 | resil imports only event/lake/fabric — never spine, never a store implementation |
2026-07-07 | letting the harness import the assembly — REJECTED under D5 layering: the assembly imports the harness and adapts to the Target seam |
The in-memory memStore is a real serialization boundary — events round-trip through canonical JSON so seals must re-verify, not be handed back as the same pointers |
BUILT |
| D05-54 | The split-brain window is MEASURED, not denied; post-heal uniqueness is asserted | 2026-07-07 | claiming partition-freedom — REJECTED: a failure detector that declares the other side dead is doing its job, and the honest artifact is the measured window | Convergence is checked under three different merge orders (the CRDT order-independence claim), and exactly one owner must hold the single-writer lake key after heal | BUILT |
| D05-55 | What in-process injection cannot cover is named and put on the ops calendar | 2026-07-07 | treating the suite as complete coverage — REJECTED: it approximates faults at the seam, and OS/device-layer semantics live below it | Five scheduled rehearsals with cadences; the device layer below fsync stays out of scope (power-loss testing needs hardware rigs) | DOCUMENTED |
| D05-56 | OS code signing is ADDITIVE to the publisher ceremony, never a substitute for it | 2026-08-02 | treating a notarized/attested binary as trusted by the agent — REJECTED: the platform's signature answers "is a revocable entity behind this", ours answers "did we build this"; they are independent questions | A signed-and-notarized binary that fails eventlake-releasesign verification is still rejected by the agent (§2.6.7) |
DOCUMENTED |
| D05-57 | The Apple ES entitlement request starts FIRST, because it is the only multi-week external dependency | 2026-08-02 | sequencing credential acquisition by cost — REJECTED: cost is instant to resolve, a review queue is not, and nothing in development is blocked on the grant | Ordering in §2.6.6 is by lead time; the request is free, so starting it early has no downside | DOCUMENTED |
| D05-58 | Every enforcement plane keeps a FREE, no-approval test path, and that constrains the design | 2026-08-02 | accepting a design that can only be exercised with a granted credential — REJECTED: an untestable plane is an unverified one | eventlake_execblock is a launchd daemon, not a sysex — which is what keeps the SIP-disable test path open (§2.6.2.4); Windows uses test-signing; Linux needs nothing |
BUILT (the property holds) |
| D05-59 | Windows drivers target ATTESTATION, not WHQL; Trusted Signing is for the agent binary only | 2026-08-02 | buying Trusted Signing expecting driver coverage — REJECTED on Microsoft's explicit exclusion of it from the Hardware Program; and WHQL — REJECTED as scope, it buys only Windows Update distribution | One EV certificate covers both drivers; the April 2026 kernel-trust change makes Partner Center the surviving path anyway (§2.6.3.4) | DOCUMENTED |
| D05-60 | The shard unit for scale-out is ONE TENANT PER BRAIN, and it is named now rather than discovered later. A brain stays single-writer over one lake; capacity beyond one node comes from placing tenants on different brains, with the API resolving tenant → brain and cross-tenant reads riding the existing gossip scatter-gather. The API-side resolver seam SHIPPED 2026-08-22au as api.TenantPlacement (api/placement.go), consulted once at the auth choke point. ⛔⛔ The mechanism this row originally named — "spine.Spine becomes an interface at the api boundary" — was REFUTED by measurement on the day it was to be built and MUST NOT be re-derived: api reaches 601 distinct members on the spine value across 838 call sites, and 18 of them are struct FIELDS (Lake, Conductor, MFA, Intel, Cases, …), which no Go interface can express — so that change is a field-to-accessor conversion on a type read across the whole module, not a one-session interface. Worse is the shape it invites: the only tractable implementation of a 601-method interface is to EMBED it and override the routed methods, and every method not overridden then answers from the LOCAL brain for a tenant that lives elsewhere — only 209 of the 583 methods even name a tenant in their signature, so 374 cannot be classified mechanically. A seam whose default is "answer from the wrong brain" is a seam against the isolation boundary this row exists to protect |
2026-08-22 | (a) a multi-writer lake — REJECTED, and this is the load-bearing rejection: single-writer is what makes the per-event seal chain (Seal.Prev), the segment digest chain and the D22 replay story sound. Two writers on one chain is not a performance problem, it is a correctness one, and recovering from it means abandoning the tamper-evidence claim the product is sold on. (b) raft/consensus over the lake — REJECTED for v1 on the same D01-02 gate every perf-tier dep faces: no measured need, and it buys agreement we do not have a conflict to resolve. (c) sharding by time or by category — REJECTED: tenancy is the isolation boundary the platform already enforces everywhere (lake shards by it, the analytics round loops on it, RBAC forces it), so any other axis introduces a second boundary that must then be reconciled with the first. (d) leaving it undecided until a tenant outgrows a node — REJECTED because that is the expensive moment to decide: api holds a concrete *spine.Spine in 627 route registrations, and the cost of introducing the seam rises with every one |
The measured ceiling is real and close: ~9k eps sustained, ~18–20k on a 4-worker probe, replicas: 1 in the shipped manifest — and the manifest says why, correctly. What was missing was not the capability but the DECISION, and an undecided shard unit is what turns scale-out into a rewrite. The cluster half already exists: gossip scatter-gathers aggregate queries across independent brains and merges HLL/DDSketch/Count-Min partials, with unreachable nodes reported rather than fatal — so the fan-out story is built and only the placement story is missing. ⛔ The interface change is the whole of the near-term work; no placement, rebalance or migration mechanism is proposed here, and a tenant moving between brains is explicitly out of scope until someone needs it |
ACCEPTED; THE SEAM IS BUILT (2026-08-22au), PLACEMENT STILL IS NOT. api.TenantPlacement + the shipped default SoleBrain (one brain, every tenant on it — true of every deployment today, replicas: 1), checked in auth() after authentication and RBAC and BEFORE the licence gate, refusing a misplaced tenant 421 Misdirected Request and counting it as platform_api_misdirected_total (aether_api_misdirected_total until 2026-10-01sg, still served as its twin through the Kind C window). ⭐ It is fail-CLOSED and its cost does not rise with the route count, which was this row's stated reason for hurrying. ⛔⛔ KNOWN BLIND SPOT, ENUMERATED AND GUARDED: the check rides auth(), which builds 585 of 632 route handlers; of the 47 it does not, 17 resolve a tenant on their own — the login/step-up/SSO/OIDC/enrolment/invitation/recovery/signup plane — and each resolves it at a different point in its own body. api/placement_coverage_test.go fails when either list moves. Those 17 doors are the next unit. ⛔ fabric.Cluster.Owner() IS THE WRONG ORACLE for this and is already in the tree: a Maglev ring reassigns keys when the alive set changes, which for a durable single-writer lake means handing a tenant to a brain holding none of its history. Placement of a lake is a DECLARED, durable assignment, never a hash |
Recorded disagreements between sources, resolved newest-wins.
deployment.md §12 predates the runtime
roll janitor and does not mention EVENTLAKE_ROLL_EVERY (shipped 2026-07-17). Newer wins: the janitor is
always-on with a 5 m default, and without it nothing rolls at runtime → unbounded RSS, so the horizon
alone is a boot-time fiction. §2.1.4 and §4.3 carry the corrected pair. EVENTLAKE_BYOK_DIR/_KEK are
likewise absent from the quick-start block while being live in the key-custody section — both are listed
in §4.3.docs/reference/build-status.md is
authoritative for both. The constants are deliberately conservative (D05-40) and are not revised here:
they re-issue as v2 when the server-class baseline lands. Do not quote 19 k as a planning constant.deployment.md §13 lists "Sensor agent self-update executor" as
SHIPPED (2026-07-10, opt-in; watchdog tier same day) while the table is titled "explicitly deferred."
That was deliberate continuity — the row had been a deferral and its closure is recorded in place. §6
keeps it as a closed row for the same reason, with its one narrowed residual intact.EVENTLAKE_METER_SCHEDULE appears as both an autonomy knob and a commercial mechanism. It is one loop:
the sealed it/platform/usage rollup is simultaneously the operator's meter and the licence's auditable
measurement. Its semantics belong to 07-commercial-and-gtm.md#licensing-model; the cadence is here.| Event type | Emitter | Notes |
|---|---|---|
it/device/health (up/down) |
snmp poller |
down is High/failure + error; metrics (uptime, gauges, counters, per-second rates) ride as attributes |
it/device/interface (down) |
snmp poller |
Medium/failure — ifOperStatus = 2 |
it/device/trap (link-down, other) |
decode/snmp |
linkDown (v2c) / generic-trap=2 (v1) is High/failure; coldStart and auth-failure are informational; all varbinds MIB-named |
it/network/flow |
decode/netflow (v5/v9/IPFIX/sFlow) |
Subject = src host, Object = dst host; counts scaled by sampling rate. Observational (SevInfo/success) — never itself an alert |
it/network/l7 |
decode/pcap |
one per dissected DNS/TLS/HTTP message |
it/network/session |
decode/pcap with WithSessionEvents() |
per-packet 5-tuple forensics record — off by default |
it/resilience/report |
resil via assembly.ResilienceDrill() |
platform-scoped, free in metering; healthy/passed/failed/skipped + per-drill metrics |
it/platform/rollout · it/platform/rollout-ack |
rollout |
full-state snapshot per transition + per-apply result, D22-replayed |
it/platform/usage |
the meter loop | the sealed per-tenant licence rollup |
it/platform/keyroot |
byok |
custody enrolments, rotations and revocations; boot replay re-destroys revoked generations |
it/enforcement/* |
the Sensor legs | dns-sinkhole · virtual-patch · edge-detection · inline-dlp · block (activity inline-waf) |
it/process/exec · it/authentication · it/network/flow |
the brain's decoders over forwarded raw frames | from -collect-auditd (the socket leg feeds the flow twin), -collect-winlog, -collect-eslogger, -collect-journald (auth only) |
it/log/syslog |
the brain's journald decoder |
every journal entry that is not one of the recognized auth forms, carrying the journal's trusted _SYSTEMD_UNIT/_COMM/_EXE/_CMDLINE/_PID/_UID provenance |
it/network/flow (connect·listen·sniff) |
the netwatch sensor via decode/netwatch |
on Linux each row also carries the owning process's container / container_runtime / pod_uid — read from /proc/<pid>/cgroup for the pid the socket's inode resolved to. An unattributed socket carries NOTHING rather than a "host" placeholder |
it/file/change |
fim over the macOS ES stream |
create/rename/unlink; a rename ships as MOVED_FROM+MOVED_TO twin records |
Full semantics live in 08-interfaces-and-extensibility.md#api-management-plane; all are Bearer <key>,
deny-by-default RBAC, tenant-forced and role-masked server-side.
| Route | Purpose |
|---|---|
GET /v1/healthz · GET /v1/readyz |
liveness (always on, unauthenticated) · readiness (503 until boot replay completes) |
GET /metrics |
platform self-observability (with EVENTLAKE_OBS=1) |
POST /v1/ingest · POST /v1/ingest/raw |
the wired telemetry paths (canonical events · raw frames from the collector legs) |
GET /v1/sensor/agents |
the fleet board — queued, spool{records,bytes,dropped}, spool_legs[], last_seen, reported version |
GET /v1/sensor/update · POST /v1/sensor/update/ack |
the pull-based update directive + the honest ack (ok:false halts the rollout in the same call) |
POST /v1/sensor/session/chunk |
brokered-session transcript chunks (strictly sequenced) |
POST /v1/rollouts/{id}/promote · GET /v1/rollouts/{id}/bundle · POST /v1/rollouts/import |
human release of a HELD ring hop (step-up) · the signed offline manifest · fail-closed air-gap import |
/v1/keyroots (enroll/rotate/revoke) · GET /v1/keyroots/attestation |
the BYOK/HYOK ceremony (admin + step-up; revoke doubly gated) · the offline-verifiable custody statement |
GET /v1/resilience · POST /v1/resilience/drill |
newest drill snapshot (answers {"ran": false} honestly) · run the hermetic suite (admin, no step-up) |
GET /v1/metering/rate-card |
the published plan ceilings the sizing calculator folds in |
POST /v1/forwarders · POST /v1/reports/schedules · POST /v1/intel/feeds |
the registrations the §2.1.7 scheduler loops drain |
The complete inventory is conduit/cmd/eventlake-api/main.go (each variable's authoritative semantics live as
comments beside its parse site, kept in lockstep by review). The operationally load-bearing set:
Durability (the five that define the lake's operational shape):
| Variable | Effect |
|---|---|
EVENTLAKE_SEGMENT_DIR |
enables durable mode: write-through hot-segment store + control-plane replay on boot |
EVENTLAKE_SEGMENT_GROUP_COMMIT |
opts into the deferred 256/5 ms group-commit shape (−3.7 % tax, ≤ 5 ms/256-event loss window) |
EVENTLAKE_MEMORY_HORIZON |
bounds RAM: boot leaves older records disk-resident, reads union them back in |
EVENTLAKE_ROLL_EVERY |
the always-on runtime roll janitor (default 5m; 0 disables) — without it nothing rolls at runtime |
EVENTLAKE_RETENTION_EXPIRE_CONTROL_PLANE |
⛔⛔ a DATA-LOSS setting, not a tuning knob (=1 only). Retention does not delete control-plane records by default (ADR D01-54) — the ones boot replay rebuilds this node FROM: enabled connectors, issued credentials, suppressions, kill-switch scopes. Setting this restores the pre-2026-08-29k behaviour, in which they age out at ColdFor (365 d) and the next restart rebuilds nothing from them, while the boot reports a clean replay (the guard keys on ControlPlaneDiscarded == 0, which is what an absent record produces). It exists because the exemption's cost is real: segment truncation is prefix-only, so one pinned segment retains every segment after it and disk reclamation can stop. Read store_segments_pinned before reaching for it — 0 means nothing is being held and this would reclaim nothing; -1 means the store cannot answer the category question, so truncation is already off |
Core & listeners: EVENTLAKE_API_ADDR (:8080) · EVENTLAKE_NODE_ID (pin it — changing it makes the
node a different brain and the boot refuses; rename via EVENTLAKE_ADOPT_NODE_ID, §2.1.8a) · EVENTLAKE_API_KEY (else minted+printed
once) · EVENTLAKE_UI_ROOT (serve the console; auto-found when run from the repo) · EVENTLAKE_PUBLIC_ADDR ·
EVENTLAKE_GRPC_ADDR + EVENTLAKE_GRPC_TLS_CERT/_KEY · EVENTLAKE_CLUSTER_ADDR/_ADVERTISE/_PEERS.
Keys & other durable state: EVENTLAKE_KMS_KEK (the persisted root KEK — in the backup set) ·
EVENTLAKE_MESSAGING_DIR · EVENTLAKE_SHRED_DIR (excluded from backups) · EVENTLAKE_BYOK_DIR + EVENTLAKE_BYOK_KEK
(the key-root contract; supersedes EVENTLAKE_SHRED_DIR when both are set; also excluded) ·
EVENTLAKE_BRAIN_SIGNING_KEY + EVENTLAKE_BRAIN_SIGNING_GRANT (the agent trust root's brain half — set together
or not at all; a half-set, mismatched or expired pair FAILS THE BOOT, §2.1.8b. The offline ROOT key is a
third custody policy: not in this host's backup set at any price) · EVENTLAKE_DISCOVERY_DOC (+ the optional
EVENTLAKE_TRUST_ROOT_PUB) — the root-signed ROUTING statement served verbatim at /v1/discover; no key
material, which is the point (§2.1.8c). Unset = this deployment does not do discovery (503, distinct from
an empty document); a document this host can see is wrong FAILS THE BOOT rather than stalling the fleet.
Content the brain SERVES to the fleet: EVENTLAKE_VULNDEFS — the vulnerability-definition bundle
cmd/vulndef-gen emits (P4). It is the SOURCE every host's per-host bundle is compiled from, scoped to that
host's inventory and signed, at GET /v1/sensor/vulndefs. Unset = no definitions, which is a real state the
sweep telemetry reports honestly; a path that is set but unreadable, invalid, or valid-and-EMPTY FAILS THE
BOOT — that operator believes assessment is armed, and a fleet reporting zero vulnerable hosts is
indistinguishable from a silent failure. Not a key and not durable state: it is content, regenerated on a patch
cadence (scripts/check-vulndef-currency.sh; runbook reference/vulndef-src/README.md), so it is not in the
backup set — a lost bundle is re-generated in seconds, not recovered. The same bundle is what the AGENTLESS
leg evaluates (EVENTLAKE_AGENTLESS_VA / POST /v1/vuln/agentless), so an unset EVENTLAKE_VULNDEFS refuses that pass
too rather than reporting a fleet clean. Since P5 the bundle also carries the hand-refined definitions in
conduit/cmd/vulndef-gen/curated/ — pass -curated ./cmd/vulndef-gen/curated when generating; on Windows they
are the ONLY definitions there are, because OSV publishes no Windows ecosystem.
Observability & paging: EVENTLAKE_OBS · EVENTLAKE_OBS_LEVEL · EVENTLAKE_ALERT_NOTIFY/_THROTTLE/_TO. · EVENTLAKE_LEAD_NOTIFY_TO.
Delivery: EVENTLAKE_WEBHOOK_URL/_SECRET ·
EVENTLAKE_SMTP_ADDR/_FROM/_USER/_PASS/_TLS/_AUTH/_TENANT_FROM · EVENTLAKE_FCM_CREDENTIALS.
Autonomy: the full table is §2.1.7 — EVENTLAKE_ANALYTICS_SCHEDULE/_WINDOW ·
EVENTLAKE_STREAMING/_WINDOW/_LATENESS · EVENTLAKE_SOURCE_HEALTH · EVENTLAKE_GOVERNANCE_SWEEP/_GRACE ·
EVENTLAKE_METER_SCHEDULE/_WINDOW · EVENTLAKE_VULN_FEED/_KEV_URL/_EPSS_URL ·
EVENTLAKE_VULN_FEED_HEALTH/_STALE_AFTER · EVENTLAKE_REPORT_SCHEDULE · EVENTLAKE_DECEPTION_ROTATE/_MAX_AGE ·
EVENTLAKE_MYTHOS_SCHEDULE/_WINDOW · EVENTLAKE_VULN_POSTURE_SNAPSHOT/_LOOKBACK ·
EVENTLAKE_HARDENING_POSTURE_SNAPSHOT/_LOOKBACK · EVENTLAKE_GOVERNANCE_POSTURE_SNAPSHOT · EVENTLAKE_AGENTLESS_VA ·
EVENTLAKE_INTEL_TAXII_POLL/_URL/_COLLECTION/_APIROOT/_TOKEN/_SOURCE · EVENTLAKE_FORWARD_TICK.
Enrichment & actuation: EVENTLAKE_GEOIP (operator-loaded IP→geo CSV; a bad table fails boot loudly) ·
EVENTLAKE_SENSOR (switch the conductor off NoopActuator) · EVENTLAKE_AGENT_KEY (sensor side, or -key).
API Setu conformance harness (the ops test rig behind /v1/apisetu, conduit/apisetu — off unless
turned on, because a probe is an outbound call to a GOVERNMENT gateway made under this deployment's
identity): EVENTLAKE_APISETU_BASE is the switch — unset, every probe is a 503 that names this variable,
never a call and never an error blamed on API Setu. The base is refused unless it is https (the response
carries a citizen's marksheet; there is no development convenience worth reading that off the wire) and, unless
EVENTLAKE_APISETU_ALLOW_PRODUCTION=1, unless its host is the public sandbox — a production call needs a
per-publisher credential and a subscription this platform does not hold, so reaching one stays a separate
explicit act rather than a consequence of setting a URL. EVENTLAKE_APISETU_ATTEMPTS (default 5, max 25) is
how many times each endpoint is called and 1 is REFUSED: this gateway fans requests across backends whose
routing tables disagree, so one attempt measures a backend, not an endpoint. EVENTLAKE_APISETU_TIMEOUT (default
10s, max 60s) bounds a single attempt — the sandbox's hung backend never answers. EVENTLAKE_APISETU_APIKEY /
EVENTLAKE_APISETU_CLIENTID carry the sandbox's published demo credentials by default. The offline
document-verification leg needs none of this and works on a node where probing is switched off entirely.
EVENTLAKE_APISETU_CRLS (P4, 2026-08-03) points at a directory of the licensed CAs' published CRLs
(*.crl/*.pem/*.der), loaded ONCE at boot and never fetched at verification time — a
mid-verification fetch would convert the one leg that works offline into one that needs a government CDN,
and would leak which document is being checked, and when, to whoever serves the distribution point.
Unset is the shipped state and it is honest, not broken: with no bundle every revocation answer is
unknown, which is explicitly NOT "not revoked" — the four states are unknown · stale · not_listed ·
revoked and only not_listed clears a certificate. A stale CRL (past its own nextUpdate, or
carrying none) can never clear one, though a serial LISTED on a stale CRL is still reported revoked —
staleness bounds what a list may DENY, never what it may ASSERT. A file that does not parse refuses the
whole bundle rather than being skipped (a half-loaded artefact answers not_listed for the issuers that
happened to parse and unknown for the rest, and nothing on any screen would say which); an unreadable
directory does not fail the boot — unlike EVENTLAKE_VULNDEFS, whose absence would make a scanner claim
CLEAN — but the failure is recorded and rendered on GET /v1/apisetu. Refresh it the way the roots are
refreshed: out of band, on the operator's schedule.
Reporting: EVENTLAKE_REPORT_FONT_DIR — a directory of regular.ttf/bold.ttf/italic.ttf/mono.ttf
(each slot optional). When set, exported report PDFs embed a GID-stable TrueType subset of those faces (full
Unicode; the receipt gains a fonts_digest); unset ⇒ the base-14 WinAnsi default. A present-but-unparseable
font file is fatal (never a silent downgrade).
LLM-security sink declaration: EVENTLAKE_LLMSEC_HTML_RENDERING_SINK=1 declares that the completions the
monitored LLM gateway serves are rendered with raw HTML passed through (a chat UI or mail body that does not
escape the model's markup). It turns on the fenced-code-aware raw-HTML <img> beacon scan — the HTML variant of
ATLAS AML.T0077 LLM Response Rendering (llmsec improper-output-handling, marker html-exfil-image). Unset or
0/false ⇒ that scan is silent, because on an escaping renderer an <img> tag is inert text and a coding
assistant emits them all day; the markdown-image leg is always on. Anything but 1/0/true/false refuses
the boot (cmd/eventlake-api/boot_llmsec.go). Declare it only for a sink that genuinely renders raw HTML.
Model seams (unset ⇒ the copilot stays extractive and embedding uses the clean-room HashEmbedder — the
air-gap defaults; semantics in 08-interfaces-and-extensibility.md#genai-agentic-integration):
EVENTLAKE_LLM_PROVIDER/_MODEL/_URL/_KEY_FILE/_TIMEOUT/_UNMASKED_EGRESS ·
EVENTLAKE_EMBED_PROVIDER/_MODEL/_URL/_KEY_FILE/_TIMEOUT/_UNMASKED_EGRESS ·
EVENTLAKE_MODEL_CANDIDATE_* (the P4 shadow-canary second endpoint — never served to a tenant, only
evaluated). Two operational rules carry across all three sets: a key is a FILE path, never the key itself
in env/args, and remote egress masks by default, fail-closed. Switching the embedding model re-embeds
the in-memory indexes on the next boot; a live index refuses mixed dimensionalities.
Dev/demo only (stable repeat testing — never production): EVENTLAKE_SUBJECT_KEY,
EVENTLAKE_SUBJECT2_KEY, EVENTLAKE_RP_KEY, EVENTLAKE_OPERATOR_KEY. On a production-stamped build these
mint nothing at all unless explicitly set, and on EVENTLAKE_POSTURE=self-managed the demo estate is
not provisioned on any build (cmd/eventlake-api/bootcreds.go, firstboot.go). Leave them unset.
⛔⛔ EVENTLAKE_PLATFORM_KEY IS NOT IN THAT LIST AND WAS FILED THERE IN ERROR (corrected
2026-08-27o). On a self-managed node it is the customer's node-admin — the only principal
that can enable their own connectors — and it is cross-tenant. It is the node's highest
authority, not a demo convenience. Pin it from custody.
Pin these from owner custody on any production node — each is minted randomly when unset, and
on a production build a minted key is written to <state-dir>/boot-credentials (mode 0600) and
printed nowhere:
| Variable | Principal | Why it must be pinned |
|---|---|---|
EVENTLAKE_API_KEY |
bootstrap-admin (admin, tenant default) |
the first console credential |
EVENTLAKE_PLATFORM_KEY |
node-admin / oem-ops (admin, cross-tenant) |
mints tenant credentials; enables connectors |
EVENTLAKE_AGENT_KEY |
mote-agent (ingestor) |
only when Sensor actuation is on |
EVENTLAKE_RADIUS_KEY |
radius-bridge (ingestor) |
only when the F5 RADIUS bridge runs |
EVENTLAKE_BREAKGLASS_KEY |
break-glass (admin, exempt from enforce_sso) |
see below |
⛔⛔ EVENTLAKE_BREAKGLASS_KEY IS THE ONLY WAY BACK IN AFTER enforce_sso. Enabling SSO
enforcement makes the tenant's IdP the sole console door: a local key can no longer mint a browser
session. The single exemption is this credential, every use of it is audited High
(sso_breakglass_sessions_total), and POST /v1/sso/idps now WARNS when a tenant turns
enforcement on without one. It is opt-in and must stay so — minting it by default would leave a
standing SSO-bypassing admin on every node in the fleet, which is worse than the gap it closes.
Provision it before you rely on enforcement, and hold it like a root key.
cmd/mote) flags (the edge contract)-brain · -key · -id · -interval (2 s) · -state · -planes · -owns · -plane · -version ·
-enforce · -enforce-exec · -inspect-addr · -ips-addr · -ips-upstream · -ips-waf block|shadow ·
-ips-redirect · -dns-addr · -dns-upstream · -dns-sinkhole-ip · -artifact-dir · -self-update ·
-watchdog · -session-dir · -wrap-session <id|env> · -collect-auditd · -collect-winlog ·
-collect-eslogger · -collect-journald <FIELD=VALUE,…|all> (Linux; cursor-backed, so it ignores
-spool-bytes) · -spool-bytes (32 MiB per stream-backed leg; 0 disables).
Authority: conduit/edgelink/egress.go; the flags above decide which rows apply. Every capability the
agent advertises at registration carries a grade, and the grade is a ceiling on the KIND of thing that
crosses the wire — never a claim about volume, retention, or who may read it afterwards:
| Grade | What leaves | Examples |
|---|---|---|
| 1 · derived | facts COMPUTED from an observation; the observation cannot be reconstructed | host-metrics (OTel numbers) · dlp-sweep/dlp-removable (identifier TYPES + counts, never values) · dlp-egress · vuln-assess · enforce-harden |
| 2 · observation | the act and its identifiers — path, domain, command line, account, peer | yara-sweep (path + sha256 + the RULE's string ids) · fim-watch (path + digests) · proc-watch · net-watch · dns-telemetry · inventory · persist-watch · the enforcement legs' result lines |
| 3 · content | verbatim bytes a person or a third-party program authored | collect-{auditd,winlog,eslogger,journald,logfile} · otlp-receiver · syslog-recv · hunt-osquery · enforce-collect-full · inline-inspect |
⚠ Grade 2 is not "harmless". dns-telemetry is a browsing history and inventory is an
installed-software census; on a personal device those are the most revealing rows on the list.
⛔ Ten capabilities are grade 3, and each one states its argument in capEgress (re-measured 2026-08-25e: 8 derived · 19 observation · 10 content, of 37 advertised). A grade-3 leg on a
private individual's device needs an explicit answer — local-only evaluation, or aggregation before
egress — before it is armed (docs/consumer-line-proposal.md §1.2). Two of the levers already exist as
separate flags by design: -enforce-collect ships parsed records while -enforce-collect-full streams
the artifact's raw bytes, and -hunt's grade is set by the operator's SQL rather than by the code.
The classification is guarded, not asserted in prose. edgelink/egress_test.go derives the capability
set from cmd/mote's own addCap calls, so a new leg fails the build until it is graded; a row for a
retired leg fails too; a grade-3 row with no argument fails; and the grade-3 set is pinned by name in
both directions. Four legs are checked against BEHAVIOUR rather than their own comments — yara-sweep
and fim-watch by planting a canary in a swept file and searching the framed bytes for it,
dlp-sweep/dlp-removable by the equivalent PAN assertions that predate this table.
| Artifact | Where | Shape / rule |
|---|---|---|
Segments + platform-keys.json |
EVENTLAKE_SEGMENT_DIR |
one unit; in the backup set. Format details: 01-substrate-plane.md#lake-storage-engine |
root-kek |
beside the segments (unless EVENTLAKE_KMS_KEK) |
32 bytes, 0600; malformed ⇒ boot fails loudly; in the backup set |
shred-keys.json |
EVENTLAKE_SHRED_DIR |
0600, atomic rewrites; the destruction point; excluded from backups |
keyroots.json |
EVENTLAKE_BYOK_DIR |
0600, atomic+fsync; DEKs wrapped under their root, roots sealed under EVENTLAKE_BYOK_KEK; excluded from backups |
<state>/spool/<connector>/ |
Sensor | FIFO segments, per stream-backed leg; adopted (and torn-tail repaired) by a successor process |
<state>/artifacts · <state>/updates · updates/self-swap.json |
Sensor | staged payload (<name>-<version>) · verified staging area · the A/B swap custody marker |
<state>/exec-deny.json |
Sensor | path→inode memory so a rollback resolves even if the file was deleted |
<state>/sessions/<id>.{cast,pid,done} |
Sensor | the PTY wrapper contract; a hand-rolled wrapper honouring it is a valid substitute |
<state>/collect |
Sensor (Windows) | wevtutil's own /bm//sbm bookmark — the cursor is the OS's, not a parse |
/etc/audit/rules.d/eventlake.rules (aether.rules on a host set up before 2026-10-01; still valid, rename rather than add) |
the host | the socket-leg rules (§2.1.10); an unloaded rule is a stream that never arrives |
deploy/ |
the repo | Dockerfile · docker-compose.yml · Caddyfile · eventlake-api.env.example · systemd/ · launchd/ · windows/ (+ windows/wfp/) · k8s/eventlake-api.yaml · linux/ebpf/ · nginx/ + stage-ui.sh |
Every figure below is labelled with the date and the box it was measured on.
docs/reference/build-status.md is authoritative for the performance baseline; this section states what
was proven, not a second scoreboard.
| Property | How it is proven |
|---|---|
| The real platform survives all six injected faults | assembly.TestPlatformResilienceDrill — real ed25519 seals, the real device-down cascade, live-lake isolation |
| A drill cannot pass vacuously | the resil unit suite's vacuous-pass guard + the "fixture failed to drive the cascade ⇒ fail loudly" rule |
| Torn-tail loss is bounded exactly | the real-segstore torn-tail drill asserts exactly one record lost and a clean-verifying prefix |
| Restore is honest about what it missed | the restore-rehearsal drill reports the RPO gap in events |
| Convergence is order-independent | partition-heal replays the heal under three merge orders and asserts one owner for the single-writer key |
| The drill wire surface enforces RBAC | api.TestResilienceAPI |
| The updater halts rather than guesses | digest mismatch ⇒ ok:false (halts) · missing payload ⇒ silence (ring holds) · adopted-then-dark ⇒ halt |
| A tampered air-gap bundle cannot import | POST /v1/rollouts/import refuses unsigned / unknown-key / tampered manifests |
| SNMP v3 crypto is spec-correct | RFC 3414 password-to-key + engine localization verified against the RFC's published test vectors; RFC 7860 tag truncation for the SHA-2 family |
| Flow volume is comparable across formats | sFlow byte/packet counts are scaled by the sampling rate before the canonical event |
| A v9/IPFIX exporter's templates survive datagram boundaries | the stateful Decoder caches templates per exporter + observation domain + template id |
| An unenforced edge command never reports success | fail-closed on every leg: unloaded eBPF driver, bad target, missing -session-dir ⇒ refused/NOT applied, never a fake block |
| Measurement | Value | Date / basis |
|---|---|---|
| Mixed-estate event cost | wire 1,475 B raw-ON / 903 B raw-OFF; stored 1,047 B / 538 B; gzip 4.3× | 2026-07-03, real sealer + real segstore v2, 2,000 varied copies per family |
Socket (auditd connect) event cost |
520 B stored · 841 B wire · 303 B agent→brain tail · 2.8× gzip · decode 55,018 events/sec/core | 2026-07-15, same method |
| RAM-resident per hot event | settled floor 1.26–1.8 KB; plan 3.5 KB (RSS under sustained ingest) | 2026-09-01 re-measurement, §2.3.7a — supersedes the withdrawn 0.39–0.69 KB |
| Sustained ingest (acceptance shape) | 8,809 EPS, ingest p50 37 µs | 2026-07-02, i7-10510U · 20 GB · win11 · go1.26.3, on AC, 175-pkg platform |
| Ceiling probe | 18,157 EPS sustained (@20 k × 4 workers) → the 10 k licensed target holds with 82 % headroom | same box/date |
| Durable-mode tax | −3.7 % (group-commit 256/5 ms) | D4 |
| Write-through vs group-commit fsync rate | ~1.6 k EPS/partition vs ~74–85 k EPS | D4 |
| Boot replay | bench ~58 k events/s · real boot ≈ 5.35 k events/s (planning constant 5 k/s until profiled — B4-D1, 2026-09-16tb) | D4 · deployment-e2e 09-13 |
| Bounded-horizon soak | flat oscillation, no monotonic growth | D17 |
| D5c single-tenant seal scaling | 3.37× on 4 workers (5,667 → 19,108 EPS; 4-tenant control 3.66×) — one tenant's ingest now scales like the whole node | 2026-07-15, same box on battery (absolutes not comparable to the AC rows; same-session deltas clean) |
| Reference-node storage envelope | 157 GB free → ~15 EPS sustains 90-day retention at the estate mean; ~30 EPS for the socket leg; RAM ceiling ~66 EPS at 24 h horizon | 2026-07-15, the deployed RHEL 9.8 brain — a worked example, not a constant |
transport + conflux), the
global-brain federation tier.netsh portproxy relay meanwhile.Inherited honestly from both runbooks; docs/production-readiness.md is the consolidated
remaining-work-to-market scoreboard and this section points at it rather than duplicating it.
| Piece | Status | Why deferred / what unblocks it |
|---|---|---|
| Published container registry images | Deferred | The release pipeline signs binaries today; image publishing (+ cosign image signing) is an additive release.yml step. Build from deploy/Dockerfile meanwhile — one extra step in the K8s/compose paths |
Ingest gateway / conflux Tier-2 aggregator as a shipped binary |
Package seam built, no cmd | transport (mTLS + PQC-hybrid) and conflux are tested packages; the standalone gateway binary ships with the distributed-brain productization. REST /v1/ingest is the wired path, and the SaaS edge proxy tier fronts it |
| Native mTLS on the cluster mesh | ✅ Shipped 2026-09-16xd (was Deferred — D05-04 superseded) |
Cluster-CA mTLS on the listener and the gossip client, https-only peer addresses, answers bound to member certificates, and a non-loopback mesh without credentials refuses to boot (§2.1.2). ⚠ Still open: node certificates are operator-issued files, not Certificates-plane issued; and membership records are unsigned, so a node holding a VALID cluster certificate can still gossip a forged record about another member. |
| Brain active-active / raft-replicated lake | Deferred by doctrine | Single-writer lake + warm standby (§2.1.9). go.etcd.io/raft is a named D5 perf-tier candidate — adopted only with a measured justification. Consequence: no zero-RTO node loss; RTO = replay, so size the horizon for it |
| Global-brain federation tier (site→global content loop) | Design complete (01-substrate-plane.md#distributed-detection) |
Site brains + island mode work today; the cross-site correlation / Compiler push tier is the F-roadmap. A hybrid sale scopes the global pane as mesh scatter reads + per-site consoles |
| Real actuation backends beyond the Sensor (Firewall fabric, identity actions) | Blocked on external targets | The Actuator seam + conductor gate are ready; the Sensor (firewall + DNS sinkhole + inline IPS/virtual patching + inline WAF) is the proven real actuator (N4 · W32) |
| Dynamic multi-OS sandbox detonation backend | Blocked on external VM targets | POST /v1/detonate + the static engine ship; the Detonator seam takes a VM/emulator backend when targets exist |
| Kafka-out egress sink | Deferred behind the forward.Sink seam |
Continuous egress ships (W40: webhook NDJSON + syslog RFC 5424/CEF via POST /v1/forwarders — the coexistence/MSSP-uplink path); a clean-room Kafka wire protocol is its own project, a named perf-tier candidate |
| OIDC/JWT on the management plane | Closed for the human login (W41, 2026-07-06) | Enterprise SSO-inbound is SHIPPED: the OIDC relying party (/v1/sso/* → the D2 session) signs the SOC in with the customer's IdP, SCIM (/scim/v2/*) drives the operator lifecycle, enforce_sso + audited break-glass govern the local door. Machine/API automation stays on bearer keys by design; SAML deferred |
| Sensor agent self-update executor | CLOSED — SHIPPED 2026-07-10 (opt-in; watchdog tier the same day) | Orchestrator (§2.1.11), agent-side executor (-self-update) and the can't-exec tier (-watchdog) are all built. Default remains stage-only. Remaining sliver: a broken payload plus a host reboot inside the seconds-wide revert window needs a manual restore |
| Windows WFP callout driver signing | Authored, runbook exists; unsigned/unbuilt is the gap | WDK-built, signed kernel artifact (deploy/windows/wfp/); the userland netsh portproxy relay is the shipped fallback. WinDivert stays rejected (LGPL). Acquisition runbook: §2.6.3 |
| Every OS signing credential and entitlement — Apple Developer Program, Developer ID, the ES entitlement, the EV certificate, the Partner Center hardware account | DOCUMENTED, none held (2026-08-02) | §2.6 is the acquisition runbook. Nothing here blocks development or testing — all three planes have a free, no-approval test path (D05-58). It blocks deployment onto any machine that is not ours. The Apple ES grant is the only multi-week external dependency (D05-57) |
| Both Windows drivers under HVCI / Memory Integrity | UNVERIFIED | Never load-tested under HVCI (the drivers are unbuilt — see the WFP row). Must be verified before shipping, since HVCI is on by default on much modern hardware. §2.6.3.5 |
| The exec-block plane under Secure Boot + kernel lockdown, per distro | UNVERIFIED, per-distro empirical | Lockdown integrity mode generally permits BPF; confidentiality mode disables the kernel-memory helpers, and some distro kernels have historically restricted bpf() far more aggressively. Verify on each target distro with Secure Boot ON rather than reasoning from doctrine. §2.6.4 |
| Signed BPF programs (kernel 6.18, keyring-based) | Recorded, not required | Merged for Linux 6.18; not a gate we must pass today. Logged in §2.6.4 so it is not re-discovered as a surprise later |
| Partner Center hardware-account standalone fee | UNVERIFIED | Registration consumes the EV certificate; no current standalone fee was confirmed at the 2026-08-02 research pass. Verify before budgeting (§2.6.5) |
| Cell autoscaling / automated tenant rebalancing | Procedural (operator action on the control plane) | SaaS fleet ops is runbook-driven today; onboarding automation depth, not a capability gap |
| SaaS control-plane productization (self-serve provision → cell assignment → billing loop) | Program surface (tracker §11/§12) | The rate-card + metering + entitlement substrate is built |
| Server-class capacity target | OPEN ratification (build-status D4 DECISION) | §2.2.6 tiers deliberately derate laptop-measured numbers; L/XL deploys want the pilot. When it lands, the §2.3.12 constants re-issue as v2 |
| Block compression in the store | SHIPPED and GATED (owner §6dw (c), retired from the deferrals 2026-10-11gc) | Built in the core as segstore v3 (stdlib flate, no perf-tier dependency); off by default, on for a release-stamped hosted cell (§6du), EVENTLAKE_SEGMENT_COMPRESS either way. zstd remains a perf-tier candidate only if flate's measured ratio disappoints |
SNMP: .mib file compiler · AgentX · SNMP-over-(D)TLS (RFC 6353) · AES-192/256 · authenticated v3 inform ack |
Honest deferrals | The MIB/OID set is a curated catalog, not a compiler (Register(name, oid) adds vendor objects). AES-192/256 are rejected on substance: their key-extension is non-standardized |
| Flow/pcap: live AF_PACKET/AF_XDP capture · TCP stream reassembly · more L7 dissectors · the never-drop raw-pcap store | Deferred behind the Capture seam |
Per-packet dissection already covers DNS, small HTTP requests and the TLS ClientHello — the common cases. eBPF/AF_XDP line-rate capture plugs in behind the same seam that the userspace IPS proxy uses today |
| Flow analytics: streaming EWMA volume baseline · east-west lateral-movement graphing | Queued augmentation | Windowed thresholds ship; the detector semantics belong to Volume 02 |
| The device layer below fsync | Explicitly out of scope | Power-loss testing needs hardware rigs; the §2.5.4 power-cycle rehearsal is the honest substitute |
| Live-node residuals | Operational, tracked in PLAN.md |
The tenant-provisioning redeploy + the auditd socket rule awaiting the same redeploy; the §2.1.8d four-name cutover (DNS + per-host ACME certs, and the ops. allowlist CIDRs the required include now forces); a real-browser click-through of every designed empty state |
Five retired documents, 2,308 source lines, re-authored into the spine above. Each former filename
resolves to the anchor named in docs/README.md's redirect table.
§2.6 is newly authored (2026-08-02), not absorbed from a retired doc — so it has no row in the table
below and no entry in docs/README.md's redirect table (which maps former filenames only). Its sources
are the four in-repo enforcement-plane READMEs (deploy/linux/ebpf/, deploy/windows/execblock/,
deploy/windows/wfp/, deploy/macos/execblock/) plus a dated external research pass: Apple's
Endpoint Security entitlement documentation and request form, Microsoft's Driver Code Signing
Requirements and Attestation Sign Windows Drivers, Microsoft's April 2026 cross-signed-driver trust
announcement, and the kernel lockdown / signed-BPF references. External facts are dated in place;
prices and policies are volatile and must be re-verified before any spend.
| Former doc | Lines | Absorbed into |
|---|---|---|
deployment.md |
902 | §2.1 (whole — what ships, the plane/TLS topology, both quick-start paths, the durable-mode runbook, health/readiness, D21 hardening, the autonomy table, backup/restore, HA + island mode, the complete Sensor section with the auditd rule block and the uplink-failure drill reproduced verbatim, fleet-update safety, the BYOK/HYOK contract, the Kubernetes reference); the TLS-at-the-proxy, durable-default, key-custody, backup-set, shutdown-ordering, per-leg-opt-in, firewall-backend, WinDivert, WAF-contract, spool and rollout decisions → §3 D05-02…D05-11, D05-14…D05-32; §4.1–§4.5 the events, routes, env vars, Sensor flags and on-disk artifacts; §5.2 the durable-tax and fsync-rate numbers; §6 the whole §13 deferral table, including the SHIPPED self-update row kept in place |
sizing-guide.md |
658 | §2.3 (whole — the six-step method, the measured event-cost tables with the deliberately-unfolded socket row, the five-part device EPS catalog, the auditd socket-leg pricing with its kernel-filter guidance and the auid tradeoff, peak/growth/pilot, the volume ready-reckoner, the three storage temperatures with the compliance presets, the compute/RAM/disk math, the worksheet, the SaaS-vs-self-managed split, all three worked examples, the calculator coupling and the normative constants table with its honesty box); the versioned-constants, derated-vCPU, unfolded-socket, bind-vs-connect, auid-tradeoff, 3×-archive, segment-granularity and calculator-lockstep decisions → §3 D05-39…D05-46; §5.2 every measured figure |
deployment-models.md |
379 | §2.2 (whole — the single-writer architectural fact and its four consequences, cells scale out / nodes scale up / edges buffer / the mesh reads across, the no-platform-ceiling corollary, the decision matrix and its four drivers, vertical/horizontal scaling with the sharding preference order and the symptom→lever table, the AWS cell reference with pooled-vs-dedicated and fleet ops, all four self-managed shapes, the quick recommendations, the reference tiers, the HA/DR and security-posture-per-model tables, and what SaaS buys you out of); the replicas-1, warm-standby, K8s-only-if-you-run-K8s, one-artifact-set, small-pooled-cell and procedural-autoscaling decisions → §3 D05-33…D05-38; §6 its §12 honest-limits rows, merged with deployment.md's §13 |
itops-nms-snmp.md |
244 | §2.4 (whole — the complete SNMP capability set with its v3 crypto and honest deferrals, the Set-safety rule, both entry directions with their code examples, interface auto-discovery and counter→rate, the trap decoder, the events/severity table, the reuse of detect.DeviceDownRule, the NOC dashboards and uptime SLA, all four clean-room flow decoders with the canonical it/network/flow event, the flow arm's four detectors and dashboards, decode/pcap file formats/frame decode/DPI dissectors/events, the edge-DPI doctrine, and the operator integration checklist); the clean-room, gated-Set, no-new-detector-engine and edge-DPI decisions → §3 D05-47…D05-50; §4.1 the device/flow/L7 streams; §6 the SNMP and capture deferrals. Detector semantics are deliberately deferred to 02-analytics-plane.md |
resilience-validation.md |
125 | §2.5 (whole — the thesis and its Jepsen benchmark, the four honesty rules, the six-drill table with every asserted invariant, suite mechanics and the D5 seam discipline, where it runs across CI/on-demand/scheduled, the five-rehearsal ops-calendar table with the single-writer restatement, the report anatomy and the four deliberate non-goals); the drills-are-tests, visible-skip, seam-discipline, measured-split-brain and named-rehearsal decisions → §3 D05-51…D05-55; §4.1 the it/resilience/report stream; §5.1 the drill proofs |
Cross-volume pointers used above: 00-charter-and-strategy.md#aether-blueprint (the scale doctrine
behind no platform ceiling) · 01-substrate-plane.md#lake-storage-engine (the D5 substrate whose disk,
horizon and backup this volume operates), #distributed-detection (the tiered site/global model behind island
mode and the carrier example), #aether-os (D27 — whether EventLake ever owns the node itself) ·
02-analytics-plane.md (the detector semantics over the SNMP/flow/pcap streams configured here) ·
03-execution-plane.md (the conductor gate every actuation and ring promotion passes, and the Sensor's enforcement
semantics) · 04-identity-and-policy-plane.md#verichain-platform (the sealed enrolments the root KEK
protects), #crypto-agility (the PQC-hybrid KEM on the edge transport) ·
06-governance-and-trust.md#supply-chain (verify before you run), #compliance-as-code (the retention
regimes and NIST CSF PR.IR evidence) · 07-commercial-and-gtm.md#licensing-model (ingested TB/day — the
commercial dimension of §2.3.6), #hosting-cost-plan (the same GB/day as per-cloud dollars) ·
08-interfaces-and-extensibility.md#api-management-plane (full REST semantics for every route in §4.2),
#grpc-surface, #genai-agentic-integration (the model seams in §4.3).