Record the distributed poller as a deferred v2 item
The owner asked whether remote pollers could work per site over an outbound-only MQTT link, keep polling while the master is down, and receive device credentials sealed to their own key. The assessment found it feasible but deferred it to v2, so the requirements, the gaps in the current code and the main risks are written down for later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -50,6 +50,8 @@ Departures from the spec are recorded in [`docs/DEVIATIONS.md`](../DEVIATIONS.md
|
||||
|
||||
- **Web push notifications (v2)**: a `webpush` receiver kind in `internal/notify` (RFC 8291 payload encryption, RFC 8292 VAPID, standard-library crypto), per-user subscriptions (needs M6 users), a service worker in the UI, and push endpoints in the outbound allow-list. Out of v1 per the spec's non-goals ("alerts go to webhooks and email only"). Owner deferred it to v2 on 2026-10-22. Until then, a webhook receiver pointed at ntfy or Gotify gives phone push.
|
||||
|
||||
- **Distributed poller (v2)**: remote pollers per site or rule that spool to disk while the master is down, send signed chunks over an outbound-only link (MQTT through a broker), and receive device credentials sealed to their own key. Assessed feasible on 2026-10-25 and deferred to v2 by the owner. Requirements, gaps and risks: [v2-distributed-poller.md](v2-distributed-poller.md).
|
||||
|
||||
- **BMC support**: SNMP profiles for Dell iDRAC, HPE iLO and Supermicro (health, temperatures, fans, PSUs, power, RAID) feeding the Health/Power/Environment tabs; a native Redfish collector later. Today, Redfish works only via an exporter pushing with a push token. Owner: "good enough for now" (2026-10-23).
|
||||
|
||||
## Post-v1 additions
|
||||
|
||||
72
docs/milestones/v2-distributed-poller.md
Normal file
72
docs/milestones/v2-distributed-poller.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# v2: distributed poller (feasibility, deferred)
|
||||
|
||||
Status: deferred to v2 by the owner on 2026-10-25. This is a feasibility assessment, not a plan; nothing is built.
|
||||
|
||||
## Requirements (owner)
|
||||
|
||||
1. A remote poller polls the devices of a site, or of a predefined rule.
|
||||
2. It sends its data to the master only as chunked payloads.
|
||||
3. While the master is down it keeps polling, and delivers when the master is back.
|
||||
4. Each poller has its own private key; the master sends the target device credentials sealed to it.
|
||||
5. It works over an outbound-only link (MQTT pub/sub through a broker). Neither side accepts a connection from
|
||||
the other. Not a data diode yet, but the design should not rule one out.
|
||||
|
||||
## Verdict
|
||||
|
||||
Feasible, about one milestone. The poller is already injectable: `poller.Options` (`internal/poller/poller.go`)
|
||||
takes the store, the metric writer, the reachability observer and the SNMP dialer, so the polling code can be
|
||||
reused unchanged. The work is in what sits around it: where targets come from, where samples go, and how
|
||||
alerting treats data that arrives late.
|
||||
|
||||
## What exists and what is missing
|
||||
|
||||
| # | Exists | Missing |
|
||||
|---|---|---|
|
||||
| 1 | `devices.site`, `store.DeviceFilter.Site`; the alert-policy matcher (`alertpolicy.Device{Slug,Profile,Role,Site,Kind}`) already selects devices by rule | a `pollers` table and assignment rules; reuse the policy matcher rather than a second rule engine |
|
||||
| 2 | `metrics.Sample`; the vmsup queue batches 50k samples per POST | a chunk format (poller id, sequence, signature). Push ingest is **not** reusable: it strips `device`/`iface`, refuses `if*`/`nmon_` names and rejects timestamps more than 1 h from now. Poller data needs its own trusted path into the hot queue |
|
||||
| 3 | the hot queue buffers in memory only (60 s / 2M samples, then drops) | a disk spool on the poller (append-only segments, bounded by size), and a local SQLite snapshot of its targets so it restarts without the master |
|
||||
| 4 | symmetric AES-256-GCM keyring with versions (`internal/secrets`, `store/cred.go` `seal`); rotation in `internal/keyrotate` | a key pair per poller (X25519 + Ed25519): the master seals the credential bundle to the poller's public key (HPKE) and signs it; the poller signs its chunks; rotation follows the keyrotate pattern |
|
||||
| 5 | none | an MQTT transport behind an interface (see below), with outbound HTTPS chunk upload as a second, brokerless option |
|
||||
|
||||
## Transport (requirement 5)
|
||||
|
||||
- Data up: `nmon/poller/<id>/data`, QoS 1. The master holds a persistent session, so the broker keeps chunks
|
||||
while the master is down.
|
||||
- Config and credentials down: a retained, sealed message on `nmon/poller/<id>/config`, read on every connect.
|
||||
- An MQTT client library (e.g. paho.golang) departs from the spec's "standard library elsewhere" and needs a
|
||||
DEVIATIONS row.
|
||||
|
||||
Rules that keep a later data diode possible:
|
||||
|
||||
1. The data path is already one-way. Chunks are self-describing, signed and sequence-numbered, and the master
|
||||
drops duplicates by (poller, seq), so a resend is harmless. The only signal the poller waits for is the
|
||||
broker's PUBACK, which frees spool space. Under a diode, spool release becomes age/size bound plus a periodic
|
||||
resend window; the format does not change.
|
||||
2. One downlink only: the sealed, signed config bundle. Make it a file format that arrives either as the
|
||||
retained message or by hand (`nmon poller import bundle.sealed`). A diode drops the subscription and keeps
|
||||
the import.
|
||||
3. Interactive tools (poll-now, test poll, SNMP debug recording: `PollTools`) need a request and a reply. Over
|
||||
the broker they work through a request topic with seconds of latency. Under a diode they are unavailable,
|
||||
and the UI must say so for remote devices.
|
||||
|
||||
## Risks
|
||||
|
||||
1. **Late data looks like an outage.** `reach.Evaluator` runs on the master and expects live
|
||||
`ObserveICMP`/`ObserveSNMP` (`internal/reach/types.go`); the API marks a device down after three silent
|
||||
intervals. A site link drop would page every device behind it. Fix: model the poller as a dependency parent
|
||||
of its devices (dependency suppression already exists) and replay spooled probes as history, never as live
|
||||
state transitions. Needs a DEVIATIONS row.
|
||||
2. **Inventory writes from the poller.** The poller writes poll state (`devloop.persistState` →
|
||||
`SetPollState`), component rows and discontinuity bases. Remotely these go to the local snapshot and are
|
||||
sent up as events.
|
||||
3. **Discovery must run remotely.** The master usually cannot reach a site's devices. Interface ULIDs are the
|
||||
frozen `iface` label; the poller can mint them (ULIDs are globally unique) and the master accepts them.
|
||||
Traps, L2 tables and checks need the same scope decision up front.
|
||||
4. **Late writes.** VictoriaMetrics accepts old timestamps within retention; dedup by (poller, seq) keeps a
|
||||
resent chunk from being written twice.
|
||||
|
||||
## Codebase constraints
|
||||
|
||||
- A new package (e.g. `internal/remotepoll`) and an `nmon poller` subcommand wired in `internal/app`.
|
||||
- Collection paths still do not import each other; time comes from `internal/clock`.
|
||||
- API routes additive only (`scripts/openapi-check.sh`); new tables through a new migration.
|
||||
Reference in New Issue
Block a user