Files
nmon/docs/spec/spec.md

36 KiB
Raw Permalink Blame History

Lightweight ISP Network Monitor — Build Spec

2026-10-21 ·

In one page

nmon is a network monitor you install as a single program on one server. You point it at your switches, routers, firewalls, access points, servers, printers and UPSs, and it gives you traffic graphs for every port, CPU, memory and disk for every device, and an alert when something goes down. It replaces LibreNMS for a small ISP or an internal IT team, and it is deliberately small:

  • One install, one folder on disk. nmon backup writes one file you can restore from. Nothing else to install.
  • It asks each device only for the numbers it needs, once a minute, instead of walking the whole device every cycle. Two thousand devices and two hundred thousand ports fit on one modest box.
  • A month of full-resolution history and two years of hourly history fits in about 60 GB. A knob extends the hourly history to ten years for roughly 40 GB more.
  • Alerts understand the network's shape: a device is only suppressed when every path to it is gone, so a dead core gives one alert, not three hundred, and a redundant core gives none.
  • Login is through your existing single sign-on. Two roles, viewer and admin.
  • Anything that can already push metrics (Proxmox, Telegraf, Prometheus exporters) can send them straight in, each sender bound to the device it is allowed to speak for. Under the hood it is written in Go, keeps its inventory in SQLite (or Postgres if you outgrow it), stores metrics in two VictoriaMetrics instances it carries inside itself and runs locally, and serves a React web UI from the same program. The rest of this tab is the design a person should be able to read in one sitting. The exact tables, labels, flags and test fixtures a build agent needs are in the second tab. Tables, labels, flags, API routes and fixtures:

Problem statement

Build a LibreNMS replacement for a small ISP that monitors thousands of devices from one box with a few tens of GB of storage. The job is traffic graphs in and out per port, CPU, memory and storage per device, and an alert when something goes down. It is designed for an ISP and sold as such, but nothing in it is ISP-specific: the same box monitors an internal IT estate of switches, servers, printers and UPSs. LibreNMS is heavy for structural reasons, not tuning ones: MySQL plus one RRD file per metric, a PHP poller that walks whole SNMP tables every cycle, and a schema of ~200 tables. At ISP scale that is hundreds of GB of RRDs and a database that needs care. "Lighter" here means three things:

  • One install, one folder on disk. Backup is one command that makes one file.
  • Opt-in metrics. Poll only the numbers a profile names. Every port is polled, up or down, because a cold counter costs almost nothing to store.
  • No table walks on the hot path. Discovery walks twice a day; polling asks for known values by index. This spec is written to be executed by an AI coding agent. It is a build order with checkable gates, not a vision document. The agent builds exactly what is written; anything it should not build is listed under Non-goals.

Decisions that don't move

These hold for every milestone. A change that breaks one is wrong even if it works.

  • Written in Go, shipped as one static binary. The two VictoriaMetrics executables it needs are embedded in that binary and unpacked into the data folder on first start, so the user installs one thing.
  • One data folder holds everything: inventory database, both metric stores, profiles, config. Backup and restore are commands, not file copies.
  • Inventory in SQLite by default, Postgres as a supported alternative tested in CI, MySQL best-effort. All database access goes through one interface and one ORM so the swap is a flag.
  • Metrics never leave the box except through nmon. The metric stores listen on localhost only.
  • The once-a-minute poll path talks to devices and to the local metric store and nothing else. No web calls, no AI calls.
  • Every port is polled, up or down. Profiles can exclude ports by name pattern; nothing else drops a port.
  • Rollups store rates, never raw counters. Long-term graphs and 95th-percentile figures come from rates.
  • Identity is stable: a device has an immutable slug and an interface has an immutable ID. Names, aliases and ifIndex can all change without splitting history.
  • Everything runs against recorded or synthetic device fixtures in CI. Real hardware is never a build dependency.
  • IPv6 is first-class, not a port. Every device has a list of management addresses in either family and nmon prefers v6 when both exist. Every path that touches a device or the network (ICMP, SNMP, traps, gNMI, push listeners, checks, Proxmox API, the web UI, the outbound allow-list) works identically on v6 and v4, and CI runs a job with v4 turned off.

Non-goals for v1

The agent does not build these, even partially, even if convenient. Each is a later plugin or a separate tool.

  • Syslog collection
  • NetFlow, sFlow, IPFIX
  • Device configuration backup or diffing (Oxidized does this)
  • Billing, invoicing, customer portals
  • Multi-tenancy or per-customer views
  • Services running on devices (process lists, package versions)
  • Neighbour auto-discovery into the inventory (LLDP is read for dependency hints only)
  • A plugin system. Profiles are YAML; code is code.
  • Mobile apps, Slack bots, chat integrations. Alerts go to webhooks and email only.
  • An AI assistant inside the product. That is a later layer over the API.

Target scale and storage budget

Design point: 2,000 devices and 200,000 interfaces (about 80,000 carrying traffic) on one box with 8 cores, 32 GB RAM and 200 GB NVMe. Must still work at 5,000 devices on the same box with a 2-minute interface poll. Poll load. 200,000 interfaces × 7 counters at 60 s is 1.4 million values a minute. At 40 values per request that is 35,000 SNMP requests a minute, about 600 a second, plus roughly 2,000 pings every 15 s and a trickle of 5-minute metrics. Budget ceiling: 50,000 SNMP requests a minute. A device that needs more than 25 requests per cycle is either huge or misconfigured, and the poller health page shows it. Storage.

Store Holds Resolution Kept for Approx. size
Hot every raw counter and gauge 60 s interfaces, 5 min the rest 30 days ~45 GB (80k busy ports ~36 GB, 120k idle ports ~4 GB, other metrics ~2 GB, index ~20%)
Long hourly avg and max rate per interface and direction, monthly p95 1 h 2 y default, 10 y max ~5 GB per year

Assumes ~1.5 bytes per sample for changing counters and ~0.1 for idle ones, which matches VictoriaMetrics' published figures for this kind of data. Total at the defaults is about 55 GB; at 10 years about 95 GB. The box is sized with 2× headroom for compaction and growth. A disk-usage alert fires at 80%, and the hot store refuses new series, not new samples, at 90%.

Architecture

Collectors write down into two stores; consumers read them. The alerter also reads the inventory for the dependency graph, and the API joins names, sites and roles from it at query time. nmon serve unpacks the embedded VictoriaMetrics binaries, starts the hot and long stores as child processes, health-checks them every 5 s, and restarts them with backoff. Everything else runs as goroutines in the nmon process. Device-down detection does not depend on the metric store (see Alerting), so losing a store degrades graphs, not outage detection.

Architecture diagram (text form). One binary collects, stores, alerts and serves.

  • Sources: network devices (SNMP v3/v2c/v1, ICMP, traps); pushers (Proxmox, Telegraf, exporters).
  • Collectors inside nmon: Discovery (GETBULK walk, 12 h or on trap); Poller (60 s GET by ifindex, 15 s ICMP); Push ingest (Influx, remote write, Graphite).
  • Stores: Inventory DB (SQLite; Postgres/MySQL by flag); VictoriaMetrics, two local processes: hot (30 days) and long (up to 10 years), never exposed.
  • Consumers: API + UI (JSON API, static SPA, /metrics); Alerter (MetricsQL rules + dependencies, reads inventory too); Rollup (5 min and 1 h, written back to long).
  • Outputs: NOC browser (OIDC, viewer or admin); Notifications (webhook JSON, SMTP).

Data model

Inventory lives in a relational database and is what the UI navigates; metrics live in VictoriaMetrics as labelled series. The join between them is by stable IDs, never by names. Devices have an immutable slug (generated on creation, never edited) and a display name you can change freely. A device has one or more management addresses, IPv6 or IPv4, with v6 preferred when both answer, and a k``ind: network (SNMP-managed), vm or container (from Proxmox, no SNMP fields), or check-only (a URL or host you ping). Deleting a device is a soft delete: the device, its interfaces and their IDs stay, marked retired, and its history stays on disk. Adding a device whose management address (either family) or sysName matches a retired one prompts to reattach (restores the slug and interface IDs, so the graphs come back) or create new. A retired device is purged, series and all, only by an explicit admin action after a configurable period, default one year. An RMA'd box that comes back with the same config and the same port names simply continues its history; the serial-number change from the entity MIB is logged as a hardware-replacement event, not treated as a new device. Interfaces have an immutable ID. On each discovery nmon matches what the device reports against what it knows, by ifName first, then by ifDescr plus MAC address, and the match must be one-to-one. A clean match updates the row, including a changed ifIndex. No match means a new interface. Two candidates for one port, or one candidate for two ports, is not guessed: both are kept, a review item is opened, and an admin merges them with one click. A split you can merge later is better than a merge you can't undo. An interface that stops appearing is marked retired with a timestamp and keeps its history. Every identity decision (matched, split, merged, retired) is in the audit log with the evidence used. Metric labels carry only the two IDs: device=<slug> and iface=<id>, plus the metric name. ifName, ifAlias, site, role and customer are joined from the database when the API answers, so renaming anything creates no new series and the series count is fixed by the number of ports, not by how often you edit descriptions. Dependencies mean one thing in v1: alternative paths to reach this device. A device can list several parents (both cores, both uplinks) and is considered cut off only when all of them are down. "Required components" (a service that needs both its firewall and its switch) is a different relationship and is deliberately not modelled yet; use a check for the service instead. The graph must be acyclic and a write that would create a cycle is rejected with the path named. Parents are set by hand, by CSV import, or accepted from LLDP suggestions in a review list. Proxmox VMs are linked to their node with a separate hosted-on relationship the alerter ignores for network suppression. Credentials are reusable objects encrypted at rest with an authenticated cipher. The key lives in a file outside the data folder, supports a current and a previous version for rotation, and is deliberately excluded from backups unless asked for. The API never returns a secret. The full table list and column names are in the Build detail tab.

Collection

Three paths on three schedules: ping for reachability, SNMP for numbers, traps and gNMI for immediacy. Push ingest for things that already export metrics. Each path has its own worker pool and its own deadline, so a slow path cannot starve another. Reachability (ICMP, every 15 s). Three consecutive misses mark ICMP down, two hits mark it up. ICMP is one signal, not the verdict; see Alerting for how "down" is decided. Discovery (every 12 h, on cold-start trap, or by hand). A GETBULK walk of the system group and the interface tables, plus whatever tables the profile lists for discovery (entity, LLDP, printer supplies). Each table has its own row limit and timeout; a table that overruns is marked partial and the previous result is kept. Every interface is monitored, up or down, including tunnels, SVIs and subinterfaces: a GRE tunnel to a scrubbing provider is the port you most want to see during an attack. Only the profile's name-pattern exclude list takes a port out, and the shipped defaults exclude loopbacks and vendor-internal interfaces only. Polling (every 60 s, 5 min, 15 min). Multi-varbind GET requests for exactly the OIDs the profile names on the interfaces discovery found. Varbinds per request start at 40 and shrink per device when a response fragments or the agent returns tooBig, then creep back up. Each device has a cycle deadline of half the interval; work still outstanding at the deadline is dropped and counted, and the next cycle starts on time. A device that keeps missing its deadline backs off to a longer interval and shows as degraded. SNMP versions. v3 authPriv (SHA-256, AES-128) is the default; SHA-1 and DES are accepted but flagged. nmon caches engine ID and time per device and re-syncs on notInTimeWindow. v2c is fine on a management VLAN. v1 is for UPSs and PDUs, and discovery on v1 uses GETNEXT. Any interface without 64-bit counters on a link of 100 Mbit/s or faster is shown as unreliable and excluded from p95 and group sums, because a 32-bit counter wraps in under six minutes at 100 Mbit/s and in seconds at 10 Gbit/s; faster polling does not fix that, so nmon does not pretend it does. The UI says why and what to enable on the device. Counters. Raw counters are stored as-is; rates are computed with rate(), which handles wraps and resets. A reboot is recorded when sysUpTime drops and either the new value is under ten minutes or hrSystemUptime also dropped; a sysUpTime wrap at 497 days is not a reboot. Interfaces that report ifCounterDiscontinuityTime have it checked each cycle. Traps (UDP 162). linkDown, linkUp, coldStart, warmStart, and authenticationFailure from devices in the inventory only; unknown senders are counted and dropped. A trap schedules one immediate poll of that device, rate-limited to one per device per 30 s, and a coldStart schedules discovery. Traps are hints; state always comes from polling. gNMI (after v1). For devices whose profile enables it, subscribe to interface counters at 10 s and write them with the same labels. SNMP then does discovery only for those devices. Push ingest. Influx line protocol, Prometheus remote write and Graphite listeners, each behind a per-source token minted by an admin and bound to the device or devices it may write as. Labels in the payload are kept except device, which nmon sets from the token binding; a host label is never trusted for identity. Each source has a series cap (default 50,000), a body-size cap and a rate limit, and exceeding any of them drops the write with a counter and an alert. Proxmox inventory (nodes, VMs, containers, with the Proxmox VMID as the stable key) comes from its API every 5 min. Checks. Ping, HTTP, DNS and TLS-expiry checks every 60 s against things that are not devices. Per-family traffic. IF-MIB counters don't distinguish IPv4 from IPv6. On devices that implement IP-MIB's ipIfStatsTable (Juniper, most Cisco, Nokia), a profile can enable it to collect per-interface octets split by address family, which gives a v4/v6 share graph per port and per group.

Device profiles

A profile is one YAML file that says how to talk to a class of device and what to collect. Profiles match on sysObjectID prefix or a sysDescr pattern; the highest-priority match wins and generic matches everything. A profile can extend another and override only some keys. Each metric in a profile declares its type (counter, gauge, status enumeration, or text), its unit and any scaling, which table column becomes a label, and which values mean "not available" so a sensor reading of −1 or 65535 is dropped rather than graphed. The loader rejects a profile that leaves any of these out, because a number stored without its meaning is the kind of bug that looks fine for a year. The generic profile collects the interface counters, uptime, and HOST-RESOURCES CPU, memory and storage where the device answers for them. Shipped alongside it: Juniper, Cisco IOS and NX-OS, Arista, MikroTik, Ubiquiti EdgeOS, VyOS, OPNsense, pfSense, FortiGate, Nokia SR OS, Huawei VRP, Aruba CX, Extreme EXOS, Linux net-snmp, Windows, APC UPS and a vendor-neutral printer profile. Devices that don't implement those MIBs simply have no CPU or disk graphs; the UI says "not reported" rather than showing zero. nmon profile test <ip> runs discovery and one poll with the matching profile and prints what it would collect, with units. That is the loop both a person and the build agent use to add gear. The YAML schema and a full example are in the Build detail tab. MIBs are for writing profiles, not for running them. A profile stores numeric OIDs with their name, type and unit, so the poller never parses a MIB. nmon ships the standard MIBs (IF, HOST-RESOURCES, ENTITY, ENTITY-SENSOR, UPS, Printer, LLDP, BGP4) and nmon profile new --mib <name> compiles vendor MIBs you drop into mibs/ into draft metric entries with the OID, type and textual-convention filled in, leaving unit and interval for you. Vendor MIBs are the operator's to fetch: Juniper, Cisco, Arista, MikroTik and Fortinet publish theirs, some behind a support login, which is why they are not bundled. From OID to gauge. For every value a poll returns: the SNMP type must match what the profile declared (Counter64 for a counter, Gauge32 or Integer for a gauge, Integer with an enum map for a status); values in the profile's not-available list are dropped; scale is applied; the label comes from the index column resolved at discovery; the result is written as name{device, iface or component}. TimeTicks become seconds. Text goes to inventory, never to metrics. A value whose type doesn't match is counted and dropped, not stored wrong. Devices SNMP can't fully cover. VyOS is net-snmp underneath, so interfaces, including WireGuard tunnels, come through IF-MIB; per-peer WireGuard state (handshake age, per-peer transfer) has no MIB anywhere and arrives by push from a small exporter on the box. The same split applies to OPNsense and pfSense (interfaces by SNMP, firewall states by push), UniFi and Meraki (controller API only), and server BMCs (Redfish). The rule: SNMP for what the device reports natively, push for the rest, and the UI shows both under one device. Shipped in v1: generic, juniper-junos, cisco-ios, cisco-nxos, arista-eos, mikrotik-routeros, ubiquiti-edgeos, fortigate, pfsense, linux-netsnmp, windows-snmp, apc-ups, printer (Printer-MIB, any vendor). Each ships with a matching snmpsim fixture.

What to collect, by device class

Everything LibreNMS collects for CPU, memory and storage comes from HOST-RESOURCES-MIB and UCD-SNMP-MIB, which almost every agent speaks. Those go in generic so every device gets them for free; vendor MIBs fill the gaps. All of it is opt-in per profile and polled at 5 min unless noted.

Class Standard MIB metrics (in generic) Vendor or extra Interval
Every device sysUpTime; hrProcessorLoad per core; hrStorageTable size/used per filesystem and RAM (type = ram, fixed disk); memTotalReal/memAvailReal (UCD) 5 min
Routers, switches IF-MIB counters, ifOperStatus; ENTITY-SENSOR-MIB temp/fan/PSU; POWER-ETHERNET-MIB PoE per port; LLDP-MIB neighbours (discovery only) BGP-4-MIB peer state and prefixes; optics DOM (jnxDom, cisco entSensor); STP topology changes; MAC/ARP table size; cpmCPUTotal, jnxOperatingCPU 60 s ifaces, 5 min rest, 15 min optics
Firewalls IF-MIB, HOST-RESOURCES session count vs max; VPN tunnel state; HA state; policy throughput (fgSys, panSession, pfSense) 60 s
APs, controllers IF-MIB client count per AP and SSID; channel utilization; noise floor; tx power; AP up/down from controller 5 min
WISP radios, CPE IF-MIB signal, SNR, modulation, link capacity (Ubiquiti, Cambium, Mimosa) 5 min
Servers, hypervisors HOST-RESOURCES, UCD load average, swap Proxmox push (per-VM CPU, mem, disk I/O); node_exporter; Ceph health and capacity; SMART via exporter 60 s via push
UPS, PDU UPS-MIB: battery charge, runtime remaining, input/output voltage, load %; output status apcUps; per-outlet current (PDU MIBs); generator status 5 min
Printers Printer-MIB: toner and drum levels per supply (prtMarkerSupplies), page count (prtMarkerLifeCount), status and alerts (prtAlertTable), paper tray levels hp, Brother, Xerox serial and model 15 min
Environment ENTITY-SENSOR-MIB where present temperature, humidity, door and water sensors (APC NetBotz, Raritan) 5 min
Services ping, HTTP, DNS, TLS expiry checks 60 s

Storage metric naming: hrStorageUsed{device, hrStorageDescr} and hrStorageSize stored raw, percent at query time, so a full-disk alert is one MetricsQL rule for every device class. Memory is hrStorageType=hrStorageRam on the same table rather than a second code path. Printers alert on supply under 10% and on any prtAlert of severity critical. Alerts added for these: cpu_high (>90% for 10 min), memory_high (>90% for 10 min), disk_full (>90%, or predicted full in 24 h via predict_linear), ups_on_battery, ups_runtime_low (<10 min), printer_supply_low, temp_high (per-profile threshold). Groups and aggregate traffic. A group holds interfaces (a customer's ports, a POP's uplinks). Each member carries a direction, whether "in" on that port means traffic toward the customer or away, and an effective date range. A historical graph or p95 uses the membership and directions as they were at that time, so adding a third port in February does not rewrite January. The group's traffic is the sum of the members' 5-minute rates after orienting them; LAG members are dropped when their LAG is also in the group; the group p95 is taken on that summed series, never by adding per-port percentiles. A group that mixes both ends of the same link is flagged, since it would double-count. A materialised group p95 records the membership it was computed from.

Retention and downsampling

Two stores, two jobs. Open-source VictoriaMetrics has one retention period per process, so nmon runs two: hot keeps every raw sample for 30 days, long keeps hourly rates for as long as retention.long says (2 years by default, 10 maximum). There is no 48-hour tier and no 5-minute tier; raw data is cheap enough to keep for a month, and that removes a whole class of "which tier answered this" problems. Rollups are rates, not counters. Every five minutes the rollup job computes, for each interface and direction, the average rate over that window from the raw counters in hot, and writes it to long as a gauge; every hour it writes the hourly average and the maximum of those 5-minute rates. It never averages a counter. The same job rolls up every non-interface gauge (CPU, memory, storage, sensors) to a 5-minute average and an hourly average, max and min. Five-minute values are kept in long for retention.fiveminute (default 4 months) so p95 can be recomputed or disputed well after raw has aged out; hourly rates are kept for retention.long. Writes to long use a per-window version tag, and a query takes the highest version for a timestamp, so re-running a window replaces it rather than relying on the store's duplicate handling. The job records a window as done only after its write is acknowledged, re-does the last two hours each run so late samples are folded in, and marks windows where the source had gaps as partial so a graph can show them honestly. nmon rollup repair <from> <to> re-runs any range still inside hot retention. 95th percentile is defined once: the 95th percentile of 5-minute average rates over a calendar month in the site's configured timezone, per interface, per direction, in bits per second, using the nearest-rank method over the samples present; a month with less than 90% of its samples is reported with a coverage figure beside it. On the first of each month the job materialises last month's figure for every interface into long. Queries use the materialised record when it exists and compute live from the 5-minute rates otherwise, which covers the current month and any month whose materialisation failed. A materialisation that is more than a day late raises an alert, and the 4-month 5-minute window means it can still be done exactly. Queries pick the store by range, and a range that straddles 30 days is answered by stitching: raw from hot where it exists, hourly from long before that, with the join point marked in the response so the UI can draw it. If both stores are pointed at an external VictoriaMetrics Enterprise or cluster with native downsampling, the rollup job can be turned off.

Alerting and dependency tree

Two kinds of alert, because they fail differently. Reachability alerts are decided inside the poller from what it just saw, in memory, and written to the database; they work even if the metric store is dead. Metric alerts are MetricsQL rules evaluated against the hot store every 30 s by nmon's own evaluator, which keeps pending and firing state per series in the database so a restart neither re-fires nor forgets. What "down" means depends on the kind of device. Each kind has a health policy; a probe the device doesn't support is unknown, never failed. Network devices: down when ICMP has failed for 45 s and the last two SNMP polls failed; ICMP down with SNMP fine is a warning (filtered or rate-limited), SNMP failing with ICMP fine is a warning (agent dead, wrong credentials); a profile can mark a device as ICMP-blocked, in which case SNMP alone decides. VMs and containers: down when Proxmox reports them stopped or missing, with their own push metrics absent as a second signal. Check-only targets: down when the check fails for its configured count. Every kind gets a no-data warning when its series stop arriving for 5 min. A device in maintenance is maintenance, not down, and a device in backoff is degraded with its last known state kept. Dependencies. A device's down alert is suppressed only while every parent is down. One core of a redundant pair failing suppresses nothing; both failing suppresses the children and the notification names the parents and the count of children. Suppression is evaluated on the parents' reachability state, not on metrics. Default metric rules: interface down (oper down while admin up, 2 min), interface errors (error ratio above 0.1% of packets, 5 min), CPU and memory above 90% for 10 min, disk above 90% or predicted full within 24 h, UPS on battery, UPS runtime under 10 min, printer supply under 10%, temperature above the profile's threshold. Each evaluation has one of four outcomes per series: below threshold, above threshold, no data, or error. An alert resolves only on below threshold. No data for two evaluation periods moves it to a separate stale state that stays visible and is cleared by a real reading or by the device being retired; a query error leaves the state untouched and counts against the store's health. Thresholds live in config and can be overridden per group. Flap damping. More than 5 state changes in 10 minutes puts a device or interface into flapping for 15 minutes: one notification in, one out. Notifications go to webhooks (JSON, templated per receiver) and SMTP. Deliveries are queued in the database, retried with backoff for an hour, and logged so you can see what was sent and when. Acks carry a reason and an expiry. Maintenance windows per device or group suppress everything inside the window. Alert history is kept for a year.

API and UI

The UI is a static single-page app served by nmon and talks straight to the JSON API; there is no BFF or middle tier. Anything the UI can show, curl can fetch. The API addresses devices by slug and interfaces by ID, pages every list, returns timestamps in UTC, and reports partial results explicitly (a graph that had to stitch two stores, a device whose last poll was incomplete) rather than hiding them. Free-form MetricsQL through /api/v1/query is read-only, limited to 30 s and 100,000 series, and available to viewers. The route list is in the Build detail tab. UI pages, in build order:

  • Device list with up/down, site, role, search. The landing page.
  • Device page: interface table with live in/out rate, errors, status; click a row for the graph.
  • Graph view: in/out area chart, range picker (1 h, 24 h, 7 d, 30 d, 1 y), p95 line toggle, max toggle.
  • Interface search: type a customer name, get every port with that ifAlias and their graphs.
  • Alerts: active, acked, history; ack button; maintenance window form.
  • Groups: site and customer groups with aggregate in/out graph.
  • Poller health: slowest devices, timeouts, PDUs/min. Stack: React 19, TypeScript, Vite, shadcn/ui on Tailwind, TanStack Query for data, TanStack Router, uPlot for charts. uPlot is chosen for rendering throughput: a 48-port device page draws 48 graphs and a NOC wall redraws them every minute. Everything it lacks (legend, tooltip, range select, the p95 line, the stitch marker and partial shading) lives in one TrafficGraph component, the only file that imports the library. No server-side rendering. Dark mode by default because NOCs. Built assets are embedded in the Go binary with embed.FS, so one binary still means one binary. OpenAPI is the contract. The Go API is written with huma, which generates openapi.json from the handler types, so the spec can never drift from the code. The frontend never hand-writes an API type: openapi-typescript generates src/api/schema.d.ts and openapi-fetch gives a typed client. CI checks the generated spec into the repo and fails on an incompatible diff against the last tagged release, which is how the "routes frozen once shipped" rule is enforced. GET /openapi.json and a Swagger UI at /docs are served by nmon for anyone scripting against it.

Auth

OIDC only, single tenant, two roles. Configured with issuer URL, client ID, client secret, and the name of a groups claim.

  • viewer: every read. admin: everything. A group in the IdP's groups claim maps to admin; any authenticated user is a viewer.
  • Sessions are HTTP-only cookies signed with a key nmon generates and rotates; every state-changing request needs a CSRF token; refresh goes through the IdP; logout revokes server-side.
  • Break-glass: one local admin whose password hash lives in a file the operator creates. Rate-limited, locked for 15 min after 5 failures, and every use is logged and alerted. It is the only way in when the IdP is the thing that broke.
  • API tokens are minted by an admin with a scope (viewer, admin, or push-for-these-devices), an expiry, and can be revoked. Stored hashed.
  • Viewers see everything, including customer names in port descriptions. That is deliberate for v1 and listed under open decisions.

Operations

Backup is a command. nmon backup <file> takes an online snapshot of the inventory database, asks both metric stores for a consistent snapshot, adds config and profiles, and writes one tar. The encryption key file is left out unless --include-key is given, and the command says so every time. nmon restore <file> is the inverse and verifies the result before it says done. A scheduled nightly backup is one line in config. Copying the data folder while nmon is running is explicitly not a backup, and the docs say why. The supervisor. nmon unpacks its embedded VictoriaMetrics binaries into the data folder on start (pinned version, checksum verified) and runs the two stores as children on localhost. Each is health-checked every 5 s. A store that fails is restarted with backoff; three restarts in ten minutes raise an alert and stop the loop. While a store is down, writes to it queue in memory for up to 60 s and are then dropped with a counter, and queries that need it fail fast with a clear error rather than hanging. Reachability alerting keeps working throughout. Shutdown flushes the queue, stops the children, and waits for them. Overload. There is one admission policy. Pollers have a fixed in-flight cap and a per-device deadline. Ingest has per-source caps. Queries have time and series limits. The rollup job yields to ingest when the write queue is above half. When disk is above 90%, the hot store stops creating new series and an alert says so; nothing else is allowed to fill the disk. Upgrades. One binary replaces another. Database migrations run on start and are reversible one version back. The embedded store version is bumped with nmon releases; downgrading is backup, replace, restore. Configuration ownership. Settings (retention, alert thresholds, IdP, receivers, outbound allow-list) are edited by an admin in the UI, which validates them, writes config.yaml atomically and reloads; editing the file by hand and sending SIGHUP does the same thing for people who prefer it. A reload that fails validation keeps the old settings and shows the error. Inventory lives in the database and is edited through the API. Profiles are files, read on start and on reload, and a device referencing a missing profile falls back to generic with a warning. Nothing is stored in both places.

Build milestones

Milestone Builds Done when (gate)
M1 Skeleton and fixtures Repo, CI, snmpsim compose, Store interface, VM supervisor, ICMP + multi-varbind GET on generic nmon serve polls the counters fixture and /api/v1/query returns rising ifHCInOctets
M2 Discovery and profiles GETBULK discovery, interface filter, HOST-RESOURCES in generic, shipped profiles, nmon profile test every fixture's expect.yaml passes, including v1 and v3, and the PDU budget holds
M3 Graphs and UI Device list, device page, graph view, interface search, poller health page a port's in/out graph renders at 1 h, 24 h and 30 d from the counters fixture
M4 Alerting MetricsQL rules, dependency suppression, flap damping, webhook + SMTP, ack, maintenance windows stopping a parent fixture sends one notification naming it and its children
M5 Push ingest, rollup, checks Influx and remote-write ingest, Proxmox inventory, rollup job, ping/HTTP/DNS/TLS checks, p95 API a replayed Proxmox sample shows under its node and a 2 y query hits the 1 h tier
M6 Auth and ops OIDC roles, API tokens, break-glass login, backup and restore, LibreNMS CSV import, loadtest a viewer gets 403 on PUT and the 2,000-device loadtest cycle finishes under 40 s

The agent gets one milestone at a time and stops at its gate for review. M1 is deliberately boring: the fixtures and the Store interface exist before any feature, so every later milestone is testable without hardware. Each gate is a command the reviewer can run, not a judgment call. gNMI is M7, after v1 ships.

Open decisions and later roadmap

Decided in this spec: Go; two embedded VictoriaMetrics stores run as children; Bun ORM over SQLite with Postgres tested in CI; SNMPv3 default; rollups store rates; stable IDs as the only metric labels; multiple parents; backup as a command; Apache 2.0 licence, with vendor MIBs fetched by the operator and never redistributed; uPlot for charts, wrapped in one TrafficGraph component so the library is replaceable. Also decided: viewers see everything, including customer names in port descriptions; no third role in v1. No LibreNMS history import, only the device and port-description CSV import. The trap receiver is on by default and the shipped systemd unit grants it port 162. Non-interface gauges (CPU, memory, environment) get the same rollup as interfaces: 5-minute averages for retention.fiveminute, hourly average, max and min for retention.long. Nothing is open. New questions go on this list as they come up. After v1, in rough order:

  • gNMI streaming (M7).
  • Vendor profiles contributed from recordings: Cambium, Ubiquiti airMAX, Mimosa, OLTs.
  • AI layer over the API: profile drafting from a walk, onboarding a subnet, alert triage summaries. Separate process, uses an API token, product works with it off.
  • Syslog and flow as separate tools that share the inventory API.