# Operating nmon How to run, secure, back up and maintain a production nmon. Design background: `docs/spec/spec.md` ("Auth", "Operations"). This page, [Adding a device profile](docs/profiles.md) and [Collection coverage](docs/coverage.md) are also built into the binary and shown in the UI under **Help** (`/help`, any signed-in user, no network needed). The help page always matches the running version; the Settings forms link their section's "?" to the matching heading here. ## Starting nmon serve --data-dir /var/lib/nmon --secret-key-file /etc/nmon/secret.key \ --tls-cert /etc/nmon/tls.crt --tls-key /etc/nmon/tls.key - **HTTPS is the default.** Pass a certificate pair (`--tls-cert`, `--tls-key`; `SIGHUP` reloads them without dropping connections) or Let's Encrypt (`--acme-host nmon.example.net` repeatable, `--acme-email you@example.net`; nmon answers the http-01 challenge on `--acme-listen`, default `:80`, certificates are cached in `/acme`). With neither (and no `--http-listen`) nmon generates a self-signed certificate, see [The self-signed certificate](#the-self-signed-certificate). `--insecure-http` is for development: plain HTTP, cookies without `Secure`. - **A sign-in method is required** over TLS (`auth.oidc` and/or `auth.break_glass_file` in `config.yaml`). On a first start with none configured nmon opens the [setup wizard](#installing-nmon-first-start) instead of refusing. With `--insecure-http` and neither configured nmon runs in *dev-open mode* (everyone is admin `dev`; a banner says so); never expose that port. - Behind a reverse proxy, see [Behind a reverse proxy](#behind-a-reverse-proxy): `--http-listen ADDR` adds an opt-in plain-HTTP listener, and the proxy goes in `auth.trusted_proxies`; `X-Forwarded-*` is ignored from anyone else. ## Installing nmon / first start On a fresh install nmon needs no `config.yaml`: start it, open `https://:8443/` and the browser lands on `/setup`, a five-step wizard (sign-in, basics, first devices, alerting, finish). Steps 2 to 4 can be skipped and done later in Settings. nmon serve --data-dir /var/lib/nmon --secret-key-file /etc/nmon/secret.key ### Install mode Install mode is on only while **no sign-in method is configured** (no `auth.oidc`, no `auth.break_glass_file`) **and setup has never been completed**. While it is on: - **Anyone who can reach nmon can run setup**, as an admin. There is no setup token and no localhost restriction: nmon is meant to sit in a DMZ or an internal network (`docs/DEVIATIONS.md` row 51). Finish setup right after starting, or keep the port firewalled until you have. - nmon logs a `WARN` at start and every 10 minutes: `INSTALL MODE: setup is open to anyone who can reach this address`. - Only the wizard works. The UI, `GET /api/v1/setup`, the settings form, webhook presets/preview/test, adding devices (and CSV import), alert policies, the profile list, secrets and `GET /api/v1/meta` are served; **every other API route answers `503` "setup not finished"** until setup completes. Install mode ends **for good** when setup completes (`POST /api/v1/setup/complete`, the wizard's last step). The completion is stored in the inventory database and nothing on the web can undo it. If you later delete the break-glass file or the `auth:` block, nmon does not reopen setup: it refuses to start with "no way to sign in", as before. An existing install that already has a sign-in method is marked completed the first time it starts on this version, so an upgrade can never open install mode. `--insecure-http` (development) never enters install mode. To skip install mode altogether, configure sign-in before the first start, as the Ansible role does (`nmon_break_glass_file` / `nmon_oidc_*`): such an install is treated as already set up. ### Setup wizard: sign-in Required. Pick the break-glass administrator (default `admin`) and a password of at least 12 characters. nmon hashes it with bcrypt, writes `/breakglass` (mode 0600, `user:hash`) and sets `auth.break_glass_file`. Optionally add an OpenID Connect provider: issuer, client id, redirect URL (prefilled from the address you are using plus `/auth/callback`), groups claim, admin group and the client secret, which is stored encrypted as the secret `oidc-client`. nmon checks the fields now and contacts the provider at the first sign-in. Sign-in to the web UI has no login page: OIDC redirects the browser to the provider, and the break-glass admin signs in at `/auth/breakglass` (HTTP Basic). ### Setup wizard: basics The time zone (site default for p95 months), retention (raw samples `retention.hot` 30d, five-minute rates 120d, hourly rates 2y) and the default poll interval (`poller.interval`, 30s to 1h, default 5m). All of them are also in Settings, General and Retention. ### Setup wizard: first devices Choose an SNMP credential (v2c community, or v3 user with auth and privacy), then paste IP addresses (one per line, IPv6 or IPv4) or upload a CSV. Each address is added like the Add device dialog does it, so a device that does not answer SNMP is reported and not added. Host names are not resolved; use the address. The step shows each device's discovery state live. ### Setup wizard: alerting Add one receiver, a webhook (with the Teams, Slack, Discord and generic JSON presets, a live preview and Send test) or email, and optionally turn on the recommended alert policies: `device_down`, `device_warning`, `interface_errors`, `cpu_high`, `memory_high` and `disk_full`, each for all devices. Receivers get every notification, so the policies need no binding to the new receiver. Running the step again creates nothing twice. Interface-down alerts stay opt-in (see below). ### Setup wizard: finish The wizard summarises what it set up and calls `POST /api/v1/setup/complete`. nmon checks that a sign-in method now exists (otherwise `409`), stores the completed flag, and switches authentication on **without a restart**: the live auth service is replaced by one built from the new configuration, and install mode closes. The browser is then sent to sign in. ### The self-signed certificate When none of `--tls-cert`/`--tls-key`, `--acme-host` or `--http-listen` is given, nmon serves HTTPS with a self-signed ECDSA P-256 certificate kept in `/tls/` (`cert.pem`, `key.pem` mode 0600). It is created once and reused on every start; it is valid for about two years and replaced within 30 days of expiry. Its names are the host name, `localhost`, `::1`, `127.0.0.1` and every non-loopback interface address (IPv6 included), as they were when it was created: delete the two files to make a new one after the host's addresses change. Browsers warn about it, and nmon logs a `WARN` at each start. Replace it with `--tls-cert`/`--tls-key` (reloaded on `SIGHUP`) or `--acme-host`. With `--http-listen` alone nmon serves plain HTTP only and creates no certificate. ## Behind a reverse proxy nmon serve --data-dir /var/lib/nmon --secret-key-file /etc/nmon/secret.key --http-listen [::1]:8080 `--http-listen ADDR` (default off; any interface is allowed, but loopback or an internal address is the point) adds a **plain-HTTP listener** that serves the same UI and API. HTTPS stays the default and is unchanged. - Beside HTTPS: with `--tls-cert/--tls-key` or `--acme-host` too, nmon serves HTTPS on `--listen` and plain HTTP on `--http-listen`, so the proxy may reach it either way. Alone: with no TLS configured nmon starts with only the HTTP listener (`--listen` is not bound) instead of refusing. SIGHUP still reloads the certificate for the HTTPS listener. - **Sign-in stays on** over this listener (OIDC, break-glass, API tokens), exactly as on HTTPS. It is not dev-open mode: with no `auth.oidc`/`auth.break_glass_file` nmon refuses to start. `--http-listen` and `--insecure-http` are mutually exclusive (`--insecure-http` is development mode with no sign-in). - List the proxy in `auth.trusted_proxies` (addresses or CIDRs, live-reloadable). Only a request from a trusted proxy has its headers honoured: `X-Forwarded-For` gives the client address used by the audit log and the break-glass lockout (the rightmost hop that is not itself a trusted proxy), and `X-Forwarded-Proto: https` marks the browser connection as HTTPS. From anyone else those headers are ignored and removed, so they cannot be spoofed. With the list empty nmon logs a WARN at startup: every client is seen as the proxy's address, so the lockout is per-proxy. - **Cookies:** the session and sign-in-flow cookies are always `Secure` on the HTTPS listener. On the HTTP listener they are `Secure` only when a trusted proxy sent `X-Forwarded-Proto: https`; a direct plain-HTTP client gets them without `Secure` (a browser would drop them otherwise). - **OIDC:** `auth.oidc.redirect_url` is configured explicitly and is never derived from the request, so set it to the public URL (`https://nmon.example.net/auth/callback`). CSRF uses a per-session token, not the Origin header, so a different public host or scheme needs nothing else. - nmon uses no websockets or long-lived streams; the proxy needs no upgrade headers. Raise the proxy's body-size limit if you import large CSV files. nginx, proxying to the plain-HTTP listener: server { listen 443 ssl; listen [::]:443 ssl; server_name nmon.example.net; ssl_certificate /etc/ssl/nmon.example.net.crt; ssl_certificate_key /etc/ssl/nmon.example.net.key; client_max_body_size 64m; location / { proxy_pass http://[::1]:8080; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 120s; } } with `auth: {trusted_proxies: ["::1"]}` in `config.yaml`. To talk TLS to nmon instead, run nmon with its normal `--tls-cert/--tls-key` (no `--http-listen` needed), use `proxy_pass https://[::1]:8443;` with `proxy_ssl_verify off;` (or `proxy_ssl_trusted_certificate` for a CA-signed pair), and the same `proxy_set_header` lines; the listener is HTTPS, so cookies are `Secure`. Ansible: set `nmon_http_listen: "[::1]:8080"` (and `nmon_tls_mode: none` to serve only that listener). ## Sign-in `config.yaml`: auth: oidc: issuer: https://idp.example.net/realms/net client_id: nmon client_secret_name: oidc-client # the secret itself is never in this file redirect_url: https://nmon.example.net/auth/callback groups_claim: groups admin_group: nmon-admins # members are admins; every other user is a viewer break_glass_file: /etc/nmon/breakglass # bcrypt hash, "hash" or "user:hash", created by you Store the client secret before the first start: `printf '%s\n' "$SECRET" | nmon secret set oidc-client --data-dir D --secret-key-file K` (it can also be set later with `PUT /api/v1/secrets/oidc-client`; OIDC stays off until the secret exists). The break-glass file holds one bcrypt hash (`$2a$`/`$2b$`), made with any bcrypt tool (the source tree's `nmon-devtools bcrypt PASSWORD` works); keep it readable by the nmon user only. - There is no login page. Opening nmon without a session redirects to the IdP and back; API clients use a token. If the IdP is down, open `/auth/breakglass`: the browser asks for the local admin user and password (HTTP Basic), and a successful sign-in lands on the page you asked for. - Break-glass: five wrong passwords lock the source address for 15 minutes; every use and every lock is audited and raises an alert. `nmon unlock [IP|all] --data-dir D` clears locks, also while nmon runs. - API tokens: an admin mints viewer, admin or push tokens (`POST /api/v1/tokens`, or Settings, API tokens). Use `Authorization: Bearer nma_...`. A push token works for ingest only. Revoking takes effect on the next request. - `GET /api/v1/audit` (admin) lists who did what. ## Backup and restore A running nmon owns the stores, so copying the data folder is **not** a backup. nmon backup /backups/nmon.tar --api https://nmon.example.net --token nma_... # key NOT included nmon backup /backups/nmon.tar --include-key --api ... --token ... # key included Every run prints whether the key file is in the archive. **Lose the key file and a restore has an inventory with no SNMP credentials.** The nightly schedule is `backup: {at: "02:30", dir: /var/backups/nmon, keep: 7}` (time in the configured timezone, default dir `/backups`); a failed run raises the `backup_failed` alert. A backup is refused when the disk is below its 5% reserve. Restore with nmon stopped, into a clean directory: nmon restore /backups/nmon.tar --data-dir /var/lib/nmon-restored --secret-key-file /etc/nmon/secret.key It verifies inventory row counts and a sample of series before it says done. Then start nmon on that directory. Restore needs the embedded stores (the normal release build). ## Keys `nmon key rotate --data-dir D --secret-key-file K` (nmon stopped) adds a new key version, re-encrypts every SNMP credential and secret under it, and keeps the old key in the file so older backups still restore. Remove old keys only after a release. ## Configuration Settings (admin) is point-and-click: a section list (General, Retention, Sites, Notifications, Authentication, Backup, Disk guard, L2 and OUI, Proxmox, Checks, Alerting) with a form per section, its own Save and Revert, and the applied / restart-required result under it. Unset fields show the default as a placeholder; durations take `30d`, `12h`, `5m`. A form that refers to a secret (OIDC client secret, Proxmox token) takes the secret's name and has a "Set secret value" button that stores the value write-only through `PUT /api/v1/secrets/{name}`; it is never shown again. The same sections are available as typed JSON: `GET /api/v1/settings/form` and `PUT /api/v1/settings/form` (each section present in the body replaces that section; the others are untouched). Alert rules and policies are not here, they are under Alerts; `alerting.rules` in `config.yaml` is preserved by the forms. The Advanced (YAML) tab and `PUT /api/v1/settings` still edit the whole document, and `config.yaml` + `SIGHUP` still works; all three go through the same validation. Saving rewrites `config.yaml` from the parsed configuration, so comments in the file are not kept. An invalid document is rejected and the running configuration stays. Alert receivers, the outbound allow-list, trusted proxies, the backup schedule, disk thresholds and the purge period apply live; everything else is listed as `restart_required` until the next restart. `disk: {alert_used: 0.8, admit_used: 0.9, reserve_free: 0.05}`: alert at 80% used, no new series at 90%, ingest throttled / rollup paused / backups refused below 5% free. ## Metric stores The supervisor restarts a crashed store; after three restarts in ten minutes it raises `store_failed` and stops. `nmon store restart [hot|long] --api URL --token T` (or `SIGHUP`) resumes it. ## Interface-down alerts are opt-in Since v1.1 the bell and the group switch only **mark** ports: `interface_down` alerts for them only where an alert policy for `interface_down` uses the `alert_down` matcher (template "Bell-marked ports: interface_down"). Mark ports in one at a time (the bell on the device page's interface table), all at once (the "Alert on down" control in the interfaces card header, or `POST /api/v1/devices/{slug}/interfaces/alert-down`), or through a group (the group's "Alert when member ports go down" switch; `PUT /api/v1/interfaces/{id}` with `{"alert_down": true}` for one port). A port in a group with the switch on alerts for as long as its membership is in force. Changes reach the evaluator at once; opting a port out closes its open alert silently as `retired`. ## Alert policies: nothing alerts until you add one Alerts fire only where an **alert policy** binds a rule to a scope. A fresh install has no policies and records no alerts at all: no metric rule, no `device_down` or `device_warning`, no system alarm (ingest caps, late p95, rollup, disk guard, break-glass, backup). The startup log says so once ("no alert policies: nothing will alert"). Upgrading from v1.0 seeds nothing: the first cycle closes every open alert that no policy binds, silently, as `retired` (no "resolved" notices go out). Manage policies under Alerts > Alert policies (admins edit, viewers read) or with `GET/POST /api/v1/alert-policies` and `GET/PUT/DELETE /api/v1/alert-policies/{id}`. `GET /api/v1/alert-rules` lists the rule names. `POST /api/v1/alert-policies/preview` counts the devices and ports a draft scope covers. Changes apply on the next cycle with no restart. Removing or disabling a policy retires its open alerts quietly. A policy has a rule (a metric rule, `device_down`, `device_warning`, or `system`), an action (`alert` or `ignore`), a priority, optional `severity`, `threshold` and `for` overrides, and a scope. Scope fields are ANDed and any entry of a list may match: `all`, `groups` (a port is in force as a member now; a device matches if any port does), `devices` (`slugs` globs, `profiles`, `roles`, `sites`, `kinds`) and `interfaces` (`if_types`, `if_name` and `if_alias` as RE2, `min_speed_mbps`, `max_speed_mbps`, `alert_down`). For one entity and rule the matching policy with the highest priority decides; on a tie `ignore` beats `alert`, then the lowest name. A policy threshold beats a rule's group override, the profile threshold and the default. Interface matchers are refused on device-level rules, and a `system` policy takes only `all: true`. Fiber and uplinks alert on any error, WiFi never alerts on errors: ``` {"name": "Fiber + uplinks: errors", "rule": "interface_errors", "action": "alert", "threshold": 0, "scope": {"interfaces": {"if_types": [6], "if_alias": "(?i)fib|uplink|core"}}} {"name": "WiFi: no error alerts", "rule": "interface_errors", "action": "ignore", "priority": 10, "scope": {"interfaces": {"if_types": [71]}}} ``` ## Alert rules: built-in and custom The shipped rules (`interface_down`, `interface_errors`, `cpu_high`, `memory_high`, `disk_full`, `ups_on_battery`, `ups_runtime_low`, `printer_supply_low`, `printer_supply_empty`, `temp_high`) are read-only **built-in** rules, listed with their generated query under Alerts > Alert rules. `cpu_high`, `ups_on_battery` and `printer_supply_empty` have **Copy to custom**, which clones them into an editable builder rule; the others join, filter or union series and say why they cannot be copied. Rules in `config.yaml` (`alerting.rules`) still work but are deprecated, and the startup log says so; define rules in the UI instead. A rule name is unique across all kinds, and `device_down`, `device_warning`, `system` and the subsystem alarm names stay reserved. Admins add their own rules with the builder (UI or `POST /api/v1/alert-rules`). There is no free-form query: nmon builds it from whitelisted parts, and the generated MetricsQL is shown read-only. **Test** runs the draft once and shows how many entities breach now. Edits apply on the next cycle; disabling a rule or deleting it retires its open alerts silently, and deleting a rule that a policy still names is refused (409, naming the policies). - Metric rule, "CPU over 80 % on role=server for 15 min": metric `hrProcessorLoad`, aggregate `max` (one value per device), op `>`, threshold `80`, hold time `15m`; then a policy for this rule with `devices.roles: [server]`. Transforms are `value`, `rate` (counters, per second) and `percent_of` (metrics with an `of`, such as `hrStorageUsed`); the window defaults to 5 min (at least twice the poll interval). Push-ingested metrics are device-level. - Inventory rule, "Uplink without description": interface fields `if_alias empty` AND `if_speed_mbps >= 10000`. - Inventory rule, "Device not seen for 1 h": device field `last_seen_age older_than 1h`. A custom rule, like a built-in, alerts nowhere until an alert policy binds it: the rule defines **what** is measured and its default threshold and hold time, and the policy defines **where** it applies and may override severity, threshold and `for`. Several policies can bind one rule to different scopes. ## Webhook templates Each webhook receiver (Settings > Notifications) has a Go `text/template` that renders the request body. It must produce one valid JSON document (max 256 KB); it is sent as `application/json`. Empty means the default, `{{json .}}`, the whole notification. SMTP receivers use the same editor for `Subject: ...`, a blank line and the body. **Fields** (a template sees nothing else): `.IncidentID`, `.AlertID`, `.Rule`, `.State` (`firing`, `resolved`, `flapping`, `stale`, `suppressed_summary`), `.Severity` (`critical`, `warning`, `info`), `.Device.Slug`, `.Device.Name`, `.Interface.ID`, `.Interface.Name`, `.Interface.Alias` (empty for device-level alerts), `.Summary`, `.Description`, `.FiredAt`, `.ClearedAt` (nil until resolved), `.Labels`, `.Parents`, `.SuppressedChildren`, `.URL` (link to the device in nmon). **Functions:** `json`, `upper`, `lower`, `join`, `timefmt` (`{{timefmt .FiredAt "2006-01-02 15:04:05 MST"}}`, UTC), plus the standard `printf`, `eq`, `or`, `if`. **The `{{json}}` rule:** wrap every value you put inside a JSON string or value in `{{json ...}}` (`"text": {{json .Summary}}`, never `"text": "{{.Summary}}"`). `json` adds the quotes and escapes quotes, newlines and control characters, so an alert text can never break the document. Secret-looking keys and `key=value` pairs are redacted from the rendered payload before it is stored or sent. **Presets:** "Start from preset" in the receiver form fills the template (it asks before replacing a non-empty one). Built in, also at `GET /api/v1/notify/presets`: | Preset | For | |---|---| | Generic JSON | the default; the whole notification | | Microsoft Teams (Workflows) | Adaptive Card 1.4 for the Workflows trigger "When a Teams webhook request is received" | | Slack | incoming webhook: text fallback and a coloured attachment with blocks | | Discord | webhook with one embed, coloured by state and severity | Teams: Microsoft is retiring the Office 365 connectors ("Incoming Webhook" in a channel's Connectors). Create a Workflow from the template "Post to a channel when a webhook request is received" and paste its URL. The preset is an Adaptive Card message and will not work with a legacy connector URL. **Preview and test:** the editor shows the server-rendered payload for a sample notification in any state as you type, or the template or JSON error. "Send test" delivers one sample immediately (no retry queue; nothing stored except an `notify.test` audit row) and shows the HTTP status, a response excerpt (4 KB, redacted), the duration and any error. Blank header values and a blank SMTP password use the stored receiver of the same name, so you can test a saved receiver without retyping credentials. Admin only; at most two tests run at once. `POST /api/v1/notify/preview` and `POST /api/v1/notify/test` take `{"receiver": {...form shape...}, "state": "firing"}`. **Outbound allow-list:** test sends and real deliveries go through the same gate. A webhook to a private, loopback or link-local address is refused unless it matches `alerting.outbound_allow` (Settings > Notifications), and the test result says so. ## Retired devices `POST /api/v1/devices/{slug}/purge` (admin) deletes a retired device's series from both stores once it has been retired for `retired_purge_after` (default 1y). ## CSV import Devices (`name,address,site,role,credential,parents,...`) and port descriptions (`device,ifname,alias`) from Settings or `POST /api/v1/devices/import`. Port descriptions are stored in the operator-owned `description` field, which discovery never overwrites. A dependency cycle is rejected with its path; nothing is written unless the whole import is valid. ## Find host: MAC, ARP and FDB lookup The **L2 table job** walks the bridge forwarding tables, VLAN names and ARP / IPv6 neighbour tables of every SNMP device every `l2.interval` and keeps them with first and last seen in the inventory, so the **Find host** page (and the header search, for a MAC, IP, CIDR or `vendor:name`) answers "which switch port is this host plugged in at, in which VLAN, and when did we last see it", with the uplinks that also saw it collapsed underneath. A MAC is seen on every switch it passes through, so the answer is the **edge** port: one that carries at most `l2.edge_max_macs` MACs, has no LLDP neighbour and is not a LAG. ```yaml l2: interval: 15m # default 15m, minimum 5m keep: 90d # history kept by last_seen, pruned daily edge_max_macs: 8 # a port with more MACs is an uplink concurrency: 4 # devices walked at once device_timeout: 2m # a slower device is skipped for the cycle oui: fetch: true # IEEE vendor registry; false never touches the network refresh: 7d ``` All of these apply from the next cycle after a reload (Settings save or SIGHUP) with no restart. The job never runs on the poll path and yields to the poller the way the rollup job does. A device that does not answer in time is skipped for that cycle and named on the Poller page (`l2` status and an `l2_devices_skipped` note); it never delays the other devices. A Cisco IOS switch keeps one bridge table per VLAN: the `cisco-ios` profile sets `l2.per_vlan_context: true`, which lists the VLANs from CISCO-VTP-MIB and walks each with community `@` (v3: context `vlan-`). Other devices use Q-BRIDGE-MIB, then BRIDGE-MIB (VLAN unknown). API: `GET /api/v1/l2/search?q=`, `GET /api/v1/devices/{slug}/{l2,fdb,arp,l2/ports}`, `GET /api/v1/interfaces/{id}/macs`. On a device, the **L2** tab (MAC table with a VLAN filter, ARP / ND, VLAN list) appears once the job has read it, and the Ports tab shows each port's MAC count linking to that port's MACs. ## MAC vendors (optional IEEE registry) If a registry is loaded, each MAC shows its vendor, a locally administered address (typical of randomised phone and VM MACs) shows a "private" badge, and `vendor:ubiquiti` lists the current MACs of a vendor. nmon works fully without it: the vendor is simply empty, and the Poller page notes `oui_unavailable` (not an alert). The registry is the IEEE's public data: MA-L (`https://standards-oui.ieee.org/oui/oui.csv`), MA-M (`.../oui28/mam.csv`) and MA-S (`.../oui36/oui36.csv`), downloaded weekly through the outbound allow-list (`standards-oui.ieee.org` must be allowed when an allow-list is configured) with timeouts, a size cap and a row-count sanity check, so a truncated or wrong download keeps the previous file. The compact result is `/oui/registry.gz` (checksummed, written atomically, included in `nmon backup`). Where nmon may not fetch, set `oui.fetch: false` and load the CSVs by hand: `nmon oui import --data-dir DIR` (a running nmon notices within an hour). The CSVs are not part of this repository. ## BGP Router profiles collect BGP session state, uptime, accepted prefixes per address family and update rates every 5 minutes (see [Collection coverage](docs/coverage.md) for which profiles and MIBs). The **BGP** page lists every session on every device, the ones that are down first, filterable by state, AS, family and text; a device's **Routing** tab shows its sessions with click-to-graph on prefixes and updates. The built-in rule `bgp_peer_down` (critical, state not established for 2 minutes) fires only where a policy binds it, like every rule: use the template "Routers: BGP peer down" under Alerts > Alert policies. Its notification names the peer and, from discovery, the peer's AS and description. A prefix-drop alert is a custom rule (v1.2); an example will be added here once custom rules are in. ## Poll interval SNMP interface counters are polled every 300 s by default. An admin can set a different interval for one device (Device settings, "Poll interval", 30 to 3600 s; blank returns it to the default) or through `PUT /api/v1/devices/{slug}` with `poll_interval_s`. The poller picks it up at once, no restart. Graph rate windows, reachability fallbacks and the rollup follow the device's own interval (see `docs/DEVIATIONS.md` row 49). ICMP is still probed every 15 s. **Port totals.** A port graph (`GET /api/v1/graph`) also returns `totals`: bytes in and out over the requested range, the average-rate series integrated over each point's own step (hourly and raw data stitched without double counting). `covered_seconds` against `range_seconds` says how much of the range had data; the UI marks a total with `*` when coverage is below 99.5%. Unreliable (32-bit fast link) ports get none. The port grid shows `total` beside avg and p95; the full graph has a Total column. Per port only. **Port graph parity with LibreNMS.** The port summary and the full graph show, per direction, Now / Avg / Max / 95th / Total. Max is the highest *plotted* point (with `max=true` the finer "peak within a step" is a separate Peak column). The aggregate 95th (`p95.aggregate_bps`) is the 95th percentile of in+out per 5-minute slot, counted over slots where both directions have a sample; it is live from the hot store (also next to a materialised monthly figure) and is absent when only one direction was asked for or the range has no raw data. When the port speed is known, the full legend prints each rate's share of that speed, and a toggle plots the y axis as percent of speed (hidden for unknown or zero speed). "Compare previous period" (device port row and graph view) fetches the same-length range just before the current one with the ordinary `from`/`to` of `GET /api/v1/graph`, shifts it forward onto the current axis and draws it faded and dashed, with its own avg / max / 95th / total rows in the legend; it is off while a p95 month is selected. The graph-type selector offers **Bits**, **Errors** and **Discards**. Errors and discards come from `GET /api/v1/graph/counters?iface=&kind=errors|discards`, the per-second rate of `ifInErrors`/`ifOutErrors` or `ifInDiscards`/`ifOutDiscards` with the same step and rate-window rules as bits. These counters have **no long-store rollup**, so the endpoint answers from the hot store only; a range that starts before hot retention is cut to its start (`clamped: true`, and the UI says so). **Packets** (unicast pps) and **Broadcast/Multicast** appear disabled: the generic profile does not poll `ifHC*UcastPkts`, `ifHC*MulticastPkts` or `ifHC*BroadcastPkts`. Adding them means polling three more columns on every port of every device and regenerating every fixture, so it is a separate decision (DEVIATIONS 56). The default itself is `poller.interval` in `config.yaml` (Settings, General, "Poll interval"; 30s to 1h, default 5m). A change applies without a restart to the poller, reachability, the rollup and the API. The dev flag `nmon serve --poll-interval`, when given, wins over the setting. ## Testing and debugging a device's polling Three admin tools answer "why is this device not polling right", from the device page (Actions menu) or the API. None of them shows or logs a credential: the trace names the SNMP version, never the community or the v3 secrets. - **Poll now** (`POST /api/v1/devices/{slug}/poll-now`, 202) runs the device's primary group right away, outside its schedule. The result lands in the normal places (graphs, reachability). It answers 409 when the device is retired, disabled, has no SNMP credential, or is not loaded by the poller yet. - **Test poll** (`POST /api/v1/devices/{slug}/poll-test`, optional `?interval_seconds=300` for one interval group) runs one synchronous cycle of the device's real poll plan: the stored credential, the resolved profile, the address and port, and the same fetch code the poller uses (the varbind count starts at the default and halves on tooBig or a lost response, exactly as in polling, but is forgotten afterwards). It writes **no metrics** and changes no poller state, so it cannot cause a reboot event, a backoff or a down alert. The answer lists each request (the session, number of varbinds, first OID, start and duration in ms, outcome `ok`, `timeout`, `too_big`, `snmp_error`, `dial_error` or `error`, and whether it was a retry in smaller pieces) and every requested OID with the series it feeds, its label (interface ULID or component), the SNMP type, the raw value and what the poller would store, or why not: `no_such_object`, `no_such_instance`, `end_of_mib`, `skipped` (text, type mismatch, not available), `not_fetched` (its request failed) or `oid_mismatch`. The `duration_ms` of a request includes the SNMP library's own retries; they are not listed separately. Limits: one test per device at a time (409), four at once overall (429), 60 s at most, 5,000 OID rows at most (`truncated: true`; failures and misses are kept first). Every test is audited as `device.poll_test`. - **SNMP debug** (`PUT /api/v1/devices/{slug}/snmp-debug` with `{"minutes": N}`; 0 turns it off, 60 at most; `GET` reads it) records every scheduled cycle of one device while it is on: the same request and OID detail as a test poll, for the last 20 cycles (500 OID rows each, failures first), newest first. It also logs one INFO line per cycle (`snmp debug cycle`, with the counts, duration and error). It switches off by itself when the minutes are up. **Recordings are kept in memory only** and are cleared on restart; starting a new recording drops the previous one, and switching off keeps what was captured so you can still read it. Turning it on or off is audited as `device.snmp_debug`. Use Poll now while it is on to get a cycle at once. From a shell: `nmon device poll --api URL --token TOKEN` queues a poll now; add `--test` for the traced test poll (`--interval SECONDS` for one group, `--json` for the full trace). Reading a trace: a request that times out with every OID `not_fetched` is a reachability, community or ACL problem (check the target address, the SNMP version and the device's allowed managers). A `too_big` request followed by `retry` requests means the device cannot answer 40 varbinds at once; the poller adapts, so this is only a note. Many `no_such_instance` rows for interface OIDs mean the stored interfaces no longer match the device: run Rediscover.