nmon/docs/operations.md

Operating nmon

How to run, secure, back up and maintain a production nmon. Design background: docs/spec/spec.md ("Auth", "Operations").

This page, Adding a device profile and Collection coverage are also built into the binary and shown in the UI under Help (/help, any signed-in user, no network needed). The help page always matches the running version; the Settings forms link their section's "?" to the matching heading here.

Starting

nmon serve --data-dir /var/lib/nmon --secret-key-file /etc/nmon/secret.key \
           --tls-cert /etc/nmon/tls.crt --tls-key /etc/nmon/tls.key

• HTTPS is the default. Pass a certificate pair (--tls-cert, --tls-key; SIGHUP reloads them without dropping connections) or Let's Encrypt (--acme-host nmon.example.net repeatable, --acme-email you@example.net; nmon answers the http-01 challenge on --acme-listen, default :80, certificates are cached in <data-dir>/acme). With neither (and no --http-listen) nmon generates a self-signed certificate, see The self-signed certificate. --insecure-http is for development: plain HTTP, cookies without Secure.
• A sign-in method is required over TLS (auth.oidc and/or auth.break_glass_file in config.yaml). On a first start with none configured nmon opens the setup wizard instead of refusing. With --insecure-http and neither configured nmon runs in dev-open mode (everyone is admin dev; a banner says so); never expose that port.
• Behind a reverse proxy, see Behind a reverse proxy: --http-listen ADDR adds an opt-in plain-HTTP listener, and the proxy goes in auth.trusted_proxies; X-Forwarded-* is ignored from anyone else.

Installing nmon / first start

On a fresh install nmon needs no config.yaml: start it, open https://<host>:8443/ and the browser lands on /setup, a five-step wizard (sign-in, basics, first devices, alerting, finish). Steps 2 to 4 can be skipped and done later in Settings.

nmon serve --data-dir /var/lib/nmon --secret-key-file /etc/nmon/secret.key

Install mode

Install mode is on only while no sign-in method is configured (no auth.oidc, no auth.break_glass_file) and setup has never been completed. While it is on:

• Anyone who can reach nmon can run setup, as an admin. There is no setup token and no localhost restriction: nmon is meant to sit in a DMZ or an internal network (docs/DEVIATIONS.md row 51). Finish setup right after starting, or keep the port firewalled until you have.
• nmon logs a WARN at start and every 10 minutes: INSTALL MODE: setup is open to anyone who can reach this address.
• Only the wizard works. The UI, GET /api/v1/setup, the settings form, webhook presets/preview/test, adding devices (and CSV import), alert policies, the profile list, secrets and GET /api/v1/meta are served; every other API route answers 503 "setup not finished" until setup completes.

Install mode ends for good when setup completes (POST /api/v1/setup/complete, the wizard's last step). The completion is stored in the inventory database and nothing on the web can undo it. If you later delete the break-glass file or the auth: block, nmon does not reopen setup: it refuses to start with "no way to sign in", as before. An existing install that already has a sign-in method is marked completed the first time it starts on this version, so an upgrade can never open install mode. --insecure-http (development) never enters install mode.

To skip install mode altogether, configure sign-in before the first start, as the Ansible role does (nmon_break_glass_file / nmon_oidc_*): such an install is treated as already set up.

Setup wizard: sign-in

Required. Pick the break-glass administrator (default admin) and a password of at least 12 characters. nmon hashes it with bcrypt, writes <data-dir>/breakglass (mode 0600, user:hash) and sets auth.break_glass_file. Optionally add an OpenID Connect provider: issuer, client id, redirect URL (prefilled from the address you are using plus /auth/callback), groups claim, admin group and the client secret, which is stored encrypted as the secret oidc-client. nmon checks the fields now and contacts the provider at the first sign-in. Sign-in to the web UI has no login page: OIDC redirects the browser to the provider, and the break-glass admin signs in at /auth/breakglass (HTTP Basic).

Setup wizard: basics

The time zone (site default for p95 months), retention (raw samples retention.hot 30d, five-minute rates 120d, hourly rates 2y) and the default poll interval (poller.interval, 30s to 1h, default 5m). All of them are also in Settings, General and Retention.

Setup wizard: first devices

Choose an SNMP credential (v2c community, or v3 user with auth and privacy), then paste IP addresses (one per line, IPv6 or IPv4) or upload a CSV. Each address is added like the Add device dialog does it, so a device that does not answer SNMP is reported and not added. Host names are not resolved; use the address. The step shows each device's discovery state live.

Setup wizard: alerting

Add one receiver, a webhook (with the Teams, Slack, Discord and generic JSON presets, a live preview and Send test) or email, and optionally turn on the recommended alert policies: device_down, device_warning, interface_errors, cpu_high, memory_high and disk_full, each for all devices. Receivers get every notification, so the policies need no binding to the new receiver. Running the step again creates nothing twice. Interface-down alerts stay opt-in (see below).

Setup wizard: finish

The wizard summarises what it set up and calls POST /api/v1/setup/complete. nmon checks that a sign-in method now exists (otherwise 409), stores the completed flag, and switches authentication on without a restart: the live auth service is replaced by one built from the new configuration, and install mode closes. The browser is then sent to sign in.

The self-signed certificate

When none of --tls-cert/--tls-key, --acme-host or --http-listen is given, nmon serves HTTPS with a self-signed ECDSA P-256 certificate kept in <data-dir>/tls/ (cert.pem, key.pem mode 0600). It is created once and reused on every start; it is valid for about two years and replaced within 30 days of expiry. Its names are the host name, localhost, ::1, 127.0.0.1 and every non-loopback interface address (IPv6 included), as they were when it was created: delete the two files to make a new one after the host's addresses change. Browsers warn about it, and nmon logs a WARN at each start. Replace it with --tls-cert/--tls-key (reloaded on SIGHUP) or --acme-host. With --http-listen alone nmon serves plain HTTP only and creates no certificate.

Behind a reverse proxy

nmon serve --data-dir /var/lib/nmon --secret-key-file /etc/nmon/secret.key --http-listen [::1]:8080

--http-listen ADDR (default off; any interface is allowed, but loopback or an internal address is the point) adds a plain-HTTP listener that serves the same UI and API. HTTPS stays the default and is unchanged.

• Beside HTTPS: with --tls-cert/--tls-key or --acme-host too, nmon serves HTTPS on --listen and plain HTTP on --http-listen, so the proxy may reach it either way. Alone: with no TLS configured nmon starts with only the HTTP listener (--listen is not bound) instead of refusing. SIGHUP still reloads the certificate for the HTTPS listener.
• Sign-in stays on over this listener (OIDC, break-glass, API tokens), exactly as on HTTPS. It is not dev-open mode: with no auth.oidc/auth.break_glass_file nmon refuses to start. --http-listen and --insecure-http are mutually exclusive (--insecure-http is development mode with no sign-in).
• List the proxy in auth.trusted_proxies (addresses or CIDRs, live-reloadable). Only a request from a trusted proxy has its headers honoured: X-Forwarded-For gives the client address used by the audit log and the break-glass lockout (the rightmost hop that is not itself a trusted proxy), and X-Forwarded-Proto: https marks the browser connection as HTTPS. From anyone else those headers are ignored and removed, so they cannot be spoofed. With the list empty nmon logs a WARN at startup: every client is seen as the proxy's address, so the lockout is per-proxy.
• Cookies: the session and sign-in-flow cookies are always Secure on the HTTPS listener. On the HTTP listener they are Secure only when a trusted proxy sent X-Forwarded-Proto: https; a direct plain-HTTP client gets them without Secure (a browser would drop them otherwise).
• OIDC: auth.oidc.redirect_url is configured explicitly and is never derived from the request, so set it to the public URL (https://nmon.example.net/auth/callback). CSRF uses a per-session token, not the Origin header, so a different public host or scheme needs nothing else.
• nmon uses no websockets or long-lived streams; the proxy needs no upgrade headers. Raise the proxy's body-size limit if you import large CSV files.

nginx, proxying to the plain-HTTP listener:

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name nmon.example.net;
    ssl_certificate     /etc/ssl/nmon.example.net.crt;
    ssl_certificate_key /etc/ssl/nmon.example.net.key;
    client_max_body_size 64m;

    location / {
        proxy_pass http://[::1]:8080;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 120s;
    }
}

with auth: {trusted_proxies: ["::1"]} in config.yaml. To talk TLS to nmon instead, run nmon with its normal --tls-cert/--tls-key (no --http-listen needed), use proxy_pass https://[::1]:8443; with proxy_ssl_verify off; (or proxy_ssl_trusted_certificate for a CA-signed pair), and the same proxy_set_header lines; the listener is HTTPS, so cookies are Secure. Ansible: set nmon_http_listen: "[::1]:8080" (and nmon_tls_mode: none to serve only that listener).

Sign-in

config.yaml:

auth:
  oidc:
    issuer: https://idp.example.net/realms/net
    client_id: nmon
    client_secret_name: oidc-client      # the secret itself is never in this file
    redirect_url: https://nmon.example.net/auth/callback
    groups_claim: groups
    admin_group: nmon-admins             # members are admins; every other user is a viewer
  break_glass_file: /etc/nmon/breakglass # bcrypt hash, "hash" or "user:hash", created by you

Store the client secret before the first start: printf '%s\n' "$SECRET" | nmon secret set oidc-client --data-dir D --secret-key-file K (it can also be set later with PUT /api/v1/secrets/oidc-client; OIDC stays off until the secret exists). The break-glass file holds one bcrypt hash ($2a$/$2b$), made with any bcrypt tool (the source tree's nmon-devtools bcrypt PASSWORD works); keep it readable by the nmon user only.

• There is no login page. Opening nmon without a session redirects to the IdP and back; API clients use a token. If the IdP is down, open /auth/breakglass: the browser asks for the local admin user and password (HTTP Basic), and a successful sign-in lands on the page you asked for.
• Break-glass: five wrong passwords lock the source address for 15 minutes; every use and every lock is audited and raises an alert. nmon unlock [IP|all] --data-dir D clears locks, also while nmon runs.
• API tokens: an admin mints viewer, admin or push tokens (POST /api/v1/tokens, or Settings, API tokens). Use Authorization: Bearer nma_.... A push token works for ingest only. Revoking takes effect on the next request.
• GET /api/v1/audit (admin) lists who did what.

Backup and restore

A running nmon owns the stores, so copying the data folder is not a backup.

nmon backup /backups/nmon.tar --api https://nmon.example.net --token nma_...   # key NOT included
nmon backup /backups/nmon.tar --include-key --api ... --token ...             # key included

Every run prints whether the key file is in the archive. Lose the key file and a restore has an inventory with no SNMP credentials. The nightly schedule is backup: {at: "02:30", dir: /var/backups/nmon, keep: 7} (time in the configured timezone, default dir <data-dir>/backups); a failed run raises the backup_failed alert. A backup is refused when the disk is below its 5% reserve.

Restore with nmon stopped, into a clean directory:

nmon restore /backups/nmon.tar --data-dir /var/lib/nmon-restored --secret-key-file /etc/nmon/secret.key

It verifies inventory row counts and a sample of series before it says done. Then start nmon on that directory. Restore needs the embedded stores (the normal release build).

Keys

nmon key rotate --data-dir D --secret-key-file K (nmon stopped) adds a new key version, re-encrypts every SNMP credential and secret under it, and keeps the old key in the file so older backups still restore. Remove old keys only after a release.

Configuration

Settings (admin) is point-and-click: a section list (General, Retention, Sites, Notifications, Authentication, Backup, Disk guard, L2 and OUI, Proxmox, Checks, Alerting) with a form per section, its own Save and Revert, and the applied / restart-required result under it. Unset fields show the default as a placeholder; durations take 30d, 12h, 5m. A form that refers to a secret (OIDC client secret, Proxmox token) takes the secret's name and has a "Set secret value" button that stores the value write-only through PUT /api/v1/secrets/{name}; it is never shown again. The same sections are available as typed JSON: GET /api/v1/settings/form and PUT /api/v1/settings/form (each section present in the body replaces that section; the others are untouched). Alert rules and policies are not here, they are under Alerts; alerting.rules in config.yaml is preserved by the forms. The Advanced (YAML) tab and PUT /api/v1/settings still edit the whole document, and config.yaml + SIGHUP still works; all three go through the same validation. Saving rewrites config.yaml from the parsed configuration, so comments in the file are not kept. An invalid document is rejected and the running configuration stays. Alert receivers, the outbound allow-list, trusted proxies, the backup schedule, disk thresholds and the purge period apply live; everything else is listed as restart_required until the next restart. disk: {alert_used: 0.8, admit_used: 0.9, reserve_free: 0.05}: alert at 80% used, no new series at 90%, ingest throttled / rollup paused / backups refused below 5% free.

Metric stores

The supervisor restarts a crashed store; after three restarts in ten minutes it raises store_failed and stops. nmon store restart [hot|long] --api URL --token T (or SIGHUP) resumes it.

Interface-down alerts are opt-in

Since v1.1 the bell and the group switch only mark ports: interface_down alerts for them only where an alert policy for interface_down uses the alert_down matcher (template "Bell-marked ports: interface_down"). Mark ports in one at a time (the bell on the device page's interface table), all at once (the "Alert on down" control in the interfaces card header, or POST /api/v1/devices/{slug}/interfaces/alert-down), or through a group (the group's "Alert when member ports go down" switch; PUT /api/v1/interfaces/{id} with {"alert_down": true} for one port). A port in a group with the switch on alerts for as long as its membership is in force. Changes reach the evaluator at once; opting a port out closes its open alert silently as retired.

Alert policies: nothing alerts until you add one

Alerts fire only where an alert policy binds a rule to a scope. A fresh install has no policies and records no alerts at all: no metric rule, no device_down or device_warning, no system alarm (ingest caps, late p95, rollup, disk guard, break-glass, backup). The startup log says so once ("no alert policies: nothing will alert"). Upgrading from v1.0 seeds nothing: the first cycle closes every open alert that no policy binds, silently, as retired (no "resolved" notices go out).

Manage policies under Alerts > Alert policies (admins edit, viewers read) or with GET/POST /api/v1/alert-policies and GET/PUT/DELETE /api/v1/alert-policies/{id}. GET /api/v1/alert-rules lists the rule names. POST /api/v1/alert-policies/preview counts the devices and ports a draft scope covers. Changes apply on the next cycle with no restart. Removing or disabling a policy retires its open alerts quietly.

A policy has a rule (a metric rule, device_down, device_warning, or system), an action (alert or ignore), a priority, optional severity, threshold and for overrides, and a scope. Scope fields are ANDed and any entry of a list may match: all, groups (a port is in force as a member now; a device matches if any port does), devices (slugs globs, profiles, roles, sites, kinds) and interfaces (if_types, if_name and if_alias as RE2, min_speed_mbps, max_speed_mbps, alert_down). For one entity and rule the matching policy with the highest priority decides; on a tie ignore beats alert, then the lowest name. A policy threshold beats a rule's group override, the profile threshold and the default. Interface matchers are refused on device-level rules, and a system policy takes only all: true.

Fiber and uplinks alert on any error, WiFi never alerts on errors:

{"name": "Fiber + uplinks: errors", "rule": "interface_errors", "action": "alert", "threshold": 0,
 "scope": {"interfaces": {"if_types": [6], "if_alias": "(?i)fib|uplink|core"}}}
{"name": "WiFi: no error alerts", "rule": "interface_errors", "action": "ignore", "priority": 10,
 "scope": {"interfaces": {"if_types": [71]}}}

Alert rules: built-in and custom

The shipped rules (interface_down, interface_errors, cpu_high, memory_high, disk_full, ups_on_battery, ups_runtime_low, printer_supply_low, printer_supply_empty, temp_high) are read-only built-in rules, listed with their generated query under Alerts > Alert rules. cpu_high, ups_on_battery and printer_supply_empty have Copy to custom, which clones them into an editable builder rule; the others join, filter or union series and say why they cannot be copied. Rules in config.yaml (alerting.rules) still work but are deprecated, and the startup log says so; define rules in the UI instead. A rule name is unique across all kinds, and device_down, device_warning, system and the subsystem alarm names stay reserved.

Admins add their own rules with the builder (UI or POST /api/v1/alert-rules). There is no free-form query: nmon builds it from whitelisted parts, and the generated MetricsQL is shown read-only. Test runs the draft once and shows how many entities breach now. Edits apply on the next cycle; disabling a rule or deleting it retires its open alerts silently, and deleting a rule that a policy still names is refused (409, naming the policies).

• Metric rule, "CPU over 80 % on role=server for 15 min": metric hrProcessorLoad, aggregate max (one value per device), op >, threshold 80, hold time 15m; then a policy for this rule with devices.roles: [server]. Transforms are value, rate (counters, per second) and percent_of (metrics with an of, such as hrStorageUsed); the window defaults to 5 min (at least twice the poll interval). Push-ingested metrics are device-level.
• Inventory rule, "Uplink without description": interface fields if_alias empty AND if_speed_mbps >= 10000.
• Inventory rule, "Device not seen for 1 h": device field last_seen_age older_than 1h.

A custom rule, like a built-in, alerts nowhere until an alert policy binds it: the rule defines what is measured and its default threshold and hold time, and the policy defines where it applies and may override severity, threshold and for. Several policies can bind one rule to different scopes.

Webhook templates

Each webhook receiver (Settings > Notifications) has a Go text/template that renders the request body. It must produce one valid JSON document (max 256 KB); it is sent as application/json. Empty means the default, {{json .}}, the whole notification. SMTP receivers use the same editor for Subject: ..., a blank line and the body.

Fields (a template sees nothing else): .IncidentID, .AlertID, .Rule, .State (firing, resolved, flapping, stale, suppressed_summary), .Severity (critical, warning, info), .Device.Slug, .Device.Name, .Interface.ID, .Interface.Name, .Interface.Alias (empty for device-level alerts), .Summary, .Description, .FiredAt, .ClearedAt (nil until resolved), .Labels, .Parents, .SuppressedChildren, .URL (link to the device in nmon).

Functions: json, upper, lower, join, timefmt ({{timefmt .FiredAt "2006-01-02 15:04:05 MST"}}, UTC), plus the standard printf, eq, or, if.

The {{json}} rule: wrap every value you put inside a JSON string or value in {{json ...}} ("text": {{json .Summary}}, never "text": "{{.Summary}}"). json adds the quotes and escapes quotes, newlines and control characters, so an alert text can never break the document. Secret-looking keys and key=value pairs are redacted from the rendered payload before it is stored or sent.

Presets: "Start from preset" in the receiver form fills the template (it asks before replacing a non-empty one). Built in, also at GET /api/v1/notify/presets:

Preset	For
Generic JSON	the default; the whole notification
Microsoft Teams (Workflows)	Adaptive Card 1.4 for the Workflows trigger "When a Teams webhook request is received"
Slack	incoming webhook: text fallback and a coloured attachment with blocks
Discord	webhook with one embed, coloured by state and severity

Teams: Microsoft is retiring the Office 365 connectors ("Incoming Webhook" in a channel's Connectors). Create a Workflow from the template "Post to a channel when a webhook request is received" and paste its URL. The preset is an Adaptive Card message and will not work with a legacy connector URL.

Preview and test: the editor shows the server-rendered payload for a sample notification in any state as you type, or the template or JSON error. "Send test" delivers one sample immediately (no retry queue; nothing stored except an notify.test audit row) and shows the HTTP status, a response excerpt (4 KB, redacted), the duration and any error. Blank header values and a blank SMTP password use the stored receiver of the same name, so you can test a saved receiver without retyping credentials. Admin only; at most two tests run at once. POST /api/v1/notify/preview and POST /api/v1/notify/test take {"receiver": {...form shape...}, "state": "firing"}.

Outbound allow-list: test sends and real deliveries go through the same gate. A webhook to a private, loopback or link-local address is refused unless it matches alerting.outbound_allow (Settings > Notifications), and the test result says so.

Retired devices

POST /api/v1/devices/{slug}/purge (admin) deletes a retired device's series from both stores once it has been retired for retired_purge_after (default 1y).

CSV import

Devices (name,address,site,role,credential,parents,...) and port descriptions (device,ifname,alias) from Settings or POST /api/v1/devices/import. Port descriptions are stored in the operator-owned description field, which discovery never overwrites. A dependency cycle is rejected with its path; nothing is written unless the whole import is valid.

Find host: MAC, ARP and FDB lookup

The L2 table job walks the bridge forwarding tables, VLAN names and ARP / IPv6 neighbour tables of every SNMP device every l2.interval and keeps them with first and last seen in the inventory, so the Find host page (and the header search, for a MAC, IP, CIDR or vendor:name) answers "which switch port is this host plugged in at, in which VLAN, and when did we last see it", with the uplinks that also saw it collapsed underneath. A MAC is seen on every switch it passes through, so the answer is the edge port: one that carries at most l2.edge_max_macs MACs, has no LLDP neighbour and is not a LAG.

l2:
  interval: 15m        # default 15m, minimum 5m
  keep: 90d            # history kept by last_seen, pruned daily
  edge_max_macs: 8     # a port with more MACs is an uplink
  concurrency: 4       # devices walked at once
  device_timeout: 2m   # a slower device is skipped for the cycle
oui:
  fetch: true          # IEEE vendor registry; false never touches the network
  refresh: 7d

All of these apply from the next cycle after a reload (Settings save or SIGHUP) with no restart. The job never runs on the poll path and yields to the poller the way the rollup job does. A device that does not answer in time is skipped for that cycle and named on the Poller page (l2 status and an l2_devices_skipped note); it never delays the other devices. A Cisco IOS switch keeps one bridge table per VLAN: the cisco-ios profile sets l2.per_vlan_context: true, which lists the VLANs from CISCO-VTP-MIB and walks each with community <community>@<vlan> (v3: context vlan-<vlan>). Other devices use Q-BRIDGE-MIB, then BRIDGE-MIB (VLAN unknown).

API: GET /api/v1/l2/search?q=, GET /api/v1/devices/{slug}/{l2,fdb,arp,l2/ports}, GET /api/v1/interfaces/{id}/macs. On a device, the L2 tab (MAC table with a VLAN filter, ARP / ND, VLAN list) appears once the job has read it, and the Ports tab shows each port's MAC count linking to that port's MACs.

MAC vendors (optional IEEE registry)

If a registry is loaded, each MAC shows its vendor, a locally administered address (typical of randomised phone and VM MACs) shows a "private" badge, and vendor:ubiquiti lists the current MACs of a vendor. nmon works fully without it: the vendor is simply empty, and the Poller page notes oui_unavailable (not an alert).

The registry is the IEEE's public data: MA-L (https://standards-oui.ieee.org/oui/oui.csv), MA-M (.../oui28/mam.csv) and MA-S (.../oui36/oui36.csv), downloaded weekly through the outbound allow-list (standards-oui.ieee.org must be allowed when an allow-list is configured) with timeouts, a size cap and a row-count sanity check, so a truncated or wrong download keeps the previous file. The compact result is <data-dir>/oui/registry.gz (checksummed, written atomically, included in nmon backup). Where nmon may not fetch, set oui.fetch: false and load the CSVs by hand: nmon oui import <dir-or-file> --data-dir DIR (a running nmon notices within an hour). The CSVs are not part of this repository.

BGP

Router profiles collect BGP session state, uptime, accepted prefixes per address family and update rates every 5 minutes (see Collection coverage for which profiles and MIBs). The BGP page lists every session on every device, the ones that are down first, filterable by state, AS, family and text; a device's Routing tab shows its sessions with click-to-graph on prefixes and updates.

The built-in rule bgp_peer_down (critical, state not established for 2 minutes) fires only where a policy binds it, like every rule: use the template "Routers: BGP peer down" under Alerts > Alert policies. Its notification names the peer and, from discovery, the peer's AS and description. A prefix-drop alert is a custom rule (v1.2); an example will be added here once custom rules are in.

Poll interval

SNMP interface counters are polled every 300 s by default. An admin can set a different interval for one device (Device settings, "Poll interval", 30 to 3600 s; blank returns it to the default) or through PUT /api/v1/devices/{slug} with poll_interval_s. The poller picks it up at once, no restart. Graph rate windows, reachability fallbacks and the rollup follow the device's own interval (see docs/DEVIATIONS.md row 49). ICMP is still probed every 15 s.

Port totals. A port graph (GET /api/v1/graph) also returns totals: bytes in and out over the requested range, the average-rate series integrated over each point's own step (hourly and raw data stitched without double counting). covered_seconds against range_seconds says how much of the range had data; the UI marks a total with * when coverage is below 99.5%. Unreliable (32-bit fast link) ports get none. The port grid shows total beside avg and p95; the full graph has a Total column. Per port only.

Port graph parity with LibreNMS. The port summary and the full graph show, per direction, Now / Avg / Max / 95th / Total. Max is the highest plotted point (with max=true the finer "peak within a step" is a separate Peak column). The aggregate 95th (p95.aggregate_bps) is the 95th percentile of in+out per 5-minute slot, counted over slots where both directions have a sample; it is live from the hot store (also next to a materialised monthly figure) and is absent when only one direction was asked for or the range has no raw data. When the port speed is known, the full legend prints each rate's share of that speed, and a toggle plots the y axis as percent of speed (hidden for unknown or zero speed). "Compare previous period" (device port row and graph view) fetches the same-length range just before the current one with the ordinary from/to of GET /api/v1/graph, shifts it forward onto the current axis and draws it faded and dashed, with its own avg / max / 95th / total rows in the legend; it is off while a p95 month is selected.

The graph-type selector offers Bits, Errors and Discards. Errors and discards come from GET /api/v1/graph/counters?iface=&kind=errors|discards, the per-second rate of ifInErrors/ifOutErrors or ifInDiscards/ifOutDiscards with the same step and rate-window rules as bits. These counters have no long-store rollup, so the endpoint answers from the hot store only; a range that starts before hot retention is cut to its start (clamped: true, and the UI says so). Packets (unicast pps) and Broadcast/Multicast appear disabled: the generic profile does not poll ifHC*UcastPkts, ifHC*MulticastPkts or ifHC*BroadcastPkts. Adding them means polling three more columns on every port of every device and regenerating every fixture, so it is a separate decision (DEVIATIONS 56).

The default itself is poller.interval in config.yaml (Settings, General, "Poll interval"; 30s to 1h, default 5m). A change applies without a restart to the poller, reachability, the rollup and the API. The dev flag nmon serve --poll-interval, when given, wins over the setting.

Testing and debugging a device's polling

Three admin tools answer "why is this device not polling right", from the device page (Actions menu) or the API. None of them shows or logs a credential: the trace names the SNMP version, never the community or the v3 secrets.

• Poll now (POST /api/v1/devices/{slug}/poll-now, 202) runs the device's primary group right away, outside its schedule. The result lands in the normal places (graphs, reachability). It answers 409 when the device is retired, disabled, has no SNMP credential, or is not loaded by the poller yet.
• Test poll (POST /api/v1/devices/{slug}/poll-test, optional ?interval_seconds=300 for one interval group) runs one synchronous cycle of the device's real poll plan: the stored credential, the resolved profile, the address and port, and the same fetch code the poller uses (the varbind count starts at the default and halves on tooBig or a lost response, exactly as in polling, but is forgotten afterwards). It writes no metrics and changes no poller state, so it cannot cause a reboot event, a backoff or a down alert. The answer lists each request (the session, number of varbinds, first OID, start and duration in ms, outcome ok, timeout, too_big, snmp_error, dial_error or error, and whether it was a retry in smaller pieces) and every requested OID with the series it feeds, its label (interface ULID or component), the SNMP type, the raw value and what the poller would store, or why not: no_such_object, no_such_instance, end_of_mib, skipped (text, type mismatch, not available), not_fetched (its request failed) or oid_mismatch. The duration_ms of a request includes the SNMP library's own retries; they are not listed separately. Limits: one test per device at a time (409), four at once overall (429), 60 s at most, 5,000 OID rows at most (truncated: true; failures and misses are kept first). Every test is audited as device.poll_test.
• SNMP debug (PUT /api/v1/devices/{slug}/snmp-debug with {"minutes": N}; 0 turns it off, 60 at most; GET reads it) records every scheduled cycle of one device while it is on: the same request and OID detail as a test poll, for the last 20 cycles (500 OID rows each, failures first), newest first. It also logs one INFO line per cycle (snmp debug cycle, with the counts, duration and error). It switches off by itself when the minutes are up. Recordings are kept in memory only and are cleared on restart; starting a new recording drops the previous one, and switching off keeps what was captured so you can still read it. Turning it on or off is audited as device.snmp_debug. Use Poll now while it is on to get a cycle at once.

From a shell: nmon device poll <slug> --api URL --token TOKEN queues a poll now; add --test for the traced test poll (--interval SECONDS for one group, --json for the full trace).

Reading a trace: a request that times out with every OID not_fetched is a reachability, community or ACL problem (check the target address, the SNMP version and the device's allowed managers). A too_big request followed by retry requests means the device cannot answer 40 varbinds at once; the poller adapts, so this is only a note. Many no_such_instance rows for interface OIDs mean the stored interfaces no longer match the device: run Rediscover.