nmon/docs/operations.md

Operating nmon

How to run, secure, back up and maintain a production nmon. Design background: docs/spec/spec.md ("Auth", "Operations").

Starting

nmon serve --data-dir /var/lib/nmon --secret-key-file /etc/nmon/secret.key \
           --tls-cert /etc/nmon/tls.crt --tls-key /etc/nmon/tls.key

• HTTPS is the default. Pass a certificate pair (--tls-cert, --tls-key; SIGHUP reloads them without dropping connections) or Let's Encrypt (--acme-host nmon.example.net repeatable, --acme-email you@example.net; nmon answers the http-01 challenge on --acme-listen, default :80, certificates are cached in <data-dir>/acme). Without either, serve refuses to start and says so. --insecure-http is for development: plain HTTP, cookies without Secure.
• A sign-in method is required over TLS (auth.oidc and/or auth.break_glass_file in config.yaml). With --insecure-http and neither configured nmon runs in dev-open mode (everyone is admin dev; a banner says so); never expose that port.
• Behind a reverse proxy, list it in auth.trusted_proxies; X-Forwarded-* is ignored from anyone else.

Sign-in

config.yaml:

auth:
  oidc:
    issuer: https://idp.example.net/realms/net
    client_id: nmon
    client_secret_name: oidc-client      # the secret itself is never in this file
    redirect_url: https://nmon.example.net/auth/callback
    groups_claim: groups
    admin_group: nmon-admins             # members are admins; every other user is a viewer
  break_glass_file: /etc/nmon/breakglass # bcrypt hash, "hash" or "user:hash", created by you

Store the client secret before the first start: printf '%s\n' "$SECRET" | nmon secret set oidc-client --data-dir D --secret-key-file K (it can also be set later with PUT /api/v1/secrets/oidc-client; OIDC stays off until the secret exists). The break-glass file holds one bcrypt hash ($2a$/$2b$), made with any bcrypt tool (the source tree's nmon-devtools bcrypt PASSWORD works); keep it readable by the nmon user only.

• There is no login page. Opening nmon without a session redirects to the IdP and back; API clients use a token. If the IdP is down, open /auth/breakglass: the browser asks for the local admin user and password (HTTP Basic), and a successful sign-in lands on the page you asked for.
• Break-glass: five wrong passwords lock the source address for 15 minutes; every use and every lock is audited and raises an alert. nmon unlock [IP|all] --data-dir D clears locks, also while nmon runs.
• API tokens: an admin mints viewer, admin or push tokens (POST /api/v1/tokens, or Settings, API tokens). Use Authorization: Bearer nma_.... A push token works for ingest only. Revoking takes effect on the next request.
• GET /api/v1/audit (admin) lists who did what.

Backup and restore

A running nmon owns the stores, so copying the data folder is not a backup.

nmon backup /backups/nmon.tar --api https://nmon.example.net --token nma_...   # key NOT included
nmon backup /backups/nmon.tar --include-key --api ... --token ...             # key included

Every run prints whether the key file is in the archive. Lose the key file and a restore has an inventory with no SNMP credentials. The nightly schedule is backup: {at: "02:30", dir: /var/backups/nmon, keep: 7} (time in the configured timezone, default dir <data-dir>/backups); a failed run raises the backup_failed alert. A backup is refused when the disk is below its 5% reserve.

Restore with nmon stopped, into a clean directory:

nmon restore /backups/nmon.tar --data-dir /var/lib/nmon-restored --secret-key-file /etc/nmon/secret.key

It verifies inventory row counts and a sample of series before it says done. Then start nmon on that directory. Restore needs the embedded stores (the normal release build).

Keys

nmon key rotate --data-dir D --secret-key-file K (nmon stopped) adds a new key version, re-encrypts every SNMP credential and secret under it, and keeps the old key in the file so older backups still restore. Remove old keys only after a release.

Configuration

Edit in Settings (admin) or config.yaml + SIGHUP. An invalid document is rejected and the running configuration stays. Alert receivers, the outbound allow-list, trusted proxies, the backup schedule, disk thresholds and the purge period apply live; everything else is listed as restart_required until the next restart. disk: {alert_used: 0.8, admit_used: 0.9, reserve_free: 0.05}: alert at 80% used, no new series at 90%, ingest throttled / rollup paused / backups refused below 5% free.

Metric stores

The supervisor restarts a crashed store; after three restarts in ten minutes it raises store_failed and stops. nmon store restart [hot|long] --api URL --token T (or SIGHUP) resumes it.

Retired devices

POST /api/v1/devices/{slug}/purge (admin) deletes a retired device's series from both stores once it has been retired for retired_purge_after (default 1y).

CSV import

Devices (name,address,site,role,credential,parents,...) and port descriptions (device,ifname,alias) from Settings or POST /api/v1/devices/import. Port descriptions are stored in the operator-owned description field, which discovery never overwrites. A dependency cycle is rejected with its path; nothing is written unless the whole import is valid.