Compare commits
19 Commits
m6
...
worktree-a
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
61b5341774 | ||
|
|
7173360346 | ||
|
|
c4d14a67d1 | ||
|
|
06a3c85c69 | ||
|
|
ddc8863a67 | ||
| c4f5ee916b | |||
| 992f09b97a | |||
| d92c6e1e08 | |||
| 9b4ab7ccbb | |||
| 457191accd | |||
| 61274c4f55 | |||
| fe53998aa6 | |||
| e25143881c | |||
| ad220eae58 | |||
| 345daaa773 | |||
| bf6dbd4923 | |||
| d97dd1d44a | |||
| 28f5444860 | |||
| 6ae7729107 |
3
.gitignore
vendored
@@ -11,3 +11,6 @@ coverage.out
|
||||
/nmon-librenms-import
|
||||
/internal/webui/dist/*
|
||||
!/internal/webui/dist/.gitkeep
|
||||
deploy/ansible/.mibs.tar.gz
|
||||
deploy/ansible/inventory.ini
|
||||
deploy/ansible/vars.yml
|
||||
|
||||
@@ -100,6 +100,7 @@ linters:
|
||||
- "**/internal/checks/**"
|
||||
- "**/internal/rollup/**"
|
||||
- "**/internal/alert/**"
|
||||
- "**/internal/l2/**"
|
||||
deny:
|
||||
- pkg: github.com/v10networks/nmon/internal/poller
|
||||
desc: collection paths share code via a lower package, not each other
|
||||
@@ -107,6 +108,8 @@ linters:
|
||||
desc: collection paths share code via a lower package, not each other
|
||||
- pkg: github.com/v10networks/nmon/internal/traps
|
||||
desc: collection paths share code via a lower package, not each other
|
||||
- pkg: github.com/v10networks/nmon/internal/l2
|
||||
desc: collection paths share code via a lower package, not each other
|
||||
devtools-not-in-nmon:
|
||||
# The product binary never links dev/test machinery. `nmon profile record` needs only the
|
||||
# .snmprec format, which lives in internal/snmprec.
|
||||
|
||||
120
CLAUDE.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# nmon: agent onboarding
|
||||
|
||||
nmon is a lightweight LibreNMS replacement for ISPs and IT teams. It is one static Go binary that polls devices
|
||||
(SNMP, ICMP, traps, push ingest, Proxmox, checks), stores metrics in two embedded VictoriaMetrics child processes
|
||||
(hot: raw 30 d; long: rollups up to 10 y), keeps inventory in SQLite or Postgres via bun, alerts with dependency
|
||||
suppression, and serves a React UI and JSON API from the same binary. Module: `github.com/v10networks/nmon`, Go 1.27.
|
||||
|
||||
## Read first (source of truth, in this order)
|
||||
|
||||
1. `docs/spec/spec.md` and `docs/spec/build-detail.md`: the owner's spec. The contract.
|
||||
2. `docs/DEVIATIONS.md`: every approved departure from the spec, with the reason. Check it before "fixing" something to match the spec.
|
||||
3. `docs/MAINTAINABILITY.md`: **hard rule**, enforced by lint. Read it before writing code.
|
||||
4. `docs/milestones/`: per-milestone scope, decisions and gate logs (M1–M6, `v1.0.1-device-tabs.md`), plus deferred items and post-v1 additions in `README.md`.
|
||||
5. `docs/operations.md` (running nmon), `docs/dev.md` (UI embedding, openapi check, devtools), `docs/coverage.md` (what each profile collects vs. the spec).
|
||||
|
||||
## Non-negotiables
|
||||
|
||||
- **`make lint` must be clean**, Go (golangci-lint v2.14.0 via `.golangci.yml`, plus `scripts/lint-filesize.sh`) and web (ESLint, `--max-warnings 0`).
|
||||
- Size limits: functions ≤ 80 lines, cyclomatic ≤ 15, cognitive ≤ 20; Go files ≤ 600 lines, web files ≤ 400.
|
||||
- `//nolint` only with the linter's name and a reason.
|
||||
- **Layering (depguard):**
|
||||
- No SQL or bun outside `internal/store`.
|
||||
- Only `cmd/*` imports `internal/app`, and only `internal/app` imports `internal/api`.
|
||||
- Collection paths don't import each other: `poller`, `discovery`, `traps`, `ingest`, `checks`, `rollup`, `alert`, `l2`. Shared code goes in a lower package (e.g. `snmptarget`, `longq`, `snmprec`).
|
||||
- `cmd/nmon` must not link dev/test machinery (`simagent`, `fixturegen`, `fixtures`, `librenms`, `testutil`); `TestNmonDoesNotLinkDevMachinery` enforces this.
|
||||
- **Time** comes only from `internal/clock` (forbidigo bans `time.Now`/`Sleep`/`After`… outside `clock`, `cmd/` and tests). Tests use `clock.NewFake`.
|
||||
- **Frozen contracts:**
|
||||
- Metric names and labels: SNMP series carry only `device=<slug>` and `iface=<ULID>`/`component`. Long-store rollups use `v` (highest wins; read via `internal/longq.LatestVersion`).
|
||||
- API routes: `scripts/openapi-check.sh` fails on incompatible changes vs the last tag. Additive changes only.
|
||||
- **Migrations are frozen:** never edit an applied migration. Add `internal/store/<YYYYMMDDNNNNNN>_<name>.go` with the next number, using frozen struct copies (see existing files). Add new store methods as a sub-interface in a new file, plus one embed line in `store.go`.
|
||||
- **API types in the UI are generated, never hand-written:**
|
||||
1. `go run ./cmd/nmon openapi > openapi.json`
|
||||
2. `cd web && npm run gen:api` (writes `web/src/api/schema.d.ts`)
|
||||
3. Use `client.GET/POST`.
|
||||
- **Only `web/src/components/TrafficGraph.tsx` imports uPlot.** Single-series graphs use its `format` prop.
|
||||
- **IPv6 first.** Everything works on v6 and v4; tests default to `::1`.
|
||||
- **Every port is polled.** Only a profile's `exclude_ifname` drops one. 32-bit counters on fast links are flagged `unreliable` and excluded from sums, p95 and rollups, never graphed as trustworthy.
|
||||
- **Rollups store rates, never raw counters.** A grep test enforces this.
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| `cmd/nmon` | the product binary (`serve`, `migrate`, `discover`, `profile test/record/new`, `backup`, `restore`, `key rotate`, `unlock`, `store restart`, `secret set`, `rollup repair`, `openapi`) |
|
||||
| `cmd/nmon-devtools` | dev-only binary (backfill, seed-demo, mocks for OIDC/Proxmox, webhook sink); never shipped |
|
||||
| `cmd/nmon-librenms-import` | offline tool: LibreNMS definitions → draft profiles |
|
||||
| `internal/app` | wiring: one `start*` per subsystem, stops in reverse order, SIGHUP reload |
|
||||
| `internal/api` | huma v2 handlers, RFC 9457 errors, paging (default 100, max 1,000); slugs and ULIDs only, never numeric ids or secrets |
|
||||
| `internal/auth` | OIDC (no login UI: browsers 302 to the IdP), sessions + CSRF, API tokens, break-glass (HTTP Basic on `/auth/breakglass`), route policy |
|
||||
| `internal/store` | bun models, migrations, every SQL query |
|
||||
| `internal/vmsup` | embedded VictoriaMetrics: extract, checksum, supervise, write queue, querier |
|
||||
| `internal/{poller,discovery,traps,ingest,checks,proxmox,rollup,p95,groups,reach,deps,alert,notify}` | collection, storage jobs, reachability, alerting |
|
||||
| `internal/{l2,hostaddr,oui}` | the MAC/ARP/FDB table job, MAC and IP parsing, the optional IEEE vendor registry |
|
||||
| `internal/profile` + `profiles/*.yaml` | device profiles (YAML, numeric OIDs, `category`/`label`/`of`/`display` per metric) |
|
||||
| `internal/mib` | SMI parser; MIBs are a sidecar in `mibs/`, never embedded |
|
||||
| `internal/{simagent,snmprec,fixturegen,fixtures,profiletest}` | the SNMP simulator, `.snmprec` format, fixture generator, expectation harness |
|
||||
| `web/` | React 19 + TS + Vite + Tailwind/shadcn + TanStack Query/Router; built into `internal/webui` |
|
||||
| `testdata/snmprec/` | per-profile and behaviour fixtures, each with `*.expect.yaml` and optional `*.agent.yaml` |
|
||||
| `deploy/ansible/` | playbook (systemd, TLS modes, break-glass/OIDC, MIB sidecar) |
|
||||
|
||||
## Commands
|
||||
|
||||
```
|
||||
make fetch-vm # once: download and checksum the VictoriaMetrics binaries (needed for vmembed and integration)
|
||||
make build # web build + static nmon with the UI and VictoriaMetrics embedded
|
||||
make devtools importer # the dev and import binaries
|
||||
make lint # Go + web; must be clean
|
||||
go test -race ./... # unit tests (fast)
|
||||
go test -race -tags 'integration vmembed' ./... # real VictoriaMetrics children
|
||||
./scripts/test-pg.sh # store suite on a throwaway Postgres 16 (Docker; safe to run concurrently)
|
||||
NMON_LIBRENMS_DIR=/home/svcuser/src/librenms go test ./internal/profile/... ./internal/fixtures/... # real-device recordings
|
||||
go run ./testdata/gen # regenerate fixtures (a test fails on drift)
|
||||
scripts/openapi-check.sh # API compatibility vs the last tag
|
||||
make gate-m1 … gate-m6 # end-to-end gates (slow, ~10 min each)
|
||||
cd web && npm run lint && npm run test && npm run build && npm run e2e # e2e = Playwright on a mocked API
|
||||
```
|
||||
|
||||
## Adding things
|
||||
|
||||
- **A device profile:**
|
||||
- Add `profiles/<name>.yaml` that `extends: generic`, with `match` (sysObjectID prefix and/or sysDescr regex) and `priority`. Higher wins; watch overlaps such as Linux-based sysDescrs and shared enterprise numbers.
|
||||
- Use numeric OIDs, verified against `mibs/`, with a provenance comment. Every pollable non-interface metric needs a `category`; the orphan test enforces it.
|
||||
- Then run `go run ./testdata/gen` for the fixture and expect file, add matcher tests, and check against LibreNMS recordings in `/home/svcuser/src/librenms/tests/snmpsim`.
|
||||
- That repo is GPL and read-only reference material. Never copy code or YAML from it.
|
||||
- **An API route:**
|
||||
- Put the handler in a new file in `internal/api`. State-changing routes are admin by default; declare read-only POSTs as viewer on the auth policy (`internal/app/authwire.go`). `TestViewerCannotMutate` walks every route automatically.
|
||||
- Regenerate `openapi.json` and the web schema, then run `scripts/openapi-check.sh`.
|
||||
- **A UI page:**
|
||||
- Use the generated client in `web/src/api/*Hooks.ts`, `NavRow` for clickable rows, and `RequireAdmin` or role checks for mutating controls.
|
||||
- Mock handlers and fixtures live in `web/e2e/mock/` for the Playwright mock suite.
|
||||
- **A deviation from the spec:** add a row to `docs/DEVIATIONS.md` explaining what and why.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Concurrent agents:**
|
||||
- Stage only your own paths, never `git add -A`.
|
||||
- `openapi.json` and `web/src/api/schema.d.ts` are shared, so regenerate them just before committing.
|
||||
- Two agents writing a migration must agree on numbers first.
|
||||
- **Screenshots and Playwright:**
|
||||
- Full Playwright runs rewrite the committed screenshots under `docs/screenshots/*`. Revert the ones you didn't mean to change (`git checkout -- docs/screenshots/<dir>`).
|
||||
- Use a non-default `E2E_PORT` if a `vite preview` may already be running, and kill any preview server you start.
|
||||
- **Gate scripts:**
|
||||
- Gate scripts run `make lint` first; a lint finding anywhere fails every gate.
|
||||
- `--insecure-http` with no sign-in configured runs **dev-open mode** (everyone is admin). The gates rely on it; never ship it.
|
||||
- **Ingest:** reserved metric prefixes (`nmon_`, `if`, `hr`, `sys` + uppercase/digit) and the labels `device`/`iface`/`v`/`push` are stripped or refused.
|
||||
- **ICMP:** `nmon serve --icmp off`, or an unsupported unprivileged ICMP socket, makes SNMP alone decide reachability, with an `icmp_unsupported` warning.
|
||||
|
||||
## Working style expected here
|
||||
|
||||
- Commit messages: imperative summary, a body explaining why, and the trailers the session asks for.
|
||||
- Don't claim something passed unless you ran it. Report skipped checks explicitly.
|
||||
- Owner preferences in force:
|
||||
- Maintainability is a hard rule.
|
||||
- MIBs are a sidecar.
|
||||
- The LibreNMS importer and devtools are separate binaries.
|
||||
- No login UI (OIDC redirect only).
|
||||
- Interface-down alerts are opt-in (per port or via groups).
|
||||
- The 2,000-device load test is skipped (approved deviation).
|
||||
- Web push is v2.
|
||||
- BMC profiles are deferred.
|
||||
47
cmd/nmon/oui.go
Normal file
@@ -0,0 +1,47 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/v10networks/nmon/internal/config"
|
||||
"github.com/v10networks/nmon/internal/oui"
|
||||
)
|
||||
|
||||
func init() {
|
||||
register("oui", "load the IEEE vendor registry from disk: nmon oui import DIR|FILE", ouiCmd)
|
||||
}
|
||||
|
||||
const ouiUsage = "usage: nmon oui import <dir-or-file> --data-dir DIR\n\n" +
|
||||
"Loads the IEEE MAC registry CSVs (oui.csv for MA-L, mam.csv for MA-M, oui36.csv for MA-S, downloaded from\n" +
|
||||
"https://standards-oui.ieee.org/) into <data-dir>/oui/registry.gz, where a running nmon picks it up within an hour.\n" +
|
||||
"Use it where nmon may not fetch (oui.fetch: false). The CSVs are public IEEE data and are not kept after import."
|
||||
|
||||
// ouiCmd is "nmon oui import": the offline way to give nmon its vendor registry.
|
||||
func ouiCmd(_ context.Context, args []string) error {
|
||||
if len(args) == 0 || args[0] != "import" {
|
||||
return fmt.Errorf("%s", ouiUsage)
|
||||
}
|
||||
var f config.Flags
|
||||
fs := newFlagSet("oui import", "oui import <dir-or-file> --data-dir DIR")
|
||||
fs.StringVar(&f.DataDir, "data-dir", "", "nmon's data directory")
|
||||
pos, err := parseInterspersed(fs, args[1:])
|
||||
if err != nil {
|
||||
return flagErr(err)
|
||||
}
|
||||
if len(pos) != 1 || f.DataDir == "" {
|
||||
return fmt.Errorf("%s", ouiUsage)
|
||||
}
|
||||
// Only the MA-L registry is required: it holds nearly every vendor. MA-M and MA-S refine it when present.
|
||||
reg, p, err := oui.Import(pos[0], oui.Minimums{MAL: oui.MinMAL})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
dst := filepath.Join(config.OUIDir(f.DataDir), oui.FileName)
|
||||
if err := oui.Write(dst, reg); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("oui import: %d MA-L, %d MA-M and %d MA-S assignments written to %s\n", p.MAL, p.MAM, p.MAS, dst)
|
||||
return nil
|
||||
}
|
||||
53
cmd/nmon/oui_test.go
Normal file
@@ -0,0 +1,53 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/v10networks/nmon/internal/config"
|
||||
"github.com/v10networks/nmon/internal/oui"
|
||||
)
|
||||
|
||||
func TestOuiImportWritesTheRegistryFileANmonLoads(t *testing.T) {
|
||||
src, data := t.TempDir(), t.TempDir()
|
||||
var b strings.Builder
|
||||
b.WriteString("Registry,Assignment,Organization Name,Organization Address\n")
|
||||
for i := 0; i < oui.MinMAL; i++ {
|
||||
fmt.Fprintf(&b, "MA-L,%06X,Vendor %d,Somewhere\n", 0x100000+i, i)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(src, "oui.csv"), []byte(b.String()), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := ouiCmd(context.Background(), []string{"import", src, "--data-dir", data}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reg, err := oui.Load(filepath.Join(config.OUIDir(data), oui.FileName))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v, ok := reg.Lookup("10:00:05:00:00:01"); !ok || v != "Vendor 5" {
|
||||
t.Errorf("lookup = %q %v", v, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOuiImportRefusesATruncatedDirectoryAndUsageErrors(t *testing.T) {
|
||||
src, data := t.TempDir(), t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(src, "oui.csv"), []byte("Registry,Assignment,Organization Name,Organization Address\nMA-L,AABBCC,One,x\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := ouiCmd(context.Background(), []string{"import", src, "--data-dir", data}); err == nil {
|
||||
t.Error("a registry with one row was imported")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(config.OUIDir(data), oui.FileName)); err == nil {
|
||||
t.Error("a refused import wrote a file")
|
||||
}
|
||||
for _, args := range [][]string{nil, {"fetch"}, {"import"}, {"import", src}} {
|
||||
if err := ouiCmd(context.Background(), args); err == nil {
|
||||
t.Errorf("args %v accepted", args)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -276,7 +276,7 @@ func renderDraft(spec draftSpec, m *mib.Module, metrics []draftMetric) string {
|
||||
var b strings.Builder
|
||||
name := "draft-" + strings.ToLower(spec.Module)
|
||||
fmt.Fprintf(&b, "# DRAFT compiled by `nmon profile new` from %s (%s).\n", spec.Module, filepath.Base(m.File))
|
||||
fmt.Fprintf(&b, "# Save as %s.yaml. Replace every TODO (unit, interval, index_label) and add a match: section.\n", name)
|
||||
fmt.Fprintf(&b, "# Save as %s.yaml. Replace every TODO (unit, interval, category, index_label) and add a match: section.\n", name)
|
||||
b.WriteString("# The loader rejects TODO on purpose, so an unreviewed draft can never be polled.\n")
|
||||
fmt.Fprintf(&b, "name: %s\nextends: generic\nprovenance: %q\nmetrics:\n", name,
|
||||
"drafted by nmon profile new from "+spec.Module+"; review before use")
|
||||
@@ -462,7 +462,8 @@ func writeDraftMetric(b *strings.Builder, dm draftMetric) {
|
||||
}
|
||||
fmt.Fprintf(b, " # %s\n", d)
|
||||
}
|
||||
fmt.Fprintf(b, " - name: %s\n oid: %s\n type: %s\n unit: %s\n interval: TODO\n", o.Name, oid, dm.typ, dm.unit)
|
||||
fmt.Fprintf(b, " - name: %s\n oid: %s\n type: %s\n unit: %s\n", o.Name, oid, dm.typ, dm.unit)
|
||||
b.WriteString(" interval: TODO\n category: TODO\n")
|
||||
if dm.scale != 0 {
|
||||
fmt.Fprintf(b, " scale: %s\n", strconv.FormatFloat(dm.scale, 'f', -1, 64))
|
||||
}
|
||||
|
||||
@@ -27,6 +27,7 @@ func bundledMIBs(t *testing.T) string {
|
||||
}
|
||||
|
||||
func fillTODOs(s string) string {
|
||||
s = strings.ReplaceAll(s, "category: TODO", "category: health")
|
||||
s = strings.ReplaceAll(s, "interval: TODO", "interval: 5m")
|
||||
return strings.ReplaceAll(s, "unit: TODO", "unit: count")
|
||||
}
|
||||
|
||||
48
deploy/ansible/README.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Deploying nmon with Ansible
|
||||
|
||||
Installs nmon as a systemd service on one Linux host (amd64 or arm64). Tested against Debian 13.
|
||||
|
||||
## Quick start
|
||||
|
||||
cd deploy/ansible
|
||||
cp inventory.example.ini inventory.ini # your host
|
||||
cp vars.example.yml vars.yml # set nmon_breakglass_password at least
|
||||
ansible-playbook site.yml -e @vars.yml
|
||||
|
||||
The first play builds the release binary on your machine (`make build build-arm64`: needs Go, Node and network).
|
||||
Pass `-e nmon_build=false` to reuse an existing `./nmon`. The second play installs it on the hosts in group `nmon`:
|
||||
|
||||
- `/usr/local/bin/nmon`, user `nmon`, data in `/var/lib/nmon`, key and TLS files in `/etc/nmon`, backups in `/var/backups/nmon`
|
||||
- the MIB sidecar in `/usr/share/nmon/mibs`, used only by `nmon profile new --mib`
|
||||
- optionally the IEEE vendor registry CSVs in `/usr/share/nmon/oui`, loaded with `nmon oui import` (`nmon_oui_download`, `nmon_oui_csv_dir`; by default nmon fetches the registry itself)
|
||||
- `config.yaml` rendered from your vars, and a hardened systemd unit (`systemctl reload nmon` sends SIGHUP to reload config and certificates)
|
||||
- `net.ipv4.ping_group_range` opened so nmon can ping without root
|
||||
|
||||
Then open `https://<host>:8443/`. Without OIDC you are sent to `/auth/breakglass` (a browser password prompt; user `admin` by default).
|
||||
|
||||
## TLS (`nmon_tls_mode`)
|
||||
|
||||
| Mode | What it does |
|
||||
|---|---|
|
||||
| `selfsigned` (default) | Generates a cert on the host. Good for trying it out; browsers will warn. |
|
||||
| `files` | Copies `nmon_tls_cert_src` / `nmon_tls_key_src` from the control machine. |
|
||||
| `acme` | Let's Encrypt for `nmon_server_name`; port 80 must reach the host. |
|
||||
| `insecure` | Plain HTTP. With no sign-in configured, nmon runs in dev-open mode where everyone is admin. Never expose this. |
|
||||
|
||||
**Reverse proxy:** nmon can sit behind one. Keep the proxy-to-nmon hop on HTTPS (`selfsigned` is fine there), set
|
||||
`nmon_trusted_proxies` to the proxy's addresses, and set the OIDC `redirect_url` to the public URL. Traps (UDP 162) and
|
||||
Graphite (TCP 2003) go to nmon directly, not through the HTTP proxy.
|
||||
|
||||
## Sign-in
|
||||
|
||||
- **OIDC:** set `nmon_oidc` (issuer, client_id, client_secret, admin_group). The secret is stored encrypted with
|
||||
`nmon secret set` and never written to `config.yaml`. Browsers go straight to the IdP; there is no login page.
|
||||
- **Break-glass:** `nmon_breakglass_password` is bcrypt-hashed on the host. Use it when the IdP is down. Five wrong
|
||||
attempts lock the source IP for 15 min; run `nmon unlock all --data-dir /var/lib/nmon` to clear.
|
||||
|
||||
## After installing
|
||||
|
||||
- **Back up `/etc/nmon/secret.key`** somewhere safe. Without it, a restore has no SNMP credentials.
|
||||
- Add devices in the UI or with `POST /api/v1/devices`, and mint API or push tokens under Settings.
|
||||
- The playbook owns `config.yaml`: edits made in the UI are overwritten on the next run, so put lasting settings in vars.
|
||||
- More: `docs/operations.md`.
|
||||
5
deploy/ansible/ansible.cfg
Normal file
@@ -0,0 +1,5 @@
|
||||
[defaults]
|
||||
inventory = inventory.ini
|
||||
roles_path = roles
|
||||
callback_result_format = yaml
|
||||
host_key_checking = True
|
||||
3
deploy/ansible/inventory.example.ini
Normal file
@@ -0,0 +1,3 @@
|
||||
# Copy to inventory.ini and edit. One nmon host is the design point.
|
||||
[nmon]
|
||||
nmon1.example.net ansible_user=admin ansible_become=true
|
||||
43
deploy/ansible/roles/nmon/defaults/main.yml
Normal file
@@ -0,0 +1,43 @@
|
||||
nmon_user: nmon
|
||||
nmon_data_dir: /var/lib/nmon
|
||||
nmon_etc_dir: /etc/nmon
|
||||
nmon_key_file: "{{ nmon_etc_dir }}/secret.key"
|
||||
nmon_backup_dir: /var/backups/nmon
|
||||
nmon_mibs_dir: /usr/share/nmon/mibs
|
||||
nmon_install_mibs: true
|
||||
# MAC vendors (optional IEEE registry). nmon downloads it itself unless nmon_oui_fetch is false; to ship it instead,
|
||||
# set nmon_oui_download (the host downloads the CSVs) or nmon_oui_csv_dir (a directory of the three CSVs on the control
|
||||
# machine). Either way they are kept next to the MIB sidecar and loaded with `nmon oui import`.
|
||||
nmon_oui_dir: /usr/share/nmon/oui
|
||||
nmon_oui_fetch: true
|
||||
nmon_oui_download: false
|
||||
nmon_oui_csv_dir: ""
|
||||
nmon_oui_urls:
|
||||
- https://standards-oui.ieee.org/oui/oui.csv
|
||||
- https://standards-oui.ieee.org/oui28/mam.csv
|
||||
- https://standards-oui.ieee.org/oui36/oui36.csv
|
||||
# MAC/ARP table job (see docs/operations.md); empty uses the defaults.
|
||||
nmon_l2: {}
|
||||
nmon_listen: "[::]:8443"
|
||||
nmon_trap_listen: "[::]:162"
|
||||
nmon_graphite_listen: "[::]:2003"
|
||||
nmon_icmp: auto
|
||||
nmon_timezone: UTC
|
||||
|
||||
nmon_tls_mode: selfsigned # selfsigned | files | acme | insecure
|
||||
nmon_server_name: "{{ inventory_hostname }}"
|
||||
nmon_acme_email: ""
|
||||
nmon_tls_extra_names: [] # more DNS names / IPs for the self-signed cert
|
||||
|
||||
nmon_breakglass_user: admin
|
||||
nmon_breakglass_password: "" # empty = no break-glass file (then OIDC must be configured)
|
||||
nmon_oidc: {}
|
||||
nmon_alerting: {}
|
||||
nmon_trusted_proxies: []
|
||||
|
||||
nmon_backup_at: "02:30"
|
||||
nmon_backup_keep: 7
|
||||
nmon_retention:
|
||||
hot: 31d
|
||||
fiveminute: 120d
|
||||
long: 2y
|
||||
14
deploy/ansible/roles/nmon/handlers/main.yml
Normal file
@@ -0,0 +1,14 @@
|
||||
- name: Restart nmon
|
||||
ansible.builtin.systemd_service:
|
||||
name: nmon
|
||||
state: restarted
|
||||
daemon_reload: true
|
||||
|
||||
- name: Reload nmon
|
||||
ansible.builtin.systemd_service:
|
||||
name: nmon
|
||||
state: reloaded
|
||||
|
||||
- name: Apply sysctl
|
||||
ansible.builtin.command: sysctl -p /etc/sysctl.d/60-nmon.conf
|
||||
changed_when: true
|
||||
182
deploy/ansible/roles/nmon/tasks/main.yml
Normal file
@@ -0,0 +1,182 @@
|
||||
- name: Check the platform (nmon ships linux/amd64 and linux/arm64)
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_system == "Linux"
|
||||
- ansible_architecture in ["x86_64", "aarch64"]
|
||||
fail_msg: "nmon v1 supports linux/amd64 and linux/arm64 only"
|
||||
|
||||
- name: Require a sign-in method when serving HTTPS (nmon refuses to start otherwise)
|
||||
ansible.builtin.assert:
|
||||
that: nmon_tls_mode == "insecure" or nmon_breakglass_password | length > 0 or nmon_oidc
|
||||
fail_msg: "Set nmon_breakglass_password and/or nmon_oidc, or use nmon_tls_mode=insecure for a dev-open playground"
|
||||
|
||||
- name: Install host packages (openssl for the self-signed cert, htpasswd for bcrypt, acl for become_user)
|
||||
ansible.builtin.package:
|
||||
name: "{{ ['openssl', 'apache2-utils', 'acl', 'procps'] if ansible_os_family == 'Debian' else ['openssl', 'httpd-tools', 'acl', 'procps-ng'] }}"
|
||||
state: present
|
||||
|
||||
- name: Create the nmon system user
|
||||
ansible.builtin.user:
|
||||
name: "{{ nmon_user }}"
|
||||
system: true
|
||||
shell: /usr/sbin/nologin
|
||||
home: "{{ nmon_data_dir }}"
|
||||
create_home: false
|
||||
|
||||
- name: Create directories
|
||||
ansible.builtin.file:
|
||||
path: "{{ item.path }}"
|
||||
state: directory
|
||||
owner: "{{ nmon_user }}"
|
||||
group: "{{ nmon_user }}"
|
||||
mode: "{{ item.mode }}"
|
||||
loop:
|
||||
- { path: "{{ nmon_data_dir }}", mode: "0750" }
|
||||
- { path: "{{ nmon_etc_dir }}", mode: "0750" }
|
||||
- { path: "{{ nmon_backup_dir }}", mode: "0750" }
|
||||
|
||||
- name: Ensure /etc/sysctl.d exists (minimal images lack it)
|
||||
ansible.builtin.file:
|
||||
path: /etc/sysctl.d
|
||||
state: directory
|
||||
mode: "0755"
|
||||
|
||||
- name: Allow unprivileged ICMP echo (nmon pings without root)
|
||||
ansible.builtin.copy:
|
||||
content: "net.ipv4.ping_group_range = 0 2147483647\n"
|
||||
dest: /etc/sysctl.d/60-nmon.conf
|
||||
mode: "0644"
|
||||
notify: Apply sysctl
|
||||
|
||||
- name: Install the nmon binary
|
||||
ansible.builtin.copy:
|
||||
src: "{{ playbook_dir }}/../../{{ 'nmon' if ansible_architecture == 'x86_64' else 'nmon-arm64' }}"
|
||||
dest: /usr/local/bin/nmon
|
||||
mode: "0755"
|
||||
notify: Restart nmon
|
||||
|
||||
- name: Install the MIB sidecar
|
||||
when: nmon_install_mibs | bool
|
||||
block:
|
||||
- name: Create the MIB directory
|
||||
ansible.builtin.file:
|
||||
path: "{{ nmon_mibs_dir }}"
|
||||
state: directory
|
||||
mode: "0755"
|
||||
- name: Unpack MIBs
|
||||
ansible.builtin.unarchive:
|
||||
src: "{{ playbook_dir }}/.mibs.tar.gz"
|
||||
dest: "{{ nmon_mibs_dir }}"
|
||||
creates: "{{ nmon_mibs_dir }}/IF-MIB"
|
||||
|
||||
- name: Ship the IEEE vendor registry (optional)
|
||||
when: nmon_oui_download | bool or nmon_oui_csv_dir | length > 0
|
||||
block:
|
||||
- name: Create the registry directory
|
||||
ansible.builtin.file:
|
||||
path: "{{ nmon_oui_dir }}"
|
||||
state: directory
|
||||
mode: "0755"
|
||||
- name: Download the registry CSVs on the host
|
||||
when: nmon_oui_download | bool
|
||||
ansible.builtin.get_url:
|
||||
url: "{{ item }}"
|
||||
dest: "{{ nmon_oui_dir }}/{{ item | basename }}"
|
||||
mode: "0644"
|
||||
timeout: 60
|
||||
loop: "{{ nmon_oui_urls }}"
|
||||
register: nmon_oui_fetched
|
||||
- name: Copy the registry CSVs from the control machine
|
||||
when: nmon_oui_csv_dir | length > 0
|
||||
ansible.builtin.copy:
|
||||
src: "{{ nmon_oui_csv_dir }}/"
|
||||
dest: "{{ nmon_oui_dir }}/"
|
||||
mode: "0644"
|
||||
register: nmon_oui_copied
|
||||
- name: Load the registry into nmon's data directory
|
||||
when: (nmon_oui_fetched is changed) or (nmon_oui_copied is changed)
|
||||
become_user: "{{ nmon_user }}"
|
||||
ansible.builtin.command:
|
||||
argv: [/usr/local/bin/nmon, oui, import, "{{ nmon_oui_dir }}", --data-dir, "{{ nmon_data_dir }}"]
|
||||
changed_when: true
|
||||
|
||||
- name: TLS certificate
|
||||
ansible.builtin.include_tasks: tls.yml
|
||||
when: nmon_tls_mode in ["selfsigned", "files"]
|
||||
|
||||
- name: Break-glass admin (bcrypt hash; the password never lands on disk)
|
||||
when: nmon_breakglass_password | length > 0
|
||||
block:
|
||||
- name: Hash the break-glass password
|
||||
ansible.builtin.command:
|
||||
argv: [htpasswd, -nbBC, "12", "{{ nmon_breakglass_user }}", "{{ nmon_breakglass_password }}"]
|
||||
register: nmon_bg_hash
|
||||
changed_when: false
|
||||
no_log: true
|
||||
- name: Write the break-glass file
|
||||
ansible.builtin.copy:
|
||||
content: "{{ nmon_bg_hash.stdout | trim }}\n"
|
||||
dest: "{{ nmon_etc_dir }}/breakglass"
|
||||
owner: "{{ nmon_user }}"
|
||||
group: "{{ nmon_user }}"
|
||||
mode: "0600"
|
||||
no_log: true
|
||||
# Re-hashing gives a new salt each run; only replace the file when it is missing.
|
||||
args:
|
||||
force: false
|
||||
|
||||
- name: Store the OIDC client secret encrypted (creates the key file on a new install)
|
||||
ansible.builtin.shell:
|
||||
cmd: >-
|
||||
set -o pipefail;
|
||||
printf '%s\n' "$NMON_OIDC_SECRET" |
|
||||
/usr/local/bin/nmon secret set oidc-client --data-dir {{ nmon_data_dir }} --secret-key-file {{ nmon_key_file }}
|
||||
executable: /bin/bash
|
||||
environment:
|
||||
NMON_OIDC_SECRET: "{{ nmon_oidc.client_secret }}"
|
||||
become: true
|
||||
become_user: "{{ nmon_user }}"
|
||||
when: nmon_oidc.client_secret is defined
|
||||
changed_when: true
|
||||
no_log: true
|
||||
notify: Restart nmon
|
||||
|
||||
- name: Write config.yaml
|
||||
ansible.builtin.template:
|
||||
src: config.yaml.j2
|
||||
dest: "{{ nmon_data_dir }}/config.yaml"
|
||||
owner: "{{ nmon_user }}"
|
||||
group: "{{ nmon_user }}"
|
||||
mode: "0640"
|
||||
notify: Reload nmon
|
||||
|
||||
- name: Install the systemd unit
|
||||
ansible.builtin.template:
|
||||
src: nmon.service.j2
|
||||
dest: /etc/systemd/system/nmon.service
|
||||
mode: "0644"
|
||||
notify: Restart nmon
|
||||
|
||||
- name: Enable and start nmon
|
||||
ansible.builtin.systemd_service:
|
||||
name: nmon
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
|
||||
- name: Wait for nmon to answer
|
||||
ansible.builtin.uri:
|
||||
url: "{{ 'http' if nmon_tls_mode == 'insecure' else 'https' }}://localhost:{{ nmon_listen | regex_replace('^.*:', '') }}/healthz"
|
||||
validate_certs: false
|
||||
status_code: 200
|
||||
register: nmon_health
|
||||
retries: 30
|
||||
delay: 2
|
||||
until: nmon_health.status == 200
|
||||
|
||||
- name: Where to go next
|
||||
ansible.builtin.debug:
|
||||
msg:
|
||||
- "nmon is up: {{ 'http' if nmon_tls_mode == 'insecure' else 'https' }}://{{ nmon_server_name }}:{{ nmon_listen | regex_replace('^.*:', '') }}/"
|
||||
- "{{ 'Signs in via your IdP; break-glass at /auth/breakglass' if nmon_oidc else 'Sign in at /auth/breakglass as ' + nmon_breakglass_user }}"
|
||||
- "Back up the key file {{ nmon_key_file }} somewhere safe; without it a restore has no SNMP credentials."
|
||||
51
deploy/ansible/roles/nmon/tasks/tls.yml
Normal file
@@ -0,0 +1,51 @@
|
||||
- name: Work out the certificate's names (IP addresses need IP:, names need DNS:)
|
||||
ansible.builtin.set_fact:
|
||||
nmon_tls_sans: >-
|
||||
{{ (([nmon_server_name] + nmon_tls_extra_names) | unique
|
||||
| map('regex_replace', '^(?=[0-9.]+$|.*:)', 'IP:')
|
||||
| map('regex_replace', '^(?!IP:)', 'DNS:') | list)
|
||||
+ ['DNS:localhost', 'IP:127.0.0.1', 'IP:::1'] }}
|
||||
when: nmon_tls_mode == "selfsigned"
|
||||
|
||||
- name: Generate a self-signed certificate (quick start; replace for production)
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- openssl
|
||||
- req
|
||||
- -x509
|
||||
- -newkey
|
||||
- ec
|
||||
- -pkeyopt
|
||||
- ec_paramgen_curve:prime256v1
|
||||
- -nodes
|
||||
- -days
|
||||
- "825"
|
||||
- -subj
|
||||
- "/CN={{ nmon_server_name }}"
|
||||
- -addext
|
||||
- "subjectAltName={{ nmon_tls_sans | join(',') }}"
|
||||
- -keyout
|
||||
- "{{ nmon_etc_dir }}/tls.key"
|
||||
- -out
|
||||
- "{{ nmon_etc_dir }}/tls.crt"
|
||||
creates: "{{ nmon_etc_dir }}/tls.crt"
|
||||
when: nmon_tls_mode == "selfsigned"
|
||||
notify: Restart nmon
|
||||
|
||||
- name: Install the provided certificate and key
|
||||
ansible.builtin.copy:
|
||||
src: "{{ item.src }}"
|
||||
dest: "{{ nmon_etc_dir }}/{{ item.dest }}"
|
||||
mode: "{{ item.mode }}"
|
||||
loop:
|
||||
- { src: "{{ nmon_tls_cert_src }}", dest: tls.crt, mode: "0644" }
|
||||
- { src: "{{ nmon_tls_key_src }}", dest: tls.key, mode: "0600" }
|
||||
when: nmon_tls_mode == "files"
|
||||
notify: Reload nmon
|
||||
|
||||
- name: Certificate ownership
|
||||
ansible.builtin.file:
|
||||
path: "{{ nmon_etc_dir }}/{{ item }}"
|
||||
owner: "{{ nmon_user }}"
|
||||
group: "{{ nmon_user }}"
|
||||
loop: [tls.crt, tls.key]
|
||||
38
deploy/ansible/roles/nmon/templates/config.yaml.j2
Normal file
@@ -0,0 +1,38 @@
|
||||
# Managed by Ansible (deploy/ansible). Admin edits in the UI are overwritten on the next run.
|
||||
retention:
|
||||
hot: {{ nmon_retention.hot }}
|
||||
fiveminute: {{ nmon_retention.fiveminute }}
|
||||
long: {{ nmon_retention.long }}
|
||||
timezone: {{ nmon_timezone }}
|
||||
{% if nmon_l2 %}
|
||||
l2:
|
||||
{{ nmon_l2 | to_nice_yaml(indent=2) | indent(2) }}
|
||||
{% endif %}
|
||||
{% if not nmon_oui_fetch | bool %}
|
||||
oui:
|
||||
fetch: false
|
||||
{% endif %}
|
||||
{% if nmon_alerting %}
|
||||
alerting:
|
||||
{{ nmon_alerting | to_nice_yaml(indent=2) | indent(2) }}
|
||||
{% endif %}
|
||||
auth:
|
||||
{% if nmon_oidc %}
|
||||
oidc:
|
||||
issuer: {{ nmon_oidc.issuer }}
|
||||
client_id: {{ nmon_oidc.client_id }}
|
||||
client_secret_name: oidc-client
|
||||
redirect_url: {{ nmon_oidc.redirect_url | default('https://' + nmon_server_name + ':' + (nmon_listen | regex_replace('^.*:', '')) + '/auth/callback') }}
|
||||
groups_claim: {{ nmon_oidc.groups_claim | default('groups') }}
|
||||
admin_group: {{ nmon_oidc.admin_group | default('nmon-admins') }}
|
||||
{% endif %}
|
||||
{% if nmon_breakglass_password | length > 0 %}
|
||||
break_glass_file: {{ nmon_etc_dir }}/breakglass
|
||||
{% endif %}
|
||||
{% if nmon_trusted_proxies %}
|
||||
trusted_proxies: {{ nmon_trusted_proxies | to_json }}
|
||||
{% endif %}
|
||||
backup:
|
||||
at: "{{ nmon_backup_at }}"
|
||||
dir: {{ nmon_backup_dir }}
|
||||
keep: {{ nmon_backup_keep }}
|
||||
47
deploy/ansible/roles/nmon/templates/nmon.service.j2
Normal file
@@ -0,0 +1,47 @@
|
||||
# Managed by Ansible (deploy/ansible).
|
||||
[Unit]
|
||||
Description=nmon network monitor
|
||||
Documentation=https://claude.ai/artifact/7FTnfA5hReTjY6cfArfJ2Z
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User={{ nmon_user }}
|
||||
Group={{ nmon_user }}
|
||||
ExecStart=/usr/local/bin/nmon serve \
|
||||
--data-dir {{ nmon_data_dir }} \
|
||||
--secret-key-file {{ nmon_key_file }} \
|
||||
--listen {{ nmon_listen }} \
|
||||
--trap-listen {{ nmon_trap_listen }} \
|
||||
--graphite-listen {{ nmon_graphite_listen }} \
|
||||
--icmp {{ nmon_icmp }} \
|
||||
{% if nmon_tls_mode == 'insecure' %}
|
||||
--insecure-http
|
||||
{% elif nmon_tls_mode == 'acme' %}
|
||||
--acme-host {{ nmon_server_name }} --acme-email {{ nmon_acme_email }}
|
||||
{% else %}
|
||||
--tls-cert {{ nmon_etc_dir }}/tls.crt --tls-key {{ nmon_etc_dir }}/tls.key
|
||||
{% endif %}
|
||||
ExecReload=/bin/kill -HUP $MAINPID
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
TimeoutStopSec=90
|
||||
LimitNOFILE=1048576
|
||||
|
||||
# Ports below 1024: SNMP traps (162) and the ACME challenge (80).
|
||||
AmbientCapabilities=CAP_NET_BIND_SERVICE
|
||||
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
PrivateDevices=true
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictSUIDSGID=true
|
||||
LockPersonality=true
|
||||
ReadWritePaths={{ nmon_data_dir }} {{ nmon_etc_dir }} {{ nmon_backup_dir }}
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
26
deploy/ansible/site.yml
Normal file
@@ -0,0 +1,26 @@
|
||||
# Deploys nmon to the hosts in group "nmon". Run from the repo root's deploy/ansible:
|
||||
# ansible-playbook site.yml -e @vars.yml
|
||||
- name: Build nmon locally
|
||||
hosts: localhost
|
||||
connection: local
|
||||
gather_facts: false
|
||||
tags: [build]
|
||||
tasks:
|
||||
- name: Build the release binary (UI and VictoriaMetrics embedded)
|
||||
ansible.builtin.command:
|
||||
cmd: make build build-arm64
|
||||
chdir: "{{ playbook_dir }}/../.."
|
||||
changed_when: true
|
||||
when: nmon_build | default(true) | bool
|
||||
|
||||
- name: Pack the MIB sidecar # noqa: command-instead-of-module (archive needs community.general; plain tar keeps this collection-free)
|
||||
ansible.builtin.command:
|
||||
cmd: tar -czf {{ playbook_dir }}/.mibs.tar.gz -C {{ playbook_dir }}/../../mibs .
|
||||
creates: "{{ playbook_dir }}/.mibs.tar.gz"
|
||||
when: nmon_install_mibs | default(true) | bool
|
||||
|
||||
- name: Install and run nmon
|
||||
hosts: nmon
|
||||
become: true
|
||||
roles:
|
||||
- nmon
|
||||
38
deploy/ansible/vars.example.yml
Normal file
@@ -0,0 +1,38 @@
|
||||
# Copy to vars.yml (keep it out of git, or encrypt with ansible-vault) and pass with -e @vars.yml.
|
||||
|
||||
# TLS: selfsigned (quick start), files (your own cert), acme (Let's Encrypt), or insecure (plain HTTP, dev only).
|
||||
nmon_tls_mode: selfsigned
|
||||
nmon_server_name: "{{ inventory_hostname }}" # a DNS name or an IP; it goes in the self-signed cert
|
||||
# nmon_tls_extra_names: ["10.0.0.5", "nmon.lan"]
|
||||
# nmon_tls_cert_src: files/nmon.crt # for tls_mode=files (paths on the control machine)
|
||||
# nmon_tls_key_src: files/nmon.key
|
||||
# nmon_acme_email: noc@example.net # for tls_mode=acme; nmon answers http-01 on :80
|
||||
|
||||
# Break-glass local admin: the way in when the IdP is down (and the only sign-in until OIDC is set).
|
||||
nmon_breakglass_user: admin
|
||||
nmon_breakglass_password: "change-me-please"
|
||||
|
||||
# OIDC (optional). Unauthenticated browsers are redirected to the IdP; there is no login page.
|
||||
# nmon_oidc:
|
||||
# issuer: https://idp.example.net/realms/net
|
||||
# client_id: nmon
|
||||
# client_secret: "s3cret" # stored encrypted via `nmon secret set`, never in config.yaml
|
||||
# groups_claim: groups
|
||||
# admin_group: nmon-admins
|
||||
|
||||
# Alert receivers (optional): webhook and/or SMTP, plus the outbound allow-list they need.
|
||||
# nmon_alerting:
|
||||
# receivers:
|
||||
# - name: noc-webhook
|
||||
# kind: webhook
|
||||
# url: https://hooks.example.net/nmon
|
||||
# outbound_allow: ["hooks.example.net"]
|
||||
|
||||
nmon_timezone: UTC
|
||||
|
||||
# MAC/ARP table job and optional IEEE vendor registry (see docs/operations.md).
|
||||
# nmon_l2: {interval: 15m, keep: 90d, edge_max_macs: 8}
|
||||
# nmon_oui_fetch: false # never let nmon download the registry...
|
||||
# nmon_oui_download: true # ...have this play download the CSVs on the host instead,
|
||||
# nmon_oui_csv_dir: files/oui # ...or ship the three CSVs from the control machine
|
||||
nmon_icmp: auto # off = decide reachability by SNMP alone
|
||||
@@ -42,3 +42,13 @@ conscious, reviewed exception.
|
||||
| 35 | Port descriptions imported from CSV | Interfaces have a new operator-owned `description` column (migration 8). The CSV import and the API write it; discovery rewrites `ifalias` from the device on every run and never touches `description`. Interface search `q` matches ifname, ifalias and description. | A description typed into `ifalias` would be overwritten at the next discovery. | M6 |
|
||||
| 36 | OIDC login with a browser flow; break-glass as a local login | **No login UI** (owner decision). Every route except `/auth/*`, `/healthz` and push ingest needs a credential, the SPA's files included: an unauthenticated page load is a 302 to `/auth/login?return=<url>` (the OIDC flow; the return target must be a same-site path), an API call without credentials is a 401 problem+json. Break-glass is `GET /auth/breakglass` with HTTP Basic, so the browser's own prompt appears; success sets a session and redirects to `return` or `/` (`POST /auth/breakglass` with JSON stays for scripts). When no IdP is configured the redirect goes to the break-glass prompt. Logout is `POST /auth/logout`, then `/auth/signedout` (static HTML). `/docs` and `/openapi*` need the viewer role. A missing OIDC client secret at start leaves OIDC off with a warning. | The IdP is the login page; a second one would be a second thing to secure. | M6 |
|
||||
| 37 | Disk guard: "ingest throttled below the reserve" | Throttle answers push ingest with 429 + Retry-After below the 5% reserve; the poller keeps writing (its volume is bounded by the inventory). Above 90% used, new series are dropped on the nmon side (poller, push and checks) while known series continue; disk alarms are re-raised each minute while the condition holds and resolve themselves 3 minutes after it clears. | VictoriaMetrics series limits are start-up flags; the guard is nmon-side admission. | M6 |
|
||||
| 38 | `interface_down` is a default rule that fires for every down port | `interface_down` is **opt-in**: it alerts only for ports whose own `alert_down` flag is on, or that belong to a group with `alert_down` on (membership in force, `GroupMember.Covers`). Migration 9 adds the columns, default off. Opting a port out closes its open alert silently as `retired` (never a resolved notification). On an access switch most unused ports are admin-up and oper-down, so the default flooded the owner with alerts. `interface_errors` is unchanged. | Owner feedback, first hands-on session. | post-v1 (v1.0.1) |
|
||||
| 39 | Spec: a profile that was matched stays ("non-generic is sticky"); no way to choose one by hand | Discovery re-matches **every unpinned device on every run**, so a newer or better profile is picked up, and at startup nmon queues a discovery for each unpinned device whose stored profile differs from what `Set.Match` picks now. An admin can pin a profile (`PUT /api/v1/devices/{slug}` with `profile`, or the Edit dialog); a pinned device keeps it, and a pinned profile that no longer exists gives generic plus the `profile_missing` warning as before. `profile: ""` returns to automatic. Migration 10 adds `devices.profile_pinned`; `GET /api/v1/profiles` lists the loaded profiles; `DeviceView` gains `profile_pinned` and `profile_match`. Profile changes are audited (`device.profile` from discovery, `device.edit` from the API). | The sticky rule meant a device matched once as generic stayed generic after an upgrade added its profile. | post-v1 (v1.0.1) |
|
||||
| 40 | Profiles shipped: the M2 set | Three more shipped profiles for the owner's gear: `ubiquiti-edgeswitch` (sysObjectID `1.3.6.1.4.1.4413` or sysDescr `^EdgeSwitch`, priority 30), `ubiquiti-unifi-switch` (sysDescr `USW`/`US-`/`USL`/`UniFi`, priority 40, so it beats edgeswitch for UniFi switches that report enterprise 4413), `vyos` (sysObjectID `1.3.6.1.4.1.44641` or sysDescr `VyOS`, extends `linux-netsnmp`, priority 20), and `ubiquiti-unifi-ap` (sysDescr `UAP`/`U6`/`U7`/`UAL`/`UK`/`E7`/`U-LTE`, UBNT-UniFi-MIB per-VAP clients, channel and tx power, per-radio channel utilisation; the MIB has no noise floor, and no per-radio channel or tx power, so those are per VAP). EdgeSwitch CPU is a text string and its temperature table has no label column, so neither is collected. | The owner's switches, routers and access points matched only `generic`. | post-v1 (v1.0.1) |
|
||||
| 41 | Default rules apply everywhere (spec §Alerting) | Alerts fire only where an **alert policy** binds a rule to a scope (groups plus device and port attribute matchers), stored in the `alert_policies` table (migration 12), edited through `/api/v1/alert-policies` and the Alerts > Alert policies tab, and applied live. With no policies nothing alerts: not metric rules, not `device_down`/`device_warning`, not system alarms (`system` policy, scope `all` only). An `ignore` policy beats an `alert` one at equal priority. Upgrading seeds nothing, so open alerts without a binding close silently as `retired` with no resolved notice. Supersedes #38's mechanism (the bell now only feeds the `alert_down` matcher) and makes `Rule.OptIn` a no-op. | Owner decision 2026-10-23: preprod alerted on everything, and a WiFi port that always sees errors is judged by the same rule as a fiber uplink where one error matters. | v1.1 |
|
||||
| 50 | The spec's collection table lists BGP-4-MIB but no profile fills it, and it names no MAC/ARP lookup | v1.3 (owner request 2026-10-24) adds both: BGP peers in the `routing` category (CISCO-BGP4 for cisco-ios/cisco-nxos, Juniper BGP4-V2, Arista BGP4V2, BGP4-MIB for the IPv4-only profiles that answer it) with a Routing tab, a BGP page, `GET /api/v1/bgp/peers` and the built-in `bgp_peer_down` rule; and a MAC/ARP/FDB lookup with a Find host page. Additive API: `/l2/search`, `/devices/{slug}/{l2,fdb,arp,l2/ports}`, `/interfaces/{id}/macs`, `/bgp/peers`; `/poller/health` gains `l2` and `notes`. | The owner wants to find where a host is plugged in and to see BGP sessions. | v1.3 |
|
||||
| 51 | (not specified) table walks run in discovery or not at all; the poll path never walks | A separate **L2 table job** (`internal/l2`, SNMP path label `l2` in `nmon_pdus_total`) walks BRIDGE-MIB, Q-BRIDGE-MIB, CISCO-VTP-MIB (per-VLAN context, profile flag `l2.per_vlan_context`, set on cisco-ios only) and IP-MIB neighbour tables every `l2.interval` (default 15m, minimum 5m) with its own concurrency cap (`l2.concurrency`, 4), a per-device deadline (`l2.device_timeout`, 2m) and the rollup job's yield rule (it pauses while the hot write queue is over half full). A device that times out or fails is skipped for the cycle, recorded on `/poller/health`, and never delays the others. Rows live in the inventory database (migration 14: `fdb_entries`, `arp_entries`, `vlans`, and `l2_walks`, one row per device holding the time of its last walk, which is how a row is "current"), are in `nmon backup` (JSONL export too) and are pruned daily by `last_seen` against `l2.keep` (90d). `interface_id` is the empty string, not NULL, when a bridge port has no ifIndex, because the unique key must not contain NULL. | The hot poll path must not walk tables; the history behind "where was this MAC last week" is inventory, not a metric. | v1.3 |
|
||||
| 52 | The component label of a table metric is a column of the table | Profiles may label rows by decoding the **row index** (`index_label.layout`, tokens `skip`, `skipinet`, `inet`, `skipaddr`, `addr`, `ipv4`, `peerindex`, `afisafi`), because BGP peer tables are indexed by an address. The label is the peer address (IPv6 compressed); a per-address-family prefix count is labelled `<peer>\|<afi>-<safi>`, for example `2001:db8::1\|ipv6-unicast`. Juniper's prefix table is keyed by a peer index, so its rows resolve through the `peer_index` attribute of the peer table (`index_label.via`). Peer AS, local AS, router id and description are component attributes (migration 11), never labels, so the SNMP label contract (`device` + `component`) is unchanged. Other profile additions: `enable_metrics` (switch on optional parent metrics), `wire: counter32` (CISCO-BGP4 declares accepted prefixes as a Counter32 but they are a count), attribute `format: ipbytes`, and `l2.per_vlan_context`. | A BGP peer is identified by its address, which only the index carries; the alternatives (a label column that does not exist, or labelling by row number) would break "each peer appears once" and the alert text. | v1.3 |
|
||||
| 53 | `bgp_peer_down` summary names the peer, AS and description | The rule's own summary names the device and the peer. The AS, router id and description are appended to the **notification** (Summary, Description and labels) when it is delivered, from the component attributes; the stored alert row keeps the plain summary. | The metric label contract allows only `device` and `component`, and the rule templates see only those; extending the evaluator's template data belongs to the v1.2 rules plumbing. | v1.3 |
|
||||
| 54 | (not specified) vendor of a MAC | An optional IEEE OUI registry (MA-L, MA-M, MA-S) shows the vendor of each MAC and flags locally administered (`local`, shown "Private / randomised") and multicast addresses. It is fetched weekly through the outbound allow-list (`oui.fetch: false` turns it off; `nmon oui import DIR` loads the CSVs offline), kept as `<data-dir>/oui/registry.gz` (checksummed, written atomically, in `nmon backup`), and looked up at query time by the longest prefix, so a refresh relabels history. Without a registry the vendor is empty, nothing fails, and `/poller/health` carries an `oui_unavailable` note (never an alert). The CSVs are public IEEE data and are not in the repository. | Owner request 2026-10-24: nice to have, never required. | v1.3 |
|
||||
| 55 | Lookup ranks edge ports "the port with the fewest MACs, then the most recent last_seen" | Within a MAC's locations the order is: edge ports before uplinks, then rows the device's last walk still saw before history, then fewest MACs, then most recent. A port is an uplink above `l2.edge_max_macs` MACs (8), with an LLDP neighbour (the lldpRemTable local port number is taken to be the ifIndex), or as a LAG (ifType 161). A port's MAC count is the MACs on it at or after the row's `last_seen`, so for a current row it is the port's MACs in the last walk. | A host that moved should show its new port first even when the old, now empty, port had fewer MACs. | v1.3 |
|
||||
|
||||
91
docs/coverage.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# Collection coverage
|
||||
|
||||
What the spec's "What to collect, by device class" table (docs/spec/spec.md) asks for, against what the shipped
|
||||
profiles in `profiles/` collect today. A metric listed here is polled, stored raw and shown on a device tab (every
|
||||
non-interface metric names a `category`; the orphan test in `internal/profile/present_test.go` enforces it). Gaps are
|
||||
stated plainly; nothing here is promised that is not collected.
|
||||
|
||||
Legend: **yes** collected and shown; **partial** some of it; **no** not collected.
|
||||
|
||||
| Class | Spec item | Status | Profile and metric |
|
||||
|---|---|---|---|
|
||||
| Every device | sysUpTime | yes | generic: sysUpTime, hrSystemUptime |
|
||||
| Every device | hrProcessorLoad per core | yes | generic: hrProcessorLoad |
|
||||
| Every device | hrStorage size/used per filesystem and RAM | yes | generic: hrStorageUsed / hrStorageSize x hrStorageAllocationUnits (bytes, percent bar) |
|
||||
| Every device | memTotalReal / memAvailReal (UCD) | yes | generic: memTotalReal, memAvailReal |
|
||||
| Servers | UCD load average | yes | linux-netsnmp, ubiquiti-edgeos: laLoadInt |
|
||||
| Servers | swap | yes | linux-netsnmp: memTotalSwap, memAvailSwap |
|
||||
| Routers, switches | IF-MIB counters, ifOperStatus | yes | generic (Ports tab) |
|
||||
| Routers, switches | ENTITY-SENSOR-MIB temp/fan/PSU | partial | generic entPhySensorValue is the raw reading only: its type, scale and precision are not applied, so units are mixed (environment tab, labelled raw). No fan or PSU state. cisco-ios/cisco-nxos entSensorValue is the same |
|
||||
| Routers, switches | POWER-ETHERNET-MIB PoE per port | no | not in any profile |
|
||||
| Routers, switches | LLDP neighbours | yes | generic discovery table only (no metric) |
|
||||
| Routers, switches | BGP-4-MIB peer state and prefixes | partial | see "BGP peers" below: state, uptime, accepted prefixes per address family and update counters where the device answers a BGP MIB; nothing on MikroTik or Linux hosts |
|
||||
| Routers, switches | optics DOM | partial | juniper-junos jnxDom* are collected but are interface metrics, so they are not on a device tab (there is no optics tab data yet); cisco entSensor DOM is not separated from the raw sensor reading |
|
||||
| Routers, switches | STP topology changes | no | |
|
||||
| Routers, switches | MAC/ARP table size | yes | the tables themselves, not just their size: the L2 table job (`internal/l2`) reads FDB, VLAN names and ARP/ND from every SNMP device (see "MAC, ARP and FDB" below) |
|
||||
| Routers, switches | cpmCPUTotal, jnxOperatingCPU | yes | cisco-ios, cisco-nxos: cpmCPUTotal5minRev; juniper-junos: jnxOperatingCPU, jnxOperatingTemp, jnxOperatingBuffer |
|
||||
| Routers, switches | Cisco memory pools | yes | cisco-ios: ciscoMemoryPoolUsed / Free (not as a percent; there is no total) |
|
||||
| Switches (Ubiquiti) | memory | partial | ubiquiti-edgeswitch, ubiquiti-unifi-switch: free / available only; CPU is a text string in that MIB and is not collected |
|
||||
| Firewalls | session count | partial | fortigate: fgSysSesCount (no maximum); pfsense has no vendor metrics |
|
||||
| Firewalls | VPN tunnel state | no | |
|
||||
| Firewalls | HA state | partial | fortigate: fgHaSystemMode only |
|
||||
| Firewalls | policy throughput (fgSys, panSession, pfSense) | no | fortigate CPU and memory only |
|
||||
| APs, controllers | client count per AP and SSID | yes | ubiquiti-unifi-ap: unifiVapNumStations per SSID |
|
||||
| APs, controllers | channel utilization | yes | ubiquiti-unifi-ap: unifiRadioCu* per radio |
|
||||
| APs, controllers | noise floor | no | the UniFi MIB has no such object |
|
||||
| APs, controllers | tx power | yes | ubiquiti-unifi-ap: unifiVapTxPower |
|
||||
| APs, controllers | AP up/down from controller | no | no controller integration |
|
||||
| WISP radios, CPE | signal, SNR, modulation, link capacity | no | no Ubiquiti airMAX, Cambium or Mimosa profile |
|
||||
| Servers, hypervisors | Proxmox push | yes | push receiver and the proxmox integration (not a profile) |
|
||||
| Servers, hypervisors | node_exporter, Ceph, SMART | no | |
|
||||
| UPS, PDU | battery charge, runtime, input/output voltage, load, output status | yes | apc-ups: UPS-MIB and PowerNet upsAdv/upsBasic metrics (power tab) |
|
||||
| UPS, PDU | per-outlet current (PDU MIBs), generator status | no | |
|
||||
| Printers | supply levels per prtMarkerSupplies | yes | printer: prtMarkerSuppliesLevel of prtMarkerSuppliesMaxCapacity; -3 "some remaining" and -2 "unknown" are stored and shown as words |
|
||||
| Printers | supply type and colour | yes | printer: prtMarkerSuppliesType and prtMarkerColorantValue, read at discovery as component attributes |
|
||||
| Printers | page count | yes | printer: prtMarkerLifeCount |
|
||||
| Printers | status and alerts | partial | printer: prtAlertSeverityLevel (severity only, not the alert description or code) |
|
||||
| Printers | paper tray levels | yes | printer: prtInputCurrentLevel of prtInputMaxCapacity, labelled by prtInputName |
|
||||
| Printers | hp, Brother, Xerox serial and model | no | serial and model are not collected; no vendor quirk profiles (printer-brother, printer-ricoh) are shipped, see below |
|
||||
| Environment | ENTITY-SENSOR-MIB | partial | as for routers: raw reading only |
|
||||
| Environment | temperature, humidity, door, water (NetBotz, Raritan) | no | no profile |
|
||||
| Environment | MikroTik health | yes | mikrotik-routeros: voltage, board and CPU temperature |
|
||||
| Services | ping, HTTP, DNS, TLS expiry | yes | checks (not profiles) |
|
||||
|
||||
Profiles with no vendor metrics beyond generic: arista-eos, pfsense, vyos, windows-snmp.
|
||||
|
||||
## Alerts named in the spec
|
||||
|
||||
| Alert | Status |
|
||||
|---|---|
|
||||
| cpu_high, memory_high, disk_full | yes |
|
||||
| ups_on_battery, ups_runtime_low | yes |
|
||||
| printer_supply_low (< 10%) | yes |
|
||||
| printer_supply_empty (level 0, critical) | yes (added in v1.0.1) |
|
||||
| temp_high (per-profile threshold) | yes, for MikroTik, UPS battery and Junos sensors only |
|
||||
| "any prtAlert of severity critical" | no rule; prtAlertSeverityLevel is collected and shown, not alerted on |
|
||||
|
||||
## Printer quirk profiles (Brother, Ricoh)
|
||||
|
||||
Not shipped. The LibreNMS recordings (`tests/snmpsim/brother*.snmprec`, `ricoh*.snmprec`) were not used to derive a
|
||||
quirk set, so none is claimed. The generic `printer` profile matches Brother (2435) and Ricoh (367) by enterprise number.
|
||||
|
||||
## BGP peers (v1.3)
|
||||
|
||||
The component is the peer address (IPv6 compressed). Metrics: `bgp_peer_state` (idle, connect, active, opensent, openconfirm, established; anything but established is critical), `bgp_peer_uptime`, `bgp_peer_in_updates` / `bgp_peer_out_updates` (counters, shown as a rate) and, where the vendor reports prefixes per address family, `bgp_prefixes_accepted` with the component `<peer>|<afi>-<safi>`. Peer AS, local AS, router id (and the description, where the MIB has one) are component attributes from discovery. A peer is reported once: a profile with a vendor table replaces the BGP4-MIB metrics of the same names.
|
||||
|
||||
| Profile | Source | IPv6 peers | Prefixes | Evidence |
|
||||
|---|---|---|---|---|
|
||||
| cisco-ios | CISCO-BGP4-MIB cbgpPeer2Table and cbgpPeer2AddrFamilyPrefixTable | yes | per AFI/SAFI | IOS XE and IOS XR recordings (`iosxe`, `iosxr`, `iosxr_asr9904`) |
|
||||
| cisco-nxos | CISCO-BGP4-MIB, as cisco-ios | yes | per AFI/SAFI | **not verified**: the NX-OS recordings carry no BGP table; kept because Cisco documents the MIB on NX-OS, and a device that does not answer simply has no peers |
|
||||
| juniper-junos | BGP4-V2-MIB-JUNIPER jnxBgpM2PeerTable, jnxBgpM2PrefixCountersTable | yes | per AFI/SAFI (resolved through jnxBgpM2PeerIndex) | `junos_mx`, `junos_vmx`, `junos_rpm`; the peer index has no length octets, as Junos sends it |
|
||||
| arista-eos | ARISTA-BGP4V2-MIB aristaBgp4V2PeerTable, aristaBgp4V2PrefixTable | yes | per AFI/SAFI | `arista_eos`, `arista_eos_7280r`; description is collected |
|
||||
| ubiquiti-edgeos, fortigate, pfsense, vyos | BGP4-MIB bgpPeerTable (generic metrics switched on with `enable_metrics`) | no (RFC 4273 is IPv4 only) | no | `edgeos`, `fortigate_60f3g4g`, `pfsense_frr-bgp` (FRR's agentx BGP4-MIB); VyOS uses FRR the same way but has no recording |
|
||||
| generic and every other profile | BGP4-MIB metrics exist but are off (`optional`) | no | no | a router profile turns them on with `enable_metrics` |
|
||||
| mikrotik-routeros | **nothing** | | | MIKROTIK-MIB has no BGP objects and none of the RouterOS recordings answers a BGP table; RouterOS BGP is not visible over SNMP |
|
||||
| linux-netsnmp, windows-snmp, apc-ups, printer, ubiquiti switches and APs | **nothing** | | | not routers |
|
||||
|
||||
FRR on Cumulus (`cumulus_bgp`) and Dell OS (`dnos_bgp`) answer BGP4-MIB, but no shipped profile matches those devices, so their peers are collected only once a profile for them enables the generic metrics.
|
||||
|
||||
## MAC, ARP and FDB (v1.3)
|
||||
|
||||
Read by the L2 table job for every SNMP device, from BRIDGE-MIB (`dot1dBasePortIfIndex`, `dot1dTpFdbPort`), Q-BRIDGE-MIB (`dot1qTpFdbPort`, `dot1qVlanFdbId`, `dot1qVlanStaticName`), CISCO-VTP-MIB (`vtpVlanState`, `vtpVlanName`, only with the cisco-ios flag `l2.per_vlan_context`, walking the bridge table once per VLAN with community `<community>@<vlan>` or v3 context `vlan-<vlan>`) and IP-MIB (`ipNetToPhysicalPhysAddress` for IPv4 ARP and IPv6 ND, `ipNetToMediaPhysAddress` as the IPv4-only fallback). Devices without a bridge MIB (servers, firewalls without a switch, most APs) have no FDB rows. The FDB is VLAN-aware only through Q-BRIDGE or the Cisco per-VLAN walk; BRIDGE-MIB alone gives VLAN 0.
|
||||
@@ -18,6 +18,7 @@ its **Gate log** is filled in when the gate passes. A milestone is done only whe
|
||||
| M4 | Alerting | stopping a parent fixture sends one notification naming it and its children | [M4](M4-alerting.md) | **passed**, tagged `m4` |
|
||||
| M5 | Push ingest, rollup, checks | a replayed Proxmox sample shows under its node and a 2 y query hits the 1 h tier | [M5](M5-ingest-rollup-checks.md) | **passed**, tagged `m5` |
|
||||
| M6 | Auth and ops | a viewer gets 403 on PUT (2,000-device loadtest skipped, approved deviation) | [M6](M6-auth-and-ops.md) | **passed**, tagged `m6` |
|
||||
| v1.3 | BGP peers and MAC/ARP/FDB lookup | MAC/IP lookup, BGP peers and `bgp_peer_down`, see the gate list in the doc | [v1.3](v1.3-bgp-and-l2-tables.md) | **built**, gate log filled in (not tagged) |
|
||||
| M7 | gNMI streaming | after v1; not built in this run | — | out of scope |
|
||||
|
||||
## How the work is run
|
||||
@@ -49,6 +50,14 @@ Departures from the spec are recorded in [`docs/DEVIATIONS.md`](../DEVIATIONS.md
|
||||
|
||||
- **Web push notifications (v2)**: a `webpush` receiver kind in `internal/notify` (RFC 8291 payload encryption, RFC 8292 VAPID, standard-library crypto), per-user subscriptions (needs M6 users), a service worker in the UI, and push endpoints in the outbound allow-list. Out of v1 per the spec's non-goals ("alerts go to webhooks and email only"). Owner deferred it to v2 on 2026-10-22. Until then, a webhook receiver pointed at ntfy or Gotify gives phone push.
|
||||
|
||||
- **BMC support**: SNMP profiles for Dell iDRAC, HPE iLO and Supermicro (health, temperatures, fans, PSUs, power, RAID) feeding the Health/Power/Environment tabs; a native Redfish collector later. Today, Redfish works only via an exporter pushing with a push token. Owner: "good enough for now" (2026-10-23).
|
||||
|
||||
## Post-v1 additions
|
||||
|
||||
- **Device management UI** (2026-10-23): the UI had no way to add a device. Added an admin-only "Add device" dialog on the device list (IPv6/IPv4 addresses, SNMP v1/v2c/v3, synchronous discovery with the reattach 409 flow) and an Actions menu on the device page (Edit with parents and SNMP replacement, Rediscover now, Retire; Purge now asks for the device name). New API route `POST /api/v1/devices/{slug}/retire` (admin, audited, additive). The SNMP fields are one shared component (`web/src/components/SnmpFields.tsx`). Screenshots: `docs/screenshots/devices-mgmt/`.
|
||||
|
||||
- **Profiles, profile pinning and row navigation** (2026-10-23): new shipped profiles `ubiquiti-edgeswitch`, `ubiquiti-unifi-switch`, `ubiquiti-unifi-ap` and `vyos`; discovery re-matches unpinned devices every run (and at startup) and an admin can pin a profile from the Edit dialog (DEVIATIONS 39, 40); the device header explains why a device is generic. On the device list, the interface search results and the poller health table, a click anywhere in a row (or Enter on a focused row) opens the device or graph; the name stays a real link. Screenshots: `docs/screenshots/v1.0.1/`.
|
||||
|
||||
## Review artifacts
|
||||
|
||||
- From M3 on, each UI milestone's gate captures Playwright screenshots of every page into `docs/screenshots/<milestone>/` for the user to review.
|
||||
|
||||
70
docs/milestones/v1.0.1-device-tabs.md
Normal file
@@ -0,0 +1,70 @@
|
||||
# v1.0.1: device tabs (Health, Wireless, Printer, …) and a printer fleet view
|
||||
|
||||
Owner request (2026-10-23): non-interface metrics (CPU, memory, storage, sensors, UPS, wireless, printer supplies) are
|
||||
collected but not shown anywhere. Build LibreNMS-style device tabs before production, driven by profile metadata so
|
||||
that any device class shows up without per-vendor UI code.
|
||||
|
||||
## Profile schema additions (internal/profile)
|
||||
|
||||
Per metric, all optional except `category`, which is **required for every pollable metric** (enforced by a test over all
|
||||
shipped profiles and by the loader for new ones; `imported: librenms` drafts default to `health`):
|
||||
|
||||
- `category`: `system | health | power | environment | wireless | printer | routing | optics | firewall`
|
||||
- `label`: human name ("CPU load", "Toner level"); defaults to the metric name
|
||||
- `of: <metric>`: the capacity/total this metric is a fraction of (hrStorageUsed `of: hrStorageSize`,
|
||||
prtMarkerSuppliesLevel `of: prtMarkerSuppliesMaxCapacity`, memAvailReal `of: memTotalReal`). The UI shows a percent bar,
|
||||
computed at query time (raw stays raw, per the spec).
|
||||
- `display`: `gauge` (default for gauge), `percent` (implied by `of`), `state` (implied by `type: status`), `rate`
|
||||
(implied by counter types), `text`
|
||||
- `scale_by: <metric>` (optional): multiply by another column, e.g. hrStorageUsed × hrStorageAllocationUnits for bytes
|
||||
- `sentinels: {<value>: <text>}` (optional): values shown as words instead of numbers, e.g. printer `-3: "some remaining"`,
|
||||
`-2: "unknown"`. These are polled and stored (not dropped as not_available).
|
||||
|
||||
Every shipped profile is annotated. Printer gains `prtMarkerSuppliesType` (enum) and `prtMarkerColorantValue`, collected at
|
||||
discovery as component attributes, plus the paper trays (`prtInputCurrentLevel` `of: prtInputMaxCapacity`, label
|
||||
`prtInputName`). Brother and Ricoh quirks go in small vendor profiles extending `printer` (verified against the LibreNMS
|
||||
recordings where they exist).
|
||||
|
||||
## API (additive)
|
||||
|
||||
- `GET /api/v1/devices/{slug}/metrics`
|
||||
```
|
||||
{ categories: [ { name: "health", label: "Health",
|
||||
groups: [ { metric: "hrProcessorLoad", label: "CPU load", unit: "percent", display: "gauge",
|
||||
components: [ { component: "CPU 0", value: 12.0, max: null, text: null, state: null, ts: "…" } ] } ] } ] }
|
||||
```
|
||||
Latest values from hot (one batched query per device). `max` is set for `of` metrics, `state` is `ok|warn|crit|unknown`
|
||||
for enums (mapping in the profile `enum_state` or a default by name), and `text` is used for sentinels and text displays.
|
||||
Categories appear only when the device has data in them, in the order system, health, power, environment, wireless, printer, routing, optics, firewall.
|
||||
- `GET /api/v1/devices/{slug}/metrics/graph?metric=&component=&from=&to=&step=` returns `{unit, display, series: [[unix_s, value|null]], max_series?}`
|
||||
for gauges (raw), with counters as a rate, using the same step rules as `/graph`. Percent displays return percent.
|
||||
- `GET /api/v1/printers?page=` (viewer): the fleet view, with
|
||||
`items: [{device{slug,name,site}, reachability, pages_total, supplies: [{component, type, color, percent|null, text|null, state}], trays: [...]}]`,
|
||||
sorted by the lowest supply percent first.
|
||||
- Alerts: add `printer_supply_empty` (critical, level 0 or "empty"). Record a "supply replaced" audit event when a supply's
|
||||
percent rises by 20 points or more between polls.
|
||||
|
||||
## UI
|
||||
|
||||
Device page tabs: **Overview** (header, small CPU/memory/storage summary, active alerts) · **Ports** (today's table) ·
|
||||
then one tab per category the device has (Health, Power, Environment, Wireless, Printer, Routing, Optics, Firewall).
|
||||
|
||||
- Generic renderers, chosen by `display`:
|
||||
- gauge: value, unit and a sparkline
|
||||
- percent: bar with value/max
|
||||
- state: coloured badge, also shown by icon or shape so it doesn't rely on colour alone
|
||||
- rate: value per second
|
||||
- text: plain text
|
||||
|
||||
Clicking a row opens the metric graph, a TrafficGraph-like component for single series. Components are grouped by metric.
|
||||
- Bespoke panels:
|
||||
- **Printer**: a card per supply with a colour swatch (C/M/Y/K from colorant), a percent bar and "some remaining" text,
|
||||
paper trays, and a page count graph.
|
||||
- **Wireless**: per-SSID table (clients, channel, tx power) with totals, and per-radio utilisation bars.
|
||||
- New nav page **Printers** (fleet), sorted by the lowest supply.
|
||||
- Viewers see everything; nothing here mutates.
|
||||
|
||||
## Guards
|
||||
|
||||
- **Orphan test:** every pollable metric in every shipped profile has a category, and its unit and display have a renderer.
|
||||
- **Coverage matrix:** `docs/coverage.md` maps the spec's "What to collect" table to what each profile collects today, with gaps listed plainly.
|
||||
113
docs/milestones/v1.1-alert-policies.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# v1.1: alert policies (alerts fire only where a policy says so)
|
||||
|
||||
Owner request (2026-10-23): preprod alerts on everything. Rules exist, but each one applies to every device and port,
|
||||
so a WiFi port that always sees errors and a fiber uplink, where one error matters, are judged by the same rule.
|
||||
The owner's decisions:
|
||||
|
||||
- **Nothing fires without a policy.** This covers metric rules, reachability (`device_down`, `device_warning`) and
|
||||
nmon's own system alarms. A fresh install records no alerts until an admin adds policies.
|
||||
- **Scope is groups + matchers.** Matchers pick ports and devices by attribute (so new ports are covered
|
||||
automatically), and groups handle the hand-curated exceptions.
|
||||
- **Policies live in the database** (a new bun migration) and are edited through the API and UI. They apply live, with no
|
||||
restart. Rule *definitions* (the MetricsQL, templates) stay in config as today.
|
||||
|
||||
## Model
|
||||
|
||||
A **policy** binds one rule to a scope:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `id` | ULID |
|
||||
| `name` | unique, human ("Fiber uplinks: errors") |
|
||||
| `rule` | a metric rule name (`interface_errors`, …), `device_down`, `device_warning`, or `system` (all subsystem alarms, `alert.SystemRule`) |
|
||||
| `action` | `alert` or `ignore`. An `ignore` policy that matches wins over any `alert` policy of the same priority or lower. |
|
||||
| `priority` | int, default 0. For one entity and rule, the matching policy with the highest priority decides; on a tie `ignore` beats `alert`, then the lowest name wins. |
|
||||
| `enabled` | bool |
|
||||
| `severity`, `threshold`, `for` | optional overrides (null = the rule's). Threshold resolution: policy > the rule's group override > profile `ThresholdFor` > rule default. |
|
||||
| `scope` | see below; must be non-empty (`all: true` is the explicit "everything") |
|
||||
|
||||
**Scope** fields are ANDed together; within a list, any entry may match. An empty field is unconstrained.
|
||||
|
||||
- `all: bool`
|
||||
- `groups: [group id]`. A port matches if it is a member now (`GroupMember.Covers`); a device matches if any of its ports is.
|
||||
- `devices`: `slugs` (globs), `profiles`, `roles`, `sites`, `kinds`
|
||||
- `interfaces`: `if_types` (ints, e.g. 6 ethernetCsmacd, 71 ieee80211), `if_name` (RE2), `if_alias` (RE2),
|
||||
`min_speed_mbps`/`max_speed_mbps`, `alert_down` (true = the port's bell or a group's alert_down is on; this keeps the
|
||||
existing opt-in UI meaningful)
|
||||
|
||||
An interface matcher on a device-level rule (`cpu_high`, `device_down`) never matches, and validation refuses it. A
|
||||
`system` policy accepts only `all: true`, because system alarms have no device.
|
||||
|
||||
The examples the owner gave:
|
||||
|
||||
```
|
||||
{name: "Fiber + uplinks: errors", rule: interface_errors, action: alert, threshold: 0,
|
||||
scope: {interfaces: {if_types: [6], if_alias: "(?i)fib|uplink|core"}}}
|
||||
{name: "WiFi: no error alerts", rule: interface_errors, action: ignore, priority: 10,
|
||||
scope: {interfaces: {if_types: [71]}}}
|
||||
```
|
||||
|
||||
## Packages
|
||||
|
||||
- `internal/alertpolicy` (new, pure: no store, no SQL): `Policy`, `Scope`, `Validate`, `Subject` (device attrs, iface
|
||||
attrs, group ids) and `Decide(policies, rule, Subject) (Binding, bool)`. Unit tests carry the precedence table.
|
||||
- `internal/store`: migration `20261023000012_alert_policies.go` (table `alert_policies`, scope as JSON text, frozen
|
||||
struct copy), plus a sub-interface `AlertPolicies` in `alert_policies.go` (List/Get/Create/Update/Delete; unique name ->
|
||||
ErrConflict) and one embed line in `store.go`.
|
||||
- `internal/alert`: `Options.Policy func(Entity) (alertpolicy.Binding, bool)`. A series without a binding is skipped,
|
||||
and its open row closes silently as `retired`, like the existing opt-in. The binding's threshold, severity and for
|
||||
apply. `Rule.OptIn` and `Options.OptedIn` are superseded by the `alert_down` matcher. Keep the yaml field so old
|
||||
configs still parse, but make it a no-op and say so in the doc comment.
|
||||
- `internal/reach`: `Options.Wants func(rule, slug string) bool`. No binding means no new row, and an open row closes
|
||||
quietly.
|
||||
- `internal/app`: a policy cache, loaded at start, refreshed every 30 s and invalidated on every API write. It builds the
|
||||
`Subject` from TTL-cached device and interface lookups and gates `sysAlarms` on a `system` policy.
|
||||
- `internal/api`, in a new file:
|
||||
- `GET/POST /api/v1/alert-policies` and `GET/PUT/DELETE /api/v1/alert-policies/{id}` (admin).
|
||||
- `POST /api/v1/alert-policies/preview` (viewer, read-only) returns the counts of devices and ports a draft scope covers,
|
||||
with up to 20 samples of each.
|
||||
- `GET /api/v1/alert-rules` (viewer) lists the rule names, their level (`device`, `interface`, `component`, `system`),
|
||||
default threshold, op and severity, for the UI dropdown.
|
||||
- All additive; regenerate `openapi.json` and the web schema.
|
||||
- `web/`: an **Alert policies** page under Alerts (list, create and edit form with a live preview count, enable toggle;
|
||||
`RequireAdmin` for mutation). It has a "Start from template" menu with UI-side constants: all devices `device_down`,
|
||||
bell-opted ports `interface_down`, uplinks `interface_errors`, wireless ignore `interface_errors`, `system` all. The
|
||||
bell tooltip changes to say that a policy using the `alert_down` matcher is needed. Add a mock and an e2e test.
|
||||
|
||||
## Upgrade behaviour
|
||||
|
||||
No policies are seeded. On the first cycle after upgrade, every open alert without a binding closes silently as
|
||||
`retired` and sends no "resolved" notifications, which is how the preprod flood clears. The startup log warns once when
|
||||
the policy table is empty ("no alert policies: nothing will alert").
|
||||
|
||||
## Docs
|
||||
|
||||
- DEVIATIONS: a new row, "Default rules apply everywhere; spec §Alerting", which this change replaces with policy-gated
|
||||
rules (owner decision 2026-10-23). It supersedes #38's mechanism but keeps the bell.
|
||||
- `docs/operations.md`: an alert policies section with the two examples above.
|
||||
|
||||
## Gate
|
||||
|
||||
- `make lint` is clean, and `go test -race ./...`, `-tags 'integration vmembed'` for `internal/alert`, `internal/reach`,
|
||||
`internal/app` and `internal/store` all pass, plus `./scripts/test-pg.sh`, `scripts/openapi-check.sh` and the web
|
||||
lint/test/build/e2e.
|
||||
- Tests prove the following:
|
||||
- With no policies, nothing is recorded (metric, reach and system).
|
||||
- The WiFi ignore and fiber alert pair decides a 71 port and a 6 port correctly.
|
||||
- Removing a policy retires open rows silently.
|
||||
- Policy threshold beats group override.
|
||||
- A policy write takes effect on the next cycle without a restart.
|
||||
|
||||
## Gate log
|
||||
|
||||
2026-10-23, run by the implementing agent:
|
||||
|
||||
- `make lint`: 0 Go issues, filesize script clean, ESLint `--max-warnings 0` clean.
|
||||
- `go test -race ./...`: all 43 packages ok.
|
||||
- `go test -race -tags 'integration vmembed' ./internal/alert/... ./internal/reach/... ./internal/app/... ./internal/store/...`: ok.
|
||||
- `./scripts/test-pg.sh` (store suite on Postgres 16, includes migration 12): ok.
|
||||
- `scripts/openapi-check.sh`: compatible with m6 (additive only). `openapi.json` and `web/src/api/schema.d.ts` regenerated.
|
||||
- Web: `npm run lint`, `npm run test` (140 tests) and `npm run build` pass. `E2E_PORT=4791 npm run e2e`: 149 passed, 6 failed. The 6 failures are device-page specs (`device-tabs`, `devices-mgmt` edit, `profiles-nav` pin, `screenshots-v101` overview, `screenshots` 32-bit warning, `ui` warning explanation) that look for the `summary-tile`, `device-header`, `generic-hint` and `device-warnings` test ids, which no longer exist in `web/src` since commit 992f09b (Overview tab). They are not caused by this change and were left alone. The new `alert-policies.spec.ts` (4 tests) and the updated `alert-down.spec.ts` pass. Screenshot dirs were reverted.
|
||||
- `make gate-m4`: the whole backend run passes (45 PASS, including device_down, suppression, restart, ack and maintenance with the two gate-created policies). It then fails in its UI steps for the same stale test ids as above (mock UI specs, plus the live `device-down` screenshot waiting for `device-header`). The first run of the gate caught a missing `Wants:` wiring in `internal/app/alerting.go` (no device_down was recorded); fixed.
|
||||
- `gate-m5.sh` and `gate-m6.sh` now create a `system` policy before the alarms they expect; only `bash -n` was run on them (not the full ~10 min gates).
|
||||
- The 2,000-device load test and `make gate-m5`/`gate-m6` were not run.
|
||||
229
docs/milestones/v1.3-bgp-and-l2-tables.md
Normal file
@@ -0,0 +1,229 @@
|
||||
# v1.3: BGP peers and MAC/ARP/FDB lookup
|
||||
|
||||
Owner request (2026-10-24), after the LibreNMS comparison: **MAC/ARP/FDB lookup is a must, and BGP peer monitoring is
|
||||
wanted.** WISP radio profiles are nice-to-have and not in this milestone.
|
||||
|
||||
Parts of this go beyond the spec. The spec lists BGP-4-MIB in the collection table but no profile fills it, and it
|
||||
names no MAC/ARP lookup feature. Add one DEVIATIONS row per feature.
|
||||
|
||||
## Part A: MAC / ARP / FDB ("where is this host plugged in?")
|
||||
|
||||
### Collection
|
||||
A new **L2 table job**, separate from the 60 s poll path (the spec forbids table walks on the hot path) and from the
|
||||
twice-daily discovery:
|
||||
|
||||
- It walks every enabled SNMP device every `l2.interval` (default 15 min, minimum 5 min, in `config.yaml`; applied on
|
||||
SIGHUP like the other job settings, or `restart_required` if that is simpler, and say which).
|
||||
- It has its own concurrency cap and a per-device deadline, and yields to the poller under the same admission policy as
|
||||
rollup.
|
||||
- A device that times out is skipped for that cycle, with a counter and a poller-health note; it never blocks the job.
|
||||
|
||||
Tables, all numeric OIDs verified against `mibs/`, each with a provenance comment:
|
||||
|
||||
| What | Source | Notes |
|
||||
|---|---|---|
|
||||
| Bridge port → ifIndex | BRIDGE-MIB `dot1dBasePortIfIndex` | needed for every FDB row |
|
||||
| FDB, VLAN-aware | Q-BRIDGE-MIB `dot1qTpFdbPort` (index: fdbId.mac) | Map fdbId to the VLAN with `dot1qVlanFdbId` where present; otherwise fdbId = VLAN id. |
|
||||
| FDB, fallback | BRIDGE-MIB `dot1dTpFdbPort` | used when Q-BRIDGE returns nothing; the VLAN is unknown (0) |
|
||||
| Cisco per-VLAN FDB | BRIDGE-MIB per VLAN | IOS only: list VLANs from CISCO-VTP-MIB `vtpVlanState`, then walk with community `<community>@<vlan>` (v2c) or context `vlan-<vlan>` (v3). Turn it on with a profile flag (`l2.per_vlan_context: true` on cisco-ios), not by detection. |
|
||||
| VLAN names | Q-BRIDGE-MIB `dot1qVlanStaticName`, CISCO-VTP-MIB `vtpVlanName` | shown in the UI |
|
||||
| ARP and IPv6 ND | IP-MIB `ipNetToPhysicalPhysAddress` (both IPv4 and IPv6, IPv6 first) | This is the primary source. Fall back to `ipNetToMediaPhysAddress` (IPv4 only) when it is empty. Skip entries of type invalid or incomplete. |
|
||||
|
||||
Recordings to verify against: `/home/svcuser/src/librenms/tests/snmpsim` (pick ones with these tables for cisco-ios,
|
||||
junos, arista, edgeswitch, unifi-switch, mikrotik and a Linux bridge). Read them as reference data only; copy no code or
|
||||
YAML from that GPL repository. Add `.snmprec` fixtures under `testdata/snmprec/` produced by our own generator (extend
|
||||
`internal/fixturegen`), including one with Q-BRIDGE, one with only BRIDGE-MIB, one Cisco per-VLAN, and an ARP table with
|
||||
IPv6 and IPv4 entries.
|
||||
|
||||
### Storage (inventory DB, not the metric stores)
|
||||
Use migration `20261024000014_l2_tables.go`. Number 13 is taken by the concurrent v1.2 work; take 14 and nothing else.
|
||||
|
||||
- `fdb_entries`: device_id, interface_id (ULID, nullable when the bridge port has no ifIndex), vlan, mac (normalised
|
||||
lower-case `aa:bb:cc:dd:ee:ff`), first_seen, last_seen.
|
||||
- Unique on (device_id, interface_id, vlan, mac).
|
||||
- A row seen again only bumps last_seen.
|
||||
- Use batched upserts: a 48-port switch can carry thousands of MACs.
|
||||
- `arp_entries`: device_id, interface_id (nullable), ip (canonical text, v6 compressed), mac, first_seen, last_seen.
|
||||
Unique on (device_id, ip, mac).
|
||||
- `vlans`: device_id, vlan, name. Replaced per walk.
|
||||
- Retention is `l2.keep` (default 90 d) on last_seen, pruned daily. This history is what lets you ask "where was this MAC
|
||||
last week".
|
||||
- Add the new tables to `backupTables`, or mark them deliberately transient with a reason. Back them up.
|
||||
- Only `internal/store` touches SQL. Add a store sub-interface in a new file plus one embed line in `store.go`. Every
|
||||
query must be indexed: lookups are by mac, by ip, by interface and by device.
|
||||
|
||||
### Trunk awareness
|
||||
A MAC is seen on every switch it passes through. For each FDB row, compute at query time the **MAC count of its port**
|
||||
(per walk), and mark a port an **uplink** when it has more than `l2.edge_max_macs` MACs (default 8), when it is an LLDP
|
||||
neighbour link, or when it is a LAG/parent. The lookup ranks **edge ports first**: the port with the fewest MACs, then
|
||||
the most recent last_seen. That is where the host is plugged in.
|
||||
|
||||
### API (additive)
|
||||
- `GET /api/v1/l2/search?q=` (viewer):
|
||||
- `q` is a MAC in any common format (`aabb.ccdd.eeff`, `aa-bb-…`, `aabbccddeeff`, partial prefixes of at least 6 hex
|
||||
digits), an IPv6 or IPv4 address, or a CIDR.
|
||||
- It returns grouped hits:
|
||||
`{ mac, ips: [{ip, device, last_seen}], locations: [{device, iface, vlan, vlan_name, edge: bool, port_mac_count, first_seen, last_seen}] }`,
|
||||
edge locations first, paged like the other routes.
|
||||
- `GET /api/v1/devices/{slug}/fdb` and `GET /api/v1/devices/{slug}/arp` (viewer): the paged tables, filterable by VLAN,
|
||||
interface and `current` (seen in the last walk) or with history.
|
||||
- `GET /api/v1/interfaces/{id}/macs` (viewer): the MACs on a port, current and history, each with its IPs from any ARP
|
||||
table.
|
||||
|
||||
### UI
|
||||
- A global search box (existing header search, or a new **Find host** page): paste a MAC or IP and get the answer
|
||||
"device X port Y, VLAN Z, last seen 3 min ago", with the full path (uplinks) collapsed below it.
|
||||
- On the device page, a new **L2** tab (FDB with a VLAN filter, ARP, VLAN list), shown only when the device has rows.
|
||||
- On the port, a MAC count on the Ports tab row with a link to that port's MACs.
|
||||
|
||||
### OUI vendor lookup (optional; nmon must work fully without it)
|
||||
Owner request (2026-10-24): show the vendor of each MAC, from the IEEE registry where it can be fetched. It is nice to
|
||||
have and never required.
|
||||
|
||||
- **Source:** the IEEE public registries MA-L (`https://standards-oui.ieee.org/oui/oui.csv`), MA-M (`…/oui28/mam.csv`) and
|
||||
MA-S (`…/oui36/oui36.csv`). Check the current URLs and CSV columns before coding against them.
|
||||
- **Fetch:**
|
||||
- a background job at start (if no database is present) and then every `oui.refresh` (default 7 d);
|
||||
- through `internal/netpolicy`, so the outbound allow-list applies;
|
||||
- with timeouts, a size cap, and a parse sanity check (a minimum row count) before replacing the current file.
|
||||
- `oui.fetch: false` turns fetching off.
|
||||
- **Offline:** `nmon oui import <dir-or-file>` loads the same CSVs from disk. The Ansible role can optionally download
|
||||
and ship them next to the MIB sidecar.
|
||||
- **Storage:** a compact file in the data dir (e.g. `oui/registry.bin`, or a gzipped normalised CSV), written atomically
|
||||
with a checksum, and included in `nmon backup`. It is not in the inventory DB, since it is reference data. Loaded into
|
||||
memory as three prefix maps (24-, 28- and 36-bit); a lookup takes the longest match.
|
||||
- **Missing or stale data never fails anything.** With no file, vendor is empty and a `oui_unavailable` note appears on
|
||||
poller health. There is no alert.
|
||||
- **Vendor flag:**
|
||||
- computed at query time and never stored per FDB or ARP row, so a refresh relabels history;
|
||||
- API hits and table rows gain `vendor` (string, empty when unknown) and `mac_kind`: `global`, `local` (the
|
||||
locally-administered bit, typical of randomised phone and VM MACs, with vendor shown as "Private / randomised"),
|
||||
or `multicast`.
|
||||
- **Search:** `q` may also be a vendor name substring (`vendor:cisco`). It returns the current MACs of that vendor,
|
||||
paged, so "find every Ubiquiti device on VLAN 20" works.
|
||||
- **UI:** a vendor column on the L2 tab, the port MAC list and the search results, plus a small "private" badge for
|
||||
`local`.
|
||||
- **Licensing:** the IEEE registry is public data. Note its source in `docs/operations.md`, and do not commit the CSVs
|
||||
to the repo. Tests use a tiny generated fixture registry.
|
||||
- **Tests:**
|
||||
- the longest prefix wins (an MA-S entry beats MA-L);
|
||||
- the local and multicast bits are detected;
|
||||
- a truncated or garbage download keeps the previous file;
|
||||
- no network and no file still lets the L2 job and search work, with an empty vendor.
|
||||
|
||||
## Part B: BGP peers
|
||||
|
||||
### Collection (profiles, 5 min interval; metrics in the `routing` category)
|
||||
The component label is the peer address (canonical, IPv6 compressed). Peer AS, local AS, description/remote router ID
|
||||
and AFI/SAFI are **component attributes** (migration 11's mechanism) collected at discovery, never metric labels. The
|
||||
SNMP label contract stays `device` + `component` only. Where a vendor reports prefixes per address family, the
|
||||
component is `<peer>|<afi>` for the prefix metrics only; document it.
|
||||
|
||||
| Source | Gives | Profiles |
|
||||
|---|---|---|
|
||||
| BGP4-MIB `bgpPeerState`, `bgpPeerFsmEstablishedTime`, `bgpPeerInUpdates`/`OutUpdates`, `bgpPeerRemoteAs` | IPv4 peers only | generic (optional, enabled on router profiles) |
|
||||
| CISCO-BGP4-MIB `cbgpPeer2State`, `cbgpPeer2FsmEstablishedTime`, `cbgpPeer2AcceptedPrefixes` (per AFI/SAFI), `cbgpPeer2RemoteAs` | IPv4 and IPv6 | cisco-ios, cisco-nxos |
|
||||
| BGP4-V2-MIB-JUNIPER `jnxBgpM2PeerState`, `jnxBgpM2PeerFsmEstablishedTime`, `jnxBgpM2PrefixInPrefixesAccepted` | IPv4 and IPv6 | juniper-junos |
|
||||
| ARISTA-BGP4V2-MIB `aristaBgp4V2PeerState`, `…PrefixInPrefixesAccepted` | IPv4 and IPv6 | arista-eos |
|
||||
|
||||
- `state` is a `status` metric with enum `idle connect active opensent openconfirm established`;
|
||||
`enum_state: established ok, others crit`.
|
||||
- Use the vendor table where one exists, and drop the BGP4-MIB duplicate on that profile, so each peer appears once.
|
||||
- VyOS/FRR, MikroTik, FortiGate and EdgeOS: check `mibs/` and the recordings (`cumulus_bgp`, `dnos_bgp`, vyos and
|
||||
mikrotik ones). Add BGP only where the device actually answers. State per profile in `docs/coverage.md` what is
|
||||
collected, and where nothing is, say so plainly.
|
||||
- Verify against the LibreNMS recordings (`arista_eos*.snmprec`, `cumulus_bgp`, `dnos_bgp`, ios and junos ones) and add
|
||||
our own generated fixtures with IPv6 peers.
|
||||
|
||||
### UI
|
||||
- Device **Routing** tab:
|
||||
- a peers table with peer, AS, description, state badge, uptime, prefixes accepted per AFI, and update rate;
|
||||
- a click-to-graph on the prefixes and updates;
|
||||
- built from the existing metrics-tab renderers where they fit.
|
||||
- A **BGP** page in the main nav listing every peer on every device, with filters for state, AS and device, so you can
|
||||
see which sessions are down right now.
|
||||
- Back it with `GET /api/v1/bgp/peers` (viewer, paged; latest values from hot, with attributes from the inventory).
|
||||
|
||||
### Alerts
|
||||
- New built-in rule `bgp_peer_down`:
|
||||
- component-level;
|
||||
- condition: state is not established for `for: 2m`;
|
||||
- severity critical;
|
||||
- its summary names the peer, AS and description.
|
||||
- Like every rule, it fires only where a policy binds it (v1.1). Add a template "Routers: BGP peer down" to the policy
|
||||
templates.
|
||||
- A prefix-drop alert is left to the custom rule builder (v1.2). Add it to the operations doc as an example once v1.2 is
|
||||
in.
|
||||
|
||||
## Constraints
|
||||
- **v1.2 is being built concurrently.** It owns `internal/alert` rules plumbing, `internal/alertrule`, migration 13, and
|
||||
the alert-rules API and UI.
|
||||
- Do not edit those files, except to append `bgp_peer_down` to `DefaultRules()` in `internal/alert/rules.go` plus its
|
||||
integration test row, and to add the policy template constant in `web/src/lib/policyModel.ts`.
|
||||
- Rebase onto whatever v1.2 commits first, if it lands before you.
|
||||
- Regenerate `openapi.json` and `web/src/api/schema.d.ts` just before committing.
|
||||
- Every port is still polled the same way; this adds a job and does not change the poller.
|
||||
- IPv6 first: ND and IPv6 BGP peers are first-class, and tests default to `::1`.
|
||||
|
||||
## Gate
|
||||
- `make lint` clean. All of these pass:
|
||||
- `go test -race ./...`
|
||||
- `go test -race -tags 'integration vmembed'` for the touched packages
|
||||
- `./scripts/test-pg.sh` and `scripts/openapi-check.sh`
|
||||
- `NMON_LIBRENMS_DIR=/home/svcuser/src/librenms go test ./internal/profile/... ./internal/fixtures/...`
|
||||
- `go run ./testdata/gen` with no drift
|
||||
- web lint, test and build, plus only the new and changed Playwright specs
|
||||
- Tests prove the following:
|
||||
- MAC normalisation accepts every format above and refuses garbage.
|
||||
- A MAC seen on an access port and on two uplinks resolves to the access port first.
|
||||
- The Cisco per-VLAN context walk is used only when the profile flag is set.
|
||||
- IPv6 ND entries are stored and searchable.
|
||||
- Retention prunes by last_seen.
|
||||
- An IPv6 BGP peer from a vendor table shows once (no BGP4-MIB duplicate).
|
||||
- `bgp_peer_down` fires through a policy and resolves on established.
|
||||
- The L2 job never runs on the poll path, and a slow device does not delay the others.
|
||||
|
||||
## Gate log
|
||||
|
||||
Built on branch `worktree-agent-a86e97dea1f9eae40` (from main at `06a3c85`), 2026-10-24. Run by the implementing agent; the
|
||||
supervisor reruns the gates after merging with v1.2.
|
||||
|
||||
### Verification
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `make lint` (golangci-lint v2.14.0, `scripts/lint-filesize.sh`, ESLint `--max-warnings 0`) | 0 issues |
|
||||
| `go test -race ./...` | all packages pass |
|
||||
| `go test -race -tags 'integration vmembed'` on alert, api, app, backup, l2, discovery, fixtures, profile, vmsup | pass (the alert integration test runs `bgp_peer_down` against a real VictoriaMetrics: fires for the active IPv6 peer, not for the established one) |
|
||||
| `./scripts/test-pg.sh` (Postgres 16, store suite incl. the L2 queries, upserts and backup/restore table list) | pass |
|
||||
| `scripts/openapi-check.sh` | compatible with m6 (additive only) |
|
||||
| `NMON_LIBRENMS_DIR=/home/svcuser/src/librenms go test ./internal/profile/... ./internal/fixtures/... ./internal/l2` | pass: real recordings produce each BGP peer exactly once (IOS XE and XR with IPv6 peers, Junos MX, Arista, EdgeOS, FortiGate, FRR on pfSense), no type mismatches; the L2 walker reads FDB and ARP/ND (IPv6 from `arista_eos_7280r` and `dell-os10`) from the Arista, Junos, IOS, UniFi/EdgeSwitch and RouterOS recordings |
|
||||
| `go run ./testdata/gen` | no drift (`TestCommittedFixturesAreCurrent` passes); 9 expect files gained only BGP series, PDU budgets unchanged |
|
||||
| `cd web && npm run lint && npm run test && npm run build` | clean; 165 tests pass |
|
||||
| Playwright, only the new `e2e/specs/l2-bgp.spec.ts` (12 tests), plus `device-tabs` and `profiles-nav` as a regression check of the pages this touches | 27 passed; `docs/screenshots` reverted; no preview server left running |
|
||||
|
||||
Not run: the `make gate-m*` scripts, the Ansible tasks (no Ansible here; the YAML parses), and the full Playwright suite.
|
||||
|
||||
### What proves each gate line
|
||||
|
||||
| Line | Test |
|
||||
|---|---|
|
||||
| MAC normalisation accepts every format and refuses garbage | `internal/hostaddr` `TestParseMACAcceptsEveryCommonFormat`, `TestParseMACRefusesGarbage`, `TestClassifySortsEveryQueryShape`; API `TestSearchAcceptsEveryMACFormatAndPrefixAndRefusesGarbage` |
|
||||
| A MAC on an access port and two uplinks resolves to the access port first | `internal/api` `TestAMACSeenOnAnAccessPortAndTwoUplinksResolvesToTheAccessPortFirst` (uplinks by MAC count and by LLDP), `TestEdgeLimitComesFromTheLiveSetting` |
|
||||
| The Cisco per-VLAN walk is used only with the profile flag | `internal/l2` `TestCiscoPerVLANContextIsUsedOnlyWhenTheProfileFlagIsSet` (dial log: no `@vlan` community without the flag); `internal/profile` `TestOnlyIOSWalksBridgeTablesPerVLAN` |
|
||||
| IPv6 ND entries are stored and searchable | `internal/l2` `TestIPv6NeighboursAreStoredAndSearchable`; store `TestL2SearchFindsMACsByPrefixIPv6AndCIDR`; API `TestIPv6AndIPv4AndCIDRSearchFindTheMAC` |
|
||||
| Retention prunes by last_seen | store `TestPruneL2DeletesByLastSeen`; l2 `TestRetentionPrunesByLastSeenOnTheDailyPass` |
|
||||
| An IPv6 BGP peer from a vendor table shows once | `internal/profile` `TestVendorBGPTablesReplaceTheBGP4MIBOnesSoAPeerAppearsOnce`; fixtures (every vendor fixture lists `2001:db8::2` once); the real-recording `checkPeers` |
|
||||
| `bgp_peer_down` fires through a policy and resolves on established | `internal/alert` `TestBGPPeerDownFiresThroughAPolicyAndResolvesOnEstablished`, `TestBGPPeerDownFiresNowhereWithoutAPolicy`; integration row |
|
||||
| The L2 job never runs on the poll path; a slow device does not delay the others | `internal/l2` (every dial is path `l2`; a poll session refuses walks, `internal/snmp`), `TestSlowDeviceTimesOutWithoutDelayingTheOthers`, `TestJobYieldsWhileTheWriteQueueIsMoreThanHalfFull` |
|
||||
| OUI: longest prefix wins; local and multicast bits; truncated or garbage download keeps the previous file; no network and no file still works | `internal/oui` `TestLongestPrefixWins`, `TestVendorFlagsLocalAndMulticast`, `TestTruncatedOrGarbageDownloadKeepsThePreviousFile`, `TestMissingRegistryNeverBreaksALookup`; API `TestVendorSearchListsCurrentMACsAndWorksEmptyWithoutARegistry` |
|
||||
|
||||
### Decisions where the spec was silent
|
||||
|
||||
- **L2 settings and reload.** `l2.*` and `oui.*` apply from the next cycle after a reload (SIGHUP or a Settings save) with no restart, and are reported as `applied`. The job re-reads `l2.interval` every minute while waiting, so a shorter interval takes effect within a minute and a longer one on the wait in progress. Test: `TestRunStartsACycleAtOnceThenEveryIntervalAndReadsTheIntervalAgain`.
|
||||
- **Ranking.** Edge before uplink, then rows the last walk still sees before history, then fewest MACs, then most recent (DEVIATIONS 55).
|
||||
- **Per-VLAN fallback.** If the first VLAN context does not answer, the walker falls back to the plain tables instead of waiting out a timeout per VLAN.
|
||||
- **`bgp_peer_down` text.** AS and description are added to the notification, not the stored alert row (DEVIATIONS 53).
|
||||
- **BGP per profile.** See `docs/coverage.md`. Nothing for mikrotik-routeros (no BGP objects in MIKROTIK-MIB or the recordings) and the non-router profiles; cisco-nxos is unverified against a recording; VyOS is covered through BGP4-MIB on the strength of the FRR recording only.
|
||||
- **Open item for v1.2.** The prefix-drop example in `docs/operations.md` is to be added once custom rules land.
|
||||
|
||||
@@ -59,6 +59,27 @@ Edit in Settings (admin) or `config.yaml` + `SIGHUP`. An invalid document is rej
|
||||
|
||||
The supervisor restarts a crashed store; after three restarts in ten minutes it raises `store_failed` and stops. `nmon store restart [hot|long] --api URL --token T` (or `SIGHUP`) resumes it.
|
||||
|
||||
## Interface-down alerts are opt-in
|
||||
|
||||
Since v1.1 the bell and the group switch only **mark** ports: `interface_down` alerts for them only where an alert policy for `interface_down` uses the `alert_down` matcher (template "Bell-marked ports: interface_down"). Mark ports in one at a time (the bell on the device page's interface table), all at once (the "Alert on down" control in the interfaces card header, or `POST /api/v1/devices/{slug}/interfaces/alert-down`), or through a group (the group's "Alert when member ports go down" switch; `PUT /api/v1/interfaces/{id}` with `{"alert_down": true}` for one port). A port in a group with the switch on alerts for as long as its membership is in force. Changes reach the evaluator at once; opting a port out closes its open alert silently as `retired`.
|
||||
|
||||
## Alert policies: nothing alerts until you add one
|
||||
|
||||
Alerts fire only where an **alert policy** binds a rule to a scope. A fresh install has no policies and records no alerts at all: no metric rule, no `device_down` or `device_warning`, no system alarm (ingest caps, late p95, rollup, disk guard, break-glass, backup). The startup log says so once ("no alert policies: nothing will alert"). Upgrading from v1.0 seeds nothing: the first cycle closes every open alert that no policy binds, silently, as `retired` (no "resolved" notices go out).
|
||||
|
||||
Manage policies under Alerts > Alert policies (admins edit, viewers read) or with `GET/POST /api/v1/alert-policies` and `GET/PUT/DELETE /api/v1/alert-policies/{id}`. `GET /api/v1/alert-rules` lists the rule names. `POST /api/v1/alert-policies/preview` counts the devices and ports a draft scope covers. Changes apply on the next cycle with no restart. Removing or disabling a policy retires its open alerts quietly.
|
||||
|
||||
A policy has a rule (a metric rule, `device_down`, `device_warning`, or `system`), an action (`alert` or `ignore`), a priority, optional `severity`, `threshold` and `for` overrides, and a scope. Scope fields are ANDed and any entry of a list may match: `all`, `groups` (a port is in force as a member now; a device matches if any port does), `devices` (`slugs` globs, `profiles`, `roles`, `sites`, `kinds`) and `interfaces` (`if_types`, `if_name` and `if_alias` as RE2, `min_speed_mbps`, `max_speed_mbps`, `alert_down`). For one entity and rule the matching policy with the highest priority decides; on a tie `ignore` beats `alert`, then the lowest name. A policy threshold beats a rule's group override, the profile threshold and the default. Interface matchers are refused on device-level rules, and a `system` policy takes only `all: true`.
|
||||
|
||||
Fiber and uplinks alert on any error, WiFi never alerts on errors:
|
||||
|
||||
```
|
||||
{"name": "Fiber + uplinks: errors", "rule": "interface_errors", "action": "alert", "threshold": 0,
|
||||
"scope": {"interfaces": {"if_types": [6], "if_alias": "(?i)fib|uplink|core"}}}
|
||||
{"name": "WiFi: no error alerts", "rule": "interface_errors", "action": "ignore", "priority": 10,
|
||||
"scope": {"interfaces": {"if_types": [71]}}}
|
||||
```
|
||||
|
||||
## Retired devices
|
||||
|
||||
`POST /api/v1/devices/{slug}/purge` (admin) deletes a retired device's series from both stores once it has been retired for `retired_purge_after` (default 1y).
|
||||
@@ -66,3 +87,35 @@ The supervisor restarts a crashed store; after three restarts in ten minutes it
|
||||
## CSV import
|
||||
|
||||
Devices (`name,address,site,role,credential,parents,...`) and port descriptions (`device,ifname,alias`) from Settings or `POST /api/v1/devices/import`. Port descriptions are stored in the operator-owned `description` field, which discovery never overwrites. A dependency cycle is rejected with its path; nothing is written unless the whole import is valid.
|
||||
|
||||
## Find host: MAC, ARP and FDB lookup
|
||||
|
||||
The **L2 table job** walks the bridge forwarding tables, VLAN names and ARP / IPv6 neighbour tables of every SNMP device every `l2.interval` and keeps them with first and last seen in the inventory, so the **Find host** page (and the header search, for a MAC, IP, CIDR or `vendor:name`) answers "which switch port is this host plugged in at, in which VLAN, and when did we last see it", with the uplinks that also saw it collapsed underneath. A MAC is seen on every switch it passes through, so the answer is the **edge** port: one that carries at most `l2.edge_max_macs` MACs, has no LLDP neighbour and is not a LAG.
|
||||
|
||||
```yaml
|
||||
l2:
|
||||
interval: 15m # default 15m, minimum 5m
|
||||
keep: 90d # history kept by last_seen, pruned daily
|
||||
edge_max_macs: 8 # a port with more MACs is an uplink
|
||||
concurrency: 4 # devices walked at once
|
||||
device_timeout: 2m # a slower device is skipped for the cycle
|
||||
oui:
|
||||
fetch: true # IEEE vendor registry; false never touches the network
|
||||
refresh: 7d
|
||||
```
|
||||
|
||||
All of these apply from the next cycle after a reload (Settings save or SIGHUP) with no restart. The job never runs on the 60 s poll path and yields to the poller the way the rollup job does. A device that does not answer in time is skipped for that cycle and named on the Poller page (`l2` status and an `l2_devices_skipped` note); it never delays the other devices. A Cisco IOS switch keeps one bridge table per VLAN: the `cisco-ios` profile sets `l2.per_vlan_context: true`, which lists the VLANs from CISCO-VTP-MIB and walks each with community `<community>@<vlan>` (v3: context `vlan-<vlan>`). Other devices use Q-BRIDGE-MIB, then BRIDGE-MIB (VLAN unknown).
|
||||
|
||||
API: `GET /api/v1/l2/search?q=`, `GET /api/v1/devices/{slug}/{l2,fdb,arp,l2/ports}`, `GET /api/v1/interfaces/{id}/macs`. On a device, the **L2** tab (MAC table with a VLAN filter, ARP / ND, VLAN list) appears once the job has read it, and the Ports tab shows each port's MAC count linking to that port's MACs.
|
||||
|
||||
## MAC vendors (optional IEEE registry)
|
||||
|
||||
If a registry is loaded, each MAC shows its vendor, a locally administered address (typical of randomised phone and VM MACs) shows a "private" badge, and `vendor:ubiquiti` lists the current MACs of a vendor. nmon works fully without it: the vendor is simply empty, and the Poller page notes `oui_unavailable` (not an alert).
|
||||
|
||||
The registry is the IEEE's public data: MA-L (`https://standards-oui.ieee.org/oui/oui.csv`), MA-M (`.../oui28/mam.csv`) and MA-S (`.../oui36/oui36.csv`), downloaded weekly through the outbound allow-list (`standards-oui.ieee.org` must be allowed when an allow-list is configured) with timeouts, a size cap and a row-count sanity check, so a truncated or wrong download keeps the previous file. The compact result is `<data-dir>/oui/registry.gz` (checksummed, written atomically, included in `nmon backup`). Where nmon may not fetch, set `oui.fetch: false` and load the CSVs by hand: `nmon oui import <dir-or-file> --data-dir DIR` (a running nmon notices within an hour). The CSVs are not part of this repository.
|
||||
|
||||
## BGP
|
||||
|
||||
Router profiles collect BGP session state, uptime, accepted prefixes per address family and update rates every 5 minutes (see `docs/coverage.md` for which profiles and MIBs). The **BGP** page lists every session on every device, the ones that are down first, filterable by state, AS, family and text; a device's **Routing** tab shows its sessions with click-to-graph on prefixes and updates.
|
||||
|
||||
The built-in rule `bgp_peer_down` (critical, state not established for 2 minutes) fires only where a policy binds it, like every rule: use the template "Routers: BGP peer down" under Alerts > Alert policies. Its notification names the peer and, from discovery, the peer's AS and description. A prefix-drop alert is a custom rule (v1.2); an example will be added here once custom rules are in.
|
||||
|
||||
BIN
docs/screenshots/devices-mgmt/action-menu.png
Normal file
|
After Width: | Height: | Size: 131 KiB |
BIN
docs/screenshots/devices-mgmt/add-dialog-empty.png
Normal file
|
After Width: | Height: | Size: 150 KiB |
BIN
docs/screenshots/devices-mgmt/add-dialog-v3.png
Normal file
|
After Width: | Height: | Size: 150 KiB |
BIN
docs/screenshots/devices-mgmt/add-discovering.png
Normal file
|
After Width: | Height: | Size: 166 KiB |
BIN
docs/screenshots/devices-mgmt/edit-dialog.png
Normal file
|
After Width: | Height: | Size: 136 KiB |
BIN
docs/screenshots/devices-mgmt/reattach-prompt.png
Normal file
|
After Width: | Height: | Size: 157 KiB |
BIN
docs/screenshots/devices-mgmt/retire-confirm.png
Normal file
|
After Width: | Height: | Size: 128 KiB |
BIN
docs/screenshots/v1.0.1/device-generic-hint.png
Normal file
|
After Width: | Height: | Size: 68 KiB |
BIN
docs/screenshots/v1.0.1/devices-row-hover.png
Normal file
|
After Width: | Height: | Size: 204 KiB |
BIN
docs/screenshots/v1.0.1/edit-profile-select.png
Normal file
|
After Width: | Height: | Size: 90 KiB |
BIN
docs/screenshots/v1.0.1/group-alert-switch.png
Normal file
|
After Width: | Height: | Size: 57 KiB |
BIN
docs/screenshots/v1.0.1/iface-alert-toggle.png
Normal file
|
After Width: | Height: | Size: 159 KiB |
BIN
docs/screenshots/v1.0.1/metric-graph.png
Normal file
|
After Width: | Height: | Size: 86 KiB |
BIN
docs/screenshots/v1.0.1/printers-fleet.png
Normal file
|
After Width: | Height: | Size: 44 KiB |
BIN
docs/screenshots/v1.0.1/tab-health.png
Normal file
|
After Width: | Height: | Size: 83 KiB |
BIN
docs/screenshots/v1.0.1/tab-overview.png
Normal file
|
After Width: | Height: | Size: 65 KiB |
BIN
docs/screenshots/v1.0.1/tab-power-ups.png
Normal file
|
After Width: | Height: | Size: 63 KiB |
BIN
docs/screenshots/v1.0.1/tab-printer-light.png
Normal file
|
After Width: | Height: | Size: 89 KiB |
BIN
docs/screenshots/v1.0.1/tab-printer.png
Normal file
|
After Width: | Height: | Size: 89 KiB |
BIN
docs/screenshots/v1.0.1/tab-wireless-ap.png
Normal file
|
After Width: | Height: | Size: 57 KiB |
62
internal/alert/bgp_rule_test.go
Normal file
@@ -0,0 +1,62 @@
|
||||
package alert
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func bgpRule(t *testing.T) Rule {
|
||||
t.Helper()
|
||||
for _, r := range DefaultRules() {
|
||||
if r.Name == "bgp_peer_down" {
|
||||
return r
|
||||
}
|
||||
}
|
||||
t.Fatal("no bgp_peer_down in DefaultRules")
|
||||
return Rule{}
|
||||
}
|
||||
|
||||
func TestBGPPeerDownIsAComponentLevelCriticalRuleHeldForTwoMinutes(t *testing.T) {
|
||||
r := bgpRule(t)
|
||||
if r.Severity != SevCritical || r.For != 2*time.Minute || r.Level() != "component" {
|
||||
t.Errorf("rule = %+v level %s", r, r.Level())
|
||||
}
|
||||
if !strings.Contains(r.Expr, "bgp_peer_state") || !strings.Contains(r.Expr, "6") {
|
||||
t.Errorf("expr = %q: the session is up only in state 6 (established)", r.Expr)
|
||||
}
|
||||
if err := r.Validate(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBGPPeerDownFiresThroughAPolicyAndResolvesOnEstablished(t *testing.T) {
|
||||
r := bgpRule(t)
|
||||
g := newGate(nil, allPolicy("routers", "bgp_peer_down"))
|
||||
h := gateHarness(t, g, r)
|
||||
// The expression is 1 for a peer whose state is not 6: an IPv6 peer in "active" and a healthy IPv4 peer.
|
||||
h.q.set(r.Expr, "rtr1|2001:db8::2|1", "rtr1|192.0.2.1|0")
|
||||
h.step(30 * time.Second)
|
||||
h.want("", "pending")
|
||||
h.step(2 * time.Minute)
|
||||
h.want("firing", "firing")
|
||||
a := h.only()
|
||||
if a.Component != "2001:db8::2" || a.Severity != SevCritical || !strings.Contains(a.Summary, "2001:db8::2") {
|
||||
t.Fatalf("alert = %+v: only the down IPv6 peer may fire, and its summary names it", a)
|
||||
}
|
||||
h.q.set(r.Expr, "rtr1|2001:db8::2|0", "rtr1|192.0.2.1|0") // established again
|
||||
h.step(30 * time.Second)
|
||||
h.want("firing,resolved", "resolved")
|
||||
}
|
||||
|
||||
func TestBGPPeerDownFiresNowhereWithoutAPolicy(t *testing.T) {
|
||||
r := bgpRule(t)
|
||||
h := gateHarness(t, newGate(nil), r)
|
||||
h.q.set(r.Expr, "rtr1|192.0.2.1|1")
|
||||
h.step(30 * time.Second)
|
||||
h.step(5 * time.Minute)
|
||||
h.want("", "")
|
||||
if n := len(h.alerts()); n != 0 {
|
||||
t.Fatalf("%d alert rows with no policy", n)
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alertpolicy"
|
||||
"github.com/v10networks/nmon/internal/clock"
|
||||
"github.com/v10networks/nmon/internal/selfmon"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
@@ -89,6 +90,10 @@ type Options struct {
|
||||
// Unreliable reports an interface whose counters cannot be trusted
|
||||
// (32-bit counters on fast links); rules with SkipUnreliable ignore it.
|
||||
Unreliable func(iface string) bool
|
||||
// Policy is the alert policy gate (required). It decides, for one entity of a rule, whether any alert policy
|
||||
// binds the rule there; no binding means the series is skipped and its open row closes silently as retired.
|
||||
// The binding's severity, threshold and for override the rule's, and its threshold beats group overrides.
|
||||
Policy func(Entity) (alertpolicy.Binding, bool)
|
||||
// Active reports whether an entity still exists (device and interface not
|
||||
// retired). Inactive entities are closed as retired, silently.
|
||||
Active func(Entity) bool
|
||||
@@ -116,8 +121,8 @@ type Evaluator struct {
|
||||
|
||||
// New validates the rules and returns an Evaluator.
|
||||
func New(o Options) (*Evaluator, error) {
|
||||
if o.Store == nil || o.Querier == nil || o.Notifier == nil || o.Clock == nil {
|
||||
return nil, errors.New("alert: Store, Querier, Notifier and Clock are required")
|
||||
if o.Store == nil || o.Querier == nil || o.Notifier == nil || o.Clock == nil || o.Policy == nil {
|
||||
return nil, errors.New("alert: Store, Querier, Notifier, Clock and Policy are required")
|
||||
}
|
||||
if o.Interval <= 0 {
|
||||
o.Interval = DefaultInterval
|
||||
@@ -168,6 +173,7 @@ func (ev *Evaluator) Run(ctx context.Context) error {
|
||||
type reading struct {
|
||||
ent Entity
|
||||
value float64
|
||||
bind alertpolicy.Binding
|
||||
}
|
||||
|
||||
// EvalOnce runs one evaluation cycle at the clock's current time.
|
||||
@@ -205,7 +211,7 @@ func (ev *Evaluator) sweepRows(ctx context.Context, now time.Time, rows []store.
|
||||
ev.deliver(ctx, now, a, ent) // closed row with an undelivered notification
|
||||
continue
|
||||
}
|
||||
if ruleByName[a.Rule] == nil || (ev.o.Active != nil && !ev.o.Active(ent)) {
|
||||
if ruleByName[a.Rule] == nil || (ev.o.Active != nil && !ev.o.Active(ent)) || !ev.bound(ent) {
|
||||
ev.retire(ctx, now, a)
|
||||
continue
|
||||
}
|
||||
@@ -235,8 +241,8 @@ func (ev *Evaluator) evalRule(ctx context.Context, now time.Time, r *compiledRul
|
||||
for _, rd := range readings {
|
||||
key := rd.ent.Key()
|
||||
seen[key] = true
|
||||
thr := r.resolveThreshold(rd.ent.Device, ev.o.Groups, ev.o.ThresholdFor)
|
||||
ev.apply(ctx, now, r, rd, thr, byKey[key])
|
||||
rr, thr := r.bound(rd.bind, rd.ent.Device, ev.o.Groups, ev.o.ThresholdFor)
|
||||
ev.apply(ctx, now, rr, rd, thr, byKey[key])
|
||||
}
|
||||
for _, a := range rows {
|
||||
if !seen[a.SeriesKey] {
|
||||
@@ -266,3 +272,11 @@ func (ev *Evaluator) SortedRuleNames() []string {
|
||||
sort.Strings(n)
|
||||
return n
|
||||
}
|
||||
|
||||
// bound reports whether an alert policy binds the entity's rule there. A policy removed, disabled or
|
||||
// narrowed away makes the entity unbound, and its open alert closes as retired, silently: switching an alert
|
||||
// off is not a recovery.
|
||||
func (ev *Evaluator) bound(ent Entity) bool {
|
||||
_, ok := ev.o.Policy(ent)
|
||||
return ok
|
||||
}
|
||||
|
||||
@@ -37,6 +37,9 @@ func (q *fakeQ) Query(_ context.Context, _ string, expr string, _ time.Time) ([]
|
||||
if p[1] != "" {
|
||||
m["component"] = p[1]
|
||||
}
|
||||
if len(p) > 3 {
|
||||
m["iface"] = p[3]
|
||||
}
|
||||
res = append(res, map[string]any{"metric": m, "value": []any{1, p[2]}})
|
||||
}
|
||||
return json.Marshal(map[string]any{"status": "success", "data": map[string]any{"resultType": "vector", "result": res}})
|
||||
@@ -112,7 +115,7 @@ func newHarness(t *testing.T, rules ...Rule) *harness {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h := &harness{t: t, st: st, clk: clk, q: &fakeQ{series: map[string][]string{}, err: map[string]error{}}, n: &fakeN{}, reg: selfmon.New()}
|
||||
h.opts = Options{Store: st, Querier: h.q, Notifier: h.n, Clock: clk, Rules: rules, Metrics: h.reg}
|
||||
h.opts = Options{Store: st, Querier: h.q, Notifier: h.n, Clock: clk, Rules: rules, Metrics: h.reg, Policy: allowAll}
|
||||
h.restart()
|
||||
return h
|
||||
}
|
||||
@@ -616,7 +619,7 @@ func TestDefaultRulesValid(t *testing.T) {
|
||||
t.Errorf("%s summary %q", r.Name, out)
|
||||
}
|
||||
}
|
||||
for _, n := range []string{"interface_down", "interface_errors", "cpu_high", "memory_high", "disk_full", "ups_on_battery", "ups_runtime_low", "printer_supply_low", "temp_high"} {
|
||||
for _, n := range []string{"interface_down", "interface_errors", "cpu_high", "memory_high", "disk_full", "ups_on_battery", "ups_runtime_low", "printer_supply_low", "temp_high", "bgp_peer_down"} {
|
||||
if !seen[n] {
|
||||
t.Errorf("missing default %s", n)
|
||||
}
|
||||
|
||||
@@ -69,7 +69,11 @@ func TestIntegrationDefaultRulesAgainstRealStore(t *testing.T) {
|
||||
add("prtMarkerSuppliesMaxCapacity", 100, "device", "prn1", "component", "Black Toner")
|
||||
add("prtMarkerSuppliesLevel", -2, "device", "prn1", "component", "Cyan Toner")
|
||||
add("prtMarkerSuppliesMaxCapacity", 100, "device", "prn1", "component", "Cyan Toner")
|
||||
add("prtMarkerSuppliesLevel", 0, "device", "prn1", "component", "Magenta Toner")
|
||||
add("prtMarkerSuppliesMaxCapacity", 100, "device", "prn1", "component", "Magenta Toner")
|
||||
add("mtxrHlTemperature", 80, "device", "rb1")
|
||||
add("bgp_peer_state", 3, "device", "rtr1", "component", "2001:db8::2") // active
|
||||
add("bgp_peer_state", 6, "device", "rtr1", "component", "192.0.2.1") // established
|
||||
add("ifInErrors", float64(i)*60, "device", "sw1", "iface", "if3")
|
||||
add("ifOutErrors", 0, "device", "sw1", "iface", "if3")
|
||||
add("ifHCInOctets", float64(i)*60*150000, "device", "sw1", "iface", "if3") // 2500 pkt/s at 1500 B... errors 1/s
|
||||
@@ -101,7 +105,7 @@ func TestIntegrationDefaultRulesAgainstRealStore(t *testing.T) {
|
||||
}
|
||||
n := &fakeN{}
|
||||
reg := selfmon.New()
|
||||
ev, err := New(Options{Store: st, Querier: vmsup.NewQuerier(sup), Notifier: n, Clock: clk, Rules: rules, Metrics: reg})
|
||||
ev, err := New(Options{Store: st, Querier: vmsup.NewQuerier(sup), Notifier: n, Clock: clk, Rules: rules, Metrics: reg, Policy: allowAll})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -109,7 +113,7 @@ func TestIntegrationDefaultRulesAgainstRealStore(t *testing.T) {
|
||||
deadline := time.Now().Add(40 * time.Second)
|
||||
for {
|
||||
ev.EvalOnce(ctx)
|
||||
if len(n.got) >= 10 || time.Now().After(deadline) {
|
||||
if len(n.got) >= 12 || time.Now().After(deadline) {
|
||||
break
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
@@ -135,14 +139,16 @@ func TestIntegrationDefaultRulesAgainstRealStore(t *testing.T) {
|
||||
"ups_on_battery/ups1//",
|
||||
"ups_runtime_low/ups1//",
|
||||
"printer_supply_low/prn1//Black Toner",
|
||||
"printer_supply_empty/prn1//Magenta Toner",
|
||||
"temp_high/rb1//mtxrHlTemperature",
|
||||
"bgp_peer_down/rtr1//2001:db8::2",
|
||||
"interface_errors/sw1/if3/",
|
||||
} {
|
||||
if !fired[want] {
|
||||
t.Errorf("did not fire: %s (fired: %v)", want, fired)
|
||||
}
|
||||
}
|
||||
for _, not := range []string{"disk_full/disk1///", "disk_full/mem1//Physical memory", "interface_down/sw1/if2/", "printer_supply_low/prn1//Cyan Toner"} {
|
||||
for _, not := range []string{"disk_full/disk1///", "disk_full/mem1//Physical memory", "interface_down/sw1/if2/", "printer_supply_low/prn1//Cyan Toner", "printer_supply_empty/prn1//Black Toner", "printer_supply_empty/prn1//Cyan Toner", "bgp_peer_down/rtr1//192.0.2.1"} {
|
||||
if fired[not] {
|
||||
t.Errorf("fired but should not: %s", not)
|
||||
}
|
||||
|
||||
146
internal/alert/policy_test.go
Normal file
@@ -0,0 +1,146 @@
|
||||
package alert
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alertpolicy"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// allowAll is the gate of tests that are not about policies: every entity is bound by a plain alert policy.
|
||||
func allowAll(Entity) (alertpolicy.Binding, bool) { return alertpolicy.Binding{PolicyID: "all"}, true }
|
||||
|
||||
// policyGate stands in for the app's policy cache: a mutable policy set and a port directory.
|
||||
type policyGate struct {
|
||||
mu sync.Mutex
|
||||
set *alertpolicy.Set
|
||||
ifaces map[string]alertpolicy.Iface
|
||||
}
|
||||
|
||||
func newGate(ifaces map[string]alertpolicy.Iface, ps ...alertpolicy.Policy) *policyGate {
|
||||
g := &policyGate{ifaces: ifaces}
|
||||
g.put(ps...)
|
||||
return g
|
||||
}
|
||||
|
||||
func (g *policyGate) put(ps ...alertpolicy.Policy) {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
g.set, _ = alertpolicy.NewSet(ps)
|
||||
}
|
||||
|
||||
func (g *policyGate) decide(e Entity) (alertpolicy.Binding, bool) {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
subj := alertpolicy.Subject{Device: alertpolicy.Device{Slug: e.Device}}
|
||||
if i, ok := g.ifaces[e.Iface]; ok {
|
||||
subj.Iface = &i
|
||||
}
|
||||
return g.set.Decide(e.Rule, subj)
|
||||
}
|
||||
|
||||
func allPolicy(name, rule string) alertpolicy.Policy {
|
||||
return alertpolicy.Policy{ID: name, Name: name, Rule: rule, Action: alertpolicy.ActionAlert, Enabled: true,
|
||||
Scope: alertpolicy.Scope{All: true}}
|
||||
}
|
||||
|
||||
func errorsRule() Rule {
|
||||
return Rule{Name: "interface_errors", Severity: SevWarning, Expr: testExpr, Op: OpGreater, Threshold: 0.1,
|
||||
Labels: []string{"device", "iface"}, Summary: "s", Description: "d"}
|
||||
}
|
||||
|
||||
func gateHarness(t *testing.T, g *policyGate, r Rule) *harness {
|
||||
h := newHarness(t, r)
|
||||
h.opts.Policy = g.decide
|
||||
h.restart()
|
||||
return h
|
||||
}
|
||||
|
||||
func TestWithoutPoliciesNothingIsRecorded(t *testing.T) {
|
||||
h := gateHarness(t, newGate(nil), errorsRule())
|
||||
h.q.set(testExpr, "d1||5|p1")
|
||||
h.step(30 * time.Second)
|
||||
h.want("", "")
|
||||
if n := len(h.alerts()); n != 0 {
|
||||
t.Fatalf("%d alert rows with no policies", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWifiIgnoreAndFiberAlertDecidePerPortType(t *testing.T) {
|
||||
zero := 0.0
|
||||
g := newGate(map[string]alertpolicy.Iface{
|
||||
"fiber": {Type: 6, Alias: "Core fiber"}, "wifi": {Type: 71, Alias: "uplink radio"}, "lan": {Type: 6, Alias: "customer"},
|
||||
},
|
||||
alertpolicy.Policy{ID: "f", Name: "Fiber", Rule: "interface_errors", Action: alertpolicy.ActionAlert, Enabled: true, Threshold: &zero,
|
||||
Scope: alertpolicy.Scope{Interfaces: alertpolicy.IfaceMatch{IfTypes: []int64{6}, IfAlias: "(?i)fib|uplink|core"}}},
|
||||
alertpolicy.Policy{ID: "w", Name: "WiFi", Rule: "interface_errors", Action: alertpolicy.ActionIgnore, Priority: 10, Enabled: true,
|
||||
Scope: alertpolicy.Scope{Interfaces: alertpolicy.IfaceMatch{IfTypes: []int64{71}}}})
|
||||
h := gateHarness(t, g, errorsRule())
|
||||
// 0.05 is under the rule's 0.1 but over the fiber policy's 0: only the fiber port, at threshold 0, fires.
|
||||
h.q.set(testExpr, "d1||0.05|fiber", "d1||0.05|wifi", "d1||0.05|lan")
|
||||
h.step(30 * time.Second)
|
||||
h.want("firing", "firing")
|
||||
if got := h.only(); got.Iface != "fiber" || got.Threshold != 0 {
|
||||
t.Fatalf("alert = %+v, want fiber at threshold 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRemovingAPolicyRetiresOpenRowsSilently(t *testing.T) {
|
||||
g := newGate(nil, allPolicy("p", "interface_errors"))
|
||||
h := gateHarness(t, g, errorsRule())
|
||||
h.q.set(testExpr, "d1||5|p1")
|
||||
h.step(30 * time.Second)
|
||||
h.want("firing", "firing")
|
||||
g.put() // the policy is deleted; no restart
|
||||
h.step(30 * time.Second)
|
||||
h.want("firing", "")
|
||||
if got := h.only().State; got != store.AlertRetired {
|
||||
t.Fatalf("state %q, want retired", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPolicyThresholdBeatsGroupOverride(t *testing.T) {
|
||||
r := errorsRule()
|
||||
r.Overrides = []Override{{Group: "dc1", Threshold: 50}}
|
||||
thr := 1.0
|
||||
p := allPolicy("p", "interface_errors")
|
||||
p.Threshold = &thr
|
||||
h := gateHarness(t, newGate(nil, p), r)
|
||||
h.opts.Groups = fakeGroups{"d1": {{"dc1", 10}}}
|
||||
h.restart()
|
||||
h.q.set(testExpr, "d1||5|p1") // over the policy's 1, under the group's 50
|
||||
h.step(30 * time.Second)
|
||||
h.want("firing", "firing")
|
||||
if got := h.only().Threshold; got != 1 {
|
||||
t.Fatalf("threshold %v, want the policy's 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPolicySeverityAndForOverrideTheRule(t *testing.T) {
|
||||
p := allPolicy("p", "interface_errors")
|
||||
p.Severity = SevCritical
|
||||
hold := 2 * time.Minute
|
||||
p.For = &hold
|
||||
h := gateHarness(t, newGate(nil, p), errorsRule())
|
||||
h.q.set(testExpr, "d1||5|p1")
|
||||
h.step(30 * time.Second)
|
||||
h.want("", "") // pending: the rule's for is 0 but the policy holds 2 minutes
|
||||
h.step(2 * time.Minute)
|
||||
h.want("firing", "firing")
|
||||
if got := h.only().Severity; got != SevCritical {
|
||||
t.Fatalf("severity %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPolicyWriteTakesEffectOnNextCycle(t *testing.T) {
|
||||
g := newGate(nil)
|
||||
h := gateHarness(t, g, errorsRule())
|
||||
h.q.set(testExpr, "d1||5|p1")
|
||||
h.step(30 * time.Second)
|
||||
h.want("", "")
|
||||
g.put(allPolicy("p", "interface_errors"))
|
||||
h.step(30 * time.Second)
|
||||
h.want("firing", "firing")
|
||||
}
|
||||
@@ -7,6 +7,8 @@ import (
|
||||
"math"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alertpolicy"
|
||||
)
|
||||
|
||||
// query runs a rule and returns one reading per active, trusted entity.
|
||||
@@ -24,7 +26,7 @@ func (ev *Evaluator) query(ctx context.Context, r *compiledRule, now time.Time)
|
||||
best := map[string]reading{}
|
||||
var order []string
|
||||
for _, s := range series {
|
||||
ent, ok := ev.entityOfSeries(r, s)
|
||||
ent, bind, ok := ev.entityOfSeries(r, s)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
@@ -35,7 +37,7 @@ func (ev *Evaluator) query(ctx context.Context, r *compiledRule, now time.Time)
|
||||
}
|
||||
// Duplicate series for one entity: keep the one nearer to alerting.
|
||||
if !seen || r.nearer(s.value, cur.value) {
|
||||
best[k] = reading{ent, s.value}
|
||||
best[k] = reading{ent, s.value, bind}
|
||||
}
|
||||
}
|
||||
out := make([]reading, 0, len(order))
|
||||
@@ -54,14 +56,14 @@ func (r *compiledRule) nearer(a, b float64) bool {
|
||||
}
|
||||
|
||||
// entityOfSeries maps a series to its entity. It reports false for series that
|
||||
// lack a label the rule needs, are on unreliable interfaces, or are inactive.
|
||||
func (ev *Evaluator) entityOfSeries(r *compiledRule, s sample) (Entity, bool) {
|
||||
// lack a label the rule needs, are on unreliable interfaces, are inactive or that no alert policy binds.
|
||||
func (ev *Evaluator) entityOfSeries(r *compiledRule, s sample) (Entity, alertpolicy.Binding, bool) {
|
||||
ent := Entity{Rule: r.Name}
|
||||
for _, l := range r.Labels {
|
||||
v := s.labels[l]
|
||||
if v == "" {
|
||||
ev.log.Debug("series without label", "rule", r.Name, "label", l)
|
||||
return Entity{}, false
|
||||
return Entity{}, alertpolicy.Binding{}, false
|
||||
}
|
||||
switch l {
|
||||
case "device":
|
||||
@@ -73,12 +75,13 @@ func (ev *Evaluator) entityOfSeries(r *compiledRule, s sample) (Entity, bool) {
|
||||
}
|
||||
}
|
||||
if r.SkipUnreliable && ent.Iface != "" && ev.o.Unreliable != nil && ev.o.Unreliable(ent.Iface) {
|
||||
return Entity{}, false
|
||||
return Entity{}, alertpolicy.Binding{}, false
|
||||
}
|
||||
if ev.o.Active != nil && !ev.o.Active(ent) {
|
||||
return Entity{}, false
|
||||
return Entity{}, alertpolicy.Binding{}, false
|
||||
}
|
||||
return ent, true
|
||||
b, ok := ev.o.Policy(ent)
|
||||
return ent, b, ok
|
||||
}
|
||||
|
||||
type sample struct {
|
||||
|
||||
@@ -8,9 +8,12 @@ import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"text/template"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alertpolicy"
|
||||
)
|
||||
|
||||
// Severities.
|
||||
@@ -50,7 +53,10 @@ type Rule struct {
|
||||
// SkipUnreliable drops series whose interface the Unreliable filter flags
|
||||
// (32-bit counters on fast links wrap too quickly to trust).
|
||||
SkipUnreliable bool `yaml:"skip_unreliable" json:"skip_unreliable"`
|
||||
Disabled bool `yaml:"disabled" json:"disabled"`
|
||||
// OptIn is a no-op kept so old configs still parse. Alert policies (the alert_down matcher) replaced it:
|
||||
// a rule fires only where a policy binds it.
|
||||
OptIn bool `yaml:"opt_in" json:"opt_in"`
|
||||
Disabled bool `yaml:"disabled" json:"disabled"`
|
||||
}
|
||||
|
||||
// Override replaces a rule's threshold for devices in a group.
|
||||
@@ -254,6 +260,35 @@ func (r Rule) resolveThreshold(device string, g GroupLookup, thresholdFor func(d
|
||||
return r.Threshold
|
||||
}
|
||||
|
||||
// bound returns the rule as one entity's policy binding sees it (a copy with the policy's severity and for)
|
||||
// and the threshold to compare with: the policy's, else the rule's group override, profile or default.
|
||||
func (c *compiledRule) bound(b alertpolicy.Binding, device string, g GroupLookup,
|
||||
thresholdFor func(device, rule string) (float64, bool)) (*compiledRule, float64) {
|
||||
rr := *c
|
||||
if b.Severity != "" {
|
||||
rr.Severity = b.Severity
|
||||
}
|
||||
if b.For != nil {
|
||||
rr.For = *b.For
|
||||
}
|
||||
if b.Threshold != nil {
|
||||
return &rr, *b.Threshold
|
||||
}
|
||||
return &rr, c.resolveThreshold(device, g, thresholdFor)
|
||||
}
|
||||
|
||||
// Level says what the rule's entity is: an interface, a component of a device, or a device. Policies use it to
|
||||
// refuse matchers that cannot apply.
|
||||
func (r Rule) Level() string {
|
||||
switch {
|
||||
case slices.Contains(r.Labels, "iface"):
|
||||
return alertpolicy.LevelInterface
|
||||
case slices.Contains(r.Labels, "component"):
|
||||
return alertpolicy.LevelComponent
|
||||
}
|
||||
return alertpolicy.LevelDevice
|
||||
}
|
||||
|
||||
// patchRule overlays the non-zero tunables of c onto d.
|
||||
func patchRule(d, c Rule) Rule {
|
||||
if c.Severity != "" {
|
||||
@@ -335,6 +370,8 @@ const (
|
||||
// or more disables the prediction half in effect.
|
||||
// - printer_supply_low: negative levels (-1 other, -2 unknown, -3 "some
|
||||
// remaining") are filtered out; the series then goes stale rather than low.
|
||||
// - printer_supply_empty: a supply level of exactly 0 (critical); the negative
|
||||
// sentinels are not 0, so they never fire it.
|
||||
// - temp_high: the temperature metrics of the shipped profiles, each tagged
|
||||
// with its metric name as component where the profile has no component
|
||||
// label. entPhySensorValue is excluded, since it mixes units. Per-profile
|
||||
@@ -421,6 +458,13 @@ func deviceRules(lab func(...string) []string) []Rule {
|
||||
Summary: `{{.Device}} {{.Component}} at {{printf "%.0f" .Value}}%`,
|
||||
Description: `Supply {{.Component}} on {{.Device}} is at {{printf "%.0f" .Value}}% (threshold {{printf "%.0f" .Threshold}}%).`,
|
||||
},
|
||||
{
|
||||
Name: "printer_supply_empty", Severity: SevCritical, Op: OpGreater, Threshold: 0, For: 0,
|
||||
Labels: lab("device", "component"),
|
||||
Expr: `last_over_time(prtMarkerSuppliesLevel[1h]) == bool 0`,
|
||||
Summary: `{{.Device}} {{.Component}} is empty`,
|
||||
Description: `Supply {{.Component}} on {{.Device}} reports a level of 0 (empty).`,
|
||||
},
|
||||
{
|
||||
Name: "temp_high", Severity: SevWarning, Op: OpGreater, Threshold: 70, For: 5 * time.Minute,
|
||||
Labels: lab("device", "component"),
|
||||
@@ -432,5 +476,14 @@ func deviceRules(lab func(...string) []string) []Rule {
|
||||
Summary: `{{.Device}} {{.Component}} at {{printf "%.0f" .Value}} C`,
|
||||
Description: `Temperature {{.Component}} on {{.Device}} is {{printf "%.1f" .Value}} C (threshold {{printf "%.0f" .Threshold}}).`,
|
||||
},
|
||||
{
|
||||
// state 6 is established in every BGP MIB the profiles read (BGP4-MIB, CISCO-BGP4, Juniper BGP4-V2, Arista
|
||||
// BGP4V2). The notification adds the peer's AS and description from the component attributes.
|
||||
Name: "bgp_peer_down", Severity: SevCritical, Op: OpGreater, Threshold: 0, For: 2 * time.Minute,
|
||||
Labels: lab("device", "component"),
|
||||
Expr: `last_over_time(bgp_peer_state[15m]) != bool 6`,
|
||||
Summary: `{{.Device}} BGP peer {{.Component}} is down`,
|
||||
Description: `The BGP session to {{.Component}} on {{.Device}} has not been established for {{.For}}.`,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
136
internal/alertpolicy/decide.go
Normal file
@@ -0,0 +1,136 @@
|
||||
package alertpolicy
|
||||
|
||||
import (
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Matcher is a compiled Scope.
|
||||
type Matcher struct {
|
||||
scope Scope
|
||||
name, alias *regexp.Regexp
|
||||
}
|
||||
|
||||
// Match reports whether the subject is in the scope. An interface matcher never matches a subject without a port.
|
||||
func (m *Matcher) Match(s Subject) bool {
|
||||
sc := m.scope
|
||||
if sc.All {
|
||||
return true
|
||||
}
|
||||
if len(sc.Groups) > 0 && !slices.ContainsFunc(sc.Groups, func(g string) bool { return slices.Contains(s.Groups, g) }) {
|
||||
return false
|
||||
}
|
||||
if !matchDevice(sc.Devices, s.Device) {
|
||||
return false
|
||||
}
|
||||
if sc.Interfaces.empty() {
|
||||
return true
|
||||
}
|
||||
return s.Iface != nil && m.matchIface(*s.Iface)
|
||||
}
|
||||
|
||||
func anyFold(list []string, v string) bool {
|
||||
return len(list) == 0 || slices.ContainsFunc(list, func(x string) bool { return strings.EqualFold(x, v) })
|
||||
}
|
||||
|
||||
func matchDevice(dm DeviceMatch, d Device) bool {
|
||||
if len(dm.Slugs) > 0 && !slices.ContainsFunc(dm.Slugs, func(g string) bool { ok, _ := path.Match(g, d.Slug); return ok }) {
|
||||
return false
|
||||
}
|
||||
return anyFold(dm.Profiles, d.Profile) && anyFold(dm.Roles, d.Role) && anyFold(dm.Sites, d.Site) && anyFold(dm.Kinds, d.Kind)
|
||||
}
|
||||
|
||||
func (m *Matcher) matchIface(i Iface) bool {
|
||||
im := m.scope.Interfaces
|
||||
switch {
|
||||
case len(im.IfTypes) > 0 && !slices.Contains(im.IfTypes, i.Type):
|
||||
return false
|
||||
case m.name != nil && !m.name.MatchString(i.Name):
|
||||
return false
|
||||
case m.alias != nil && !m.alias.MatchString(i.Alias):
|
||||
return false
|
||||
case im.MinSpeedMbps != nil && i.SpeedMbps < *im.MinSpeedMbps:
|
||||
return false
|
||||
case im.MaxSpeedMbps != nil && i.SpeedMbps > *im.MaxSpeedMbps:
|
||||
return false
|
||||
case im.AlertDown != nil && *im.AlertDown != i.AlertDown:
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
type entry struct {
|
||||
p Policy
|
||||
m *Matcher
|
||||
}
|
||||
|
||||
// Set is a compiled list of policies.
|
||||
type Set struct{ entries []entry }
|
||||
|
||||
// NewSet compiles the enabled policies. A policy that does not compile is left out, and the error says which;
|
||||
// the rest still work, so one bad row cannot silence every other policy.
|
||||
func NewSet(policies []Policy) (*Set, error) {
|
||||
s := &Set{}
|
||||
var bad []string
|
||||
for _, p := range policies {
|
||||
if !p.Enabled {
|
||||
continue
|
||||
}
|
||||
m, err := p.Scope.Compile()
|
||||
if err != nil {
|
||||
bad = append(bad, p.Name+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
s.entries = append(s.entries, entry{p, m})
|
||||
}
|
||||
if len(bad) > 0 {
|
||||
return s, &CompileError{Bad: bad}
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// CompileError lists the policies NewSet skipped.
|
||||
type CompileError struct{ Bad []string }
|
||||
|
||||
func (e *CompileError) Error() string { return "alert policies skipped: " + strings.Join(e.Bad, "; ") }
|
||||
|
||||
// Len is the number of enabled, compiled policies.
|
||||
func (s *Set) Len() int { return len(s.entries) }
|
||||
|
||||
// Decide finds the policy that rules over (rule, subject). The matching policy with the highest priority
|
||||
// decides; on a tie ignore beats alert, then the lowest name wins. It reports false when nothing matches or
|
||||
// the decider is an ignore policy: the entity raises nothing.
|
||||
func (s *Set) Decide(rule string, subj Subject) (Binding, bool) {
|
||||
var best *Policy
|
||||
for i := range s.entries {
|
||||
e := &s.entries[i]
|
||||
if e.p.Rule != rule || !e.m.Match(subj) {
|
||||
continue
|
||||
}
|
||||
if best == nil || outranks(e.p, *best) {
|
||||
best = &e.p
|
||||
}
|
||||
}
|
||||
if best == nil || best.Action != ActionAlert {
|
||||
return Binding{}, false
|
||||
}
|
||||
return Binding{PolicyID: best.ID, PolicyName: best.Name, Severity: best.Severity, Threshold: best.Threshold, For: best.For}, true
|
||||
}
|
||||
|
||||
func outranks(a, b Policy) bool {
|
||||
if a.Priority != b.Priority {
|
||||
return a.Priority > b.Priority
|
||||
}
|
||||
if a.Action != b.Action {
|
||||
return a.Action == ActionIgnore
|
||||
}
|
||||
return a.Name < b.Name
|
||||
}
|
||||
|
||||
// Decide compiles policies and decides once; the evaluators keep a Set instead.
|
||||
func Decide(policies []Policy, rule string, subj Subject) (Binding, bool) {
|
||||
s, _ := NewSet(policies)
|
||||
return s.Decide(rule, subj)
|
||||
}
|
||||
123
internal/alertpolicy/decide_test.go
Normal file
@@ -0,0 +1,123 @@
|
||||
package alertpolicy
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func f64(v float64) *float64 { return &v }
|
||||
|
||||
func port(typ int64, alias string) Subject {
|
||||
return Subject{Device: Device{Slug: "sw1"}, Iface: &Iface{Type: typ, Name: "e1", Alias: alias, SpeedMbps: 1000}}
|
||||
}
|
||||
|
||||
func ownerPair() []Policy {
|
||||
return []Policy{
|
||||
{ID: "a", Name: "Fiber + uplinks: errors", Rule: "interface_errors", Action: ActionAlert, Enabled: true, Threshold: f64(0),
|
||||
Scope: Scope{Interfaces: IfaceMatch{IfTypes: []int64{6}, IfAlias: "(?i)fib|uplink|core"}}},
|
||||
{ID: "b", Name: "WiFi: no error alerts", Rule: "interface_errors", Action: ActionIgnore, Priority: 10, Enabled: true,
|
||||
Scope: Scope{Interfaces: IfaceMatch{IfTypes: []int64{71}}}},
|
||||
}
|
||||
}
|
||||
|
||||
func TestOwnerExamplesDecideWifiAndFiber(t *testing.T) {
|
||||
ps := ownerPair()
|
||||
if b, ok := Decide(ps, "interface_errors", port(6, "Core Fiber to X")); !ok || b.Threshold == nil || *b.Threshold != 0 {
|
||||
t.Fatalf("fiber port: %+v %v", b, ok)
|
||||
}
|
||||
if _, ok := Decide(ps, "interface_errors", port(71, "uplink wifi")); ok {
|
||||
t.Fatal("wifi port must not alert")
|
||||
}
|
||||
if _, ok := Decide(ps, "interface_errors", port(6, "customer lan")); ok {
|
||||
t.Fatal("unmatched ethernet port must not alert")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoPoliciesNothingAlerts(t *testing.T) {
|
||||
if _, ok := Decide(nil, "device_down", Subject{}); ok {
|
||||
t.Fatal("alerted without policies")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrecedence(t *testing.T) {
|
||||
all := Scope{All: true}
|
||||
mk := func(name, action string, prio int) Policy {
|
||||
return Policy{ID: name, Name: name, Rule: "r", Action: action, Priority: prio, Enabled: true, Scope: all}
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
ps []Policy
|
||||
want string // "" = no alert
|
||||
}{
|
||||
{"higher priority alert beats ignore", []Policy{mk("i", ActionIgnore, 0), mk("a", ActionAlert, 5)}, "a"},
|
||||
{"tie: ignore wins", []Policy{mk("a", ActionAlert, 1), mk("i", ActionIgnore, 1)}, ""},
|
||||
{"tie alerts: lowest name", []Policy{mk("b", ActionAlert, 1), mk("a", ActionAlert, 1)}, "a"},
|
||||
{"higher ignore beats alert", []Policy{mk("a", ActionAlert, 1), mk("i", ActionIgnore, 2)}, ""},
|
||||
{"disabled skipped", []Policy{{ID: "x", Name: "x", Rule: "r", Action: ActionAlert, Scope: all}}, ""},
|
||||
{"other rule skipped", []Policy{{ID: "x", Name: "x", Rule: "q", Action: ActionAlert, Enabled: true, Scope: all}}, ""},
|
||||
}
|
||||
for _, c := range cases {
|
||||
b, ok := Decide(c.ps, "r", Subject{})
|
||||
if got := b.PolicyID; (c.want == "") == ok || got != c.want {
|
||||
t.Errorf("%s: got %q ok=%v, want %q", c.name, got, ok, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchers(t *testing.T) {
|
||||
yes := true
|
||||
cases := []struct {
|
||||
name string
|
||||
scope Scope
|
||||
subj Subject
|
||||
want bool
|
||||
}{
|
||||
{"group member", Scope{Groups: []string{"3"}}, Subject{Groups: []string{"1", "3"}}, true},
|
||||
{"group miss", Scope{Groups: []string{"3"}}, Subject{Groups: []string{"1"}}, false},
|
||||
{"slug glob", Scope{Devices: DeviceMatch{Slugs: []string{"core-*"}}}, Subject{Device: Device{Slug: "core-1"}}, true},
|
||||
{"role fold", Scope{Devices: DeviceMatch{Roles: []string{"CORE"}}}, Subject{Device: Device{Role: "core"}}, true},
|
||||
{"iface matcher on device subject", Scope{Interfaces: IfaceMatch{IfTypes: []int64{6}}}, Subject{}, false},
|
||||
{"alert_down", Scope{Interfaces: IfaceMatch{AlertDown: &yes}}, Subject{Iface: &Iface{AlertDown: true}}, true},
|
||||
{"min speed", Scope{Interfaces: IfaceMatch{MinSpeedMbps: new(int64)}}, Subject{Iface: &Iface{}}, true},
|
||||
{"and across fields", Scope{Groups: []string{"1"}, Devices: DeviceMatch{Kinds: []string{"switch"}}},
|
||||
Subject{Groups: []string{"1"}, Device: Device{Kind: "router"}}, false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
m, err := c.scope.Compile()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := m.Match(c.subj); got != c.want {
|
||||
t.Errorf("%s: got %v", c.name, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidate(t *testing.T) {
|
||||
levels := map[string]string{"interface_errors": LevelInterface, "cpu_high": LevelDevice, RuleSystem: LevelSystem}
|
||||
ok := Policy{Name: "n", Rule: "interface_errors", Action: ActionAlert, Scope: Scope{Interfaces: IfaceMatch{IfTypes: []int64{6}}}}
|
||||
if err := Validate(ok, levels); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
neg := -time.Second
|
||||
bad := map[string]Policy{
|
||||
"empty scope": {Name: "n", Rule: "cpu_high", Action: ActionAlert},
|
||||
"unknown rule": {Name: "n", Rule: "nope", Action: ActionAlert, Scope: Scope{All: true}},
|
||||
"iface on device": {Name: "n", Rule: "cpu_high", Action: ActionAlert, Scope: Scope{Interfaces: IfaceMatch{IfName: "x"}}},
|
||||
"system not all": {Name: "n", Rule: RuleSystem, Action: ActionAlert, Scope: Scope{Groups: []string{"1"}}},
|
||||
"bad regex": {Name: "n", Rule: "interface_errors", Action: ActionAlert, Scope: Scope{Interfaces: IfaceMatch{IfAlias: "("}}},
|
||||
"all plus more": {Name: "n", Rule: "cpu_high", Action: ActionAlert, Scope: Scope{All: true, Groups: []string{"1"}}},
|
||||
"bad action": {Name: "n", Rule: "cpu_high", Action: "x", Scope: Scope{All: true}},
|
||||
"negative for": {Name: "n", Rule: "cpu_high", Action: ActionAlert, For: &neg, Scope: Scope{All: true}},
|
||||
"missing name": {Rule: "cpu_high", Action: ActionAlert, Scope: Scope{All: true}},
|
||||
"bad severity": {Name: "n", Rule: "cpu_high", Action: ActionAlert, Severity: "x", Scope: Scope{All: true}},
|
||||
"min above max speed": {Name: "n", Rule: "interface_errors", Action: ActionAlert, Scope: Scope{Interfaces: IfaceMatch{MinSpeedMbps: i64(5), MaxSpeedMbps: i64(1)}}},
|
||||
}
|
||||
for n, p := range bad {
|
||||
if Validate(p, levels) == nil {
|
||||
t.Errorf("%s: accepted", n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func i64(v int64) *int64 { return &v }
|
||||
116
internal/alertpolicy/policy.go
Normal file
@@ -0,0 +1,116 @@
|
||||
// Package alertpolicy decides where alerts fire. A policy binds one rule to a scope (groups plus device and
|
||||
// port attribute matchers); with no matching policy an entity raises nothing. The package is pure: it knows no
|
||||
// store and no clock, so the evaluators, the API's preview and the tests all use the same decision.
|
||||
package alertpolicy
|
||||
|
||||
import "time"
|
||||
|
||||
// Actions of a policy.
|
||||
const (
|
||||
ActionAlert = "alert"
|
||||
ActionIgnore = "ignore"
|
||||
)
|
||||
|
||||
// Rule names that are not metric rules. RuleSystem stands for every subsystem alarm (alert.SystemRule).
|
||||
const (
|
||||
RuleDeviceDown = "device_down"
|
||||
RuleDeviceWarning = "device_warning"
|
||||
RuleSystem = "system"
|
||||
)
|
||||
|
||||
// Levels say what a rule's entity is, and so which matchers can apply to it.
|
||||
const (
|
||||
LevelDevice = "device"
|
||||
LevelInterface = "interface"
|
||||
LevelComponent = "component"
|
||||
LevelSystem = "system"
|
||||
)
|
||||
|
||||
// Policy binds one rule to a scope.
|
||||
type Policy struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Rule string `json:"rule"`
|
||||
Action string `json:"action"`
|
||||
Priority int `json:"priority"`
|
||||
Enabled bool `json:"enabled"`
|
||||
// Severity, Threshold and For override the rule's; nil or empty keeps the rule's.
|
||||
Severity string `json:"severity,omitempty"`
|
||||
Threshold *float64 `json:"threshold,omitempty"`
|
||||
For *time.Duration `json:"for,omitempty"`
|
||||
Scope Scope `json:"scope"`
|
||||
}
|
||||
|
||||
// Scope selects entities. Its fields are ANDed; within a list any entry may match; an empty field is
|
||||
// unconstrained. A scope with no field set is invalid: All is the explicit "everything".
|
||||
type Scope struct {
|
||||
All bool `json:"all,omitempty"`
|
||||
Groups []string `json:"groups,omitempty"`
|
||||
Devices DeviceMatch `json:"devices,omitempty"`
|
||||
Interfaces IfaceMatch `json:"interfaces,omitempty"`
|
||||
}
|
||||
|
||||
// DeviceMatch picks devices by attribute. Slugs are path.Match globs; the others compare case-insensitively.
|
||||
type DeviceMatch struct {
|
||||
Slugs []string `json:"slugs,omitempty"`
|
||||
Profiles []string `json:"profiles,omitempty"`
|
||||
Roles []string `json:"roles,omitempty"`
|
||||
Sites []string `json:"sites,omitempty"`
|
||||
Kinds []string `json:"kinds,omitempty"`
|
||||
}
|
||||
|
||||
// IfaceMatch picks ports by attribute. IfName and IfAlias are RE2 patterns searched anywhere in the text.
|
||||
type IfaceMatch struct {
|
||||
IfTypes []int64 `json:"if_types,omitempty"`
|
||||
IfName string `json:"if_name,omitempty"`
|
||||
IfAlias string `json:"if_alias,omitempty"`
|
||||
MinSpeedMbps *int64 `json:"min_speed_mbps,omitempty"`
|
||||
MaxSpeedMbps *int64 `json:"max_speed_mbps,omitempty"`
|
||||
// AlertDown true matches ports whose bell, or a group's alert_down, is on; false matches the others.
|
||||
AlertDown *bool `json:"alert_down,omitempty"`
|
||||
}
|
||||
|
||||
// Binding is the outcome of a deciding "alert" policy: who decided and what it overrides.
|
||||
type Binding struct {
|
||||
PolicyID, PolicyName string
|
||||
Severity string
|
||||
Threshold *float64
|
||||
For *time.Duration
|
||||
}
|
||||
|
||||
// Subject is the thing a rule would alert on. Iface is nil for a device-level or component-level entity.
|
||||
type Subject struct {
|
||||
Device Device
|
||||
Iface *Iface
|
||||
// Groups are the ids of the groups the port is a member of now; for a device subject, the groups of any
|
||||
// of its ports.
|
||||
Groups []string
|
||||
}
|
||||
|
||||
// Device holds the device attributes matchers read.
|
||||
type Device struct{ Slug, Profile, Role, Site, Kind string }
|
||||
|
||||
// Iface holds the port attributes matchers read.
|
||||
type Iface struct {
|
||||
Type int64
|
||||
Name string
|
||||
Alias string
|
||||
SpeedMbps int64
|
||||
AlertDown bool
|
||||
}
|
||||
|
||||
func (m IfaceMatch) empty() bool {
|
||||
return len(m.IfTypes) == 0 && m.IfName == "" && m.IfAlias == "" && m.MinSpeedMbps == nil && m.MaxSpeedMbps == nil && m.AlertDown == nil
|
||||
}
|
||||
|
||||
func (m DeviceMatch) empty() bool {
|
||||
return len(m.Slugs)+len(m.Profiles)+len(m.Roles)+len(m.Sites)+len(m.Kinds) == 0
|
||||
}
|
||||
|
||||
// Empty reports a scope that constrains nothing and is not All.
|
||||
func (s Scope) Empty() bool {
|
||||
return !s.All && len(s.Groups) == 0 && s.Devices.empty() && s.Interfaces.empty()
|
||||
}
|
||||
|
||||
// HasInterfaceMatchers reports a scope that only a port can satisfy.
|
||||
func (s Scope) HasInterfaceMatchers() bool { return !s.Interfaces.empty() }
|
||||
24
internal/alertpolicy/scopejson.go
Normal file
@@ -0,0 +1,24 @@
|
||||
package alertpolicy
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// FormatScope renders a scope as the JSON the store keeps.
|
||||
func FormatScope(s Scope) string {
|
||||
b, err := json.Marshal(s)
|
||||
if err != nil { // a Scope holds only strings, numbers and bools
|
||||
panic(fmt.Sprintf("alertpolicy: marshal scope: %v", err))
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// ParseScope reads what FormatScope wrote.
|
||||
func ParseScope(raw string) (Scope, error) {
|
||||
var s Scope
|
||||
if err := json.Unmarshal([]byte(raw), &s); err != nil {
|
||||
return Scope{}, fmt.Errorf("alert policy scope: %w", err)
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
103
internal/alertpolicy/validate.go
Normal file
@@ -0,0 +1,103 @@
|
||||
package alertpolicy
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"path"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Validate checks a policy. levels maps every rule name a policy may name to its level (LevelDevice, ...);
|
||||
// the caller builds it from the configured rules plus device_down, device_warning and system.
|
||||
func Validate(p Policy, levels map[string]string) error {
|
||||
var errs []error
|
||||
add := func(f string, a ...any) { errs = append(errs, fmt.Errorf(f, a...)) }
|
||||
if strings.TrimSpace(p.Name) == "" {
|
||||
add("name is required")
|
||||
}
|
||||
level, known := levels[p.Rule]
|
||||
if !known {
|
||||
add("rule %q is not a known rule", p.Rule)
|
||||
}
|
||||
if p.Action != ActionAlert && p.Action != ActionIgnore {
|
||||
add("action %q is not alert or ignore", p.Action)
|
||||
}
|
||||
switch p.Severity {
|
||||
case "", "critical", "warning", "info":
|
||||
default:
|
||||
add("severity %q is not critical, warning or info", p.Severity)
|
||||
}
|
||||
if p.Threshold != nil && (math.IsNaN(*p.Threshold) || math.IsInf(*p.Threshold, 0)) {
|
||||
add("threshold is not a number")
|
||||
}
|
||||
if p.For != nil && *p.For < 0 {
|
||||
add("for is negative")
|
||||
}
|
||||
if known {
|
||||
errs = append(errs, validateScopeFor(p.Scope, level)...)
|
||||
}
|
||||
errs = append(errs, validateScope(p.Scope)...)
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
// validateScopeFor checks the scope against what the rule's entity can be.
|
||||
func validateScopeFor(s Scope, level string) []error {
|
||||
var errs []error
|
||||
if level == LevelSystem && !s.All {
|
||||
errs = append(errs, errors.New("a system policy accepts only scope.all: true, because system alarms have no device"))
|
||||
}
|
||||
if level != LevelInterface && s.HasInterfaceMatchers() {
|
||||
errs = append(errs, fmt.Errorf("interface matchers cannot apply to a %s rule", level))
|
||||
}
|
||||
return errs
|
||||
}
|
||||
|
||||
func validateScope(s Scope) []error {
|
||||
var errs []error
|
||||
if s.Empty() {
|
||||
return []error{errors.New("scope is empty: set all: true to mean everything")}
|
||||
}
|
||||
if s.All && (len(s.Groups) > 0 || !s.Devices.empty() || s.Interfaces.HasAny()) {
|
||||
errs = append(errs, errors.New("scope.all excludes every other scope field"))
|
||||
}
|
||||
for _, g := range s.Groups {
|
||||
if strings.TrimSpace(g) == "" {
|
||||
errs = append(errs, errors.New("scope.groups holds an empty group id"))
|
||||
}
|
||||
}
|
||||
for _, g := range s.Devices.Slugs {
|
||||
if _, err := path.Match(g, ""); err != nil {
|
||||
errs = append(errs, fmt.Errorf("device slug glob %q: %w", g, err))
|
||||
}
|
||||
}
|
||||
if _, err := s.Compile(); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
if mn, mx := s.Interfaces.MinSpeedMbps, s.Interfaces.MaxSpeedMbps; mn != nil && mx != nil && *mn > *mx {
|
||||
errs = append(errs, errors.New("min_speed_mbps is above max_speed_mbps"))
|
||||
}
|
||||
return errs
|
||||
}
|
||||
|
||||
// HasAny reports whether any interface matcher is set.
|
||||
func (m IfaceMatch) HasAny() bool { return !m.empty() }
|
||||
|
||||
// Compile prepares a scope for matching; it fails on a bad regular expression.
|
||||
func (s Scope) Compile() (*Matcher, error) {
|
||||
m := &Matcher{scope: s}
|
||||
var errs []error
|
||||
var err error
|
||||
if s.Interfaces.IfName != "" {
|
||||
if m.name, err = regexp.Compile(s.Interfaces.IfName); err != nil {
|
||||
errs = append(errs, fmt.Errorf("if_name: %w", err))
|
||||
}
|
||||
}
|
||||
if s.Interfaces.IfAlias != "" {
|
||||
if m.alias, err = regexp.Compile(s.Interfaces.IfAlias); err != nil {
|
||||
errs = append(errs, fmt.Errorf("if_alias: %w", err))
|
||||
}
|
||||
}
|
||||
return m, errors.Join(errs...)
|
||||
}
|
||||
155
internal/api/alert_down.go
Normal file
@@ -0,0 +1,155 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// AlertDownState says whether an interface alerts when it goes down. Own is the port's flag; Effective adds
|
||||
// membership of a group with the flag on, and Via names the source ("port", or "group:<name>").
|
||||
type AlertDownState struct {
|
||||
AlertDown bool `json:"alert_down" doc:"The port's own opt-in for interface_down alerts."`
|
||||
AlertDownEffective bool `json:"alert_down_effective" doc:"The port alerts when down: own flag, or a group it is in now."`
|
||||
AlertDownVia string `json:"alert_down_via" doc:"Why it is effective: port, or group:<name>. Empty when it is not."`
|
||||
}
|
||||
|
||||
func alertDownState(own bool, via string) AlertDownState {
|
||||
return AlertDownState{AlertDown: own, AlertDownEffective: via != "", AlertDownVia: via}
|
||||
}
|
||||
|
||||
// InterfaceAlertBody is the body of PUT /interfaces/{id}.
|
||||
type InterfaceAlertBody struct {
|
||||
AlertDown bool `json:"alert_down" doc:"Opt the port in (true) or out (false) of interface_down alerts."`
|
||||
}
|
||||
|
||||
// DeviceAlertDownBody is the body of POST /devices/{slug}/interfaces/alert-down.
|
||||
type DeviceAlertDownBody struct {
|
||||
On bool `json:"on" doc:"Opt the ports in (true) or out (false)."`
|
||||
Ifaces []string `json:"ifaces,omitempty" doc:"Interface ids of this device. Omitted means every active port of the device."`
|
||||
}
|
||||
|
||||
// AlertDownResult reports how many ports a bulk change touched.
|
||||
type AlertDownResult struct {
|
||||
Changed int `json:"changed"`
|
||||
}
|
||||
|
||||
func (s *server) registerAlertDown(api huma.API) {
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "set-interface-alert-down", Method: http.MethodPut, Path: "/api/v1/interfaces/{id}", Tags: []string{"interfaces"},
|
||||
Summary: "Opt one interface in or out of interface_down alerts",
|
||||
Description: "Admin only. interface_down alerts are opt-in: a down port raises nothing until its own flag is on " +
|
||||
"or a group it belongs to has alert_down on.",
|
||||
Errors: []int{http.StatusNotFound},
|
||||
}, s.setInterfaceAlertDown)
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "set-device-alert-down", Method: http.MethodPost, Path: "/api/v1/devices/{slug}/interfaces/alert-down",
|
||||
Tags: []string{"devices"},
|
||||
Summary: "Opt a device's interfaces in or out of interface_down alerts",
|
||||
Description: "Admin only. Without ifaces the change applies to every active port of the device. " +
|
||||
"Turning a port off closes its open interface_down alert silently.",
|
||||
Errors: []int{http.StatusNotFound, http.StatusUnprocessableEntity},
|
||||
}, s.setDeviceAlertDown)
|
||||
}
|
||||
|
||||
func (s *server) alertDownChanged() {
|
||||
if s.OnAlertDownChange != nil {
|
||||
s.OnAlertDownChange()
|
||||
}
|
||||
}
|
||||
|
||||
func (s *server) setInterfaceAlertDown(ctx context.Context, in *struct {
|
||||
ID string `path:"id" doc:"Interface ULID."`
|
||||
Body InterfaceAlertBody
|
||||
}) (*struct{ Body InterfaceView }, error) {
|
||||
err := s.Store.WithTx(ctx, func(tx store.Store) error {
|
||||
if err := tx.SetInterfaceAlertDown(ctx, []string{in.ID}, in.Body.AlertDown); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.AppendAudit(ctx, actorOf(ctx), "interface.alert_down", in.ID, nil, in.Body)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.alertDownChanged()
|
||||
i, err := s.Store.GetInterface(ctx, in.ID)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
via, err := s.Store.InterfaceAlertDownVia(ctx, []string{in.ID})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
v := interfaceView(i, nil)
|
||||
v.AlertDownState = alertDownState(i.AlertDown, via[in.ID])
|
||||
return &struct{ Body InterfaceView }{v}, nil
|
||||
}
|
||||
|
||||
func (s *server) setDeviceAlertDown(ctx context.Context, in *struct {
|
||||
Slug string `path:"slug" doc:"Device slug."`
|
||||
Body DeviceAlertDownBody
|
||||
}) (*struct{ Body AlertDownResult }, error) {
|
||||
d, err := s.Store.GetDeviceBySlug(ctx, in.Slug)
|
||||
if err != nil || d.PurgedAt != nil {
|
||||
return nil, huma.Error404NotFound("not found")
|
||||
}
|
||||
ifs, err := s.Store.ListInterfaces(ctx, d.ID, false)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
ids, err := alertDownTargets(ifs, in.Body.Ifaces)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
err = s.Store.WithTx(ctx, func(tx store.Store) error {
|
||||
if err := tx.SetInterfaceAlertDown(ctx, ids, in.Body.On); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.AppendAudit(ctx, actorOf(ctx), "device.interfaces.alert_down", d.Slug, nil,
|
||||
map[string]any{"on": in.Body.On, "count": len(ids), "all": in.Body.Ifaces == nil})
|
||||
})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.alertDownChanged()
|
||||
return &struct{ Body AlertDownResult }{AlertDownResult{Changed: len(ids)}}, nil
|
||||
}
|
||||
|
||||
// alertDownTargets picks the port ids a bulk change applies to: every active port, or the listed ones, which
|
||||
// must all belong to the device.
|
||||
func alertDownTargets(ifs []store.Interface, want []string) ([]string, error) {
|
||||
have := make(map[string]bool, len(ifs))
|
||||
all := make([]string, 0, len(ifs))
|
||||
for _, i := range ifs {
|
||||
have[i.ID] = true
|
||||
all = append(all, i.ID)
|
||||
}
|
||||
if want == nil {
|
||||
return all, nil
|
||||
}
|
||||
for _, id := range want {
|
||||
if !have[id] {
|
||||
return nil, huma.Error422UnprocessableEntity("interface " + id + " is not an active port of this device")
|
||||
}
|
||||
}
|
||||
return want, nil
|
||||
}
|
||||
|
||||
// withAlertDown fills the alert_down fields of interface views from one store query.
|
||||
func (s *server) withAlertDown(ctx context.Context, ifs []store.Interface, views []InterfaceView) error {
|
||||
ids := make([]string, len(ifs))
|
||||
for i := range ifs {
|
||||
ids[i] = ifs[i].ID
|
||||
}
|
||||
via, err := s.Store.InterfaceAlertDownVia(ctx, ids)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range views {
|
||||
views[i].AlertDownState = alertDownState(ifs[i].AlertDown, via[ifs[i].ID])
|
||||
}
|
||||
return nil
|
||||
}
|
||||
121
internal/api/alert_down_test.go
Normal file
@@ -0,0 +1,121 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/v10networks/nmon/internal/api"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
func alertDownEnv(t *testing.T) (*env, *store.Device, store.Interface, store.Interface, *int) {
|
||||
changes := new(int)
|
||||
e := newEnvDeps(t, 0, func(d *api.Deps) { d.OnAlertDownChange = func() { *changes++ } })
|
||||
d := e.device(t, "sw1", "yyz", "access")
|
||||
return e, d, e.iface(t, d, 1, "e1", ""), e.iface(t, d, 2, "e2", ""), changes
|
||||
}
|
||||
|
||||
func TestInterfaceAlertDownIsOffByDefaultAndTogglesPerPort(t *testing.T) {
|
||||
e, d, a, b, changes := alertDownEnv(t)
|
||||
var dv api.DeviceView
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &dv)
|
||||
for _, i := range dv.Interfaces {
|
||||
if i.AlertDown || i.AlertDownEffective || i.AlertDownVia != "" {
|
||||
t.Fatalf("%s alerts by default: %+v", i.IfName, i.AlertDownState)
|
||||
}
|
||||
}
|
||||
rec := e.do("PUT", "/api/v1/interfaces/"+a.ID, `{"alert_down":true}`)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("PUT: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
var iv api.InterfaceView
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &iv); err != nil || !iv.AlertDown || !iv.AlertDownEffective || iv.AlertDownVia != "port" {
|
||||
t.Fatalf("PUT body %s (%v)", rec.Body, err)
|
||||
}
|
||||
if *changes != 1 {
|
||||
t.Errorf("evaluator told %d times, want 1", *changes)
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &dv)
|
||||
if !dv.Interfaces[0].AlertDown || dv.Interfaces[1].AlertDown {
|
||||
t.Fatalf("device view flags: %+v %+v", dv.Interfaces[0].AlertDownState, dv.Interfaces[1].AlertDownState)
|
||||
}
|
||||
var hits api.InterfaceList
|
||||
e.getJSON(t, "/api/v1/interfaces?q=e", &hits)
|
||||
got := map[string]string{}
|
||||
for _, h := range hits.Items {
|
||||
got[h.ID] = h.AlertDownVia
|
||||
}
|
||||
if got[a.ID] != "port" || got[b.ID] != "" {
|
||||
t.Fatalf("search hits via = %v", got)
|
||||
}
|
||||
isProblem(t, e.do("PUT", "/api/v1/interfaces/nope", `{"alert_down":true}`), http.StatusNotFound)
|
||||
audit, _, _ := e.st.ListAudit(t.Context(), store.Page{})
|
||||
if len(audit) == 0 || audit[0].Action != "interface.alert_down" {
|
||||
t.Fatalf("audit = %+v", audit)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeviceBulkAlertDownAllOrListedPorts(t *testing.T) {
|
||||
e, d, a, _, _ := alertDownEnv(t)
|
||||
url := "/api/v1/devices/" + d.Slug + "/interfaces/alert-down"
|
||||
var res api.AlertDownResult
|
||||
rec := e.do("POST", url, `{"on":true}`)
|
||||
if rec.Code != http.StatusOK || json.Unmarshal(rec.Body.Bytes(), &res) != nil || res.Changed != 2 {
|
||||
t.Fatalf("all on: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
rec = e.do("POST", url, fmt.Sprintf(`{"on":false,"ifaces":[%q]}`, a.ID))
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("one off: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
var dv api.DeviceView
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &dv)
|
||||
if dv.Interfaces[0].AlertDown || !dv.Interfaces[1].AlertDown {
|
||||
t.Fatalf("flags after partial off: %+v %+v", dv.Interfaces[0].AlertDownState, dv.Interfaces[1].AlertDownState)
|
||||
}
|
||||
other := e.device(t, "sw2", "yyz", "access")
|
||||
foreign := e.iface(t, other, 1, "x1", "")
|
||||
isProblem(t, e.do("POST", url, fmt.Sprintf(`{"on":true,"ifaces":[%q]}`, foreign.ID)), http.StatusUnprocessableEntity)
|
||||
isProblem(t, e.do("POST", "/api/v1/devices/nope/interfaces/alert-down", `{"on":true}`), http.StatusNotFound)
|
||||
rec = e.do("POST", url, `{"on":false}`)
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &dv)
|
||||
if rec.Code != http.StatusOK || dv.Interfaces[1].AlertDown {
|
||||
t.Fatalf("all off: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGroupAlertDownMakesMemberPortsEffectiveVia(t *testing.T) {
|
||||
e, d, a, _, changes := alertDownEnv(t)
|
||||
rec := e.do("POST", "/api/v1/groups", `{"name":"Uplinks","kind":"role","alert_down":true}`)
|
||||
var g api.GroupSummary
|
||||
if rec.Code != http.StatusCreated || json.Unmarshal(rec.Body.Bytes(), &g) != nil || !g.AlertDown {
|
||||
t.Fatalf("create: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
rec = e.do("POST", "/api/v1/groups/"+g.ID+"/members", fmt.Sprintf(`{"interface_id":%q,"direction":"toward"}`, a.ID))
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("member: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
var dv api.DeviceView
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &dv)
|
||||
if i := dv.Interfaces[0]; i.AlertDown || !i.AlertDownEffective || i.AlertDownVia != "group:Uplinks" {
|
||||
t.Fatalf("member port: %+v", i.AlertDownState)
|
||||
}
|
||||
// A PUT without alert_down leaves the flag; an explicit false clears it.
|
||||
rec = e.do("PUT", "/api/v1/groups/"+g.ID, `{"name":"Uplinks 2","kind":"role"}`)
|
||||
if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), `"alert_down":true`) {
|
||||
t.Fatalf("rename: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
rec = e.do("PUT", "/api/v1/groups/"+g.ID, `{"name":"Uplinks 2","kind":"role","alert_down":false}`)
|
||||
if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), `"alert_down":false`) {
|
||||
t.Fatalf("clear: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &dv)
|
||||
if dv.Interfaces[0].AlertDownEffective {
|
||||
t.Fatal("still effective after the group flag cleared")
|
||||
}
|
||||
if *changes < 4 {
|
||||
t.Errorf("evaluator told %d times; group writes must tell it", *changes)
|
||||
}
|
||||
}
|
||||
347
internal/api/alert_policies.go
Normal file
@@ -0,0 +1,347 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alertpolicy"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// AlertRuleInfo describes a rule a policy can bind, for the UI's dropdown.
|
||||
type AlertRuleInfo struct {
|
||||
Name string `json:"name"`
|
||||
Level string `json:"level" enum:"device,interface,component,system" doc:"What the rule's entity is; port matchers need interface."`
|
||||
Op string `json:"op,omitempty" enum:">,<" doc:"Comparison of a metric rule; empty for reachability and system rules."`
|
||||
Threshold *float64 `json:"threshold" nullable:"true" doc:"The rule's default threshold; null for reachability and system rules."`
|
||||
Severity string `json:"severity" enum:"critical,warning,info"`
|
||||
}
|
||||
|
||||
// AlertRuleList is every rule a policy can bind.
|
||||
type AlertRuleList struct {
|
||||
Items []AlertRuleInfo `json:"items"`
|
||||
}
|
||||
|
||||
// AlertPolicyHost is the running alerting engine as the API sees it.
|
||||
type AlertPolicyHost interface {
|
||||
// Rules lists the rules a policy may name.
|
||||
Rules() []AlertRuleInfo
|
||||
// Changed tells the evaluators a policy was written, so it applies on their next cycle.
|
||||
Changed()
|
||||
// Preview counts what a scope covers right now.
|
||||
Preview(ctx context.Context, rule string, scope alertpolicy.Scope) (PolicyPreview, error)
|
||||
}
|
||||
|
||||
// PolicyPreview is what a draft scope covers.
|
||||
type PolicyPreview struct {
|
||||
Devices PreviewDevices `json:"devices"`
|
||||
Interfaces PreviewInterfaces `json:"interfaces"`
|
||||
}
|
||||
|
||||
// PreviewDevices counts devices; for an interface rule it counts devices with a covered port.
|
||||
type PreviewDevices struct {
|
||||
Count int `json:"count"`
|
||||
Samples []DeviceRef `json:"samples" doc:"Up to 20."`
|
||||
}
|
||||
|
||||
// PreviewInterfaces counts ports; it is zero for rules that have no port.
|
||||
type PreviewInterfaces struct {
|
||||
Count int `json:"count"`
|
||||
Samples []PreviewInterface `json:"samples" doc:"Up to 20."`
|
||||
}
|
||||
|
||||
// PreviewInterface names a covered port.
|
||||
type PreviewInterface struct {
|
||||
ID string `json:"id"`
|
||||
Device string `json:"device" doc:"Device slug."`
|
||||
IfName string `json:"ifname"`
|
||||
IfAlias string `json:"ifalias"`
|
||||
}
|
||||
|
||||
// AlertPolicyView is a policy as the API returns it.
|
||||
type AlertPolicyView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Rule string `json:"rule"`
|
||||
Action string `json:"action" enum:"alert,ignore"`
|
||||
Priority int `json:"priority"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Severity *string `json:"severity" nullable:"true" doc:"Overrides the rule's severity; null keeps it."`
|
||||
Threshold *float64 `json:"threshold" nullable:"true" doc:"Overrides the rule's threshold; null keeps it."`
|
||||
For *string `json:"for" nullable:"true" doc:"Overrides the rule's hold time, a duration such as 5m; null keeps it."`
|
||||
Scope alertpolicy.Scope `json:"scope"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
// AlertPolicyList is every policy, by name.
|
||||
type AlertPolicyList struct {
|
||||
Items []AlertPolicyView `json:"items"`
|
||||
}
|
||||
|
||||
// AlertPolicyBody is the body of POST and PUT /alert-policies.
|
||||
type AlertPolicyBody struct {
|
||||
Name string `json:"name" minLength:"1" maxLength:"200"`
|
||||
Rule string `json:"rule" minLength:"1" doc:"A name from GET /alert-rules."`
|
||||
Action string `json:"action" enum:"alert,ignore" default:"alert"`
|
||||
Priority int `json:"priority,omitempty" doc:"The highest priority decides; on a tie ignore wins, then the lowest name."`
|
||||
Enabled *bool `json:"enabled,omitempty" doc:"Omitted: on for create, kept for update."`
|
||||
Severity *string `json:"severity,omitempty" enum:"critical,warning,info"`
|
||||
Threshold *float64 `json:"threshold,omitempty"`
|
||||
For *string `json:"for,omitempty" doc:"A duration such as 5m."`
|
||||
Scope alertpolicy.Scope `json:"scope" doc:"Must not be empty; all: true means everything."`
|
||||
}
|
||||
|
||||
// AlertPolicyPreviewBody is the body of POST /alert-policies/preview.
|
||||
type AlertPolicyPreviewBody struct {
|
||||
Rule string `json:"rule" minLength:"1"`
|
||||
Scope alertpolicy.Scope `json:"scope"`
|
||||
}
|
||||
|
||||
func (s *server) registerAlertPolicies(api huma.API) {
|
||||
tags := []string{"alerts"}
|
||||
conflict := []int{http.StatusConflict, http.StatusUnprocessableEntity, http.StatusServiceUnavailable}
|
||||
huma.Register(api, huma.Operation{OperationID: "list-alert-rules", Method: http.MethodGet, Path: "/api/v1/alert-rules", Tags: tags,
|
||||
Summary: "Rules an alert policy can bind", Errors: []int{http.StatusServiceUnavailable}}, s.listAlertRules)
|
||||
huma.Register(api, huma.Operation{OperationID: "list-alert-policies", Method: http.MethodGet, Path: "/api/v1/alert-policies", Tags: tags,
|
||||
Summary: "List alert policies",
|
||||
Description: "Alerts fire only where an enabled policy binds a rule to a scope; with no policies nothing alerts."}, s.listAlertPolicies)
|
||||
huma.Register(api, huma.Operation{OperationID: "create-alert-policy", Method: http.MethodPost, Path: "/api/v1/alert-policies", Tags: tags,
|
||||
DefaultStatus: http.StatusCreated, Summary: "Create an alert policy", Errors: conflict}, s.createAlertPolicy)
|
||||
huma.Register(api, huma.Operation{OperationID: "preview-alert-policy", Method: http.MethodPost, Path: "/api/v1/alert-policies/preview",
|
||||
Tags: tags, Summary: "Count the devices and ports a draft scope covers",
|
||||
Errors: []int{http.StatusUnprocessableEntity, http.StatusServiceUnavailable}}, s.previewAlertPolicy)
|
||||
huma.Register(api, huma.Operation{OperationID: "get-alert-policy", Method: http.MethodGet, Path: "/api/v1/alert-policies/{id}", Tags: tags,
|
||||
Summary: "An alert policy", Errors: []int{http.StatusNotFound}}, s.getAlertPolicy)
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "update-alert-policy", Method: http.MethodPut, Path: "/api/v1/alert-policies/{id}", Tags: tags,
|
||||
Summary: "Replace an alert policy", Errors: append([]int{http.StatusNotFound}, conflict...),
|
||||
}, s.updateAlertPolicy)
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "delete-alert-policy", Method: http.MethodDelete, Path: "/api/v1/alert-policies/{id}",
|
||||
Tags: tags, DefaultStatus: http.StatusNoContent, Summary: "Delete an alert policy",
|
||||
Description: "Open alerts the policy bound close silently as retired on the next cycle.", Errors: []int{http.StatusNotFound},
|
||||
}, s.deleteAlertPolicy)
|
||||
}
|
||||
|
||||
func (s *server) policyHost() (AlertPolicyHost, error) {
|
||||
if s.AlertPolicy == nil {
|
||||
return nil, errNotConfigured("alert policies")
|
||||
}
|
||||
return s.AlertPolicy, nil
|
||||
}
|
||||
|
||||
func (s *server) listAlertRules(_ context.Context, _ *struct{}) (*struct{ Body AlertRuleList }, error) {
|
||||
h, err := s.policyHost()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &struct{ Body AlertRuleList }{AlertRuleList{Items: h.Rules()}}, nil
|
||||
}
|
||||
|
||||
func policyView(r store.AlertPolicy) (AlertPolicyView, error) {
|
||||
scope, err := alertpolicy.ParseScope(r.Scope)
|
||||
if err != nil {
|
||||
return AlertPolicyView{}, fmt.Errorf("policy %s: %w", r.ID, err)
|
||||
}
|
||||
v := AlertPolicyView{ID: r.ID, Name: r.Name, Rule: r.Rule, Action: r.Action, Priority: r.Priority, Enabled: r.Enabled,
|
||||
Threshold: r.Threshold, Scope: scope, CreatedAt: r.CreatedAt, UpdatedAt: r.UpdatedAt}
|
||||
if r.Severity != "" {
|
||||
v.Severity = &r.Severity
|
||||
}
|
||||
if r.ForSeconds != nil {
|
||||
d := (time.Duration(*r.ForSeconds) * time.Second).String()
|
||||
v.For = &d
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func (s *server) listAlertPolicies(ctx context.Context, _ *struct{}) (*struct{ Body AlertPolicyList }, error) {
|
||||
rows, err := s.Store.ListAlertPolicies(ctx)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
out := AlertPolicyList{Items: make([]AlertPolicyView, 0, len(rows))}
|
||||
for _, r := range rows {
|
||||
v, err := policyView(r)
|
||||
if err != nil {
|
||||
s.Logger.Error("alert policy unreadable", "err", err)
|
||||
continue
|
||||
}
|
||||
out.Items = append(out.Items, v)
|
||||
}
|
||||
return &struct{ Body AlertPolicyList }{out}, nil
|
||||
}
|
||||
|
||||
func (s *server) getAlertPolicy(ctx context.Context, in *struct {
|
||||
ID string `path:"id" doc:"Policy ULID."`
|
||||
}) (*struct{ Body AlertPolicyView }, error) {
|
||||
r, err := s.Store.GetAlertPolicy(ctx, in.ID)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
v, err := policyView(*r)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
return &struct{ Body AlertPolicyView }{v}, nil
|
||||
}
|
||||
|
||||
// policyRow validates a body against the known rules and converts it to a store row. enabled is the value to
|
||||
// keep when the body omits it.
|
||||
func (s *server) policyRow(b AlertPolicyBody, enabled bool) (store.AlertPolicy, error) {
|
||||
h, err := s.policyHost()
|
||||
if err != nil {
|
||||
return store.AlertPolicy{}, err
|
||||
}
|
||||
p := alertpolicy.Policy{Name: strings.TrimSpace(b.Name), Rule: b.Rule, Action: b.Action, Priority: b.Priority,
|
||||
Enabled: enabled, Threshold: b.Threshold, Scope: b.Scope}
|
||||
if b.Enabled != nil {
|
||||
p.Enabled = *b.Enabled
|
||||
}
|
||||
if p.Action == "" {
|
||||
p.Action = alertpolicy.ActionAlert
|
||||
}
|
||||
if b.Severity != nil {
|
||||
p.Severity = *b.Severity
|
||||
}
|
||||
if b.For != nil {
|
||||
d, err := time.ParseDuration(*b.For)
|
||||
if err != nil {
|
||||
return store.AlertPolicy{}, huma.Error422UnprocessableEntity("for: " + err.Error())
|
||||
}
|
||||
p.For = &d
|
||||
}
|
||||
if err := alertpolicy.Validate(p, ruleLevels(h.Rules())); err != nil {
|
||||
return store.AlertPolicy{}, huma.Error422UnprocessableEntity(flatten(err))
|
||||
}
|
||||
row := store.AlertPolicy{Name: p.Name, Rule: p.Rule, Action: p.Action, Priority: p.Priority, Enabled: p.Enabled,
|
||||
Severity: p.Severity, Threshold: p.Threshold, Scope: alertpolicy.FormatScope(p.Scope)}
|
||||
if p.For != nil {
|
||||
secs := int64(p.For.Seconds())
|
||||
row.ForSeconds = &secs
|
||||
}
|
||||
return row, nil
|
||||
}
|
||||
|
||||
func ruleLevels(rules []AlertRuleInfo) map[string]string {
|
||||
m := make(map[string]string, len(rules))
|
||||
for _, r := range rules {
|
||||
m[r.Name] = r.Level
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func flatten(err error) string {
|
||||
var parts []string
|
||||
for _, line := range strings.Split(err.Error(), "\n") {
|
||||
if line != "" {
|
||||
parts = append(parts, line)
|
||||
}
|
||||
}
|
||||
return strings.Join(parts, "; ")
|
||||
}
|
||||
|
||||
func (s *server) policyChanged() {
|
||||
if s.AlertPolicy != nil {
|
||||
s.AlertPolicy.Changed()
|
||||
}
|
||||
}
|
||||
|
||||
func (s *server) createAlertPolicy(ctx context.Context, in *struct{ Body AlertPolicyBody }) (*struct{ Body AlertPolicyView }, error) {
|
||||
row, err := s.policyRow(in.Body, true)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
err = s.Store.WithTx(ctx, func(tx store.Store) error {
|
||||
if err := tx.CreateAlertPolicy(ctx, &row); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.AppendAudit(ctx, actorOf(ctx), "alert_policy.create", row.ID, nil, in.Body)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.policyChanged()
|
||||
return s.policyResult(row)
|
||||
}
|
||||
|
||||
func (s *server) updateAlertPolicy(ctx context.Context, in *struct {
|
||||
ID string `path:"id" doc:"Policy ULID."`
|
||||
Body AlertPolicyBody
|
||||
}) (*struct{ Body AlertPolicyView }, error) {
|
||||
cur, err := s.Store.GetAlertPolicy(ctx, in.ID)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
row, err := s.policyRow(in.Body, cur.Enabled)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
row.ID, row.CreatedAt = cur.ID, cur.CreatedAt
|
||||
err = s.Store.WithTx(ctx, func(tx store.Store) error {
|
||||
if err := tx.UpdateAlertPolicy(ctx, &row); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.AppendAudit(ctx, actorOf(ctx), "alert_policy.update", row.ID, cur, in.Body)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.policyChanged()
|
||||
return s.policyResult(row)
|
||||
}
|
||||
|
||||
func (s *server) policyResult(row store.AlertPolicy) (*struct{ Body AlertPolicyView }, error) {
|
||||
v, err := policyView(row)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
return &struct{ Body AlertPolicyView }{v}, nil
|
||||
}
|
||||
|
||||
func (s *server) deleteAlertPolicy(ctx context.Context, in *struct {
|
||||
ID string `path:"id" doc:"Policy ULID."`
|
||||
}) (*struct{}, error) {
|
||||
err := s.Store.WithTx(ctx, func(tx store.Store) error {
|
||||
cur, err := tx.GetAlertPolicy(ctx, in.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.DeleteAlertPolicy(ctx, in.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.AppendAudit(ctx, actorOf(ctx), "alert_policy.delete", in.ID, cur, nil)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.policyChanged()
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (s *server) previewAlertPolicy(ctx context.Context, in *struct{ Body AlertPolicyPreviewBody }) (*struct{ Body PolicyPreview }, error) {
|
||||
h, err := s.policyHost()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
draft := alertpolicy.Policy{Name: "preview", Rule: in.Body.Rule, Action: alertpolicy.ActionAlert, Scope: in.Body.Scope}
|
||||
if err := alertpolicy.Validate(draft, ruleLevels(h.Rules())); err != nil {
|
||||
return nil, huma.Error422UnprocessableEntity(flatten(err))
|
||||
}
|
||||
p, err := h.Preview(ctx, in.Body.Rule, in.Body.Scope)
|
||||
if err != nil {
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
return nil, huma.Error404NotFound("not found")
|
||||
}
|
||||
s.Logger.Error("alert policy preview", "err", err)
|
||||
return nil, huma.Error500InternalServerError("internal error")
|
||||
}
|
||||
return &struct{ Body PolicyPreview }{p}, nil
|
||||
}
|
||||
112
internal/api/alert_policies_test.go
Normal file
@@ -0,0 +1,112 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alertpolicy"
|
||||
"github.com/v10networks/nmon/internal/api"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
type fakePolicyHost struct {
|
||||
changes int
|
||||
preview api.PolicyPreview
|
||||
}
|
||||
|
||||
func (f *fakePolicyHost) Rules() []api.AlertRuleInfo {
|
||||
thr := 0.1
|
||||
return []api.AlertRuleInfo{
|
||||
{Name: "interface_errors", Level: "interface", Op: ">", Threshold: &thr, Severity: "warning"},
|
||||
{Name: "cpu_high", Level: "device", Op: ">", Threshold: &thr, Severity: "warning"},
|
||||
{Name: "device_down", Level: "device", Severity: "critical"},
|
||||
{Name: "system", Level: "system", Severity: "warning"},
|
||||
}
|
||||
}
|
||||
func (f *fakePolicyHost) Changed() { f.changes++ }
|
||||
func (f *fakePolicyHost) Preview(context.Context, string, alertpolicy.Scope) (api.PolicyPreview, error) {
|
||||
return f.preview, nil
|
||||
}
|
||||
|
||||
func policyEnv(t *testing.T) (*env, *fakePolicyHost) {
|
||||
h := &fakePolicyHost{preview: api.PolicyPreview{Devices: api.PreviewDevices{Count: 2, Samples: []api.DeviceRef{}}}}
|
||||
return newEnvDeps(t, 0, func(d *api.Deps) { d.AlertPolicy = h }), h
|
||||
}
|
||||
|
||||
const fiberPolicy = `{"name":"Fiber","rule":"interface_errors","action":"alert","threshold":0,"for":"5m","severity":"critical",
|
||||
"scope":{"interfaces":{"if_types":[6],"if_alias":"(?i)fib"}}}`
|
||||
|
||||
func TestAlertPolicyCRUDIsAuditedAndTellsTheEngine(t *testing.T) {
|
||||
e, host := policyEnv(t)
|
||||
rec := e.do("POST", "/api/v1/alert-policies", fiberPolicy)
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("POST: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
var v api.AlertPolicyView
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &v); err != nil || v.ID == "" || !v.Enabled || *v.For != "5m0s" || *v.Threshold != 0 {
|
||||
t.Fatalf("created = %s (%v)", rec.Body, err)
|
||||
}
|
||||
isProblem(t, e.do("POST", "/api/v1/alert-policies", fiberPolicy), http.StatusConflict)
|
||||
upd := `{"name":"Fiber","rule":"interface_errors","action":"ignore","priority":5,"enabled":false,"scope":{"all":true}}`
|
||||
rec = e.do("PUT", "/api/v1/alert-policies/"+v.ID, upd)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("PUT: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
var list api.AlertPolicyList
|
||||
e.getJSON(t, "/api/v1/alert-policies", &list)
|
||||
if len(list.Items) != 1 || list.Items[0].Action != "ignore" || list.Items[0].Enabled || list.Items[0].Threshold != nil || !list.Items[0].Scope.All {
|
||||
t.Fatalf("list = %+v", list)
|
||||
}
|
||||
if rec := e.do("DELETE", "/api/v1/alert-policies/"+v.ID, ""); rec.Code != http.StatusNoContent {
|
||||
t.Fatalf("DELETE: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
isProblem(t, e.do("GET", "/api/v1/alert-policies/"+v.ID, ""), http.StatusNotFound)
|
||||
if host.changes != 3 {
|
||||
t.Errorf("engine told %d times, want 3", host.changes)
|
||||
}
|
||||
audit, _, _ := e.st.ListAudit(t.Context(), store.Page{})
|
||||
if len(audit) < 3 || audit[0].Action != "alert_policy.delete" {
|
||||
t.Fatalf("audit = %+v", audit)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAlertPolicyValidationRefusesBadScopes(t *testing.T) {
|
||||
e, _ := policyEnv(t)
|
||||
for name, body := range map[string]string{
|
||||
"empty scope": `{"name":"a","rule":"cpu_high","action":"alert","scope":{}}`,
|
||||
"unknown rule": `{"name":"a","rule":"nope","action":"alert","scope":{"all":true}}`,
|
||||
"interface on device": `{"name":"a","rule":"cpu_high","action":"alert","scope":{"interfaces":{"if_types":[6]}}}`,
|
||||
"system needs all": `{"name":"a","rule":"system","action":"alert","scope":{"groups":["1"]}}`,
|
||||
"bad regex": `{"name":"a","rule":"interface_errors","action":"alert","scope":{"interfaces":{"if_name":"("}}}`,
|
||||
"bad duration": `{"name":"a","rule":"cpu_high","action":"alert","for":"soon","scope":{"all":true}}`,
|
||||
"unknown action": `{"name":"a","rule":"cpu_high","action":"maybe","scope":{"all":true}}`,
|
||||
} {
|
||||
rec := e.do("POST", "/api/v1/alert-policies", body)
|
||||
if rec.Code != http.StatusUnprocessableEntity {
|
||||
t.Errorf("%s: %d %s", name, rec.Code, rec.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAlertRulesAndPreview(t *testing.T) {
|
||||
e, _ := policyEnv(t)
|
||||
var rules api.AlertRuleList
|
||||
e.getJSON(t, "/api/v1/alert-rules", &rules)
|
||||
if len(rules.Items) != 4 {
|
||||
t.Fatalf("rules = %+v", rules)
|
||||
}
|
||||
rec := e.do("POST", "/api/v1/alert-policies/preview", `{"rule":"device_down","scope":{"all":true}}`)
|
||||
var p api.PolicyPreview
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &p); rec.Code != http.StatusOK || err != nil || p.Devices.Count != 2 {
|
||||
t.Fatalf("preview = %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
isProblem(t, e.do("POST", "/api/v1/alert-policies/preview", `{"rule":"device_down","scope":{}}`), http.StatusUnprocessableEntity)
|
||||
}
|
||||
|
||||
func TestAlertPolicyRoutesAnswer503WithoutAnEngine(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, nil)
|
||||
isProblem(t, e.do("GET", "/api/v1/alert-rules", ""), http.StatusServiceUnavailable)
|
||||
isProblem(t, e.do("POST", "/api/v1/alert-policies", fiberPolicy), http.StatusServiceUnavailable)
|
||||
}
|
||||
@@ -78,6 +78,9 @@ type Deps struct {
|
||||
// OnMaintenanceChange is called after a maintenance window is created or deleted, so the
|
||||
// evaluators see it at once instead of at their next refresh.
|
||||
OnMaintenanceChange func(ctx context.Context)
|
||||
// OnAlertDownChange is called after a write that can change which ports alert on interface_down (a port or
|
||||
// group flag, group membership), so the evaluator sees it at once instead of after its cache expires.
|
||||
OnAlertDownChange func()
|
||||
|
||||
// PollInterval is the normal poll cadence; a device unheard of for three of them is down. Default 60 s.
|
||||
PollInterval time.Duration
|
||||
@@ -117,8 +120,17 @@ type Deps struct {
|
||||
Series SeriesDeleter
|
||||
// PurgeAfter is how long a device must have been retired before it can be purged. Default one year.
|
||||
PurgeAfter func() time.Duration
|
||||
// AlertPolicy is the running policy engine; routes under /alert-policies and /alert-rules answer 503 without it.
|
||||
AlertPolicy AlertPolicyHost
|
||||
// OnImported is told the ids of devices a CSV import created, so they can be discovered at once.
|
||||
OnImported func(ids []int64)
|
||||
|
||||
// v1.3 collaborators; all optional. Vendors names MAC vendors from the IEEE registry (nil: vendor is empty
|
||||
// everywhere). L2EdgeMax returns l2.edge_max_macs. L2Health and HealthNotes feed GET /poller/health.
|
||||
Vendors VendorLookup
|
||||
L2EdgeMax func() int
|
||||
L2Health func() *L2Health
|
||||
HealthNotes func() []HealthNote
|
||||
}
|
||||
|
||||
type server struct {
|
||||
@@ -185,14 +197,20 @@ func (s *server) register(api huma.API) {
|
||||
s.registerDevices(api)
|
||||
s.registerDeviceList(api)
|
||||
s.registerDeviceViews(api)
|
||||
s.registerProfiles(api)
|
||||
s.registerInterfaces(api)
|
||||
s.registerAlertDown(api)
|
||||
s.registerGraph(api)
|
||||
s.registerDeviceMetrics(api)
|
||||
s.registerOverview(api)
|
||||
s.registerPrinters(api)
|
||||
s.registerOps(api)
|
||||
s.registerHealth(api)
|
||||
s.registerReviews(api)
|
||||
s.registerQuery(api)
|
||||
s.registerMeta(api)
|
||||
s.registerAlerts(api)
|
||||
s.registerAlertPolicies(api)
|
||||
s.registerMaintenance(api)
|
||||
s.registerMetrics(api)
|
||||
s.registerTokens(api)
|
||||
@@ -207,6 +225,9 @@ func (s *server) register(api huma.API) {
|
||||
s.registerSettings(api)
|
||||
s.registerImport(api)
|
||||
s.registerPurge(api)
|
||||
s.registerL2(api)
|
||||
s.registerBGP(api)
|
||||
s.registerRetire(api)
|
||||
s.registerAdmin(api)
|
||||
s.registerAudit(api)
|
||||
}
|
||||
|
||||
335
internal/api/bgp.go
Normal file
@@ -0,0 +1,335 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/profile"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// The profile metrics BGP peers are built from. Every BGP profile uses these names whatever MIB feeds them, so
|
||||
// the rule, the API and the UI never care which vendor a peer comes from.
|
||||
const (
|
||||
bgpState = "bgp_peer_state"
|
||||
bgpUptime = "bgp_peer_uptime"
|
||||
bgpPrefixes = "bgp_prefixes_accepted"
|
||||
bgpInUpd = "bgp_peer_in_updates"
|
||||
bgpOutUpd = "bgp_peer_out_updates"
|
||||
|
||||
stateEstablished = "established"
|
||||
stateUnknown = "unknown"
|
||||
// rateWindow is long enough to hold several 5-minute samples of an update counter.
|
||||
rateWindow = "15m"
|
||||
)
|
||||
|
||||
func (s *server) registerBGP(api huma.API) {
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "list-bgp-peers", Method: http.MethodGet, Path: "/api/v1/bgp/peers", Tags: []string{"bgp"},
|
||||
Summary: "BGP sessions of every device",
|
||||
Description: "Every BGP peer the profiles collect, with the latest state, uptime, accepted prefixes per address family and " +
|
||||
"update rates from the hot store, and the peer AS, local AS, router ID and description from discovery. Sessions that " +
|
||||
"are down come first. state is established, down (anything else) or an FSM state name.",
|
||||
Errors: []int{http.StatusTooManyRequests, http.StatusServiceUnavailable},
|
||||
}, s.listBGPPeers)
|
||||
}
|
||||
|
||||
// bgpDevice is a device whose profile collects BGP, with its peer rows.
|
||||
type bgpDevice struct {
|
||||
dev store.Device
|
||||
state profile.Metric
|
||||
comps []store.Component
|
||||
}
|
||||
|
||||
func (s *server) listBGPPeers(ctx context.Context, in *struct {
|
||||
Device string `query:"device" doc:"Only this device (slug)."`
|
||||
State string `query:"state" doc:"established, down, or an FSM state: idle, connect, active, opensent, openconfirm."`
|
||||
AS string `query:"as" pattern:"^[0-9]{1,10}$" doc:"Only sessions with this remote AS."`
|
||||
Family string `query:"family" enum:"ipv4,ipv6" doc:"Only IPv4 or only IPv6 peers."`
|
||||
Q string `query:"q" doc:"Substring of the peer address, description or device name."`
|
||||
PageParams
|
||||
}) (*struct{ Body BGPPeerList }, error) {
|
||||
devs, err := s.bgpDevices(ctx, in.Device)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
peers, err := s.bgpPeers(ctx, devs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
kept := peers[:0]
|
||||
var counts BGPCounts
|
||||
for _, p := range peers {
|
||||
if !bgpMatches(p, optInt(in.AS), in.Family, in.Q) {
|
||||
continue
|
||||
}
|
||||
counts.Total++
|
||||
if p.Established {
|
||||
counts.Established++
|
||||
} else {
|
||||
counts.Down++
|
||||
}
|
||||
if bgpStateMatches(p, in.State) {
|
||||
kept = append(kept, p)
|
||||
}
|
||||
}
|
||||
sortPeers(kept)
|
||||
total := int64(len(kept))
|
||||
lo := min(in.offset(), total)
|
||||
page := kept[lo:min(lo+in.limit(), total)]
|
||||
out := BGPPeerList{Items: slices.Clone(page), Page: in.pageNo(), PerPage: in.limit(), Total: total, Counts: counts}
|
||||
return &struct{ Body BGPPeerList }{out}, nil
|
||||
}
|
||||
|
||||
// bgpDevices lists the devices whose profile collects BGP and which have peer rows.
|
||||
func (s *server) bgpDevices(ctx context.Context, only string) ([]bgpDevice, error) {
|
||||
if s.Profiles == nil {
|
||||
return nil, nil
|
||||
}
|
||||
var out []bgpDevice
|
||||
for off := int64(0); ; off += store.MaxLimit {
|
||||
rows, n, err := s.Store.ListDevices(ctx, store.DeviceFilter{Kind: store.KindNetwork}, store.Page{Offset: off, Limit: store.MaxLimit})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
for _, d := range rows {
|
||||
if only != "" && d.Slug != only {
|
||||
continue
|
||||
}
|
||||
if bd, ok, err := s.bgpDeviceOf(ctx, d); err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
} else if ok {
|
||||
out = append(out, bd)
|
||||
}
|
||||
}
|
||||
if off+store.MaxLimit >= n {
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *server) bgpDeviceOf(ctx context.Context, d store.Device) (bgpDevice, bool, error) {
|
||||
res := s.Profiles.Get(d.Profile)
|
||||
if res == nil {
|
||||
return bgpDevice{}, false, nil
|
||||
}
|
||||
m, ok := res.Metric(bgpState)
|
||||
if !ok || !m.Pollable() {
|
||||
return bgpDevice{}, false, nil
|
||||
}
|
||||
all, err := s.Store.ListComponents(ctx, d.ID)
|
||||
if err != nil {
|
||||
return bgpDevice{}, false, err
|
||||
}
|
||||
bd := bgpDevice{dev: d, state: m}
|
||||
for _, c := range all {
|
||||
if c.TableName == m.TableKey() {
|
||||
bd.comps = append(bd.comps, c)
|
||||
}
|
||||
}
|
||||
return bd, len(bd.comps) > 0, nil
|
||||
}
|
||||
|
||||
// bgpPeers builds the peers of the devices from the inventory and the hot store.
|
||||
func (s *server) bgpPeers(ctx context.Context, devs []bgpDevice) ([]BGPPeer, error) {
|
||||
if len(devs) == 0 {
|
||||
return []BGPPeer{}, nil
|
||||
}
|
||||
slugs := make([]string, len(devs))
|
||||
for i, d := range devs {
|
||||
slugs[i] = d.dev.Slug
|
||||
}
|
||||
match := `device=~` + quote(regexAlt(slugs))
|
||||
latest, err := s.queryLatest(ctx, match, []string{bgpState, bgpUptime, bgpPrefixes})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rates, err := s.queryRates(ctx, match, []string{bgpInUpd, bgpOutUpd})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := []BGPPeer{}
|
||||
for _, d := range devs {
|
||||
out = append(out, peersOf(d, latest[d.dev.Slug], rates[d.dev.Slug])...)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// rateReadings is metric -> component -> per-second rate.
|
||||
type rateReadings map[string]map[string]float64
|
||||
|
||||
// queryRates reads the per-second rate of counters over rateWindow for every device in match.
|
||||
func (s *server) queryRates(ctx context.Context, match string, names []string) (map[string]rateReadings, error) {
|
||||
parts := make([]string, len(names))
|
||||
for i, n := range names {
|
||||
parts[i] = `label_set(rate(` + n + `{` + match + `}[` + rateWindow + `]), "k", ` + quote(n) + `)`
|
||||
}
|
||||
release, err := s.admit(ctx, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer release()
|
||||
qctx, cancel := context.WithTimeout(ctx, QueryTimeout)
|
||||
defer cancel()
|
||||
body, err := s.Querier.Query(qctx, "hot", strings.Join(parts, " or "), s.Clock.Now())
|
||||
if err != nil {
|
||||
return nil, s.vmErr(qctx, err)
|
||||
}
|
||||
rows, err := parseVM(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("bgp rates: %w", err)
|
||||
}
|
||||
out := map[string]rateReadings{}
|
||||
for _, r := range rows {
|
||||
if r.Value == nil || !r.Value.OK {
|
||||
continue
|
||||
}
|
||||
dev, k := r.Metric["device"], r.Metric["k"]
|
||||
if out[dev] == nil {
|
||||
out[dev] = rateReadings{}
|
||||
}
|
||||
if out[dev][k] == nil {
|
||||
out[dev][k] = map[string]float64{}
|
||||
}
|
||||
out[dev][k][r.Metric["component"]] = r.Value.V
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// peersOf builds the peers of one device.
|
||||
func peersOf(d bgpDevice, rd readings, rates rateReadings) []BGPPeer {
|
||||
seen := map[string]int{}
|
||||
for _, c := range d.comps {
|
||||
seen[componentLabel(c)]++
|
||||
}
|
||||
out := make([]BGPPeer, 0, len(d.comps))
|
||||
for _, c := range d.comps {
|
||||
peer := componentLabel(c)
|
||||
series := peer // the poller keeps rows with equal labels apart with the index
|
||||
if seen[peer] > 1 {
|
||||
series += "#" + c.Idx
|
||||
}
|
||||
out = append(out, buildPeer(d, c, peer, series, rd, rates))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func componentLabel(c store.Component) string {
|
||||
if c.Label != "" {
|
||||
return c.Label
|
||||
}
|
||||
return c.Idx
|
||||
}
|
||||
|
||||
func buildPeer(d bgpDevice, c store.Component, peer, series string, rd readings, rates rateReadings) BGPPeer {
|
||||
var attrs map[string]string
|
||||
_ = json.Unmarshal([]byte(c.Attrs), &attrs) // attrs are written by discovery; unreadable ones are simply absent
|
||||
p := BGPPeer{
|
||||
Device: DeviceRef{Slug: d.dev.Slug, Name: d.dev.Name}, Peer: peer, Component: series, Family: peerFamily(peer),
|
||||
RemoteAS: parseAS(attrs["remote_as"]), LocalAS: parseAS(attrs["local_as"]), RouterID: attrs["router_id"],
|
||||
Description: attrs["description"], State: stateUnknown, StateClass: stateUnknown, Prefixes: []BGPFamily{},
|
||||
}
|
||||
if r := rd.at(bgpState, series); r != nil {
|
||||
name := d.state.Enum[int(r.v)]
|
||||
if name == "" {
|
||||
name = stateUnknown
|
||||
}
|
||||
p.State, p.StateClass, p.Established = name, stateOf(d.state, name), name == stateEstablished
|
||||
t := r.ts
|
||||
p.Sampled = &t
|
||||
}
|
||||
if r := rd.at(bgpUptime, series); r != nil {
|
||||
v := r.v
|
||||
p.UptimeS = &v
|
||||
}
|
||||
p.InUpdatesPerS, p.OutUpdatesPerS = rateAt(rates, bgpInUpd, series), rateAt(rates, bgpOutUpd, series)
|
||||
p.Prefixes = prefixFamilies(rd[bgpPrefixes], series)
|
||||
return p
|
||||
}
|
||||
|
||||
func rateAt(rates rateReadings, metric, comp string) *float64 {
|
||||
if v, ok := rates[metric][comp]; ok {
|
||||
return &v
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// prefixFamilies reads the "<peer>|<family>" components of the prefix metric that belong to the peer.
|
||||
func prefixFamilies(by map[string]*reading, series string) []BGPFamily {
|
||||
out := []BGPFamily{}
|
||||
for comp, r := range by {
|
||||
if fam, ok := strings.CutPrefix(comp, series+"|"); ok {
|
||||
v := r.v
|
||||
out = append(out, BGPFamily{Family: fam, Accepted: &v})
|
||||
}
|
||||
}
|
||||
slices.SortFunc(out, func(a, b BGPFamily) int { return strings.Compare(a.Family, b.Family) })
|
||||
return out
|
||||
}
|
||||
|
||||
func peerFamily(peer string) string {
|
||||
if a, err := netip.ParseAddr(peer); err == nil && a.Is6() {
|
||||
return "ipv6"
|
||||
}
|
||||
return "ipv4"
|
||||
}
|
||||
|
||||
func parseAS(s string) *int64 {
|
||||
if v, err := strconv.ParseInt(s, 10, 64); err == nil && v >= 0 {
|
||||
return &v
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func bgpMatches(p BGPPeer, as *int64, family, q string) bool {
|
||||
if as != nil && (p.RemoteAS == nil || *p.RemoteAS != *as) {
|
||||
return false
|
||||
}
|
||||
if family != "" && p.Family != family {
|
||||
return false
|
||||
}
|
||||
if q == "" {
|
||||
return true
|
||||
}
|
||||
q = strings.ToLower(q)
|
||||
for _, f := range []string{p.Peer, p.Description, p.Device.Name, p.Device.Slug, p.RouterID} {
|
||||
if strings.Contains(strings.ToLower(f), q) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func bgpStateMatches(p BGPPeer, state string) bool {
|
||||
switch state {
|
||||
case "":
|
||||
return true
|
||||
case "down":
|
||||
return !p.Established
|
||||
}
|
||||
return p.State == state
|
||||
}
|
||||
|
||||
// sortPeers puts the sessions that are down first, then orders by device and peer.
|
||||
func sortPeers(ps []BGPPeer) {
|
||||
slices.SortStableFunc(ps, func(a, b BGPPeer) int {
|
||||
if a.Established != b.Established {
|
||||
if a.Established {
|
||||
return 1
|
||||
}
|
||||
return -1
|
||||
}
|
||||
if c := strings.Compare(a.Device.Slug, b.Device.Slug); c != 0 {
|
||||
return c
|
||||
}
|
||||
return strings.Compare(a.Peer, b.Peer)
|
||||
})
|
||||
}
|
||||
116
internal/api/bgp_test.go
Normal file
@@ -0,0 +1,116 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/api"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// bgpRouter seeds an Arista with an IPv4 and an IPv6 peer (the latter down) and the readings the hot store would hold.
|
||||
func bgpRouter(t *testing.T, e *env, name, profileName string) *store.Device {
|
||||
t.Helper()
|
||||
d := e.device(t, name, "yyz", "router", func(d *store.Device) { d.Profile = profileName })
|
||||
table := "ARISTA-BGP4V2-MIB::aristaBgp4V2PeerTable"
|
||||
err := e.st.ReplaceComponents(context.Background(), d.ID, table, []store.Component{
|
||||
{Idx: "1.1.4.192.0.2.1", Label: "192.0.2.1", Attrs: `{"remote_as":"65001","local_as":"65000","description":"transit-a","router_id":"198.51.100.1"}`},
|
||||
{Idx: "1.2.16.32.1.13.184.0.0.0.0.0.0.0.0.0.0.0.2", Label: "2001:db8::2", Attrs: `{"remote_as":"65002","local_as":"65000"}`},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func bgpBody(dev string) func(string, string, time.Time, time.Time, time.Duration) ([]byte, error) {
|
||||
return func(_, expr string, _, _ time.Time, _ time.Duration) ([]byte, error) {
|
||||
if strings.Contains(expr, "rate(") {
|
||||
return vector(
|
||||
vrow{lbl(dev, "bgp_peer_in_updates", "192.0.2.1"), 0.25}, vrow{lbl(dev, "bgp_peer_out_updates", "192.0.2.1"), 0.5},
|
||||
), nil
|
||||
}
|
||||
var rows []vrow
|
||||
rows = append(rows, rd(dev, "bgp_peer_state", "192.0.2.1", 6)...)
|
||||
rows = append(rows, rd(dev, "bgp_peer_state", "2001:db8::2", 3)...) // active
|
||||
rows = append(rows, rd(dev, "bgp_peer_uptime", "192.0.2.1", 86400)...)
|
||||
rows = append(rows, rd(dev, "bgp_prefixes_accepted", "192.0.2.1|ipv4-unicast", 540000)...)
|
||||
rows = append(rows, rd(dev, "bgp_prefixes_accepted", "192.0.2.1|ipv6-unicast", 12)...)
|
||||
return vector(rows...), nil
|
||||
}
|
||||
}
|
||||
|
||||
func TestBGPPeersListsSessionsDownFirstWithAttributesFromTheInventory(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
d := bgpRouter(t, e, "edge1", "arista-eos")
|
||||
e.device(t, "plain-switch", "yyz", "access") // generic: no BGP, never listed
|
||||
e.q.fn = bgpBody(d.Slug)
|
||||
|
||||
var got api.BGPPeerList
|
||||
e.getJSON(t, "/api/v1/bgp/peers", &got)
|
||||
if got.Total != 2 || len(got.Items) != 2 || got.Counts.Total != 2 || got.Counts.Established != 1 || got.Counts.Down != 1 {
|
||||
t.Fatalf("list = %+v", got)
|
||||
}
|
||||
down, up := got.Items[0], got.Items[1]
|
||||
if down.Peer != "2001:db8::2" || down.Family != "ipv6" || down.State != "active" || down.StateClass != "crit" || down.Established {
|
||||
t.Errorf("the down IPv6 session must come first: %+v", down)
|
||||
}
|
||||
if *down.RemoteAS != 65002 || *down.LocalAS != 65000 || down.Description != "" {
|
||||
t.Errorf("down attrs = %+v", down)
|
||||
}
|
||||
if up.Component != "192.0.2.1" {
|
||||
t.Errorf("component = %q", up.Component)
|
||||
}
|
||||
if up.Peer != "192.0.2.1" || up.State != "established" || up.StateClass != "ok" || !up.Established || up.Description != "transit-a" ||
|
||||
up.RouterID != "198.51.100.1" || *up.RemoteAS != 65001 || *up.UptimeS != 86400 {
|
||||
t.Errorf("up = %+v", up)
|
||||
}
|
||||
if len(up.Prefixes) != 2 || up.Prefixes[0].Family != "ipv4-unicast" || *up.Prefixes[0].Accepted != 540000 || up.Prefixes[1].Family != "ipv6-unicast" {
|
||||
t.Errorf("prefixes = %+v", up.Prefixes)
|
||||
}
|
||||
if up.InUpdatesPerS == nil || *up.InUpdatesPerS != 0.25 || *up.OutUpdatesPerS != 0.5 || down.InUpdatesPerS != nil {
|
||||
t.Errorf("update rates = %v %v / %v", up.InUpdatesPerS, up.OutUpdatesPerS, down.InUpdatesPerS)
|
||||
}
|
||||
if up.Device.Slug != d.Slug || up.Sampled == nil {
|
||||
t.Errorf("device/sampled = %+v", up)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBGPPeersFiltersByStateASFamilyDeviceAndText(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
d := bgpRouter(t, e, "edge1", "arista-eos")
|
||||
e.q.fn = bgpBody(d.Slug)
|
||||
for path, want := range map[string]int64{
|
||||
"?state=down": 1, "?state=established": 1, "?state=active": 1, "?state=idle": 0, "?as=65001": 1, "?as=1": 0,
|
||||
"?family=ipv6": 1, "?family=ipv4": 1, "?device=" + d.Slug: 2, "?device=nope": 0, "?q=transit": 1, "?q=EDGE1": 2, "?q=2001:db8": 1,
|
||||
} {
|
||||
var got api.BGPPeerList
|
||||
e.getJSON(t, "/api/v1/bgp/peers"+path, &got)
|
||||
if got.Total != want {
|
||||
t.Errorf("%s: total %d, want %d", path, got.Total, want)
|
||||
}
|
||||
}
|
||||
var got api.BGPPeerList
|
||||
e.getJSON(t, "/api/v1/bgp/peers?state=down", &got)
|
||||
if got.Counts.Total != 2 {
|
||||
t.Errorf("counts must ignore the state filter: %+v", got.Counts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBGPPeersWithNoReadingIsUnknownAndCountsAsDown(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
bgpRouter(t, e, "edge1", "juniper-junos") // profile with a different table key: no components match
|
||||
var got api.BGPPeerList
|
||||
e.getJSON(t, "/api/v1/bgp/peers", &got)
|
||||
if got.Total != 0 || got.Items == nil {
|
||||
t.Errorf("a router whose rows are in another table lists nothing: %+v", got)
|
||||
}
|
||||
e2 := newEnvDeps(t, 0, withShipped(t))
|
||||
bgpRouter(t, e2, "edge2", "arista-eos") // store has no readings at all
|
||||
e2.getJSON(t, "/api/v1/bgp/peers", &got)
|
||||
if got.Total != 2 || got.Items[0].State != "unknown" || got.Counts.Down != 2 {
|
||||
t.Errorf("no readings: %+v", got)
|
||||
}
|
||||
}
|
||||
52
internal/api/bgp_types.go
Normal file
@@ -0,0 +1,52 @@
|
||||
package api
|
||||
|
||||
import "time"
|
||||
|
||||
// BGPFamily is the prefixes a peer has accepted in one address family.
|
||||
type BGPFamily struct {
|
||||
Family string `json:"family" example:"ipv6-unicast" doc:"Address family and subsequent family, for example ipv4-unicast."`
|
||||
Accepted *float64 `json:"accepted" nullable:"true" doc:"Prefixes accepted from the peer; null when unread."`
|
||||
}
|
||||
|
||||
// BGPPeer is one BGP session of one device.
|
||||
type BGPPeer struct {
|
||||
Device DeviceRef `json:"device"`
|
||||
Peer string `json:"peer" doc:"The peer address, canonical; IPv6 compressed."`
|
||||
// Component is the label of the peer's metric series: the peer address, with #<index> added when two rows of
|
||||
// the device share an address (two routing instances). Pass it to /devices/{slug}/metrics/graph.
|
||||
Component string `json:"component"`
|
||||
// Family is the address family of the peer address itself: ipv4 or ipv6.
|
||||
Family string `json:"family" enum:"ipv4,ipv6"`
|
||||
RemoteAS *int64 `json:"remote_as" nullable:"true"`
|
||||
LocalAS *int64 `json:"local_as" nullable:"true"`
|
||||
RouterID string `json:"router_id" doc:"The peer's BGP identifier, when the device reports it."`
|
||||
Description string `json:"description" doc:"The configured neighbor description, where the device's MIB carries it (Arista)."`
|
||||
// State is the BGP FSM state: idle, connect, active, opensent, openconfirm or established; "unknown" without a reading.
|
||||
State string `json:"state"`
|
||||
StateClass string `json:"state_class" enum:"ok,warn,crit,unknown"`
|
||||
Established bool `json:"established"`
|
||||
// UptimeS is how long the session has been established, in seconds.
|
||||
UptimeS *float64 `json:"uptime_s" nullable:"true"`
|
||||
Prefixes []BGPFamily `json:"prefixes" doc:"Accepted prefixes per address family, where the device reports them."`
|
||||
InUpdatesPerS *float64 `json:"in_updates_per_s" nullable:"true" doc:"UPDATE messages received per second over the last 15 minutes."`
|
||||
OutUpdatesPerS *float64 `json:"out_updates_per_s" nullable:"true"`
|
||||
// Sampled is when the state was last read.
|
||||
Sampled *time.Time `json:"sampled" nullable:"true"`
|
||||
}
|
||||
|
||||
// BGPCounts are the session counts of a listing before the state filter.
|
||||
type BGPCounts struct {
|
||||
Total int64 `json:"total"`
|
||||
Established int64 `json:"established"`
|
||||
Down int64 `json:"down" doc:"Sessions not established, including those with no reading."`
|
||||
}
|
||||
|
||||
// BGPPeerList is a page of BGP peers, sessions that are down first.
|
||||
type BGPPeerList struct {
|
||||
Items []BGPPeer `json:"items"`
|
||||
Page int64 `json:"page"`
|
||||
PerPage int64 `json:"per_page"`
|
||||
Total int64 `json:"total"`
|
||||
// Counts cover the device, AS and q filters but not the state filter, for the page's summary line.
|
||||
Counts BGPCounts `json:"counts"`
|
||||
}
|
||||
344
internal/api/device_metrics.go
Normal file
@@ -0,0 +1,344 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/profile"
|
||||
)
|
||||
|
||||
// latestWindow is how far back the latest reading of a metric may be; the slowest profile interval is 15 minutes.
|
||||
const latestWindow = "1h"
|
||||
|
||||
var categoryLabels = map[string]string{
|
||||
"system": "System", "health": "Health", "power": "Power", "environment": "Environment", "wireless": "Wireless",
|
||||
"printer": "Printer", "routing": "Routing", "optics": "Optics", "firewall": "Firewall",
|
||||
}
|
||||
|
||||
func (s *server) registerDeviceMetrics(api huma.API) {
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "get-device-metrics", Method: http.MethodGet, Path: "/api/v1/devices/{slug}/metrics", Tags: []string{"devices"},
|
||||
Summary: "Latest non-interface metrics of a device, by category",
|
||||
Description: "Driven by the device's profile: each pollable metric names a category, a label and a display. Values come " +
|
||||
"from the hot store in one batched query. max is set for metrics that are a fraction of another (profile of), percent " +
|
||||
"is computed here, sentinel values (a printer's -3) come back as text, and enum metrics carry a state. Categories " +
|
||||
"with no data are left out.",
|
||||
Errors: []int{http.StatusNotFound, http.StatusTooManyRequests, http.StatusServiceUnavailable},
|
||||
}, s.deviceMetrics)
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "get-device-metric-graph", Method: http.MethodGet, Path: "/api/v1/devices/{slug}/metrics/graph", Tags: []string{"graph"},
|
||||
Summary: "One metric component over time",
|
||||
Description: "Gauges raw, counters as a per-second rate, percent displays as percent. The step rules are those of /graph; " +
|
||||
"only the hot store's retention is searched.",
|
||||
Errors: []int{http.StatusNotFound, http.StatusUnprocessableEntity, http.StatusTooManyRequests, http.StatusServiceUnavailable},
|
||||
}, s.metricGraph)
|
||||
}
|
||||
|
||||
// reading is the latest sample of one metric component.
|
||||
type reading struct {
|
||||
v float64
|
||||
ts time.Time
|
||||
hasTS bool
|
||||
}
|
||||
|
||||
// readings is metric -> component -> latest sample.
|
||||
type readings map[string]map[string]*reading
|
||||
|
||||
func (r readings) at(metric, comp string) *reading { return r[metric][comp] }
|
||||
|
||||
// listedMetrics are the metrics a device's tabs can show: polled, not interface metrics, with a category.
|
||||
func listedMetrics(res *profile.Resolved) []profile.Metric {
|
||||
var out []profile.Metric
|
||||
for _, m := range res.Metrics {
|
||||
if m.NeedsCategory() && m.Category != "" {
|
||||
out = append(out, m)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// latestExpr is one query yielding, for every metric, its last value (k=name) and the time of that
|
||||
// sample (k=@name) for the device.
|
||||
func latestExpr(match string, names []string) string {
|
||||
parts := make([]string, 0, 2*len(names))
|
||||
for _, n := range names {
|
||||
sel := n + `{` + match + `}[` + latestWindow + `]`
|
||||
parts = append(parts,
|
||||
`label_set(last_over_time(`+sel+`), "k", `+quote(n)+`)`,
|
||||
`label_set(tlast_over_time(`+sel+`), "k", `+quote("@"+n)+`)`)
|
||||
}
|
||||
return strings.Join(parts, " or ")
|
||||
}
|
||||
|
||||
// queryLatest runs the batched query and indexes the result by device.
|
||||
func (s *server) queryLatest(ctx context.Context, match string, names []string) (map[string]readings, error) {
|
||||
if len(names) == 0 {
|
||||
return map[string]readings{}, nil
|
||||
}
|
||||
release, err := s.admit(ctx, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer release()
|
||||
qctx, cancel := context.WithTimeout(ctx, QueryTimeout)
|
||||
defer cancel()
|
||||
body, err := s.Querier.Query(qctx, "hot", latestExpr(match, names), s.Clock.Now())
|
||||
if err != nil {
|
||||
return nil, s.vmErr(qctx, err)
|
||||
}
|
||||
rows, err := parseVM(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("metrics: %w", err)
|
||||
}
|
||||
return indexReadings(rows), nil
|
||||
}
|
||||
|
||||
// indexReadings sorts result rows by device, metric and component.
|
||||
func indexReadings(rows []vmSeries) map[string]readings {
|
||||
all := map[string]readings{}
|
||||
for _, r := range rows {
|
||||
out := all[r.Metric["device"]]
|
||||
if out == nil {
|
||||
out = readings{}
|
||||
all[r.Metric["device"]] = out
|
||||
}
|
||||
if r.Value == nil || !r.Value.OK {
|
||||
continue
|
||||
}
|
||||
k := r.Metric["k"]
|
||||
isTS := strings.HasPrefix(k, "@")
|
||||
name := strings.TrimPrefix(k, "@")
|
||||
if out[name] == nil {
|
||||
out[name] = map[string]*reading{}
|
||||
}
|
||||
rd := out[name][r.Metric["component"]]
|
||||
if rd == nil {
|
||||
rd = &reading{}
|
||||
out[name][r.Metric["component"]] = rd
|
||||
}
|
||||
if isTS {
|
||||
rd.ts, rd.hasTS = time.Unix(int64(r.Value.V), 0).UTC(), true
|
||||
} else {
|
||||
rd.v = r.Value.V
|
||||
if !rd.hasTS {
|
||||
rd.ts = time.Unix(r.Value.TS, 0).UTC()
|
||||
}
|
||||
}
|
||||
}
|
||||
return all
|
||||
}
|
||||
|
||||
func (s *server) deviceMetrics(ctx context.Context, in *struct {
|
||||
Slug string `path:"slug" doc:"Device slug."`
|
||||
}) (*struct{ Body MetricsResponse }, error) {
|
||||
d, err := s.Store.GetDeviceBySlug(ctx, in.Slug)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
if d.PurgedAt != nil {
|
||||
return nil, huma.Error404NotFound("not found")
|
||||
}
|
||||
out := MetricsResponse{Categories: []MetricCategory{}}
|
||||
if s.Profiles == nil {
|
||||
return &struct{ Body MetricsResponse }{out}, nil
|
||||
}
|
||||
res := s.Profiles.Get(d.Profile)
|
||||
if res == nil {
|
||||
return &struct{ Body MetricsResponse }{out}, nil
|
||||
}
|
||||
ms := listedMetrics(res)
|
||||
names := make([]string, len(ms))
|
||||
for i, m := range ms {
|
||||
names[i] = m.Name
|
||||
}
|
||||
all, err := s.queryLatest(ctx, `device=`+quote(d.Slug), names)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
attrs, err := s.componentAttrs(ctx, d.ID)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
b := &metricsBuilder{res: res, rd: all[d.Slug], attrs: attrs}
|
||||
out.Categories = b.categories(ms)
|
||||
return &struct{ Body MetricsResponse }{out}, nil
|
||||
}
|
||||
|
||||
// componentAttrs loads the discovery attributes of a device's rows, keyed by table and label.
|
||||
func (s *server) componentAttrs(ctx context.Context, deviceID int64) (map[string]map[string]string, error) {
|
||||
rows, err := s.Store.ListComponents(ctx, deviceID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := map[string]map[string]string{}
|
||||
for _, c := range rows {
|
||||
if c.Attrs == "" || c.Attrs == "{}" {
|
||||
continue
|
||||
}
|
||||
var m map[string]string
|
||||
if json.Unmarshal([]byte(c.Attrs), &m) == nil && len(m) > 0 {
|
||||
out[attrKey(c.TableName, c.Label)] = m
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func attrKey(table, label string) string { return table + "\x00" + label }
|
||||
|
||||
// metricsBuilder turns readings into the response, using the profile's presentation fields.
|
||||
type metricsBuilder struct {
|
||||
res *profile.Resolved
|
||||
rd readings
|
||||
attrs map[string]map[string]string
|
||||
}
|
||||
|
||||
func (b *metricsBuilder) categories(ms []profile.Metric) []MetricCategory {
|
||||
byCat := map[string][]MetricGroup{}
|
||||
for _, m := range ms {
|
||||
if b.res.Referenced(m.Name) {
|
||||
continue // a denominator is shown through the metric that refers to it
|
||||
}
|
||||
if g, ok := b.group(m); ok {
|
||||
byCat[m.Category] = append(byCat[m.Category], g)
|
||||
}
|
||||
}
|
||||
out := []MetricCategory{}
|
||||
for _, c := range profile.Categories {
|
||||
if gs := byCat[c]; len(gs) > 0 {
|
||||
out = append(out, MetricCategory{Name: c, Label: categoryLabels[c], Groups: gs})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (b *metricsBuilder) group(m profile.Metric) (MetricGroup, bool) {
|
||||
g := MetricGroup{Metric: m.Name, Label: m.Title(), Unit: b.unitOf(m), Display: m.EffectiveDisplay(), Components: []MetricComponent{}}
|
||||
for comp, rd := range b.rd[m.Name] {
|
||||
g.Components = append(g.Components, b.component(m, comp, rd))
|
||||
}
|
||||
sort.Slice(g.Components, func(i, j int) bool { return g.Components[i].Component < g.Components[j].Component })
|
||||
return g, len(g.Components) > 0
|
||||
}
|
||||
|
||||
// unitOf is the metric's unit, or the scale_by metric's when the value is multiplied by it.
|
||||
func (b *metricsBuilder) unitOf(m profile.Metric) string {
|
||||
if m.ScaleBy != "" {
|
||||
if sb, ok := b.res.Metric(m.ScaleBy); ok {
|
||||
return sb.Unit
|
||||
}
|
||||
}
|
||||
return m.Unit
|
||||
}
|
||||
|
||||
// scaled is the metric's value at comp after scale_by.
|
||||
func (b *metricsBuilder) scaled(m profile.Metric, comp string) (float64, bool) {
|
||||
rd := b.rd.at(m.Name, comp)
|
||||
if rd == nil {
|
||||
return 0, false
|
||||
}
|
||||
if m.ScaleBy == "" {
|
||||
return rd.v, true
|
||||
}
|
||||
if f := b.rd.at(m.ScaleBy, comp); f != nil {
|
||||
return rd.v * f.v, true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
func (b *metricsBuilder) component(m profile.Metric, comp string, rd *reading) MetricComponent {
|
||||
c := MetricComponent{Component: comp, TS: rd.ts}
|
||||
if words, ok := sentinelText(m, rd.v); ok {
|
||||
c.Text = &words
|
||||
} else if v, ok := b.scaled(m, comp); ok && !math.IsNaN(v) {
|
||||
c.Value = &v
|
||||
}
|
||||
b.fillMax(&c, m)
|
||||
b.fillState(&c, m, rd.v)
|
||||
if m.EffectiveDisplay() == "text" && c.Text == nil && c.Value != nil {
|
||||
t := strconv.FormatFloat(*c.Value, 'f', -1, 64)
|
||||
c.Text = &t
|
||||
}
|
||||
c.Attrs = b.attrs[attrKey(m.TableKey(), comp)]
|
||||
return c
|
||||
}
|
||||
|
||||
// sentinelText is the profile's words for a raw value like -3.
|
||||
func sentinelText(m profile.Metric, v float64) (string, bool) {
|
||||
if v != math.Trunc(v) {
|
||||
return "", false
|
||||
}
|
||||
t, ok := m.Sentinels[int(v)]
|
||||
return t, ok
|
||||
}
|
||||
|
||||
// fillMax sets max and percent for a metric that is a fraction of another.
|
||||
func (b *metricsBuilder) fillMax(c *MetricComponent, m profile.Metric) {
|
||||
if m.Of == "" {
|
||||
return
|
||||
}
|
||||
om, ok := b.res.Metric(m.Of)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
mx, ok := b.scaled(om, c.Component)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
c.Max = &mx
|
||||
if c.Value != nil && mx > 0 && *c.Value >= 0 {
|
||||
p := 100 * *c.Value / mx
|
||||
c.Percent = &p
|
||||
}
|
||||
}
|
||||
|
||||
func (b *metricsBuilder) fillState(c *MetricComponent, m profile.Metric, raw float64) {
|
||||
if m.Type != profile.Status {
|
||||
return
|
||||
}
|
||||
name := m.Enum[int(raw)]
|
||||
st := stateOf(m, name)
|
||||
c.State = &st
|
||||
}
|
||||
|
||||
var (
|
||||
critWords = []string{"crit", "fail", "fault", "depleted", "error", "down"}
|
||||
warnWords = []string{"warn", "low", "degrad", "bypass", "test"}
|
||||
unknownWords = []string{"unknown", "other", "notpresent"}
|
||||
)
|
||||
|
||||
// stateOf maps an enum name to ok, warn, crit or unknown: the profile's enum_state, else by name.
|
||||
func stateOf(m profile.Metric, name string) string {
|
||||
if st, ok := m.EnumState[name]; ok {
|
||||
return st
|
||||
}
|
||||
l := strings.ToLower(name)
|
||||
switch {
|
||||
case name == "":
|
||||
return "unknown"
|
||||
case containsAny(l, unknownWords):
|
||||
return "unknown"
|
||||
case containsAny(l, critWords):
|
||||
return "crit"
|
||||
case containsAny(l, warnWords):
|
||||
return "warn"
|
||||
}
|
||||
return "ok"
|
||||
}
|
||||
|
||||
func containsAny(s string, words []string) bool {
|
||||
for _, w := range words {
|
||||
if strings.Contains(s, w) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
167
internal/api/device_metrics_graph.go
Normal file
@@ -0,0 +1,167 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/profile"
|
||||
)
|
||||
|
||||
type metricGraphIn struct {
|
||||
Slug string `path:"slug" doc:"Device slug."`
|
||||
Metric string `query:"metric" required:"true" minLength:"1" doc:"Profile metric name, as /metrics lists it."`
|
||||
Component string `query:"component" doc:"Row label; empty for a device-wide scalar."`
|
||||
From time.Time `query:"from" doc:"Start, RFC 3339. Default one hour before to."`
|
||||
To time.Time `query:"to" doc:"End, RFC 3339. Default now."`
|
||||
Step int64 `query:"step" minimum:"1" doc:"Step in seconds. Default max(60, range/1000) rounded up to whole minutes."`
|
||||
Aggregate string `query:"aggregate" enum:"avg,sum,max" doc:"Without component: one series over all components."`
|
||||
}
|
||||
|
||||
// allComponents, as the component of metricSel, selects every component of the metric.
|
||||
const allComponents = "\x00all"
|
||||
|
||||
// metricSel is the selector of one component of a metric.
|
||||
func metricSel(name, slug, comp string) string {
|
||||
if comp == allComponents {
|
||||
return name + `{device=` + quote(slug) + `}`
|
||||
}
|
||||
return name + `{device=` + quote(slug) + `,component=` + quote(comp) + `}`
|
||||
}
|
||||
|
||||
// valueExpr is the metric's value at one component with scale_by applied. A metric with sentinels
|
||||
// drops its negative values, which are words rather than numbers.
|
||||
func valueExpr(m profile.Metric, slug, comp string) string {
|
||||
e := metricSel(m.Name, slug, comp)
|
||||
if len(m.Sentinels) > 0 {
|
||||
e = `(` + e + ` >= 0)`
|
||||
}
|
||||
if m.ScaleBy != "" {
|
||||
e = `(` + e + ` * on(device, component) ` + metricSel(m.ScaleBy, slug, comp) + `)`
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
// graphExprs returns the series expression and, for a metric with a capacity that is not drawn as
|
||||
// percent, the capacity expression.
|
||||
func graphExprs(res *profile.Resolved, m profile.Metric, slug, comp string, step time.Duration) (expr, maxExpr string) {
|
||||
display := m.EffectiveDisplay()
|
||||
val := valueExpr(m, slug, comp)
|
||||
counter := m.Type == profile.Counter32 || m.Type == profile.Counter64
|
||||
if counter && display == "rate" {
|
||||
val = `rate(` + metricSel(m.Name, slug, comp) + `[` + secs(max(step, minRateWindow)) + `])`
|
||||
}
|
||||
om, hasOf := res.Metric(m.Of)
|
||||
if !hasOf {
|
||||
return val, ""
|
||||
}
|
||||
den := valueExpr(om, slug, comp)
|
||||
if display == "percent" {
|
||||
return `100 * ` + val + ` / on(device, component) (` + den + ` > 0)`, ""
|
||||
}
|
||||
return val, den
|
||||
}
|
||||
|
||||
func (s *server) metricGraph(ctx context.Context, in *metricGraphIn) (*struct{ Body MetricGraphResponse }, error) {
|
||||
d, err := s.Store.GetDeviceBySlug(ctx, in.Slug)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
if d.PurgedAt != nil || s.Profiles == nil {
|
||||
return nil, huma.Error404NotFound("not found")
|
||||
}
|
||||
res := s.Profiles.Get(d.Profile)
|
||||
m, ok := metricOf(res, in.Metric)
|
||||
if !ok {
|
||||
return nil, huma.Error404NotFound("no such metric in the device's profile")
|
||||
}
|
||||
plan, err := s.metricPlan(in)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
step := plan.newest().step
|
||||
comp := in.Component
|
||||
if in.Aggregate != "" && comp == "" {
|
||||
comp = allComponents
|
||||
}
|
||||
expr, maxExpr := graphExprs(res, m, d.Slug, comp, step)
|
||||
if comp == allComponents {
|
||||
expr, maxExpr = in.Aggregate+`(`+expr+`)`, ""
|
||||
}
|
||||
resp := MetricGraphResponse{
|
||||
Unit: unitOfMetric(res, m), Display: m.EffectiveDisplay(), StepSeconds: int64(step / time.Second), Series: []Point{},
|
||||
}
|
||||
grid := gridOf(plan)
|
||||
series, err := s.fetchMetricSeries(ctx, expr, plan)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp.Series = pointsOf(grid, series)
|
||||
if maxExpr != "" {
|
||||
mx, err := s.fetchMetricSeries(ctx, maxExpr, plan)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp.MaxSeries = pointsOf(grid, mx)
|
||||
}
|
||||
return &struct{ Body MetricGraphResponse }{resp}, nil
|
||||
}
|
||||
|
||||
func metricOf(res *profile.Resolved, name string) (profile.Metric, bool) {
|
||||
if res == nil {
|
||||
return profile.Metric{}, false
|
||||
}
|
||||
m, ok := res.Metric(name)
|
||||
return m, ok && m.NeedsCategory()
|
||||
}
|
||||
|
||||
func unitOfMetric(res *profile.Resolved, m profile.Metric) string {
|
||||
return (&metricsBuilder{res: res}).unitOf(m)
|
||||
}
|
||||
|
||||
// metricPlan applies the /graph range defaults and step rules, limited to the hot store.
|
||||
func (s *server) metricPlan(in *metricGraphIn) (*graphPlan, error) {
|
||||
to := in.To
|
||||
if to.IsZero() {
|
||||
to = s.Clock.Now()
|
||||
}
|
||||
from := in.From
|
||||
if from.IsZero() {
|
||||
from = to.Add(-time.Hour)
|
||||
}
|
||||
hotStart := s.Clock.Now().Add(-s.HotRetention)
|
||||
from = from.UTC()
|
||||
if from.Before(hotStart) {
|
||||
from = hotStart
|
||||
}
|
||||
if !from.Before(to) {
|
||||
return nil, huma.Error422UnprocessableEntity("from must be before to, within the hot retention")
|
||||
}
|
||||
plan, err := planGraph(from, to.UTC(), hotStart, time.Duration(in.Step)*time.Second)
|
||||
if err != nil {
|
||||
return nil, huma.Error422UnprocessableEntity(err.Error())
|
||||
}
|
||||
return plan, nil
|
||||
}
|
||||
|
||||
// fetchMetricSeries runs the expression over the plan's hot windows and returns ts -> value.
|
||||
func (s *server) fetchMetricSeries(ctx context.Context, expr string, p *graphPlan) (map[int64]float64, error) {
|
||||
release, err := s.admit(ctx, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer release()
|
||||
qctx, cancel := context.WithTimeout(ctx, QueryTimeout)
|
||||
defer cancel()
|
||||
out := map[int64]float64{}
|
||||
for _, w := range p.tiers {
|
||||
if w.tier != tierRaw {
|
||||
continue // only the hot store is searched
|
||||
}
|
||||
if err := s.fetchOne(qctx, "hot", expr, w, func(ts int64, v float64) { out[ts] = v }); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
208
internal/api/device_metrics_test.go
Normal file
@@ -0,0 +1,208 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/v10networks/nmon/internal/api"
|
||||
"github.com/v10networks/nmon/internal/profile"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
func withShipped(t *testing.T) func(*api.Deps) {
|
||||
t.Helper()
|
||||
set, err := profile.Load("")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return func(d *api.Deps) { d.Profiles = set }
|
||||
}
|
||||
|
||||
func lbl(dev, k, comp string) map[string]string {
|
||||
m := map[string]string{"device": dev, "k": k}
|
||||
if comp != "" {
|
||||
m["component"] = comp
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// rd is a value row plus its timestamp row.
|
||||
func rd(dev, metric, comp string, v float64) []vrow {
|
||||
return []vrow{{lbl(dev, metric, comp), v}, {lbl(dev, "@"+metric, comp), 1_790_000_000}}
|
||||
}
|
||||
|
||||
func TestDeviceMetricsByCategory(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
d := e.device(t, "srv1", "yyz", "server")
|
||||
var rows []vrow
|
||||
rows = append(rows, rd(d.Slug, "hrProcessorLoad", "CPU 0", 12)...)
|
||||
rows = append(rows, rd(d.Slug, "hrStorageUsed", "/", 50)...)
|
||||
rows = append(rows, rd(d.Slug, "hrStorageSize", "/", 100)...)
|
||||
rows = append(rows, rd(d.Slug, "hrStorageAllocationUnits", "/", 1024)...)
|
||||
e.q.body = vector(rows...)
|
||||
var got api.MetricsResponse
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/metrics", &got)
|
||||
if len(got.Categories) != 1 || got.Categories[0].Name != "health" || got.Categories[0].Label != "Health" {
|
||||
t.Fatalf("categories = %+v", got.Categories)
|
||||
}
|
||||
groups := map[string]api.MetricGroup{}
|
||||
for _, g := range got.Categories[0].Groups {
|
||||
groups[g.Metric] = g
|
||||
}
|
||||
if _, ok := groups["hrStorageSize"]; ok {
|
||||
t.Error("a denominator is listed on its own")
|
||||
}
|
||||
cpu := groups["hrProcessorLoad"]
|
||||
if cpu.Unit != "percent" || cpu.Display != "gauge" || len(cpu.Components) != 1 || *cpu.Components[0].Value != 12 {
|
||||
t.Errorf("cpu = %+v", cpu)
|
||||
}
|
||||
st := groups["hrStorageUsed"]
|
||||
c := st.Components[0]
|
||||
if st.Unit != "bytes" || st.Display != "percent" || *c.Value != 51200 || *c.Max != 102400 || *c.Percent != 50 {
|
||||
t.Errorf("storage = %+v %+v", st, c)
|
||||
}
|
||||
if c.TS.Unix() != 1_790_000_000 {
|
||||
t.Errorf("ts = %v", c.TS)
|
||||
}
|
||||
qs := e.q.queries()
|
||||
if len(qs) != 1 || qs[0].store != "hot" || !strings.Contains(qs[0].expr, `device="`+d.Slug+`"`) {
|
||||
t.Errorf("queries = %+v", qs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeviceMetricsEmptyAndUnknown(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
d := e.device(t, "srv1", "yyz", "server")
|
||||
var got api.MetricsResponse
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/metrics", &got)
|
||||
if got.Categories == nil || len(got.Categories) != 0 {
|
||||
t.Errorf("categories = %#v, want empty list", got.Categories)
|
||||
}
|
||||
isProblem(t, e.do("GET", "/api/v1/devices/nope/metrics", ""), 404)
|
||||
}
|
||||
|
||||
func printerRows(dev string) []vrow {
|
||||
var rows []vrow
|
||||
rows = append(rows, rd(dev, "prtMarkerSuppliesLevel", "Black", 8)...)
|
||||
rows = append(rows, rd(dev, "prtMarkerSuppliesMaxCapacity", "Black", 100)...)
|
||||
rows = append(rows, rd(dev, "prtMarkerSuppliesLevel", "Cyan", -3)...)
|
||||
rows = append(rows, rd(dev, "prtMarkerSuppliesMaxCapacity", "Cyan", 100)...)
|
||||
rows = append(rows, rd(dev, "prtMarkerSuppliesLevel", "Magenta", -2)...)
|
||||
rows = append(rows, rd(dev, "prtAlertSeverityLevel", "1", 3)...)
|
||||
rows = append(rows, rd(dev, "prtMarkerLifeCount", "1", 4321)...)
|
||||
return rows
|
||||
}
|
||||
|
||||
func TestPrinterMetricsSentinelsAndState(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
d := e.device(t, "prn1", "yyz", "printer", func(d *store.Device) { d.Profile = "printer" })
|
||||
e.q.body = vector(printerRows(d.Slug)...)
|
||||
var got api.MetricsResponse
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/metrics", &got)
|
||||
if len(got.Categories) != 1 || got.Categories[0].Name != "printer" {
|
||||
t.Fatalf("categories = %+v", got.Categories)
|
||||
}
|
||||
by := map[string]map[string]api.MetricComponent{}
|
||||
for _, g := range got.Categories[0].Groups {
|
||||
by[g.Metric] = map[string]api.MetricComponent{}
|
||||
for _, c := range g.Components {
|
||||
by[g.Metric][c.Component] = c
|
||||
}
|
||||
}
|
||||
lv := by["prtMarkerSuppliesLevel"]
|
||||
if p := lv["Black"].Percent; p == nil || *p != 8 {
|
||||
t.Errorf("black = %+v", lv["Black"])
|
||||
}
|
||||
if c := lv["Cyan"]; c.Value != nil || c.Text == nil || *c.Text != "some remaining" || c.Percent != nil {
|
||||
t.Errorf("cyan = %+v", c)
|
||||
}
|
||||
if c := lv["Magenta"]; c.Text == nil || *c.Text != "unknown" {
|
||||
t.Errorf("magenta = %+v", c)
|
||||
}
|
||||
if s := by["prtAlertSeverityLevel"]["1"].State; s == nil || *s != "crit" {
|
||||
t.Errorf("alert state = %v", s)
|
||||
}
|
||||
if _, listed := by["prtMarkerSuppliesMaxCapacity"]; listed {
|
||||
t.Error("capacity listed on its own")
|
||||
}
|
||||
if v := by["prtMarkerLifeCount"]["1"].Value; v == nil || *v != 4321 {
|
||||
t.Errorf("pages = %v", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrinterFleetSortedByLowestSupply(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
asPrinter := func(d *store.Device) { d.Profile = "printer" }
|
||||
low := e.device(t, "lowprn", "yyz", "printer", asPrinter)
|
||||
full := e.device(t, "fullprn", "yyz", "printer", asPrinter)
|
||||
e.device(t, "a-switch", "yyz", "switch")
|
||||
var rows []vrow
|
||||
rows = append(rows, printerRows(low.Slug)...)
|
||||
rows = append(rows, rd(full.Slug, "prtMarkerSuppliesLevel", "Black", 90)...)
|
||||
rows = append(rows, rd(full.Slug, "prtMarkerSuppliesMaxCapacity", "Black", 100)...)
|
||||
e.q.body = vector(rows...)
|
||||
var got api.PrinterList
|
||||
e.getJSON(t, "/api/v1/printers", &got)
|
||||
if got.Total != 2 || len(got.Items) != 2 || got.Items[0].Device.Slug != low.Slug {
|
||||
t.Fatalf("fleet = %+v", got)
|
||||
}
|
||||
sup := got.Items[0].Supplies
|
||||
if len(sup) != 3 || sup[0].Component != "Black" || sup[0].State != "warn" || *sup[0].Percent != 8 {
|
||||
t.Errorf("supplies = %+v", sup)
|
||||
}
|
||||
if sup[1].Component != "Cyan" || *sup[1].Text != "some remaining" || sup[1].State != "ok" {
|
||||
t.Errorf("cyan = %+v", sup[1])
|
||||
}
|
||||
if sup[2].State != "unknown" {
|
||||
t.Errorf("magenta = %+v", sup[2])
|
||||
}
|
||||
if p := got.Items[0].PagesTotal; p == nil || *p != 4321 {
|
||||
t.Errorf("pages = %v", p)
|
||||
}
|
||||
if got.Items[1].Supplies[0].State != "ok" || got.Items[0].Reachability == "" {
|
||||
t.Errorf("second = %+v", got.Items[1])
|
||||
}
|
||||
n := 0
|
||||
for _, q := range e.q.queries() {
|
||||
if strings.Contains(q.expr, "prtMarkerSuppliesLevel") {
|
||||
n++
|
||||
}
|
||||
}
|
||||
if n != 1 {
|
||||
t.Errorf("%d supply queries for the fleet, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetricGraphExpressions(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
d := e.device(t, "srv1", "yyz", "server")
|
||||
now := e.clk.Now().Unix()
|
||||
e.q.body = matrix(mrow{lbl(d.Slug, "", "/"), map[int64]float64{now - 600: 25, now - 540: 30}})
|
||||
var got api.MetricGraphResponse
|
||||
path := "/api/v1/devices/" + d.Slug + "/metrics/graph?metric=hrStorageUsed&component=%2F"
|
||||
e.getJSON(t, path, &got)
|
||||
expr := e.q.q
|
||||
for _, want := range []string{`hrStorageUsed{device="` + d.Slug + `",component="/"}`, `hrStorageSize{`, `hrStorageAllocationUnits{`, `100 *`} {
|
||||
if !strings.Contains(expr, want) {
|
||||
t.Errorf("expr %q lacks %q", expr, want)
|
||||
}
|
||||
}
|
||||
if got.Display != "percent" || got.Unit != "bytes" || got.StepSeconds != 60 || len(got.Series) == 0 {
|
||||
t.Errorf("graph = %+v", got)
|
||||
}
|
||||
var withValue int
|
||||
for _, p := range got.Series {
|
||||
if p.V != nil {
|
||||
withValue++
|
||||
}
|
||||
}
|
||||
if withValue != 2 {
|
||||
t.Errorf("%d points with values, want 2", withValue)
|
||||
}
|
||||
// A counter is a rate; a gauge is raw.
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/metrics/graph?metric=hrProcessorLoad&component=CPU+0", &got)
|
||||
if strings.Contains(e.q.q, "rate(") || !strings.Contains(e.q.q, `hrProcessorLoad{`) {
|
||||
t.Errorf("gauge expr %q", e.q.q)
|
||||
}
|
||||
isProblem(t, e.do("GET", "/api/v1/devices/"+d.Slug+"/metrics/graph?metric=nope", ""), 404)
|
||||
}
|
||||
79
internal/api/device_metrics_types.go
Normal file
@@ -0,0 +1,79 @@
|
||||
package api
|
||||
|
||||
import "time"
|
||||
|
||||
// MetricsResponse is a device's latest non-interface metrics, grouped into the tabs of the device page.
|
||||
type MetricsResponse struct {
|
||||
Categories []MetricCategory `json:"categories" doc:"Only categories with data, in the fixed tab order (system first, firewall last)."`
|
||||
}
|
||||
|
||||
// MetricCategory is one tab: the metrics of one category.
|
||||
type MetricCategory struct {
|
||||
Name string `json:"name" enum:"system,health,power,environment,wireless,printer,routing,optics,firewall"`
|
||||
Label string `json:"label"`
|
||||
Groups []MetricGroup `json:"groups" doc:"One group per metric, in profile order; components are its rows (cores, filesystems, supplies)."`
|
||||
}
|
||||
|
||||
// MetricGroup is one metric with its latest value per component.
|
||||
type MetricGroup struct {
|
||||
Metric string `json:"metric" doc:"The profile metric name; pass it to /metrics/graph."`
|
||||
Label string `json:"label" doc:"The profile's human name for the metric."`
|
||||
Unit string `json:"unit" doc:"Unit of value and max. With scale_by this is the scaled unit (bytes)."`
|
||||
Display string `json:"display" enum:"gauge,percent,state,rate,text" doc:"Which renderer to use."`
|
||||
Components []MetricComponent `json:"components"`
|
||||
}
|
||||
|
||||
// MetricComponent is the latest reading of one row of a metric.
|
||||
type MetricComponent struct {
|
||||
Component string `json:"component" doc:"The row label; empty for a device-wide scalar."`
|
||||
Value *float64 `json:"value" doc:"Value after the profile's scale and scale_by; null for a sentinel or no reading."`
|
||||
Max *float64 `json:"max" doc:"The capacity this value is a fraction of (profile of), same unit; null otherwise."`
|
||||
Percent *float64 `json:"percent,omitempty" doc:"100 * value / max, computed at query time, for display percent."`
|
||||
Text *string `json:"text" doc:"A sentinel's words (for example some remaining), or the number as text for display text."`
|
||||
State *string `json:"state" enum:"ok,warn,crit,unknown" doc:"For enum (status) metrics."`
|
||||
TS time.Time `json:"ts" doc:"When the reading was taken."`
|
||||
Attrs map[string]string `json:"attrs,omitempty" doc:"Discovery attributes of the row (a printer supply's type and color)."`
|
||||
}
|
||||
|
||||
// MetricGraphResponse is a single-series graph of one metric component.
|
||||
type MetricGraphResponse struct {
|
||||
Unit string `json:"unit"`
|
||||
Display string `json:"display" enum:"gauge,percent,state,rate,text"`
|
||||
StepSeconds int64 `json:"step_seconds"`
|
||||
Series []Point `json:"series" doc:"[unix_s, value|null] pairs. Counters are a per-second rate; percent displays are percent."`
|
||||
MaxSeries []Point `json:"max_series,omitempty" doc:"The capacity over time, when the metric has one and the display is not percent."`
|
||||
}
|
||||
|
||||
// PrinterSupply is one supply or paper tray in the fleet view.
|
||||
type PrinterSupply struct {
|
||||
Component string `json:"component"`
|
||||
Type string `json:"type,omitempty" doc:"Supply type (toner, drum, fuser, ...), from discovery."`
|
||||
Color string `json:"color,omitempty" doc:"Colorant as the printer names it (black, cyan, ...), from discovery."`
|
||||
Percent *float64 `json:"percent" doc:"Level as a percent of capacity; null for a sentinel or no reading."`
|
||||
Text *string `json:"text" doc:"A sentinel's words, for example some remaining."`
|
||||
State string `json:"state" enum:"ok,warn,crit,unknown"`
|
||||
}
|
||||
|
||||
// PrinterRow is one printer of the fleet view.
|
||||
type PrinterRow struct {
|
||||
Device DeviceRefSite `json:"device"`
|
||||
Reachability string `json:"reachability" enum:"up,down,warning,unknown,maintenance"`
|
||||
PagesTotal *float64 `json:"pages_total" doc:"Lifetime page count; null when unread."`
|
||||
Supplies []PrinterSupply `json:"supplies"`
|
||||
Trays []PrinterSupply `json:"trays"`
|
||||
}
|
||||
|
||||
// DeviceRefSite is a device reference with its site.
|
||||
type DeviceRefSite struct {
|
||||
Slug string `json:"slug"`
|
||||
Name string `json:"name"`
|
||||
Site string `json:"site"`
|
||||
}
|
||||
|
||||
// PrinterList is a page of printers, the one with the lowest supply first.
|
||||
type PrinterList struct {
|
||||
Items []PrinterRow `json:"items"`
|
||||
Page int64 `json:"page"`
|
||||
PerPage int64 `json:"per_page"`
|
||||
Total int64 `json:"total"`
|
||||
}
|
||||
@@ -19,6 +19,7 @@ type PutDeviceBody struct {
|
||||
Enabled *bool `json:"enabled,omitempty"`
|
||||
Parents []string `json:"parents,omitempty" doc:"Parent device slugs, replacing the current parents."`
|
||||
SNMP *SNMPBody `json:"snmp,omitempty" doc:"Replaces the SNMP credential; secrets are write-only."`
|
||||
Profile *string `json:"profile,omitempty" doc:"A profile name pins it; an empty string returns the device to automatic matching. A change triggers one discovery."` //nolint:lll // struct tag text is the OpenAPI doc; cannot wrap
|
||||
}
|
||||
|
||||
func isNotFound(err error) bool { return errors.Is(err, store.ErrNotFound) }
|
||||
@@ -45,6 +46,9 @@ func (s *server) putDevice(ctx context.Context, in *struct {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.inventoryChanged(ctx)
|
||||
if plan.rediscover {
|
||||
s.rediscoverAfterEdit(ctx, dev.Slug)
|
||||
}
|
||||
return s.view(ctx, dev.Slug)
|
||||
}
|
||||
|
||||
@@ -56,6 +60,7 @@ type editPlan struct {
|
||||
parents []*store.Device
|
||||
cred *store.SNMPCredential
|
||||
change map[string]any // what the audit row records; never a secret
|
||||
rediscover bool // the profile or its pin changed: run one discovery after the write
|
||||
}
|
||||
|
||||
// planEdit validates body against the device without writing anything.
|
||||
@@ -85,6 +90,11 @@ func (s *server) planEdit(ctx context.Context, dev *store.Device, b PutDeviceBod
|
||||
p.setParents, p.parents = true, parents
|
||||
p.change["parents"] = b.Parents
|
||||
}
|
||||
if b.Profile != nil {
|
||||
if err := s.planProfile(p, dev, *b.Profile); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if b.SNMP != nil {
|
||||
if dev.Kind != store.KindNetwork {
|
||||
return nil, huma.Error422UnprocessableEntity("snmp applies to network devices only")
|
||||
|
||||
@@ -49,34 +49,37 @@ type InterfaceView struct {
|
||||
OutBPS *float64 `json:"out_bps" nullable:"true"`
|
||||
InErrorsPS *float64 `json:"in_errors_ps" nullable:"true" doc:"Inbound errors per second over 5 minutes."`
|
||||
OutErrorsPS *float64 `json:"out_errors_ps" nullable:"true"`
|
||||
AlertDownState
|
||||
}
|
||||
|
||||
// DeviceView is a device as the API shows it. It carries no numeric ID and no credential.
|
||||
type DeviceView struct {
|
||||
Slug string `json:"slug" doc:"Immutable public key."`
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"`
|
||||
Site string `json:"site"`
|
||||
Role string `json:"role"`
|
||||
Profile string `json:"profile"`
|
||||
Enabled bool `json:"enabled"`
|
||||
MgmtIP string `json:"mgmt_ip"`
|
||||
Addresses []string `json:"addresses"`
|
||||
SNMPVersion string `json:"snmp_version,omitempty"`
|
||||
SysName string `json:"sys_name"`
|
||||
SysObjectID string `json:"sys_object_id"`
|
||||
SysDescr string `json:"sys_descr"`
|
||||
HWSerial string `json:"hw_serial"`
|
||||
UptimeSeconds *float64 `json:"uptime_seconds" nullable:"true" doc:"sysUpTime from the hot store; null when unknown."`
|
||||
Reachability string `json:"reachability" enum:"up,down,warning,unknown,maintenance"`
|
||||
PollState string `json:"poll_state" enum:"ok,degraded,backoff"`
|
||||
Warnings []string `json:"warnings"`
|
||||
LastSeen *time.Time `json:"last_seen" nullable:"true"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
RetiredAt *time.Time `json:"retired_at,omitempty" nullable:"true"`
|
||||
Parents []DeviceRef `json:"parents"`
|
||||
Groups []GroupView `json:"groups"`
|
||||
Interfaces []InterfaceView `json:"interfaces"`
|
||||
Slug string `json:"slug" doc:"Immutable public key."`
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"`
|
||||
Site string `json:"site"`
|
||||
Role string `json:"role"`
|
||||
Profile string `json:"profile"`
|
||||
ProfilePinned bool `json:"profile_pinned" doc:"An operator chose the profile; discovery keeps it."`
|
||||
ProfileMatch *ProfileMatchView `json:"profile_match,omitempty"`
|
||||
Enabled bool `json:"enabled"`
|
||||
MgmtIP string `json:"mgmt_ip"`
|
||||
Addresses []string `json:"addresses"`
|
||||
SNMPVersion string `json:"snmp_version,omitempty"`
|
||||
SysName string `json:"sys_name"`
|
||||
SysObjectID string `json:"sys_object_id"`
|
||||
SysDescr string `json:"sys_descr"`
|
||||
HWSerial string `json:"hw_serial"`
|
||||
UptimeSeconds *float64 `json:"uptime_seconds" nullable:"true" doc:"sysUpTime from the hot store; null when unknown."`
|
||||
Reachability string `json:"reachability" enum:"up,down,warning,unknown,maintenance"`
|
||||
PollState string `json:"poll_state" enum:"ok,degraded,backoff"`
|
||||
Warnings []string `json:"warnings"`
|
||||
LastSeen *time.Time `json:"last_seen" nullable:"true"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
RetiredAt *time.Time `json:"retired_at,omitempty" nullable:"true"`
|
||||
Parents []DeviceRef `json:"parents"`
|
||||
Groups []GroupView `json:"groups"`
|
||||
Interfaces []InterfaceView `json:"interfaces"`
|
||||
// M5 additions.
|
||||
HostedOn *DeviceRef `json:"hosted_on,omitempty" doc:"The host node of a VM or container."`
|
||||
Guests []DeviceRef `json:"guests" doc:"VMs and containers hosted on this device."`
|
||||
@@ -118,7 +121,8 @@ func (s *server) view(ctx context.Context, slug string) (*deviceOut, error) {
|
||||
return nil, huma.Error404NotFound("not found")
|
||||
}
|
||||
v := DeviceView{
|
||||
Slug: d.Slug, Name: d.Name, Kind: d.Kind, Site: d.Site, Role: d.Role, Profile: d.Profile, Enabled: d.Enabled,
|
||||
Slug: d.Slug, Name: d.Name, Kind: d.Kind, Site: d.Site, Role: d.Role, Profile: d.Profile,
|
||||
ProfilePinned: d.ProfilePinned, ProfileMatch: s.profileMatch(d), Enabled: d.Enabled,
|
||||
MgmtIP: d.MgmtIP, SysName: d.SysName, SysObjectID: d.SysObjectID, SysDescr: d.SysDescr, HWSerial: d.HWSerial,
|
||||
PollState: pollStateOf(d), Warnings: s.warningsOf(d), LastSeen: utcp(d.LastSeen), CreatedAt: d.CreatedAt.UTC(),
|
||||
RetiredAt: utcp(d.RetiredAt), Addresses: []string{}, Parents: []DeviceRef{}, Groups: []GroupView{},
|
||||
@@ -192,7 +196,7 @@ func (s *server) fillInterfaces(ctx context.Context, d *store.Device, v *DeviceV
|
||||
for i := range ifs {
|
||||
v.Interfaces = append(v.Interfaces, interfaceView(&ifs[i], rates[ifs[i].ID]))
|
||||
}
|
||||
return nil
|
||||
return s.withAlertDown(ctx, ifs, v.Interfaces)
|
||||
}
|
||||
|
||||
func interfaceView(i *store.Interface, r *ifRates) InterfaceView {
|
||||
|
||||
@@ -15,7 +15,8 @@ import (
|
||||
// GroupSummary is a group in a list.
|
||||
type GroupSummary struct {
|
||||
GroupView
|
||||
MemberCount int `json:"member_count" doc:"Interfaces in the group now."`
|
||||
MemberCount int `json:"member_count" doc:"Interfaces in the group now."`
|
||||
AlertDown bool `json:"alert_down" doc:"Member ports alert on interface_down while they are members."`
|
||||
}
|
||||
|
||||
// GroupList is every group, by name.
|
||||
@@ -57,6 +58,8 @@ type GroupDetail struct {
|
||||
type GroupBody struct {
|
||||
Name string `json:"name" minLength:"1" maxLength:"100"`
|
||||
Kind string `json:"kind" enum:"site,customer,role,custom" default:"custom"`
|
||||
// AlertDown is a pointer so a PUT that omits it leaves the flag alone.
|
||||
AlertDown *bool `json:"alert_down,omitempty" doc:"Opt member ports into interface_down alerts. Omitted: off on create, kept on update."`
|
||||
}
|
||||
|
||||
// GroupMemberBody is the body of POST /groups/{id}/members.
|
||||
@@ -110,7 +113,8 @@ func summaryOf(now time.Time, g store.Group, members []store.GroupMember) GroupS
|
||||
n++
|
||||
}
|
||||
}
|
||||
return GroupSummary{GroupView: GroupView{ID: strconv.FormatInt(g.ID, 10), Name: g.Name, Kind: g.Kind}, MemberCount: n}
|
||||
gv := GroupView{ID: strconv.FormatInt(g.ID, 10), Name: g.Name, Kind: g.Kind}
|
||||
return GroupSummary{GroupView: gv, MemberCount: n, AlertDown: g.AlertDown}
|
||||
}
|
||||
|
||||
func (s *server) listGroups(ctx context.Context, _ *struct{}) (*struct{ Body GroupList }, error) {
|
||||
@@ -138,6 +142,7 @@ func groupBody(b GroupBody) (store.Group, error) {
|
||||
if g.Name == "" {
|
||||
return g, huma.Error422UnprocessableEntity("name must not be empty")
|
||||
}
|
||||
g.AlertDown = b.AlertDown != nil && *b.AlertDown
|
||||
return g, nil
|
||||
}
|
||||
|
||||
@@ -155,6 +160,7 @@ func (s *server) createGroup(ctx context.Context, in *struct{ Body GroupBody })
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.alertDownChanged()
|
||||
return &struct{ Body GroupSummary }{summaryOf(s.Clock.Now(), g, nil)}, nil
|
||||
}
|
||||
|
||||
@@ -173,7 +179,8 @@ func (s *server) getGroup(ctx context.Context, in *struct {
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
d := GroupDetail{GroupSummary: summaryOf(s.Clock.Now(), store.Group{ID: ref.ID, Name: ref.Name, Kind: ref.Kind}, members),
|
||||
g := store.Group{ID: ref.ID, Name: ref.Name, Kind: ref.Kind, AlertDown: ref.AlertDown}
|
||||
d := GroupDetail{GroupSummary: summaryOf(s.Clock.Now(), g, members),
|
||||
Members: make([]GroupMemberView, 0, len(members)), Flags: []GroupFlag{}}
|
||||
for _, m := range members {
|
||||
d.Members = append(d.Members, s.memberView(ctx, m))
|
||||
@@ -226,6 +233,13 @@ func (s *server) updateGroup(ctx context.Context, in *struct {
|
||||
return nil, err
|
||||
}
|
||||
g.ID = id
|
||||
if in.Body.AlertDown == nil {
|
||||
cur, err := s.Store.GetGroup(ctx, id)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
g.AlertDown = cur.AlertDown
|
||||
}
|
||||
err = s.Store.WithTx(ctx, func(tx store.Store) error {
|
||||
if err := tx.UpdateGroup(ctx, g); err != nil {
|
||||
return err
|
||||
@@ -235,6 +249,7 @@ func (s *server) updateGroup(ctx context.Context, in *struct {
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.alertDownChanged()
|
||||
members, err := s.Store.GroupMembers(ctx, id, time.Time{}, time.Time{})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
@@ -258,6 +273,7 @@ func (s *server) deleteGroup(ctx context.Context, in *struct {
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.alertDownChanged()
|
||||
return &struct{}{}, nil
|
||||
}
|
||||
|
||||
@@ -287,6 +303,7 @@ func (s *server) addGroupMember(ctx context.Context, in *struct {
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.alertDownChanged()
|
||||
return &struct{ Body GroupMemberView }{s.memberView(ctx, m)}, nil
|
||||
}
|
||||
|
||||
@@ -312,5 +329,6 @@ func (s *server) endGroupMember(ctx context.Context, in *struct {
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.alertDownChanged()
|
||||
return &struct{}{}, nil
|
||||
}
|
||||
|
||||
@@ -39,10 +39,33 @@ type PollerTotals struct {
|
||||
DevicesDegraded int64 `json:"devices_degraded"`
|
||||
}
|
||||
|
||||
// HealthNote is a condition worth showing on the health page that is not an alert: optional data is missing, or
|
||||
// a background job is skipping devices.
|
||||
type HealthNote struct {
|
||||
Code string `json:"code" example:"oui_unavailable"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// L2Health is the MAC/ARP/FDB table job's last cycle.
|
||||
type L2Health struct {
|
||||
Running bool `json:"running"`
|
||||
LastCycle *time.Time `json:"last_cycle" nullable:"true" doc:"When the last cycle ended."`
|
||||
IntervalS int64 `json:"interval_s"`
|
||||
OK int64 `json:"devices_ok"`
|
||||
TimedOut int64 `json:"devices_timed_out" doc:"Devices skipped in the last cycle because they did not answer in time."`
|
||||
Failed int64 `json:"devices_failed"`
|
||||
// Skipped names the devices that timed out or failed in the last cycle.
|
||||
Skipped []string `json:"skipped"`
|
||||
}
|
||||
|
||||
// PollerHealth is the poller health page.
|
||||
type PollerHealth struct {
|
||||
Devices []PollerDeviceHealth `json:"devices"`
|
||||
Totals PollerTotals `json:"totals"`
|
||||
// L2 is the MAC/ARP/FDB table job; absent when it is not running.
|
||||
L2 *L2Health `json:"l2,omitempty"`
|
||||
// Notes are conditions that are not alerts, such as oui_unavailable (no vendor registry loaded).
|
||||
Notes []HealthNote `json:"notes"`
|
||||
}
|
||||
|
||||
func (s *server) registerHealth(api huma.API) {
|
||||
@@ -58,7 +81,13 @@ func (s *server) pollerHealth(ctx context.Context, _ *struct{}) (*struct{ Body P
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
out := PollerHealth{Devices: []PollerDeviceHealth{}}
|
||||
out := PollerHealth{Devices: []PollerDeviceHealth{}, Notes: []HealthNote{}}
|
||||
if s.L2Health != nil {
|
||||
out.L2 = s.L2Health()
|
||||
}
|
||||
if s.HealthNotes != nil {
|
||||
out.Notes = append(out.Notes, s.HealthNotes()...)
|
||||
}
|
||||
if s.PollerStatus != nil {
|
||||
for _, d := range s.PollerStatus() {
|
||||
h := PollerDeviceHealth{
|
||||
|
||||
@@ -22,6 +22,7 @@ type InterfaceHit struct {
|
||||
SpeedBPS int64 `json:"speed_bps" doc:"Link speed in bits per second (ifHighSpeed when set)."`
|
||||
InBPS *float64 `json:"in_bps" nullable:"true"`
|
||||
OutBPS *float64 `json:"out_bps" nullable:"true"`
|
||||
AlertDownState
|
||||
}
|
||||
|
||||
// InterfaceList is a page of InterfaceHit.
|
||||
@@ -52,11 +53,11 @@ func (s *server) listInterfaces(ctx context.Context, in *struct {
|
||||
out := InterfaceList{Items: make([]InterfaceHit, 0, len(rows)), Page: in.pageNo(), PerPage: in.limit(), Total: total}
|
||||
var rates map[string]*ifRates
|
||||
if len(rows) > 0 {
|
||||
ids := make([]string, len(rows))
|
||||
for i, r := range rows {
|
||||
ids[i] = r.ID
|
||||
}
|
||||
rates, _ = s.liveRates(ctx, `iface=~`+quote(regexAlt(ids)), false)
|
||||
rates, _ = s.liveRates(ctx, `iface=~`+quote(regexAlt(hitIDs(rows))), false)
|
||||
}
|
||||
via, err := s.Store.InterfaceAlertDownVia(ctx, hitIDs(rows))
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
for i := range rows {
|
||||
r := &rows[i]
|
||||
@@ -64,7 +65,7 @@ func (s *server) listInterfaces(ctx context.Context, in *struct {
|
||||
out.Items = append(out.Items, InterfaceHit{
|
||||
ID: r.ID, Device: DeviceRef{Slug: r.DeviceSlug, Name: r.DeviceName}, IfName: r.IfName, IfAlias: r.IfAlias, Description: r.Description,
|
||||
IfDescr: r.IfDescr, OperStatus: r.OperStatus, AdminStatus: r.AdminStatus, SpeedBPS: speedBPS(&r.Interface),
|
||||
InBPS: inB, OutBPS: outB,
|
||||
InBPS: inB, OutBPS: outB, AlertDownState: alertDownState(r.AlertDown, via[r.ID]),
|
||||
})
|
||||
}
|
||||
return &struct{ Body InterfaceList }{out}, nil
|
||||
@@ -78,3 +79,11 @@ func speedBPS(i *store.Interface) int64 {
|
||||
}
|
||||
return i.IfSpeed
|
||||
}
|
||||
|
||||
func hitIDs(rows []store.InterfaceHit) []string {
|
||||
ids := make([]string, len(rows))
|
||||
for i := range rows {
|
||||
ids[i] = rows[i].ID
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
83
internal/api/l2_rank.go
Normal file
@@ -0,0 +1,83 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"cmp"
|
||||
"slices"
|
||||
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// ifTypeLAG is IANAifType ieee8023adLag: a link aggregate carries the MACs of its members, so it is an uplink.
|
||||
const ifTypeLAG = 161
|
||||
|
||||
// Reasons a port is not an edge port.
|
||||
const (
|
||||
uplinkMACs = "macs"
|
||||
uplinkLLDP = "lldp"
|
||||
uplinkLAG = "lag"
|
||||
)
|
||||
|
||||
// portFacts is what decides whether a port is where a host is plugged in: a MAC is seen on every switch it passes
|
||||
// through, so only a port with few MACs and no switch behind it is the host's own.
|
||||
type portFacts struct {
|
||||
counts map[store.PortKey]int64
|
||||
lldp map[int64]map[int64]bool // device -> ifIndexes with an LLDP neighbour
|
||||
edgeMax int64 // l2.edge_max_macs
|
||||
}
|
||||
|
||||
// key is the MAC count key of a row: the port's MACs seen at or after the row's last_seen, which for a current
|
||||
// row is every MAC of the device's last walk on that port.
|
||||
func portKeyOf(r *store.FDBRow) store.PortKey {
|
||||
return store.PortKey{DeviceID: r.DeviceID, InterfaceID: r.InterfaceID, Since: r.LastSeen}
|
||||
}
|
||||
|
||||
// uplink says why the port of r is not an edge port, or "" when it is one.
|
||||
func (f portFacts) uplink(r *store.FDBRow, count int64) string {
|
||||
switch {
|
||||
case count > f.edgeMax:
|
||||
return uplinkMACs
|
||||
case r.IfIndex != 0 && f.lldp[r.DeviceID][r.IfIndex]:
|
||||
return uplinkLLDP
|
||||
case r.IfType == ifTypeLAG:
|
||||
return uplinkLAG
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// location builds the API view of a row.
|
||||
func (f portFacts) location(r *store.FDBRow) L2Location {
|
||||
n := f.counts[portKeyOf(r)]
|
||||
why := f.uplink(r, n)
|
||||
loc := L2Location{
|
||||
Device: DeviceRef{Slug: r.DeviceSlug, Name: r.DeviceName}, VLAN: r.VLAN, VLANName: r.VLANName,
|
||||
Edge: why == "", UplinkReason: why, PortMACCount: n, Current: r.Current, FirstSeen: r.FirstSeen, LastSeen: r.LastSeen,
|
||||
}
|
||||
if r.InterfaceID != "" {
|
||||
loc.Iface = &L2IfaceRef{ID: r.InterfaceID, IfName: r.IfName}
|
||||
}
|
||||
return loc
|
||||
}
|
||||
|
||||
// sortLocations puts the host's own port first: edge ports before uplinks, then rows the last walk still sees
|
||||
// before history, then the port with the fewest MACs, then the most recently seen.
|
||||
func sortLocations(locs []L2Location) {
|
||||
slices.SortStableFunc(locs, func(a, b L2Location) int {
|
||||
if c := cmp.Compare(boolRank(b.Edge), boolRank(a.Edge)); c != 0 {
|
||||
return c
|
||||
}
|
||||
if c := cmp.Compare(boolRank(b.Current), boolRank(a.Current)); c != 0 {
|
||||
return c
|
||||
}
|
||||
if c := cmp.Compare(a.PortMACCount, b.PortMACCount); c != 0 {
|
||||
return c
|
||||
}
|
||||
return b.LastSeen.Compare(a.LastSeen)
|
||||
})
|
||||
}
|
||||
|
||||
func boolRank(b bool) int {
|
||||
if b {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
200
internal/api/l2_search.go
Normal file
@@ -0,0 +1,200 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strconv"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/config"
|
||||
"github.com/v10networks/nmon/internal/hostaddr"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
func (s *server) registerL2(api huma.API) {
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "search-l2", Method: http.MethodGet, Path: "/api/v1/l2/search", Tags: []string{"l2"},
|
||||
Summary: "Find where a MAC or IP address is plugged in",
|
||||
Description: "q is a MAC address in any common notation (aabb.ccdd.eeff, aa-bb-..., aabbccddeeff), a MAC prefix of at least " +
|
||||
"six hex digits, an IPv4 or IPv6 address, a CIDR (matched against ARP and neighbour tables), or vendor:<name> " +
|
||||
"(the current MACs of a vendor in the IEEE registry). Each hit is a MAC with the addresses it answers to and every " +
|
||||
"place it was seen, history included: edge ports first (a port that carries few MACs, has no LLDP neighbour and " +
|
||||
"is not a LAG is where the host is plugged in), then uplinks.",
|
||||
Errors: []int{http.StatusUnprocessableEntity},
|
||||
}, s.l2Search)
|
||||
s.registerL2Tables(api)
|
||||
}
|
||||
|
||||
// maxVendorHits caps how many MACs one vendor search considers before paging.
|
||||
const maxVendorHits = 20_000
|
||||
|
||||
func (s *server) l2Search(ctx context.Context, in *struct {
|
||||
Q string `query:"q" minLength:"1" maxLength:"200" doc:"MAC, MAC prefix, IP, CIDR or vendor:name."`
|
||||
VLAN string `query:"vlan" pattern:"^[0-9]{1,4}$" doc:"Only MACs seen in this VLAN."`
|
||||
PageParams
|
||||
}) (*struct{ Body L2SearchResult }, error) {
|
||||
q, err := hostaddr.Classify(in.Q)
|
||||
if err != nil {
|
||||
return nil, huma.Error422UnprocessableEntity(err.Error())
|
||||
}
|
||||
res := L2SearchResult{Items: []L2Hit{}, Page: in.pageNo(), PerPage: in.limit(), Kind: q.Kind}
|
||||
res.VendorsAvailable = s.Vendors != nil && s.Vendors.Available()
|
||||
var hits []store.MACHit
|
||||
vlan := optInt(in.VLAN)
|
||||
if q.Kind == hostaddr.QueryVendor {
|
||||
hits, res.Total, err = s.vendorHits(ctx, q.Vendor, vlan, in.PageParams)
|
||||
} else {
|
||||
hits, res.Total, err = s.Store.FindMACs(ctx, finderFor(q, vlan), store.Page{Offset: in.offset(), Limit: in.limit()})
|
||||
}
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
if res.Items, err = s.hitsFor(ctx, hits); err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
return &struct{ Body L2SearchResult }{res}, nil
|
||||
}
|
||||
|
||||
// optInt reads an optional non-negative integer query value: nil when empty.
|
||||
func optInt(s string) *int64 {
|
||||
if v, err := strconv.ParseInt(s, 10, 64); err == nil {
|
||||
return &v
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func finderFor(q hostaddr.Query, vlan *int64) store.MACFinder {
|
||||
f := store.MACFinder{VLAN: vlan}
|
||||
switch q.Kind {
|
||||
case hostaddr.QueryMAC:
|
||||
f.MAC = q.MAC
|
||||
case hostaddr.QueryPrefix:
|
||||
f.MACPrefix = q.MAC
|
||||
case hostaddr.QueryIP:
|
||||
f.IP = q.IP
|
||||
case hostaddr.QueryCIDR:
|
||||
f.Prefix = q.Prefix
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// vendorHits finds the current MACs whose registered vendor contains name, and pages them. The registry is in
|
||||
// memory, so the filter runs here on the distinct current MACs the store streams.
|
||||
func (s *server) vendorHits(ctx context.Context, name string, vlan *int64, p PageParams) ([]store.MACHit, int64, error) {
|
||||
if s.Vendors == nil {
|
||||
return nil, 0, nil
|
||||
}
|
||||
match := s.Vendors.Match(name)
|
||||
var macs []string
|
||||
err := s.Store.WalkCurrentMACs(ctx, vlan, func(mac string) bool {
|
||||
if match(mac) {
|
||||
macs = append(macs, mac)
|
||||
}
|
||||
return len(macs) < maxVendorHits
|
||||
})
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
total := int64(len(macs))
|
||||
lo := min(p.offset(), total)
|
||||
hi := min(lo+p.limit(), total)
|
||||
out := make([]store.MACHit, 0, hi-lo)
|
||||
for _, m := range macs[lo:hi] {
|
||||
out = append(out, store.MACHit{Mac: m})
|
||||
}
|
||||
return out, total, nil
|
||||
}
|
||||
|
||||
// vendorOf is the vendor of a canonical MAC, "" without a registry.
|
||||
func (s *server) vendorOf(mac string) string {
|
||||
if s.Vendors == nil {
|
||||
return ""
|
||||
}
|
||||
return s.Vendors.Vendor(mac)
|
||||
}
|
||||
|
||||
// edgeMax is l2.edge_max_macs.
|
||||
func (s *server) edgeMax() int64 {
|
||||
if s.L2EdgeMax != nil {
|
||||
return int64(s.L2EdgeMax())
|
||||
}
|
||||
return config.DefaultL2EdgeMaxMACs
|
||||
}
|
||||
|
||||
// hitsFor loads the FDB and ARP rows of the MACs and builds the grouped hits in the order given.
|
||||
func (s *server) hitsFor(ctx context.Context, hits []store.MACHit) ([]L2Hit, error) {
|
||||
macs := make([]string, len(hits))
|
||||
for i, h := range hits {
|
||||
macs[i] = h.Mac
|
||||
}
|
||||
fdb, err := s.Store.FDBForMACs(ctx, macs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
arp, err := s.Store.ARPForMACs(ctx, macs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
facts, err := s.facts(ctx, fdb)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
locs := map[string][]L2Location{}
|
||||
for i := range fdb {
|
||||
locs[fdb[i].Mac] = append(locs[fdb[i].Mac], facts.location(&fdb[i]))
|
||||
}
|
||||
ips := map[string][]L2IP{}
|
||||
for _, a := range arp {
|
||||
ips[a.Mac] = append(ips[a.Mac], L2IP{IP: a.IP, Device: DeviceRef{Slug: a.DeviceSlug, Name: a.DeviceName}, LastSeen: a.LastSeen})
|
||||
}
|
||||
out := make([]L2Hit, 0, len(hits))
|
||||
for _, h := range hits {
|
||||
l := locs[h.Mac]
|
||||
sortLocations(l)
|
||||
out = append(out, L2Hit{
|
||||
MAC: h.Mac, Vendor: s.vendorOf(h.Mac), MACKind: hostaddr.Kind(h.Mac),
|
||||
IPs: nonNilIPs(ips[h.Mac]), Locations: nonNilLocs(l),
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func nonNilIPs(v []L2IP) []L2IP {
|
||||
if v == nil {
|
||||
return []L2IP{}
|
||||
}
|
||||
slices.SortStableFunc(v, func(a, b L2IP) int { return b.LastSeen.Compare(a.LastSeen) })
|
||||
return v
|
||||
}
|
||||
|
||||
func nonNilLocs(v []L2Location) []L2Location {
|
||||
if v == nil {
|
||||
return []L2Location{}
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// facts gathers the per-port MAC counts and LLDP links that decide edge versus uplink for the given rows.
|
||||
func (s *server) facts(ctx context.Context, rows []store.FDBRow) (portFacts, error) {
|
||||
keys := make([]store.PortKey, 0, len(rows))
|
||||
devs := map[int64]bool{}
|
||||
for i := range rows {
|
||||
keys = append(keys, portKeyOf(&rows[i]))
|
||||
devs[rows[i].DeviceID] = true
|
||||
}
|
||||
counts, err := s.Store.PortMACCounts(ctx, keys)
|
||||
if err != nil {
|
||||
return portFacts{}, err
|
||||
}
|
||||
ids := make([]int64, 0, len(devs))
|
||||
for id := range devs {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
lldp, err := s.Store.LLDPIfIndexes(ctx, ids)
|
||||
if err != nil {
|
||||
return portFacts{}, err
|
||||
}
|
||||
return portFacts{counts: counts, lldp: lldp, edgeMax: s.edgeMax()}, nil
|
||||
}
|
||||
283
internal/api/l2_tables.go
Normal file
@@ -0,0 +1,283 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/hostaddr"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
func (s *server) registerL2Tables(api huma.API) {
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "get-device-l2", Method: http.MethodGet, Path: "/api/v1/devices/{slug}/l2", Tags: []string{"l2"},
|
||||
Summary: "What the L2 job knows about a device",
|
||||
Description: "Row counts, the time and source of the last walk and the VLAN list. The device page shows its L2 tab " +
|
||||
"when has_data is true.",
|
||||
Errors: []int{http.StatusNotFound},
|
||||
}, s.deviceL2)
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "list-device-fdb", Method: http.MethodGet, Path: "/api/v1/devices/{slug}/fdb", Tags: []string{"l2"},
|
||||
Summary: "A device's forwarding table (MACs by port and VLAN)",
|
||||
Description: "current=true (the default) lists the rows the last walk saw; current=false adds history, each row with its first and " +
|
||||
"last seen. Ordered by VLAN, then MAC.",
|
||||
Errors: []int{http.StatusNotFound, http.StatusUnprocessableEntity},
|
||||
}, s.deviceFDB)
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "list-device-arp", Method: http.MethodGet, Path: "/api/v1/devices/{slug}/arp", Tags: []string{"l2"},
|
||||
Summary: "A device's ARP and IPv6 neighbour table",
|
||||
Description: "current=true (the default) lists the rows the last walk saw; current=false adds history. ip is an address or a CIDR.",
|
||||
Errors: []int{http.StatusNotFound, http.StatusUnprocessableEntity},
|
||||
}, s.deviceARP)
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "list-device-l2-ports", Method: http.MethodGet, Path: "/api/v1/devices/{slug}/l2/ports", Tags: []string{"l2"},
|
||||
Summary: "How many MACs each port of a device carries now",
|
||||
Description: "For the Ports tab: ports that carry MACs, with whether they look like uplinks " +
|
||||
"(too many MACs, an LLDP neighbour, or a LAG).",
|
||||
Errors: []int{http.StatusNotFound},
|
||||
}, s.deviceL2Ports)
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "list-interface-macs", Method: http.MethodGet, Path: "/api/v1/interfaces/{id}/macs", Tags: []string{"l2"},
|
||||
Summary: "The MACs on one port, with their IP addresses",
|
||||
Description: "current=true (the default) lists the MACs the last walk saw on the port; current=false adds history.",
|
||||
Errors: []int{http.StatusNotFound},
|
||||
}, s.interfaceMACs)
|
||||
}
|
||||
|
||||
// l2Device loads a device for the L2 routes.
|
||||
func (s *server) l2Device(ctx context.Context, slug string) (*store.Device, error) {
|
||||
d, err := s.Store.GetDeviceBySlug(ctx, slug)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
if d.PurgedAt != nil {
|
||||
return nil, huma.Error404NotFound("not found")
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
func (s *server) deviceL2(ctx context.Context, in *struct {
|
||||
Slug string `path:"slug" doc:"Device slug."`
|
||||
}) (*struct{ Body L2Summary }, error) {
|
||||
d, err := s.l2Device(ctx, in.Slug)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c, err := s.Store.L2Counts(ctx, d.ID)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
vl, err := s.Store.ListVLANs(ctx, d.ID)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
out := L2Summary{
|
||||
HasData: c.FDBTotal+c.ARPTotal > 0, FDBCurrent: c.FDBCurrent, FDBTotal: c.FDBTotal, ARPCurrent: c.ARPCurrent, ARPTotal: c.ARPTotal,
|
||||
VLANs: make([]L2VLAN, 0, len(vl)), EdgeMaxMACs: s.edgeMax(),
|
||||
}
|
||||
if c.Walk != nil {
|
||||
t := c.Walk.WalkedAt.UTC()
|
||||
out.LastWalk, out.Source = &t, c.Walk.Source
|
||||
}
|
||||
for _, v := range vl {
|
||||
out.VLANs = append(out.VLANs, L2VLAN{VLAN: v.VLAN, Name: v.Name})
|
||||
}
|
||||
return &struct{ Body L2Summary }{out}, nil
|
||||
}
|
||||
|
||||
// ifaceRef is the interface reference of a row, nil without an interface.
|
||||
func ifaceRef(id, name string) *L2IfaceRef {
|
||||
if id == "" {
|
||||
return nil
|
||||
}
|
||||
return &L2IfaceRef{ID: id, IfName: name}
|
||||
}
|
||||
|
||||
// ipsByMAC maps each MAC to the addresses ARP and neighbour tables give it, most recent first.
|
||||
func (s *server) ipsByMAC(ctx context.Context, macs []string) (map[string][]string, error) {
|
||||
arp, err := s.Store.ARPForMACs(ctx, macs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := map[string][]string{}
|
||||
for _, a := range arp {
|
||||
if !contains(out[a.Mac], a.IP) {
|
||||
out[a.Mac] = append(out[a.Mac], a.IP)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func contains(s []string, v string) bool {
|
||||
for _, x := range s {
|
||||
if x == v {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// fdbRows builds the API rows of a page of FDB rows with their IPs.
|
||||
func (s *server) fdbRows(ctx context.Context, rows []store.FDBRow) ([]L2FDBRow, error) {
|
||||
macs := make([]string, len(rows))
|
||||
for i := range rows {
|
||||
macs[i] = rows[i].Mac
|
||||
}
|
||||
ips, err := s.ipsByMAC(ctx, macs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]L2FDBRow, 0, len(rows))
|
||||
for i := range rows {
|
||||
r := &rows[i]
|
||||
out = append(out, L2FDBRow{
|
||||
MAC: r.Mac, Vendor: s.vendorOf(r.Mac), MACKind: hostaddr.Kind(r.Mac), VLAN: r.VLAN, VLANName: r.VLANName,
|
||||
Iface: ifaceRef(r.InterfaceID, r.IfName), IPs: nonNilStrings(ips[r.Mac]), Current: r.Current,
|
||||
FirstSeen: r.FirstSeen, LastSeen: r.LastSeen,
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func nonNilStrings(v []string) []string {
|
||||
if v == nil {
|
||||
return []string{}
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func (s *server) deviceFDB(ctx context.Context, in *struct {
|
||||
Slug string `path:"slug" doc:"Device slug."`
|
||||
VLAN string `query:"vlan" pattern:"^[0-9]{1,4}$" doc:"Only this VLAN."`
|
||||
Interface string `query:"interface" doc:"Only this interface (ULID)."`
|
||||
MAC string `query:"mac" doc:"Only MACs starting with this prefix (at least 6 hex digits)."`
|
||||
Current bool `query:"current" default:"true" doc:"true (default): rows the last walk saw. false: with history."`
|
||||
PageParams
|
||||
}) (*struct{ Body L2FDBList }, error) {
|
||||
d, err := s.l2Device(ctx, in.Slug)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
f := store.FDBFilter{VLAN: optInt(in.VLAN), InterfaceID: in.Interface, Current: in.Current}
|
||||
if in.MAC != "" {
|
||||
if f.MACPrefix, err = hostaddr.ParsePrefix(in.MAC); err != nil {
|
||||
return nil, huma.Error422UnprocessableEntity(err.Error())
|
||||
}
|
||||
}
|
||||
rows, total, err := s.Store.ListFDB(ctx, d.ID, f, store.Page{Offset: in.offset(), Limit: in.limit()})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
items, err := s.fdbRows(ctx, rows)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
return &struct{ Body L2FDBList }{L2FDBList{Items: items, Page: in.pageNo(), PerPage: in.limit(), Total: total}}, nil
|
||||
}
|
||||
|
||||
func (s *server) deviceARP(ctx context.Context, in *struct {
|
||||
Slug string `path:"slug" doc:"Device slug."`
|
||||
IP string `query:"ip" doc:"An address or a CIDR."`
|
||||
Interface string `query:"interface" doc:"Only this interface (ULID)."`
|
||||
MAC string `query:"mac" doc:"Only MACs starting with this prefix (at least 6 hex digits)."`
|
||||
Current bool `query:"current" default:"true" doc:"true (default): rows the last walk saw. false: with history."`
|
||||
PageParams
|
||||
}) (*struct{ Body L2ARPList }, error) {
|
||||
d, err := s.l2Device(ctx, in.Slug)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
f := store.ARPFilter{InterfaceID: in.Interface, Current: in.Current}
|
||||
if in.IP != "" {
|
||||
if f.IP, err = parseIPOrCIDR(in.IP); err != nil {
|
||||
return nil, huma.Error422UnprocessableEntity(err.Error())
|
||||
}
|
||||
}
|
||||
if in.MAC != "" {
|
||||
if f.MACPrefix, err = hostaddr.ParsePrefix(in.MAC); err != nil {
|
||||
return nil, huma.Error422UnprocessableEntity(err.Error())
|
||||
}
|
||||
}
|
||||
rows, total, err := s.Store.ListARP(ctx, d.ID, f, store.Page{Offset: in.offset(), Limit: in.limit()})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
items := make([]L2ARPRow, 0, len(rows))
|
||||
for _, r := range rows {
|
||||
items = append(items, L2ARPRow{
|
||||
IP: r.IP, MAC: r.Mac, Vendor: s.vendorOf(r.Mac), MACKind: hostaddr.Kind(r.Mac), Iface: ifaceRef(r.InterfaceID, r.IfName),
|
||||
Current: r.Current, FirstSeen: r.FirstSeen, LastSeen: r.LastSeen,
|
||||
})
|
||||
}
|
||||
return &struct{ Body L2ARPList }{L2ARPList{Items: items, Page: in.pageNo(), PerPage: in.limit(), Total: total}}, nil
|
||||
}
|
||||
|
||||
// parseIPOrCIDR reads an address as a host prefix or a CIDR as given.
|
||||
func parseIPOrCIDR(s string) (netip.Prefix, error) {
|
||||
q, err := hostaddr.Classify(s)
|
||||
switch {
|
||||
case err != nil:
|
||||
return netip.Prefix{}, err
|
||||
case q.Kind == hostaddr.QueryIP:
|
||||
return netip.PrefixFrom(q.IP, q.IP.BitLen()), nil
|
||||
case q.Kind == hostaddr.QueryCIDR:
|
||||
return q.Prefix, nil
|
||||
}
|
||||
return netip.Prefix{}, hostaddr.ErrBadQuery
|
||||
}
|
||||
|
||||
func (s *server) deviceL2Ports(ctx context.Context, in *struct {
|
||||
Slug string `path:"slug" doc:"Device slug."`
|
||||
}) (*struct{ Body L2PortList }, error) {
|
||||
d, err := s.l2Device(ctx, in.Slug)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
counts, err := s.Store.CurrentMACCounts(ctx, d.ID)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
ifs, err := s.Store.ListInterfaces(ctx, d.ID, false)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
lldp, err := s.Store.LLDPIfIndexes(ctx, []int64{d.ID})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
f := portFacts{lldp: lldp, edgeMax: s.edgeMax()}
|
||||
out := L2PortList{Items: []L2Port{}}
|
||||
for i := range ifs {
|
||||
n := counts[ifs[i].ID]
|
||||
if n == 0 {
|
||||
continue
|
||||
}
|
||||
row := store.FDBRow{DeviceID: d.ID, IfIndex: ifs[i].IfIndex, IfType: ifs[i].IfType}
|
||||
why := f.uplink(&row, n)
|
||||
ref := L2IfaceRef{ID: ifs[i].ID, IfName: ifs[i].IfName}
|
||||
out.Items = append(out.Items, L2Port{Iface: ref, MACs: n, Uplink: why != "", UplinkReason: why})
|
||||
}
|
||||
return &struct{ Body L2PortList }{out}, nil
|
||||
}
|
||||
|
||||
func (s *server) interfaceMACs(ctx context.Context, in *struct {
|
||||
ID string `path:"id" doc:"Interface ULID."`
|
||||
Current bool `query:"current" default:"true" doc:"true (default): MACs the last walk saw. false: with history."`
|
||||
PageParams
|
||||
}) (*struct{ Body L2FDBList }, error) {
|
||||
if _, err := s.Store.GetInterface(ctx, in.ID); err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
rows, total, err := s.Store.InterfaceMACs(ctx, in.ID, in.Current, store.Page{Offset: in.offset(), Limit: in.limit()})
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
items, err := s.fdbRows(ctx, rows)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
return &struct{ Body L2FDBList }{L2FDBList{Items: items, Page: in.pageNo(), PerPage: in.limit(), Total: total}}, nil
|
||||
}
|
||||
302
internal/api/l2_test.go
Normal file
@@ -0,0 +1,302 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/api"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// stubVendors is a registry of two vendors.
|
||||
type stubVendors struct{}
|
||||
|
||||
func (stubVendors) Vendor(mac string) string {
|
||||
switch {
|
||||
case strings.HasPrefix(mac, "f0:9f:c2"):
|
||||
return "Ubiquiti Inc"
|
||||
case mac[1] == '2' || mac[1] == '6' || mac[1] == 'a' || mac[1] == 'e':
|
||||
return "Private / randomised"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (stubVendors) Available() bool { return true }
|
||||
|
||||
func (stubVendors) Match(sub string) func(string) bool {
|
||||
return func(mac string) bool { return strings.Contains(strings.ToLower((stubVendors{}).Vendor(mac)), sub) }
|
||||
}
|
||||
|
||||
// seedL2Net seeds three switches that all see the same host: its own access switch, a distribution switch where the
|
||||
// host's MAC is one of ten behind an uplink, and a core switch whose port toward the distribution layer has an
|
||||
// LLDP neighbour.
|
||||
const host = "f0:9f:c2:11:22:33"
|
||||
|
||||
func seedL2Net(t *testing.T, e *env) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
access := e.device(t, "access1", "yyz", "access")
|
||||
dist := e.device(t, "dist1", "yyz", "distribution")
|
||||
core := e.device(t, "core1", "yyz", "core")
|
||||
aif := e.iface(t, access, 7, "Gi1/0/7", "")
|
||||
dif := e.iface(t, dist, 48, "Te1/1", "")
|
||||
cif := e.iface(t, core, 3, "Eth3", "")
|
||||
now := e.clk.Now()
|
||||
rec := func(d *store.Device, entries []store.FDBEntry, arp []store.ARPEntry) {
|
||||
if err := e.st.RecordL2Walk(ctx, d.ID, store.L2WalkResult{At: now, Source: "qbridge", FDB: entries, ARP: arp,
|
||||
VLANs: []store.VLANName{{VLAN: 20, Name: "cameras"}}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
rec(access, []store.FDBEntry{{InterfaceID: aif.ID, VLAN: 20, Mac: host}}, nil)
|
||||
var many []store.FDBEntry
|
||||
many = append(many, store.FDBEntry{InterfaceID: dif.ID, VLAN: 20, Mac: host})
|
||||
for i := 1; i <= 9; i++ {
|
||||
many = append(many, store.FDBEntry{InterfaceID: dif.ID, VLAN: 20, Mac: fmt.Sprintf("02:00:00:00:00:%02x", i)})
|
||||
}
|
||||
rec(dist, many, nil)
|
||||
rec(core, []store.FDBEntry{{InterfaceID: cif.ID, VLAN: 20, Mac: host}}, []store.ARPEntry{
|
||||
{IP: "192.0.2.77", Mac: host, InterfaceID: cif.ID}, {IP: "2001:db8::77", Mac: host, InterfaceID: cif.ID},
|
||||
})
|
||||
// The core's port toward the distribution layer has an LLDP neighbour (local port number = ifIndex).
|
||||
if err := e.st.ReplaceComponents(ctx, core.ID, "LLDP-MIB::lldpRemTable", []store.Component{{Idx: "0.3.1", Label: "dist1"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func searchL2(t *testing.T, e *env, q string) api.L2SearchResult {
|
||||
t.Helper()
|
||||
var got api.L2SearchResult
|
||||
e.getJSON(t, "/api/v1/l2/search?q="+url.QueryEscape(q), &got)
|
||||
return got
|
||||
}
|
||||
|
||||
func TestAMACSeenOnAnAccessPortAndTwoUplinksResolvesToTheAccessPortFirst(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, func(d *api.Deps) { d.Vendors = stubVendors{} })
|
||||
seedL2Net(t, e)
|
||||
got := searchL2(t, e, host)
|
||||
if got.Total != 1 || len(got.Items) != 1 {
|
||||
t.Fatalf("hits = %+v", got)
|
||||
}
|
||||
h := got.Items[0]
|
||||
if h.MAC != host || h.Vendor != "Ubiquiti Inc" || h.MACKind != "global" {
|
||||
t.Errorf("hit = %+v", h)
|
||||
}
|
||||
if len(h.Locations) != 3 {
|
||||
t.Fatalf("locations = %+v", h.Locations)
|
||||
}
|
||||
first := h.Locations[0]
|
||||
if !first.Edge || first.Device.Slug == "" || first.Iface == nil || first.Iface.IfName != "Gi1/0/7" || first.PortMACCount != 1 ||
|
||||
first.VLAN != 20 || first.VLANName != "cameras" || !first.Current {
|
||||
t.Errorf("the access port must come first: %+v", first)
|
||||
}
|
||||
reasons := map[string]string{}
|
||||
for _, l := range h.Locations[1:] {
|
||||
if l.Edge {
|
||||
t.Errorf("an uplink was reported as an edge port: %+v", l)
|
||||
}
|
||||
reasons[l.Device.Name] = l.UplinkReason
|
||||
}
|
||||
if reasons["dist1"] != "macs" || reasons["core1"] != "lldp" {
|
||||
t.Errorf("uplink reasons = %v", reasons)
|
||||
}
|
||||
if len(h.IPs) != 2 {
|
||||
t.Errorf("ips = %+v", h.IPs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSearchAcceptsEveryMACFormatAndPrefixAndRefusesGarbage(t *testing.T) {
|
||||
e := newEnv(t, 0)
|
||||
seedL2Net(t, e)
|
||||
for _, q := range []string{"f09f.c211.2233", "F0-9F-C2-11-22-33", "f09fc2112233", "f0:9f:c2", "f09fc2", "F0:9F:C2:11:2"} {
|
||||
if got := searchL2(t, e, q); got.Total != 1 || got.Items[0].MAC != host {
|
||||
t.Errorf("q=%q: %+v", q, got)
|
||||
}
|
||||
}
|
||||
for _, q := range []string{"hello", "f09f", "zz:zz:zz:zz:zz:zz", "1.2.3", "10.0.0.0/99", "vendor:"} {
|
||||
rec := e.do("GET", "/api/v1/l2/search?q="+url.QueryEscape(q), "")
|
||||
isProblem(t, rec, 422)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIPv6AndIPv4AndCIDRSearchFindTheMAC(t *testing.T) {
|
||||
e := newEnv(t, 0)
|
||||
seedL2Net(t, e)
|
||||
for q, kind := range map[string]string{"2001:db8::77": "ip", "192.0.2.77": "ip", "192.0.2.0/24": "cidr", "2001:db8::/32": "cidr"} {
|
||||
got := searchL2(t, e, q)
|
||||
if got.Kind != kind || got.Total != 1 || got.Items[0].MAC != host {
|
||||
t.Errorf("q=%q: %+v", q, got)
|
||||
}
|
||||
}
|
||||
if got := searchL2(t, e, "198.51.100.0/24"); got.Total != 0 || len(got.Items) != 0 {
|
||||
t.Errorf("a CIDR with no entries matched: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVendorSearchListsCurrentMACsAndWorksEmptyWithoutARegistry(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, func(d *api.Deps) { d.Vendors = stubVendors{} })
|
||||
seedL2Net(t, e)
|
||||
got := searchL2(t, e, "vendor:ubiquiti")
|
||||
if got.Kind != "vendor" || !got.VendorsAvailable || got.Total != 1 || got.Items[0].MAC != host {
|
||||
t.Fatalf("vendor search = %+v", got)
|
||||
}
|
||||
var vlan20 api.L2SearchResult
|
||||
e.getJSON(t, "/api/v1/l2/search?q=vendor:ubiquiti&vlan=30", &vlan20)
|
||||
if vlan20.Total != 0 {
|
||||
t.Errorf("a vendor search restricted to another VLAN matched: %+v", vlan20)
|
||||
}
|
||||
|
||||
none := newEnv(t, 0)
|
||||
seedL2Net(t, none)
|
||||
got = searchL2(t, none, "vendor:ubiquiti")
|
||||
if got.VendorsAvailable || got.Total != 0 || len(got.Items) != 0 {
|
||||
t.Errorf("without a registry the vendor search must find nothing, not fail: %+v", got)
|
||||
}
|
||||
// The same lookup by MAC still works, with an empty vendor.
|
||||
if h := searchL2(t, none, host).Items[0]; h.Vendor != "" || h.MACKind != "global" || len(h.Locations) != 3 {
|
||||
t.Errorf("hit without a registry = %+v", h)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocallyAdministeredMACsAreFlaggedPrivate(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, func(d *api.Deps) { d.Vendors = stubVendors{} })
|
||||
seedL2Net(t, e)
|
||||
h := searchL2(t, e, "02:00:00:00:00:01").Items[0]
|
||||
if h.MACKind != "local" || h.Vendor != "Private / randomised" {
|
||||
t.Errorf("hit = %+v", h)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeviceTablesFilterByVLANInterfaceAndCurrent(t *testing.T) {
|
||||
e := newEnv(t, 0)
|
||||
seedL2Net(t, e)
|
||||
devs, _, _ := e.st.ListDevices(context.Background(), store.DeviceFilter{}, store.Page{})
|
||||
var dist, core *store.Device
|
||||
for i := range devs {
|
||||
switch devs[i].Name {
|
||||
case "dist1":
|
||||
dist = &devs[i]
|
||||
case "core1":
|
||||
core = &devs[i]
|
||||
}
|
||||
}
|
||||
var fdb api.L2FDBList
|
||||
e.getJSON(t, "/api/v1/devices/"+dist.Slug+"/fdb", &fdb)
|
||||
if fdb.Total != 10 || len(fdb.Items) != 10 || !fdb.Items[0].Current {
|
||||
t.Fatalf("fdb = %+v", fdb)
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+dist.Slug+"/fdb?vlan=30", &fdb)
|
||||
if fdb.Total != 0 {
|
||||
t.Errorf("vlan filter: %+v", fdb)
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+dist.Slug+"/fdb?mac=f09fc2", &fdb)
|
||||
if fdb.Total != 1 || fdb.Items[0].MAC != host {
|
||||
t.Errorf("mac prefix filter: %+v", fdb)
|
||||
}
|
||||
isProblem(t, e.do("GET", "/api/v1/devices/"+dist.Slug+"/fdb?mac=zz", ""), 422)
|
||||
|
||||
var arp api.L2ARPList
|
||||
e.getJSON(t, "/api/v1/devices/"+core.Slug+"/arp", &arp)
|
||||
if arp.Total != 2 {
|
||||
t.Fatalf("arp = %+v", arp)
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+core.Slug+"/arp?ip=2001:db8::/32", &arp)
|
||||
if arp.Total != 1 || arp.Items[0].IP != "2001:db8::77" {
|
||||
t.Errorf("arp ip filter: %+v", arp)
|
||||
}
|
||||
|
||||
// History: a later walk that no longer sees the host leaves the row, no longer current.
|
||||
if err := e.st.RecordL2Walk(context.Background(), core.ID, store.L2WalkResult{At: e.clk.Now().Add(time.Hour)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+core.Slug+"/fdb", &fdb)
|
||||
if fdb.Total != 0 {
|
||||
t.Errorf("current=true must hide rows the last walk missed: %+v", fdb)
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+core.Slug+"/fdb?current=false", &fdb)
|
||||
if fdb.Total != 1 || fdb.Items[0].Current {
|
||||
t.Errorf("history: %+v", fdb)
|
||||
}
|
||||
isProblem(t, e.do("GET", "/api/v1/devices/nope/fdb", ""), 404)
|
||||
}
|
||||
|
||||
func TestInterfaceMACsWithTheirIPsAndPortsSummary(t *testing.T) {
|
||||
e := newEnv(t, 0)
|
||||
seedL2Net(t, e)
|
||||
ifs, _, _ := e.st.SearchInterfaces(context.Background(), "Eth3", false, store.Page{})
|
||||
var macs api.L2FDBList
|
||||
e.getJSON(t, "/api/v1/interfaces/"+ifs[0].ID+"/macs", &macs)
|
||||
if macs.Total != 1 || macs.Items[0].MAC != host || len(macs.Items[0].IPs) != 2 {
|
||||
t.Fatalf("macs = %+v", macs)
|
||||
}
|
||||
isProblem(t, e.do("GET", "/api/v1/interfaces/01ARZ3NDEKTSV4RRFFQ69G5FAV/macs", ""), 404)
|
||||
|
||||
devs, _, _ := e.st.ListDevices(context.Background(), store.DeviceFilter{}, store.Page{})
|
||||
for _, d := range devs {
|
||||
var sum api.L2Summary
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/l2", &sum)
|
||||
if !sum.HasData || sum.LastWalk == nil || sum.Source != "qbridge" || len(sum.VLANs) != 1 || sum.EdgeMaxMACs != 8 {
|
||||
t.Errorf("%s summary = %+v", d.Name, sum)
|
||||
}
|
||||
var ports api.L2PortList
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/l2/ports", &ports)
|
||||
if len(ports.Items) != 1 {
|
||||
t.Fatalf("%s ports = %+v", d.Name, ports)
|
||||
}
|
||||
p := ports.Items[0]
|
||||
wantUp := map[string]string{"access1": "", "dist1": "macs", "core1": "lldp"}[d.Name]
|
||||
if p.UplinkReason != wantUp || p.Uplink != (wantUp != "") {
|
||||
t.Errorf("%s port = %+v", d.Name, p)
|
||||
}
|
||||
}
|
||||
other := e.device(t, "plain", "yyz", "server")
|
||||
var sum api.L2Summary
|
||||
e.getJSON(t, "/api/v1/devices/"+other.Slug+"/l2", &sum)
|
||||
if sum.HasData || sum.LastWalk != nil || sum.VLANs == nil {
|
||||
t.Errorf("a device the job has not walked: %+v", sum)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEdgeLimitComesFromTheLiveSetting(t *testing.T) {
|
||||
limit := 8
|
||||
e := newEnvDeps(t, 0, func(d *api.Deps) { d.L2EdgeMax = func() int { return limit } })
|
||||
seedL2Net(t, e)
|
||||
if h := searchL2(t, e, host).Items[0]; h.Locations[1].UplinkReason == "" || h.Locations[0].Device.Name != "access1" {
|
||||
t.Fatalf("baseline = %+v", h.Locations)
|
||||
}
|
||||
limit = 20 // the distribution port now counts as an edge port: it carries 10 MACs
|
||||
h := searchL2(t, e, host).Items[0]
|
||||
edge := 0
|
||||
for _, l := range h.Locations {
|
||||
if l.Edge {
|
||||
edge++
|
||||
}
|
||||
}
|
||||
if edge != 2 || h.Locations[0].Device.Name != "access1" {
|
||||
t.Errorf("with a limit of 20 the 10-MAC port is an edge port, ranked after the 1-MAC one: %+v", h.Locations)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPollerHealthCarriesL2StatusAndNotes(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, func(d *api.Deps) {
|
||||
d.L2Health = func() *api.L2Health { return &api.L2Health{OK: 3, TimedOut: 1, Skipped: []string{"slow1"}} }
|
||||
d.HealthNotes = func() []api.HealthNote {
|
||||
return []api.HealthNote{{Code: "oui_unavailable", Message: "no IEEE vendor registry is loaded"}}
|
||||
}
|
||||
})
|
||||
var h api.PollerHealth
|
||||
e.getJSON(t, "/api/v1/poller/health", &h)
|
||||
if h.L2 == nil || h.L2.TimedOut != 1 || len(h.Notes) != 1 || h.Notes[0].Code != "oui_unavailable" {
|
||||
t.Errorf("health = %+v", h)
|
||||
}
|
||||
plain := newEnv(t, 0)
|
||||
h = api.PollerHealth{}
|
||||
plain.getJSON(t, "/api/v1/poller/health", &h)
|
||||
if h.L2 != nil || h.Notes == nil || len(h.Notes) != 0 {
|
||||
t.Errorf("a server without the L2 job: %+v", h)
|
||||
}
|
||||
}
|
||||
146
internal/api/l2_types.go
Normal file
@@ -0,0 +1,146 @@
|
||||
package api
|
||||
|
||||
import "time"
|
||||
|
||||
// VendorLookup names the vendor of a MAC from the IEEE registry. It is optional: with none, vendor is empty
|
||||
// everywhere. *oui.Service implements it.
|
||||
type VendorLookup interface {
|
||||
// Vendor is the registered vendor of a canonical MAC, "Private / randomised" for a locally administered
|
||||
// one, and "" when unknown.
|
||||
Vendor(mac string) string
|
||||
// Match returns a predicate for MACs whose vendor contains the (lower-case) substring.
|
||||
Match(sub string) func(mac string) bool
|
||||
// Available reports whether a registry is loaded.
|
||||
Available() bool
|
||||
}
|
||||
|
||||
// L2IfaceRef is the port a MAC was learned on.
|
||||
type L2IfaceRef struct {
|
||||
ID string `json:"id" doc:"Interface ULID."`
|
||||
IfName string `json:"ifname"`
|
||||
}
|
||||
|
||||
// L2IP is an address an ARP or IPv6 neighbour table maps to a MAC.
|
||||
type L2IP struct {
|
||||
IP string `json:"ip" doc:"Canonical text; IPv6 compressed."`
|
||||
Device DeviceRef `json:"device" doc:"The device whose table holds the entry (usually the gateway)."`
|
||||
LastSeen time.Time `json:"last_seen"`
|
||||
}
|
||||
|
||||
// L2Location is one place a MAC was seen: a port of a device in a VLAN.
|
||||
type L2Location struct {
|
||||
Device DeviceRef `json:"device"`
|
||||
Iface *L2IfaceRef `json:"iface,omitempty" doc:"Absent when the bridge port has no ifIndex."`
|
||||
VLAN int64 `json:"vlan" doc:"0 when the device's BRIDGE-MIB table does not say."`
|
||||
// VLANName is empty when the device reports no name for the VLAN.
|
||||
VLANName string `json:"vlan_name"`
|
||||
// Edge is true for a port where a host is plugged in: not an uplink. A port is an uplink when it carries more MACs
|
||||
// than l2.edge_max_macs, has an LLDP neighbour, or is a LAG.
|
||||
Edge bool `json:"edge"`
|
||||
// UplinkReason is why the port is not an edge port: macs, lldp or lag; empty for an edge port.
|
||||
UplinkReason string `json:"uplink_reason"`
|
||||
// PortMACCount is how many MACs the port carried in the walk this row was last seen in.
|
||||
PortMACCount int64 `json:"port_mac_count"`
|
||||
// Current is true when the device's last walk saw this row.
|
||||
Current bool `json:"current"`
|
||||
FirstSeen time.Time `json:"first_seen"`
|
||||
LastSeen time.Time `json:"last_seen"`
|
||||
}
|
||||
|
||||
// L2Hit is one MAC with the addresses it answers to and the places it was seen, edge ports first.
|
||||
type L2Hit struct {
|
||||
MAC string `json:"mac" doc:"Lower-case aa:bb:cc:dd:ee:ff."`
|
||||
// Vendor is the IEEE-registered vendor, "Private / randomised" for a locally administered MAC, empty when
|
||||
// unknown or when no registry is loaded.
|
||||
Vendor string `json:"vendor"`
|
||||
MACKind string `json:"mac_kind" enum:"global,local,multicast"`
|
||||
IPs []L2IP `json:"ips"`
|
||||
Locations []L2Location `json:"locations" doc:"Edge ports first (current before history, then fewest MACs, then newest), then uplinks."`
|
||||
}
|
||||
|
||||
// L2SearchResult is a page of MAC hits.
|
||||
type L2SearchResult struct {
|
||||
Items []L2Hit `json:"items"`
|
||||
Page int64 `json:"page"`
|
||||
PerPage int64 `json:"per_page"`
|
||||
Total int64 `json:"total"`
|
||||
// Kind says how q was read: mac, prefix, ip, cidr or vendor.
|
||||
Kind string `json:"kind" enum:"mac,prefix,ip,cidr,vendor"`
|
||||
// VendorsAvailable is false when no IEEE registry is loaded, in which case a vendor: search finds nothing.
|
||||
VendorsAvailable bool `json:"vendors_available"`
|
||||
}
|
||||
|
||||
// L2FDBRow is a row of a device's forwarding table.
|
||||
type L2FDBRow struct {
|
||||
MAC string `json:"mac"`
|
||||
Vendor string `json:"vendor"`
|
||||
MACKind string `json:"mac_kind" enum:"global,local,multicast"`
|
||||
VLAN int64 `json:"vlan"`
|
||||
VLANName string `json:"vlan_name"`
|
||||
Iface *L2IfaceRef `json:"iface,omitempty"`
|
||||
IPs []string `json:"ips" doc:"Addresses any ARP or neighbour table maps this MAC to."`
|
||||
Current bool `json:"current"`
|
||||
FirstSeen time.Time `json:"first_seen"`
|
||||
LastSeen time.Time `json:"last_seen"`
|
||||
}
|
||||
|
||||
// L2FDBList is a page of L2FDBRow.
|
||||
type L2FDBList struct {
|
||||
Items []L2FDBRow `json:"items"`
|
||||
Page int64 `json:"page"`
|
||||
PerPage int64 `json:"per_page"`
|
||||
Total int64 `json:"total"`
|
||||
}
|
||||
|
||||
// L2ARPRow is a row of a device's ARP or IPv6 neighbour table.
|
||||
type L2ARPRow struct {
|
||||
IP string `json:"ip"`
|
||||
MAC string `json:"mac"`
|
||||
Vendor string `json:"vendor"`
|
||||
MACKind string `json:"mac_kind" enum:"global,local,multicast"`
|
||||
Iface *L2IfaceRef `json:"iface,omitempty"`
|
||||
Current bool `json:"current"`
|
||||
FirstSeen time.Time `json:"first_seen"`
|
||||
LastSeen time.Time `json:"last_seen"`
|
||||
}
|
||||
|
||||
// L2ARPList is a page of L2ARPRow.
|
||||
type L2ARPList struct {
|
||||
Items []L2ARPRow `json:"items"`
|
||||
Page int64 `json:"page"`
|
||||
PerPage int64 `json:"per_page"`
|
||||
Total int64 `json:"total"`
|
||||
}
|
||||
|
||||
// L2VLAN is a VLAN of a device with its name.
|
||||
type L2VLAN struct {
|
||||
VLAN int64 `json:"vlan"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// L2Summary says what the L2 job knows about a device; the device page shows its L2 tab only when there is something.
|
||||
type L2Summary struct {
|
||||
HasData bool `json:"has_data" doc:"True when the device has any FDB or ARP row, current or historical."`
|
||||
LastWalk *time.Time `json:"last_walk" nullable:"true"`
|
||||
Source string `json:"source" doc:"What supplied the FDB: qbridge, bridge or per-vlan."`
|
||||
FDBCurrent int64 `json:"fdb_current"`
|
||||
FDBTotal int64 `json:"fdb_total"`
|
||||
ARPCurrent int64 `json:"arp_current"`
|
||||
ARPTotal int64 `json:"arp_total"`
|
||||
VLANs []L2VLAN `json:"vlans"`
|
||||
EdgeMaxMACs int64 `json:"edge_max_macs" doc:"l2.edge_max_macs: ports with more MACs are uplinks."`
|
||||
}
|
||||
|
||||
// L2Port is how many MACs a port carries now, for the Ports tab.
|
||||
type L2Port struct {
|
||||
Iface L2IfaceRef `json:"iface"`
|
||||
MACs int64 `json:"macs" doc:"MACs seen on the port by the device's last walk."`
|
||||
// Uplink is true when the port has more MACs than l2.edge_max_macs, an LLDP neighbour, or is a LAG.
|
||||
Uplink bool `json:"uplink"`
|
||||
UplinkReason string `json:"uplink_reason"`
|
||||
}
|
||||
|
||||
// L2PortList lists the ports of a device that carry MACs.
|
||||
type L2PortList struct {
|
||||
Items []L2Port `json:"items"`
|
||||
}
|
||||
144
internal/api/overview.go
Normal file
@@ -0,0 +1,144 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
var overviewRanges = map[string]time.Duration{"24h": 24 * time.Hour, "7d": 7 * 24 * time.Hour, "30d": 30 * 24 * time.Hour}
|
||||
|
||||
func (s *server) registerOverview(api huma.API) {
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "get-device-overview", Method: http.MethodGet, Path: "/api/v1/devices/{slug}/overview", Tags: []string{"devices"},
|
||||
Summary: "Data of the device Overview tab",
|
||||
Description: "Aggregate traffic (the sum over the device's non-excluded, non-retired, reliable ports; one query per direction), " +
|
||||
"port counts, availability over 24h, 7d, 30d and 1y from device_down alerts, ICMP round-trip time and the last 20 events. " +
|
||||
"The step rules are those of /graph, limited to the hot store. If the metric store cannot answer, traffic and rtt are " +
|
||||
"empty and partial says so.",
|
||||
Errors: []int{http.StatusNotFound, http.StatusUnprocessableEntity},
|
||||
}, s.overview)
|
||||
}
|
||||
|
||||
type overviewIn struct {
|
||||
Slug string `path:"slug" doc:"Device slug."`
|
||||
Range string `query:"range" enum:"24h,7d,30d" default:"24h" doc:"Window ending now."`
|
||||
}
|
||||
|
||||
func (s *server) overview(ctx context.Context, in *overviewIn) (*struct{ Body OverviewResponse }, error) {
|
||||
d, err := s.Store.GetDeviceBySlug(ctx, in.Slug)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
if d.PurgedAt != nil {
|
||||
return nil, huma.Error404NotFound("not found")
|
||||
}
|
||||
now := s.Clock.Now()
|
||||
ifs, err := s.Store.ListInterfaces(ctx, d.ID, false)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
resp := OverviewResponse{Range: in.Range, Ports: portCounts(ifs), Traffic: OverviewTraffic{In: []Point{}, Out: []Point{}}, RTT: []Point{}}
|
||||
if err := s.overviewHistory(ctx, &resp, d, ifs, now); err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
s.overviewMetrics(ctx, &resp, d, ifs, now)
|
||||
return &struct{ Body OverviewResponse }{resp}, nil
|
||||
}
|
||||
|
||||
// overviewHistory fills availability and events from the inventory.
|
||||
func (s *server) overviewHistory(ctx context.Context, resp *OverviewResponse, d *store.Device, ifs []store.Interface, now time.Time) error {
|
||||
rows, err := s.Store.ListDeviceDownAlerts(ctx, d.Slug, now.Add(-year))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp.Availability = availabilityOf(rows, now, d.CreatedAt)
|
||||
alerts, _, err := s.Store.ListAlerts(ctx, store.AlertFilter{Device: d.Slug}, store.Page{Limit: overviewEvents})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
objects := []string{d.Slug}
|
||||
for _, i := range ifs {
|
||||
objects = append(objects, i.ID)
|
||||
}
|
||||
audit, err := s.Store.ListAuditByObjects(ctx, objects, overviewEvents)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp.Events = mergeEvents(alerts, audit)
|
||||
return nil
|
||||
}
|
||||
|
||||
// overviewMetrics fills the traffic and rtt series; a metric store that cannot answer leaves them empty.
|
||||
func (s *server) overviewMetrics(ctx context.Context, resp *OverviewResponse, d *store.Device, ifs []store.Interface, now time.Time) {
|
||||
plan, err := s.metricPlan(&metricGraphIn{From: now.Add(-overviewRanges[resp.Range]), To: now})
|
||||
if err != nil {
|
||||
resp.Partial = append(resp.Partial, "range")
|
||||
return
|
||||
}
|
||||
resp.StepSeconds = int64(plan.newest().step / time.Second)
|
||||
grid := gridOf(plan)
|
||||
step := plan.newest().step
|
||||
hc, soft := countedIfaces(ifs, &resp.Traffic)
|
||||
for _, dir := range []string{dirIn, dirOut} {
|
||||
if len(hc)+len(soft) == 0 {
|
||||
break
|
||||
}
|
||||
vals, err := s.fetchMetricSeries(ctx, trafficExpr(hc, soft, dir, step), plan)
|
||||
if err != nil {
|
||||
resp.Partial = append(resp.Partial, "traffic")
|
||||
resp.Traffic.In, resp.Traffic.Out = []Point{}, []Point{}
|
||||
break
|
||||
}
|
||||
if dir == dirIn {
|
||||
resp.Traffic.In = pointsOf(grid, vals)
|
||||
} else {
|
||||
resp.Traffic.Out = pointsOf(grid, vals)
|
||||
}
|
||||
}
|
||||
rtt, err := s.fetchMetricSeries(ctx, `nmon_rtt_seconds{device=`+quote(d.Slug)+`}`, plan)
|
||||
if err != nil {
|
||||
resp.Partial = append(resp.Partial, "rtt")
|
||||
return
|
||||
}
|
||||
if len(rtt) > 0 {
|
||||
resp.RTT = pointsOf(grid, rtt)
|
||||
}
|
||||
}
|
||||
|
||||
// countedIfaces splits the ports that count toward traffic by counter width and counts the unreliable ones left out.
|
||||
func countedIfaces(ifs []store.Interface, t *OverviewTraffic) (hc, soft []string) {
|
||||
for _, i := range ifs {
|
||||
switch {
|
||||
case i.RetiredAt != nil || i.Excluded:
|
||||
case i.Unreliable:
|
||||
t.UnreliableExcluded++
|
||||
case i.HasHCCounters:
|
||||
hc = append(hc, i.ID)
|
||||
default:
|
||||
soft = append(soft, i.ID)
|
||||
}
|
||||
}
|
||||
return hc, soft
|
||||
}
|
||||
|
||||
// trafficExpr is the summed bits per second of dir; HC counters where a port has them, else the 32-bit ones.
|
||||
func trafficExpr(hc, soft []string, dir string, step time.Duration) string {
|
||||
names := map[string][2]string{dirIn: {"ifHCInOctets", "ifInOctets"}, dirOut: {"ifHCOutOctets", "ifOutOctets"}}[dir]
|
||||
win := secs(max(step, minRateWindow))
|
||||
var parts []string
|
||||
for n, ids := range [][]string{hc, soft} {
|
||||
if len(ids) > 0 {
|
||||
parts = append(parts, `rate(`+names[n]+`{iface=~`+ifaceMatcher(ids)+`}[`+win+`])*8`)
|
||||
}
|
||||
}
|
||||
switch len(parts) {
|
||||
case 1:
|
||||
return `sum(` + parts[0] + `)`
|
||||
}
|
||||
return `sum(` + parts[0] + ` or ` + parts[1] + `)`
|
||||
}
|
||||
145
internal/api/overview_calc.go
Normal file
@@ -0,0 +1,145 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
const (
|
||||
adminDown = 2
|
||||
operUp = 1
|
||||
overviewEvents = 20
|
||||
year = 365 * 24 * time.Hour
|
||||
)
|
||||
|
||||
// availabilityPeriods are the keys of the availability block and their lengths.
|
||||
var availabilityPeriods = []time.Duration{24 * time.Hour, 7 * 24 * time.Hour, 30 * 24 * time.Hour, year}
|
||||
|
||||
type span struct{ from, to time.Time }
|
||||
|
||||
// downtime sums the time inside [from, to] covered by the incidents; overlaps count once.
|
||||
func downtime(incidents []span, from, to time.Time) time.Duration {
|
||||
clipped := make([]span, 0, len(incidents))
|
||||
for _, s := range incidents {
|
||||
s.from, s.to = laterOf(s.from, from), earlierOf(s.to, to)
|
||||
if s.from.Before(s.to) {
|
||||
clipped = append(clipped, s)
|
||||
}
|
||||
}
|
||||
sort.Slice(clipped, func(i, j int) bool { return clipped[i].from.Before(clipped[j].from) })
|
||||
var total time.Duration
|
||||
var end time.Time
|
||||
for _, s := range clipped {
|
||||
if s.from.Before(end) {
|
||||
s.from = end
|
||||
}
|
||||
if s.to.After(s.from) {
|
||||
total += s.to.Sub(s.from)
|
||||
end = s.to
|
||||
}
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
func laterOf(a, b time.Time) time.Time {
|
||||
if a.After(b) {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func earlierOf(a, b time.Time) time.Time {
|
||||
if a.Before(b) {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// availabilityPct is 100 x (1 - downtime/period) over the period ending at now, counted from the device's creation
|
||||
// if that is later; nil when the device has existed for no time.
|
||||
func availabilityPct(incidents []span, now, created time.Time, period time.Duration) *float64 {
|
||||
start := laterOf(now.Add(-period), created)
|
||||
if !start.Before(now) {
|
||||
return nil
|
||||
}
|
||||
v := 100 * (1 - float64(downtime(incidents, start, now))/float64(now.Sub(start)))
|
||||
return &v
|
||||
}
|
||||
|
||||
// incidentSpans turns device_down rows into spans; an open row lasts until now.
|
||||
func incidentSpans(rows []store.Alert, now time.Time) []span {
|
||||
out := make([]span, 0, len(rows))
|
||||
for _, a := range rows {
|
||||
if a.FiredAt == nil {
|
||||
continue
|
||||
}
|
||||
end := now
|
||||
if a.ClearedAt != nil {
|
||||
end = *a.ClearedAt
|
||||
}
|
||||
out = append(out, span{from: *a.FiredAt, to: end})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func availabilityOf(rows []store.Alert, now, created time.Time) OverviewAvailability {
|
||||
sp := incidentSpans(rows, now)
|
||||
at := func(i int) *float64 { return availabilityPct(sp, now, created, availabilityPeriods[i]) }
|
||||
return OverviewAvailability{Day: at(0), Week: at(1), Month: at(2), Year: at(3)}
|
||||
}
|
||||
|
||||
// portCounts classifies the ports that count: admin down first, then oper up, else down.
|
||||
func portCounts(ifs []store.Interface) OverviewPorts {
|
||||
var p OverviewPorts
|
||||
for _, i := range ifs {
|
||||
if i.RetiredAt != nil || i.Excluded {
|
||||
continue
|
||||
}
|
||||
p.Total++
|
||||
switch {
|
||||
case i.AdminStatus == adminDown:
|
||||
p.AdminDown++
|
||||
case i.OperStatus == operUp:
|
||||
p.Up++
|
||||
default:
|
||||
p.Down++
|
||||
}
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// mergeEvents turns alerts (a fired and a resolved line each) and audit entries into the newest-first list.
|
||||
func mergeEvents(alerts []store.Alert, audit []store.AuditEntry) []OverviewEvent {
|
||||
ev := []OverviewEvent{}
|
||||
for _, a := range alerts {
|
||||
if a.FiredAt != nil {
|
||||
ev = append(ev, OverviewEvent{At: a.FiredAt.UTC(), Kind: "alert", Severity: a.Severity, Summary: "Fired: " + a.Summary})
|
||||
}
|
||||
if a.ClearedAt != nil {
|
||||
sum := closedWord(a.State) + ": " + a.Summary
|
||||
ev = append(ev, OverviewEvent{At: a.ClearedAt.UTC(), Kind: "alert", Severity: a.Severity, Summary: sum})
|
||||
}
|
||||
}
|
||||
for _, e := range audit {
|
||||
ev = append(ev, OverviewEvent{At: e.At.UTC(), Kind: "audit", Summary: auditSummary(e)})
|
||||
}
|
||||
sort.SliceStable(ev, func(i, j int) bool { return ev[i].At.After(ev[j].At) })
|
||||
return ev[:min(len(ev), overviewEvents)]
|
||||
}
|
||||
|
||||
func closedWord(state string) string {
|
||||
if state == store.AlertRetired {
|
||||
return "Retired"
|
||||
}
|
||||
return "Resolved"
|
||||
}
|
||||
|
||||
func auditSummary(e store.AuditEntry) string {
|
||||
return fmt.Sprintf("%s by %s", e.Action, e.Actor)
|
||||
}
|
||||
|
||||
// ifaceMatcher is a regexp alternation of interface ids, for an iface=~ matcher.
|
||||
func ifaceMatcher(ids []string) string { return quote(regexAlt(ids)) }
|
||||
149
internal/api/overview_test.go
Normal file
@@ -0,0 +1,149 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/api"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
func overviewEnv(t *testing.T) (*env, *store.Device) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
d := e.device(t, "sw1", "yyz", "access")
|
||||
return e, d
|
||||
}
|
||||
|
||||
func hoursAgo(e *env, n float64) *time.Time {
|
||||
x := e.clk.Now().Add(-time.Duration(n * float64(time.Hour)))
|
||||
return &x
|
||||
}
|
||||
|
||||
func TestOverviewShape(t *testing.T) {
|
||||
e, d := overviewEnv(t)
|
||||
e.iface(t, d, 1, "up1", "")
|
||||
e.iface(t, d, 2, "dn1", "", func(i *store.Interface) { i.OperStatus = 2 })
|
||||
e.iface(t, d, 3, "ad1", "", func(i *store.Interface) { i.AdminStatus, i.OperStatus = 2, 2 })
|
||||
e.iface(t, d, 4, "soft", "", func(i *store.Interface) { i.HasHCCounters = false })
|
||||
e.iface(t, d, 5, "bad", "", func(i *store.Interface) { i.Unreliable = true })
|
||||
e.iface(t, d, 6, "ex", "", func(i *store.Interface) { i.Excluded = true })
|
||||
now := e.clk.Now().Unix()
|
||||
e.q.fn = func(_, expr string, _, _ time.Time, _ time.Duration) ([]byte, error) {
|
||||
if strings.HasPrefix(expr, "nmon_rtt") {
|
||||
return matrix(mrow{map[string]string{}, map[int64]float64{now - now%300: 0.002}}), nil
|
||||
}
|
||||
return matrix(mrow{map[string]string{}, map[int64]float64{now - now%300: 5e6}}), nil
|
||||
}
|
||||
var got api.OverviewResponse
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/overview?range=7d", &got)
|
||||
if got.Range != "7d" || got.StepSeconds != 660 || got.Ports != (api.OverviewPorts{Total: 5, Up: 3, Down: 1, AdminDown: 1}) {
|
||||
t.Fatalf("got %+v", got)
|
||||
}
|
||||
if got.Traffic.UnreliableExcluded != 1 || len(got.Traffic.In) == 0 || len(got.Traffic.Out) != len(got.Traffic.In) || len(got.RTT) != len(got.Traffic.In) {
|
||||
t.Fatalf("traffic %d/%d rtt %d excl %d", len(got.Traffic.In), len(got.Traffic.Out), len(got.RTT), got.Traffic.UnreliableExcluded)
|
||||
}
|
||||
var traffic []string
|
||||
for _, c := range e.q.queries() {
|
||||
if strings.HasPrefix(c.expr, "sum(") {
|
||||
traffic = append(traffic, c.expr)
|
||||
}
|
||||
}
|
||||
if len(traffic) != 2 || !strings.Contains(traffic[0], "ifHCInOctets") || !strings.Contains(traffic[0], "ifInOctets{") ||
|
||||
strings.Contains(traffic[0], "ifOutOctets") || !strings.Contains(traffic[1], "ifHCOutOctets") {
|
||||
t.Fatalf("traffic queries = %q", traffic)
|
||||
}
|
||||
if got.Availability.Day != nil || got.Events == nil {
|
||||
t.Errorf("availability %+v events %v", got.Availability, got.Events)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOverviewHotStoreDownIsPartial(t *testing.T) {
|
||||
e, d := overviewEnv(t)
|
||||
e.iface(t, d, 1, "up1", "")
|
||||
e.q.err = errors.New("boom")
|
||||
e.q.fn = nil
|
||||
var got api.OverviewResponse
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/overview", &got)
|
||||
if got.Range != "24h" || got.Ports.Total != 1 || len(got.Traffic.In) != 0 || len(got.RTT) != 0 || len(got.Partial) == 0 {
|
||||
t.Fatalf("got %+v", got)
|
||||
}
|
||||
isProblem(t, e.do("GET", "/api/v1/devices/nope/overview", ""), 404)
|
||||
isProblem(t, e.do("GET", "/api/v1/devices/"+d.Slug+"/overview?range=9d", ""), 422)
|
||||
}
|
||||
|
||||
func TestOverviewAvailability(t *testing.T) {
|
||||
e, d := overviewEnv(t)
|
||||
ctx := t.Context()
|
||||
e.clk.Advance(48 * time.Hour) // the device is two days old
|
||||
add := func(i int, fired, cleared *time.Time) {
|
||||
a := &store.Alert{Rule: "device_down", DeviceSlug: d.Slug, SeriesKey: "device_down|" + d.Slug + "|" + string(rune('a'+i)),
|
||||
State: store.AlertResolved, BaseState: store.AlertResolved, FiredAt: fired, ClearedAt: cleared, FlapChanges: "[]", Severity: "critical", Summary: "down"}
|
||||
if err := e.st.UpsertAlertState(ctx, a); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
add(0, hoursAgo(e, 10), hoursAgo(e, 8)) // 2h
|
||||
add(1, hoursAgo(e, 9), hoursAgo(e, 7)) // overlaps: union 10h..7h = 3h
|
||||
add(2, hoursAgo(e, 1), nil) // open: 1h
|
||||
add(3, hoursAgo(e, 300), hoursAgo(e, 100)) // before the device existed: no effect
|
||||
var got api.OverviewResponse
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/overview", &got)
|
||||
a := got.Availability
|
||||
day := 100 * (1 - 4.0/24)
|
||||
age := 100 * (1 - 4.0/48)
|
||||
if a.Day == nil || *a.Day < day-0.001 || *a.Day > day+0.001 {
|
||||
t.Errorf("24h = %v want %v", a.Day, day)
|
||||
}
|
||||
for name, v := range map[string]*float64{"7d": a.Week, "30d": a.Month, "1y": a.Year} { // clipped to created_at: 48h
|
||||
if v == nil || *v < age-0.001 || *v > age+0.001 {
|
||||
t.Errorf("%s = %v want %v", name, v, age)
|
||||
}
|
||||
}
|
||||
if len(got.Events) != 5 && len(got.Events) != 7 {
|
||||
t.Errorf("events = %+v", got.Events)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOverviewEventsMergeNewestFirst(t *testing.T) {
|
||||
e, d := overviewEnv(t)
|
||||
i := e.iface(t, d, 1, "up1", "")
|
||||
ctx := t.Context()
|
||||
a := &store.Alert{Rule: "device_down", DeviceSlug: d.Slug, SeriesKey: "k", State: store.AlertResolved, BaseState: store.AlertResolved,
|
||||
FiredAt: hoursAgo(e, 3), ClearedAt: hoursAgo(e, 1), FlapChanges: "[]", Severity: "critical", Summary: "sw1 down"}
|
||||
if err := e.st.UpsertAlertState(ctx, a); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = e.st.AppendAudit(ctx, "ann", "interface.alert_down", i.ID, nil, nil)
|
||||
_ = e.st.AppendAudit(ctx, "ann", "device.edit", "other", nil, nil)
|
||||
var got api.OverviewResponse
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/overview", &got)
|
||||
if len(got.Events) != 3 {
|
||||
t.Fatalf("events = %+v", got.Events)
|
||||
}
|
||||
kinds := got.Events[0].Kind + got.Events[1].Kind + got.Events[2].Kind
|
||||
if kinds != "auditalertalert" || got.Events[1].Severity != "critical" || got.Events[0].Severity != "" ||
|
||||
!strings.HasPrefix(got.Events[2].Summary, "Fired") || !strings.HasPrefix(got.Events[1].Summary, "Resolved") {
|
||||
t.Fatalf("events = %+v", got.Events)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetricGraphAggregate(t *testing.T) {
|
||||
e := newEnvDeps(t, 0, withShipped(t))
|
||||
d := e.device(t, "srv1", "yyz", "server")
|
||||
now := e.clk.Now().Unix()
|
||||
e.q.body = matrix(mrow{map[string]string{}, map[int64]float64{now - 600: 40}})
|
||||
var got api.MetricGraphResponse
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/metrics/graph?metric=hrProcessorLoad&aggregate=avg", &got)
|
||||
if want := `avg(hrProcessorLoad{device="` + d.Slug + `"})`; e.q.q != want {
|
||||
t.Errorf("expr %q want %q", e.q.q, want)
|
||||
}
|
||||
if len(got.Series) == 0 {
|
||||
t.Error("no series")
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug+"/metrics/graph?metric=hrProcessorLoad&aggregate=max&component=CPU+0", &got)
|
||||
if !strings.Contains(e.q.q, `component="CPU 0"`) || strings.Contains(e.q.q, "max(") {
|
||||
t.Errorf("explicit component must win: %q", e.q.q)
|
||||
}
|
||||
}
|
||||
46
internal/api/overview_types.go
Normal file
@@ -0,0 +1,46 @@
|
||||
package api
|
||||
|
||||
import "time"
|
||||
|
||||
// OverviewTraffic is the device's aggregate traffic over the range.
|
||||
type OverviewTraffic struct {
|
||||
In []Point `json:"in" doc:"[unix_s, bits per second|null] pairs: the sum over the device's counted ports."`
|
||||
Out []Point `json:"out"`
|
||||
UnreliableExcluded int `json:"unreliable_excluded" doc:"Ports left out of the sums because their counters are unreliable."`
|
||||
}
|
||||
|
||||
// OverviewPorts counts the device's ports by state.
|
||||
type OverviewPorts struct {
|
||||
Total int `json:"total"`
|
||||
Up int `json:"up"`
|
||||
Down int `json:"down" doc:"Administratively up but not operationally up."`
|
||||
AdminDown int `json:"admin_down"`
|
||||
}
|
||||
|
||||
// OverviewAvailability is the percent of time the device was not down over each period; null for no time yet.
|
||||
type OverviewAvailability struct {
|
||||
Day *float64 `json:"24h" nullable:"true"`
|
||||
Week *float64 `json:"7d" nullable:"true"`
|
||||
Month *float64 `json:"30d" nullable:"true"`
|
||||
Year *float64 `json:"1y" nullable:"true"`
|
||||
}
|
||||
|
||||
// OverviewEvent is one line of the recent-events list.
|
||||
type OverviewEvent struct {
|
||||
At time.Time `json:"at"`
|
||||
Kind string `json:"kind" enum:"alert,audit"`
|
||||
Severity string `json:"severity,omitempty" enum:"critical,warning,info"`
|
||||
Summary string `json:"summary"`
|
||||
}
|
||||
|
||||
// OverviewResponse is the data of the device Overview tab.
|
||||
type OverviewResponse struct {
|
||||
Range string `json:"range" enum:"24h,7d,30d"`
|
||||
StepSeconds int64 `json:"step_seconds"`
|
||||
Traffic OverviewTraffic `json:"traffic"`
|
||||
Ports OverviewPorts `json:"ports"`
|
||||
Availability OverviewAvailability `json:"availability"`
|
||||
RTT []Point `json:"rtt" doc:"[unix_s, seconds|null] pairs of ICMP round-trip time; empty when ICMP is off."`
|
||||
Events []OverviewEvent `json:"events" doc:"The last 20 alert transitions and audit entries of the device or its ports."`
|
||||
Partial []string `json:"partial,omitempty" doc:"Parts that could not be read, for example the metric store."`
|
||||
}
|
||||
@@ -41,9 +41,8 @@ func Routes() []Route {
|
||||
// adminReads are GET routes only an admin may call: they reveal who can reach nmon or how it is configured.
|
||||
var adminReads = []string{"GET /api/v1/tokens", "GET /api/v1/settings", "GET /api/v1/audit"}
|
||||
|
||||
// viewerPosts are non-GET routes that only read, so a viewer may call them. None today; a query endpoint taking
|
||||
// a body would be listed here.
|
||||
var viewerPosts []string
|
||||
// viewerPosts are non-GET routes that only read, so a viewer may call them (a query endpoint taking a body).
|
||||
var viewerPosts = []string{"POST /api/v1/alert-policies/preview"}
|
||||
|
||||
// DeclarePolicy states the minimum role of each route that departs from the default (GET and HEAD need viewer,
|
||||
// everything else admin).
|
||||
|
||||
192
internal/api/printers.go
Normal file
@@ -0,0 +1,192 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"math"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
const (
|
||||
supplyTable = "Printer-MIB::prtMarkerSuppliesTable"
|
||||
// supplyWarnPercent is the level under which a supply reads as low, as the printer_supply_low rule does.
|
||||
supplyWarnPercent = 10
|
||||
)
|
||||
|
||||
func (s *server) registerPrinters(api huma.API) {
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "list-printers", Method: http.MethodGet, Path: "/api/v1/printers", Tags: []string{"devices"},
|
||||
Summary: "Printer fleet: supplies, trays and page counts",
|
||||
Description: "Every device whose profile is or extends printer, with its supply and paper tray levels from the hot store. " +
|
||||
"The printer with the lowest supply comes first; printers with no reading come last.",
|
||||
Errors: []int{http.StatusTooManyRequests, http.StatusServiceUnavailable},
|
||||
}, s.listPrinters)
|
||||
}
|
||||
|
||||
// printerNames are the metrics the fleet view reads.
|
||||
var printerNames = []string{
|
||||
"prtMarkerSuppliesLevel", "prtMarkerSuppliesMaxCapacity", "prtInputCurrentLevel", "prtInputMaxCapacity", "prtMarkerLifeCount",
|
||||
}
|
||||
|
||||
// printerDevices lists the devices whose profile chain includes printer.
|
||||
func (s *server) printerDevices(ctx context.Context) ([]store.Device, error) {
|
||||
if s.Profiles == nil {
|
||||
return nil, nil
|
||||
}
|
||||
var out []store.Device
|
||||
for off := int64(0); ; off += store.MaxLimit {
|
||||
rows, n, err := s.Store.ListDevices(ctx, store.DeviceFilter{}, store.Page{Offset: off, Limit: store.MaxLimit})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, d := range rows {
|
||||
if res := s.Profiles.Get(d.Profile); res != nil && containsString(res.Chain, "printer") {
|
||||
out = append(out, d)
|
||||
}
|
||||
}
|
||||
if off+store.MaxLimit >= n {
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func containsString(l []string, s string) bool {
|
||||
for _, x := range l {
|
||||
if x == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
type printerEntry struct {
|
||||
dev store.Device
|
||||
row PrinterRow
|
||||
lowest float64 // lowest supply percent; +Inf with no reading
|
||||
sentinel map[int]string
|
||||
}
|
||||
|
||||
// supplyState: empty is critical, under supplyWarnPercent a warning, a sentinel or no reading unknown or ok by its words.
|
||||
func supplyState(pct *float64, text *string) string {
|
||||
switch {
|
||||
case pct != nil && *pct <= 0:
|
||||
return "crit"
|
||||
case pct != nil && *pct < supplyWarnPercent:
|
||||
return "warn"
|
||||
case pct != nil:
|
||||
return "ok"
|
||||
case text != nil && *text != "unknown":
|
||||
return "ok"
|
||||
}
|
||||
return "unknown"
|
||||
}
|
||||
|
||||
// levels builds the rows of one level/capacity metric pair, lowest first.
|
||||
func levels(rd readings, level, capacity string, sent map[int]string) []PrinterSupply {
|
||||
out := []PrinterSupply{}
|
||||
for comp, lv := range rd[level] {
|
||||
sup := PrinterSupply{Component: comp}
|
||||
if words, ok := sent[int(lv.v)]; ok && lv.v == math.Trunc(lv.v) {
|
||||
sup.Text = &words
|
||||
} else if c := rd.at(capacity, comp); c != nil && c.v > 0 && lv.v >= 0 {
|
||||
p := 100 * lv.v / c.v
|
||||
sup.Percent = &p
|
||||
}
|
||||
sup.State = supplyState(sup.Percent, sup.Text)
|
||||
out = append(out, sup)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
a, b := out[i].Percent, out[j].Percent
|
||||
switch {
|
||||
case a != nil && b != nil && *a != *b:
|
||||
return *a < *b
|
||||
case (a == nil) != (b == nil):
|
||||
return a != nil
|
||||
}
|
||||
return out[i].Component < out[j].Component
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *server) printerEntry(d store.Device, rd readings) printerEntry {
|
||||
sent := map[int]string{}
|
||||
if res := s.Profiles.Get(d.Profile); res != nil {
|
||||
if m, ok := res.Metric("prtMarkerSuppliesLevel"); ok {
|
||||
sent = m.Sentinels
|
||||
}
|
||||
}
|
||||
e := printerEntry{dev: d, lowest: math.Inf(1), sentinel: sent}
|
||||
e.row = PrinterRow{
|
||||
Device: DeviceRefSite{Slug: d.Slug, Name: d.Name, Site: d.Site},
|
||||
Supplies: levels(rd, "prtMarkerSuppliesLevel", "prtMarkerSuppliesMaxCapacity", sent),
|
||||
Trays: levels(rd, "prtInputCurrentLevel", "prtInputMaxCapacity", sent),
|
||||
}
|
||||
for _, sup := range e.row.Supplies {
|
||||
if sup.Percent != nil {
|
||||
e.lowest = min(e.lowest, *sup.Percent)
|
||||
}
|
||||
}
|
||||
var pages float64
|
||||
for _, c := range rd["prtMarkerLifeCount"] {
|
||||
pages += c.v
|
||||
e.row.PagesTotal = &pages
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
func (s *server) listPrinters(ctx context.Context, in *PageParams) (*struct{ Body PrinterList }, error) {
|
||||
devs, err := s.printerDevices(ctx)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
var all map[string]readings
|
||||
if len(devs) > 0 {
|
||||
slugs := make([]string, len(devs))
|
||||
for i, d := range devs {
|
||||
slugs[i] = d.Slug
|
||||
}
|
||||
if all, err = s.queryLatest(ctx, `device=~`+quote(regexAlt(slugs)), printerNames); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
entries := make([]printerEntry, len(devs))
|
||||
reach := s.Reachability.Reachability(ctx, devs)
|
||||
for i, d := range devs {
|
||||
entries[i] = s.printerEntry(d, all[d.Slug])
|
||||
entries[i].row.Reachability = reach[d.Slug]
|
||||
}
|
||||
sort.SliceStable(entries, func(i, j int) bool {
|
||||
if entries[i].lowest != entries[j].lowest {
|
||||
return entries[i].lowest < entries[j].lowest
|
||||
}
|
||||
return strings.ToLower(entries[i].dev.Name) < strings.ToLower(entries[j].dev.Name)
|
||||
})
|
||||
lo := min(in.offset(), int64(len(entries)))
|
||||
hi := min(lo+in.limit(), int64(len(entries)))
|
||||
out := PrinterList{Items: []PrinterRow{}, Page: in.pageNo(), PerPage: in.limit(), Total: int64(len(entries))}
|
||||
for _, e := range entries[lo:hi] {
|
||||
if err := s.addSupplyAttrs(ctx, &e); err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
out.Items = append(out.Items, e.row)
|
||||
}
|
||||
return &struct{ Body PrinterList }{out}, nil
|
||||
}
|
||||
|
||||
// addSupplyAttrs fills each supply's type and color from the discovery attributes of its row.
|
||||
func (s *server) addSupplyAttrs(ctx context.Context, e *printerEntry) error {
|
||||
attrs, err := s.componentAttrs(ctx, e.dev.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range e.row.Supplies {
|
||||
a := attrs[attrKey(supplyTable, e.row.Supplies[i].Component)]
|
||||
e.row.Supplies[i].Type, e.row.Supplies[i].Color = a["type"], a["color"]
|
||||
}
|
||||
return nil
|
||||
}
|
||||
143
internal/api/profiles.go
Normal file
@@ -0,0 +1,143 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
|
||||
"github.com/v10networks/nmon/internal/discovery"
|
||||
"github.com/v10networks/nmon/internal/profile"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// ProfileMatchRules are a profile's own match rules (they are not inherited through extends).
|
||||
type ProfileMatchRules struct {
|
||||
SysObjectID []string `json:"sysObjectID" doc:"sysObjectID prefixes, matched on a dot boundary."`
|
||||
SysDescr []string `json:"sysDescr" doc:"Regular expressions tried against sysDescr."`
|
||||
}
|
||||
|
||||
// ProfileView is one profile as GET /profiles lists it.
|
||||
type ProfileView struct {
|
||||
Name string `json:"name"`
|
||||
Priority int `json:"priority" doc:"The highest priority wins when several profiles match a device."`
|
||||
Extends string `json:"extends,omitempty"`
|
||||
Source string `json:"source" enum:"shipped,operator" doc:"operator: the file comes from the operator's profiles directory."`
|
||||
Match ProfileMatchRules `json:"match"`
|
||||
}
|
||||
|
||||
// ProfileMatchView says how a device's profile was chosen.
|
||||
type ProfileMatchView struct {
|
||||
MatchedBy string `json:"matched_by" enum:"sysObjectID,sysDescr,default,pinned" doc:"The rule that picks it."`
|
||||
GenericReason string `json:"generic_reason,omitempty" doc:"Why no vendor profile applies, when the profile is generic."`
|
||||
}
|
||||
|
||||
func (s *server) registerProfiles(api huma.API) {
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "list-profiles", Method: http.MethodGet, Path: "/api/v1/profiles", Tags: []string{"profiles"},
|
||||
Summary: "List the loaded device profiles with their match rules",
|
||||
}, s.listProfiles)
|
||||
}
|
||||
|
||||
func (s *server) listProfiles(context.Context, *struct{}) (*struct{ Body []ProfileView }, error) {
|
||||
out := []ProfileView{}
|
||||
if s.Profiles != nil {
|
||||
for _, n := range s.Profiles.Names() {
|
||||
r, _ := s.Profiles.Lookup(n)
|
||||
v := ProfileView{
|
||||
Name: r.Name, Priority: r.Priority, Source: "shipped",
|
||||
Match: ProfileMatchRules{SysObjectID: nonNil(r.Match.SysObjectID), SysDescr: nonNil(r.Match.SysDescr)},
|
||||
}
|
||||
if r.Operator {
|
||||
v.Source = "operator"
|
||||
}
|
||||
if len(r.Chain) > 1 {
|
||||
v.Extends = r.Chain[1]
|
||||
}
|
||||
out = append(out, v)
|
||||
}
|
||||
}
|
||||
return &struct{ Body []ProfileView }{out}, nil
|
||||
}
|
||||
|
||||
// profileMatch explains the device's profile choice; nil when no profiles are loaded.
|
||||
func (s *server) profileMatch(d *store.Device) *ProfileMatchView {
|
||||
if d.ProfilePinned {
|
||||
return &ProfileMatchView{MatchedBy: "pinned"}
|
||||
}
|
||||
if s.Profiles == nil {
|
||||
return nil
|
||||
}
|
||||
_, by := s.Profiles.MatchWhy(d.SysObjectID, d.SysDescr)
|
||||
v := &ProfileMatchView{MatchedBy: string(by)}
|
||||
if by == profile.ByDefault {
|
||||
if d.SysObjectID == "" {
|
||||
v.GenericReason = "not discovered yet: no sysObjectID to match"
|
||||
} else {
|
||||
v.GenericReason = fmt.Sprintf("no vendor profile matched sysObjectID %s or sysDescr", d.SysObjectID)
|
||||
}
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// planProfile applies the body's profile to the plan: a name pins it, "" returns the device to automatic
|
||||
// matching (the profile is re-matched from the stored sysObjectID/sysDescr now and again at each discovery).
|
||||
func (s *server) planProfile(p *editPlan, dev *store.Device, name string) error {
|
||||
if name == "" {
|
||||
p.after.ProfilePinned, p.after.Profile = false, s.autoProfile(dev)
|
||||
p.change["profile"] = "automatic"
|
||||
} else {
|
||||
if err := s.knownProfile(name); err != nil {
|
||||
return err
|
||||
}
|
||||
p.after.Profile, p.after.ProfilePinned = name, true
|
||||
p.change["profile"] = name
|
||||
}
|
||||
p.rediscover = p.after.Profile != dev.Profile || p.after.ProfilePinned != dev.ProfilePinned
|
||||
if !p.rediscover {
|
||||
delete(p.change, "profile")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *server) knownProfile(name string) error {
|
||||
if s.Profiles == nil {
|
||||
return nil
|
||||
}
|
||||
if _, ok := s.Profiles.Lookup(name); !ok {
|
||||
return huma.Error422UnprocessableEntity("unknown profile " + name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// autoProfile is the profile Match picks for the device's stored sysObjectID and sysDescr, or its current one
|
||||
// when nothing was discovered yet.
|
||||
func (s *server) autoProfile(dev *store.Device) string {
|
||||
if s.Profiles == nil || (dev.SysObjectID == "" && dev.SysDescr == "") {
|
||||
return dev.Profile
|
||||
}
|
||||
if m := s.Profiles.Match(dev.SysObjectID, dev.SysDescr); m != nil {
|
||||
return m.Name
|
||||
}
|
||||
return dev.Profile
|
||||
}
|
||||
|
||||
// rediscoverAfterEdit runs one discovery so the new profile's tables are read at once. A device that cannot be
|
||||
// reached keeps its edit; the next scheduled run applies it.
|
||||
func (s *server) rediscoverAfterEdit(ctx context.Context, slug string) {
|
||||
dev, err := s.Store.GetDeviceBySlug(ctx, slug)
|
||||
if err != nil || dev.RetiredAt != nil || dev.Kind != store.KindNetwork || dev.SNMPCredentialID == nil {
|
||||
return
|
||||
}
|
||||
dctx, cancel := context.WithTimeout(ctx, s.DiscoveryTimeout)
|
||||
defer cancel()
|
||||
deps := discovery.Deps{Store: s.Store, Profiles: s.Profiles, Clock: s.Clock, SNMPPort: s.SNMPPort, Logger: s.Logger}
|
||||
if _, err := discovery.Discover(dctx, deps, dev); err != nil {
|
||||
s.Logger.Warn("discovery after profile change failed", "device", slug, "err", err)
|
||||
return
|
||||
}
|
||||
if s.AfterDiscover != nil {
|
||||
s.AfterDiscover(dev.ID)
|
||||
}
|
||||
}
|
||||
79
internal/api/profiles_test.go
Normal file
@@ -0,0 +1,79 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"github.com/v10networks/nmon/internal/api"
|
||||
"github.com/v10networks/nmon/internal/profile"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
func profileEnv(t *testing.T) *env {
|
||||
t.Helper()
|
||||
set, err := profile.Load("")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return newEnvDeps(t, 0, func(d *api.Deps) { d.Profiles = set })
|
||||
}
|
||||
|
||||
func TestListProfiles(t *testing.T) {
|
||||
e := profileEnv(t)
|
||||
var got []api.ProfileView
|
||||
e.getJSON(t, "/api/v1/profiles", &got)
|
||||
by := map[string]api.ProfileView{}
|
||||
for _, p := range got {
|
||||
by[p.Name] = p
|
||||
}
|
||||
es, ok := by["ubiquiti-edgeswitch"]
|
||||
if !ok || es.Source != "shipped" || es.Priority != 30 || len(es.Match.SysObjectID) != 1 || es.Extends != "generic" {
|
||||
t.Fatalf("edgeswitch = %+v", es)
|
||||
}
|
||||
if _, ok := by["generic"]; !ok || len(by) < 15 {
|
||||
t.Fatalf("%d profiles", len(by))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPutDeviceProfile(t *testing.T) {
|
||||
e := profileEnv(t)
|
||||
d := e.device(t, "sw", "", "", func(d *store.Device) { d.SysObjectID, d.SysDescr = "1.3.6.1.4.1.4413", "EdgeSwitch 12 Fiber" })
|
||||
var v api.DeviceView
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &v)
|
||||
if v.ProfilePinned || v.ProfileMatch == nil || v.ProfileMatch.MatchedBy != "sysObjectID" {
|
||||
t.Fatalf("view %+v %+v", v, v.ProfileMatch)
|
||||
}
|
||||
if rec := e.do(http.MethodPut, "/api/v1/devices/"+d.Slug, `{"profile":"nope"}`); rec.Code != http.StatusUnprocessableEntity {
|
||||
t.Fatalf("unknown profile: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
if rec := e.do(http.MethodPut, "/api/v1/devices/"+d.Slug, `{"profile":"vyos"}`); rec.Code != http.StatusOK {
|
||||
t.Fatalf("pin: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &v)
|
||||
if v.Profile != "vyos" || !v.ProfilePinned || v.ProfileMatch.MatchedBy != "pinned" {
|
||||
t.Fatalf("after pin %+v", v)
|
||||
}
|
||||
if rec := e.do(http.MethodPut, "/api/v1/devices/"+d.Slug, `{"profile":""}`); rec.Code != http.StatusOK {
|
||||
t.Fatalf("auto: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &v)
|
||||
if v.Profile != "ubiquiti-edgeswitch" || v.ProfilePinned {
|
||||
t.Fatalf("after unpin %+v", v)
|
||||
}
|
||||
audits, _, err := e.st.ListAudit(context.Background(), store.Page{Limit: 10})
|
||||
if err != nil || len(audits) < 2 || audits[0].Action != "device.edit" {
|
||||
t.Fatalf("audit rows %d, %v", len(audits), err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenericDeviceExplainsWhy(t *testing.T) {
|
||||
e := profileEnv(t)
|
||||
d := e.device(t, "odd", "", "", func(d *store.Device) { d.SysObjectID, d.SysDescr = "1.3.6.1.4.1.99999.1", "mystery box" })
|
||||
var v api.DeviceView
|
||||
e.getJSON(t, "/api/v1/devices/"+d.Slug, &v)
|
||||
if v.ProfileMatch == nil || v.ProfileMatch.MatchedBy != "default" ||
|
||||
v.ProfileMatch.GenericReason != "no vendor profile matched sysObjectID 1.3.6.1.4.1.99999.1 or sysDescr" {
|
||||
t.Fatalf("%+v", v.ProfileMatch)
|
||||
}
|
||||
}
|
||||
41
internal/api/retire.go
Normal file
@@ -0,0 +1,41 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
|
||||
"github.com/danielgtaylor/huma/v2"
|
||||
)
|
||||
|
||||
func (s *server) registerRetire(api huma.API) {
|
||||
huma.Register(api, huma.Operation{
|
||||
OperationID: "retire-device", Method: http.MethodPost, Path: "/api/v1/devices/{slug}/retire", Tags: []string{"devices"},
|
||||
Summary: "Retire a device",
|
||||
Description: "Admin only. Soft-deletes the device and its interfaces: polling stops, history is kept, and adding the " +
|
||||
"same address again offers to restore it. Retiring a retired device is a 409.",
|
||||
Errors: []int{http.StatusNotFound, http.StatusConflict},
|
||||
}, s.retireDevice)
|
||||
}
|
||||
|
||||
func (s *server) retireDevice(ctx context.Context, in *struct {
|
||||
Slug string `path:"slug" doc:"Device slug."`
|
||||
}) (*deviceOut, error) {
|
||||
d, err := s.Store.GetDeviceBySlug(ctx, in.Slug)
|
||||
if err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
if d.PurgedAt != nil {
|
||||
return nil, huma.Error404NotFound("not found")
|
||||
}
|
||||
if d.RetiredAt != nil {
|
||||
return nil, huma.Error409Conflict("device is already retired")
|
||||
}
|
||||
if err := s.Store.RetireDevice(ctx, d.ID); err != nil {
|
||||
return nil, s.storeErr(err)
|
||||
}
|
||||
if err := s.Store.AppendAudit(ctx, actorOf(ctx), "device.retire", d.Slug, nil, map[string]any{"name": d.Name}); err != nil {
|
||||
s.Logger.Warn("audit device.retire", "err", err)
|
||||
}
|
||||
s.inventoryChanged(ctx)
|
||||
return s.view(ctx, d.Slug)
|
||||
}
|
||||
24
internal/api/retire_test.go
Normal file
@@ -0,0 +1,24 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRetireDevice(t *testing.T) {
|
||||
e := newEnv(t, 0)
|
||||
d := e.device(t, "Old Box", "", "")
|
||||
rec := e.do("POST", "/api/v1/devices/"+d.Slug+"/retire", "")
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("retire: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
got, err := e.st.GetDeviceBySlug(t.Context(), d.Slug)
|
||||
if err != nil || got.RetiredAt == nil {
|
||||
t.Fatalf("not retired: %+v %v", got, err)
|
||||
}
|
||||
if e.ref.n.Load() == 0 {
|
||||
t.Error("poller was not refreshed")
|
||||
}
|
||||
isProblem(t, e.do("POST", "/api/v1/devices/"+d.Slug+"/retire", ""), http.StatusConflict)
|
||||
isProblem(t, e.do("POST", "/api/v1/devices/nope/retire", ""), http.StatusNotFound)
|
||||
}
|
||||
45
internal/app/alert_optin.go
Normal file
@@ -0,0 +1,45 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/clock"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// optInSet answers alert.Options.OptedIn from the store's effective opt-in set (a port's own flag, or a group
|
||||
// holding it now with the flag on). The set is loaded once per entityTTL, and API writes invalidate it.
|
||||
type optInSet struct {
|
||||
st store.AlertDown
|
||||
cache *ttlCache[struct{}, map[string]bool]
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
last map[string]bool // the last set that loaded; used while the store fails
|
||||
}
|
||||
|
||||
func newOptInSet(st store.AlertDown, clk clock.Clock, ttl time.Duration, log *slog.Logger) *optInSet {
|
||||
return &optInSet{st: st, cache: newTTLCache[struct{}, map[string]bool](clk, ttl), log: log}
|
||||
}
|
||||
|
||||
// opted reports whether the interface is opted in to interface_down alerts. While the store fails it keeps
|
||||
// answering from the last good set, so a database hiccup neither fires nor closes alerts.
|
||||
func (o *optInSet) opted(iface string) bool {
|
||||
set, err := o.cache.get(struct{}{}, func() (map[string]bool, error) {
|
||||
return o.st.InterfacesAlertDown(context.Background())
|
||||
})
|
||||
o.mu.Lock()
|
||||
defer o.mu.Unlock()
|
||||
if err != nil {
|
||||
o.log.Warn("load alert-down opt-in set", "err", err)
|
||||
return o.last[iface]
|
||||
}
|
||||
o.last = set
|
||||
return set[iface]
|
||||
}
|
||||
|
||||
// invalidate drops the cached set; the API calls it after a write that can change it.
|
||||
func (o *optInSet) invalidate() { o.cache.invalidate() }
|
||||
52
internal/app/alert_optin_test.go
Normal file
@@ -0,0 +1,52 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/clock"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
type fakeAlertDown struct {
|
||||
set map[string]bool
|
||||
err error
|
||||
loads int
|
||||
}
|
||||
|
||||
func (f *fakeAlertDown) SetInterfaceAlertDown(context.Context, []string, bool) error { return nil }
|
||||
func (f *fakeAlertDown) InterfaceAlertDownVia(context.Context, []string) (map[string]string, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (f *fakeAlertDown) InterfacesAlertDown(context.Context) (map[string]bool, error) {
|
||||
f.loads++
|
||||
return f.set, f.err
|
||||
}
|
||||
|
||||
var _ store.AlertDown = (*fakeAlertDown)(nil)
|
||||
|
||||
func TestOptInSetCachesInvalidatesAndSurvivesStoreErrors(t *testing.T) {
|
||||
clk := clock.NewFake(time.Date(2026, 10, 23, 12, 0, 0, 0, time.UTC))
|
||||
f := &fakeAlertDown{set: map[string]bool{"p1": true}}
|
||||
o := newOptInSet(f, clk, time.Minute, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
if !o.opted("p1") || o.opted("p2") || f.loads != 1 {
|
||||
t.Fatalf("first answers wrong or loaded %d times", f.loads)
|
||||
}
|
||||
f.set = map[string]bool{"p2": true}
|
||||
if !o.opted("p1") {
|
||||
t.Fatal("answer changed inside the TTL")
|
||||
}
|
||||
o.invalidate()
|
||||
if o.opted("p1") || !o.opted("p2") {
|
||||
t.Fatal("invalidate did not reload")
|
||||
}
|
||||
clk.Advance(2 * time.Minute)
|
||||
f.err = errors.New("db down")
|
||||
if !o.opted("p2") {
|
||||
t.Fatal("a failing store must keep answering from the last good set")
|
||||
}
|
||||
}
|
||||
@@ -26,10 +26,12 @@ const entityTTL = time.Minute
|
||||
// queue, the reachability evaluator, the metric-rule evaluator and the
|
||||
// maintenance windows they consult.
|
||||
type alerting struct {
|
||||
queue *notify.Queue
|
||||
reach *reach.Evaluator
|
||||
rules *alert.Evaluator
|
||||
maint *maintenanceCache
|
||||
queue *notify.Queue
|
||||
reach *reach.Evaluator
|
||||
rules *alert.Evaluator
|
||||
maint *maintenanceCache
|
||||
optIn *optInSet
|
||||
policies *policyEngine
|
||||
|
||||
nt *notifier
|
||||
sys *sysAlarms
|
||||
@@ -54,9 +56,12 @@ func (a *App) newAlerting(s *settings) error {
|
||||
devices := newTTLCache[string, store.Device](clk, entityTTL)
|
||||
n := newNotifier(q, s.alerting.Receivers, a.st, devices, a.log)
|
||||
maint := newMaintenanceCache(a.st, clk, a.log)
|
||||
a.alerting = &alerting{queue: q, maint: maint, devices: devices}
|
||||
optIn := newOptInSet(a.st, clk, entityTTL, a.log)
|
||||
pe := newPolicyEngine(a.st, clk, optIn, s.alerting.Rules, a.log)
|
||||
a.alerting = &alerting{queue: q, maint: maint, devices: devices, optIn: optIn, policies: pe}
|
||||
a.alerting.reach = reach.New(reach.Options{
|
||||
Store: a.st, Notifier: reachNotifier{n}, Clock: clk, Log: a.log, PollInterval: s.PollInterval, ICMPOff: s.ICMP == config.ICMPOff,
|
||||
Store: a.st, Notifier: reachNotifier{n}, Clock: clk, Log: a.log, Wants: pe.Wants,
|
||||
PollInterval: s.PollInterval, ICMPOff: s.ICMP == config.ICMPOff,
|
||||
ICMPBlocked: func(d *store.Device) bool { return a.icmpBlocked(d) },
|
||||
InMaintenance: func(slug string, _ time.Time) bool {
|
||||
_, ok := maint.Covered(slug)
|
||||
@@ -64,7 +69,7 @@ func (a *App) newAlerting(s *settings) error {
|
||||
},
|
||||
})
|
||||
a.alerting.nt = n
|
||||
a.alerting.sys = newSysAlarms(a.st, clk, alertNotifier{n}, a.log)
|
||||
a.alerting.sys = newSysAlarms(a.st, clk, alertNotifier{n}, pe.SystemWanted, a.log)
|
||||
a.policy = pol
|
||||
if s.ICMP == config.ICMPOff {
|
||||
a.log.Info("ICMP probing is off (--icmp off): SNMP alone decides whether a device is down; " +
|
||||
@@ -94,7 +99,7 @@ func (a *App) startAlerting(s *settings) error {
|
||||
var err error
|
||||
al.rules, err = alert.New(alert.Options{
|
||||
Store: a.st, Querier: vmsup.NewQuerier(a.sup), Notifier: alertNotifier{al.nt}, Clock: clock.Real(), Rules: s.alerting.Rules,
|
||||
Groups: groupLookup{a.st}, Unreliable: a.unreliable(), Active: a.entityActive(), Suppressed: a.suppressed(),
|
||||
Groups: groupLookup{a.st}, Unreliable: a.unreliable(), Policy: al.policies.Metric, Active: a.entityActive(), Suppressed: a.suppressed(),
|
||||
StoreHealth: storeHealth(), Interval: s.alerting.Interval, Logger: a.log,
|
||||
})
|
||||
if err != nil {
|
||||
|
||||
@@ -25,14 +25,23 @@ func (a *App) apiDeps(s *settings) api.Deps {
|
||||
HotRetention: s.cfg.Retention.Hot.D(), FiveMinuteRetention: s.cfg.Retention.FiveMinute.D(), LongRetention: s.cfg.Retention.Long.D(),
|
||||
Reachability: a.alerting.reach, DeviceWarnings: a.alerting.reach.Warnings,
|
||||
Dependencies: depGraph{a.alerting.reach}, OnMaintenanceChange: a.alerting.maint.Refresh,
|
||||
OnAlertDownChange: a.alertDownChanged, AlertPolicy: a.alerting.policies,
|
||||
Tokens: a.svc.tokens, Secrets: a.st, Rollup: a.svc.rollup, P95: a.svc.p95, Groups: a.svc.groups, Zones: a.svc.zones,
|
||||
Checks: a.svc.checks, Proxmox: a.svc.proxmox,
|
||||
APITokens: a.auth.Tokens, Settings: a, Backups: a.backups, Stores: a.sup,
|
||||
Series: seriesDeleter{a.sup}, PurgeAfter: func() time.Duration { return a.live.get().PurgeAfter() },
|
||||
OnImported: a.discoverNew,
|
||||
OnImported: a.discoverNew, Vendors: a.vendors(), L2EdgeMax: a.l2EdgeMax, L2Health: a.l2Health, HealthNotes: a.healthNotes,
|
||||
}
|
||||
}
|
||||
|
||||
// vendors is the vendor lookup for the API, or nil (an interface holding a nil *oui.Service is not nil).
|
||||
func (a *App) vendors() api.VendorLookup {
|
||||
if a.oui == nil {
|
||||
return nil
|
||||
}
|
||||
return a.oui
|
||||
}
|
||||
|
||||
// discoverNew starts discovery for freshly imported devices.
|
||||
func (a *App) discoverNew(ids []int64) {
|
||||
for _, id := range ids {
|
||||
@@ -133,3 +142,9 @@ func (a *App) OnSIGHUP() {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// alertDownChanged drops the caches an alert_down flag, a group flag or a membership change makes stale.
|
||||
func (a *App) alertDownChanged() {
|
||||
a.alerting.optIn.invalidate()
|
||||
a.alerting.policies.Changed()
|
||||
}
|
||||
|
||||
@@ -20,8 +20,10 @@ import (
|
||||
"github.com/v10networks/nmon/internal/discovery"
|
||||
"github.com/v10networks/nmon/internal/diskguard"
|
||||
"github.com/v10networks/nmon/internal/ingest"
|
||||
"github.com/v10networks/nmon/internal/l2"
|
||||
"github.com/v10networks/nmon/internal/metrics"
|
||||
"github.com/v10networks/nmon/internal/netpolicy"
|
||||
"github.com/v10networks/nmon/internal/oui"
|
||||
"github.com/v10networks/nmon/internal/poller"
|
||||
"github.com/v10networks/nmon/internal/profile"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
@@ -59,6 +61,8 @@ type App struct {
|
||||
alerting *alerting
|
||||
policy *netpolicy.Policy
|
||||
svc services
|
||||
l2 *l2.Job // the MAC/ARP/FDB table job
|
||||
oui *oui.Service // the optional IEEE vendor registry
|
||||
ingester *ingest.Ingester
|
||||
|
||||
serveErr chan error // the HTTP server stopped on its own
|
||||
@@ -92,14 +96,17 @@ func (a *App) onStop(name string, fn func(ctx context.Context) error) {
|
||||
// Run starts nmon and blocks until ctx is cancelled or the HTTP server fails,
|
||||
// then shuts everything down gracefully.
|
||||
func Run(ctx context.Context, f config.Flags) error {
|
||||
// Catch SIGHUP before starting: Go's default action is to exit, and startup takes
|
||||
// seconds (metric stores), so a `systemctl reload` then would kill nmon. A SIGHUP
|
||||
// that arrives during startup is queued and applied once Start returns.
|
||||
hup := make(chan os.Signal, 1)
|
||||
signal.Notify(hup, syscall.SIGHUP)
|
||||
defer signal.Stop(hup)
|
||||
a, err := Start(ctx, f)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var runErr error
|
||||
hup := make(chan os.Signal, 1)
|
||||
signal.Notify(hup, syscall.SIGHUP)
|
||||
defer signal.Stop(hup)
|
||||
wait:
|
||||
for {
|
||||
select {
|
||||
@@ -122,7 +129,7 @@ wait:
|
||||
//
|
||||
// Startup order: inventory, metric stores, alerting (built), write queues and the
|
||||
// disk guard, profiles, poller, alerting (started), discovery scheduler, trap receiver,
|
||||
// rollup, p95 and group jobs, Proxmox sync and checks, push ingest (HTTP and Graphite),
|
||||
// rollup, p95 and group jobs, the MAC/ARP table job and vendor registry, Proxmox sync and checks, push ingest (HTTP and Graphite),
|
||||
// authentication, HTTP API, the nightly backup schedule.
|
||||
// Stop reverses it.
|
||||
func Start(ctx context.Context, f config.Flags) (_ *App, err error) {
|
||||
@@ -168,10 +175,7 @@ func Start(ctx context.Context, f config.Flags) (_ *App, err error) {
|
||||
a.newTraps(s)
|
||||
a.startScheduler()
|
||||
a.startTraps(s)
|
||||
if err = a.startRollup(s); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err = a.startCollectors(s); err != nil {
|
||||
if err = a.startJobs(s); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err = a.startIngest(s); err != nil {
|
||||
@@ -188,6 +192,17 @@ func Start(ctx context.Context, f config.Flags) (_ *App, err error) {
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// startJobs starts the storage and collection jobs that follow the poller: rollup with the p95 jobs, the MAC/ARP
|
||||
// table job with the vendor registry, then Proxmox sync and service checks.
|
||||
func (a *App) startJobs(s *settings) error {
|
||||
for _, start := range []func(*settings) error{a.startRollup, a.startL2, a.startCollectors} {
|
||||
if err := start(s); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Addr is the address the API listens on.
|
||||
func (a *App) Addr() string { return a.ln.Addr().String() }
|
||||
|
||||
|
||||
59
internal/app/bgp_alert.go
Normal file
@@ -0,0 +1,59 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alert"
|
||||
"github.com/v10networks/nmon/internal/notify"
|
||||
)
|
||||
|
||||
// ruleBGPPeerDown is the built-in rule whose notifications name the peer's AS and description.
|
||||
const ruleBGPPeerDown = "bgp_peer_down"
|
||||
|
||||
// describePeer adds the peer's AS, router id and description to a bgp_peer_down notification. The SNMP label
|
||||
// contract allows only device and component on a series, so these are component attributes kept from discovery
|
||||
// and are joined here, at delivery, instead of being carried in the rule's template.
|
||||
func (n *notifier) describePeer(ctx context.Context, note *notify.Notification, t alert.Transition) {
|
||||
if t.Rule != ruleBGPPeerDown || t.Component == "" {
|
||||
return
|
||||
}
|
||||
dev, err := n.st.GetDeviceBySlug(ctx, t.Device)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
comps, err := n.st.ListComponents(ctx, dev.ID)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
for _, c := range comps {
|
||||
if c.Label != t.Component {
|
||||
continue
|
||||
}
|
||||
var attrs map[string]string
|
||||
if json.Unmarshal([]byte(c.Attrs), &attrs) != nil || (attrs["remote_as"] == "" && attrs["description"] == "") {
|
||||
continue
|
||||
}
|
||||
detail := peerDetail(attrs)
|
||||
note.Summary += " " + detail
|
||||
note.Description += " " + detail
|
||||
for _, k := range []string{"remote_as", "local_as", "router_id", "description"} {
|
||||
if attrs[k] != "" {
|
||||
note.Labels[k] = attrs[k]
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// peerDetail renders "(AS 65001, transit-a)".
|
||||
func peerDetail(attrs map[string]string) string {
|
||||
switch {
|
||||
case attrs["remote_as"] != "" && attrs["description"] != "":
|
||||
return fmt.Sprintf("(AS %s, %s)", attrs["remote_as"], attrs["description"])
|
||||
case attrs["remote_as"] != "":
|
||||
return fmt.Sprintf("(AS %s)", attrs["remote_as"])
|
||||
}
|
||||
return fmt.Sprintf("(%s)", attrs["description"])
|
||||
}
|
||||
94
internal/app/l2wire.go
Normal file
@@ -0,0 +1,94 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/api"
|
||||
"github.com/v10networks/nmon/internal/clock"
|
||||
"github.com/v10networks/nmon/internal/config"
|
||||
"github.com/v10networks/nmon/internal/l2"
|
||||
"github.com/v10networks/nmon/internal/oui"
|
||||
)
|
||||
|
||||
// ouiHTTPTimeout bounds one registry download through the outbound allow-list.
|
||||
const ouiHTTPTimeout = 2 * time.Minute
|
||||
|
||||
// startL2 starts the MAC/ARP/FDB table job and the optional IEEE vendor registry. Both read their settings from the
|
||||
// live configuration on every pass, so a reload applies without a restart. Neither can stop nmon from starting.
|
||||
func (a *App) startL2(s *settings) error {
|
||||
clk := clock.Real()
|
||||
a.oui = oui.New(oui.Options{
|
||||
Dir: config.OUIDir(s.DataDir), Clock: clk, Logger: a.log,
|
||||
Settings: func() oui.Settings {
|
||||
c := a.live.get().OUI
|
||||
return oui.Settings{Fetch: c.FetchEnabled(), Refresh: c.RefreshOrDefault()}
|
||||
},
|
||||
Client: a.policy.HTTPClient(ouiHTTPTimeout),
|
||||
})
|
||||
job, err := l2.New(l2.Options{
|
||||
Store: a.st, Profiles: a.profiles, Clock: clk, Logger: a.log, Queue: a.hot, Paused: a.guard.PauseRollup,
|
||||
SNMPPort: s.SNMPPort, Settings: a.l2Settings,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("l2 job: %w", err)
|
||||
}
|
||||
a.l2 = job
|
||||
a.goStop("oui registry", a.oui.Run)
|
||||
a.goStop("l2 job", job.Run)
|
||||
return nil
|
||||
}
|
||||
|
||||
// l2Settings is the L2 job's view of the live configuration.
|
||||
func (a *App) l2Settings() l2.Settings {
|
||||
c := a.live.get().L2
|
||||
return l2.Settings{
|
||||
Interval: c.IntervalOrDefault(), Keep: c.KeepOrDefault(), Concurrency: c.Workers(), DeviceTimeout: c.DeviceTimeoutOrDefault(),
|
||||
}
|
||||
}
|
||||
|
||||
// l2EdgeMax is l2.edge_max_macs from the live configuration.
|
||||
func (a *App) l2EdgeMax() int { return a.live.get().L2.EdgeMax() }
|
||||
|
||||
// l2Health adapts the job's status for the poller health page.
|
||||
func (a *App) l2Health() *api.L2Health {
|
||||
if a.l2 == nil {
|
||||
return nil
|
||||
}
|
||||
st := a.l2.Status()
|
||||
h := &api.L2Health{
|
||||
Running: st.Running, IntervalS: int64(st.Interval.Seconds()), OK: int64(st.OK), TimedOut: int64(st.Timeouts),
|
||||
Failed: int64(st.Errors), Skipped: []string{},
|
||||
}
|
||||
if !st.LastEnd.IsZero() {
|
||||
t := st.LastEnd.UTC()
|
||||
h.LastCycle = &t
|
||||
}
|
||||
for _, d := range st.Devices {
|
||||
if d.Result != "ok" {
|
||||
h.Skipped = append(h.Skipped, d.Slug)
|
||||
}
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
// healthNotes lists conditions for the poller health page that are not alerts: no alert is ever raised for them.
|
||||
func (a *App) healthNotes() []api.HealthNote {
|
||||
var out []api.HealthNote
|
||||
if !a.oui.Available() {
|
||||
why := "no IEEE vendor registry is loaded, so MAC vendors are not shown"
|
||||
if !a.live.get().OUI.FetchEnabled() {
|
||||
why += " (oui.fetch is off; load one with `nmon oui import`)"
|
||||
} else if st := a.oui.Status(); st.LastError != "" {
|
||||
why += ": " + st.LastError
|
||||
}
|
||||
out = append(out, api.HealthNote{Code: "oui_unavailable", Message: why})
|
||||
}
|
||||
if h := a.l2Health(); h != nil && len(h.Skipped) > 0 {
|
||||
out = append(out, api.HealthNote{Code: "l2_devices_skipped",
|
||||
Message: fmt.Sprintf("the last MAC/ARP table cycle skipped %d device(s) that did not answer in time or failed: %s",
|
||||
len(h.Skipped), strings.Join(h.Skipped, ", "))})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -122,6 +122,7 @@ func (a alertNotifier) Notify(ctx context.Context, t alert.Transition) error {
|
||||
if t.Component != "" {
|
||||
note.Labels["component"] = t.Component
|
||||
}
|
||||
a.describePeer(ctx, ¬e, t)
|
||||
a.device(¬e, t.Device)
|
||||
if t.Iface != "" {
|
||||
note.Interface.ID, note.Interface.Name = t.Iface, t.Iface
|
||||
|
||||
277
internal/app/policies.go
Normal file
@@ -0,0 +1,277 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alert"
|
||||
"github.com/v10networks/nmon/internal/alertpolicy"
|
||||
"github.com/v10networks/nmon/internal/clock"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// policyRefresh is how often the policy table is re-read; an API write invalidates it at once.
|
||||
const policyRefresh = 30 * time.Second
|
||||
|
||||
// policyStore is what the policy engine reads.
|
||||
type policyStore interface {
|
||||
store.AlertPolicies
|
||||
store.Devices
|
||||
store.Interfaces
|
||||
store.Groups
|
||||
store.AlertDown
|
||||
}
|
||||
|
||||
// policyEngine decides where alerts fire. It caches the compiled policy set (30 s, or until an API write) and
|
||||
// the device, port and group lookups a Subject is built from, and keeps the last good answer while the store
|
||||
// fails so a database hiccup neither fires nor closes alerts.
|
||||
type policyEngine struct {
|
||||
st policyStore
|
||||
log *slog.Logger
|
||||
optIn *optInSet
|
||||
rules []alert.Rule
|
||||
|
||||
set *ttlCache[struct{}, *alertpolicy.Set]
|
||||
devices *ttlCache[string, store.Device]
|
||||
ifaces *ttlCache[string, store.Interface]
|
||||
groups *ttlCache[struct{}, map[string][]string]
|
||||
devGroups *ttlCache[string, []string]
|
||||
|
||||
clk clock.Clock
|
||||
|
||||
mu sync.Mutex
|
||||
lastSet *alertpolicy.Set
|
||||
lastDev map[string]store.Device // last good lookups, used while the store fails
|
||||
lastIf map[string]store.Interface // (a database hiccup must not close alerts)
|
||||
warned bool
|
||||
}
|
||||
|
||||
func newPolicyEngine(st policyStore, clk clock.Clock, optIn *optInSet, rules []alert.Rule, log *slog.Logger) *policyEngine {
|
||||
return &policyEngine{
|
||||
st: st, log: log, optIn: optIn, rules: rules, clk: clk,
|
||||
lastDev: map[string]store.Device{}, lastIf: map[string]store.Interface{},
|
||||
set: newTTLCache[struct{}, *alertpolicy.Set](clk, policyRefresh),
|
||||
devices: newTTLCache[string, store.Device](clk, entityTTL),
|
||||
ifaces: newTTLCache[string, store.Interface](clk, entityTTL),
|
||||
groups: newTTLCache[struct{}, map[string][]string](clk, entityTTL),
|
||||
devGroups: newTTLCache[string, []string](clk, entityTTL),
|
||||
}
|
||||
}
|
||||
|
||||
// Changed drops every cache, so a policy, group or flag write applies on the next cycle.
|
||||
func (p *policyEngine) Changed() {
|
||||
p.set.invalidate()
|
||||
p.groups.invalidate()
|
||||
p.devGroups.invalidate()
|
||||
p.ifaces.invalidate()
|
||||
p.devices.invalidate()
|
||||
}
|
||||
|
||||
// current returns the compiled policy set. While the store fails it keeps the last one that loaded.
|
||||
func (p *policyEngine) current() *alertpolicy.Set {
|
||||
set, err := p.set.get(struct{}{}, func() (*alertpolicy.Set, error) { return p.load(context.Background()) })
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if err != nil {
|
||||
p.log.Warn("load alert policies", "err", err)
|
||||
if p.lastSet == nil {
|
||||
p.lastSet, _ = alertpolicy.NewSet(nil)
|
||||
}
|
||||
return p.lastSet
|
||||
}
|
||||
p.lastSet = set
|
||||
return set
|
||||
}
|
||||
|
||||
func (p *policyEngine) load(ctx context.Context) (*alertpolicy.Set, error) {
|
||||
rows, err := p.st.ListAlertPolicies(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
policies := make([]alertpolicy.Policy, 0, len(rows))
|
||||
for _, r := range rows {
|
||||
pol, err := policyOfRow(r)
|
||||
if err != nil {
|
||||
p.log.Error("alert policy unreadable, skipped", "policy", r.Name, "err", err)
|
||||
continue
|
||||
}
|
||||
policies = append(policies, pol)
|
||||
}
|
||||
set, err := alertpolicy.NewSet(policies)
|
||||
if err != nil {
|
||||
p.log.Error("alert policy does not compile", "err", err)
|
||||
}
|
||||
p.warnIfEmpty(len(rows))
|
||||
return set, nil
|
||||
}
|
||||
|
||||
// warnIfEmpty says once that an empty policy table means silence.
|
||||
func (p *policyEngine) warnIfEmpty(n int) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if n == 0 && !p.warned {
|
||||
p.warned = true
|
||||
p.log.Warn("no alert policies: nothing will alert; add policies under Alerts > Alert policies")
|
||||
}
|
||||
}
|
||||
|
||||
func policyOfRow(r store.AlertPolicy) (alertpolicy.Policy, error) {
|
||||
scope, err := alertpolicy.ParseScope(r.Scope)
|
||||
if err != nil {
|
||||
return alertpolicy.Policy{}, err
|
||||
}
|
||||
pol := alertpolicy.Policy{ID: r.ID, Name: r.Name, Rule: r.Rule, Action: r.Action, Priority: r.Priority, Enabled: r.Enabled,
|
||||
Severity: r.Severity, Threshold: r.Threshold, Scope: scope}
|
||||
if r.ForSeconds != nil {
|
||||
d := time.Duration(*r.ForSeconds) * time.Second
|
||||
pol.For = &d
|
||||
}
|
||||
return pol, nil
|
||||
}
|
||||
|
||||
// Metric is alert.Options.Policy.
|
||||
func (p *policyEngine) Metric(e alert.Entity) (alertpolicy.Binding, bool) {
|
||||
subj, ok := p.subject(e.Device, e.Iface)
|
||||
if !ok {
|
||||
return alertpolicy.Binding{}, false
|
||||
}
|
||||
return p.current().Decide(e.Rule, subj)
|
||||
}
|
||||
|
||||
// Wants is reach.Options.Wants.
|
||||
func (p *policyEngine) Wants(rule, slug string) bool {
|
||||
subj, ok := p.subject(slug, "")
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
_, ok = p.current().Decide(rule, subj)
|
||||
return ok
|
||||
}
|
||||
|
||||
// SystemWanted reports whether a system policy binds subsystem alarms. They have no device, so only a scope of
|
||||
// all can match.
|
||||
func (p *policyEngine) SystemWanted() bool {
|
||||
_, ok := p.current().Decide(alertpolicy.RuleSystem, alertpolicy.Subject{})
|
||||
return ok
|
||||
}
|
||||
|
||||
// subject builds what policies match against; ok false means the device or port is unknown or unreadable.
|
||||
func (p *policyEngine) subject(slug, iface string) (alertpolicy.Subject, bool) {
|
||||
d, err := p.devices.get(slug, func() (store.Device, error) {
|
||||
dev, err := p.st.GetDeviceBySlug(context.Background(), slug)
|
||||
return remember(p, p.lastDev, slug, dev, err)
|
||||
})
|
||||
if err != nil {
|
||||
p.logLookup("device", slug, err)
|
||||
return alertpolicy.Subject{}, false
|
||||
}
|
||||
subj := alertpolicy.Subject{Device: deviceSubject(d)}
|
||||
if iface == "" {
|
||||
subj.Groups = p.deviceGroups(d)
|
||||
return subj, true
|
||||
}
|
||||
i, err := p.ifaces.get(iface, func() (store.Interface, error) {
|
||||
x, err := p.st.GetInterface(context.Background(), iface)
|
||||
return remember(p, p.lastIf, iface, x, err)
|
||||
})
|
||||
if err != nil {
|
||||
p.logLookup("interface", iface, err)
|
||||
return alertpolicy.Subject{}, false
|
||||
}
|
||||
port := ifaceSubject(i, p.optIn.opted(i.ID))
|
||||
subj.Iface = &port
|
||||
subj.Groups = p.ifaceGroups()[i.ID]
|
||||
return subj, true
|
||||
}
|
||||
|
||||
func (p *policyEngine) logLookup(what, key string, err error) {
|
||||
if !errors.Is(err, store.ErrNotFound) {
|
||||
p.log.Warn("alert policy lookup failed", "what", what, "key", key, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
func deviceSubject(d store.Device) alertpolicy.Device {
|
||||
return alertpolicy.Device{Slug: d.Slug, Profile: d.Profile, Role: d.Role, Site: d.Site, Kind: d.Kind}
|
||||
}
|
||||
|
||||
func ifaceSubject(i store.Interface, alertDown bool) alertpolicy.Iface {
|
||||
speed := i.IfHighSpeed
|
||||
if speed == 0 {
|
||||
speed = i.IfSpeed / 1_000_000
|
||||
}
|
||||
return alertpolicy.Iface{Type: i.IfType, Name: i.IfName, Alias: i.IfAlias, SpeedMbps: speed, AlertDown: alertDown}
|
||||
}
|
||||
|
||||
// ifaceGroups maps each interface id to the ids of the groups holding it now.
|
||||
func (p *policyEngine) ifaceGroups() map[string][]string {
|
||||
m, err := p.groups.get(struct{}{}, func() (map[string][]string, error) { return p.loadGroups(context.Background(), p.clk.Now()) })
|
||||
if err != nil {
|
||||
p.log.Warn("load group membership for alert policies", "err", err)
|
||||
return nil
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func (p *policyEngine) loadGroups(ctx context.Context, now time.Time) (map[string][]string, error) {
|
||||
members, err := p.st.GroupMembers(ctx, 0, now, time.Time{})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := map[string][]string{}
|
||||
for _, m := range members {
|
||||
if m.Covers(now) {
|
||||
out[m.InterfaceID] = append(out[m.InterfaceID], fmt.Sprint(m.GroupID))
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// deviceGroups is the union of the groups of the device's ports.
|
||||
func (p *policyEngine) deviceGroups(d store.Device) []string {
|
||||
g, err := p.devGroups.get(d.Slug, func() ([]string, error) {
|
||||
ifs, err := p.st.ListInterfaces(context.Background(), d.ID, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return unionGroups(ifs, p.ifaceGroups()), nil
|
||||
})
|
||||
if err != nil {
|
||||
p.log.Warn("load device groups for alert policies", "device", d.Slug, "err", err)
|
||||
return nil
|
||||
}
|
||||
return g
|
||||
}
|
||||
|
||||
func unionGroups(ifs []store.Interface, byIface map[string][]string) []string {
|
||||
seen := map[string]bool{}
|
||||
var out []string
|
||||
for _, i := range ifs {
|
||||
for _, g := range byIface[i.ID] {
|
||||
if !seen[g] {
|
||||
seen[g] = true
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// remember records a successful lookup and, when the store fails with anything but "not found", answers from the
|
||||
// last good one so a database hiccup neither fires nor closes alerts.
|
||||
func remember[V any](p *policyEngine, last map[string]V, key string, v *V, err error) (V, error) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if err == nil {
|
||||
last[key] = *v
|
||||
return *v, nil
|
||||
}
|
||||
if old, ok := last[key]; ok && !errors.Is(err, store.ErrNotFound) {
|
||||
return old, nil
|
||||
}
|
||||
var zero V
|
||||
return zero, err
|
||||
}
|
||||
142
internal/app/policies_test.go
Normal file
@@ -0,0 +1,142 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alert"
|
||||
"github.com/v10networks/nmon/internal/alertpolicy"
|
||||
"github.com/v10networks/nmon/internal/clock"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
"github.com/v10networks/nmon/internal/testutil"
|
||||
"github.com/v10networks/nmon/internal/ulid"
|
||||
)
|
||||
|
||||
type policyEnv struct {
|
||||
t *testing.T
|
||||
st store.Store
|
||||
clk *clock.Fake
|
||||
pe *policyEngine
|
||||
dev *store.Device
|
||||
}
|
||||
|
||||
func newPolicyEnv(t *testing.T) *policyEnv {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
clk := clock.NewFake(time.Date(2026, 10, 23, 12, 0, 0, 0, time.UTC))
|
||||
st := testutil.OpenStore(t, clk)
|
||||
d := &store.Device{Name: "sw1", Kind: store.KindNetwork, Role: "access", Site: "hq", Enabled: true}
|
||||
if err := st.CreateDevice(ctx, d); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e := &policyEnv{t: t, st: st, clk: clk, dev: d}
|
||||
e.pe = newPolicyEngine(st, clk, newOptInSet(st, clk, entityTTL, testLogger()), alert.DefaultRules(), testLogger())
|
||||
return e
|
||||
}
|
||||
|
||||
func (e *policyEnv) iface(idx int64, name, alias string, typ int64) string {
|
||||
e.t.Helper()
|
||||
id := ulid.New(e.clk.Now())
|
||||
i := store.Interface{ID: id, DeviceID: e.dev.ID, IfIndex: idx, IfName: name, IfAlias: alias, IfType: typ, AdminStatus: 1, OperStatus: 1}
|
||||
if err := e.st.UpsertInterfaces(context.Background(), e.dev.ID, []store.Interface{i}); err != nil {
|
||||
e.t.Fatal(err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func (e *policyEnv) policy(name, rule, action string, prio int, scope alertpolicy.Scope) {
|
||||
e.t.Helper()
|
||||
p := &store.AlertPolicy{Name: name, Rule: rule, Action: action, Priority: prio, Enabled: true, Scope: alertpolicy.FormatScope(scope)}
|
||||
if err := e.st.CreateAlertPolicy(context.Background(), p); err != nil {
|
||||
e.t.Fatal(err)
|
||||
}
|
||||
e.pe.Changed()
|
||||
}
|
||||
|
||||
func TestNoPoliciesMeansNothingAlertsAnywhere(t *testing.T) {
|
||||
e := newPolicyEnv(t)
|
||||
id := e.iface(1, "e1", "", 6)
|
||||
if _, ok := e.pe.Metric(alert.Entity{Rule: "interface_errors", Device: e.dev.Slug, Iface: id}); ok {
|
||||
t.Fatal("metric rule bound without policies")
|
||||
}
|
||||
if e.pe.Wants("device_down", e.dev.Slug) || e.pe.Wants("device_warning", e.dev.Slug) {
|
||||
t.Fatal("reachability wanted without policies")
|
||||
}
|
||||
if e.pe.SystemWanted() {
|
||||
t.Fatal("system alarms wanted without policies")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPoliciesDecideByTypeAliasGroupAndBellAndApplyAfterChanged(t *testing.T) {
|
||||
e := newPolicyEnv(t)
|
||||
fiber := e.iface(1, "e1", "Core fiber", 6)
|
||||
wifi := e.iface(2, "w1", "Core radio", 71)
|
||||
e.policy("fiber", "interface_errors", alertpolicy.ActionAlert, 0,
|
||||
alertpolicy.Scope{Interfaces: alertpolicy.IfaceMatch{IfTypes: []int64{6}, IfAlias: "(?i)fib|core"}})
|
||||
e.policy("wifi", "interface_errors", alertpolicy.ActionIgnore, 10, alertpolicy.Scope{Interfaces: alertpolicy.IfaceMatch{IfTypes: []int64{71}}})
|
||||
ent := func(id string) alert.Entity {
|
||||
return alert.Entity{Rule: "interface_errors", Device: e.dev.Slug, Iface: id}
|
||||
}
|
||||
if _, ok := e.pe.Metric(ent(fiber)); !ok {
|
||||
t.Fatal("fiber port not bound")
|
||||
}
|
||||
if _, ok := e.pe.Metric(ent(wifi)); ok {
|
||||
t.Fatal("wifi port bound")
|
||||
}
|
||||
|
||||
// A group matcher and the bell matcher follow live changes once Changed is called.
|
||||
g := &store.Group{Name: "vip", Kind: store.GroupCustom}
|
||||
if err := e.st.CreateGroup(context.Background(), g); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
yes := true
|
||||
e.policy("vip", "interface_down", alertpolicy.ActionAlert, 0, alertpolicy.Scope{Groups: []string{strconv.FormatInt(g.ID, 10)}})
|
||||
e.policy("bell", "interface_down", alertpolicy.ActionAlert, 0,
|
||||
alertpolicy.Scope{Interfaces: alertpolicy.IfaceMatch{AlertDown: &yes}})
|
||||
down := alert.Entity{Rule: "interface_down", Device: e.dev.Slug, Iface: wifi}
|
||||
if _, ok := e.pe.Metric(down); ok {
|
||||
t.Fatal("bound before membership or bell")
|
||||
}
|
||||
if err := e.st.AddGroupMember(context.Background(), store.GroupMember{GroupID: g.ID, InterfaceID: wifi, Direction: store.DirToward}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.pe.Changed()
|
||||
if _, ok := e.pe.Metric(down); !ok {
|
||||
t.Fatal("group member not bound")
|
||||
}
|
||||
if !e.pe.Wants("interface_down", e.dev.Slug) {
|
||||
t.Fatal("a device with a member port should match a group policy")
|
||||
}
|
||||
if err := e.st.SetInterfaceAlertDown(context.Background(), []string{fiber}, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.pe.optIn.invalidate()
|
||||
e.pe.Changed()
|
||||
if _, ok := e.pe.Metric(alert.Entity{Rule: "interface_down", Device: e.dev.Slug, Iface: fiber}); !ok {
|
||||
t.Fatal("belled port not bound")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSystemPolicyAndPreview(t *testing.T) {
|
||||
e := newPolicyEnv(t)
|
||||
e.iface(1, "e1", "uplink", 6)
|
||||
e.iface(2, "w1", "", 71)
|
||||
e.policy("sys", alertpolicy.RuleSystem, alertpolicy.ActionAlert, 0, alertpolicy.Scope{All: true})
|
||||
if !e.pe.SystemWanted() {
|
||||
t.Fatal("system policy ignored")
|
||||
}
|
||||
p, err := e.pe.Preview(context.Background(), "interface_errors",
|
||||
alertpolicy.Scope{Interfaces: alertpolicy.IfaceMatch{IfTypes: []int64{6}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if p.Interfaces.Count != 1 || p.Devices.Count != 1 || p.Interfaces.Samples[0].IfAlias != "uplink" {
|
||||
t.Fatalf("preview = %+v", p)
|
||||
}
|
||||
d, err := e.pe.Preview(context.Background(), "device_down", alertpolicy.Scope{Devices: alertpolicy.DeviceMatch{Sites: []string{"nowhere"}}})
|
||||
if err != nil || d.Devices.Count != 0 {
|
||||
t.Fatalf("device preview = %+v, %v", d, err)
|
||||
}
|
||||
}
|
||||
145
internal/app/policypreview.go
Normal file
@@ -0,0 +1,145 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/v10networks/nmon/internal/alertpolicy"
|
||||
"github.com/v10networks/nmon/internal/api"
|
||||
"github.com/v10networks/nmon/internal/store"
|
||||
)
|
||||
|
||||
// previewSamples is how many covered devices and ports a preview names.
|
||||
const previewSamples = 20
|
||||
|
||||
// Rules implements api.AlertPolicyHost: the metric rules plus the reachability and system rules.
|
||||
func (p *policyEngine) Rules() []api.AlertRuleInfo {
|
||||
out := make([]api.AlertRuleInfo, 0, len(p.rules)+3)
|
||||
for _, r := range p.rules {
|
||||
if r.Disabled {
|
||||
continue
|
||||
}
|
||||
thr := r.Threshold
|
||||
out = append(out, api.AlertRuleInfo{Name: r.Name, Level: r.Level(), Op: r.Op, Threshold: &thr, Severity: r.Severity})
|
||||
}
|
||||
return append(out,
|
||||
api.AlertRuleInfo{Name: alertpolicy.RuleDeviceDown, Level: alertpolicy.LevelDevice, Severity: "critical"},
|
||||
api.AlertRuleInfo{Name: alertpolicy.RuleDeviceWarning, Level: alertpolicy.LevelDevice, Severity: "warning"},
|
||||
api.AlertRuleInfo{Name: alertpolicy.RuleSystem, Level: alertpolicy.LevelSystem, Severity: "warning"})
|
||||
}
|
||||
|
||||
func (p *policyEngine) level(rule string) (string, bool) {
|
||||
for _, r := range p.Rules() {
|
||||
if r.Name == rule {
|
||||
return r.Level, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// Preview implements api.AlertPolicyHost. It reads the inventory afresh, so the counts match what the policy
|
||||
// would cover on the next cycle.
|
||||
func (p *policyEngine) Preview(ctx context.Context, rule string, scope alertpolicy.Scope) (api.PolicyPreview, error) {
|
||||
level, ok := p.level(rule)
|
||||
if !ok {
|
||||
return api.PolicyPreview{}, fmt.Errorf("%w: rule %q", store.ErrNotFound, rule)
|
||||
}
|
||||
m, err := scope.Compile()
|
||||
if err != nil {
|
||||
return api.PolicyPreview{}, err
|
||||
}
|
||||
out := api.PolicyPreview{
|
||||
Devices: api.PreviewDevices{Samples: []api.DeviceRef{}},
|
||||
Interfaces: api.PreviewInterfaces{Samples: []api.PreviewInterface{}},
|
||||
}
|
||||
if level == alertpolicy.LevelSystem {
|
||||
return out, nil
|
||||
}
|
||||
byIface, err := p.loadGroups(ctx, p.clk.Now())
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
down, err := p.st.InterfacesAlertDown(ctx)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
devs, err := p.activeDevices(ctx)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for _, d := range devs {
|
||||
ifs, err := p.st.ListInterfaces(ctx, d.ID, false)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
pv := previewOf{m: m, level: level, byIface: byIface, down: down}
|
||||
pv.device(&out, d, ifs)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (p *policyEngine) activeDevices(ctx context.Context) ([]store.Device, error) {
|
||||
var all []store.Device
|
||||
for off := int64(0); ; off += store.MaxLimit {
|
||||
page, total, err := p.st.ListDevices(ctx, store.DeviceFilter{}, store.Page{Offset: off, Limit: store.MaxLimit})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, d := range page {
|
||||
if d.RetiredAt == nil && d.PurgedAt == nil && d.Enabled {
|
||||
all = append(all, d)
|
||||
}
|
||||
}
|
||||
if len(page) == 0 || off+int64(len(page)) >= total {
|
||||
return all, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// previewOf evaluates one scope over devices and ports.
|
||||
type previewOf struct {
|
||||
m *alertpolicy.Matcher
|
||||
level string
|
||||
byIface map[string][]string
|
||||
down map[string]bool
|
||||
}
|
||||
|
||||
// device adds a device (and, for an interface rule, its covered ports) to the preview.
|
||||
func (pv previewOf) device(out *api.PolicyPreview, d store.Device, ifs []store.Interface) {
|
||||
base := alertpolicy.Subject{Device: deviceSubject(d)}
|
||||
if pv.level != alertpolicy.LevelInterface {
|
||||
base.Groups = unionGroups(ifs, pv.byIface)
|
||||
if pv.m.Match(base) {
|
||||
addDevice(out, d)
|
||||
}
|
||||
return
|
||||
}
|
||||
covered := false
|
||||
for _, i := range ifs {
|
||||
if i.Excluded {
|
||||
continue
|
||||
}
|
||||
port := ifaceSubject(i, pv.down[i.ID])
|
||||
s := base
|
||||
s.Iface, s.Groups = &port, pv.byIface[i.ID]
|
||||
if !pv.m.Match(s) {
|
||||
continue
|
||||
}
|
||||
covered = true
|
||||
out.Interfaces.Count++
|
||||
if len(out.Interfaces.Samples) < previewSamples {
|
||||
out.Interfaces.Samples = append(out.Interfaces.Samples,
|
||||
api.PreviewInterface{ID: i.ID, Device: d.Slug, IfName: i.IfName, IfAlias: i.IfAlias})
|
||||
}
|
||||
}
|
||||
if covered {
|
||||
addDevice(out, d)
|
||||
}
|
||||
}
|
||||
|
||||
func addDevice(out *api.PolicyPreview, d store.Device) {
|
||||
out.Devices.Count++
|
||||
if len(out.Devices.Samples) < previewSamples {
|
||||
out.Devices.Samples = append(out.Devices.Samples, api.DeviceRef{Slug: d.Slug, Name: d.Name})
|
||||
}
|
||||
}
|
||||