Audience: Contributor / operator
Optional OSS Grafana stack for ops signals (up / RED / logs / alerts). It is not Insights, Server status, or product analytics. Product UIs stay on 8010 / 8011 / 8012. Grafana is the only new published UI (8005 by default).
In-app Server status, Insights, and Usage stay in LokAkrt — they are not Grafana.
Runbook in the monorepo: server/observability/README.md. Product events vs ops plane: tools/TELEMETRY.md.
lokakrt_network, lokakrt_ai_network, lokakrt_store_network).server/observability/.env.example → server/observability/.env. Set GF_SECURITY_ADMIN_PASSWORD, scrape tokens (MAIN_METRICS_SCRAPE_TOKEN / STORE_METRICS_SCRAPE_TOKEN / AI_METRICS_SCRAPE_TOKEN), and blackbox LAN URLs.docker compose -f server/observability/docker-compose.yml --env-file server/observability/.env up -d
http://<host>:8005 (or GRAFANA_PORT). Login: GF_SECURITY_ADMIN_USER (default admin) + GF_SECURITY_ADMIN_PASSWORD.Grafana home is the Overview dashboard (status-first). Apps stay up if this stack is down.
MariaDB dashboards need the optional mariadb compose profile and MYSQLD_* DSNs in .env — see the repo README.
| Question | Where to look |
|---|---|
| Is anything wrong? | Top stat row (scrape up, blackbox probe, HTTP 5xx burn) — red/yellow means yes |
| Which app? | Values are labeled main / store / ai |
| What is wrong? | Firing alerts panel under the stats (title + summary name the app and cause) |
| Detail / traffic | RED charts below (rate, 5xx, p95, heap). AI also has AI agent failures |
| Logs for that app | Dashboard link Logs, or Explore → Loki {service="main\|store\|ai"} |
| Symptom | Likely cause |
|---|---|
| Scrape down | Missing/mismatched security.metricsScrapeToken in that app’s Settings → Security vs Alloy env (MAIN_METRICS_SCRAPE_TOKEN / STORE_METRICS_SCRAPE_TOKEN / AI_METRICS_SCRAPE_TOKEN). security.restrictMetrics defaults true. |
| Blackbox fail | App process down, or BLACKBOX_*_URL uses localhost / Docker DNS instead of a LAN IP |
| HTTP 5xx burn | Application errors — open Logs for that service |
| AI agent failures | Ollama/agent path (reason= on the alert) — AI may still scrape “up” |
| Restart loop | Map container image/name to main/store/ai, then check that app’s logs |
| Disk high | Host filesystem — affects all apps |
Grafana unified alerting (no Alertmanager / Slack in v1):
service labeled)http_request_duration_seconds_* (main, store, and ai after AI rebuild with Express RED)lokakrt_agent_failures_total by reasonNo Loki error-rate alerts in v1 (too noisy). Correlate by service + time window (no trace_id until v2).
AI emits the same Express RED histogram as main/store (http_request_duration_seconds_*). Rebuild the AI image/module after pulling that change so Overview 5xx/p95 and the 5xx alert cover service=ai.
node server/observability/check-observability.mjs
node --test server/observability/probe-url.spec.mjs server/observability/observability-contract.spec.mjs
reasonaiTempo / trace_id, Slack/email contact points, in-app ops health page, Loki error-rate alerts, deep DB/setup/inference probes beyond blackbox HTTP 200.