Server Cluster Monitoring

The monitoring module provides a comprehensive observability stack for the server cluster using Prometheus (metrics), Loki (logs), Grafana (visualization), and Grafana Alloy for authenticated OTLP ingestion. All components are configured as reusable NixOS modules with automatic cross-host discovery.

Overview

The system consists of three layers:

Exporters (run on all servers)
- node_exporter for system-level metrics (CPU, memory, disk, network, per-process stats)
- Grafana Alloy for shipping journald logs and Caddy access logs to Loki
- Caddy access logs are parsed as JSON at ingest time so detected_level, logger, and status are available in Loki
- Ingest-time log parsing for journal stdout entries and Caddy access logs to infer detected_level and normalize common timestamp formats
- Application-specific exporters (Caddy, PostgreSQL, Redis) enabled automatically
- fail2ban exporter available on the IO primary host (when fail2ban is enabled)
Collectors (run on the monitoring primary host)
- Prometheus for metrics aggregation with 90-day retention
- Loki for log aggregation with 90-day retention
- Alertmanager for alert routing and notifications
- OTLP/HTTP ingestion on otlp.<domain> with bearer-token authentication
Visualization (runs on the monitoring primary host)
- Grafana with provisioned datasources and dashboards
- Native Kanidm OAuth2 authentication

Architecture

┌─────────────────────────────────────────────────────┐
│                    nixmon (Monitoring Primary)      │
│  ┌──────────┐  ┌──────┐  ┌─────────┐  ┌──────────┐  │
│  │Prometheus│  │ Loki │  │ Grafana │  │Alertmgr  │  │
│  │  :9090   │  │:3100 │  │  :3000  │  │  :9093   │  │
│  └────┬──┬──┘  └──┬───┘  └─────────┘  └────┬─────┘  │
│       │  │        │                        │        │
│  ┌────┘  │   ┌────┘        ┌───────────────┘        │
│  │ scrape│   │ push        │ webhooks               │
├──┼───────┼───┼─────────────┼────────────────────────┤
│  ▼       ▼   ▼             ▼                        │
│  All servers:          Home Assistant / Nextcloud   │
│  - node_exporter :9100                              │
│  - alloy → Loki                                     │
│  - OTLP/HTTP → Alloy :4318                          │
│  - caddy metrics :2019 (if proxy configured)        │
│  - fail2ban_exporter :9191 (if fail2ban enabled)    │
│  - postgres_exporter :9187 (if postgres configured) │
│  - redis_exporter :9121 (if redis configured)       │
│  - pve_exporter :9221 (nixmon only, Proxmox API)    │
└─────────────────────────────────────────────────────┘

Configuration

Enabling Monitoring

Monitoring is enabled by default on all servers (server.monitoring.enable = true). The monitoring primary host is configured via the allocations.server.monitoringPrimaryHost option, currently set to nixmon.

Options

`server.monitoring.collector.alerting.enable`


Type	`boolean`
Default	`cfg.enable`
Example	`true`

Whether to enable Alertmanager and alert rules.

`server.monitoring.collector.alerting.homeAssistant.enable`


Type	`boolean`
Default	`false`
Example	`true`

Whether to enable Home Assistant webhook alerting.

`server.monitoring.collector.alerting.nextcloudTalk.enable`


Type	`boolean`
Default	`false`
Example	`true`

Whether to enable Nextcloud Talk webhook alerting.

`server.monitoring.collector.enable`


Type	`boolean`
Default	`thisIsMonitoringPrimaryHost && cfg.enable`
Example	`true`

Whether to enable monitoring collector services (Prometheus, Loki, Grafana).

`server.monitoring.collector.grafana.kanidm.enable`


Type	`boolean`
Default	`true`
Example	`true`

Whether to enable Kanidm OAuth2 authentication for Grafana.

`server.monitoring.collector.otlp.bearerTokenSecret`


Type	`string`
Default	`"MONITORING/OLTP/BEARER_TOKEN"`

SOPS secret path used as the bearer token for OTLP/HTTP ingestion.

`server.monitoring.collector.otlp.enable`


Type	`boolean`
Default	`isThisMonitoringPrimaryHost && cfg.enable`
Example	`true`

Whether to enable OTLP/HTTP ingestion via Grafana Alloy.

`server.monitoring.collector.otlp.port`


Type	`signed integer`
Default	`4318`

Port for the OTLP/HTTP ingestion endpoint.

`server.monitoring.collector.otlp.subdomain`


Type	`string`
Default	`"otlp"`

Subdomain used for the OTLP/HTTP ingestion endpoint.

`server.monitoring.collector.proxmox.enable`


Type	`boolean`
Default	`isThisMonitoringPrimaryHost && cfg.enable`
Example	`true`

Whether to enable Proxmox VE metrics collection.

`server.monitoring.enable`


Type	`boolean`
Default	`true`
Example	`true`

Whether to enable monitoring for this server.

`server.monitoring.exporters.caddy.enable`


Type	`boolean`
Default	`cfg.enable && config.services.caddy.enable`
Example	`true`

Whether to enable Caddy metrics exporter.

`server.monitoring.exporters.fail2ban.enable`


Type	`boolean`
Default	`cfg.enable && isThisIOPrimaryHost && config.server.fail2ban.enable`
Example	`true`

Whether to enable fail2ban metrics exporter.

`server.monitoring.exporters.node.enable`


Type	`boolean`
Default	`cfg.enable`
Example	`true`

Whether to enable node_exporter for system-level metrics.

`server.monitoring.exporters.postgres.enable`


Type	`boolean`
Default	`cfg.enable && thisIsIOPrimaryHost && hasPostgresDatabases`
Example	`true`

Whether to enable PostgreSQL exporter.

`server.monitoring.exporters.process.enable`


Type	`boolean`
Default	`cfg.enable`
Example	`true`

Whether to enable Process exporter for monitoring specific processes.

`server.monitoring.exporters.redis.enable`


Type	`boolean`
Default	`cfg.enable && thisIsIOPrimaryHost && hasRedisInstances`
Example	`true`

Whether to enable Redis exporter.

`server.monitoring.logs.enable`


Type	`boolean`
Default	`cfg.enable`
Example	`true`

Whether to enable Alloy log shipping.

`server.monitoring.logs.extraConfiguration`


Type	`strings concatenated with "\n"`
Default	`""`

Additional configuration for the alloy log processor. This is useful for adding custom Loki stages, relabeling rules, or write targets.

Note that the default configuration for processing the system journal is always included and does not need to be specified here.

`server.monitoring.retention.logs`


Type	`string`
Default	`"90d"`

Loki log retention period.

`server.monitoring.retention.metrics`


Type	`string`
Default	`"90d"`

Prometheus TSDB retention period.

`server.monitoring.scrapeConfigs`


Type	`attribute set of (submodule)`
Default	`{ }`

Declarative scrape configs for services running on this host. These are collected by the monitoring primary host and converted into Prometheus scrape configurations.

`server.monitoring.scrapeConfigs.<name>.bearer_token_secret`


Type	`null or string`
Default	`null`

SOPS secret path for bearer token authentication. When set, the secret will be created on the monitoring primary host.

`server.monitoring.scrapeConfigs.<name>.host`


Type	`string`
Default	`config.host.name`

Host to scrape metrics from.

`server.monitoring.scrapeConfigs.<name>.job_name`


Type	`string`
Default	`"‹name›"`

Prometheus job name for this scrape target.

`server.monitoring.scrapeConfigs.<name>.metrics_path`


Type	`string`
Default	`"/metrics"`

HTTP path to the metrics endpoint.

`server.monitoring.scrapeConfigs.<name>.port`


Type	`signed integer`

Port the metrics endpoint listens on.

`server.monitoring.scrapeConfigs.<name>.scheme`


Type	`one of "http", "https"`
Default	`"http"`

URL scheme for scraping.

Auto-Detection

The module automatically detects and enables exporters based on host role:

Caddy exporter: Enabled when server.proxy.virtualHosts is non-empty
PostgreSQL exporter: Enabled on the IO primary host when postgres databases are configured
Redis exporter: Enabled on the IO primary host when redis instances are configured
Caddy access logs: Enabled when Caddy metrics/logs are enabled; each access log file under /var/log/caddy-access-* is shipped to Loki and parsed as JSON
node_exporter process collector: Enabled on all servers via the processes collector to expose per-process stats
fail2ban exporter: Enabled on the IO primary host when fail2ban intrusion detection is enabled
Collector services: Enabled only on the monitoring primary host

Secrets

The monitoring module requires the following secrets in hosts/server/nixmon/secrets.yaml:

MONITORING:
  OLTP:
    BEARER_TOKEN: <random-secret-key>
  GRAFANA:
    SECRET_KEY: <random-secret-key>
    OAUTH_SECRET: <kanidm-oauth2-secret>
  HOME_ASSISTANT:
    WEBHOOK_URL: <ha-webhook-url>
  NEXTCLOUD_TALK:
    WEBHOOK_URL: <nc-talk-webhook-url>
PROXMOX:
  USER: <proxmox-user-at-realm>
  TOKEN_ID: <proxmox-token-name>
  TOKEN_SECRET: <proxmox-token-secret>

Generating Secrets

Generating secure random secrets can be done with the following command:

cat /dev/urandom | tr -dc 'A-Za-z0-9' | head -c 48

The MONITORING/GRAFANA/OAUTH_SECRET must match the value in hosts/server/nixcloud/secrets.yaml under KANIDM/OAUTH2/GRAFANA_SECRET (the Kanidm provisioning side).

Caddy Virtual Hosts

The module configures four virtual hosts on nixmon:

Service	Subdomain	Access
Grafana	`grafana.<domain>`	Public
OTLP	`otlp.<domain>`	Public, bearer token required
Prometheus	`prometheus.<domain>`	LAN
Loki	`loki.<domain>`	LAN

Grafana remains protected by the existing Kanidm-backed login flow. The OTLP ingestion endpoint is intended for machine-to-machine clients and requires an Authorization: Bearer <token> header on every request. The exposed OTLP/HTTP paths are the standard /v1/metrics and /v1/logs endpoints.

These are defined in hosts/server/nixmon/default.nix and collected by the IO primary host’s Caddy configuration.

Alert Rules

The following alerts are configured by default:

Alert	Condition	Severity
`HostDown`	`up{job="node"} == 0` for 2 minutes	Critical
`DiskSpaceCritical`	Root filesystem < 10% free for 5 minutes	Critical
`HighCPUUsage`	CPU usage > 90% for 5 minutes	Warning
`HighMemoryUsage`	Memory usage > 90% for 5 minutes	Warning
`ServiceDown`	`up{job!="node"} == 0` for 2 minutes	Critical

Alerts are routed to:

Home Assistant: All critical and warning alerts via webhook (requires collector.alerting.homeAssistant.enable = true)
Nextcloud Talk: Critical alerts only via webhook (requires collector.alerting.nextcloudTalk.enable = true)

Module Structure

modules/nixos/server/monitoring/
├── default.nix              # Entry point, imports sub-modules
├── options.nix              # All server.monitoring.* options
├── collector/
│   ├── default.nix          # Imports collector sub-modules
│   ├── prometheus.nix       # Prometheus server + scrape targets
│   ├── loki.nix             # Loki server + storage config
│   ├── grafana.nix          # Grafana + Kanidm OAuth2
│   ├── otlp.nix             # OTLP ingestion
│   ├── alerting.nix         # Alertmanager + alert rules
│   └── dashboards.nix       # Dashboard provisioning
├── exporters/
│   ├── default.nix          # Imports exporter sub-modules
│   ├── node.nix             # node_exporter
│   ├── caddy.nix            # Caddy metrics
│   ├── postgres.nix         # PostgreSQL exporter
│   ├── redis.nix            # Redis exporter
│   └── fail2ban.nix         # fail2ban metrics exporter
├── logs/
│   └── alloy.nix            # Alloy log shipping
└── integrations/
    └── proxmox.nix          # PVE exporter for Proxmox API

Troubleshooting

Checking Service Status

On the monitoring host (nixmon):

systemctl status prometheus.service
systemctl status loki.service
systemctl status grafana.service
systemctl status prometheus-alertmanager.service
systemctl status prometheus-pve-exporter.service

On any server:

systemctl status prometheus-node-exporter.service
systemctl status prometheus-fail2ban-exporter.service
systemctl status alloy.service

Verifying Metrics Collection

Check Prometheus targets are up:

curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {instance: .labels.instance, health: .health}'

Verifying Log Collection

Alloy applies ingest-time parsing for journal stdout logs and Caddy access logs before forwarding to Loki:

Caddy access logs are read as JSON, not plain text
Legacy timestamps in form YYYY/MM/DD HH:MM:SS are parsed and used as event timestamps
ISO-8601 timestamps with a log level prefix are parsed and normalized
detected_level defaults to info when the source log line does not provide one
Caddy JSON fields level, ts, logger, and status are extracted into Loki labels and timestamps
Caddy access logs are read from /var/log/caddy-access-*.log and use the timestamp and level prefix in each line when present

node_exporter also enables the processes collector, which exposes per-process metrics such as CPU and memory usage for running processes.

Check Alloy is shipping logs:

journalctl -u alloy.service -f

Query Loki directly:

curl -s 'http://localhost:3100/loki/api/v1/labels' | jq

Common Issues

Grafana OAuth login fails:

Verify GRAFANA_OAUTH_SECRET in nixmon matches KANIDM/OAUTH2/GRAFANA_SECRET in nixcloud
Check Kanidm provisioning has the grafana OAuth2 client configured
Verify DNS resolves auth.<domain> correctly

Prometheus targets showing as down:

Check firewall rules allow traffic on exporter ports from the monitoring host
Verify the exporter service is running on the target host
Check network connectivity between nixmon and the target host

Proxmox metrics missing:

Verify proxmox/token_id and proxmox/token_secret are valid
Check PVE API is accessible from nixmon: curl -k https://pve.<domain>/api2/json
Review PVE exporter logs: journalctl -u prometheus-pve-exporter.service

Keyboard shortcuts

RacciDev