ch-analyzer — ClickHouse monitoring in a single Go binary

The problem

ClickHouse fails quietly, then all at once.

Parts pile up. A materialized view silently falls behind. S3 latency creeps. By the time queries time out, the root cause is buried three layers deep in system.*. ch-analyzer watches all of it, continuously.

// too-many-parts

Merges can't keep up

Small inserts outpace background merges, active parts climb past thresholds, and inserts start throwing TOO_MANY_PARTS before anyone notices.

ch-analyzer — tracks active parts per table & partition, merge throughput, and the small-insert anti-pattern per minute.

// mv-lag

Materialized views drift

A chained MV breaks or bloats. Downstream tables serve stale data for hours, and nothing in the query log looks wrong.

ch-analyzer — measures MV lag, failures, bloat ratio, and detects chained-MV breakage across the DAG.

// replica-delay

Replicas fall behind

Absolute delay grows, a replica flips read-only, the replication queue backs up — and reads quietly start returning inconsistent results.

ch-analyzer — per-table replica status, absolute delay, read-only detection, and queue-backlog alerts.

// no-central-tsdb

Monitoring the monitor

Standing up Prometheus + a TSDB + exporters + Grafana just to watch ClickHouse means one more fragile system to operate.

ch-analyzer — stores its metrics back into ClickHouse itself. One binary. Each node owns its own data.

How it works

One reconcile loop, every minute.

No agents on your nodes. ch-analyzer connects as a read-only monitoring user, runs every collector in parallel per instance, then reconciles alert state in a single writer.

01 — Collect

Poll in parallel

Every instance, every collector, concurrently each cycle. A circuit breaker skips nodes that fail 5× in a row.

→

02 — Analyze

Correlate signals

Std-dev baselines auto-learn per metric. Cross-collector rules catch OOM risk, merge overload, S3 contention.

→

03 — Reconcile

Dedup & layer

Single-writer diff against DB state. Snooze, ack, inhibition, maintenance and escalation applied on top.

→

04 — Notify

Route & store

Slack, PagerDuty, webhook by severity. Metrics & alerts written back into ch_analyzer.* on each node.

What you get

Not just metrics — an operator's console.

Detection is the easy half. ch-analyzer is built around what you do next: triage, investigate, remediate, and communicate.

Alerting engine that thinks like an on-call

20+ categories with fire-count tracking and first-seen timestamps — plus the logic to keep noise down.

Inhibition — a CPU spike suppresses the query-slowdown alerts it caused
Escalation — sustained warn promotes to critical after N clean-of-recovery polls
Snooze with expiry + acknowledge-with-reason, all written to an audit log
Maintenance windows silence a single instance or everything, globally

AI Analyzer

A context-aware chat that sees the view you're on and uses agentic tool calls to dig through live cluster state.

Slack Socket-Mode app

Pinned live-health message, inline Resolve / Snooze buttons, and /status · /alerts · /runcheck. No public endpoint.

Cost Explorer

Storage broken down by table and tier with monthly cost estimates — find the expensive tables before the bill does.

Query Analyzer & Table Scanner

Patterns, live queries, users, anti-patterns, failures, S3 latency and merges in one view — plus multi-instance table search by pattern, engine, or size.

Compare & Advisor

Diff DDL, settings, metrics and query patterns across instances. Per-category remediation suggestions, fully customizable via a suggestions.yaml.

Architecture

No central control plane.

Each ClickHouse node is the source of truth for its own observability data. ch-analyzer reads system.*, writes ch_analyzer.* right back to the same node, and fans everything out.

Sources

Your CH instances

system.* read · ch_analyzer.* written per node. Each node owns its schema.

→

Engine

ch-analyzer binary

~20 collectors → analyzer → single-writer reconcile loop → store.

→

Surfaces

Everywhere you look

React SPA · Slack socket app · Prometheus · PagerDuty · webhook.

Six tables on each node — metrics · alerts · health_snapshots · digest_snapshots · query_samples · audit_log. No single bottleneck, no cross-instance writes.

20+

parallel collectors

Go binary, embedded SPA

tables, zero external TSDB

default poll interval

Quick start

From clone to dashboard in four steps.

Create a read-only monitoring user, load the schema, point the config at your instances, and run. Binary, Docker, or Kubernetes.

setup.sh

# 1 · create the read-only monitoring user on each node
CREATE USER IF NOT EXISTS monitoring IDENTIFIED BY '<secret>';
GRANT SELECT ON system.* TO monitoring;
GRANT SELECT, INSERT ON ch_analyzer.* TO monitoring;

# 2 · load the schema (tables are NOT auto-created)
clickhouse-client --host your-host --secure \
  --multiquery < schema.sql

# 3 · configure & 4 · run as a service
make build-linux
sudo ./setup.sh          # user + schema + binary + systemd
sudo systemctl enable --now ch-analyzer

docker

# mount your config, expose the dashboard
docker run -d \
  -v /etc/ch-analyzer/config.yaml:/etc/ch-analyzer/config.yaml:ro \
  -p 8080:8080 \
  ch-analyzer:latest

# dashboard → http://localhost:8080

kubectl

# edit credentials, then apply
vi deploy/k8s.yaml
kubectl apply -f deploy/k8s.yaml

kubectl -n ch-analyzer \
  port-forward svc/ch-analyzer 8080:8080

# set k8s.enabled: true in config for OOMKill + restart tracking

local dev

# terminal 1 — frontend hot reload
cd web/frontend && npm run dev

# terminal 2 — Go backend
make build-go
./bin/ch-analyzer -config configs/ch-analyzer.yaml

# dev dashboard → http://localhost:5173

Requirements

Go1.23+

Node (frontend build)22+

ClickHouse22.x+

clickhouse-clientsetup.sh only

External TSDBnone

Public HTTP for Slacknone

Durations use Go syntax — "1m", "5m", "8760h". Every threshold in the table above is tunable in config.yaml.

ClickHouse tells you what broke.
ch-analyzer tells you why.