Monitoring and observability

A monitoring stack with Prometheus, Grafana and the ELK stack, for real-time application and infrastructure monitoring.

Without shared signals, incidents begin with people asking whether something is really wrong. The first job of observability is to shorten that question.

What the stack must answer quickly.

• –Are users affected right now, and how many?
• –Which service and which change is responsible?
• –Is the budget being used faster than it can afford?

Prometheus scrapes metrics and evaluates alert rules. Logs flow separately into Elasticsearch through a shipper. Grafana brings both together, so one view can move from a metric spike to the matching logs.

Why this split of tools.

• –Metrics for detection because they are cheap to keep and fast to query.
• –Logs for diagnosis, retained for less time because they cost more.
• –Alerts on symptoms, with dashboards for causes.

Start from what users do, define SLIs for those journeys, then add alerts. Instrument the top services first, and remove any alert that does not lead to an action.

Dashboards and alert rules live in version control and deploy like any other change, so they are reviewed, reproducible and restorable.

Logs get scrubbed of secrets and personal data before storage. Dashboards sit behind SSO, and metric endpoints are not exposed outside the cluster.

The monitoring must fail independently of what it watches. An external synthetic check and a dead man's switch alert reveal when the alerting path itself has stopped working.

What commonly goes wrong.

• –Alert fatigue: page only on budget burn, as shown in the Error budget lab.
• –Cardinality explosions: keep unbounded values out of labels.
• –Log costs: apply retention tiers and drop what no one queries.

No verified figures are published for this project. The numbers that would prove it worked: time to detect, time to restore, pages per on-call shift, and the share of pages that needed action.