Prometheus is a great monitoring and alerting system. At Spreaker (self promotion – we’re hiring!) we’re slowly migrating to Prometheus, and thus a correct understanding in how it works is very important.
Today, me and my colleague Rocco were experimenting on the delays introduced by our Prometheus setup. In particular, it was our intention to measure how much time it takes to get an event, occured on a monitored target, notified through our notification channels. This article sums up what we’ve learned.
Scraping, Evaluation and Alerting
Prometheus scrape metrics from monitored targets at regular intervals, defined by the
). The scrape interval can be configured globally, and then overriden per job. Scraped metrics are then stored persistently on its local storage.
Prometheus has another loop, whose clock is independent from the scraping one, that evaluates alerting rules at a regular interval, defined by
). At each evaluation cycle, Prometheus runs the expression defined in each alerting rule and updates the alert state.
An alert can have the following states:
: the state of an alert that is neither firing nor pending
: the state of an alert that has been active for less than the configured threshold duration
: the state of an alert that has been active for longer than the configured threshold duration
An alert transitions from a state to another only during the evaluation cycle. How the transition occurs, depends if the alert’s
clause has been set or not. From the doc:
The optional FOR clause causes Prometheus to wait for a certain duration between first encountering a new expression output vector element (like an instance with a high HTTP error rate) and counting an alert as firing for this element.
- Alerts without the
clause (or set to0
) will immediately transition tofiring
- Alerts with the
clause will transition first topending
, thus it will take at least two evaluation cycles before an alert with theFOR
clause is fired.
The Alert Lifecycle
Let’s do an example to better explain the lifecycle of an alert. We do have a simple alert that monitors the load 1m of a node, and fires when it’s higher than 20 for at least 1 minute.
Prometheus is configured to scrape metrics every 20 seconds, and the evaluation interval is 1 minute.
Question: how much time does it take to fire
, once the avgload on the machine is higher than 20?
Answer: it takes a time between
. The upper bound is probably higher than you may expect when you set
, but it actually makes much sense in the Prometheus architecture.
The reason of such worst-case timing is explained by the lifecycle of an alert. The following diagram shows the sequence of events over a timeline:
- The load of a node constantly changes, but it gets scraped by Prometheus every
- Alert rules are then evaluated against the scraped metrics every
- When an alert rule expression is true (ie.
node_load1 > 20
), the alert switches topending
, in order to honor theFOR
- At the next evaluation cycles, if the alert expression is still true, once the
clause is honored the alert finally switches tofiring
and a notification is pushed to the alert manager
The Alert Manager
The Alert Manager is the last piece in the pipeline, before an alert notification gets dispatched. It receives in input the alerts once switched to
or switched back to
, and then it dispatch notifications about the alert (please note that
alerts are not forwarded to the Alert Manager).
The Alert Manager has several features, including grouping similar alerts together into a single notification. The grouping feature is particularly important to avoid bombing notification receivers when multiple alerts of the same type occur (ie. when the same alert condition occurring on multiple nodes, we may want to receive just one notification that groups all nodes together instead of a single notification per node).
The downside of grouping notifications is that it may introduce further delays.
The Alert Manager notifications grouping is basically controlled by the following settings:
When a new alert is fired, it waits for
time until the notification gets dispatched. This waiting time is necessary to eventually group further upcoming alerts that match the same
condition (in other terms, two alerts are grouped together if they have the same exact values for the
This means that if
is set to
, the alert manager will buffer the alert notification for 30 seconds, waiting for other potential notifications to be appended to the buffered notification. Once the
time expires, the notification is finally sent.
Great! The notification is finally sent.
But what happens if at the next evaluation cycle further alerts of the same group get fired? They will not wait for
before getting dispatched, but they will wait for
setting controls how long to wait before dispatching further notifications of the same group, and the time interval is calculated starting from the last notification sent.
What bothered me, though, was how long it took for the resolution message to arrive. By default, it takes 5 minutes; luckily, though, you can customize this in AlertManager’s global settings:
This would set the timeout to only 20 seconds, feels much more usable to me given that most of our check intervals are somewhere in the 5-15s range and the alerts are set to something between a 10-20s range. I’m pretty sure I will be tuning this setting in the future but for now this should do 😉