Home Ask Login Register

Developers Planet

Your answer is one click away!

user1788078 February 2016

Prometheus alert not peding/firing at second time

I have this simple Prometheus alert configured

ALERT MyServiceDown
  IF my_custom_metric == 0
  FOR 15s

Scenario is as follows:

  1. In the correct runtime my_custom_metric is 1, and MyServiceDown alert is green at Alert tab.
  2. Then I set my_custom_metric 0 and Propetheus retrieves it as 0, which I can see at Graph -> Console tab.
  3. MyServiceDown immediately becomes Pending (yellow) and after 15 seconds becomes Firing (red).
  4. Then I "fix bug", my_custom_metric is 1 again, and MyServiceDown alert is green.

Here comes strange part. I break app second time so that my_custom_metric is 0 again. It is shown at Graph -> Console. But MyServiceDown alert never pending or firing. When I restart Prometheus everything works good, but only once.

What am I doing wrong?

Answers


user1788078 February 2016

There was bug in version 0.17rc1

https://github.com/prometheus/prometheus/issues/1372

It fixed in 0.17rc2

Problem solved.

Post Status

Asked in February 2016
Viewed 2,010 times
Voted 8
Answered 1 times

Search




Leave an answer


Quote of the day: live life