I have this simple Prometheus alert configured
IF my_custom_metric == 0
Scenario is as follows:
- In the correct runtime my_custom_metric is 1, and MyServiceDown alert is green at Alert tab.
- Then I set my_custom_metric 0 and Propetheus retrieves it as 0, which I can see at Graph -> Console tab.
- MyServiceDown immediately becomes Pending (yellow) and after 15 seconds becomes Firing (red).
- Then I "fix bug", my_custom_metric is 1 again, and MyServiceDown alert is green.
Here comes strange part. I break app second time so that my_custom_metric is 0 again. It is shown at Graph -> Console. But MyServiceDown alert never pending or firing. When I restart Prometheus everything works good, but only once.
What am I doing wrong?