Add option to handle watchdog alerts

[martialblog]

In Prometheus a "watchdog" or "dead man's switch" is an alert that is always firing to ensure alerting pipeline is working. The -W, --watchdog flag can be used to flip/negate the exit state of the plugin for these kind of alerts:

$ check_prometheus alert --name Watchdog -W --no-alerts-state 2
[OK] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [OK] [Watchdog] is firing - value: 1.00 - {"alertname":"Watchdog","severity":"none"}
|total=1 firing=1 pending=0 inactive=0

$ check_prometheus alert --name Watchdog -W --no-alerts-state 2
[CRITICAL] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] No alerts retrieved
|total=0 firing=0 pending=0 inactive=0

Fixes #123

See also #61 (comment)

[martialblog]

@wattebausch @gianmarco-mameli

I'm planing to add this feature in the next release. Any feedback and testing is welcome.

Regards
Markus

[gianmarco-mameli]

Hi @martialblog, I don't have this kind of alerts but I try to find some time to manage a test

[wattebausch]

@martialblog i re-enabled the watchdog on my kube-prometheus-stack and it works like a charm

$ ./check_prometheus -H test.example.org -p 443 -s alert -P -W
[OK] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [OK] [Watchdog] is firing - value: 1.00 - {"alertname":"Watchdog","severity":"none"}
|total=1 firing=1 pending=0 inactive=0

$ ./check_prometheus -H test.example.org -p 443 -s alert -P 
[CRITICAL] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] [Watchdog] is firing - value: 1.00 - {"alertname":"Watchdog","severity":"none"}
|total=1 firing=1 pending=0 inactive=0

many thanks