Solved: How to Send Custom Prometheus Alerts to Discord via Webhooks

🚀 Executive Summary

This article details integrating Prometheus, Alertmanager, and Discord via webhooks for real-time alerting. Traditional alerting methods often lead to alert fatigue or high costs; this solution offers a cost-effective alternative.

Why This Matters

Ideal monitoring systems provide immediate, actionable insights, but real-world implementations often struggle with alert fatigue or the expense of SaaS solutions. Alert fatigue can lead to missed critical incidents, while proprietary tools can become prohibitively expensive, especially at scale. Integrating open-source tools like Prometheus and Alertmanager with a readily available communication platform like Discord offers a balance between reliability and cost-effectiveness.

Key Insights

• Discord webhooks provide a simple integration point for sending messages to channels without bots, 2016.
• Prometheus alerting rules use PromQL expressions, for duration, and labels/annotations for context, 2012.
• Alertmanager’s configuration uses routes for alert grouping and receivers with webhook_configs for output destinations, 2015.

Working Example

# alert.rules.yml
groups:
- name: server_cpu_alerts
  rules:
  - alert: HighCPUUsage
    expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
    for: 5m
    labels:
      severity: critical
      team: infrastructure
    annotations:
      summary: "High CPU usage detected on {{ $labels.instance }}"
      description: "The CPU usage on instance {{ $labels.instance }} has been above 90% for more than 5 minutes. Current value: {{ $value | humanize }}%."

# alertmanager.yml
global:
  resolve_timeout: 5m
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'discord-default'
receivers:
- name: 'discord-default'
  webhook_configs:
  - url: 'YOUR_DISCORD_WEBHOOK_URL_HERE'
    send_resolved: true

Practical Applications

• Company/system: TechResolve uses this setup to immediately notify DevOps teams of critical infrastructure issues via Discord.
• Pitfall: Using overly broad PromQL expressions can generate excessive alerts, leading to alert fatigue and desensitization.

References:

• https://dev.to/techresolve/solved-how-to-send-custom-prometheus-alerts-to-discord-via-webhooks-47ie