Solving Alert Fatigue with the Grafana Cloud Kubernetes Operator

We Had 100 Dead Alerts Firing for Services That No Longer Existed. So I Built a Kubernetes Operator.

The Grafana Cloud Operator was developed to synchronize observability resources directly with Kubernetes service lifecycles. It specifically addressed a backlog of 100 orphaned alert rules that remained active months after their associated services were decommissioned. This system ensures that monitoring state is reconciled automatically against the desired infrastructure state.

Why This Matters

In high-scale environments, manual management of dashboards and alerts leads to permanent drift and orphaned resources that erode system trust. The technical reality of 2 AM manual tweaks often bypasses version control, creating a disconnect between infrastructure state and monitoring. Using a controller to reconcile these resources ensures that deletion of a Kubernetes namespace automatically triggers the removal of its Grafana counterparts, preventing the accumulation of ‘dead’ alerts that ignore decommissioned services.

Key Insights

• Lifecycle Coupling: Deleting a Kubernetes namespace triggers the operator to scan for resources tagged with ‘createdby=operator’ and ‘cluster=id’ to purge orphaned Grafana alerts.
• Hash-based Idempotency: The operator uses SHA1 hashes of payloads to prevent redundant API calls during frequent controller reconciliation loops.
• Multi-cluster Safety: By using a CLUSTER_ID environment variable, the operator prevents cross-cluster resource deletion in shared Grafana Cloud organizations.
• Provisioning API Locking: Alert rules created via the operator are locked in the Grafana UI, enforcing Git as the sole source of truth for monitoring configuration.
• SLO Support: The operator manages the Grafana Cloud SLO plugin, handling definitions for targets and indicators that the official Grafana operator currently lacks.

Working Examples

Definition of a Grafana alert rule as a Kubernetes Custom Resource.

apiVersion: monitoring.grafana-operator.io/v1alpha1
kind: GrafanaAlertRule
metadata:
  name: high-error-rate
  namespace: payments-service
spec:
  title: "High Error Rate"
  folder: "payments-service"
  datasourceUid: "grafanacloud-prom"
  condition: "C"
  for: "5m"
  notificationSettings:
    receiver: "payments-pagerduty"
  data:
    # ... Grafana alert query blocks

Simplified Go logic for handling resource deletion with ownership and cluster checks.

func handleAlertDelete(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  uid := grafanautil.GenerateStableUID(req.Namespace, req.Name)
  existing := fetchAlertFromGrafana(uid)
  if existing.Labels["createdby"] != "operator" {
    return ctrl.Result{}, nil
  }
  if existing.Labels["cluster"] != os.Getenv("CLUSTER_ID") {
    return ctrl.Result{}, nil
  }
  deleteAlertFromGrafana(uid)
}

Implementation of hash-based idempotency to reduce API overhead.

hash := computeSHA1(payload)
if alert.Status.AlertHash == hash {
  logger.Info("No change in alert rule, skipping sync")
  return ctrl.Result{}, nil
}

CRD for managing Service Level Objectives (SLOs) within Grafana Cloud.

apiVersion: monitoring.grafana-operator.io/v1alpha1
kind: GrafanaSLO
metadata:
  name: api-availability
  namespace: payments-service
spec:
  title: "API Availability"
  target: 99.9
  timeWindow: "30d"
  indicator:
    type: ratio
    source: "grafanacloud-prom"
    params:
      good: 'http_requests_total{status!~"5.."}'
      total: "http_requests_total"

Practical Applications

• Platform teams managing multi-cluster environments can use the cluster label to ensure staging deletions don’t impact production dashboards. Pitfall: Failing to scope resources by cluster ID can lead to accidental deletion of identical resources across environments.
• Developers can use the included CLI generator to produce valid Grafana Alert YAML in under three minutes, bypassing complex nested JSON schemas. Pitfall: Manually writing Grafana query data structures often leads to syntax errors that are difficult to debug.
• SREs can enforce drift correction by using the force-sync annotation to overwrite manual UI changes with the Git-defined state. Pitfall: Allowing manual UI edits creates undocumented monitoring states that lead to confusion during incidents.

References:

• https://dev.to/infra_tools_97d10de984ee0/we-had-100-dead-alerts-firing-for-services-that-no-longer-existed-so-i-built-a-kubernetes-operator-5e6d
• github.com/nidhirai968/grafana-cloud-operator