How to Monitor Cron Jobs to Prevent Silent Failures

How to monitor cron jobs so they don’t fail silently

Scheduled background jobs often lack a UI and disappear into background logs, making failures difficult to detect immediately. A job might fail due to an expired API token or a database connection error, resulting in missing data or stale reports. Krasimir Petkov proposes a ping-based monitoring approach to make these failures visible.

Why This Matters

Cron jobs run on servers where they are easily forgotten until a downstream dependency fails. While logs capture errors, they are passive and require manual intervention to review, creating a lag between the incident and its discovery. By implementing a best-effort reporting system, developers can distinguish between a script that crashed and a job that never started. This proactive visibility ensures that critical tasks like database backups and billing syncs remain healthy without the monitoring tool itself becoming a fragile dependency that breaks the core job.

Key Insights

• The ping approach involves sending start, success, and failure signals to a monitoring endpoint to track execution status.
• Monitoring calls should use non-blocking patterns like ’|| true’ in shell or try-except in Python to prevent monitoring downtime from affecting the job.
• Distinguishing between ‘failed’ and ‘missed’ states allows developers to separate script logic errors from environment-level execution failures.
• Useful monitoring states include ‘running’, ‘healthy’, ‘failed’, ‘late’, and ‘missed’ to provide a complete operational picture of job health.
• MissedRun is a specialized tool developed to provide ping URLs and monitor history for recurring background tasks.

Working Examples

A simple shell wrapper that pings start, success, and failure endpoints.

#!/bin/bash\nSTART_URL="https://example.com/ping/YOUR_TOKEN/start"\nSUCCESS_URL="https://example.com/ping/YOUR_TOKEN"\nFAIL_URL="https://example.com/ping/YOUR_TOKEN/fail"\ncurl -fsS -X POST --max-time 5 "$START_URL" >/dev/null || true\nyour-real-command-here\nEXIT_CODE=$?\nif [ $EXIT_CODE -eq 0 ]; then\ncurl -fsS -X POST --max-time 5 "$SUCCESS_URL" >/dev/null || true\nelse\ncurl -fsS -X POST --max-time 5 "$FAIL_URL" >/dev/null || true\nfi\nexit $EXIT_CODE

Python implementation of best-effort ping monitoring using the requests library.

import requests\nSTART_URL = "https://example.com/ping/YOUR_TOKEN/start"\nSUCCESS_URL = "https://example.com/ping/YOUR_TOKEN"\nFAIL_URL = "https://example.com/ping/YOUR_TOKEN/fail"\ndef safe_ping(url: str) -> None:\n    try:\n        requests.post(url, timeout=5)\n    except requests.RequestException:\n        pass\ndef run_job() -> None:\n    print("Running job...")\n    safe_ping(START_URL)\n    try:\n        # Replace this with your real scheduled task.\n        pass\n    except Exception:\n        safe_ping(FAIL_URL)\n        raise\n    else:\n        safe_ping(SUCCESS_URL)

Practical Applications

• Use Case: Database backups and ETL jobs where ‘nothing happened’ indicates a major failure. Pitfall: Relying on passive logs which are not checked until data loss is discovered.
• Use Case: Billing syncs and email digests that run in the background. Pitfall: Expired API tokens causing silent failures that go unnoticed for days.
• Use Case: Cache refreshes and background cleanup scripts. Pitfall: Server restarts preventing the cron job from firing without any explicit error report.

References:

• https://dev.to/krasimir_petkov_c14f3b461/how-to-monitor-cron-jobs-so-dont-fail-silently-2a1f