Grafana Alerts Restore

This role restores Grafana alert rules from backup archives created by `grafana_alerts_backup`.

Ansible Bash Grafana HTTPS NAS SSL Synology YAML

Grafana Alerts Restore Role

Overview

This role restores Grafana alert rules from backup archives created by grafana_alerts_backup. It intelligently handles folder creation, contact point setup, and alert rule import/update using the Grafana Provisioning API. The role automatically identifies the latest backup and handles conflicts gracefully by updating existing alerts.

Purpose

  • Disaster Recovery: Restore alert configurations after data loss
  • Migration: Transfer alert rules to new Grafana instances
  • Environment Cloning: Replicate production alerts in testing environments
  • Version Rollback: Restore previous alert configurations
  • Automated Folder Management: Creates required folders if they don’t exist
  • Conflict Resolution: Updates existing alerts instead of failing

Requirements

  • Ansible 2.9 or higher
  • Grafana with API access enabled
  • Grafana API token with appropriate permissions (stored in Ansible Vault)
  • Alert backup archives created by grafana_alerts_backup role
  • NAS mount point accessible with backup files
  • Network connectivity to Grafana server (typically port 3000)

Role Variables

Required Variables

VariableRequiredDescription
vault_grafana_ansibleuser_api_tokenYesGrafana API token (stored in Ansible Vault)

Optional Variables

VariableDefaultDescription
grafana_alerts_restore_backup_path/mnt/synology-ds418/grafana/alertsDirectory containing backup archives
grafana_alerts_restore_host{{ hostvars['grafana']['ansible_host'] }}:3000Grafana host and port
grafana_alerts_restore_port3000Grafana HTTP port
grafana_alerts_restore_validate_certsfalseValidate SSL certificates
grafana_alerts_restore_contact_point_nameJoffreyContact point name to create
grafana_alerts_restore_contact_point_typeemailContact point type (email, slack, etc.)
grafana_alerts_restore_contact_point_emailadmin@example.comEmail address for notifications

Variable Details

vault_grafana_ansibleuser_api_token

Grafana API token with permissions to create/update alert rules and folders.

Required permissions:

  • Create/update alert rules
  • Create folders
  • Create contact points

Example vault variable:

vault_grafana_ansibleuser_api_token: "glsa_1234567890abcdefghijklmnopqrstuvwxyz"

grafana_alerts_restore_contact_point_name

Name of the contact point (notification channel) to create. The role checks if it exists and only creates it if missing.

Example:

grafana_alerts_restore_contact_point_name: "OnCall Team"

grafana_alerts_restore_contact_point_type

Type of notification channel. Supported types depend on Grafana version.

Common types:

  • email - Email notifications
  • slack - Slack webhooks
  • webhook - Generic webhooks
  • pagerduty - PagerDuty integration
  • teams - Microsoft Teams

Example:

grafana_alerts_restore_contact_point_type: "slack"

Dependencies

This role has no dependencies on other Ansible roles, but requires:

  • Grafana instance accessible via HTTPS
  • Alert backups created by grafana_alerts_backup role
  • NAS mount configured (if restoring from network storage)

Example Playbook

Basic Usage (Restore Latest Backup)

---
- name: Restore Grafana Alert Rules
  hosts: localhost

  roles:
    - grafana_alerts_restore

Restore to Different Grafana Instance

---
- name: Restore Alerts to New Grafana Server
  hosts: localhost

  vars:
    grafana_alerts_restore_host: "new-grafana.example.com:3000"
    grafana_alerts_restore_validate_certs: true
    grafana_alerts_restore_contact_point_name: "Production Alerts"
    grafana_alerts_restore_contact_point_email: "alerts@example.com"

  roles:
    - grafana_alerts_restore

Restore with Custom Backup Location

---
- name: Restore Alerts from Custom Backup
  hosts: localhost

  vars:
    grafana_alerts_restore_backup_path: /mnt/backup2/grafana/alerts

  roles:
    - grafana_alerts_restore

What This Role Does

  1. Finds latest backup archive matching pattern grafana_alerts_*.tar.gz
  2. Fails gracefully if no backups are found
  3. Creates temporary directory for extraction
  4. Extracts backup archive to temporary location
  5. Finds all YAML files in extracted backup
  6. Parses YAML provisioning files and extracts alert rule groups
  7. Identifies required folders from alert rules
  8. Retrieves existing folders from Grafana API
  9. Creates missing folders only if they don’t already exist
  10. Builds folder name-to-UID mapping from both new and existing folders
  11. Extracts individual alert rules with correct folderUIDs and rule groups
  12. Checks for contact point existence, creates if missing
  13. Imports alert rules via Grafana Provisioning API (POST)
  14. Updates existing alerts if conflicts occur (PUT with UID)
  15. Cleans up temporary files
  16. Displays restoration summary with statistics

Backup File Selection

The role automatically selects the most recent backup using these criteria:

  • Searches for files matching pattern: grafana_alerts_*.tar.gz
  • Sorts by modification time (mtime)
  • Uses the newest file

Example backup files (newest first):

grafana_alerts_20260107T190530.tar.gz  <- This will be used
grafana_alerts_20260107T143022.tar.gz
grafana_alerts_20260106T020000.tar.gz

Conflict Handling

The role intelligently handles conflicts when alert rules already exist:

Initial Import (Status 201)

New alert rules are created successfully.

Conflict Detection (Status 409)

If an alert with the same UID already exists, the role:

  1. Receives HTTP 409 Conflict response
  2. Automatically retries with PUT request to update the existing alert
  3. Uses the alert UID to target the specific rule
  4. Updates the alert with values from the backup

This means:

  • ✅ You can safely re-run the restore role multiple times
  • ✅ Existing alerts are updated, not duplicated
  • ✅ No manual cleanup required

Contact Point Creation

Before importing alerts, the role ensures a contact point exists:

Contact Point Configuration:
  Name: Joffrey
  Type: email
  Email: admin@example.com

Behavior:

  • Checks if contact point with name “Joffrey” exists
  • Creates it only if missing
  • Uses HTTP status 202 (Accepted) for creation

Why this matters: Alert rules reference contact points for notifications. Without a contact point, alerts may not send notifications.

Folder Management

Alert rules belong to folders. The role:

  1. Extracts folder names from backup files
  2. Gets existing folders from Grafana
  3. Creates only missing folders
  4. Builds UID mapping for both new and existing folders
  5. Assigns correct folderUID to each alert rule

Example:

Backup contains alerts in folders:
- Production Alerts (doesn't exist) → Created with UID: abc123
- Infrastructure (exists with UID: def456) → Reused
- Application Monitoring (doesn't exist) → Created with UID: ghi789

Restoration Summary

After completion, the role displays a summary:

✓ Grafana alerts restored from: grafana_alerts_20260107T190530.tar.gz
Total alerts processed: 15
Successfully created: 10
Already existed: 5

Note: If alerts are not visible in Grafana, check:
  - The notification receiver exists in Grafana (e.g., 'Joffrey')
  - Datasource UIDs match your current Grafana instance
  - Alert rule folders exist

What Gets Restored

The restoration includes:

  • All alert rules from the backup
  • Alert conditions (queries and thresholds)
  • Annotations (descriptions, runbook URLs)
  • Labels (for routing and grouping)
  • Notification settings (for duration, etc.)
  • Folder structure (folders created as needed)

The restoration does NOT restore:

  • Notification policies (alert routing rules)
  • Silences (muted alerts)
  • Alert history/state
  • Dashboards (use grafana_dashboards_restore role)

Post-Restoration Verification

After restoration, verify in Grafana UI:

  1. Check alert rules: Alerting → Alert rules
  2. Verify folders: Confirm all folders were created
  3. Test contact point: Alerting → Contact points → Test
  4. Review datasources: Ensure datasource UIDs match
  5. Check notification policies: Configure routing if needed

Common Post-Restore Issues

Alerts Exist But Don’t Fire

Problem: Datasource UIDs don’t match

Solution:

# Old datasource UID in backup: abc123
# New datasource UID: xyz789

# Update alert queries manually or use Grafana UI to reconnect

Notifications Not Sent

Problem: Contact point doesn’t exist or isn’t configured

Solution:

  • Verify contact point in Grafana: Alerting → Contact points
  • Configure notification policy: Alerting → Notification policies
  • Test contact point with “Test” button

Alerts Show “No Data”

Problem: Datasource connection issues

Solution:

  • Check datasource configuration
  • Verify queries return data
  • Review datasource permissions

API Endpoints Used

1. Get Folders

GET /api/folders
Authorization: Bearer {api_token}

2. Create Folder

POST /api/folders
Body: {"title": "Folder Name"}

3. Get Contact Points

GET /api/v1/provisioning/contact-points

4. Create Contact Point

POST /api/v1/provisioning/contact-points
Body: {contact point configuration}

5. Create Alert Rule

POST /api/v1/provisioning/alert-rules
Body: {alert rule configuration}

6. Update Alert Rule

PUT /api/v1/provisioning/alert-rules/{uid}
Body: {alert rule configuration}

Security Considerations

  • API Token Protection: Token used in HTTP headers (HTTPS recommended)
  • Vault Storage: API token must be in Ansible Vault
  • Certificate Validation: Disabled by default (set to true for production)
  • Email Exposure: Contact point email is in plaintext (consider using secrets)

Tags

This role does not define any tags. Use playbook-level tags if needed:

- hosts: localhost
  roles:
    - grafana_alerts_restore
  tags:
    - restore
    - grafana
    - disaster-recovery
    - alerts

Notes

  • The role uses Grafana Provisioning API (v1) which is stable
  • Alert rules are processed individually for better error handling
  • Folder UIDs are dynamically mapped to handle different Grafana instances
  • Temporary files are always cleaned up, even if restoration fails
  • The role is idempotent - safe to run multiple times

Troubleshooting

”No backup files found”

Verify:

  • NAS is mounted: mount | grep synology
  • Backup directory exists: ls -l /mnt/synology-ds418/grafana/alerts/
  • Backup files exist: ls -l /mnt/synology-ds418/grafana/alerts/grafana_alerts_*.tar.gz

”Unauthorized” or 401 errors

Verify API token:

  • Check token is valid in Grafana UI
  • Ensure token has required permissions (alert rules, folders, contact points)
  • Confirm token hasn’t expired

Test manually:

curl -H "Authorization: Bearer YOUR_TOKEN" https://grafana:3000/api/v1/provisioning/alert-rules

Alerts imported but not visible in UI

Check:

  • Correct Grafana organization (multi-tenant setup)
  • User has permission to view the alert folders
  • Browser cache (try hard refresh: Ctrl+Shift+R)

“Contact point creation failed”

Check:

  • Email address is valid (for email type)
  • Grafana email configuration is setup (for sending emails)
  • Required fields for contact point type are provided

Datasource UID mismatch errors

Alert rules reference datasources by UID. If the UID doesn’t match:

Manual fix:

  1. Note the datasource UID error
  2. Find your datasource UID: Connections → Data sources → Click datasource → URL shows UID
  3. Update alert rule via Grafana UI to use correct datasource

Automated fix (advanced): Pre-process backup files to replace old datasource UIDs with new ones before import.

Performance Considerations

  • Processing time depends on number of alert rules (typically 1-2 seconds per rule)
  • API rate limiting may apply for large imports (hundreds of rules)
  • Folder creation is optimized (only creates missing folders)

Grafana Version Compatibility

  • Grafana 8.0+: Full support (Provisioning API v1)
  • Grafana 9.0+: Full support with enhanced features
  • Grafana 10.0+: Full support
  • grafana_alerts_backup: Creates the backup files this role restores
  • grafana_dashboards_restore: Restores Grafana dashboards
  • grafana_datasource_create: Configures datasources for alerts

License

MIT

Author

Created for homelab infrastructure management.