Grafana Alerts Backup Role

Overview

This role backs up all Grafana alert rules using the Grafana API to dual NAS storage. It exports alerts in YAML provisioning format, organized by folder, and automatically manages retention by keeping only the last 5 backups per NAS. The backup format is compatible with Grafana’s alert rule provisioning, making restoration straightforward.

Purpose

Disaster Recovery: Protect alert configurations against data loss
Version Control: Maintain timestamped history of alert rule changes
Migration: Transfer alert rules to new Grafana instances
Dual Redundancy: Backs up to two separate NAS devices (Synology DS418 and Proxmox OMV)
API-Based: Uses Grafana API for programmatic backup (no file system access needed)
Folder Organization: Preserves alert folder structure for organized restoration

Requirements

Ansible 2.9 or higher
Collection: community.general (for archive module)
Grafana with API access enabled
Grafana API token with appropriate permissions (stored in Ansible Vault)
NAS mount points must be accessible and writable
Network connectivity to Grafana server (typically port 3000)

Role Variables

Required Variables

Variable	Required	Description
`vault_grafana_ansibleuser_api_token`	Yes	Grafana API token (stored in Ansible Vault)

Optional Variables

Variable	Default	Description
`grafana_alerts_backup_host`	`{{ hostvars['grafana']['ansible_host'] }}:3000`	Grafana host and port
`grafana_alerts_backup_port`	`3000`	Grafana web interface port
`grafana_alerts_backup_validate_certs`	`true`	Validate SSL certificates
`grafana_alerts_backup_syno_mount_point`	`/mnt/synology-ds418/grafana/alerts`	Synology NAS backup destination
`grafana_alerts_backup_prxmxomv_mount_point`	`/mnt/prxmx-omv/grafana/alerts`	Proxmox OMV NAS backup destination

Variable Details

vault_grafana_ansibleuser_api_token

Grafana API token with permissions to read alert rules. Create in Grafana UI: Configuration → API Keys.

Required permissions:

Read alert rules
Read folders

Example vault variable:

vault_grafana_ansibleuser_api_token: "glsa_1234567890abcdefghijklmnopqrstuvwxyz"

grafana_alerts_backup_validate_certs

Set to false if using self-signed certificates.

Example:

grafana_alerts_backup_validate_certs: false

Dependencies

This role has no dependencies on other Ansible roles, but requires:

Grafana instance accessible via HTTPS
NAS mounts configured (see nas_mount or nas_mount_systemd roles)
Grafana API token configured

Example Playbook

Basic Usage

---
- name: Backup Grafana Alert Rules
  hosts: localhost
  become: true

  roles:
    - grafana_alerts_backup

With Custom Grafana Host

---
- name: Backup Grafana Alerts from Custom Host
  hosts: localhost
  become: true

  vars:
    grafana_alerts_backup_host: "monitoring.example.com:3000"
    grafana_alerts_backup_validate_certs: false

  roles:
    - grafana_alerts_backup

Scheduled Backup with Cron

---
- name: Schedule Daily Grafana Alerts Backup
  hosts: localhost
  become: true

  tasks:
    - name: Create cron job for daily Grafana alerts backup
      ansible.builtin.cron:
        name: "Daily Grafana Alerts Backup"
        minute: "0"
        hour: "3"
        job: "ansible-playbook /path/to/grafana_alerts_backup.yml"

What This Role Does

Ensures backup directories exist on both NAS mount points
Retrieves folder list from Grafana API (/api/folders)
Creates temporary directory for staging backup files
Retrieves all alert rules from Grafana Provisioning API (/api/v1/provisioning/alert-rules)
Groups alert rules by folder and saves each as YAML file
Skips empty folders (folders with no alert rules)
Creates timestamped tar.gz archive of all YAML files
Copies archive to both NAS devices
Cleans up temporary files and local archive
Removes old backups, keeping only the 5 most recent per NAS

Backup Format

Alert rules are saved in Grafana YAML provisioning format, one file per folder:

Directory Structure (inside archive)

grafana_alerts_20260107T143022.tar.gz
├── abc123_Production_Alerts_alerts.yaml
├── def456_Infrastructure_alerts.yaml
└── ghi789_Application_Monitoring_alerts.yaml

YAML File Format

groups:
  - name: Production Alerts
    folder: Production Alerts
    rules:
      - uid: alert_uid_1
        title: High CPU Usage
        condition: A
        data:
          - refId: A
            queryType: ''
            model:
              expr: 'cpu_usage > 80'
        for: 5m
        annotations:
          description: CPU usage is above 80%
        labels:
          severity: warning
      - uid: alert_uid_2
        title: Low Disk Space
        ...

This format is:

Compatible with Grafana alert rule provisioning
Human-readable and diff-friendly
Can be restored using grafana_alerts_restore role
Can be imported directly via Grafana UI

Backup Content

The backup includes:

Alert rules (all rules from all folders)
Alert conditions (queries and thresholds)
Annotations (descriptions, runbook URLs, etc.)
Labels (for routing and grouping)
Notification settings (for duration, etc.)
Folder organization (preserved in filename)

The backup does NOT include:

Contact points (notification channels)
Notification policies (alert routing)
Silences
Alert history/state
Dashboards (use grafana_dashboards_backup role)

Empty Folder Handling

If no alert rules are found:

TASK [Display message when no alerts found]
ok: [localhost] => {
    "msg": "No alert rules found to backup - skipping archive creation"
}

This prevents creating empty archives and is normal for new Grafana instances or when all alerts have been deleted.

Backup Retention

The role implements automatic retention management:

Keeps: Last 5 backups per NAS
Deletes: Backups older than the 5th most recent
Per-NAS logic: Each NAS independently maintains 5 backups

This ensures you have:

5 backups on Synology DS418
5 backups on Proxmox OMV
Total: Up to 10 backup copies across both devices

File Naming Convention

Backup files are named using ISO 8601 basic short format:

grafana_alerts_YYYYMMDDTHHMMSS.tar.gz

Examples:

grafana_alerts_20260107T143022.tar.gz
grafana_alerts_20260107T190530.tar.gz

Individual YAML files inside the archive:

{folder_uid}_{folder_title}_alerts.yaml

Examples:

abc123_Production_Alerts_alerts.yaml
def456_Infrastructure_alerts.yaml

API Endpoints Used

1. Get Folders

GET /api/folders
Authorization: Bearer {api_token}

Returns all folders in Grafana, including:

Folder UID
Folder title
Folder ID

2. Get Alert Rules

GET /api/v1/provisioning/alert-rules
Authorization: Bearer {api_token}

Returns all alert rules in provisioning format, including:

Alert UID, title, condition
Folder UID (for grouping)
Query data, annotations, labels
Notification settings

Restoration

To restore from a backup, use the companion grafana_alerts_restore role or manually:

# Extract the archive
tar -xzf grafana_alerts_20260107T143022.tar.gz

# Place YAML files in Grafana provisioning directory
sudo cp *.yaml /etc/grafana/provisioning/alerting/

# Restart Grafana
sudo systemctl restart grafana-server

Or use the Grafana API to import each rule.

Security Considerations

API Token Protection: Token is used in HTTP headers (HTTPS recommended)
Vault Storage: API token must be stored in Ansible Vault
Certificate Validation: Enabled by default, only disable for trusted networks
File Permissions: Backup files created with mode 0644
Sensitive Data: Alert rules may contain sensitive query logic

Notes

The role uses Grafana Provisioning API (v1) which is stable and recommended
Alert rules are grouped by folder for easier organization
Folder names with special characters are sanitized in filenames (e.g., spaces become underscores)
Empty folders are automatically skipped to avoid clutter
Temporary files are always cleaned up, even if the backup fails

Troubleshooting

”Unauthorized” or 401 errors

Verify the API token:

Check token is valid in Grafana UI
Ensure token has read permissions for alert rules
Confirm token hasn’t expired

Test manually:

curl -H "Authorization: Bearer YOUR_TOKEN" https://grafana:3000/api/v1/provisioning/alert-rules

“Connection refused” errors

Verify:

Grafana is running: systemctl status grafana-server
Port 3000 is accessible: nc -zv grafana-host 3000
Firewall allows connections

”Permission denied” when writing to NAS

Ensure:

NAS is mounted: mount | grep synology
Backup directories exist and are writable
User has write permissions

No alert rules backed up (but you have alerts)

Check:

API token permissions include alert rule access
Alert rules are not in “General” folder (may have different UID)
Grafana version supports Provisioning API (v8.0+)

SSL certificate verification fails

If using self-signed certificates:

grafana_alerts_backup_validate_certs: false

Performance Considerations

API calls are minimal (1 for folders, 1 for all rules)
Alert rules are processed in memory (fast)
Archive creation depends on number of alerts (typically < 1 second)
Network transfer time depends on NAS speed

Grafana Version Compatibility

Grafana 8.0+: Full support (Provisioning API v1)
Grafana 7.x: Limited support (use legacy API)
Grafana 9.0+: Full support with enhanced features

grafana_alerts_restore: Restores alert rules from backups created by this role
grafana_dashboards_backup: Backs up Grafana dashboards
grafana_install: Installs and configures Grafana server

License

MIT

Author

Created for homelab infrastructure management.

Grafana Alerts Backup