Automating AWS EBS Snapshot Management with Python and Boto3

As an AWS user, you know that creating EBS snapshots is essential for data protection and disaster recovery. However, as your snapshot collection grows, it becomes increasingly important to manage and delete old snapshots to optimize costs and maintain a clean backup environment. Manually managing snapshots can be time-consuming and error-prone, especially when dealing with a large number of volumes across multiple regions. This is where Python comes to the rescue! With its simplicity, versatility, and extensive ecosystem, Python is an excellent choice for automating EBS snapshot management tasks. In this blog post, we’ll explore how to use Python and the AWS SDK (Boto3) to automate the process of listing, retaining, and deleting EBS snapshots based on specific criteria. By leveraging the power of Python, you can save costs, reduce manual effort, and ensure a more efficient and reliable backup strategy.

Why Manage and Delete Old Snapshots?

Before we dive into the code, let’s understand why managing and deleting old snapshots is crucial:

• Cost Savings: EBS snapshots consume storage space, and the costs can add up quickly, especially if you have a large number of snapshots. By regularly deleting old and unnecessary snapshots, you can significantly reduce your storage costs and optimize your AWS spending.
• Compliance and Data Retention: Depending on your industry and regulatory requirements, you may need to adhere to specific data retention policies. Managing snapshots allows you to ensure compliance by retaining snapshots for the required duration and deleting them when they are no longer needed.
• Simplified Backup Management: As your infrastructure grows, managing a large number of snapshots can become complex and overwhelming. By automating snapshot management, you can simplify your backup strategy, reduce clutter, and focus on more critical tasks.

Now that we understand the importance of managing and deleting old snapshots let’s explore how Python and Boto3 can help us achieve this efficiently.

Prerequisites

Before we dive into the code, make sure you have the following:

• Python installed on your machine
• AWS SDK for Python (Boto3) installed (pip install boto3)
• AWS credentials configured with appropriate permissions to manage EBS snapshots

Listing EBS Snapshots

The first step is to list all the EBS snapshots in a specific AWS region. Here’s a Python function that accomplishes this:

def list_snapshots(region):
    ec2 = boto3.client('ec2', region_name=region)
    try:
        snapshots_response = ec2.describe_snapshots(OwnerIds=['self'])
        snapshots = snapshots_response.get('Snapshots', [])
        now = datetime.now(timezone.utc)
        print(f"Listing snapshots in {region}:")
        for snapshot in snapshots:
            start_time = snapshot['StartTime']
            snapshot_age_days = (now - start_time).days
            tags = snapshot.get('Tags', [])
            print(f"Snapshot ID: {snapshot['SnapshotId']} | Region: {region} | Tags: {tags} | Age (days): {snapshot_age_days}")
    except Exception as e:
        print(f"An error occurred while listing snapshots in {region}: {e}")

This function uses the Boto3 EC2 client to retrieve all snapshots owned by the current account in the specified region. It then prints the snapshot details, including the snapshot ID, region, tags, and age in days. Deleting Old Snapshots Next, let’s automate the deletion of old snapshots based on a specified age threshold and mandatory tag filtering. Here’s a Python function to achieve this:

def delete_old_snapshots(region, tag_key, tag_value):
    ec2 = boto3.client('ec2', region_name=region)
    try:
        snapshots_response = ec2.describe_snapshots(OwnerIds=['self'])
        snapshots = snapshots_response.get('Snapshots', [])
        now = datetime.now(timezone.utc)
        snapshots_to_delete = [snapshot['SnapshotId'] for snapshot in snapshots
                               if (now - snapshot['StartTime']).days > 30
                               and any(tag.get('Key') == tag_key and tag.get('Value') == tag_value for tag in snapshot.get('Tags', []))]
        for snapshot_id in snapshots_to_delete:
            ec2.delete_snapshot(SnapshotId=snapshot_id)
            print(f"Deleted snapshot {snapshot_id} in {region}")
        print(f"Deleted {len(snapshots_to_delete)} snapshots in {region} older than 30 days with the specified tag.")
    except Exception as e:
        print(f"An error occurred while processing {region}: {e}")

This function retrieves all snapshots owned by the current account in the specified region and filters them based on the age threshold (30 days in this example) and the provided mandatory tag key and value. It then deletes each snapshot individually and prints a summary of the deleted snapshots.

Scheduling Snapshot Management

To automate the snapshot listing and deletion process, we can use the schedule library to run the script at regular intervals:

import schedule

def run_snapshot_management():
    for region in regions:
        list_snapshots(region)
        delete_old_snapshots(region, tag_key, tag_value)

schedule.every().day.at("02:00").do(run_snapshot_management)

while True:
    schedule.run_pending()
    time.sleep(60)

This code defines a run_snapshot_management() function that iterates over each region, lists snapshots, and deletes old snapshots with the specified mandatory tag. The schedule library is used to run this function daily at 2:00 AM. For those interested in implementing or adapting this script, here’s the complete code:

import boto3
import schedule
import time
from datetime import datetime, timezone

def list_snapshots(region):
    # Create an EC2 client for the specified region
    ec2 = boto3.client('ec2', region_name=region)

    try:
        # Retrieve all snapshots owned by the current account
        snapshots_response = ec2.describe_snapshots(OwnerIds=['self'])
        snapshots = snapshots_response.get('Snapshots', [])

        # Get the current timestamp in UTC
        now = datetime.now(timezone.utc)
        print(f"Listing snapshots in {region}:")
        for snapshot in snapshots:
            # Get the start time of the snapshot
            start_time = snapshot['StartTime']
            # Calculate the age of the snapshot in days
            snapshot_age_days = (now - start_time).days
            # Get the tags associated with the snapshot
            tags = snapshot.get('Tags', [])
            # Print the snapshot details
            print(f"Snapshot ID: {snapshot['SnapshotId']} | Region: {region} | Tags: {tags} | Age (days): {snapshot_age_days}")
    except Exception as e:
        # Print an error message if an exception occurs while listing snapshots
        print(f"An error occurred while listing snapshots in {region}: {e}")

def delete_old_snapshots(region, tag_key=None, tag_value=None):
    # Create an EC2 client for the specified region
    ec2 = boto3.client('ec2', region_name=region)

    try:
        # Retrieve all snapshots owned by the current account
        snapshots_response = ec2.describe_snapshots(OwnerIds=['self'])
        snapshots = snapshots_response.get('Snapshots', [])

        # Get the current timestamp in UTC
        now = datetime.now(timezone.utc)
        # Filter snapshots older than or equal to 30 days and with the specified tag.
        snapshots_to_delete = [snapshot['SnapshotId'] for snapshot in snapshots
                               if (now - snapshot['StartTime']).days > 30
                               and any(tag.get('Key') == tag_key and tag.get('Value') == tag_value for tag in snapshot.get('Tags', []))]
        for snapshot_id in snapshots_to_delete:
            # Delete each snapshot individually
            ec2.delete_snapshot(SnapshotId=snapshot_id)
            print(f"Deleted snapshot {snapshot_id} in {region}")

        print(f"Deleted {len(snapshots_to_delete)} snapshots in {region} older than 30 days with the specified tag.")
    except Exception as e:
        # Print an error message if an exception occurs while processing snapshots
        print(f"An error occurred while processing {region}: {e}")

# List of regions to process
regions = ['us-east-1', 'eu-west-1']  # Add more regions as needed
# Tag key for filtering snapshots (compulsory)
tag_key = 'env'  # Specify the tag key to filter snapshots
# Tag value for filtering snapshots (compulsory)
tag_value = 'prod'  # Specify the tag value to filter snapshots

def run_snapshot_management():
    # Iterate over each region
    for region in regions:
        # List all snapshots in the region
        list_snapshots(region)
        # Delete snapshots older than or equal to 30 days with the specified tag in the region
        delete_old_snapshots(region, tag_key, tag_value)


# Schedule the snapshot management task to run daily at 2:00 AM
schedule.every().day.at("02:00").do(run_snapshot_management)

while True:
    schedule.run_pending()
    time.sleep(60)  # Wait for 60 seconds before checking for the next scheduled task

Best Practices

When automating EBS snapshot management, consider the following best practices:

• Use meaningful and consistent tag keys and values to identify snapshots for deletion.
• Regularly review and test your snapshot deletion scripts in a non-production environment before running them in production.
• Implement monitoring and alerting to track the script’s execution and receive notifications about any errors or deleted snapshots.
• Consider using AWS Data Lifecycle Manager (DLM) for more advanced snapshot lifecycle management features.

Conclusion

Automating EBS snapshot management using Python and Boto3 helps streamline your backup processes, reduce operational complexity, and ensure data protection. By leveraging tags and scheduling, you can implement a robust and efficient snapshot management strategy. Remember to carefully plan your snapshot retention policies, regularly review your scripts, and follow best practices to maintain a reliable and cost-effective backup solution. Happy automating!