Posted inNuggets

Bash script to automate EBS volume snapshot

11 Comments
automate EBS volume snapshot

This will be a nugget on how to automate EBS volume snapshot using a bash script.

Need for EBS snapshot:

Data backup is of utmost importance to avoid single point of failure and to restore to a specific point in time in case of corruption. EBS volume snapshot is one of the most resilient way by which you can backup your data in AWS. Using a snapshot, you can create a new AMI and in turn launch a new instance with that AMI. You’ll be ready with the setup identical to the time when the snapshot was taken. It is recommended to take EBS snapshot periodically for data backup

You can read more on what EBS snapshot is and how incremental snapshot works here

 

Where Can I run this script from ?

You can run this script from your local, a Jump / Bastion server or even from any of the server for which you would like to take EBS snapshot for. We will run the script as part of cron job to make it run at specified interval. So running it from your local machine is not recommended.

 

Pre-Requisites

There are few pre-requisites that you need to complete for the script to work as expected.

  • Firstly you need to have the “aws cli” installed in the server from which the script will be run.
  • You should have API keys of an IAM user who has access to Read EC2 information, create and delete EBS snapshots.
  • jq to parse the json output of aws cli commands

 

Install aws cli:

You can refer this for manually installing aws cli for various distribution. If you are running the script from an EC2 instance, then installing aws cli is simple.

 

Debian/Ubuntu:

# apt-get install awscli

 

RedHat/CentOS:

# yum install awscli

 

Create IAM user:

You can refer this for creating an IAM user. Attach a policy that contains the below set of rules for the IAM user.

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:Describe*",
                "ec2:CreateTags",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot",
                "ec2:DescribeSnapshots"
            ],
            "Resource": "*"
        }
    ]
}

 

Configure aws cli

Execute the below command to configure aws cli

# aws configure
AWS Access Key ID [None]: <Access Key>
AWS Secret Access Key [None]: <Secret Key>
Default region name [None]: <region>
Default output format [None]: <ENTER>

The configuration wizard will ask you to enter the Access Key, Secret Key, Default Region and the output format. You can pass in the Access and Secret Key of IAM user which you have created in the previous step. By default the output format is json and you can leave it as default.

 

Install jq

Debian/Ubuntu:

# apt-get install jq

 

RedHat/CentOS:

# yum install jq

 

Bash script to automate EBS volume snapshot

#!/bin/bash
 
ACTION=$1
AGE=$2
 
if [ -z $ACTION ];
then
echo "Usage $1: Define ACTION of backup or delete"
exit 1
fi
 
if [ "$ACTION" == "delete" ] &amp;&amp; [ -z $AGE ];
then
	echo "Please enter the age of backups you would like to delete"
	exit 1
fi
 
function backup_ebs () {
 
	prod_instances=`aws ec2 describe-instances --filters "Name=tag-value,Values=prod*" | jq -r ".Reservations[].Instances[].InstanceId"`
 
	for instance in $prod_instances
	do
 
		volumes=`aws ec2 describe-volumes --filter Name=attachment.instance-id,Values=$instance | jq .Volumes[].VolumeId | sed 's/\"//g'`
 
		for volume in $volumes
		do
			echo Creating snapshot for $volume $(aws ec2 create-snapshot --volume-id $volume --description "ebs-backup-script")
		done
 
	done
}
 
function delete_snapshots () {
 
	for snapshot in $(aws ec2 describe-snapshots --filters Name=description,Values=ebs-backup-script | jq .Snapshots[].SnapshotId | sed 's/\"//g')
	do
 
		SNAPSHOTDATE=$(aws ec2 describe-snapshots --filters Name=snapshot-id,Values=$snapshot | jq .Snapshots[].StartTime | cut -d T -f1 | sed 's/\"//g')
		STARTDATE=$(date +%s)
		ENDDATE=$(date -d $SNAPSHOTDATE +%s)
		INTERVAL=$[ (STARTDATE - ENDDATE) / (60*60*24) ]
 
		if (( $INTERVAL &gt;= $AGE ));
	then
		echo "Deleting snapshot --&gt; $snapshot"
		aws ec2 delete-snapshot --snapshot-id $snapshot
	fi
 
	done
}
 
case $ACTION in
 
"backup")
backup_ebs
;;
 
"delete")
delete_snapshots
;;
 
esac

 

Script run through

  • The script takes two parameters – ACTION and AGE. ACTION can either be backup or delete. ( Refer Step 3 and 4 )
  • If the ACTION is delete, then the script expects another parameter AGE. ( Refer Step 12 to 16 )
  • AGE takes a number and  defines the snapshot to be removed in terms of days. For eg, value of 5 will remove snapshots older than 5 days.
  • Based on ACTION, relevant function is called through case statement. ( Refer Step 54 to 64 )
  • The backup_ebs function first gets the list of instance that has a tag-value that starts with prod ( —filters “Name=tag-value,Values=prod*”). The resulting list is assigned to a variable called prod_instances. You can customize the filter as per your need. If you do not enforce a filter, then you will get a list of all the instances running in the Region. ( Refer Step 20 )
  • Based on the instance id list generated in the previous step, it is then passed through a loop to get the id’s of the volume attached to each instance. ( Refer Step 22 to 25 )
  • The volume list is further passed into another nested loop where the snapshot is taken against each volume. ( Refer Step 27 to 30 ).
  • While taking snapshot, it is given the description “ebs-backup-script“ in order to identify that the snapshot was taken by the script (—description “ebs-backup-script”).
  • The delete_snapshots function first gets a list of snapshots available with the description “ebs_backup-script“. This is to make sure that we do not delete snapshots that were generated manually by a different team/member. ( Refer Step 37 )
  • The snapshot start time is stored in the variable SNAPSHOTDATE. ( Refer Step 40 )
  • The present time is stored in seconds in the variable STARTDATE. ( Refer Step 41 )
  • The end time is stored in seconds based of of the snapshot start time in the variable ENDDATE. ( Refer Step 42 )
  • The INTERVAL of the snapshot is calculated based on STARTDATE and ENDTIME. ( Refer Step 43 )
  • It then checks for the condition whether the value of  INTERVAL is greater than or equal to the AGE. If the condition satisfies, it deletes the snapshot. Else, it then checks for the next item in the list. ( Refer Step 45 to 48 )

 

This script provides the basic functionality of automating the process of EBS volume snapshot and life cycle. This can further be customized and improved to handle higher needs such as handling snapshots across different regions, logging the output, trigger a mail if the script fails, etc.

 

Schedule it as cron job

You can schedule this script to run as part of cron.

# Backup EBS Snapshot daily at 3 am
00 3 * * * /path/to/ebs_snapshot.sh backup

# Manage Life cycle policy of EBS Snapshot 
15 3 * * * /path/to/ebs_snapshot.sh delete 5

 

Note:

The script is available in the git repository. You can clone it by running the below command.

# git clone https://github.com/chrissam/AWS_scripts.git

 

You can post your questions/suggestions in the comment section.

 

0 Shares
Tags: