NerdVana

Moving multiple files out of AWS Glacier

Written 10 years ago by Eric Schwimmer

I needed to de-glacier-ize a bunch of files in one of our S3 buckets; unfortunately Amazon makes this kind of hard to do on a large scale. Hence this:

#!/bin/bash

[[ $# != 1 ]] && echo "Usage: deglacierize <s3 path>" && exit 1
S3PATH=$1
[[ "${S3PATH: -1}" == "/" ]] || S3PATH="${S3PATH}/"

if [[ "$S3PATH" =~ ^s3://([^/]+)/(.*)?/$ ]]; then
    BUCKET=${BASH_REMATCH[1]}
else
    echo "S3 path must be in the format 's3://bucket/path'"
fi

TMPFILE1=/tmp/deglacierize.$$.1
TMPFILE2=/tmp/deglacierize.$$.2

quit() {
    rm -f $TMPFILE1 $TMPFILE2
    exit $1
}

aws s3 ls --recursive "${S3PATH}"  > $TMPFILE1
[[ $? != 0 ]] && echo "Error when querying S3; quitting" && quit 1
awk '{if ($4) print $4}' $TMPFILE1 > $TMPFILE2

NUM_FILES=$(wc -l $TMPFILE2 | cut -d" " -f1)
read -p "About to restore ${NUM_FILES} files; proceed? [y|N]: " go
[[ $go != "y" ]] && echo "Aborting!" && quit 1

while read KEY; do
    echo -n "Restoring s3://$/$... "
    aws s3api restore-object \
        --bucket "$" \
        --key "$" \
        --restore-request '{"Days":30}'
    [[ $? != 0 ]] && echo "Error!" && quit 1
    echo "ok!"
done < $TMPFILE2

quit 0