Tuesday, September 08, 2009

How to create and save an AMI image from a running instance

One snag I encountered early on in my migration of Cragwag and Sybilline to Amazon's EC2 Cloud, was that I needed to take a snapshot of my running instance and save it as a new Amazon Machine Image (AMI).

I'd created a bare-bones Debian image from a public AMI (32-bit Lenny, 5.0, not much else) and then installed a few standard software packages on it - mysql, ruby, apache, etc etc etc. Once I'd got them configured the way I wanted, it had taken a couple of hours (I'll go into the configuration relating to EBS in a separate post) so I wanted to snapshot this instance as a new AMI image. That way, if and when I needed to create a new instance, all of this work would already have been done.

It actually took a fair amount of time to find out (well, more than a few seconds Googling, which is just eternity these days, y'know?) so I'll save you the pain and just give you the solution.

First, install Amazon's AMI tools, and API tools:

export EC2_TOOLS_DIR=~/.ec2 #(or choose a directory here)
mkdir ec2-ami-tools
cd ec2-ami-tools
wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
unzip ec2-ami-tools.zip
ln -s ec2-ami-tools-* current
cd ..
mkdir ec2-api-tools
cd ec2-api-tools
wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip ec2-api-tools.zip
ln -s ec2-api-tools-* current

echo "export EC2_AMITOOL_HOME=`dirname $EC2_TOOLS_DIR`/ec2-ami-tools/current" >> ~/.bashrc
echo "export EC2_APITOOL_HOME=`dirname $EC2_TOOLS_DIR`/ec2-api-tools/current" >> ~/.bashrc
echo "export PATH=${PATH}:`dirname $AMI_TOOLS_DIR`/ec2-ami-tools/current/bin:`dirname $AMI_TOOLS_DIR`/ec2-api-tools/current/bin" >> ~/.bashrc
source ~/.bashrc

Next, you'll need to get your security credentials. You can get a reminder of - or create as needed - these on the AWS "Your Account" > "Security Credentials" page.

I recommend you saving your X.509 certificate and your private key somewhere under /mnt/ - this directory is excluded from the bundled image. Quite important that, as otherwise your credentials would be bundled up in the image - and if you ever shared that image with anyone else, you'd be sharing your credentials too!

You'll also need to note your AWS access details - especially your access key and secret key - plus your Amazon account ID.

Now, we're at the main event.

To take a snapshot of your running instance:

First, choose a name for your AMI snapshot. We'll call it ami-instance-name :)

# make a directory for your image:
mkdir /mnt/ami-instance-name

# create the image (this will take a while!)
ec2-bundle-vol -d /mnt/ami-instance-name -k /path/to/your/pk-(long string).pem -c /path/to/your/cert-(long string).pem -u YOUR_AMAZON_ACCOUNT_ID_WITHOUT_DASHES

Once that's done, you should have a file called image.manifest.xml in your /mnt/ami-instance-name directory, along with all the bundle parts. Sometimes it will say Unable to read instance meta-data for product-codes - but this doesn't seem to cause any problems, and I've successfully ignored it so far :)

Next, upload the AMI image to S3. This command will create an S3 bucket of the given name if it doesn't exist - I've found it convenient to call my buckets the same as the instance name:

ec2-upload-bundle -b ami-instance-name -m /mnt/ami-instance-name/image.manifest.xml -a YOUR_AWS_ACCESS_KEY -s YOUR_AWS_SECRET_KEY

You should then be able to register the instance. I've done that using the rather spiffy AWS Management Console web UI, but you can also do it from the command line using:

ec2-register ami-instance-name/image.manifest.xml

And that's it!

Of course, you could be cunning and create a script that does it all in one. I've got my AWS/EC2 credentials stored in environment variables from my .bashrc:

export EC2_PRIVATE_KEY=/mnt/keys/pk-(long string).pem
export EC2_CERT=/mnt/keys/cert-(long string).pem
export AWS_ACCOUNT_ID=(my account id)
export AWS_ACCESS_KEY=(my AWS access key)
export AWS_SECRET_KEY=(my AWS secret key)

which means I can make, upload and register an instance in one, by running this script:



ec2-bundle-vol -d /mnt/images/$1 -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $AWS_ACCOUNT_ID
ec2-upload-bundle -b $1 -m /mnt/images/$1/image.manifest.xml -a $AWS_ACCESS_KEY -s $AWS_SECRET_KEY
ec2-register $1/image.manifest.xml

...and giving it a parameter of ami-instance-name. I have that script saved as make_ami.sh, so I can just call, for instance:

make_ami.sh webserver-with-sites-up-and-running

...and go have a cup of coffee while it does it's thing.


Alistair Davidson said...

PS - I should also mention that I found Rob Rohan's post extremely useful. Thanks Rob!

marcus said...

Thanks for the tutorial, took the pain out of that for me :)

One point, you have a broken line in the first few commands and you get a
ashrc on a single line.

cire333 said...

Hi have run into a snag from trying your instructions out. I only get one large image file called "image" under the mnt directory and then I get another directory that called "img-mnt" in the mnt directory that looks like the copy of my instance? Have you seen this before or can you shed some light on the problem.

Anonymous said...

You have a syntax error in your line:

export $EC2_TOOLS_DIR=~/.ec2

Should be

export EC2_TOOLS_DIR=~/.ec2

(No $ before EC2_TOOLS_DIR)

Alistair Davidson said...

Anonymous - fixed, thanks!

Raju Singh said...

Nice, but you miss the steps as how to run the vm on a para-virtualized xen environment. I reitherat the same steps but getting errors while spinning the virtual machine from aws console. I think, if you can explain, how to troubleshoot the error which the imported vm faces while spinning on the aws instance, it would really helpful to all of us.


Alistair Davidson said...

UPDATE: when I wrote this post a few years ago, this was the only way to do it. However since then, Amazon have cottoned on to the fact that this should simple, and you can now do it from within the AWS Management Console!

Just right-click on your instance, and under "Instance Management" you'll see an option for "Create Image (EBS AMI)"


صور حب said...

Thanks for the tutorial

Ramesh K said...

Thanks for sharing this informative information....You may also refer...AWK Scripting
This white paper explains about the AWK programming.

AWS snapshot said...

Great blog... This blog clearly show the importance of AWS snapshot and how it work. Thanks for sharing