Get Amazon S3 metadata in Python using boto

Another example of something that costs time because it is simply not well documented (at least at the moment of writing). When working with Amazon S3 objects (identified by keys in buckets) you may want to get meta-data associated with those keys. In my case I was storing versions (revisions) of backups in meta-data.

Looking at the examples and sample online documentation I’ve started with

keys = bucket.list() # Store for later use
...
backups = {}
for key in keys:
    backups[key.name] = key.get_metadata('revision')

Hoping to get a nice dictionary with file names and proper revisions. Right. All keys were equals to None. ‘Wattaheck’…

Inspection of S3 bucket using Amazon UI shows keys are nicely present, so there should be something with API. OK, Google time again. Apparently you need to issue a call to key.get_key() first, otherwise call to key.get_metadata('name') will always return None! Strangely enough no error is reported and this simply so ‘by design’.

So I ended up with having the following code:

backups = {}
for key in bucket:
    key = bucket.get_key(key.name)
    backups[key.name] = key.get_metadata('revision')

Which gave me nicely list of backups with associates metadata. By the way, the key on server will be prefixed with "x-amz-meta-", so ‘revision’ will become 'x-amz-meta-revision', but this is transparent for the Python interface.