Check out how we implemented the boto3 connection.

2. Validate your data

DynamoDB supports several data types natively that your language will probably handle a bit differently.

In our case, it's nice to convert our data to native python objects where appropriate:

• date and datetime objects, for dates and timestamps (we always store date-like data as integers, for easy comparison)
• booleans
• empty attributes (to avoid pesky "AttributeValue may not contain an empty string" errors)
• Automatic UUID generation (see #3)

Fortunately, we have that work all done and tested as part of cc_dynamodb3. Check out models.py here and here especially.

We used schematics to create a light ORM.

3. Consider a UUID HashKey for your primary key

A table's primary key cannot be changed once created. There are lots of resources out there to help you design your tables, and I suggest you research your use cases ahead of time.

In our experience, the safest choice is a string UUID Hash Key. It gives you 100% flexibility over how you're going to uniquely identify your data. You can always add, modify or remove GSIs to improve performance for specific query operations.

Here are some easily avoidable mistakes:

• Don't include a RangeKey in an index unless you actually have multiple records with the same HashKey. It makes it a pain to GetItem directly, since you'll need both the Hash and Range keys to find it. (PS. range key = sort key)
• Choose the primary key carefully. You cannot change the primary key once the table is created, and then you'll have to deal with migrating your data to a new table
• When you update an item via PutItem, DynamoDB will create a new entry if the primary key has changed. The old item will remain unchanged!

4. Consider a composite RangeKey for extra flexibility

You can use a begins_with filter on a RangeKey, but you cannot use it on a HashKey.

For example, say you have a book library. If you first and foremost care to keep your books by publisher, that could be your HashKey. Then, you may sort the books by year of publication, and uniquely identify them by their ISBN. So you can have:

• HashKey: publisher ID (or publisher name), e.g. HarperBusiness
• RangeKey: year + ISBN, e.g. 1995-9512512

You can find all HarperBusiness books published in 1995 via a single query using begins_with, what great performance!

5. Scans and queries do not return all data in the table.

Here is an example of how we do this in the light ORM from cc_dynamodb3:

class DynamoDBModel(Model):
  # moar stuff here...

  @classmethod
  def all(cls):
      response = cls.table().scan()
      # DynamoDB scan only returns up to 1MB of data, so we need to keep scanning.
      while True:
          metadata = response.get('ResponseMetadata', {})
          for row in response['Items']:
              yield cls.from_row(row, metadata)
          if response.get('LastEvaluatedKey'):
              response = cls.table().scan(
                  ExclusiveStartKey=response['LastEvaluatedKey'],
              )
          else:
              break

Caveat: Item.all() may do multiple queries to DynamoDB, and thus has a hidden cost and time. Please note the use of yield to lazy-evaluate the results. This gives you an opportunity to retrieve data only until you need to, and avoids scanning the whole table in one go.

In practice, you don't want to perform table scans or large queries on the main sever thread anyway.

6. Include created and updated columns in each table

These days being the days of big data and analytics, I suggest always having a created and an updated column.

In our light ORM, we can do this with a BaseModel:

from schematics import types as fields
from cc_dynamodb3.models import DynamoDBModel


class BaseModel(DynamoDBModel):
    created = fields.DateTimeType(default=DynamoDBModel.utcnow)
    updated = fields.DateTimeType(default=DynamoDBModel.utcnow)

    def save(self, overwrite=False):
        self.updated = DynamoDBModel.utcnow()
        return super(DynamoDBModel, self).save(overwrite=overwrite)

Then, in your code:

from myproject.db import BaseModel


class Book(BaseModel):
    publisher = fields.StringType(required=True)

This makes sure all your models inheriting from BaseModel will have the two columns automatically populated. Ta-da!

If you'd enjoy working on these types of projects while having a chance to help make aging better, ClearCare is hiring full-time engineers! Check out their careers section.