How DynamoDB expressions work
DynamoDB requests describe what to read, compare or change with expressions - short strings in a syntax of their own. There are five kinds, and one request can use several:
| Expression | Used by | What it does |
|---|---|---|
KeyConditionExpression | Query | Selects the items to read: the partition key with =, optionally the sort key with =, <, <=, >, >=, BETWEEN or begins_with. |
FilterExpression | Query, Scan | Drops items from the result after they were read, by any attribute. |
ConditionExpression | PutItem, UpdateItem, DeleteItem | Lets the write happen only when the item matches; otherwise it fails with ConditionalCheckFailedException. |
UpdateExpression | UpdateItem | Changes attributes with the SET, REMOVE, ADD and DELETE clauses. |
ProjectionExpression | Query, Scan, GetItem | Lists the attributes to return, e.g. #orderDate, #total, #address.#city; all when missing. |
Attribute names and values: # and : placeholders
An expression never contains a value itself. Each value is a placeholder starting with a colon, such as :status, defined in ExpressionAttributeValues. Attribute names may be written directly, except names that are DynamoDB reserved words (there are over 570 of them, including status, name, date, size and count), names with special characters such as a dot or a dash, and names starting with a digit. Those need a placeholder starting with #, defined in ExpressionAttributeNames. Otherwise the request fails:
ValidationException: Invalid FilterExpression: Attribute name is a reserved keyword; reserved keyword: status The builder gives every attribute name a placeholder, so no reserved word can break the expression. A nested path gets one placeholder per name: address.city becomes #address.#city, and list indexes stay as they are - items[0].price becomes #items[0].#price.
Key condition vs filter expression
A key condition decides which items DynamoDB reads. A filter only decides which of the read items it returns: read capacity is consumed for every item read, including the ones the filter drops, and the 1 MB page limit applies before filtering, so a page can come back with few or no items and a LastEvaluatedKey.
- Put what you query by into the key - the table's or a secondary index's.
- Use a filter for what remains, on a result the key condition already narrowed down.
- A Scan with a filter still reads the whole table; check what it costs with the item size calculator.
Sort order, Limit and pagination
A query returns items in sort key order, ascending by default; ScanIndexForward: false reverses it, which with a date as the sort key gives the newest items first. Limit caps how many items one request reads - before the filter, so a query with Limit: 25 and a filter can return fewer than 25 items even when more match.
One request returns at most 1 MB or Limit items. When there is more, the response carries a LastEvaluatedKey; the next request passes it as ExclusiveStartKey and continues from there, until a response comes without one. With "Read every page" the builder writes that loop: a paginator (paginateQuery, paginateScan) in JavaScript and a while loop in Python. The AWS CLI paginates by itself; --max-items stops it after a number of returned items.
A ProjectionExpression makes the response smaller, not the request cheaper: read capacity is counted from the full size of every item read.
Update expressions
| Clause | Example | Effect |
|---|---|---|
SET | SET #status = :status | Adds or replaces an attribute. |
SET with arithmetic | SET #stock = #stock - :qty | Increments or decrements a number that already exists. |
if_not_exists | SET #views = if_not_exists(#views, :zero) | Sets a value only when the attribute is missing. |
list_append | SET #history = list_append(#history, :event) | Appends a list (L) to a list. |
REMOVE | REMOVE #trackingNumber | Deletes an attribute, or a list element by index. |
ADD | ADD #count :one | Adds to a number, starting from 0 when it is missing, or adds elements to a set. |
DELETE | DELETE #tags :old | Removes elements from a set. |
Each clause appears once, with its actions separated by commas: SET #a = :a, #b = :b REMOVE #c. One expression cannot change the same attribute twice.
Comparison operators and functions
Filters and conditions support =, <>, <, <=, >, >=, BETWEEN, IN (up to 100 values) and the functions attribute_exists, attribute_not_exists, attribute_type, begins_with, contains (a substring, or an element of a set or list) and size, joined with AND, OR and NOT.
Conditional writes: PutItem, UpdateItem and DeleteItem
PutItem, UpdateItem and DeleteItem - and GetItem, which reads one item by its full key - take the key of a single item. A write with a ConditionExpression happens only when the item as it is stored matches the condition; otherwise DynamoDB rejects it with ConditionalCheckFailedException and changes nothing. The check and the write are one atomic operation, so no other request can change the item in between. Typical conditions:
| Goal | Operation | Condition |
|---|---|---|
| Create an item, never overwrite one with the same key | PutItem | attribute_not_exists(#pk) |
| Update an item only if it exists (UpdateItem would create it otherwise) | UpdateItem | attribute_exists(#pk) |
| Optimistic locking: write only if nobody changed the item since you read it | PutItem, UpdateItem | #version = :version, with the version increased in the write |
| Delete only in a given state | DeleteItem | #status = :status |
attribute_not_exists on the partition key is enough even for a table with a sort key: the condition is evaluated against the one item with the full key being written, and every stored item has a partition key.
What a write returns: ReturnValues
A write returns nothing by default. ReturnValues makes it return the item in the Attributes field of the response, in the same request - no separate read, and no chance for another write to come in between:
| Value | Operations | Returns |
|---|---|---|
ALL_OLD | PutItem, UpdateItem, DeleteItem | The whole item as it was before the write - what a put replaced or a delete removed. |
ALL_NEW | UpdateItem | The whole item after the update. |
UPDATED_OLD | UpdateItem | Only the updated attributes, as they were before. |
UPDATED_NEW | UpdateItem | Only the updated attributes, as they are after - e.g. a counter's new value. |
Frequently asked questions
Why do I get "Attribute name is a reserved keyword"?
The expression uses a reserved word such as status or name directly. Replace it with a # placeholder and define the placeholder in ExpressionAttributeNames, as the builder does.
Are my table and attribute names sent anywhere?
No. The expressions and the code are built in your browser: nothing you enter is uploaded, processed on a server, stored or logged. The page only counts that the builder was used, never what you entered.
Why does my query return fewer items than the filter matches?
DynamoDB reads at most 1 MB (or Limit items) per request and filters afterwards. Keep requesting pages with ExclusiveStartKey set to the LastEvaluatedKey of the previous page until it is missing - in the builder, check "Read every page".
How do I make PutItem fail if the item already exists?
Add ConditionExpression: attribute_not_exists(#pk) with #pk mapped to the partition key name - in the builder, choose PutItem and "Only if the item does not exist". An existing item makes the put fail with ConditionalCheckFailedException instead of being replaced.
How do I get the new value of a counter after incrementing it?
Add ReturnValues: UPDATED_NEW to the UpdateItem request with SET #count = #count + :one or ADD #count :one; the response's Attributes holds the value after the increment.
Can a key condition use OR or a function other than begins_with?
No. A key condition takes exactly one partition key value and at most one sort key comparison. Anything else belongs in a filter - or in a secondary index keyed by that attribute.
Do I need the placeholders with the document client or boto3's Table?
Yes, for the expressions. They convert the values in ExpressionAttributeValues for you, so you pass plain values instead of DynamoDB JSON, but the expression syntax is the same. boto3 also has Key and Attr in boto3.dynamodb.conditions, which build the key condition and filter expressions and their placeholders for you.
References
Using expressions in DynamoDB
Condition and filter expressions, operators, and functions
Using update expressions in DynamoDB
Using projection expressions in DynamoDB
Paginating table query results
Reserved words in DynamoDB