Using Node.js, I'm querying DynamoDB for objects using a secondary index:
TableName: "Products",
IndexName: 'MerchantAndDateIndex',
KeyConditionExpression: "#creator = :creatorId",
ExpressionAttributeNames:{
"#creator": "createdBy"
},
ExpressionAttributeValues: {
":creatorId": uuid
}
What I want to do is query the "Products" table with the "createdBy" hash key accepting several different possible strings, something like this:
ExpressionAttributeValues: {
":creatorId": ["multiple","valid","uuids","here"]
}
Where every element in my dynamic array is checked against the ID in the table when running a query.
Of course, this specific implementation returns a type mismatch, but I'm not sure how to do this correctly
This is not supported by the DynamoDB API, and it would still count as multiple read requests from DynamoDB's throughput perspective.
If you need to query for multiple values you would need to implement that logic yourself in your application. Basically write a loop over the values to query and issue a query for each. That's what Dynamo would do server-side anyway.
Related
I want to store the information of user and their total_score.
Each user has a facebook_id and it will be the primary key.
I designed the table and index like that:
Table name: UserDataTest
Primary partition key: facebook_id (String)
Now I want to get top 50 user follow by total_ranking.
I wrote the query params like that:
var params = {
TableName : "UserDataTest",
ProjectionExpression:"facebook_id, facebook_name, #lev, total_star, total_crown, total_ranking, isVip",
IndexName: "facebook_id-total_ranking-index",
ExpressionAttributeNames: {
"#lev" : "level"
}
ScanIndexForward: false,
Limit: 50
};
But i could not get the data that I need. Can you give me a solution for this? Any idea for rewrite the query or redesign the table?
Thanks
It's possible that I'm not quite understanding how hash/primary keys work in DynamoDB, but I'm trying to create a model (using Serverless + Dynogels/NodeJS) for a messaging service.
The model looks like this:
const ConversationORM = dynogels.define('Conversation', {
hashKey: 'id',
timestamps: true,
tableName: config.CONVERSATION_TABLE,
schema: {
id: Joi.string(),
users: Joi.array(), // e.g. ['foo', 'bar', 'moo']
messages: Joi.array()
}
})
As you can see, users is an array, which lists the userIds of the conversation's participants.
I need to create a service which finds all conversations that a user is participating in. In MongoDB (which I'm far more familiar with), I'd do something like:
Conversation.find({users: {"$in": ['foo']} }).then(....
Is there something equivalent I can do in DynamoDB? This is an API call that will happen quite often so I'm hoping to make it as efficient as possible.
This answer takes into account a comment on Hunter Frazier's answer saying you don't want to use a Scan.
When using a Query you need specify a single partition key in the operation. In your schema this would mean partitioning on the userid attribute, which is a set. Partition keys in DynamoDB must be a top-level scalar attribute. As userid is not scalar (its a set), you cannot use this attribute as an index, and therefore you cannot do a Query for the conversations a user is part of.
If you need to do this Query, I would suggest revisiting your schema. Specifically I would suggest implementing the Adjacency list pattern which works well in databases containing many-to-many relationships.
You can see some additional notes on the article above I have written on this answer DynamoDB M-M Adjacency List Design Pattern
In your case you would have:
Primary Key: ConversationID
Sort Key: UserID
GSI Primary Key: UserID
You can then use the GSI Primary key in a Query to return all conversations the user is part of.
I'm not familiar with Dynogels or Serverless but if it uses the regular API this might work:
var params = {
ExpressionAttributeNames: {
"U": "users"
},
ExpressionAttributeValues: {
":a": {
S: "John Doe"
}
},
FilterExpression: "Author = :a",
ProjectionExpression: "#U",
TableName: "Conversations"
};
dynamodb.scan(params, function (err, data) {
if (err) console.log(err, err.stack);
else console.log(data);
});
I am trying to query dynamodb using the following code:
const AWS = require('aws-sdk');
let dynamo = new AWS.DynamoDB.DocumentClient({
service: new AWS.DynamoDB(
{
apiVersion: "2012-08-10",
region: "us-east-1"
}),
convertEmptyValues: true
});
dynamo.query({
TableName: "Jobs",
KeyConditionExpression: 'sstatus = :st',
ExpressionAttributeValues: {
':st': 'processing'
}
}, (err, resp) => {
console.log(err, resp);
});
When I run this, I get an error saying:
ValidationException: Query condition missed key schema element: id
I do not understand this. I have defined id as the partition key for the jobs table and need to find all the jobs that are in processing status.
You're trying to run a query using a condition that does not include the primary key. This is how queries work in DynamoDB. You would need to do a scan for the info in your case, however, I don't think that is the best option.
I think you want to set up a global secondary index and use that to query for the processing status.
In another answer #smcstewart responded to this question. But he provides a link instead of commenting why this error occurs. I want to add a brief comment hoping it will save your time.
AWS docs on Querying a Table states that you can do WHERE condition queries (e.g. SQL query SELECT * FROM Music WHERE Artist='No One You Know') in the DynamoDB way, but with one important caveat:
You MUST specify an EQUALITY condition for the PARTITION key, and you can optionally provide another condition for the SORT key.
Meaning you can only use key attributes with Query. Doing it in any other way would mean that DynamoDB would run a full scan for you which is NOT efficient - less efficient than using Global secondary indexes.
So if you need to query on non-key attributes using Query is usually NOT an option - best option is using Global Secondary Indexes as suggested by #smcstewart.
I found this guide to be useful to create a Global secondary index manually.
If you need to add it using CloudFormation here is a relevant page.
I was getting this error for a different scenario. Here is my scenario.
(It's very unlikely that anyone else ends up with this case, but incase)
I had a query working on a Table (say table A). Table A had a partition key m_id and sort key u_id.
I had a query to fetch data using m_id. The query was working.
'''
var queryParams = {
ExpressionAttributeValues: {
':m_id': mId
},
KeyConditionExpression: 'm_id = :m_id',
TableName: "A"
};
let connections = await docClient.query(queryParams).promise();
'''
I created another Table say Table B. I made some errors in naming keys so I simply deleted and created a table with the same name again, Table B. Table B had partition key m_id, and sort key s_id.
I copied pasted the same query which I was using for Table A, I changed Table name only because partition key had the same name.
To my shock, I get this expectation.
"ValidationException: Query condition missed key schema element"
I rechecked all the names, I compared the query with the working query. Everything was fine.
I thought maybe because, I was deleting recreating Table B, it could be something with that. So I create a fresh Table with a new Name Table B2 with the same key names as Table B.
In my query that was throwing exceptions, I changed only the Table name from B to B2.
And the Exception was gone.
If you are getting this on a fresh table, where no query has worked earlier, creating a new Table with a new name is an option.
If you delete a Table only to change partition key names, it may be safer to use a new name for Table as well (Dynamo could be referring metadata by table names and not by internal identifiers, it is possible that old metadata stays even if you delete a table. Just a guess given I faced this case).
EDIT:2022-July-12
This error does not leave me. My own answer was helpful but one more case, there was a trailing space in name of Key in the table. And Dynamo does not even check for spaces in key names.
You have to create an global secondary index for the status field.
Then, you code could look like smth like this:
dynamo.query({
TableName: "Jobs",
IndexName: 'status',
KeyConditionExpression: '#s = :st',
ExpressionAttributeValues: {
':st': 'processing'
},
ExpressionAttributeNames: {
'#s': 'status',
},
}, (err, resp) => {
console.log(err, resp);
});
Note: scan operation is indeed very costly, especially if you table is huge in size
i solved the problem using AWS.DynamoDB.DocumentClient() with scan, for sample (nodejs):
var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
TableName: "product",
FilterExpression: "#cg = :data",
ExpressionAttributeNames: {
"#cg": "categoria",
},
ExpressionAttributeValues: {
":data": category,
}
};
docClient.scan(params, onScan);
function onScan(err, data) {
if (err) {
// for the log in server
console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2));
res.json(err);
} else {
console.log("Scan succeeded.");
res.json(data);
}
}
I am using Lambda (nodeJS 4.3) to query my DynamoDB with the following:
var params = {
TableName : "shoes",
KeyConditionExpression: "gender = :gender AND support = :support AND terrain = :terrain",
ExpressionAttributeValues: {
":input": inputGender,
":input": inputSupport,
":input": inputTerrain
}
}
When this runs I am getting an error that I am missing "Query condition missed key schema element: Id". It may just be a fundamental misunderstanding on my part, but if I want to query 2 or more of the fields in the DynamoDB do I need to make them a key or create indexes on all of them??
Thanks in advance.
A query must include the partition key of the table (or a global secondary index). Your table's partition key is id and you didn't include that in the query. Given the query you are trying to run I don't think it makes sense to create a GSI on your table. You will need to perform a full table scan operation instead of a query operation.
I have a small nodejs script connected to my DynamoDB and one of my primary sort keys is datetime, which is represented in UTC.
I would like to filter the results by a certain date and time and am having issues with my Filter Expression.
var params = {
TableName: "realtimeusers",
ProjectionExpression: "brand, datetime, activeusers",
KeyConditionExpression: "brand = :brand",
FilterExpression: "datetime > :today",
ExpressionAttributeValues: {
":brand": "BRAND A",
":today": 1464705900
},
};
I get the error message:
"message": "Invalid FilterExpression: Attribute name is a reserved keyword; reserved keyword: datetime",
I think I am missing something in my filter expression that I don't fully grasp.
Can some one please help?
"datetime" is a reserved dynamo keyword. The reserved keywords are not allowed in dynamo query expressions.
The way around this is to use Expression attribute names when querying for such properties.
Reference:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ReservedWords.html
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ExpressionPlaceholders.html#ExpressionAttributeNames