I m using node typescript and local dynamodb. I'm unable to get a specific nested document.
I m trying to get a post on the basis of id. I m using email as pk.
let params1:any= {
TableName: "userTable",
// Key: {
// email: event.body.email
// },
FilterExpression: "#posts[0].#id = :postId",
// KeyConditionExpression: `#postId= :postId`,
ExpressionAttributeNames: {
"#posts": "posts",
"#id": "id",
},
ExpressionAttributeValues: {
":postId": event.body.id
},
}
As per comments.
you need to remodel your table design by using composite key structure.
composite key = email = hash key, postId = range/sort key
note:- if you are using post-operation to query DB, event.body.id should be req.body.id.
you can simply use dynamodb get operation to grab single record
const getRecordParams = {
TableName: tableName,
Key: {
email: req.body.id, // hash key
postId: req.body.postId //range key
},
};
const dynamoDbGetResults = await dynamoDb
.get(getRecordParams)
.promise();
dynamoDbGetResults will have your entire record from which you can extract your desired values.
Note:- if you have constraints in changing table design an alternate solution would be to create index based on postId. Using that you can query directly on index using postId.
dynamoDb supports local secondary index and Global secondary Index.
they both have difference in terms of pricing, finding differences between them priorly for your use case would be good. https://stackoverflow.com/a/21383613/13126651
Related
I want to store the information of user and their total_score.
Each user has a facebook_id and it will be the primary key.
I designed the table and index like that:
Table name: UserDataTest
Primary partition key: facebook_id (String)
Now I want to get top 50 user follow by total_ranking.
I wrote the query params like that:
var params = {
TableName : "UserDataTest",
ProjectionExpression:"facebook_id, facebook_name, #lev, total_star, total_crown, total_ranking, isVip",
IndexName: "facebook_id-total_ranking-index",
ExpressionAttributeNames: {
"#lev" : "level"
}
ScanIndexForward: false,
Limit: 50
};
But i could not get the data that I need. Can you give me a solution for this? Any idea for rewrite the query or redesign the table?
Thanks
It's possible that I'm not quite understanding how hash/primary keys work in DynamoDB, but I'm trying to create a model (using Serverless + Dynogels/NodeJS) for a messaging service.
The model looks like this:
const ConversationORM = dynogels.define('Conversation', {
hashKey: 'id',
timestamps: true,
tableName: config.CONVERSATION_TABLE,
schema: {
id: Joi.string(),
users: Joi.array(), // e.g. ['foo', 'bar', 'moo']
messages: Joi.array()
}
})
As you can see, users is an array, which lists the userIds of the conversation's participants.
I need to create a service which finds all conversations that a user is participating in. In MongoDB (which I'm far more familiar with), I'd do something like:
Conversation.find({users: {"$in": ['foo']} }).then(....
Is there something equivalent I can do in DynamoDB? This is an API call that will happen quite often so I'm hoping to make it as efficient as possible.
This answer takes into account a comment on Hunter Frazier's answer saying you don't want to use a Scan.
When using a Query you need specify a single partition key in the operation. In your schema this would mean partitioning on the userid attribute, which is a set. Partition keys in DynamoDB must be a top-level scalar attribute. As userid is not scalar (its a set), you cannot use this attribute as an index, and therefore you cannot do a Query for the conversations a user is part of.
If you need to do this Query, I would suggest revisiting your schema. Specifically I would suggest implementing the Adjacency list pattern which works well in databases containing many-to-many relationships.
You can see some additional notes on the article above I have written on this answer DynamoDB M-M Adjacency List Design Pattern
In your case you would have:
Primary Key: ConversationID
Sort Key: UserID
GSI Primary Key: UserID
You can then use the GSI Primary key in a Query to return all conversations the user is part of.
I'm not familiar with Dynogels or Serverless but if it uses the regular API this might work:
var params = {
ExpressionAttributeNames: {
"U": "users"
},
ExpressionAttributeValues: {
":a": {
S: "John Doe"
}
},
FilterExpression: "Author = :a",
ProjectionExpression: "#U",
TableName: "Conversations"
};
dynamodb.scan(params, function (err, data) {
if (err) console.log(err, err.stack);
else console.log(data);
});
I am trying to query dynamodb using the following code:
const AWS = require('aws-sdk');
let dynamo = new AWS.DynamoDB.DocumentClient({
service: new AWS.DynamoDB(
{
apiVersion: "2012-08-10",
region: "us-east-1"
}),
convertEmptyValues: true
});
dynamo.query({
TableName: "Jobs",
KeyConditionExpression: 'sstatus = :st',
ExpressionAttributeValues: {
':st': 'processing'
}
}, (err, resp) => {
console.log(err, resp);
});
When I run this, I get an error saying:
ValidationException: Query condition missed key schema element: id
I do not understand this. I have defined id as the partition key for the jobs table and need to find all the jobs that are in processing status.
You're trying to run a query using a condition that does not include the primary key. This is how queries work in DynamoDB. You would need to do a scan for the info in your case, however, I don't think that is the best option.
I think you want to set up a global secondary index and use that to query for the processing status.
In another answer #smcstewart responded to this question. But he provides a link instead of commenting why this error occurs. I want to add a brief comment hoping it will save your time.
AWS docs on Querying a Table states that you can do WHERE condition queries (e.g. SQL query SELECT * FROM Music WHERE Artist='No One You Know') in the DynamoDB way, but with one important caveat:
You MUST specify an EQUALITY condition for the PARTITION key, and you can optionally provide another condition for the SORT key.
Meaning you can only use key attributes with Query. Doing it in any other way would mean that DynamoDB would run a full scan for you which is NOT efficient - less efficient than using Global secondary indexes.
So if you need to query on non-key attributes using Query is usually NOT an option - best option is using Global Secondary Indexes as suggested by #smcstewart.
I found this guide to be useful to create a Global secondary index manually.
If you need to add it using CloudFormation here is a relevant page.
I was getting this error for a different scenario. Here is my scenario.
(It's very unlikely that anyone else ends up with this case, but incase)
I had a query working on a Table (say table A). Table A had a partition key m_id and sort key u_id.
I had a query to fetch data using m_id. The query was working.
'''
var queryParams = {
ExpressionAttributeValues: {
':m_id': mId
},
KeyConditionExpression: 'm_id = :m_id',
TableName: "A"
};
let connections = await docClient.query(queryParams).promise();
'''
I created another Table say Table B. I made some errors in naming keys so I simply deleted and created a table with the same name again, Table B. Table B had partition key m_id, and sort key s_id.
I copied pasted the same query which I was using for Table A, I changed Table name only because partition key had the same name.
To my shock, I get this expectation.
"ValidationException: Query condition missed key schema element"
I rechecked all the names, I compared the query with the working query. Everything was fine.
I thought maybe because, I was deleting recreating Table B, it could be something with that. So I create a fresh Table with a new Name Table B2 with the same key names as Table B.
In my query that was throwing exceptions, I changed only the Table name from B to B2.
And the Exception was gone.
If you are getting this on a fresh table, where no query has worked earlier, creating a new Table with a new name is an option.
If you delete a Table only to change partition key names, it may be safer to use a new name for Table as well (Dynamo could be referring metadata by table names and not by internal identifiers, it is possible that old metadata stays even if you delete a table. Just a guess given I faced this case).
EDIT:2022-July-12
This error does not leave me. My own answer was helpful but one more case, there was a trailing space in name of Key in the table. And Dynamo does not even check for spaces in key names.
You have to create an global secondary index for the status field.
Then, you code could look like smth like this:
dynamo.query({
TableName: "Jobs",
IndexName: 'status',
KeyConditionExpression: '#s = :st',
ExpressionAttributeValues: {
':st': 'processing'
},
ExpressionAttributeNames: {
'#s': 'status',
},
}, (err, resp) => {
console.log(err, resp);
});
Note: scan operation is indeed very costly, especially if you table is huge in size
i solved the problem using AWS.DynamoDB.DocumentClient() with scan, for sample (nodejs):
var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
TableName: "product",
FilterExpression: "#cg = :data",
ExpressionAttributeNames: {
"#cg": "categoria",
},
ExpressionAttributeValues: {
":data": category,
}
};
docClient.scan(params, onScan);
function onScan(err, data) {
if (err) {
// for the log in server
console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2));
res.json(err);
} else {
console.log("Scan succeeded.");
res.json(data);
}
}
I have a table called followProduct in Dynamodb and it has following strucure
id - item id
email - user email
product - product id
Whenever a user follows a product I am making an entry in the table. I am trying to stop duplicate entry and using the following code
let params = {
TableName: "followProduct",
ConditionExpression: "email <> :email AND product <> :pid",
Item: {
email: "a#a.com",
product: req.body.productId,
id: shortid.generate()
},
ExpressionAttributeValues: {
':email': "a#a.com",
":pid": req.body.productId
}
};
createItemInDDB(params).then(() => {
res.status(200).send("Company Added");
}, err => {
console.log(err);
res.sendStatus(500);
});
CreateItemInDDB is just a function that takes params as input and run put function provided by document client. This params is still making a duplicate entry. I want that for every email each product id should be entered only once.
can you describe your table hash-range keys?
Dynamodb can force uniqueness only for hash-range table keys (not for global secondary index keys)
from http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html
To prevent a new item from replacing an existing item, use a conditional expression that contains the attribute_not_exists function with the name of the attribute being used as the partition key for the table. Since every record must contain that attribute, the attribute_not_exists function will only succeed if no matching item exists.
and http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html:
The PutItem operation will overwrite an item with the same key (if it exists). If you want to avoid this, use a condition expression. This will allows the write to proceed only if the item in question does not already have the same key:
Im currently having an issue where i want to specify to the bookshelf model which field to use to make the data relation.
it seems to always use the id (primary key) of the model table; as far as i've found out it's only possible to set column for the relation but not which to use from model.
for example:
var StopsWithCustomer = bookshelf.Model.extend({
tableName: 'stops',
customers: function () {
return this.hasOne(customerWithStop, 'id');
}
});
it has to match on customerWithStop on the column id but it has to use the column customer_id from 'stops' to make this relation; is there any way to specify this?
Besides tableName Bookshelf.js also provides an idAttribute property. That will allow you to override Bookshelf.js' id default.
Note the second argument of the relationship (like your hasOne()) is the foreign key, not the target's primary key.
Example:
var Language = bookshelf.Model.extend({
tableName: 'languages',
idAttribute: 'languageid'
});
var Post = bookshelf.Model.extend({
tableName: 'posts',
idAttribute: 'postid',
Language: function() {
return this.belongsTo(Language,'languageid');
}
});