I want to retrieve items from the table where an attribute is equal to a value I specify using dynomodb.
The SQL equivalent is.
SELECT * FROM tokens where type='app_access_token`
Code:
copnst db = new Dynamodb.DocumentClient(credentials);
const params = {
TableName: 'TOKEN',
Key: {
type: 'app_access_token'
}
};
const response = db.get(params).promise();
But I believe this will only let me get via primary key, is that right?
No SQL WorkBench Structure
DynamoDB's equivalent of SELECT * is the scan method. They are both scanning all of the table records (and hence, slow and expensive).
You can learn more about the scan method in Amazon Developer Guide or in this useful thread.
If you can, I would use a GSI with the attribute type as key, but if you still want to do it with scan method, here's how you do it:
const params = {
TableName : 'TOKEN',
FilterExpression : 'type = :type',
ExpressionAttributeValues : {':type' : 'app_access_token'}
};
const documentClient = new AWS.DynamoDB.DocumentClient();
const response = documentClient.scan(params).promise();
AWS JavaScript SDK - DocumentClient Scan()
Related
I want to get data form dynamoDB, shorted by timestamp. Anyone can help? My code is given below.
const AWS = require("aws-sdk");
const dynamoDbClient = new AWS.DynamoDB.DocumentClient();
const USERS_TABLE = process.env.USERS_TABLE;
const getNews = async (req, res) => {
try {
//dynamodb params
const params = {
TableName: USERS_TABLE,
FilterExpression: "PK = :this",
ExpressionAttributeValues: { ":this": "newsTable" },
};
//get dynamodb data
const data = await dynamoDbClient.scan(params).promise();
res.status(200).send({ data: data });
} catch (e) {
return res.status(400).send({ message: e.message });
}
};
module.exports = { getNews };
Option 1: Keep Scan; Sort client-side
Works for small tables only. Single Scan call will scan only the first 1 MB of data in the table.
If you're doing scan operation as in your code example, it's impossible to get results sorted from DynamoDB. The only way to sort them is on client-side after you download all your data from database.
Replace:
res.status(200).send({ data: data });
With:
res.status(200).send({data: data.sort((a, b) => b.date - a.date)});
However, this is not recommended, since Scan operation without pagination will scan only 1st MB of data in your table. So you could get partial results. Possible solutions are:
Option 2: (recommended) Don't use Scan; Use Query; Sort by secondary key
This will work if you have your timestamp in the secondary key of the table
Don't use Scan; Use Query -- that way you can sort your data by SK (secondary key) by passing the ScanIndexForward: false to get the most recent results first.
Assuming you have such a table schema, where a timestamp is in the secondary key:
PK
SK
email
newsTable
2022-01-01
some-1#example.com
newsTable
2022-02-01
some-2#example.com
newsTable
2022-03-01
some-3#example.com
You can change your code from:
const params = {
TableName: USERS_TABLE,
FilterExpression: 'PK = :this',
ExpressionAttributeValues: {':this': 'newsTable'},
};
//get dynamodb data
const data = await dynamoDbClient.scan(params).promise();
To:
const params = {
TableName: USERS_TABLE,
KeyConditionExpression: 'PK = :this',
ExpressionAttributeValues: {':this': 'newsTable'},
ScanIndexForward: false,
};
//get dynamodb data
const data = await dynamoDbClient.query(params).promise();
And it will return results sorted from database already.
If you don't have a timestamp in your secondary key, and you cannot add it, you can add Local Secondary Index or Global Secondary Index.
Option 3: Keep Scan, but paginate; Sort client-side
Works if you cannot change DB schema and cannot switch your code to the Query operation.
Beware, it will be much more expensive, much slower. The larger the table, the slower it gets.
If you absolutely need to use Scan, you need to paginate through all the pages of the Scan operation, and then sort results in the JS code, like I described before. I've developed a handy library that makes scanning in parallel and supports pagination.
I am trying to update my existing record in my dynamodb table.
Like below I have an Item in my table
let params = {
TableName: proces.env.dynamoDbTable,
Item: {
productId: "id",
att1: val1,
att2: val2
}
}
I want to perform an update. I am using the aws dynamodb sdk's update method and passing it params like below
let aws = require('aws-sdk');
let dbb = new aws.DynamoDb.DocumentClient();
let params = {
TableName: process.env.tableName,
Key: {productID}
ExpressionAttributeNames: { "#updatedAt" : "updatedAt" }
ExpressionAttributeValues: {":u":moment().unix(), ":val1" : a, ":val2": b}
UpdateExpression: "SET att1 = :val1, att2: val2, #updatedAt: :u"
}
// a, b are passed as argument to function and are optional
dbb.update(params).promise()
When an argument goes missing the dynamo raises ExpressionAttributeValue missing exception and I know it is straight. Is there a way I can update my Item with the attributes provided
Unfortunately dynamodb does not make this easy. But you can use some fancy js to create the params object dynamically:
// for some object `attrs` we find which keys we need to update
const keys = Object.keys(attrs);
const values = Object.values(attrs);
// get a list of key names and value names to match the dynamodb syntax
const attributeKeyNames = keys.map((k) => '#key_' + k);
const attributeValueNames = keys.map((k) => ':val_' + k);
// create individual expressions for each attribute that needs to be updated
const expressions = attributeValueNames.map((attr, i) => {
return `${attributeKeyNames[i]} = ${attr}`;
});
// add the SET keyword to the beginning of the string
// and join all the expressions with a comma
const UpdateExpression = 'SET ' + expressions.join(', ');
// I use `zipObject()` from lodash https://lodash.com/docs/4.17.15#zipObject
// it makes an object map from two arrays where the first is the keys and
// the second is the values
const ExpressionAttributeValues = _.zipObject(attributeValueNames, values);
const ExpressionAttributeNames = _.zipObject(attributeKeyNames, keys);
// now you have all the params
const params = {
TableName,
Key: {
uuid,
},
UpdateExpression,
ExpressionAttributeValues,
ExpressionAttributeNames,
};
Lucas D's answer works great. A couple of things to keep in mind though:
If you're sending an object with a unique key into your update function, you must remove that key before mapping your expression arrays or it will be included in the query and will get an error:
const newObject = {...originalObject}
delete newObject.unique_key
If you can't or don't want to use Lodash, you can use Object.assign to map your keys/values arrays:
const ExpressionAttributeValues = Object.assign(...attributeValueNames.map((k, i) => ({[k]: values[i]})));
const ExpressionAttributeNames = Object.assign(...attributeKeyNames.map((k, i) => ({[k]: keys[i]})));
Below is the code of my Lambda function. I'm having trouble querying rows based on the timestamps. My plan is to get all the rows from 5 seconds before the current time to the current time in milliseconds. TimeMillis(Number) stores the current time in miiliseconds and it is the primary key and the range key is PhoneId(String). Please help me with the solution or is there any way to overcome the problem?
I'm not able to get the output, it is throwing error.
'use strict';
var AWS = require("aws-sdk");
AWS.config.update({
region: "us-east-1",
});
var docClient = new AWS.DynamoDB.DocumentClient();
exports.handler = function(event, context, callback) {
var timemillis = new Date().getTime();
var timemillis1 = timemillis - 5000;
var params = {
TableName: 'Readings',
KeyConditionExpression: "TimeMillis = :tm and TimeMillis BETWEEN :from AND :to",
ExpressionAttributeValues: {
":tm" : "TimMillis",
":from" : timemillis1,
":to" : timemillis
}
};
docClient.query(params, function(err, data) {
if(err){
callback(err, null);
}
else{
callback(null, data);
}
});
};
Here is my DynamoDB table image.
You cannot have multiple conditions inside a KeyConditionExpression. What you can do is use a FilterExpression with KeyConditionExpression to narrow down the result set.
Quoting from the documentation,
Use the KeyConditionExpression parameter to provide a specific value
for the partition key. The Query operation will return all of the
items from the table or index with that partition key value. You can
optionally narrow the scope of the Query operation by specifying a
sort key value and a comparison operator in KeyConditionExpression. To
further refine the Query results, you can optionally provide a
FilterExpression. A FilterExpression determines which items within the
results should be returned to you. All of the other results are
discarded.
Also for the test, only supported test for partition key is equality. Other conditions can be applied to sort key.
partitionKeyName = :partitionkeyval AND sortKeyName = :sortkeyval
Another way is to create a GSI which support further querying. By the way, traditional RDBMS thinking would not work best with DynamoDB. You can read about best practices here.
I am using ancestor query to retrieve the entity from google datastore using nodejs
query = datastore.createQuery(entity).hasAncestor(key)
where the key is
key = datastore.key([kind_name_of_parent, id_of_parent])
I am able to retrieve the objects but i would like to get the complete key of the retrieved object, whereas the returned array only contains the returned objects and the endCursor.
How can i get the complete key? or, can i get the complete key
from the endCursor?
An example for my query result is:
[{ modTS: 1481006473081,
modLoc: null,
modUid: 0,
createTS: 1481006473081 } ],
{ moreResults: 'NO_MORE_RESULTS',
endCursor: 'CloSVGoTc350ZXN0cHJvamVjdC0zN2ZiNnI9CxIEdXNlchiAgID409OICgwLEgRzaW1zGICAgICAgIAKDAsSDmNsaWVudFNldHRwsdrfGICAgICA5NEKDBgAIAA=' } ]
Since datastore client v0.42.2 the key is now referred using a Symbol on the datastore client datastoreClient.KEY.
Run this on the CLI, if it doesn't work the first time run it again (first time might fail because of 'eventual consistency').
'use strict';
const Datastore = require('#google-cloud/datastore'),
projectId = 'your-project-id',
datastore = Datastore({
projectId: projectId
}),
pkind = 'Foo',
pname = 'foo',
kind = 'Bar',
name = 'bar',
parentKey = datastore.key([pkind, pname ]),
entityKey = datastore.key([pkind, pname, kind, name]),
entity = {
key: entityKey,
data: {
propa: 'valuea'
}
},
query = datastore.createQuery().hasAncestor(parentKey).limit(5);
let complete = false;
datastore.save(entity).then(() => {
datastore.runQuery(query).then((res) => {
try {
console.log('parent key ', res[0][0][datastore.KEY].parent);
} finally {
complete = true;
}
});
});
function waitUntilComplete() {
if (!complete)
setTimeout(waitUntilComplete, 1000);
}
waitUntilComplete();
The latest update to the datastore SDK has changed the way keys are accessed in an entity.
Earlier the entity had a key called key which had a complete JSON object with information of the key.
After the update, the key is now referred using Symbols, a new ES6 datatype.
The key will be referenced using entity[datastoreClient.KEY] assuming datastoreClient is correctly authenticated/initialised object of the datasotore SDK.
The keys were retrieved by:
res[0][0][datastore.KEY]
I have this table:
DomainId string HashKey
EmailId string RangeKey
I was wondering if it's possible query this table with HashKey only, like this:
var AWS = require("aws-sdk");
var client = new AWS.DynamoDB.DocumentClient();
var dm = 'infodinamica.cl';
//Set params
var params = {
TableName : 'table-name',
KeyConditionExpression: "DomainId = :dm",
ExpressionAttributeValues: {
":dm": dm
},
Select: 'COUNT'
};
client.query(params, (err, data) => {
if(err)
console.log(JSON.stringify(err, null, 2));
else
console.log(JSON.stringify(data, null, 2));
}
ps: note that this table has HashKey and RangeKey.
Yes, it is possible to query the data using Hash Key only using query API.
Use the KeyConditionExpression parameter to provide a specific value
for the partition key. The Query operation will return all of the
items from the table or index with that partition key value. You can
optionally narrow the scope of the Query operation by specifying a
sort key value and a comparison operator in KeyConditionExpression.
You can use the ScanIndexForward parameter to get results in forward
or reverse order, by sort key.