This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 4 years ago.
Can someone give me a hand? I tried to figure it out but i ran out of ideas.
-------------dynamo.js....
module.exports.readUser = function (user_id) {
AWS.config = new AWS.Config();
AWS.config.update({region: "eu-west-1"});
var docClient = new AWS.DynamoDB.DocumentClient();
var table = "user";
var user_id = user_id;
var params = {
TableName: table,
Key:{
"user_id": user_id
}
};
docClient.get(params, function(err, data) {
if (err) {
return err;
} else {
//console.log(data); <-- data is filled
return data;
}
});
var dynamo = require("./dynamo.js");
console.log(dynamo.readUser(4711998));
The data you log out inside the function is only the returned value for the anonymous function given to docClient.get as the second argument. The exported readUser function has no return statement and therefore it is undefined.
Because of the async nature of javascript you have to access the result via a callback like so:
module.exports.readUser = function (user_id, cb) {
...
docClient.get(params, function(err, data) {
if (err) {
return err;
} else {
return cb(data);
}
});
}
var dynamo = require("./dynamo.js");
dynamo.readUser(4711998, function (user) {
console.log(user);
});
docClient.get is an asynchronous function. You need promisify this function or use callback.
Use callback:
module.exports.readUser = function (user_id, cb) {
AWS.config = new AWS.Config();
AWS.config.update({ region: "eu-west-1" });
var docClient = new AWS.DynamoDB.DocumentClient();
var table = "user";
var user_id = user_id;
var params = {
TableName: table,
Key: {
"user_id": user_id
}
};
docClient.get(params, cb);
}
var dynamo = require("./dynamo.js");
dynamo.readUser(4711998, function (err, user) {
if(err)
console.error(err);
else
console.log(user);
});
Use promise:
module.exports.readUser = function (user_id) {
AWS.config = new AWS.Config();
AWS.config.update({ region: "eu-west-1" });
var docClient = new AWS.DynamoDB.DocumentClient();
var table = "user";
var user_id = user_id;
var params = {
TableName: table,
Key: {
"user_id": user_id
}
};
return new Promise(function(resolve, reject) {
docClient.get(params, function(err, data) {
if (err) {
reject(err);
} else {
resolve(data);
}
});
});
}
var dynamo = require("./dynamo.js");
dynamo.readUser(4711998).then(function (user) {
console.log(user);
}).catch(function(err) {
console.error(err);
});
Read more:
How do I promisify the AWS JavaScript SDK?
https://aws.amazon.com/ru/blogs/developer/support-for-promises-in-the-sdk/
Related
batchWrite doesn't work with async in Laqmbda. The code is going to insert one record tho, it can't. However, when I remove async, It works.
const AWS = require("aws-sdk");
const documentClient = new AWS.DynamoDB.DocumentClient();
AWS.config.update({ region: "us-west-2" });
const tableName = "BlrSession-56pfbzohnvdqpac6asb627z2wu-dev";
exports.handler = async (event, context, callback) => {
try {
let games = [];
games.push({
PutRequest: {
Item: {
id: Math.random().toString(36).substring(2) + Date.now().toString(36),
},
},
});
let params = {
RequestItems: {
[tableName]: games,
},
};
documentClient.batchWrite(params, function (err, data) {
if (err) {
callback(err);
} else {
callback(null, data);
}
});
} catch (err) {
return err;
}
};
The result is below. There is no error.
Ensuring latest function changes are built...
Starting execution...
Result:
null
Finished execution.
Have you guys got the same behavior?
You can't combine the callback method with the async/await method. The easiest thing to do here is to make it all async/await (and don't forget the .promise() on the call).
const AWS = require("aws-sdk");
const documentClient = new AWS.DynamoDB.DocumentClient();
AWS.config.update({ region: "us-west-2" });
const tableName = "BlrSession-56pfbzohnvdqpac6asb627z2wu-dev";
exports.handler = async (event, context, callback) => {
try {
let games = [];
games.push({
PutRequest: {
Item: {
id: Math.random().toString(36).substring(2) + Date.now().toString(36),
},
},
});
let params = {
RequestItems: {
[tableName]: games,
},
};
return await documentClient.batchWrite(params).promise();
} catch (err) {
return err;
}
};
Consider the code :
const AWS = require('aws-sdk');
const dynamoDB = new AWS.DynamoDB({
region : 'us-east-2' ,
apiVersion: '2012-08-10'
});
exports.handler = async (event , context , callback) => {
const type = event.type;
if (type === 'all') {
// Which table we want to scan
const params = {
TableName : 'compare-yourself'
};
await dynamoDB.scan(params ,(err , data) => {
if (err) {
callback(err);
}
else{
callback(null , data);
}
});
// callback(null , "All Data");
}
else if (type === 'single') {
callback(null , "Single data");
}
else{
callback(null , "Everything else...");
}
};
When I test this piece of code on AWS Lambda with the value :
The result is always "Response: null" , even through there is data in the DynamoDB table compare-yourself.
What might be the problem here ?
Promise will solve the issue. I prefer documentclient.
const AWS = require('aws-sdk');
const documentClient = new AWS.DynamoDB.DocumentClient();
exports.handler = async (event, context, callback) => {
const type = event.type;
if (type === 'all') {
const params = {
TableName: 'compare-yourself'
};
let data = await documentClient.scan(params).promise();
callback(null, data);
} else if (type === 'single') {
callback(null, "Single data");
} else {
callback(null, "Everything else...");
}
};
The data will be an object with table values in the Items,
{
Items: [],
Count:
ScannedCount:
.....
}
try use DocumentClient();
var documentClient = new AWS.DynamoDB.DocumentClient();
const params = {
TableName : 'compare-yourself'
};
documentClient.scan(params, function(err, data) {
if (err) console.log(err);
else console.log(data);
});
Mistake:
The problem is you are doing both callback and await at the same.
Fix:
So you could put the scan function in a try block, so on error
we do callback error in the catch block and stop the function without proceeding further (return) else you will send the data.
Code:
const AWS = require('aws-sdk');
const dynamoDB = new AWS.DynamoDB({
region : 'us-east-2' ,
apiVersion: '2012-08-10'
});
exports.handler = async (event , context , callback) => {
const type = event.type;
if (type === 'all') {
// Which table we want to scan
const params = {
TableName : 'compare-yourself'
};
try {
await dynamoDB.scan(params).promise();
} catch (e) {
callback(e);
return;
}
callback(null , data);
}
else if (type === 'single') {
callback(null , "Single data");
}
else{
callback(null , "Everything else...");
}
};
Note:
Just wanted to add this, when performing scan be aware of the limits, if the limit exceeds you won't get the entire entries in the table.
Using Lambda (node.js) - how to delete all the items in the Dynamodb table?
There are 500K rows in the table
I have tried using scan method and then loop through each item and then using delete method. It only allow up to 3000 rows only.
Code
exports.handler = function(context, callback) {
getRecords().then((data) => {
data.Items.forEach(function(item) {
deleteItem(item.Id).then((data1) => {
});
});
});
};
var deleteItem = function(id) {
var params = {
TableName: "TableName",
Key: {
"Id": id
},
};
return new Promise(function(resolve, reject) {
client.delete(params, function(err, data) {
if (err) {
reject(err);
} else {
resolve();
}
});
});
}
function getRecords() {
var params = {
TableName: 'TableName',
IndexName: 'Type-index',
KeyConditionExpression: 'Type = :ty',
ExpressionAttributeValues: {
':ty': "1"
},
ProjectionExpression: "Id",
};
return new Promise(function(resolve, reject) {
client.query(params, function(err, data) {
if (err) {
reject(err);
} else {
resolve(data);
}
});
});
}
There is already one right answer, but here is another code snippet to delete all records from Dynamo DB.
const AWS = require("aws-sdk");
AWS.config.update({
region: "us-east-1",
});
const docClient = new AWS.DynamoDB.DocumentClient();
const getAllRecords = async (table) => {
let params = {
TableName: table,
};
let items = [];
let data = await docClient.scan(params).promise();
items = [...items, ...data.Items];
while (typeof data.LastEvaluatedKey != "undefined") {
params.ExclusiveStartKey = data.LastEvaluatedKey;
data = await docClient.scan(params).promise();
items = [...items, ...data.Items];
}
return items;
};
const deleteItem = (table, id) => {
var params = {
TableName: table,
Key: {
id: id,
},
};
return new Promise(function (resolve, reject) {
docClient.delete(params, function (err, data) {
if (err) {
console.log("Error Deleting ", id,err);
reject(err);
} else {
console.log("Success Deleting ", id,err);
resolve();
}
});
});
};
exports.handler = async function (event, context, callback) {
try {
const tableName = "<table>";
// scan and get all items
const allRecords = await getAllRecords(tableName);
// delete one by one
for (const item of allRecords) {
await deleteItem(tableName, item.id);
}
callback(null, {
msg: "All records are deleted.",
});
} catch (e) {
callback(null, JSON.stringify(e, null, 2));
}
};
A Scan operation consumes Read capacity. Each Read returns up to 4 kb of data. When this limit is reached, the Scan returns only what it has found until there. If you need more, you need to issue another Scan request.
This, you'll need two loops: 1) loop to delete all records returned at each Scan; 2) loop to keep scanning multiple times, until you reach the end of the table
Make sure you use consistent Reads or wait 1 or 2 second(s) before issuing another Scan, otherwise you may get repeated items in different Scans.
exports.handler = function(context, callback) {
clearRecords();
};
clearRecords = function() {
getRecords().then((data) => {
data.Items.forEach(function(item) {
deleteItem(item.Id).then((data1) => {});
});
clearRecords(); // Will call the same function over and over
});
}
Observe that Lambda has a timeout limit of 15 minutes. Since you have 500K items in your table, it's likely that your Lambda will timeout and you'll need to trigger it more than once. You could also make your Lambda call itself after 14:50, for example, just take a look at the AWS SDK documentation for triggering Lambda functions. For this matter, you might also want to check the getRemainingTimeInMillis() method from the context object.
I'm working with DynamoDB for the first time and so far, not too bad. Having some understanding of Node is making this easier, but having none of noSQL, well that hurts!. My application relies on quickly searching a GROUPS list and returning members from that group.
When I add the data using the console, I am able to create a JSON document like this:
"GROUP_ID":"GroupA", "MEMBER": ["MemberA", "MemberB", "MemberC"], "STATUS": ["OWNER", "MEMBER", "INVITED"], "ADD_DATE":[1234567, 2345671, 3456712]
I am using this to create the GROUP_ID as the primary key:
const AWS = require('aws-sdk');
const docClient = new AWS.DynamoDB.DocumentClient({region:'us-east-1'});
exports.handler = (event, context, callback) => {
var tableName = "USER_GROUPS";
var checkParams = {
TableName: tableName,
Key:{"GROUP_NAME": event.group_id}
};
var createParams = {
TableName: tableName,
Key:{"GROUP_NAME": event.group_id},
Item:{"GROUP_NAME": event.group_id, "MEMBER_NAME":[event.device_id], "MEMBER_SATUS":["OWNER"], "MEMBER_DATE" : [event.date]}
};
var checkGroupExists = new Promise((resolve, reject) => {
docClient.get(checkParams, (err, data) => {
if(err){
reject(err);
}
if(data.Item){ // exists...
console.log("found Group ID");
reject();
}
else{
console.log("Group ID not found...");
resolve();
}
});
});
checkGroupExists.then((err, data) => {
console.log("adding Group: ");
console.log(data);
docClient.put(createParams, (err, data) =>{
if(err){
console.log(err);
callback(err, null);
}else{
console.log("added device: " + createParams.Key);
callback(null, {"created":createParams.Key.GROUP_NAME});
}
});
}).catch((err) => {
if(err){
callback(err, null);
}
else{
callback(err, {"used":createParams.Key.GROUP_NAME});
}
});
};
but when I use update() I am adding to the array and end up with a type name for the arrays
["S":"MemberA", "S":"MemberB", "S":"MemberC"]
kind of thing....
This is where I have left off, quite frustrated here!
const AWS = require('aws-sdk');
const docClient = new AWS.DynamoDB.DocumentClient({region:'us-east-1'});
exports.handler = (event, context, callback) => {
var tableName = "USER_GROUPS";
var checkParams = {
TableName: tableName,
Key:{"GROUP_NAME": event.group_id},
Item:{"MEMBER_NAME": event.member_name}
};
var updateParams = {
TableName: tableName,
Key:{"GROUP_NAME": event.group_id},
//Item:{"MEMBER_NAME":[event.device_id], "MEMBER_STATUS":["OWNER"], "MEMBER_DATE" : [event.date]},
AttributeUpdates: {
"MEMBER_NAME": {
Action: 'ADD',
Value: event.device_id
},
"MEMBER_STATUS":{
Action : 'ADD',
Value: "ENVITED"
},
"MEMBER_DATE":{
Action: 'ADD',
Value: event.date
}
},
};
var checkGroupExists = new Promise((resolve, reject) => {
docClient.get(checkParams, (err, data) => {
console.log(data);
if(err){
reject(err);
}
if(data.Item.GROUP_NAME){ // exists...
console.log("found the group...");
resolve();
}
else{
reject();
}
});
});
checkGroupExists.then((err, data) => {
docClient.update(updateParams, (err, data) =>{
if(err){
console.log(err);
callback(err, null);
}else{
callback(null, {"updated" : updateParams.Key.GROUP_NAME});
}
});
}).catch((err) => {
if(err){
callback(err, null);
}
else{
callback(err, {"noGroup":createParams.Key.GROUP_NAME});
}
});
};
It seems I am lacking the understanding of how to process updates to include NEW item in the member array. Any assistance would be appreciated!
Use list_append() together in an UpdateExpression to append to a list column along with a ConditionExpression to prevent updates to a non-existent group:
const AWS = require('aws-sdk');
const DB = new AWS.DynamoDB.DocumentClient({region:'us-east-1'});
const UpdateExpression = 'SET ' + [
'#members = list_append(#members, :member)',
'#statuses = list_append(#statuses, :status)',
'#dates = list_append(#dates, :date)',
].join(', ')
const ConditionExpression = [
'attribute_type(#members, :L)',
'attribute_type(#statuses, :L)',
'attribute_type(#dates, :L)'
].join(' AND ')
const ExpressionAttributeNames = {
'#members': 'MEMBER_NAME',
'#statuses': 'MEMBER_STATUS',
'#dates': 'MEMBER_DATE'
}
exports.handler = (event, context, callback) => {
DB.update({
TableName: 'USER_GROUPS',
Key: { GROUP_NAME: e.group_id },
ReturnValues: 'ALL_NEW',
UpdateExpression: UpdateExpression,
ConditionExpression: ConditionExpression,
ExpressionAttributeNames: ExpressionAttributeNames,
ExpressionAttributeValues: {
':member': [e.member_name],
':status': ['ENVITED'],
':date': [e.date],
':L': 'L'
}
}, callback)
}
So I need a recursive function in node.js for replacing this function call:
docClient.scan(params, callback)
More info see http://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/GettingStarted.NodeJs.04.html
Here is the recursive code to execute the scan until LastEvaluatedKey is available.
var AWS = require("aws-sdk");
var creds = new AWS.Credentials('akid', 'secret', 'session');
AWS.config.update({
region: "us-west-2",
endpoint: "http://localhost:8000",
credentials : creds
});
var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
TableName: "Movies"
};
console.log("Scanning Movies table.");
docClient.scan(params, onScan);
var count = 0;
function onScan(err, data) {
if (err) {
console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2));
} else {
// print all the movies
console.log("Scan succeeded.");
data.Items.forEach(function(movie) {
console.log("Item :", ++count,JSON.stringify(movie));
});
// continue scanning if we have more movies
if (typeof data.LastEvaluatedKey != "undefined") {
console.log("Scanning for more...");
params.ExclusiveStartKey = data.LastEvaluatedKey;
docClient.scan(params, onScan);
}
}
}
Recursive DynamoDB Scan using AWS SDK V3.
import {
DynamoDBClient,
ScanCommand,
ScanCommandOutput,
ScanCommandInput,
} from '#aws-sdk/client-dynamodb';
import { unmarshall, marshall } from '#aws-sdk/util-dynamodb';
const client = new DynamoDBClient({ region: 'eu-west-1' });
export const scanTable = async<ResultType>(params: ScanCommandInput): Promise<ResultType[]> => {
const scanParams: ScanCommandInput = params;
const results = [];
let lastEvaluatedKey: string;
do {
console.log(`Running query: ${JSON.stringify(scanParams)}`);
const { Items, LastEvaluatedKey } = await client.send(new ScanCommand(scanParams));
lastEvaluatedKey = LastEvaluatedKey;
Items.forEach((item) => results.push(unmarshall(item)));
scanParams.ExclusiveStartKey = LastEvaluatedKey;
} while (typeof lastEvaluatedKey !== 'undefined');
return results as ResultType[];
};
const items = await scanTable({ TableName: 'dynamo-table' })