AWS Documentation Sucks, I can't seem to get an idea of how to "scan" DynamoDb with two or more where conditions, in NodeJS ?
Here's my code
const AWS = require("aws-sdk");
let documentClient = new AWS.DynamoDB.DocumentClient();
module.exports.RetryIncompleteCSV = async (event, context, callback) => {
console.log('In RetryIncompleteCSV');
const cmpId = event.queryStringParameters.campaign;
console.log('cmpId = ', cmpId);
var params = {
TableName : 'processed_csv_developments',
FilterExpression : 'campaignId = :campaignId',
ExpressionAttributeValues : {':campaignId' : cmpId}
};
let incompleteCsvData = await scanIncompleteProcessedCsv(params);
console.log('incompleteCsvData = ', incompleteCsvData);
}
async function scanIncompleteProcessedCsv(params) { // get method fetch data from dynamodb
console.log(params)
const queryExecute = await documentClient.scan(params, function(err, data) {
if (err) console.log(err);
else console.log(data);
console.log('queryExecute=',queryExecute)
});
}
I need scan to filter out data, based on following two conditions:
campaignId must be equal to the campaignId that I get from query params.
isComplete should be equal to false (Boolean)
The code written above neither give any error nor a success message. Attaching the cloudwatch logs screenshot below :
What am I missing. I looked at a lot of questions and tutorials, but nothing worked for me. Please Help!
scanIncompleteProcessedCsv should return a promise. Now it returns void, this mean RetryIncompleteCSV function will finish before a querying to dynamoDB complete.
async function scanIncompleteProcessedCsv(params) { // get method fetch data from dynamodb
console.log(params)
return await documentClient.scan(params).promise();
}
About multiple conditions, I found to many documents about that, just try:
var params = {
TableName : 'processed_csv_developments',
FilterExpression : 'campaignId = :campaignId AND isComplete = false',
ExpressionAttributeValues : {':campaignId' : cmpId}
};
Ref: Link
Related
So for anyone familiar with CW Logs..this is not the typical DB query call that returns with results.
You send an API call to start a query, that returns a Query ID.
You then send a different API call to get the query results, with the "hope" that the query completed, and if it hasn't, you're out of luck.
That's where I am right now.
I have a query that takes a bit of time and I'm "guessing" the way to handle this is to keep looping the DescribeQueries call until I find a match among that returned array of Completed queries, then go on with the rest of the code. I am unable to pull this off! Grrrr!!
I have tried While and Do..While loops that completely do NOT work. I tried setting the escape condition value when a match is found.. but it never gets set and the Lambda function times out.
function checkQueryStatus (logGroup, qID, callback) {
var params = {
logGroupName: logGroup,
maxResults: '3',
status: 'Complete'
};
let found = 0;
do {
cwlogs.describeQueries(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
// console.log(data.queries); // successful response
const qList = data.queries;
if (qList.some(query => query.queryId === qID)) {
console.log('Query Done');
callback(1);
found = 1;
} else {
console.log('Query not done');
}
}
});
} while (found == 0);
}
checkQueryStatus(logGroupName, queryID, function(qStatus) {
console.log('Query Status: ', qStatus);
if (qStatus == 1) {
console.log('Query Done');
<do other code...>
How can I do it? I'm now looking into Promises to see what thats all about..
If the DescribeQueries find a match, I want to trigger the GetQueryResults API call.
I've run into a similar problem with AWS Athena. When I start a query, I get back the response that is has started but get no notification when it finishes. The best solution I came up with was to use setTimeout to check its status every 100ms or so and continue when the query is completed. Hope that helps.
Here is a generic function which you can use to await until the query finish and returns the query result.
async function queryCloudWatch(queryRequest: StartQueryRequest): Promise<GetQueryResultsResponse> {
const queryResponse: StartQueryResponse = await cloudwatchLogs.startQuery(queryRequest).promise()
if (!queryResponse.queryId) return {}
let response: GetQueryResultsResponse | undefined = undefined
while (!response || response.status === 'Running') {
response = await cloudwatchLogs.getQueryResults({
"queryId": queryResponse.queryId
}).promise()
}
return response;
}
You have two options : query log Insights or query log groups
If you would like to query log Insights :
Use cloudwatchlogs.startQuery and cloudwatchlogs.getQueryResults APIs
A very good example can be found at https://gist.github.com/zenoyu/f63799a9079a5df376d5daf3cea27be4
To query log groups use filterLogEvents API :
const AWS = require('aws-sdk');
AWS.config.setPromisesDependency(require('bluebird'));
AWS.config.update({region: 'us-west-2'});
const cloudWatchLogs = new AWS.CloudWatchLogs({apiVersion: '2014-03-28'});
const timestamp = new Date();
const endtTime = timestamp.getTime();
const params = {
endTime: endtTime,
filterPattern: `"${stringToSearch}"`,
startTime: new Date (endtTime - 5 * 60 * 60* 24 * 1000).getTime(), // Last 5 days
logGroupName: 'myLogGroup',
limit : 10
};
const events = await cloudWatchLogs.filterLogEvents(params).promise();
console.log(`successfully queryCloudWatchLogs ${stringToSearch} results: ${JSON.stringify(events)}`);
const results = events.events.map(e => e.message)
console.log(`successfully queryCloudWatchLogs ${stringToSearch} results (${results.length}): ${JSON.stringify(results)}`);
I built an AWS Lambda that scans and filters my DynamoDB and returns to my AWS API. Thereby I had three possible search parameters (itemname, author and type), whereas I don’t know which of them are used in a query. At first, I implemented a version, in which all search parameters where hard coded. As a result, I got errors if not all search parameters where defined. At the end I reworked the code to build individual scan params dependent on the inputted search parameter.
The code works fine, but I think that there are better implementations for this problem, maybe you can give me some improvement advice. Otherwise this will help people which face the same issues with their optional search parameters.
var AWS = require('aws-sdk');
var docClient = new AWS.DynamoDB.DocumentClient();
//This is the Lambda function
exports.handler = function(event, context, callback)
{
//In case we query without query attributes
if(!event.hasOwnProperty("queryStringParameters"))
{
console.log("NO queryStringParameters FOUND");
var emptyparams =
{
TableName: "blackboard-items",
};
docClient.scan(emptyparams, onScan);
return;
}
//we want to tailor this attributes for the params for docClient.scan(params, onScan);
var queryParam = event["queryStringParameters"];
var filterexpression = "";
var expressionAttributeNames = {}; //Instantiate
var expressionAttributeValues = {};
console.log("QUERY PARAMETERS: " + JSON.stringify(queryParam));
//Do we look for an author?
if(queryParam.hasOwnProperty("author"))
{
console.log("FOUND AUTHOR");
filterexpression += "contains(#author, :author)"; //Collect scan params
expressionAttributeNames['#author'] = 'author';
expressionAttributeValues[':author'] = event["queryStringParameters"]["author"];
}
//Do we look for an itemname?
if(queryParam.hasOwnProperty("itemname"))
{
console.log("FOUND ITEMNAME");
if(filterexpression !== "")
filterexpression += " AND contains(#itemname, :itemname)";
else
filterexpression += "contains(#itemname, :itemname)";
expressionAttributeNames['#itemname'] = 'itemname';
expressionAttributeValues[':itemname'] = queryParam["itemname"];
}
//Do we look for a type?
if(queryParam.hasOwnProperty("type"))
{
console.log("FOUND TYPE");
if(filterexpression !== "")
filterexpression += " AND #type = :type";
else
filterexpression += "#type = :type";
expressionAttributeNames['#type'] = 'type';
expressionAttributeValues[':type'] = event["queryStringParameters"]["type"];
}
//Build params based on the tailored parts
var params =
{
TableName: "blackboard-items",
FilterExpression: filterexpression,
ExpressionAttributeNames: expressionAttributeNames,
ExpressionAttributeValues: expressionAttributeValues,
};
//Use tailored params for scan()
docClient.scan(params, onScan);
var count = 0;
function onScan(err, data)
{
if (err)
{
console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2));
}
else
{
console.log("Scan succeeded.");
data.Items.forEach(function(itemdata)
{
console.log("Item :", ++count,JSON.stringify(itemdata));
});
// continue scanning if we have more items
if (typeof data.LastEvaluatedKey != "undefined")
{
console.log("Scanning for more...");
params.ExclusiveStartKey = data.LastEvaluatedKey;
docClient.scan(params, onScan);
}
}
var response =
{
"isBase64Encoded": false,
"statusCode": "200",
"headers": { },
"body": JSON.stringify(data.Items)
};
callback(null, response);
}
};
Note:
The primary key of the DB is "itemname", but I will rework the DB design soon to have a sort key.
DynamoDB is very limited on its query capabilities and, as such, you should avoid scan at ALL costs. Every scan operation will consume your RCUs for every item it reads. If your table has many items, it can use up your RCUs quite quickly.
If you want to query by those 3 attributes, then DynamoDB may not be the best database for your use case. If you can narrow your query down to 1 attribute at a time instead, you can then use Global Secondary Indexes. That way you can query based on the author or type. You can see this answer on how to query for GSIs in DynamoDB
I want to merge data from two tables and then send the result as the response.
I'm new to nodejs and lambda and I'm unable to figure out how I can merge json data from both scan calls and send it as the response.
If I uncomment the callback then response for only one table is sent.
My code is below, can someone please help in completing it
'use strict';
const AWS = require("aws-sdk");
const dynamodb = new AWS.DynamoDB();
const docClient = new AWS.DynamoDB.DocumentClient();
exports.handler = function(event, ctx, callback) {
var params= {
TableName:'x',
FilterExpression:'SessionId = :SessionId',
ExpressionAttributeValues:{ ":SessionId" : 'ca47a131'},
};
var params1= {
TableName:'y',
FilterExpression:'sessionid = :SessionId',
ExpressionAttributeValues:{ ":SessionId" : 'ca47a131'},
};
docClient.scan(params, onScan);
docClient.scan(params1, onScan1);
function onScan(err, data){
if(err){
callback(err, null);
}else{
//callback(null, data);
}
}
function onScan1(err, data){
if(err){
callback(err, null);
}else{
//callback(null, data);
}
}
}
You can use the following modification to the code so that you can send the response in a single callback.
'use strict';
const AWS = require("aws-sdk");
const dynamodb = new AWS.DynamoDB();
const docClient = new AWS.DynamoDB.DocumentClient();
exports.handler = function(event, ctx, callback) {
var params= {
TableName:'x',
FilterExpression:'SessionId = :SessionId',
ExpressionAttributeValues:{ ":SessionId" : 'ca47a131'},
};
var params1= {
TableName:'y',
FilterExpression:'sessionid = :SessionId',
ExpressionAttributeValues:{ ":SessionId" : 'ca47a131'},
};
docClient.scan(params, onScan);
docClient.scan(params1, onScan1);
var firstResultData = false;
function runAfterBothCallbacks(data){
if(!firstResultData){
firstResultData = data;
}else{
// Combine firstResultData with data and return in the callback
callback(null,{ dataX: firstResultData, dataY: data });
// Note: The order of scan and scan1 result coming cannot be guaranteed so, dataX can be the results of scan or scan1. If you can identify the result based on the scan, either pass it as another parameter to the runAfterBothCallbacks method or identify the scan based on data result (If possible).
}
}
function onScan(err, data){
if(err){
callback(err, null);
}else{
runAfterBothCallbacks(data);
}
}
function onScan1(err, data){
if(err){
callback(err, null);
}else{
runAfterBothCallbacks(data);
}
}
}
Welcome to JavaScript asynchronous callbacks (aka callback hell).
Fortunately, the AWS SDK supports promises so you can use Promise.all() to wait for multiple promises to be resolved. When that happens, merge the JSON results and return the merged result via the Lambda function's callback() method.
console.log('starting function');
var dynamodb = new AWS.DynamoDB();
var AWS = require('aws-sdk');
exports.handler = function (e, ctx, callback) {
var params = {
Item: {
"Name": {
S: "Dalton Warden"
},
"PhoneNumber": {
S: "796-353-1416",
}
},
ReturnConsumedCapacity: "TOTAL",
TableName: "CustomerInfo"
};
dynamodb.putItem(params, function (err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
};
I'm pretty new to lambda and dynamoDB but i'm trying to send data to a table I have set up. I've been through amazon's documentation and looked for similar instances on this site and The formatting looks like it would return the correct JSON but I'm still having trouble. The error I'm getting is Cannot read property 'DynamoDB' of undefined".
Please change the order of the below statements as mentioned below (first AWS and then dynamodb).
var AWS = require('aws-sdk');
var dynamodb = new AWS.DynamoDB();
I have written simple Lambda function to scan data from dynamodb, but data is getting retrieved with error message tag and lambda function shows message as execution failed
var AWS = require('aws-sdk');
var DOC = require("dynamodb-doc");
var dynamo = new DOC.DynamoDB();
exports.handler = function (event, context, callback) {
var params = {
TableName: "Movies",
// ProjectionExpression: "#yr, Movie",
FilterExpression: "#yr = :thisyear",
ExpressionAttributeNames: {
"#yr": "year",
},
ExpressionAttributeValues: {
":thisyear" : 2009
}
};
dynamo.scan(params, function(err, data){
if (err){
callback("error occoured");
}
else{
callback(JSON.stringify(data.Items));
}
});
};
Result
{
"errorMessage": "[{\"year\":2009,\"Movie\":\"Jab tak hai jaan\"}]"
}
nodejs callback are typically error-first callbacks. It means that the first parameter is the error message and the second parameter is the result. So when returning the result you need to pass null as the first argument.
eg: callback(null, JSON.stringify(data.Items));
Please refer this article