How can we get all the items by invoking dynamodb.query?
The documentation states that we need to look for the presence of LastEvaluatedKey. Just wondering how we could aggregate all the Items in an efficient way?
app.get(path, function (req, res) {
var allItems = [];
var params = {
TableName: tableName,
"IndexName": "status-index",
"KeyConditionExpression": "#attrib_name = :attrib_value",
"ExpressionAttributeNames": { "#attrib_name": "status" },
"ExpressionAttributeValues": { ":attrib_value": req.query.status },
"ScanIndexForward": false
};
dynamodb.query(params, onQuery);
function onQuery(err, data) {
if (err) {
res.json({ error: 'Could not load items: ' + err });
} else {
// Should I be aggregating all the items like this?
allItems = allItems.concat(data.Items);
// Then should I set it to res like this to return all the items?
res.json(allItems);
if (typeof data.LastEvaluatedKey != 'undefined') {
params.ExclusiveStartKey = data.LastEvaluatedKey;
dynamodb.query(params, onQuery);
}
}
}
});
Please look at comments within the code. That is where I think we need to have the appropriate code to aggregate all the items and return back the response.
I have not found a way to debug this yet as I'm fairly new to DynamoDB and AWS Amplify. Let me as well know if there is an easier way to debug this in an AWS amplify backed up GET API.
This is not a direct answer to you question, but a suggestion. I wrote an article "How To Use AWS AppSync in Lambda Functions
".
The TLDR of it is:
Create a Lambda function, which uses the AppSync client to perform
GraphQL operations. Use polyfills and install all necessary
dependencies.
Ensure the Lambda function has the right execution policy.
Use AppSync’s multi auth to allow both requests that are signed by
Amazon Cognito User Pools as well as requests that are signed using
Amazon’s IAM. This way, both the client and the server (aka. the
Lambda function) will be authenticated and can have different CRUD
permissions.
If I were you and wanted to access my data base through a Lambda function, I would follow that tutorial and do it using AppSync. One of the advantages which matters to you is that you don't have to care about LastEvaluatedKey and you can instead use AppSync's nextToken which is way more safe.
Query returns paginated results - if you want all data then you need to keep querying and aggregating until your LastEvaluatedKey is empty.
Refer: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
Related
I would like to build a node application using REST, need to read data from readymade api and store it in class for temp, and then save it to MySQL. Can anyone have any idea about this?
This is a very simple job.
Let's consider Typescript, but you can achieve the same result with JavaScript. I'll be using node-fetch as an example of the rest API library. Do note that the code might not be syntactically correct.
First: Create interfaces/classes that reflect the data you will receive from the REST API
interface Food {
id: number,
name: string,
...
}
Second:
Create a Repository
Create a class Repository which you will use to communicate with the rest API
class Repository {
async function getFoods(...args): List<Food> {
let foods = await fetch({url: "url"});
return foods;
}
async function addFood(food: Food): Response {
let response = await fetch({
url: "url-to-add-food",
method: "post",
data: JSON.stringify(food)
});
}
}
Third:
Use the repository to fetch the data and use conventional methods to save it to a MySQL database
let foods = await repository.getFoods();
foods.forEach(food => {
connection.query('INSERT INTO foods SET ?', food,
function (err, resp) {
if (err) throw err;
}
);
});
I'm working on caching strategies for an application that uses knex.js for all sql related stuff.
Is there a way to intercept the query to check if it can be fetched from a cache instead of querying the database?
Briefly looked into knex.js events, which has a query event.
Doc:
A query event is fired just before a query takes place, providing data about the query, including the connection's __knexUid / __knexTxId properties and any other information about the query as described in toSQL. Useful for logging all queries throughout your application.
Which means that it's possible to do something like (also from docs)
.from('users')
.on('query', function(data) {
app.log(data);
})
.then(function() {
// ...
});
But is it possible to make the on query method intercept and do some logic before actually executing the query towards the database?
I note that this suggestion is attached to a Knex GitHub issue (credit to Arian Santrach) which seems relevant:
knex.QueryBuilder.extend('cache', async function () {
try {
const cacheKey = this.toString()
if(cache[cacheKey]) {
return cache[cacheKey]
}
const data = await this
cache[cacheKey] = data
return data
} catch (e) {
throw new Error(e)
}
});
This would allow:
knex('tablename').where(criteria).cache()
to check for cached data for the same query. I would think a similar sort of structure could be used for whatever your caching solution was, using the query's string representation as the key.
I've got a Dialogflow agent for which I'm using the Inline Editor (powered by Cloud Functions for Firebase). When I try to get external api data by using request-promise-native I keep getting Ignoring exception from a finished function in my firebase console.
function video(agent) {
agent.add(`You are now being handled by the productivity intent`);
const url = "https://reqres.in/api/users?page=2";
return request.get(url)
.then(jsonBody => {
var body = JSON.parse(jsonBody);
agent.add(body.data[0].first_name)
return Promise.resolve(agent);
});
}
Your code looks correct. The exception in this case might be that you're not using a paid account, so network access outside Google is blocked. You can probably see the exact exception by adding a catch block:
function video(agent) {
agent.add(`You are now being handled by the productivity intent`);
const url = "https://reqres.in/api/users?page=2";
return request.get(url)
.then(jsonBody => {
var body = JSON.parse(jsonBody);
agent.add(body.data[0].first_name)
return Promise.resolve(agent);
})
.catch(err => {
console.error('Problem making network call', err);
agent.add('Unable to get result');
return Promise.resolve(agent);
});
}
(If you do this, you may want to update your question with the exact error from the logs.)
Inline Editor uses Firebase. If you do not have a paid account with Firebase, you will not be able to access external APIs.
I am creating application which takes nodejs code from the user, and I am creating lambda function on the fly using that code.
eg: The code can be
var http = require('http');
exports.handler = function(event, context) {
console.log('start request to ' + event.url)
http.get('http://##someapi', function(res) {
console.log("Any Response : " + res.statusCode);
}).on('error', function(e) {
console.log("Error from API : " + e.message);
});
console.log('end request to ' + event.url)
context.done(null);
}
But some how I want to restrict http/https calls to be made from that code , as I don't have control on what code will passed by the user.
So is there any way to restrict that, like some sort of ROLE or POLICY or any configuration to achieve that?
I am able to restrict DynamoDB access by specifying Policy in Role. So I have control over db access but not http calls.
Simply prepend the user's code with the following:
(function(){
function onlyAWS (module) {
var isAWS = /amazonaws.com$/i
var orig = module.request
module.request = function restrictedRequest (opts, done) {
if (typeof opts === 'string') opts = require('url').parse(opts)
if (isAWS.test(opts.host || opts.hostname)) {
return orig.call(module, opts, done)
} else {
throw new Error('No HTTP requests allowed')
}
}
}
onlyAWS(require('http'))
onlyAWS(require('https'))
})()
One alternative would be, putting these lambdas in a VPC with restricted Outbound access.
It sounds like funny solution but I found simple solution to my problem. I am adding below code along with code entered by User.
var require = function(){
return "You are not allowed to do this operation";
}
Now if used user tries to include any 3rd party library like required('http') , then it will not allow to instantiate http lib in the node code.
using this solution I am able to block loading all 3rd party library which i don't want User to use in AWS lambda function.
I am still searching for proper solution instead of using that hack in code.
This is my first post here so please don't get mad if my formatting is a bit off ;-)
I'm trying to develop a backend solution using Azure mobile apps and node.js for server side scripts. It is a steep curve as I am new to javaScript and node.js coming from the embedded world. What I have made is a custom API that can add users to a MSSQL table, which is working fine using the tables object. However, I also need to be able to delete users from the same table. My code for adding a user is:
var userTable = req.azureMobile.tables('MyfUserInfo');
item.id = uuid.v4();
userTable.insert(item).then( function (){
console.log("inserted data");
res.status(200).send(item);
});
It works. The Azure node.js documentation is really not in good shape and I keep searching for good example on how to do simple things. Pretty annoying and time consuming.
The SDK documentation on delete operations says it works the same way as read, but that is not true. Or I am dumb as a wet door. My code for deleting looks like this - it results in exception
query = queries.create('MyfUserInfo')
.where({ id: results[i].id });
userTable.delete(query).then( function(delet){
console.log("deleted id ", delet);
});
I have also tried this and no success either
userTable.where({ id: item.id }).read()
.then( function(results) {
if (results.length > 0)
{
for (var i = 0; i < results.length; i++)
{
userTable.delete(results[i].id);
});
}
}
Can somebody please point me in the right direction on the correct syntax for this and explain why it has to be so difficult doing basic stuff here ;-) It seems like there are many ways of doing the exact same thing, which really confuses me.
Thanks alot
Martin
You could issue SQL in your api
var api = {
get: (request, response, next) => {
var query = {
sql: 'UPDATE TodoItem SET complete=#completed',
parameters: [
{ name: 'completed', value: request.params.completed }
]
};
request.azureMobile.data.execute(query)
.then(function (results) {
response.json(results);
});
}};
module.exports = api;
That is from their sample on GitHub
Here is the full list of samples to take a look at
Why are you doing a custom API for a table? Just define the table within the tables directory and add any custom authorization / authentication.