i am new in NodeJs development,
i want to ask, surely this is kinda basic, but i dont know how to do it.
i have a task to read request one field that can filled with multiple values,
on json array like this :
{
"email" : "first#mail.com" , "second#mail.com", "third#mail.com"
}
how to get each value from that "email" field and processing it to some function ?
i have a query select to get username from one table
select username from [dbo].[users] where email=#email (first, second, and third)
this is my code for now only that read one value, not multiple :
async getValueFunction(req, res) {
res.setHeader('Content-Type', 'application/json');
try {
if (req.body.email != "") {
const pool = await poolPromise
const result = await pool.request()
.input('email', sql.VarChar, req.body.email)
.query(queries.getUserScoreByEmail)
var showUserScore = result.recordset;
res.json(showUserScore);
} else if (req.body.email == "") {
const pool = await poolPromise
const result = await pool.request()
.query(queries.getUserScore)
var showAllUserScore = result.recordset;
res.json(showAllUserScore);
}
} catch (error) {
res.status(500)
res.send(error.message)
}
}
how to do the loop (iteration) and collect the result/recordset before send as one response (json) ??
You should update your structure because it is not an key value pair.
What you can do is storing the E-Mail Adresses in an Array like this
const data ={
"email" : ["first#mail.com" , "second#mail.com", "third#mail.com" ]
}
And then you access it with data.email
Related
REVISED QUESTION
I've revised the question, in the hope of getting a clearer answer.
I'm trying to process data in ExpressJS, based on the incoming req.body and the existing data in the table.
I'm receiving a req.body that contains a JSON list of updated fields. Some of those fields are stored as JSONB in Postgres. If an incoming field is JSONB, then the form (external code) that is making the request has already run a jsonpatch.compare() to generate the list of patches, and it is these patches and not the full values that are being passed in. For any non-JSONB values, incoming values just need to be passed through to the UPDATE query.
I have a working version, as below, that pretends that the existing JSONB values in the table ARE NULL. Clearly, this is NOT what is needed. I need to pull the values from the db. The non-querying-of-current-values version and a bare minimum router, looks like this:
const express = require('express')
const bodyParser = require('body-parser')
const SQL = require('sql-template-strings')
const { Client } = require('pg')
const dbConfig = require('../db')
const jsonpatch = require('fast-json-patch')
const FormRouter = express.Router()
I have some update code:
````javascript
const patchFormsRoute = (req, res) => {
const client = new Client(dbConfig)
const { id } = req.body
const parts = []
const params = [id]
// list of JSONB fields for the 'forms' table
const jsonFields = [
'sections',
'editors',
'descriptions',
]
// list of all fields, including JSONB fields in the 'forms' table
const possibleFields = [
'status',
'version',
'detail',
'materials',
...jsonFields,
]
// this is a DUMMY RECORD instead of the result of a client.query
let currentRecord = { 'sections':[], 'editors':[], 'descriptions':[] }
possibleFields.forEach(myProp => {
if (req.body[myProp] != undefined) {
parts.push(`${myProp} = $${params.length + 1}`)
if (jsonFields.indexOf(myProp) > -1) {
val = currentRecord[myProp]
jsonpatch.applyPatch(val, req.body[myProp])
params.push(JSON.stringify(val))
} else {
params.push(req.body[myProp])
}
}
})
const updateQuery = 'UPDATE forms SET ' + parts.join(', ') + ' WHERE id = $1'
client.connect()
return client
.query(updateQuery, params)
.then(result => res.status(200).json(result.rowCount))
.catch(err => res.status(400).json(err.severity))
.then(() => client.end())
}
FormRouter.route('/')
.patch(bodyParser.json({ limit: '50mb' }), patchFormsRoute)
exports.FormRouter = FormRouter
I promise, that this is working code, which does almost what I need. However, I want to replace the dummy record with the data already in the table, fetched contemporaneously. My issue, is because multiple clients could be updating a row at the same time (but looking at orthogonal elements of the JSONB values), I need the fetch, calc, and update to happen as a SINGLE TRANSACTIOn. My plan is to:
BEGIN a transaction
Query Postgres for the current row value, based on the incoming id
For any JSONB fields, apply the patch to generate the correct value for that field in the UPDATE statement.
Run the UPDATE statement with the appropriate param values (either from the req.body or the patched row, depending on whether the field is JSONB or not)
COMMIT the transaction, or ROLLBACK on error.
I've tried implementing the answer from #midrizi; maybe it's just me, but the combination of awaits and plain testing of res sends the server off into Hyperspace... and ends in a timeout.
In case anyone is still awake, here's a working solution to my issue.
TLDR; RTFM: A pooled client with async/await minus the pooling (for now).
const patchFormsRoute = (req, res) => {
const { id } = req.body
// list of JSONB fields for the 'forms' table
const jsonFields = [
'sections',
'editors',
'descriptions',
]
// list of all fields, including JSONB fields in the 'forms' table
const possibleFields = [
'status',
'version',
'detail',
'materials',
...jsonFields,
]
const parts = []
const params = [id]
;(async () => {
const client = await new Client(dbConfig)
await client.connect()
try {
// begin a transaction
await client.query('BEGIN')
// get the current form data from DB
const fetchResult = await client.query(
SQL`SELECT * FROM forms WHERE id = ${id}`,
)
if (fetchResult.rowCount === 0) {
res.status(400).json(0)
await client.query('ROLLBACK')
} else {
const currentRecord = fetchResult.rows[0]
// patch JSONB values or update non-JSONB values
let val = []
possibleFields.forEach(myProp => {
if (req.body[myProp] != undefined) {
parts.push(`${myProp} = $${params.length + 1}`)
if (jsonFields.indexOf(myProp) > -1) {
val = currentRecord[myProp]
jsonpatch.applyPatch(val, req.body[myProp])
params.push(JSON.stringify(val))
} else {
params.push(req.body[myProp])
}
}
})
const updateQuery =
'UPDATE forms SET ' + parts.join(', ') + ' WHERE id = $1'
// update record in DB
const result = await client.query(updateQuery, params)
// commit transaction
await client.query('COMMIT')
res.status(200).json(result.rowCount)
}
} catch (err) {
await client.query('ROLLBACK')
res.status(400).json(err.severity)
throw err
} finally {
client.end()
}
})().catch(err => console.error(err.stack))
}
I want to create a function with node.js but I've got stuck at a point.
Explanation of what I want to do:
First, the function will trigger when a new document added to the path profile/{profileID}/posts/{newDocument}
the function will send a notification to all the following users. the problem comes here.
I've another collection in the profile collection which is followers and contains documents of the field followerID.
I want to take this followerID and use it as a document id to access the tokenID field with I've added to the profile document.
like this:
..(profile/followerID).get(); and then access the field value of tokenID field.
My current Code:- Index.js
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.fcmTester = functions.firestore.document('profile/{profileID}/posts/{postID}').onCreate((snapshot, context) => {
const notificationMessageData = snapshot.data();
var x = firestore.doc('profile/{profileID}/followers/');
var follower;
x.get().then(snapshot => {
follower = snapshot.followerID;
});
return admin.firestore().collection('profile').get()
.then(snapshot => {
var tokens = [];
if (snapshot.empty) {
console.log('No Devices');
throw new Error('No Devices');
} else {
for (var token of snapshot.docs) {
tokens.push(token.data().tokenID);
}
var payload = {
"notification": {
"title": notificationMessageData.title,
"body": notificationMessageData.title,
"sound": "default"
},
"data": {
"sendername": notificationMessageData.title,
"message": notificationMessageData.title
}
}
return admin.messaging().sendToDevice(tokens, payload)
}
})
.catch((err) => {
console.log(err);
return null;
})
});
my firestore database explanation.
profile | profileDocuments | posts & followers | followers collection documents & posts collection documents
I have a parent collection called profile and it contains documents as any collection these documents contain a field called tokenID and that I want to access, but I will not do this for all users only for followers (the users who follwed that profile) so I've created a new collection called followers and it contains all the followers IDs, I want to take every followerID and for each id push tokenID to tokens list.
If I understand correctly your question, you should do as follows. See the explanations below.
exports.fcmTester = functions.firestore.document('profile/{profileID}/posts/{postID}').onCreate((snapshot, context) => {
const notificationMessageData = snapshot.data();
const profileID = context.params.profileID;
// var x = firestore.doc('profile/{profileID}/followers/'); //This does not point to a document since your path is composed of 3 elements
var followersCollecRef = admin.firestore().collection('profile/' + profileID + '/followers/');
//You could also use Template literals `profile/${profileID}/followers/`
return followersCollecRef.get()
.then(querySnapshot => {
var tokens = [];
querySnapshot.forEach(doc => {
// doc.data() is never undefined for query doc snapshots
tokens.push(doc.data().tokenID);
});
var payload = {
"notification": {
"title": notificationMessageData.title,
"body": notificationMessageData.title,
"sound": "default"
},
"data": {
"sendername": notificationMessageData.title,
"message": notificationMessageData.title
}
}
return admin.messaging().sendToDevice(tokens, payload)
});
First by doing var x = firestore.doc('profile/{profileID}/followers/'); you don't declare a DocumentReference because your path is composed of 3 elements (i.e. Collection/Doc/Collection). Note also that,in a Cloud Function, you need to use the Admin SDK in order to read other Firestore documents/collections: So you need to do admin.firestore() (var x = firestore.doc(...) will not work).
Secondly, you cannot get the value of profileID just by doing {profileID}: you need to use the context object, as follows const profileID = context.params.profileID;.
So, applying the above, we declare a CollectionReference followersCollecRef and we call the get() method. Then we loop over all the docs of this Collection with querySnapshot.forEach() to populate the tokens array.
The remaining part is easy and in line with your code.
Finally, note that since v1.0 you should initialize your Cloud Functions simple with admin.initializeApp();, see https://firebase.google.com/docs/functions/beta-v1-diff#new_initialization_syntax_for_firebase-admin
Update following your comments
The following Cloud Function code will lookup the Profile document of each follower and use the value of the tokenID field from this document.
(Note that you could also store the tokenID directly in the Follower document. You would duplicate data but this is quite common in the NoSQL world.)
exports.fcmTester = functions.firestore.document('profile/{profileID}/posts/{postID}').onCreate((snapshot, context) => {
const notificationMessageData = snapshot.data();
const profileID = context.params.profileID;
// var x = firestore.doc('profile/{profileID}/followers/'); //This does not point to a document but to a collectrion since your path is composed of 3 elements
const followersCollecRef = admin.firestore().collection('profile/' + profileID + '/followers/');
//You could also use Template literals `profile/${profileID}/followers/`
return followersCollecRef.get()
.then(querySnapshot => {
//For each Follower document we need to query it's corresponding Profile document. We will use Promise.all()
const promises = [];
querySnapshot.forEach(doc => {
const followerDocID = doc.id;
promises.push(admin.firestore().doc(`profile/${followerDocID}`).get()); //We use the id property of the DocumentSnapshot to build a DocumentReference and we call get() on it.
});
return Promise.all(promises);
})
.then(results => {
//results is an array of DocumentSnapshots
//We will iterate over this array to get the values of tokenID
const tokens = [];
results.forEach(doc => {
if (doc.exists) {
tokens.push(doc.data().tokenID);
} else {
//It's up to you to decide what you want to to do in case a Follower doc doesn't have a corresponding Profile doc
//Ignore it or throw an error
}
});
const payload = {
"notification": {
"title": notificationMessageData.title,
"body": notificationMessageData.title,
"sound": "default"
},
"data": {
"sendername": notificationMessageData.title,
"message": notificationMessageData.title
}
}
return admin.messaging().sendToDevice(tokens, payload)
})
.catch((err) => {
console.log(err);
return null;
});
});
Incase I have to execute more queries in a single API in what method I have to proceed
Is this a better way to write sub queries inside main queries
without using db.task or db.tx or else I have to use these methods.
like this below
function userChangePassword(req, res) {
const changePwd = req.swagger.params.body.value;
const token = req.headers.authorization;
token.replace('Bearer ', '');
const decoded = jwt_decode(token);
const newpass = changePwd.newPassword;
const user_id = decoded.userId;
const userSel = `select * from xx."yy" where "userId" = '${user_id}' AND "status" = 1`;
const updatePwd = `update xx."yy" set "password" = '${newpass}' where "userId" = '${user_id}' `;
db.query(userSel).then((usrResult) => {
if (usrResult.length > 0) {
db.query(updatePwd).then(() => {
res.send(response.success("The password has been changed successfully", []));
});
} else {
res.send(response.success("The password has not changed successfully", []));
}
})
.catch((err) => {
if (util.isError(err)) res.error('NotFoundError', err); // return 404
else res.error('InternalServerError', err); // else 500
});
}
kindly help me to get solution for this confusion..
Thanks.
If you have a dependency between queries, then you have to use methods task/tx. Otherwise, you can concatenate queries into one, using helpers.concat, and execute them as one query, which will be faster. And you can use methods multi / multiResult, if you are expecting return data from your multi-query.
Looking at the correct/best/better way to use AWAIT with MySQL2 in a Node.js/Express.js app when I need to run multiple queries in a single request.
Early on in my app I create a Promise Pool from my Database config
const promisePool = db.promise();
Then, on a POST request I accept 2 values, both of which I need to verify are valid, and then take the returned ID's and INSERT them in to another table.
Below is was my first attempt but I am missing out on the JS's concurrently goodness. (I've overly simplified all the calls/SQL for demonstration purposes),
app.post('/addUserToDepartment', async (req, res) => {
// Get the POST variables
let email = 'example#example.com';
let departmentname = 'sales';
let insertParams = [];
// Need to check if Department ID is even valid
const [departments] = await promisePool.query( "SELECT ? AS deptid", [departmentname] );
// Need to check if Email address is valid
const [user] = await promisePool.query( "SELECT ? AS userid", [email] );
// This would normall be an INSERT or UPDATE statement
if(departments.length && user.length){
const [rows] = await promisePool.query( "SELECT ? AS passedDeptId,? AS passedUserid", [departments[0].deptid, user[0].userid] );
}
res.send( rows )
}
Here is my second stab at it, now wrapping the promises up.
app.post('/addUserToDepartment', async (req, res) => {
// Get the POST variables
let email = 'example#example.com';
let departmentname = 'sales';
let insertParams = [];
// Need to check if Department ID is even valid
let [[departments],[user]] =
await Promise.all([
promisePool.query( "SELECT ? AS deptid", [departmentname] ),
promisePool.query( "SELECT ? AS userid", [email] )
])
// This would normall be an INSERT or UPDATE statement
if(departments.length && user.length){
let [rows] = await promisePool.query( "SELECT ? AS passedDeptId,? AS passedUserid", [departments[0].deptid, user[0].userid] );
}
res.send( rows )
}
The IF at the end still doesn't 'feel' right, but I need to know that the first two queries are valid otherwise I'll send the user to an error page.
What would be a better way to achieve the above result without forfeiting readability too much?
first: both snippets are broken as the rows variable needs to be declared outside if if.
Aside from that, what you're doing is mostly fine, but the big issue here is that if length of either is 0, you return nothing.
Is that really the behavior you want? If I call /addUserToDepartment and there is a problem in your database, do you want this to silently fail?
I think a better approach is to return appropriate errors when something goes wrong. Ideally you should just throw an exception, (but you're using Express, and I'm not sure if they support catching exceptions).
Here is what I went with in the end. I added catches, I also did my last query as a part of the Promise.all() chain.
app.get('/test2', async (req, res) => {
// Get the POST variables
let email = 'example#example.com';
let departmentname = 'sales';
let insertParams = [];
let rtn = {
status : '',
errors : [],
values : []
}
console.clear();
// Need to check if Department ID is even valid
let arrayOfPromises = [
promisePool.query( "SELECT ? AS did", [departmentname] ),
promisePool.query( "SELECT ? AS uid", [email] )
]
await Promise.all(arrayOfPromises)
.then( ([d,u] ) => {
// Get the values back from the queries
let did = d[0][0].did;
let uid = u[0][0].uid;
let arrayOfValues = [did,uid];
// Check the values
if(did == 'sales'){
rtn.values.push( did );
} else{
rtn.errors.push( `${did} is not a valid department`);
}
if(uid == 'example#example.com'){
rtn.values.push( uid );
} else{
rtn.errors.push( `${did} is not a valid department`);
}
if( rtn.errors.length === 0){
return arrayOfValues;
} else{
return Promise.reject();
}
})
.then( async ( val ) => {
// By this point everything is ok
let [rows] = await promisePool.query( "SELECT ? AS passedDeptId,? AS passedUserid", val );
res.send( rtn )
})
.catch((err) => {
console.error(err)
rtn.status = 'APPLICATION ERROR';
rtn.errors.push( err.message);
res.send( rtn )
});
});
I've had a read through AWS's docs around pagination:
As their docs specify:
In a response, DynamoDB returns all the matching results within the scope of the Limit value. For example, if you issue a Query or a Scan request with a Limit value of 6 and without a filter expression, DynamoDB returns the first six items in the table that match the specified key conditions in the request (or just the first six items in the case of a Scan with no filter)
Which means that given I have a table called Questions with an attribute called difficulty(that can take any numeric value ranging from 0 to 2) I might end up with the following conundrum:
A client makes a request, think GET /questions?difficulty=0&limit=3
I forward that 3 to the DynamoDB query, which might return 0 items as the first 3 in the collection might not be of difficulty == 0
I then have to perform a new query to fetch more questions that match that criteria without knowing I might return duplicates
How can I then paginate based on a query correctly? Something where I'll get as many results as I asked for whilst having the correct offset
Using async/await.
const getAllData = async (params) => {
console.log("Querying Table");
let data = await docClient.query(params).promise();
if(data['Items'].length > 0) {
allData = [...allData, ...data['Items']];
}
if (data.LastEvaluatedKey) {
params.ExclusiveStartKey = data.LastEvaluatedKey;
return await getAllData(params);
} else {
return data;
}
}
I am using a global variable allData to collect all the data.
Calling this function is enclosed within a try-catch
try {
await getAllData(params);
console.log("Processing Completed");
// console.log(allData);
} catch(error) {
console.log(error);
}
I am using this from within a Lambda and it works fine.
The article here really helped and guided me. Thanks.
Here is an example of how to iterate over a paginated result set from
a DynamoDB scan (can be easily adapted for query as well) in Node.js.
You could save the LastEvaluatedKey state serverside and pass an identifier back to your client, which it would send with its next request and your server would pass that value as ExclusiveStartKey in the next request to DynamoDB.
const AWS = require('aws-sdk');
AWS.config.logger = console;
const dynamodb = new AWS.DynamoDB({ apiVersion: '2012-08-10' });
let val = 'some value';
let params = {
TableName: "MyTable",
ExpressionAttributeValues: {
':val': {
S: val,
},
},
Limit: 1000,
FilterExpression: 'MyAttribute = :val',
// ExclusiveStartKey: thisUsersScans[someRequestParamScanID]
};
dynamodb.scan(scanParams, function scanUntilDone(err, data) {
if (err) {
console.log(err, err.stack);
} else {
// do something with data
if (data.LastEvaluatedKey) {
params.ExclusiveStartKey = data.LastEvaluatedKey;
dynamodb.scan(params, scanUntilDone);
} else {
// all results scanned. done!
someCallback();
}
}
});
Avoid using recursion to prevent call stack overflow. An iterative solution extending #Roshan Khandelwal's approach:
const getAllData = async (params) => {
const _getAllData = async (params, startKey) => {
if (startKey) {
params.ExclusiveStartKey = startKey
}
return this.documentClient.query(params).promise()
}
let lastEvaluatedKey = null
let rows = []
do {
const result = await _getAllData(params, lastEvaluatedKey)
rows = rows.concat(result.Items)
lastEvaluatedKey = result.LastEvaluatedKey
} while (lastEvaluatedKey)
return rows
}
I hope you figured out. So just in case others might find it useful. AWS has QueryPaginator/ScanPaginator as simple as below:
const paginator = new QueryPaginator(dynamoDb, queryInput);
for await (const page of paginator) {
// do something with the first page of results
break
}
See more details at https://github.com/awslabs/dynamodb-data-mapper-js/tree/master/packages/dynamodb-query-iterator
2022-05-19:
For AWS SDK v3 see how to use paginateXXXX at this blog post https://aws.amazon.com/blogs/developer/pagination-using-async-iterators-in-modular-aws-sdk-for-javascript/
Query and Scan operations return LastEvaluatedKey in their responses. Absent concurrent insertions, you will not miss items nor will you encounter items multiple times, as long as you iterate calls to Query/Scan and set ExclusiveStartKey to the LastEvaluatedKey of the previous call.
For create pagination in dynamodb scan like
var params = {
"TableName" : "abcd",
"FilterExpression" : "#someexperssion=:someexperssion",
"ExpressionAttributeNames" : {"#someexperssion":"someexperssion"},
"ExpressionAttributeValues" : {":someexperssion" : "value"},
"Limit" : 20,
"ExclusiveStartKey" : {"id": "9ee10f6e-ce6d-4820-9fcd-cabb0d93e8da"}
};
DB.scan(params).promise();
where ExclusiveStartKey is LastEvaluatedKey return by this query last execution time
Using async/await, returning the data in await.
Elaboration on #Roshan Khandelwal's answer.
const getAllData = async (params, allData = []) => {
const data = await dynamodbDocClient.scan(params).promise()
if (data['Items'].length > 0) {
allData = [...allData, ...data['Items']]
}
if (data.LastEvaluatedKey) {
params.ExclusiveStartKey = data.LastEvaluatedKey
return await getAllData(params, allData)
} else {
return allData
}
}
Call inside a try/catch:
try {
const data = await getAllData(params);
console.log("my data: ", data);
} catch(error) {
console.log(error);
}
you can do a index secundary by difficulty and at query set KeyConditionExpression where difficulty = 0. Like this
var params = {
TableName: questions,
IndexName: 'difficulty-index',
KeyConditionExpression: 'difficulty = :difficulty ',
ExpressionAttributeValues: {':difficulty':0}
}
You can also achieve this using recrusion instead of a global variable, like:
const getAllData = async (params, allData = []) => {
let data = await db.scan(params).promise();
return (data.LastEvaluatedKey) ?
getAllData({...params, ExclusiveStartKey: data.LastEvaluatedKey}, [...allData, ...data['Items']]) :
[...allData, ...data['Items']];
};
Then you can simply call it like:
let test = await getAllData({ "TableName": "test-table"}); // feel free to add try/catch
Using DynamoDB pagination with async generators:
let items = []
let params = {
TableName: 'mytable',
Limit: 1000,
KeyConditionExpression: 'mykey = :key',
ExpressionAttributeValues: {
':key': { S: 'myvalue' },
},
}
async function* fetchData({
params
}) {
let data
do {
data = await dynamodb.query(params).promise()
yield data.Items
params.ExclusiveStartKey = data.LastEvaluatedKey
} while (typeof data.LastEvaluatedKey != 'undefined')
}
for await (const data of fetchData(params)) {
items = [...items, ...data]
}