I'm fairly new to Node and Express and am struggling with creating a route that takes a user uploaded file and processes it into another file. The problem is that the second time the user sends a request, I am having to wait for the first process to complete to allow the user to upload a new file.
The basic structure of route that I have is below
THE PROBLEM is that the function convertFile below is a time taking process and it keeps the server busy from accepting new requests. How do I make it so that once the project is saved in mongo db at newProject.save() - the process to convertFile runs in the background while the server accepts new requests from the same route?
I'm sending a response back the user after newProject.save() and was hoping that would allow the user to send another request. And although this sends another request, the server doesn't accept it since its busy with the previous process to convertFile
router.post('/', upload.fields([{ name: 'zip' }]), async (req, res, next) => {
let data = {
title: req.body.title,
description: req.body.description,
}
if (req.files && req.files.zip) {
const newProject = new MongoProject(data);
newProject.save()
.then(async (project) => {
res.send(project);
console.log("Project created");
const uploadedFilePath = path.normalize(req.files.zip[0].path);
// below method - "convertFile" is a time taking method
const extractZipinfo = await convertFile(uploadedFilePath , data.masterFile).then((zipInfo) => {
console.log({ zipInfo })
data.zipInfo = {
sizes: zipInfo.sizes
}
})
})
.catch(err => {
console.log("Error creating project");
return res.send(err);
})
}
})
Below is the simplified version of code in convertFile function (code modified for brevity):
I know that this can be improvised a lot, but i'm struggling with getting it to function as expected first (allowing multiple routes)
async function convertFile(inputFilePath, outputInfo) {
const outputFilePath = "output.abc";
const jsonFilePath = "output.json";
const doc = new Document(); // this is a class to store all data of the output file that we will write at the end
const _FileAPI = new fileAPI();
const outputFinalData = await _FileAPI.Init() // this is an async method
.then(() => {
const dataClass = initiateClass(); // this is a class to store data in JSON format
const paragraphs = _FileAPI.GetallParagraphs(inputFilePath);
for (let i = 0, len = paragraphs.size(); i < len; i++) {
for (let j = 0, lenj = paragraphs.size(); j < lenj; j++) {
const para = paragraphs.get(j);
// read each para and Capitalize each word
dataClass.paragraphs.push(para);
}
}
fs.writeFileSync(jsonFilePath, JSON.stringify(dataClass, null, 2), 'utf-8');
console.log("then")
}).then(() => {
const io = new NodeIO(); // this class helps in writing the file in the desired output format
const outData = io.write(outputFilePath, doc).then(() => {
outputInfo.sizes.push(fs.statSync(outputFilePath).size);
return outputInfo;
});
return outData;
});
return outputFinalData;
}
I am new to Microsoft Bot Framework .I have created a bot using Microsoft Bot Framework. How do I create a session for individual user. Currently the problem I am facing is whenever multiple users are creating connection, the values in the variables are getting over written thus giving wrong values to the users.
Here is the code
if (turnContext.activity.text === 'CCP') {
const url = await this.connectToAgent(members);
const initialMessage = {
topic: "aws/subscribe",
content: {
topics: ["aws/chat"]
}
};
ws = new WebSocket(url[1]);
ws.addEventListener("open", () => {
ws.send(JSON.stringify(initialMessage));
});
await turnContext.sendActivity("Please wait while we connect you to an agent.");
conversationReferences[currentUser] = TurnContext.getConversationReference(turnContext.activity);
adapter = turnContext.adapter;
ws.addEventListener('message', async function (event) {
const msg = JSON.parse(event.data);
});
let sendChatHistory = (function() {
let executed = false;
return function() {
if (!executed) {
executed = true;
const param = {
ConnectionToken: connectionToken, /* required */
Content: sendHistory, /* required */
ContentType: 'text/plain', /* required */
};
connectparticipant.sendMessage(param,async function (err, data) {
chatHistory = '';
sendHistory = '';
if (err) {
errorMsg = "Error while sending a message";
}
});
}
};
})();
sendChatHistory();
}
According to the official documentation we have the procedure of StateClient where we can give separate activity holder for individual users. This works when we are creating multiple users with multiple connections.
StateClient sc = activity.GetStateClient();
userData.SetProperty<string>("MyDetails", < some value >);
// How to save the BotUserData
await sc.BotState.SetUserDataAsync(activity.ChannelId, activity.From.Id, userData);
// Getting User data from bot
BotData userData = await sc.BotState.GetUserDataAsync(activity.ChannelId, activity.From.Id);
As this is a parallel operation, the method Async is required to get the data of user simultaneously.
I'm using Back4app.
My Profile class schema has 4 File columns containing pictures.
So when I retrieve an object , I have to make an HTTP request for each file URL and get the byte data like this.
const data = await Parse.Cloud.httpRequest({url:profilePhoto.url()});
return data.buffer.toString('base64');
But for all four files I have to do 4 HTTP requests to the server.
Is there anyway to do a batch HTTP request so that with just 1 request I can get data for all 4 files ?
My main aim is to do the least amount of requests to the server as possible.
There is no out-of-the-box way to retrieve multiple files with one request in Parse Server.
You could implement your own Parse Cloud Code function to retrieve multiple files, but you would have to manually combine them server side and separate them client side.
As a starting point you could look at packages like multistream that allow you to combine multiple file streams into one to get some inspiration.
You might be able to do something similar to what I've done in cloud code.
I had to load up a bunch of information at the start of my application, requiring many round trips to the server.
So I wrote a function called getUserData().
This does many unrelated queries, and jams all of the results into one big object. I then return the object from the function.
Here is the entire function:
console.log("startig getUserData");
var callCount = 0;
var lastLoadTime=0;
// Given a user, load all friends. Save the objects to ret.objects,
// and save the objectIds to ret.friends
//
// Note: we always load the exhaustive friend list, because
// otherwise, we would have no way of recognizing
// removed friendships.
//
async function loadFriends(user, ret) {
const friendQuery = user.relation("friends").query();
const friends = await findFully(friendQuery);
for(var i=0;i<friends.length;i++){
ret.friends[friends[i].id]=1;
ret.objects[friends[i].id]=friends[i];
};
}
// Given a user, load all owned cells. Save the objects to ret.owned,
// and save their objectIds to ret.ownedCells.
//
// Also, save the ids of members, which we will use to flesh out ret.objects with
// the objects who are not friends, but share a cell with the current user.
async function loadPublicCells(user, ret, memberIds) {
const ownedCellQ = new Parse.Query('PublicCell');
ownedCellQ.equalTo('owner',user);
const joinedCellQ = new Parse.Query('PublicCell');
joinedCellQ.equalTo('members',user);
const publicCellQ = Parse.Query.or(ownedCellQ,joinedCellQ);
publicCellQ.greaterThan("updatedAt",new Date(lastLoadTime));
const publicCells=await findFully(publicCellQ);
for(var i=0;i<publicCells.length;i++) {
const cell = publicCells[i];
ret.ownedCells[cell.id]=cell;
const owner = cell.get("owner");
if(owner==null)
continue;
ret.objects[cell.id]=cell;
if(owner.id === user.id) {
ret.ownedCells[cell.id]=1;
} else {
ret.joinedCells[cell.id]=1;
};
const memberQ = cell.relation("members").query();
const members = await findFully(memberQ);
if(ret.memberMap[cell.id]==null)
ret.memberMap[cell.id]={};
const map = ret.memberMap[cell.id];
for(var j=0;j<members.length;j++){
const member=members[j];
map[member.id]=1;
ret.objects[member.id]=member;
};
};
};
// given a list of all members of all cells, load those objects and store
// them in ret.objects. We do not have to record which cells they belong
// to, because that information is in ret.memberMap
async function loadMembers(memberIds, ret) {
const memberQ = new Parse.Query(Parse.User);
var partIds;
while(memberIds.length){
partIds = memberIds.splice(0,100);
memberQ.containedIn('objectId',partIds);
const part = await findFully(memberQ);
for(var i=0;i<part.length;i++) {
ret.objects[part[i].id]=part[i];
}
};
};
// given a user, save all of the objectIds of people who have annoyed him with
// spam. We save only the ids, they don't go on ret.objects, because we only
// need to filter them out of things. The objectIds are sufficient.
//
// We always send all spam objects, otherwise we would not recognize deletions
async function loadUserSpams(user, ret) {
const userSpamsQ = new Parse.Query("_User");
userSpamsQ.equalTo("spamUsers",user);
userSpamsQ.greaterThan("updatedAt", new Date(lastLoadTime));
const userSpams = await findFully(userSpamsQ);
for(var i=0;i<userSpams.length;i++){
ret.userSpams[userSpams[i].id]=1;
};
};
// given a user, save all of the objectIds of people who have been annoyed *BY*
// him with spam. We save only the ids, they don't go on ret.objects, because we
// only need to filter them out of things. The objectIds are sufficient.
//
// We always send all spam objects, otherwise we would not recognize deletions
async function loadSpamUsers(user, ret) {
const spamUserR = user.relation('spamUsers');
const spamUserQ = spamUserR.query();
spamUserQ.greaterThan("updatedAt", new Date(lastLoadTime));
const spamUsers = await findFully(spamUserQ);
for(var i=0;i<spamUsers.length;i++){
ret.spamUsers[spamUsers[i].id]=1;
};
};
// given a user, save all of the objectIds of people to whom he has sent a
// friend request which is still pending. We save only the ids, they don't go
// on ret.objects, because we only need to filter them out of things. The
// objectIds are sufficient.
async function loadPendingFriends(user, ret) {
const request1Q = new Parse.Query('Request');
request1Q.equalTo("owner",user);
const request2Q = new Parse.Query('Request');
request2Q.equalTo("sentTo",user);
const requestQ = Parse.Query.or(request1Q,request2Q);
requestQ.equalTo("status",'PENDING');
const requests = await findFully(requestQ);
for(var i=0;i<requests.length;i++){
const request = requests[i];
const sentBy = request.get("owner");
if(sentBy==null){
console.warn("sentBy==null");
continue;
};
const sentTo = request.get("sentTo");
if(sentTo==null){
console.warn("sentTo==null");
continue;
};
console.dump({sentTo,sentBy});
if(sentBy.id==user.id){
ret["pendingFriends"][sentTo.id]=sentTo;
} else if ( sentTo.id==user.id ) {
ret["friendingPends"][sentBy.id]=sentBy;
};
};
};
// given a user, load all of his private cells. We do not store
// the user objects, because only friends will be in your private cells.
async function loadPrivateCells(user, ret) {
const privateCellQ = new Parse.Query('PrivateCell');
privateCellQ.equalTo("owner", user);
privateCellQ.greaterThan("updatedAt", new Date(lastLoadTime));
const privateCells = await findFully(privateCellQ);
for(var i=0;i<privateCells.length;i++) {
const cell = privateCells[i];
ret.objects[cell.id]=cell;
ret.privateCells[cell.id]=cell;
if(ret.memberMap[cell.id]==null)
ret.memberMap[cell.id]={};
const map = ret.memberMap[cell.id];
const memberQ = cell.relation("members").query();
const members = await findFully(memberQ);
for(var j=0;j<members.length;j++){
const member=members[j];
map[member.id]=1;
ret.objects[member.id]=member;
};
};
//});
}
// we use objects as maps to weed out duplicate objects and cells.
// when we are done, we use this function to replace the object
// with an array of objects. we don't need to send the keys, since
// they already exist within the objects.
function objToValueList(k,ret){
const objs = [];
for( var id in ret[k] )
objs.push(ret[k][id]);
ret[k]=objs;
ret.counts[k]=objs.length;
};
// convert the objects which have been used to accumulate key lists
// to arrays of objectIds. k is the name of the list we are working
// on. ret[k] is the list itself.
function objToKeyList(k,ret) {
const objs = [];
for( var id in ret[k] ) {
objs.push(id);
};
ret[k]=objs;
ret.counts[k]=objs.length;
};
async function checkUserConsent(user){
const query = new Parse.Query("PrivacyPolicy");
query.descending("createdAt");
query.limit(1);
const res = await query.find();
if(res.length==0) {
return true;
};
const policy=res[0];
console.dump(policy);
console.log(policy);
const userConsent=user.get("lastConsent");
return userConsent!=null && userConsent.id == policy.id;
};
async function loadAlerts(user,ret) {
const q1 = new Parse.Query("Alert");
q1.equalTo("owner", user);
const q2 = new Parse.Query("Response");
q2.equalTo("owner", user);
const q3 = new Parse.Query("Alert");
q3.matchesKeyInQuery("objectId", "alert", q2);
const q = Parse.Query.or(q1,q3);
const list = await q.find();
var time = new Date().getTime();
time -= 1000*86400;
time=Math.max(lastLoadTime, time);
q.greaterThan("updatedAt",time);
for(var i=0;i<list.length;i++) {
const item=list[i];
ret.alerts[item.id]=1;
ret.objects[item.id]=item;
};
}
async function doGetUserData(user) {
if(!user)
return {fatal: 'not logged in!' };
const ret = {
owner: {},
privateCells: {},
friends: {},
alerts: {},
objects: {},
ownedCells: {},
joinedCells: {},
spamUsers: {},
userSpams: {},
pendingFriends: {},
friendingPends: {},
memberMap: {},
loadTime: lastLoadTime,
counts: {callCount: callCount++},
};
{
user.fetch();
ret.owner=user.id;
const memberIds={};
ret.objects[user.id]=user;
console.log("loadFriends");
await loadFriends(user,ret);
console.log("loadPrivateCells");
await loadPrivateCells(user,ret,memberIds);
console.log("loadPublicCells");
await loadPublicCells(user,ret,memberIds);
console.log("loadPendingFriends");
await loadPendingFriends(user,ret);
console.log("loadUserSpams");
await loadUserSpams(user,ret);
console.log("loadSpamUsers");
await loadSpamUsers(user,ret);
console.log("loadAlerts");
await loadAlerts(user,ret);
const memberList=[];
for( var id in memberIds ) {
console.log(ret.objects[id]);
memberList.push(id);
};
console.log("loadMembers");
await loadMembers(memberList,ret);
}
for(var cell in ret.memberMap) {
var map = ret.memberMap[cell];
var list = [];
ret.memberMap[cell]=list;
for(var member in map) {
list.push(member);
};
}
delete ret.objects[user.id];
[
'friends', "friendingPends", 'pendingFriends',
'privateCells', 'ownedCells', 'joinedCells',
'userSpams', 'spamUsers', "alerts"
].forEach((k)=>{
objToKeyList(k,ret);
});
objToValueList('objects',ret);
delete ret.counts;
return ret;
}
async function getUserData(req) {
try {
var nextLoadTime=new Date().getTime();
const user = req.user;
console.log(user);
lastLoadTime = req.params.lastLoadTime;
if(lastLoadTime==null)
lastLoadTime=0;
lastLoadTime = new Date(lastLoadTime);
const ret = await doGetUserData(user);
ret.loadTime=nextLoadTime;
return ret;
} catch ( err ) {
console.log(err);
try {
console.log(err.stack());
} catch ( xxx ) {
console.log(err);
};
throw (`error getting data: ${err}`);
};
};
Parse.Cloud.define("getUserData", getUserData);
Something like this could easily be done to get your data for you. Like this solution, it is unlikely to be entirely pretty, but it would probably work.
I have a firebase function that's supposed to return Items that are sold by a seller. I want to get the seller's profile picture via firebase authentication. But whenever I AWAIT the function
edit: worth noting that mAuth is firebase authentication*
await mAuth.geUser(sellerData.UID);
the application returns me an empty json or []
Here is the full code for the function, the error occurs on line 11 or somewhere around there.
export const getHottestItems = functions.region("asia-east2").https.onRequest(async (data, response) => {
try {
var arrayItem = new Array<Item>();
let itemSeller: Seller;
const sellerSnapshot = await db.collection("users").get();
// this is the list of promises/awaitables for all items
const promises = new Array<Promise<FirebaseFirestore.QuerySnapshot<FirebaseFirestore.DocumentData>>>();
sellerSnapshot.forEach(async (sellerDoc) => {
const sellerData = sellerDoc.data();
// THIS PART CAUSES THE API TO RETURN []
const sellerAuth = await mAuth.getUser(sellerData.UID);
// check for non null / empty strings
if (sellerData.Name as string && sellerData.UID as string) {
// this is all the seller information we need
itemSeller = new Seller(sellerData.Name, sellerData.UID, sellerAuth.photoURL); // placeholder profile picture
const refItem = sellerDoc.ref.collection("Items");
// push all the promises to a list so we can run all our queries in parallel
promises.push(refItem.get());
}
});
// wait for all promises to finish and get a list of snapshots
const itemSnapshots = await Promise.all(promises);
itemSnapshots.forEach((ItemSnapshot) => {
ItemSnapshot.forEach((ItemDoc) => {
// get the data
const itemData = ItemDoc.data();
// if title is not null, the rest of the fields are unlikely to be.
if (itemData.Title as string) {
// the rest of the logic to convert from database to model is in the constructor
arrayItem.push(new Item(ItemDoc.id, itemData.Title, itemSeller, itemData.Likes, itemData.ListedTime, itemData.Rating, itemData.Description, itemData.TransactionInformation, itemData.ProcurementInformation, itemData.Category, itemData.Stock, itemData.Image1, itemData.Image2, itemData.Image3, itemData.Image4, itemData.AdvertisementPoints, itemData.isDiscounted, itemData.isRestocked));
}
});
});
// sort by performance level
arrayItem = arrayItem.sort(x => x.Performance);
if (data.body.userID) {
arrayItem = await markLikedItems(data.body.userID, arrayItem);
}
//send the responseafter all the final modifications
response.send(arrayItem);
} catch (err) {
// log the error
console.log(err);
response.status(500).send(err);
}
});
I am building a chatbot with WATSON API where I use the async/await method in order to fetch the data from MongoDB and attain the result, which then I send it back to the user.
The function artpromise is the promise that collects data from mongo DB. And the function randomartist is a function that fetches 3 random document from the DB. However, the WATSON BLUEMIX Cloud service supports Nodejs SDK of 6.1.3 which does not support the async method. Is there any way to update the SDK version on Blumix or should I use a difference approach in fetching data from the server?
let getConversationResponse = (message, context) => {
let payload = {
workspace_id: process.env.WORKSPACE_ID,
context: context || {},
input: message || {}
};
payload = preProcess(payload);
return new Promise((resolved, rejected) => {
// Send the input to the conversation service
conversation.message(payload, async function(err, data) {
if (err) {
rejected(err);
}
else{
if(data.context.type == 'ask'){
let artist = data.context.name;
let result = await artpromise(artist);
console.log(result);
data.context.name = result[0].name;
data.context.nationality = result[0].nationality;
data.context.birth = result[0].years;
data.context.url = result[0].art_link;
data.output.text = data.context.name+' is a '+data.context.nationality+' artist from '+data.context.birth+'. Check out a painting at '+data.context.url;
}
else if(data.context.type == 'random_artist'){
let result = await randomArtist();
console.log(result);
data.output.text = 'Let\'s find some random artists for you! \n'+result;
}
let processed = postProcess(data);
if(processed){
// return 값이 Promise 일 경우
if(typeof processed.then === 'function'){
processed.then(data => {
resolved(data);
}).catch(err => {
rejected(err);
})
}
// return 값이 변경된 data일 경우
else{
resolved(processed);
}
}
else{
// return 값이 없을 경우
resolved(data);
}
}
});
})
}
Using Node's util.promisify() utility, you can transform a callback-style function into a Promise-based one.
Somewhere outside of your getConversationResponse-function, assign it to a local variable:
const util = require('util');
const messagePromise = util.promisify(conversation.message);
And use that function instead. Something like this should work:
const util = require('util');
const messagePromise = util.promisify(conversation.message);
let getConversationResponse = async (message, context) => {
let payload = preprocess({
workspace_id: process.env.WORKSPACE_ID,
context: context || {},
input: message || {}
});
let data = await messagePromise(payload);
if (data.context.type == 'ask') {
let artist = data.context.name;
let result = await artpromise(artist);
console.log(result)
data.context.name = result[0].name;
data.context.nationality = result[0].nationality;
data.context.birth = result[0].years;
data.context.url = result[0].art_link;
data.output.text = data.context.name+' is a '+data.context.nationality+' artist from '+data.context.birth+'. Check out a painting at '+data.context.url;
} else if (data.context.type == 'random_artist'){
let result = await randomArtist();
console.log(result);
data.output.text = 'Let\'s find some random artists for you! \n'+result;
}
return postProcess(data) || data;
};
Note that if the return value of postProcess is falsy, it will return the data variable instead. Additionally, an async function always returns a Promise, so to call this function, you'll do:
getConversationResponse(message, context).then((data) => {
// Do something with the data
}).catch((e) => {
// Handle the error!
});
or if you call it from another async function:
let data = await getConversationResponse(message, context);
or if you need to specifically catch errors in the calling async function:
try {
let data = await getConversationResponse(message, context);
} catch (e) {
// Handle error
}
Just like regular synchronous code, any error thrown in the function call chain "trickles up" to the top-most callee. If you're confused about this, I suggest reading up on error handling.
If you want to use the Watson API in an async Promise-based fashion throughout your code, it might be feasible to write a small wrapper library and use that directly instead.
A Promise-only implementation:
const util = require('util');
const messagePromise = util.promisify(conversation.message);
let getConversationResponse = (message, context) => {
let payload = preprocess({
workspace_id: process.env.WORKSPACE_ID,
context: context || {},
input: message || {}
});
return messagePromise(payload).then((data) => {
if (data.context.type == 'ask') {
let artist = data.context.name;
return artpromise(artist).then((result) => {
data.context.name = result[0].name;
data.context.nationality = result[0].nationality;
data.context.birth = result[0].years;
data.context.url = result[0].art_link;
data.output.text = data.context.name+' is a '+data.context.nationality+' artist from '+data.context.birth+'. Check out a painting at '+data.context.url;
return data;
});
} else if (data.context.type == 'random_artist') {
return randomArtist().then((result) => {
data.output.text = 'Let\'s find some random artists for you! \n' + result;
return data;
});
}
}).then((data) => {
return postProcess(data) || data;
});
};
Calling it is the exact same as the async/await implementation.