I'm uploading a file using AUI ajax file upload and I want to make the same ajax request in loop once the previous response is received. What I observed is that, the next loop is getting called without receiving the response from previous request. How can make sure that, I call next loop only after receiving response?
var loopCount = 0;
var requiredLoops = 10;
function uploadFile(){
loopCount = loopCount + 1;
A.io.request('${fileUploadCSV}', {
method: 'post',
form: {
id: '<portlet:namespace />uploadForm',
upload: true,
},
on: {
complete : function(e){
if(requiredLoops <= loopcount)
uploadFile(); // LOOP.
}
}
});
I made it work by adding uploadFile(); in setTimeout();
setTimeout(function(){uploadFile()}, 100);
Related
I'm attempting to write a very basic scraper that loops through a few pages and outputs all the data from each url to a single json file. The url structure goes as follows:
http://url/1
http://url/2
http://url/n
Each of the urls has a table, which contains information pertaining to the ID of the url. This is the data I am attempting to retrieve and store inside a json file.
I am still extremely new to this and having a difficult time moving forward. So far, my code looks as follows:
app.get('/scrape', function(req, res){
var json;
for (var i = 1163; i < 1166; i++){
url = 'https://urlgoeshere.com' + i;
request(url, function(error, response, html){
if(!error){
var $ = cheerio.load(html);
var mN, mL, iD;
var json = { mN : "", mL : "", iD: ""};
$('html body div#wrap h2').filter(function(){
var data = $(this);
mN = data.text();
json.mN = mN;
})
$('table.vertical-table:nth-child(7)').filter(function(){
var data = $(this);
mL = data.text();
json.mL = mL;
})
$('table.vertical-table:nth-child(8)').filter(function(){
var data = $(this);
iD = data.text();
json.iD = iD;
})
}
fs.writeFile('output' + i + '.json', JSON.stringify(json, null, 4), function(err){
console.log('File successfully written! - Check your project directory for the output' + i + '.json file');
})
});
}
res.send(json);
})
app.listen('8081')
console.log('Magic happens on port 8081');
exports = module.exports = app;
When I run the code as displayed above, the output within the output.json file only contains data for the last url. I presume that's because I attempt to save all the data within the same variable?
If I include res.send() inside the loop, so the data writes after each page, I receive the error that multiple headers cannot be sent.
Can someone provide some pointers as to what I'm doing wrong? Thanks in advance.
Ideal output I would like to see:
Page ID: 1
Page Name: First Page
Color: Blue
Page ID: 2
Page Name: Second Page
Color: Red
Page ID: n
Page Name: Nth Page
Color: Green
I can see a number of problems:
Your loop doesn't wait for the asynchronous operations in the loop, thus you do some things like res.send() before the asynchronous operations in the loop have completed.
In appropriate use of cheerio's .filter().
Your json variable is constantly being overwritten so it only has the last data in it.
Your loop variable i would lose its value by the time you tried to use it in the fs.writeFile() statement.
Here's one way to deal with those issues:
const rp = require('request-promise');
const fsp = require('fs').promises;
app.get('/scrape', async function(req, res) {
let data = [];
for (let i = 1163; i < 1166; i++) {
const url = 'https://urlgoeshere.com/' + i;
try {
const html = await rp(url)
const $ = cheerio.load(html);
const mN = $('html body div#wrap h2').first().text();
const mL = $('table.vertical-table:nth-child(7)').first().text();
const iD = $('table.vertical-table:nth-child(8)').first().text();
// create object for this iteration of the loop
const obj = {iD, mN, mL};
// add this object to our overall array of all the data
data.push(obj);
// write a file specifically for this invocation of the loop
await fsp.writeFile('output' + i + '.json', JSON.stringify(obj, null, 4));
console.log('File successfully written! - Check your project directory for the output' + i + '.json file');
} catch(e) {
// stop further processing on an error
console.log("Error scraping ", url, e);
res.sendStatus(500);
return;
}
}
// send all the data we accumulated (in an array) as the final result
res.send(data);
});
Things different in this code:
Switch over all variable declarations to let or const
Declare route handler as async so we can use await inside.
Use the request-promise module instead of request. It has the same features, but returns a promise instead of using a plain callback.
Use the promise-based fs module (in latest versions of node.js).
Use await in order to serialize our two asynchronous (now promise-returning) operations so the for loop will pause for them and we can have proper sequencing.
Catch errors and stop further processing and return an error status.
Accumulate an object of data for each iteration of the for loop into an array.
Change .filter() to .first().
Make the response to the request handler be a JSON array of data.
FYI, you can tweak the organization of the data in obj however you want, but the point here is that you end up with an array of objects, one for each iteration of the for loop.
EDIT Jan, 2020 - request() module in maintenance mode
FYI, the request module and its derivatives like request-promise are now in maintenance mode and will not be actively developed to add new features. You can read more about the reasoning here. There is a list of alternatives in this table with some discussion of each one. I have been using got() myself and it's built from the beginning to use promises and is simple to use.
I'm trying to write a function to use callbacks to send a message in Facebook messenger. I need to do this because I'm having problems sending text from an array. The messages are sent, but not in the correct order. I THINK this is because Nodejs is looping over the elements faster than it can send the text. See my question about this here.
So now I am trying to rewrite my send functions using callbacks, in the vain hope that I can somehow FORCE NodeJS to actually WAIT before jumping to the next element!
So far I have the following code:
Main send function:
sendWithCallback: function(messageData, callback) {
request({
uri: 'https://graph.facebook.com/v2.6/me/messages',
qs: {
access_token: config.FB_PAGE_TOKEN
},
method: 'POST',
json: messageData
}, function (error, response, body) {
if (!error && response.statusCode === 200) {
let recipientId = body.recipient_id;
console.log("Message sent to recipient '%s'", recipientId);
callback(true);
} else {
console.error("Could not send message: ", response.statusCode, response.statusMessage, body.error)
callback(false);
}
}
);
},
Function for sending a "multi part" message (i.e. an Array of text):
sendMultipartMessage: function(recipientId, textArray) {
let messageData, msgPart, msgLength, count = 0;
msgLength = textArray.length;
while (count < msgLength) {
msgPart = textArray[count];
messageData = {
recipient: {
id: recipientId
},
message: {
text: msgPart
}
};
}
self.sendWithCallback(messageData, function(sent) {
if (sent) {
count++;
console.log("Message part %s sent.", msgPart);
}
else {
console.log("Couldn't send message");
}
});
},
In my head, this code works properly! It sends the text (taken from the array), then increments the count until it is equal to messageLength. But in reality it DOESN'T do that. Instead, it just goes into an infinite loop (which I can't see happening in my logs) and then crashes the app.
WHAT am I doing wrong?
If we simplify your loop it essentially becomes this:
let count = 0;
while (count < msgLength) {
messageData = data;
}
You never increment count.
I think that you intend to move the self.sendWithCallback call inside of the while loop. However, this still won't do what you want and will run forever. Even if it did do what you wanted, it wouldn't solve the problem of sending messages out in order.
JavaScript's concurrency model uses an event loop with "run-to-completion." You can pass messages to the event queue using something like request which you call by sendWithCallback. This only adds a message to the queue, but that message is not processed until the current running block completes. That means that your while loop actually has to complete before any of your requests start running. We can construct a simpler example with setTimeout:
let count = 0;
while (count < 1) {
setTimeout(() => {
count++;
}, 1000);
}
console.log('while loop completed');
In the above the while loop never completes because count never gets incremented in the same block (console.log will never be called). It needs to complete before it can start processing the asynchronous messages you are creating with setTimeout.
You could actually just rewrite it like this:
textArray.forEach(msgPart => self.sendWithCallback(msgPart, sent => {
if (!sent) console.error('Could not send message');
});
However this doesn't guarantee the order that the messages were sent and it will send messages even if one of the messages triggers an error. If you want to send them in order you will have to recursively call sendWithCallback within the callback for the next message once the previous one completes. That might look something like this:
let count = 0;
const sendMessage = (textArray, count) => {
self.sendMessageWithCallback(textArray[count], sent => {
count++;
if (sent && count < textArray.length) {
sendMessages(textArray, count);
}
});
}
sendMessages(textArray, 0);
If you were using promises and async/await you could write this much more simply as something like:
for (count = 0; count < msgLength; count++) {
await self.sendMessageAsync(textArray[count]);
}
However this would require a larger rewrite of the surrounding code and using something like request-promise instead of just request.
I have been trying to retrieve the data using the MongoJS driver for node.js.The Code which I am using is as follows
req.on('end', function(){
var decodedBody = querystring.parse(fullBody);
story=decodedBody.name;
var z=new Array();
console.log(story);
res.writeHead(200,{'Content-Type': 'text/html'});
res.write('<html><body>');
db.frames.find({str_id:story}).toArray(function(err,doc){
console.log(doc);
for(var t=0;t<doc.length;t++)
{
var picid=doc[t].pic_id;
console.log(picid);
db.pictures.find({_id:picid}).toArray(function(err,pic){
res.write('<img src="'+pic[0].name+'"/>');
});
}
})
res.end('</body></html>');
});
The problem here is that because of the asynchronous nature of the code the response gets ends first and then the code inside the block of database gets executed and because of that nothing gets displayed on the browser i.e an image in this case .Thankx in advance.
Don't fight the asynchronous nature of node.js, embrace it!
So you should fire off all your requests, marking each one as completed when the response arrives. When all requests are completed, render your images and body/html closing tags.
I don't usually work with node.js, so I can make some mistakes, but it may look like this:
res.write('<html><body>');
db.frames.find({str_id:story}).toArray(function(err,doc){
console.log(doc);
var completed = {};
for(var t = 0; t < doc.length; t++) {
var picid = doc[t].pic_id;
completed.picid = false;
console.log(picid);
db.pictures.find({_id: picid}).toArray(function(err, pic) {
// mark request as completed
completed.picid = pic;
// check if all requests completed
var all_finished = true;
for(var k in completed) {
if(completed[k] === false) {
all_finished = false;
break;
}
}
// render final markup
if(all_finished) {
for(var k in completed) {
var pic = completed[k];
res.write('<img src="'+pic[0].name+'"/>');
}
res.end('</body></html>);
}
});
}
})
just put the res.end('</body></html>'); inside your db.frames.find function. Check when you reached doc.length - 1 and then send the end command.
In my chrome extension i was checking for a function which can stop my for loop from processing till it gets a response from content scripts. Sharing the sample code below
function abc() {
chrome.tabs.query({'status': 'complete'}, function(tabArray) {
for (var i = 0, tab; tab = tabArray[i]; i++) {
var currentUrl = tab.url;
var tabId = tab.id;
if (currentUrl.match(otherthing)) {
chrome.tabs.sendRequest(tabId, {'type': 'getrequiredthing'},
function(response) {
if (response.isrequiredthind) {
callfunction(tabId);
}
}
);
}
}
});
}
Here when i get the matching url in else if i'm sending a request to the page for getting some info, if my info is positive i need to callfunction. But here in the for loop tabId is iterating very fastly and even if the response is positive it is calling the callfunction with next(or next) tabId.
Can you give your opinions on solving this keep waiting the for loop this response is received.
Thanks
The problem is that the third argument to sendRequest does not block on the request being ready. By design, JavaScript almost never blocks. This is a Good Thing. Instead, it uses an "event-driven" model.
Another problem is due to lexical scoping: When callfunction is called, tabId has the most recent value, not the value when sendRequest was called. To get around this, you need to create a separate scope for each loop iteration e.g.
for (...) {
var tabId = ...;
if (...) {
(function (localTabId) {
chrome.tabs.SendRequest(..., function (response) {
if (response.isrequiredthind) {
callfunction(localTabId);
}
}
})(tabId);
}
}
I am newbie to nodejs.It's very hard to handle callbacks at nodejs level. I have code like this,
getItems(request,function(jsonObject){
var itemData={};
var itemDetails=new Array();
for(var i=0;i < jsonObject.length;i++){
getItemDetails(jsonObject[i].value.item_id,function(jsonObject){
itemDetails.push(jsonObject);
});
}
itemData["itemDetails"]=itemDetails;
response.contentType("application/json");
response.send({"data":itemData});
});
while executing the above code, the for loop is continuing with out getting callback from getItemDetails method and response sent to client. My requirement is the loop will wait until getting the call back from the getItemDetails then response should send.
I have tried with process.nextTick(), but i am unable to find where i have to use that process.nextTick().. Please anybody provide suggestions.
Thanks in advance.
You need to send the response only after you get all the items, so modify your code like so:
getItems(request,function(jsonObject) {
var itemData = {},
itemDetails = [],
itemsLeft = len = jsonObject.length,
i;
function sendResponse(itemDetails) {
itemData["itemDetails"] = itemDetails;
response.contentType("application/json");
response.send({ "data": itemData });
}
for (i = 0; i < len; i++) {
getItemDetails(jsonObject[i].value.item_id, function(jsonObject) {
itemDetails.push(jsonObject);
// send response after all callbacks have been executed
if (!--itemsLeft) {
sendResponse(itemDetails);
}
});
}
});
Note: I've used itemLeft here since it's a more generic way to solve these kind of problems, but Ianzz approach is also ok since you can compare the length of the two arrays.
You can't get the loop to wait, but you can modify your code to get the behavior you are expecting:
getItems(request,function(outerJsonObject){
var itemData={};
var itemDetails=new Array();
for(var i=0;i < outerJsonObject.length;i++){
getItemDetails(jsonObject[i].value.item_id,function(innerJsonObject){
itemDetails.push(innerJsonObject);
if (itemDetails.length == outerJsonObject.length) {
// got all item details
itemData["itemDetails"]=itemDetails;
response.contentType("application/json");
response.send({"data":itemData});
}
});
}
});