Provide progress on data processing - node.js

I've written a Node package that performs some intensive data processing. I'm able to watch progress via console.log, but I'd like to provide the end user with some way of monitoring it in an event-driven way. Basically, it would be like returning a promise, except instead of one then event it would have an event for each "step", and then finally resolve with the data.
Unfortunately, I don't know enough about Node streams (which I'm guessing is the thing I need) to do this. Can I get a few pointers? How can I create a stream which is updated every time there is, say, 1% more progress, and then finally gives the computed data?
EDIT: As an example, consider this loop:
for(let i = 0; i < N; i++) {
intensiveFunction();
console.log(`${i} of ${N} completed`);
}
What I want to do instead is
for(let i = 0; i < N; i++) {
intensiveFunction();
// send user a signal that i/N of the task has been completed
}

You don't need to use streams, You can use an EventEmitter, and emit the current progress, or any event you may want.
my-package.js
const EventEmitter = require('events');
// Create a class that extends from EventEmitter
// And now you can emit events when something happens, e.g., progress update
class MyPackage extends EventEmitter {
async intensiveFunction() {
// Something
}
async process() {
for(let i = 0; i < N; i++) {
await this.intensiveFunction();
this.emit('step', i, N); // Or send progress in % or whatever you want
}
}
}
module.exports = MyPackage;
index.js
const MyPackage = require('my-package');
const package = new MyPackage();
package.on('step', (step, total) => console.log(`${step}/${total}`));
package.process();
You can either provide a full events API, or mix it with promises. Meaning process can resolve once is done, or you can emit an end event, or do both.
async process() {
for(let i = 0; i < N; i++) {
await this.intensiveFunction();
this.emit('step', i, N); // Or send progress in % or whatever you want
}
// Emit the end event
this.emit('end', result);
// And resolve the promise
return result; // Whatever result is
}

Related

Should `drain` event be registered before the result of the write is available

I have this snippet of code:
const file = fs.createWriteStream('./test.txt');
let written = true;
// handler is added before even an attempt to write is made
file.on('drain', function () {
written = true;
console.log('drained');
});
const interval = setInterval(function () {
if (Date.now() - time > 10000) {
clearInterval(interval);
}
if (written) {
written = file.write(new Array(1000000).join('z'));
}
}, 100);
I'm wondering if that a standard practice to add handler even an attempt to write is made?
In case of using file.on('drain') listener you set up general listener to drain event of your stream.
Notice: This listener will be removed after closing of writable stream.
Generally that code will work proper, but most common practice in Node.js is to use stream.once('drain') handler for each case of internal buffer exceeding. That approach is covered in Node.js documentation for Event: 'drain':
function writeOneMillionTimes(writer, data, encoding, callback) {
var i = 1000000;
write();
function write() {
var ok = true;
do {
i -= 1;
if (i === 0) {
// last time!
writer.write(data, encoding, callback);
} else {
// see if we should continue, or wait
// don't pass the callback, because we're not done yet.
ok = writer.write(data, encoding);
}
} while (i > 0 && ok);
if (i > 0) {
// had to stop early!
// write some more once it drains
writer.once('drain', write);
}
}
}

turning a synchronous function into asynchronous function

I have a for loop which does many iterations .I would like to put that piece of code in a custom async function as it is blocking.Is there anyway I can write a function so it will call a callback once the loop iteration is over?.
Use asynchronous-function-inside-a-loop paradigm. This ensures that the asynchronous functions get called with the correct value of the index variable.
var total = someObject.list.length;
var count = 0;
for(var i = 0; i < total; i++){
(function(foo){
myobj.get(someObject.list[foo], function(err, response) {
do_something(foo);
count++;
if (count > total - 1) done();
});
}(i)); //To immediately invoke the function passing 'i' as parameter
}
function done() {
console.log('All data loaded');
}

c# Using SslStream.WriteAsync at high speeds

i am facing an annoying problem with SslStream.WriteAsync
here is the code
public void Send(PacketWriter writer)
{
var buffer = writer.GetWorkspace();
_sslStream.WriteAsync(buffer, 0, buffer.Length);
}
When writing the data at extremely high speed it tells me
The beginwrite method cannot be called when another write operation is pending
[NOTE] : I mean by high speed something like this
for (var i = 0 ; i <= 1000; i++)
{
Send(somedata);
}
You must wait until the previous asynchronous write has finished.
Change your send method to return a Task:
public Task Send(PacketWriter writer)
{
var buffer = writer.GetWorkspace();
return _sslStream.WriteAsync(buffer, 0, buffer.Length);
}
Make the calling method async, then await until each send operation completes:
for (var i = 0 ; i <= 1000; i++)
{
await Send(somedata);
}
May be try this-
private static void send_data(SslStream socket_stream, byte[] data)
{
Task task = new Task(() => { socket_stream.Write(data, 0, data.Length); });
task.RunSynchronously();
task.Wait();
}
This will make sure all the messaages are sent in oderly manner without any exception

spawn multiple threads in loop and knowing when all finished to do certain task

I have to spawn multiple threads in loop for configured no. of sites and then each thread has multiple child threads. After finishing all threads I need to do certain task to save data into database. How I come to know whether all task has finished.
my current code:
List<SocialDataConfig> configList = SetConfiguration();
foreach (SocialDataConfig config in configList)
{
new Thread(delegate()
{
FetchSocialData(config);
}).Start();
}
// save fetched data into database and log fetched information in database
private void FetchSocialData(SocialDataConfig config)
{
for (int i = 0; i < config.ThreadCount; i++)
{
Thread thread = new Thread(delegate()
{
FetchData(54764876, config);
});
thread.Start();
}
}
Instead of creating your own threads you can use the new Task library in C#.
You can then use something like this:
int n = 10;
// Construct started tasks
Task<int>[] tasks = new Task<int>[n];
for (int i = 0; i < n; i++)
{
tasks[i] = Task<int>.Factory.StartNew(action, i);
}
// Wait for all the tasks to finish.
Task.WaitAll(tasks);
Here is the MSDN documentation on WaitAll with a complete example.

Phased execution recursive cycle in Node.js

I'm new to Node.js and asynchronous programming in general, and I have a problem. I need for continuous work server in a recursive loop, which steps would be implemented consistently, and each stage should consist of an asynchronous receive data over websotskets (this is already has understood), or go to next stage on the expiration of timer.
I am aware that is it not a trivial task, but how to implement it, what libraries can help?
I tried to understand the Step.js and Node.js events, this is what i need?
If I wrote all this in sync:
//CODE FOR STACKOVERFLOW
var previous_round = 'qwerty';// this is string, selected in previous cycle
var round_users = users;// 'users' is array of soceket.id all connected users and here a put them into new array 'round_users' with sockets for current cycle
io.sockets.emit('users_round', { number: round_users.length }); // send to users information about number of users in new round
for (var i = 0; i < round_users.length; i++) {
io.sockets.socket(round_users[i]).emit('round', { text: previous_round });//for each socket in 'round_users' emit event to enter chat message
} // and send selected in previous round message
var messages = []; // array of users messages
//now we must listen users events and start timer for next stage
//listen users events
for (var i = 0; i < round_users.length; i++) {
io.sockets.socket(round_users[i]).on('message', function (data) {
messages[messages.length] = data.text;//write text from users to array
if (messages.length == round_users.length) { /* GO TO NEXT STAGE */ } // it's after previous operation in this function
});
}
//or set timeout
setTimeout(/* GO TO NEXT STAGE */,15000);
for (var i = 0; i < round_users.length; i++) {
io.sockets.socket(round_users[i]).emit('voting', { messages_array: messages });//for each socket in 'round_users' emit event to vote perfect chat message
} // and send messages, which they send to server
//i'm not quite sure about this decision of vote counting :-)
var votes = []; //array with users votes
for (var i = 0; i < messages.length; i++) {
votes[i] = 0;
}
//now we must listen users events and start timer
//listen users events
for (var i = 0; i < round_users.length; i++) {
io.sockets.socket(round_users[i]).on('vote', function (data) {
votes[data.number]++;//increment selected message
if (votes.length == messages.length) { /* GO TO NEXT STAGE */ } // it's after previous operation in this function
});
}
//or set timeout
setTimeout(/* GO TO NEXT STAGE */,10000);
var max_id = 0; //now select max number from 'votes' array
for (var i = 0; i < votes.length; i++) {
if (votes[i]>votes[max_id]) {max_id = i;} //yet without the same values
}
var previous_round = messages[max_id]; //set up string value of selected message
//restart cycle
this code on pastebin with syntax highlighting
Recursion in the most basic sense is just a function that calls itself over and again, but in this instance with node you may want to utilize process.nextTick() to allow other events to occur during this recursion.
Here is a simple example of one way you might be able to go about it:
function someFunction(args, callback) {
// Check requirements
if (typeof callback !== 'function') return console.log('Callback required');
if (typeof args === 'undefined') return callback('Arguments required');
// Set defaults
var err = false;
var result = false;
// Do other stuff
anotherFunctionWithAcallback(args, function(err, result) {
if (err) return callback(err);
// This way, the nextTick only occurs after processing completes
return callback(err, result);
}
}
(function loop(stage) {
// When the stage argument is not provided, default to stage 1
stage = (typeof stage === 'undefined') ? 1 : stage;
switch(stage) {
case 1:
/* stage 1 */
someFunction(args, function(err, result){
process.nextTick(loop(2));
});
break;
case 2:
/* stage 2 */
someFunction(args, function(err, result){
process.nextTick(loop(3));
});
break;
case 3:
/* stage 3 */
someFunction(args, function(err, result){
// No stage argument, restart at stage 1 by default
process.nextTick(loop());
});
break;
}
})(); // Execute this function immediately

Resources