I use the library node-redis: https://github.com/NodeRedis/node-redis
let client = redis.createClient();
let blpopAsync = util.promisify(client.blpop).bind(client);
let rpushAsync = util.promisify(client.rpush).bind(client);
async function consume() {
let data = await blpopAsync('queue', 0); // the program is blocked at here
}
async function otherLongRunningTasks() {
// call some redis method
await rpushAsync('queue', data);
}
I expect that method blpopAsync will be blocked until a element is popped. And in this time, event loop will run other async functions. However, my program is blocked at await blpopAsync forever.
Could you tell me how to use this function correctly to not block event-loop?
I find that blpop blocks other non-blocking methods of redis client, so other functions which use redis will wait forever. The event loop is still running.
As #AnonyMouze mentioned, doing client.duplicate().blpop(...) is the trick to freeing the connection.
i currently cannot comment so i will have to post it here. blpop will block when there is no data. Another thing to consider is the Redis's version . blpop behave differently between versions. You can refer to redis documentation .
Related
As i understand Nodejs has a event loop based mechanism, therefore the main nodejs code is single threaded which just tries to execute the methods and callbacks present in the call stack, therefore at a single point of time in the nodejs runtime only one thread executes(main thread), so if i define a data structure like a queue or map i need not to worry about locking it or blocking others accessing it when one of the callbacks is using because
unless a callback is done executing the main thread will keep executing ( unlike multithreaded application where a thread could execute for a bit and then contexted switched out to give chance to some other thread.)
no 2 callbacks are executed in parallel
Example:
let arr = [];
const fun = async (website) => {
const result = await fetch(website, {}, "POST");
arr.push(...result); }
const getData = async () => {
let promises = [];
for (let i = 0; i < 10; i++) {
promises.push(fun(`website${i}`));
}
Promise.all(promises);
}
So my question is do i ever need to lock a resource on the nodejs side of thing,
i do understand the libuv is multithreaded therefore when calling async operations that use libuv (ex writing to file) could cause problems if i don't lock.
But my question is do i ever need a lock in the nodejs runtime.
Is it possible to send a response from an endpoint and then continue running ?(in the background) So when the endpoint gets called the caller gets some response but the server continues doing things :
app.at("/endpoint").post(myendpoint);
async fn myendpoint(mut req: Request<State>) -> tide::Result {
let body= Body::from_json(some_json).unwrap();
Ok(body.into()) //continue doing stuff after this (calling another function)
}
You can spawn a thread. Even after you process your request and return the response, the thread will continue doing what it needs to do for as long as needed.
An example:
use std::thread;
let hnd = thread::spawn(|| {
// Put your thread code here
});
That's a regular thread, no async stuff. Generally, if you expect hundreds or thousands of concurrent calls, you might consider another, more-scalable approach.
Read more on thread spawning here.
I am very new to node js and socket io. Can this code lead to a race condition on counter variable. Should I use a locking library for safely updating the counter variable.
"use strict";
module.exports = function (opts) {
var module = {};
var io = opts.io;
var counter = 0;
io.on('connection', function (socket) {
socket.on("inc", function (msg) {
counter += 1;
});
socket.on("dec" , function (msg) {
counter -= 1;
});
});
return module;
};
No, there is no race condition here. Javascript in node.js is single threaded and event driven so only one socket.io event handler is ever executing at a time. This is one of the nice programming simplifications that come from the single threaded model. It runs a given thread of execution to completion and then and only then does it grab the next event from the event queue and run it.
Hopefully you do realize that the same counter variable is accessed by all socket.io connections. While this isn't a race condition, it means that there's only one counter that all socket.io connections are capable of modifying.
If you wanted a per-connection counter (separeate counter for each connection), then you could define the counter variable inside the io.on('connection', ....) handler.
The race conditions you do have to watch out for in node.js are when you make an async call and then continue the rest of your coding logic in the async callback. While the async operation is underway, other node.js code can run and can change publicly accessible variables you may be using. That is not the case in your counter example, but it does occur with lots of other types of node.js programming.
For example, this could be an issue:
var flag = false;
function doSomething() {
// set flag indicating we are in a fs.readFile() operation
flag = true;
fs.readFile("somefile.txt", function(err, data) {
// do something with data
// clear flag
flag = false;
});
}
In this case, immediately after we call fs.readFile(), we are returning control back to the node.js. It is free at that time to run other operations. If another operation could also run this code, then it will tromp on the value of flag and we'd have a concurrency issue.
So, you have to be aware that anytime you make an async operation and then the rest of your logic continues in the callback for the async operation that other code can run and any shared variables can be accessed at that time. You either need to make a local copy of shared data or you need to provide appropriate protections for shared data.
In this particular case, the flag could be incremented and decremented rather than simply set to true or false and it would probably serve the desired purpose of keeping track of whether this file is current being read or not.
Shorter answer:
"Race condition" is when you execute a series of ordered asynchronous functions and because of their async nature they won't finish processing in their original order.
In your code, you are executing a series of ordered synchronous process (increasing or decreasing the counter), So they finish instantly after they start, resulting in ordered output. So no racing here!
What does the line of code below do?
Npm.require('fibers/future');
I looked online for examples and I came across a few like this:
Future = Npm.require('fibers/future');
var accessToken = new Future();
What will accessToken variable be in this case?
Question is a bit old but my 2 cents:
As Molda said in the comment, Future's main purpose is to make async things work synchronously.
future instance comes with 3 methods:
future.wait() basically tells your thread to basically pause until told to resume.
future.return(value), first way to tell waiting future he can resume, it's also very useful since it returns a value wait can then be assigned with, hence lines like const ret = future.wait() where ret becomes your returned value once resumed.
future.throw(error), quite explicit too, makes your blocking line throw with given error.
Making things synchronous in javascript might sound a bit disturbing but it is sometimes useful. In Meteor, it's quite useful when you are chaining async calls in a Meteor.method and you want its result to be returned to the client. You could also use Promises which are now fully supported by Meteor too, I've used both and it works, it's up to your liking.
A quick example:
Meteor.methods({
foo: function() {
const future = new Future();
someAsyncCall(foo, function bar(error, result) {
if (error) future.throw(error);
future.return(result);
});
// Execution is paused until callback arrives
const ret = future.wait(); // Wait on future not Future
return ret;
}
});
When I receive an "on" event on the server side, I want to start a task in parallel so it does not block the current event loop thread. Is it possible to do so? How?
I don't want to block the server side loop and I want to be able to send back a message to the client once the task is done, something such as:
client.on('execute-parallel-task', function(msg) {
setTimeout(function() {
// do something that takes a while
client.emit('finished-that-task');
},0);
// this block should return asap, not waiting for the previous call
});
I am not sure if setTimeout will do the job.
It depends what the takes a while is. If it takes a while asynchronously (you can tell because you'll have to register a callback or complete handler), and takes a while because it's blocked on something like IO, rather than CPU bound, it'll inherently be parallel.
If however, its something synchronous or CPU bound, whilst you can use setTimeout, setImmediate etc. to send back a message immediately, once the handler for setTimeout or setImmediate executes, your single thread of execution will be stuck handling that; you're not really fixing the problem, merely deferring it.
To exhibit true parallel behaviour, you'll need to launch a child process. You can use the message passing functionality to notify your worker what work to do, and to notify the parent process once the work is complete.
var cp = require('child_process');
var child = cp.fork(__dirname + '/my-child-worker.js');
n.on('message', function(m) {
if (m === "done") {
// Whey!
}
});
n.send(/* Job id, or something */);
Then in my-child-worker.js;
process.on('message', function (m) {
switch (m) {
case 'get-x':
// blah
break;
// other jobs
}
process.send('done');
});
you do not need the setTimeout.
Your function(msg) will be called once the execute parallel task finishes.
if you are designing a task to run in an async manner, you can look at something like the async lib for node.js
Async Node JS Link