Redis is very fast. For most part on my machine it is as fast as say native Javascript statements or function calls in node.js. It is easy/painless to write regular Javascript code in node.js because no callbacks are needed. I don't see why it should not be that easy to get/set key/value data in Redis using node.js.
Assuming node.js and Redis are on the same machine, are there any npm libraries out there that allow interacting with Redis on node.js using blocking calls? I know this has to be a C/C++ library interfacing with V8.
I suppose you want to ensure all your redis insert operations have been performed. To achieve that, you can use the MULTI commands to insert keys or perform other operations.
The https://github.com/mranney/node_redis module queues up the commands pushed in multi object, and executes them accordingly.
That way you only require one callback, at the end of exec call.
This seems like a common bear-trap for developers who are trying to get used to Node's evented programming model.
What happens is this: you run into a situation where the async/callback pattern isn't a good fit, you figure what you need is some way of doing blocking code, you ask Google/StackExchange about blocking in Node, and all you get is admonishment on how bad blocking is.
They're right - blocking, ("wait for the result of this before doing anything else"), isn't something you should try to do in Node. But what I think is more helpful is to realize that 99.9% of the time, you're not really looking for a way to do blocking, you're just looking for a way to make your app, "wait for the result of this before going on to do that," which is not exactly the same thing.
Try looking into the idea of "flow control" in Node rather than "blocking" for some design patterns that could be a clearer fit for what you're trying to do. Here's a list of libraries to check out:
https://github.com/joyent/node/wiki/modules#wiki-async-flow
I'm new to Node too, but I'm really digging Async: https://github.com/caolan/async
Blocking code creates a MASSIVE bottleneck.
If you use blocking code your server will become INCREDIBLY slow.
Remember, node is single threaded. So any blocking code, will block node for every connected client.
Your own benchmarking shows it's fast enough for one client. Have you benchmarked it with a 1000 clients? If you try this you will see why blocking code is bad
Whilst Redis is quick it is not instantaneous ... this is why you must use a callback if you want to continue execution ensuring your values are there.
The only way I think you could (and am not suggesting you do) achieve this use a callback with a variable that is the predicate for leaving a timer.
Related
I'm trying to connect to a 3rd party library, that has a function that can block. I would like to use it, but without blocking. Is it possible to wrap a blocking call that I don't have the control of, to make it async?
// calling this function will block the nodejs thread
blockingCall();
What I would like would be something like this.
// wrapper for the blocking call
var wrapper = wrapBlockingCall(blockingCall);
wrapper.on('complete', function() {});
Is this possible? Does this make sense?
There is no way to make a blocking JavaScript code non-blocking in Node.js - the mechanism which Node uses for its non-blocking behaviour is implemented in the C/C++ layer, which in turn is used only when doing I/O operations (reading from disk, networking etc.).
In reality, every line of JavaScript your program uses will be executed one-by-one, because it is always executed on the same thread, no matter what you do.
The only option I see is to execute the offending code in a separate Node process using the built-in Child Process module. However, this will have significant performance impact, even bigger one if the code needs to be executed frequently.
Note:
After reading the comments under your question, it seems that you are actually the author of the blocking function, which in turn calls a C API which performs blocking I/O. There are ways of calling C functions which would normally block in a manner which does not block the upper JavaScript layer.
While I am not a C expert, I think this is accomplished using the libuv library included in Node - have a look at the addons documentation for more info.
Why and when should we prefer read/writeFileSync to the asynchronous ones on Nodejs in particular for server applications?
For them are blocking functions, you should ever prefer the async versions in production environments.
Anyway, it could make sense to use them during the bootstrap of your application, as an example if you use to load configurations from files and you don't want either to let your fully initialized components to interact with partially initialized ones or design them to be able to work with each other in such a case.
I cannot see any other meaningful use for them.
I prefer to use writeFileSync or readFileSync if it does not make any sense for the program to continue in the event of a failure and there is no other asynchronous work to do. Typically this is at the beginning/end of a program, but I have also used them while "checkpointing" a long running process to make it clear that nothing is changing while the checkpoint is being written to disk.
Of course you can implement this with the asynchronous versions too, it is just more verbose and prone to mistakes.
Does Go have an equivalent of node.js' "emitter"?
I'm teaching myself Go by porting over a node.js library I wrote. In the node version, the library emits an event once something happens (e.g. it listens on UDP port 1234 and when "ABC" is received, "abcreceived" is emitted so the calling code can respond as necessary (e.g. sending back "DEF")
I've seen channels in Go (and am currently reading up on them), but as I'm still new to this language, I don't know if (or how, for that matter) that can be used to communicate with whatever code is using my library.
I've also seen https://github.com/chuckpreslar/emission, but am not sure if this is acceptable, or if there's a better ("Best practice") way of doing things.
Go and Node.js are very different. Node.js supports concurrency only via callbacks. There might be various ways of dressing them up, but they're fundamentally callbacks.
In Node.js, there is no parallelism; Node.js has a single-threaded runtime. When Node.js async is used to achieve what is called 'parallel' execution, it isn't parallel in the sense used in Go, but concurrent.
Concurrency is not parallelism in the Go world.
Go has explicit concurrency based on Communicating Sequential Processes (CSP), a mathematical basis conceived by Tony Hoare at Oxford. The runtime interleaves cooperating processes called goroutines by time-slicing them onto the available CPU cores. Within each goroutine, the code is single threaded, so is easy to write. In the simple case, no data is shared between goroutines; instead messages pass between them along channels. In this way, there is no need for callbacks.
When goroutines get blocked waiting for I/O, that's OK because they don't use any CPU time until they're unblocked. Their memory footprint is slight and you can have very large numbers of them. So callbacks are not needed for I/O operations either.
Because the execution models of Go and Node.js are about as different as they could be, attempting to port code from one to the other is very likely to lead to very clumsy solutions. It's better to start from the original requirements and implement from scratch.
It would be possible to distort the Go concurrency model using function arguments to behave like callbacks. This would be a bad idea because it would not be idiomatic and would lose the benefits that CSP gives.
So by reading others' Go code and some links in the comments to my question, I think channels are the way to go.
In my library code (semi pseudo-code):
// Make a new channel called "Events"
var Events = make(chan
func doSomething() {
// ...
Events <-"abcreceived" // Add "abcreceived" to the Events channel
}
And in the code that will use my library:
evt := <-mylib.Events
switch evt {
case "abcreceived":
sendBackDEF()
break
// ...
}
I still prefer node.js' EventEmitter (because you can transfer data back easily) but for simple things, this should suffice.
This is a general question about testing, but I will frame it in the context of Node.js. I'm not as concerned with a particular technology, but it may matter.
In my application, I have several modules that are called upon to do work when my web server receives a request. In the case of some of these modules, I close the request before I call upon them.
What is a good way to test that these modules are doing what they are supposed to do?
The advice here for RSpec is to mock out the work these modules are doing and just ensure that the appropriate methods are being called. This makes sense to me, but in Node.js, since my modules are not global, I don't think I cannot mock out functions without changing my program architecture so that every instance receives instances of objects that it needs1.
[1] This is a well known programming paradigm, but I cannot remember its name right now.
The other option I see is to use setTimeout and take my best guess at when these modules are done with their work.
Neither of these seems ideal.
Am I missing something? Are background processes not tested?
Since you are speaking of integration tests of these background components, a few strategies come to mind.
Take all the asynchronicity out of their operation for test mode. I'm imagining you have some sort of queueing process (that could be a faulty assumption), you toss work into the queue, and then your modules pick up that work and do their task. You could rework your test harness such that the test harness stands in as the queuing mechanism and you effectively get direct control over when the modules execute.
Refactor your modules to take some sort of next callback function. They would end up functioning a bit like Express's middleware layer or how async's each function works, but into each module you'd pass some callback that you call when that module's task is complete. Once all of the modules have reported in, then you can check the state of the program.
Exactly what you already suggested-- wait some amount of time, and if it still isn't done, consider that a failure. Mocha sort of does that, in that if a given test is over a definable threshold, then it's a failure. I don't like this way though, because if you add more tests, they all have to wait the same amount of time.
So, after reading a little about non-blocking code, does...
response.write(thisWillTakeALongTime());
...block the process? If so, do we need to pass the response into pretty much every slow function call we make, and have that function handle the response?
Thanks for helping to clarify!
Yes, it will block the event loop. And passing the response object into the slow function won't help, no matter where you call the slow function you will be blocking the event loop.
As to how to fix it, we will need more information.
What is making your slow function slow? Are you performing large calculations?
Are you doing sync versions of file/database calls?
It depends on what you mean by process. The web server has already finished serving the page at this point you js will execute however the request is synchronous so the javascript will continue to devote its in your function until it returns regardless even if take years. (hopefully by this point the browser will detect your script is taking too long and give you the opportunity to kill it). Even still you suffer the embarrassment of the user having to kill your javascript functionality and them not being able to use the page.
So how do you solve the problem. The time when this gets particular important is when your js is making the problem because at the point a whole host of things of things can go wrong. Imagine that your user is on the other side of the earth. the network latency could make your js painfully slow. when using ajax its preferable to use Asynchronous requests which get around this. I personally recommend using jquery as it makes async ajax calls really easy and the documentation on the side is quite straight forward. The other thing I recommend is making the return output small. It made be better to return json output and build the needed html from that.