i found the plugin mongo-sync which can make me to use mongoDB synchronized
on the git,there show :
It is a thin wrapper around the official MongoDB driver for Node. Here is a quick usage example that you can use with Common Node:
var Server = require("mongo-sync").Server;
var server = new Server('127.0.0.1');
var result = server.db("test").getCollection("posts").find().toArray();
console.log(result);
server.close();
how can i use like this?
it's mentioned that use with Common Node
whether it's means common-node?
so, how can i use it ? or use mongo-sync straightforwardly?
It means you have to follow Common-Node installation instructions and use common-node command instead of the plain-old node to run your program.
As the docs mention, to use it with plain-old node you need to use node-fibers and make queries inside a Fiber.
No way around node-fibers I'm afraid, as mongo-sync is just a "synchronous" wrapper around asynchronous mongo driver, and it's hard to make async js code synchronous without some low level monkey-patching.
Related
Let's say I'd want to control (start/stop) several other NodeJS scripts from within one "main" NodeJS app. However, not necessarly NodeJS Scripts exclusively, but also simple bash scripts.
I'm looking into the following solution, using execa
// simplified
const managedProcesses = [];
async function Start (pkg) {
const runningProcess = await execa(pkg.run_command, {
cwd : pkg.path
});
return runningProcess;
}
async function Stop (pkg) {
// somehow stop the child process
return
}
const someProcess = await Start({
run_command : 'node app.js',
path : './path/to/my/script/'
});
// Keep Reference of process
managedProcesses.push(someProcess);
I first thought pm2 would be a viable solution, but I guess this would only fit for NodeJS-only scripts.
What Problems could I run into using the approach above ?
Should I consider moving forward with this idea ?
For node.js subprocesses there is the cluster module and I strongly recommend using this. For general subprocesses (e.g. bash scripts as you mentioned) you have to use child_process (-> execa). Communication between processes may then be accomplished via grpc. Your approach is fine, so you can consider moving forward with it.
I decided to go full with pm2 for the time being, as they have an excellent programmatic API - also (which I only just learned about) you can specify different interpreters to run your script. So not only node apps are possible but also bash, python, php and so on - which is exactly what I am looking for.
FYI - I know how to use the MongoDB driver and know this is not how one would use it in a web app, but this is not for a web app. My aim is to emulate the MongoDB shell in NodeJS
I'm writing a DB GUI and would like to execute a raw MongoDB query, eg db.tableName.find({ col: 'value' }). Can I achieve this using the native MongoDB driver? I'm using v2.2, which is current the latest version.
If not, how can I achieve this in NodeJS?
Note: The question has changed - see the updates below.
Original answer:
Yes.
Instead of:
db.tableName.find({ col: 'value' })
You use it as:
db.collection('tableName').find({ col: 'value' }, (err, data) => {
if (err) {
// handle error
} else {
// you have data here
}
});
See: http://mongodb.github.io/node-mongodb-native/2.2/api/Collection.html#find
Update
After you changed your question and posted some comments it is more clear what you want to do.
To achieve your goal of emulating the Mongo shell in Node you would need to parse the command typed by the user and execute the appropriate command while keeping in mind:
the difference between SpiderMonkey used by the Mongo shell and Node with V8 and libuv
the difference between BSON and JSON
the fact that Mongo shell works synchronously and the Node driver works asynchronously
The last part will probably be the hardest part for you. Remember that in the Mongo shell this is perfectly legal:
db.test.find()[0].x;
In Node the .find() method doesn't return the value but it either takes a callback or returns a promise. It will be tricky. The db.test.find()[0].x; case may be relatively easy to handle with promises (if you understand the promises well) but this will be harder:
db.test.find({x: db.test.find()[0].x});
and remember that you need to handle arbitrarily nested levels.
The Mongo protocol
After reading some of the comments I think it's worth noting that what you actually send to the Mongo server has nothing to do with the JavaScript that you write in the Mongo shell. The Mongo shell uses SpiderMonkey with a number of predefined functions and objects.
But you don't actually send JavaScript to the Mongo server so you can't send things like db.collection.find(). Rather you send a binary OP_QUERY struct with a collection name encoded as a cstring and a query encoded as BSON plus a bunch of binary flags. See:
https://docs.mongodb.com/manual/reference/mongodb-wire-protocol/#wire-op-query
The BSON is itself a binary format with a number of low level values defined as bytes:
http://bsonspec.org/spec.html
The bottom line is that you don't send to the Mongo server anything resembling what you enter in the Mongo shell. The Mongo shell parses the things that you type using the SpiderMonkey parser and sends binary requests to the actual Mongo server. The Mongo shell uses JavaScript but you don't communicate with the Mongo server in JavaScript.
Example
Even the JSON query object is not sent to Mongo as JSON. For example, when you are searching for a document with a hello property equal to "world" you would use {hello: 'world'} in JavaScript or {"hello": "world"} in JSON but this is what gets send to the Mongo server - by the Mongo shell or by any other Mongo client:
\x16\x00\x00\x00\x02hello\x00\x06\x00\x00\x00world\x00\x00
Why it's so different
To understand why the syntax used in Node is so different from the Mongo shell, see this answer:
Why nodejs-mongodb middleware has different syntax than mongo shell?
(Using Sails.js)
I am testing webworker-threads ( https://www.npmjs.com/package/webworker-threads ) for long running processes on Node and the following example looks good:
var Worker = require('webworker-threads').Worker;
var fibo = new Worker(function() {
function fibo (n) {
return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}
this.onmessage = function (event) {
try{
postMessage(fibo(event.data));
}catch (e){
console.log(e);
}
}
});
fibo.onmessage = function (event) {
//my return callback
};
fibo.postMessage(40);
But as soon as I add any code to query Mongodb, it throws an exception:
(not using the Sails model in the query, just to make sure the code could run on its own -- db has no password)
var Worker = require('webworker-threads').Worker;
var fibo = new Worker(function() {
function fibo (n) {
return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}
// MY DB TEST -- THIS WORKS FINE OUTSIDE THE WORKER
function callDb(event){
var db = require('monk')('localhost/mydb');
var users = db.get('users');
users.find({ "firstName" : "John"}, function (err, docs){
console.log(("serviceSuccess"));
return fibo(event.data);
});
}
this.onmessage = function (event) {
try{
postMessage(callDb(event.data)); // calling db function now
}catch (e){
console.log(e);
}
}
});
fibo.onmessage = function (event) {
//my return callback
};
fibo.postMessage(40);
Since the DB code works perfectly fine outside the Worker, I think it has something to do with the require. I've tried something that also works outside the Worker, like
var moment = require("moment");
var deadline = moment().add(30, "s");
And the code also throws an exception. Unfortunately, console.log only shows this for all types of errors:
{Object}
{/Object}
So, the questions are: is there any restriction or guideline for using require inside a Worker? What could I be doing wrong here?
UPDATE
it seems Threads will not allow external modules
https://github.com/xk/node-threads-a-gogo/issues/22
TL:DR I think that if you need to require, you should use a node's
cluster or child process. If you want to offload some cpu busy work,
you should use tagg and the load function to grab any helpers you
need.
Upon reading this thread, I see that this question is similar to this one:
Load Nodejs Module into A Web Worker
To which Audreyt, the webworker-threads author answered:
author of webworker-threads here. Thank you for using the module!
There is a default native_fs_ object with the readFileSync you can use
to read files.
Beyond that, I've mostly relied on onejs to compile all required
modules in package.json into a single JS file for importScripts to
use, just like one would do when deploying to a client-side web worker
environment. (There are also many alternatives to onejs -- browserify,
etc.)
Hope this helps!
So it seems importScripts is the way to go. But at this point, it might be too hacky for what I want to do, so probably KUE is a more mature solution.
I'm a collaborator on the node-webworker-threads project.
You can't require in node-webworker-threads
You are correct in your update: node-webworker-threads does not (currently) support requireing external modules.
It has limited support for some of the built-ins, including file system calls and a version of console.log. As you've found, the version of console.log implemented in node-webworker-threads is not identical to the built-in console.log in Node.js; it does not, for example, automatically make nice string representations of the components of an Object.
In some cases you can use external modules, as outlined by audreyt in her response. Clearly this is not ideal, and I view the incomplete require as the primary "dealbreaker" of node-webworker-threads. I'm hoping to work on it this summer.
When to use node-webworker-threads
node-webworker-threads allows you to code against the WebWorker API and run the same code in the client (browser) and the server (Node.js). This is why you would use node-webworker-threads over node-threads-a-gogo.
node-webworker-threads is great if you want the most lightweight possible JavaScript-based workers, to do something CPU-bound. Examples: prime numbers, Fibonacci, a Monte Carlo simulation, offloading built-in but potentially-expensive operations like regular expression matching.
When not to use node-webworker-threads
node-webworker-threads emphasizes portability over convenience. For a Node.js-only solution, this means that node-webworker-threads is not the way to go.
If you're willing to compromise on full-stack portability, there are two ways to go: speed and convenience.
For speed, try a C++ add-on. Use NaN. I recommend Scott Frees's C++ and Node.js Integration book to learn how to do this, it'll save you a lot of time. You'll pay for it in needing to brush up on your C++ skills, and if you want to work with MongoDB then this probably isn't a good idea.
For convenience, use a Child Process-based worker pool like fork-pool. In this case, each worker is a full-fledged Node.js instance. You can then require to your heart's content. You'll pay for it in a larger application footprint and in higher communication costs compared to node-webworker-threads or a C++ add-on.
i'm using the request module to fetch image from website and pipe it into local file, the code is like:
url = http:/xxx.com/x.jpg;
if(url){
request(url).pipe(localFilePath);
}
if(xxx){
// save the localFilePath to db;
redirect('/index');
}
// The question is the filePath is needed in the index page, so if the file has not downloaded yet, then it can not show the file on index page.
i tried.
request(url).pipe(...).on('end',function(){
....
});
but it seems does't work..
so, i wonder how to do like :
yield xxxxx in node v0.11.x to pause the process until the file is already downloaded completely?
thanks
Yield is only presently available in Node 0.11 (when using the –harmony flag), but this is an unstable release that is probably not suitable for any kind of production use. Node 0.12 shouldn’t be too far away though, as 0.11 has been in development for a while now. So the good news is generators will be available in a stable Node.js release near you very soon!
If you want to stick with 0.10.x you will have to use callbacks or promises for now.
I am new to the 'nodejs' world.So wanting to explore the various technologies,frameworks involved i am building a simple user posts system(users posting something everybody else seeing the posts) backed by redis.I am using express framework which is recommended by most tutorials.But i have some difficulty in gettting data from the redis server i need to do 3 queries from the redis server to display the posts.In which case have to use neested callback after each redis call.So i wanted to use streamline.js to simplify the callbacks.But i am unable to get it to work even after i used npm install streamline -g and require('streamline').register(); before calling
var keys=['comments','timestamp','id'];
var posts=[];
for(var key in keys){
var post=client.sort("posts",'by','nosort',"get","POST:*->"+keys[key],_);
posts.push(post);
}
i get the error ReferenceError: _ is not defined.
Please point me in the right direction or point to any resources i might have missed.
The require('streamline').register() call should be in the file that starts your application (with a .js extension). The streamline code should be in another file with a ._js extension, which is required by the main script.
Streamline only allows you to have async calls (calls with _ argument) at the top level in a main script. Here, your streamline code is in a module required by the main script. So you need to put it inside a function. Something like:
exports.myFunction = function(_) {
var keys=['comments','timestamp','id'];
var posts=[];
for(var key in keys){
var post=client.sort("posts",'by','nosort',"get","POST:*->"+keys[key],_);
posts.push(post);
}
}
This is because require is synchronous. So you cannot put asynchronous code at the top level of a script which is required by another script.