So, I'm going through the Node.js learnyounode tutorial (which is pretty fun), but I'm no stranger to javascript. One of the requests is to have the user write a program that accepts a directory and a file extension as arguments, and then proceeds to ls the files which match the given extension.
In checking my arguments in various places where data is passed through functions, I cannot get Node.js to allow me to pass additional arguments to a callback function for a built-in function.
For example, I can run the following snippet, pass an err, path, and extension, and would like to see the all the passed arguments, but I only see err and data in the arguments object while the "ext" variable is ignored.
var fs = require("fs"), path = require("path")
var dir = process.argv[2], ext = process.argv[3];
fs.readdir(dir, function (err, data, ext) {
console.log(arguments);
});
The reason for wanting this functionality is to avoid breaking the function scope to retrieve the value of ext.
I can do this type of thing with custom functions all day, but the built-in functions are a little less forgiving.
Thanks,
Bob
The easiest way is to just use the built-in path.extname() function on each entry name:
fs.readdir(dir, function(err, entries) {
if (err) throw err;
for (var i = 0; i < entries.length; ++i) {
if (path.extname(entries[i]) === ext)
console.log(entries[i]);
}
});
Talked to the node.js people in irc, turns out the way javascript works such a method to keep scope in bounds is not necessary. Node.js, by default, puts modules in their own closures, so the variables can be accessed outside the function and not interfere with global namespace.
Found this answer on node js and variable scope which probably best matches my situation, but didn't know it's what I needed when I posted my question.
Also found some nice closure examples on Mozilla.
On a personal note I thought I knew javascript a little better than this, and I hope this answer helps people who are coming from C / PHP backgrounds into node.js
Related
I was wondering what's the best practice and if I should create:
a directory in which declare statically all the errors my application uses, like api/errors/custom1Error
declare them directly inside the files
or put the files directly inside the dir that needs that error, like api/controller/error/formInvalidError
other options!?
A neat way of going about this would be to simply add the errors as custom responses under api/responses. This way even the invocation becomes pretty neat. Although the doc says you should add them directly in the responses directory, I'm sure there must be a way to nest them under, say, responses/errors. I'll try that out and post an update in a bit.
Alright, off a quick search, I couldn't find any way to nest the responses, but you can use a small workaround that's not quite as neat:
Create the responses/errors directory with all the custom error response handlers. Create a custom response and name it something like custom.js. Then specify the response name while calling res.custom().
I'm adding a short snippet just for illustration:
api/responses/custom.js:
var customErrors = {
customError1: require('./errors/customError1'),
customError2: require('./errors/customError2')
};
module.exports = function custom (errorName, data) {
var req = this.req;
var res = this.res;
if (customErrors[errorName]) return customErrors[errorName](req, res, data);
else return res.negotiate();
}
From the controller:
res.custom('authError', data);
If you don't need logical processing for different errors, you can do away with the whole errors/ directory and directly invoke the respective views from custom.js:
module.exports = function custom (viewName, data) {
var req = this.req;
var res = this.res;
return res.view('errors/' + viewName, data);//assuming you have error views in views/errors
}
(You should first check if the view exists. Find out how on the linked page.)
Although I'm using something like this for certain purposes (dividing routes and so on), there definitely should be a way to include response handlers defined in different directories. (Perhaps by reconfiguring some grunt task?) I'll try to find that out and update if I find any success.
Good luck!
Update
Okay, so I found that the responses hook adds all files to res without checking if they are directories. So adding a directory under responses results in a TypeError from lodash. I may be reading this wrong but I guess it's reasonable to conclude that currently it's not possible to add a directory there, so I guess you'll have to stick to one of the above solutions.
I installed sqlite3 with npm
npm install sqlite3 --save
I have written some basic functions that I would like to be performed synchronously rather then with async.
for instance I would like to get column names in one function and the row count in another.
I would like to simply return these values. currently I am using a callback like so
d.cinfo = function(table, callback){
var o = {};
db.each("PRAGMA table_info(" + table + ")", function(err, col){
o[col.name] = col.type;
}, function(){
if(typeof callback == 'function') callback(o);
});
}
d is an object which is later exposed
however Id like to return the values
d.cinfo = function(table, callback){
var o = {};
db.each("PRAGMA table_info(" + table + ")", function(err, col){
o[col.name] = col.type;
}, function(){
return o;
});
}
is there a way I can achieve this. I found documentation saying it was possible but then I learned that it was outdated and im not sure its meant for the same api
I have implemented bluebird however Promise.promisifyAll(middleware) returns and error "Object # has no method .then() anyone know what im doing wrong
I've come across a problem like this and many others while using Express and it really drives me crazy where I just can't avoid callback hells.
Some of the possible solutions can be:
Using the same old event driven style. Instead of calling nested functions, emit events for them. But I'm assuming you wouldn't like that very much, but that atleast gets rid of the nested callbacks.
There's a npm module called bluebird that allows you to extend existing modules with promises. (Using promisifyAll). So promises will still trim down the code and you'll have much cleaner code than you had before using promises.
If you don't mind switching frameworks, you can try KoaJS that allows you to yield(fun stuff from ECS 6) functions using generator functions(another cool feature). You can have more reading about this here.
So the above code can be re-written like:
var rows = yield db.each() //or something like that
this is an amazing feature but the drawback is that you have to use unstable versions of node (0.11.x). We also faced some issues setting up https using 0.11.x and had to downgrade to 0.10.x (we're not using koa either)
If there are only a few places where you need to do such kind of stuff, I would recommend using bluebird, but if you're tired of having lots of callbacks in your code, better go for Koa, although I've already mentioned you the tradeoffs.
I have a nodejs application and I want to write data into hadoop HDFS file system. I have seen two main nodejs libraries that can do it: node-hdfs and node-webhdfs. Someone have tried it? Any hints? Which one should I use in production?
I am inclined to use node-webhdfs since it uses WebHDFS REST API. node-hdfs seem to be a c++ binding.
Any help will be greatly appreciated.
You may want to check out webhdfs library. It provides nice and straightforward (similar to fs module API) interface for WebHDFS REST API calls.
Writing to the remote file:
var WebHDFS = require('webhdfs');
var hdfs = WebHDFS.createClient();
var localFileStream = fs.createReadStream('/path/to/local/file');
var remoteFileStream = hdfs.createWriteStream('/path/to/remote/file');
localFileStream.pipe(remoteFileStream);
remoteFileStream.on('error', function onError (err) {
// Do something with the error
});
remoteFileStream.on('finish', function onFinish () {
// Upload is done
});
Reading from the remote file:
var WebHDFS = require('webhdfs');
var hdfs = WebHDFS.createClient();
var remoteFileStream = hdfs.createReadStream('/path/to/remote/file');
remoteFileStream.on('error', function onError (err) {
// Do something with the error
});
remoteFileStream.on('data', function onChunk (chunk) {
// Do something with the data chunk
});
remoteFileStream.on('finish', function onFinish () {
// Upload is done
});
Not good news!!!
Do not use node-hdfs. Although it seems promising, it is now two years obsolete. I've tried to compile it but it does not match the symbols of current libhdfs. If you want to use something like that you'll have to make your own nodejs binding.
You can use node-webhdfs but IMHO there's not much advantage on that. It is better to use an http nodejs lib to make your own requests. The hardest part here is try to hold the very async nature of nodejs, since you might want first to create a folder, and then after successfully create it, create a file and then, at last, write or append data. Everything through http requests that you must send and wait the for answer to then go on....
At least node-webhdfs might be a good reference to you take a look and start your own code.
Br,
Fabio Moreira
im having problems understanding how to talk 'up and down' between app.js and modules...
I think its with a callback but i've also seen things like self._send(), this.send() and module.exports.emit
I'm quite confused.
I recently installed pdfkit from npm (quite good 6/10 :p) I want to learn by improving it slightly though by adding a done event/callback for doc.write().
I know its not that important but i've been looking through my installed modules and that is probably the easiest example of code that wouldn't hurt to have a 'DONE' I also figured this function would be good to learn from as it uses fs.writeFile which has a function(){} that fires when its finished writing so the fact that i can see where in the code it ends makes it an easy learning tool.
I've modified the code a few times tried to compare modules to see where similar things have been done but i just keep breaking it with errors, i don't feel like i'm getting anywhere:
inside the pdfkit module document.js i've made changes:
var EventEmitter = require('events').EventEmitter;//ben
module.exports = new EventEmitter();//ben
PDFDocument.prototype.write = function(filename, fn, callback) {//ben added callback
return this.output(function(out) {
return fs.writeFile(filename, out, 'binary', fn, function(){//ben added finished function
//module.exports.emit('pdf:saved');//ben
callback();//ben
});
});
};
in my app.js:
doc.write('public_html/img/'+_.c+'_'+_.propertyid+'.pdf',function(){console.log('pdf:saved');});
//doc.on('pdf:saved',function(){console.log('pdf:saved');});
I'm also not really sure what i'm querying on google, please can someone help me?
EventEmitter is required to create objects with event storage, and emit/catch events.
While what you are using in your example is called 'callback'.
It is two different ways, and can be sort of cross used. Sometimes it is good to use events, but sometimes just callback is enough.
The best way to have head around it: play with callbacks (forget about events for now). Try to think of different uses of callbacks and maybe even have callback function and pass it around. Then come to callback chains. And only after start playing with EventEmitter. Remember that EventEmitter is different thing from callbacks, with sometimes compatible use cases, but generally is used in different cases.
Here is your code, simplified with same functionality as you have/need atm:
PDFDocument.prototype.write = function(filename, callback) {
this.output(function(out) {
fs.writeFile(filename, out, 'binary', callback);
});
};
And use it the way you already do.
Do not try to generate garbage of code that will only complicate everything - it is better to study speficic areas seperatelly, and then switch to next one. Otherwise will mess up in mind.
FIXED
fn is the callback function!
PDFDocument.prototype.write = function(filename, fn) {
return this.output(function(out) {
return fs.writeFile(filename, out, 'binary', fn);
});
};
and by naming my callback function to fn it works!
doc.write('public_html/img/'+_.c+'_'+_.propertyid+'.pdf',function fn(){console.log('pdf:saved');});
for me that is a massive learning mountain climbed!!
I managed to create a module to handle all the database call. It uses this lib: https://github.com/developmentseed/node-sqlite3
My issues are the following.
Everytime I make a call, I need to make sure the database exist, and if not to create it.
Plus, as all the calls are asynchronous, I end up having loads of functions in functions in callbacks ... etc.
It pretty much looks like this:
getUsers : function (callback){
var _aUsers = [];
var that = this;
this._setupDb(function(){
var db = that.db;
db.all("SELECT * FROM t_client", function(err, rows) {
rows.forEach(function (row) {
_aUsers.push({"cli_id":row.id,"cli_name":row.cli_name,"cli_path":row.cli_path});
});
callback(_aUsers);
});
});
},
So, is there any way I can export my module only when the database is ready and fully created if it does not exist yet?
Does anyone see a way around the "asynchronous" issue?
You could also try using promises or fibers ...
I don't think so. If you make it synchronous, you are taking away the advantage. Javascript functions are meant to be that way. Such a situation is referred to as callback hell. If you are facing problems managing callbacks then you can use these libraries :
underscore
async
See these guides to understand basics of asynchronous programming
Node.js: Style and structure
Using underscore.js managing-callback-spaghetti-in-nodejs
Using async.js node-js-async-programming