nodejs: nedb filename path causing problems; how to fix? - node.js

I'm an experienced programmer, but relatively new to nodejs, nedb, Mocha.
nedb is a nodejs npm module for small data usages, that replicates some of the interface used in Mongodb. I chose it because I'm building an app to run on a Beaglebone Black or Raspberry Pi and didn't want the overhead for Mongodb for my small data needs.
However, I'm having a problem with the path portion of the filename for nedb. It works fine if the myapp.js I'm working on is invoked local to the current path, but if I invoke it from a different directory, like my Mocha test directory, then nedb won't work with the same path. It won't work with relative path either; I have to specify the full path, starting with /home before I can get it to work from the Mocha test directory. Needless to say, this is both inconvenient, and prohibitave for production, as it's not running with a base of /home in production.
In the code that follows, myapp.js is the application I'm developing and testing. useMyapp.js is a utility test to use myapp.js for simplicity. These are all in the same directory, and work properly when I invoke useMyapp.js.
myapp.js is located in /home/geek/project/public/javascripts/
Here is the code for myapp.js:
var Datastore = require("nedb");
var db = new Datastore({filename: "./calEvents.db", autoload: true});
function Myapp(year, month) {
console.log('myapp: year: ' + year + ', month: ' + month);
this.year = year;
this.month = month;
}
Myapp.prototype.getAllData = function() {
db.find({}, function(err, docs) {
if(err) {
console.error('error: ', err);
} else {
console.log(" docs found: ", docs);
}
});
};
module.exports = Myapp;
useMyapp.js is located in the same directory at /home/geek/project/public/javascripts/
Here is the code for useMyapp.js, which works correctly, outputing the expected nedb records.
var Myapp = require("./myapp");
var myapp = new Myapp(2016, "Mar");
myapp.getAllData();
I copied useMyapp.js to my Mocha test directory, renamed it to be TestUseMyapp.js, into /home/geek/project/test/
I adjusted the require statment for relative path to myapp.js
Here's the contents of TestUseMyapp.js in /home/geek/project/test/
var Myapp = require("../public/javascripts/myapp");
var myapp = new Myapp(2016, "Mar");
myapp.getAllData();
The output from running this via nodejs is an empty data set:
docs found: []
THE PROBLEM:
TestUseMyapp.js is able to find and load myapp.js per the 'require' statement, but the filename parameter for nedb is not working when myapp.js is invoked from a different directory. This is causing me problems between development, Mocha testing, and running in production, as the file path are all different.
I created a duplicate of myapp.js, named it myapp2.js, and kept it in the same directory as myapp.js at /home/geek/project/public/javascripts. In this, I changed the path for the nedb filename parameter.
Here's the code in myapp2.js PLEASE NOTE THE ONLY CHANGE IS TO THE filename PARAMATER PATH:
var Datastore = require("nedb");
var db = new Datastore({filename: "/home/geek/project/public/javascripts/calEvents.db", autoload: true});
function Myapp(year, month) {
console.log('myapp: year: ' + year + ', month: ' + month);
this.year = year;
this.month = month;
}
Myapp.prototype.getAllData = function() {
db.find({}, function(err, docs) {
if(err) {
console.error('error: ', err);
} else {
console.log(" docs found: ", docs);
}
});
};
module.exports = Myapp;
This works as expected when I invoked the test against myapp2.js with the full path.
THE QUESTION:
How do I resolve the filename path issue with nedb so that I don't have to change it between development, Mocha testing, and deployment to production?
Thanks!

I was able to solve this problem by inserting __dirname as part of the filename parameter.

Related

reading a packaged file in aws lambda package

I have a very simple node lambda function which reads the contents of packaged file in it. I upload the code as zip file. The directory structure is as follows.
index.js
readme.txt
Then have in my index.js file:
fs.readFile('/var/task/readme.txt', function (err, data) {
if (err) throw err;
});
I keep getting the following error NOENT: no such file or directory, open '/var/task/readme.txt'.
I tried ./readme.txt also.
What am I missing ?
Try this, it works for me:
'use strict'
let fs = require("fs");
let path = require("path");
exports.handler = (event, context, callback) => {
// To debug your problem
console.log(path.resolve("./readme.txt"));
// Solution is to use absolute path using `__dirname`
fs.readFile(__dirname +'/readme.txt', function (err, data) {
if (err) throw err;
});
};
to debug why your code is not working, add below link in your handler
console.log(path.resolve("./readme.txt"));
On AWS Lambda node process might be running from some other folder and it looks for readme.txt file from that folder as you have provided relative path, solution is to use absolute path.
What worked for me was the comment by Vadorrequest to use process.env.LAMBDA_TASK_ROOT. I wrote a function to get a template file in a /templates directory when I'm running it locally on my machine with __dirname or with the process.env.LAMBDA_TASK_ROOT variable when running on Lambda:
function loadTemplateFile(templateName) {
const fileName = `./templates/${templateName}`
let resolved
if (process.env.LAMBDA_TASK_ROOT) {
resolved = path.resolve(process.env.LAMBDA_TASK_ROOT, fileName)
} else {
resolved = path.resolve(__dirname, fileName)
}
console.log(`Loading template at: ${resolved}`)
try {
const data = fs.readFileSync(resolved, 'utf8')
return data
} catch (error) {
const message = `Could not load template at: ${resolved}, error: ${JSON.stringify(error, null, 2)}`
console.error(message)
throw new Error(message)
}
}
This is an oldish question but comes up first when attempting to sort out whats going on with file paths on Lambda.
Additional Steps for Serverless Framework
For anyone using Serverless framework to deploy (which probably uses webpack to build) you will also need to add the following to your webpack config file (just after target: node):
// assume target: 'node', is here
node: {
__dirname: false,
},
Without this piece using __dirname with Serverless will STILL not get you the desired absolute directory path.
I went through this using serverless framework and it really was the file that was not sent in the compression. Just add the following line in serverless.yml:
package:
individually: false
include:
- src/**
const filepath = path.resolve('../../filename.text');
const fileData2 = fs.readFileSync(process.env.LAMBDA_TASK_ROOT + filepath, 'utf-8');
I was using fs.promises.readFile(). Couldn't get it to error out at out. The file was there, and LAMBDA_TASK_ROOT seemed right to me as well. After I changed to fs.readFileSync(), it worked.
I hade the same problem and I tried applying all these wonderful solutions above - which didn't work.
The problem was that I setup one of the folder name with one letter in upper case which was really lowercase.
So when I tried to fetch the content of /src/SOmething/some_file.txt
While the folder was really /src/Something/ - I got this error...
Windows (local environment) is case insensitive while AWS is not!!!....

How to create a directory in the current directory in Node.js

I am new to Node.js, so I'm not familiar with a lot of stuff.
So basically I want to create a directory in the current working directory:
var mkdirp = require('mkdirp');
console.log("Going to create directory /tmp/test");
mkdirp('/tmp/test',function(err){
if (err) {
return console.error(err);
}
console.log("Directory created successfully!");
});
My current directory is C:\Users\Owner\Desktop\Tutorials\NodeJS on Windows, which means I run node main.js in that directory.
(main.js is in C:\Users\Owner\Desktop\Tutorials\NodeJS)
After I run the code, it generates C:\tmp\test, which is in C:\.
But I want to create it in the current directory, so the result I want is C:\Users\Owner\Desktop\Tutorials\NodeJS\tmp\test.
I just don't know how to do that...
You can use process.cwd() to output the directory where your command has been executed (in your case, the directory where you run node main.js) so your code might look like this:
var mkdirp = require('mkdirp');
var path = require('path');
console.log("Going to create directory /tmp/test");
mkdirp(path.join(process.cwd(), '/tmp/test'), function(err){
if (err) {
return console.error(err);
}
console.log("Directory created successfully!");
});
If you need just the directory where the main.js file is located and not where you execute it (by calling node main.js), you can use the __dirname variable instead of process.cwd().
It's a good idea to use the path.join() function to make sure the path delimiters are set correctly, especially when you're on a Windows system which may treat forward slashes as options.
var mkdirp = require('mkdirp');
var path = require('path');
console.log("Going to create directory /tmp/test");
mkdirp(path.join(__dirname, '/tmp/test'),function(err){
if (err) {
return console.error(err);
}
console.log("Directory created successfully!");
});
You could use path.join(__dirname, '/tmp/test') where __dirname would return The name of the directory that the currently executing script resides in.
You need to include module 'path' to make path.join() work.
Reference
__dirname

pre-load / pre-require directories of .js route files

Using Express with Node.js, we might do something like this:
app.use('api/:controller/:action/:id', function(req,res,next){
var controller = req.params.controller;
var action = req.params.action;
var route = require('./routes/' + controller + '/' + action);
route(req,res,next);
}
now this is all fine and well, except there is at least one problem: the route file is dynamically loaded at runtime if this file has not been 'require'd yet. Which means it's a little bit slower at least.
Does someone have a script that recurses through a directory and pre-loads/pre-requires all the .js files when a server first starts up?
I have a similar problem for the front-end as well, using RequireJS. The solution seems to be to write a bash script that writes out all the .js filepaths in a directory and its subdirectories to a text file. then when the server starts up, it reads that text file and requires all the files in the directory that are listed in the text file. Is that the best way to do it?
If you can use io.js, it can preload modules using command-line -r or --require:
iojs -r <module_name> server.js
I created an NPM module that does this for the front-end, doing it for Node.js / CommonJS is another story.
https://www.npmjs.com/package/requirejs-metagen
you can use it like so:
var grm = require('requirejs-metagen'); //you can use with Gulp
var controllersOpts = {
inputFolder: './public/static/app/js/controllers/all',
appendThisToDependencies: 'app/js/controllers/',
appendThisToReturnedItems: '',
eliminateSharedFolder: true,
output: './public/static/app/js/meta/allControllers.js'
};
grm(controllersOpts,function(err){
//handle errors your own way
});
it generates a corresponding AMD/RequireJS module like so:
define(
[
"app/js/controllers/all/jobs",
"app/js/controllers/all/users"
],
function(){
return {
"jobs": arguments[0],
"users": arguments[1]
}
});
you can also require subdirectories and all that stuff like so:
var allViewsOpts = {
inputFolder: './public/static/app/js/jsx',
appendThisToDependencies: 'app/js/',
appendThisToReturnedItems: '',
eliminateSharedFolder: true,
output: './public/static/app/js/meta/allViews.js'
}
grm(allViewsOpts );
which generates output like so:
define([
"app/js/jsx/BaseView",
"app/js/jsx/reactComponents/FluxCart",
"app/js/jsx/reactComponents/FluxCartApp",
"app/js/jsx/reactComponents/FluxProduct",
"app/js/jsx/reactComponents/Item",
"app/js/jsx/reactComponents/Job",
"app/js/jsx/reactComponents/JobsList",
"app/js/jsx/reactComponents/listView",
"app/js/jsx/reactComponents/Picture",
"app/js/jsx/reactComponents/PictureList",
"app/js/jsx/reactComponents/RealTimeSearchView",
"app/js/jsx/reactComponents/Service",
"app/js/jsx/reactComponents/ServiceChooser",
"app/js/jsx/reactComponents/todoList",
"app/js/jsx/relViews/getAll/getAll",
"app/js/jsx/relViews/jobs/jobsView",
"app/js/jsx/standardViews/dashboardView",
"app/js/jsx/standardViews/overviewView",
"app/js/jsx/standardViews/pictureView",
"app/js/jsx/standardViews/portalView",
"app/js/jsx/standardViews/registeredUsersView",
"app/js/jsx/standardViews/userProfileView"
],
function(){
return {
"BaseView": arguments[0],
"reactComponents/FluxCart": arguments[1],
"reactComponents/FluxCartApp": arguments[2],
"reactComponents/FluxProduct": arguments[3],
"reactComponents/Item": arguments[4],
"reactComponents/Job": arguments[5],
"reactComponents/JobsList": arguments[6],
"reactComponents/listView": arguments[7],
"reactComponents/Picture": arguments[9],
"reactComponents/PictureList": arguments[10],
"reactComponents/RealTimeSearchView": arguments[11],
"reactComponents/Service": arguments[12],
"reactComponents/ServiceChooser": arguments[13],
"relViews/getAll/getAll": arguments[14],
"relViews/jobs/jobsView": arguments[15],
"standardViews/dashboardView": arguments[16],
"standardViews/overviewView": arguments[17],
"standardViews/pictureView": arguments[18],
"standardViews/portalView": arguments[19],
"standardViews/registeredUsersView": arguments[20],
"standardViews/userProfileView": arguments[21]
}
});
I need to update the library so it returns the stream so you can handle when it completes, otherwise it works great.

How ensure default data in NeDB?

I'm trying to use NeDB as storage for my data in node-webkit application. I have the single collection named config.db:
var Datastore = require('nedb')
, path = require('path')
, db = new Datastore({ filename: path.join(require('nw.gui').App.dataPath, 'config.db') });
When user opens node-webkit application first time my config.db should have default data like:
{
color: "red",
font: 'bold'
...
}
Does NeDB have option for providing default data if there are no yet? Or What it the best way to save it if config.db is empty (in case if user opens node-webkit application first time)?
As far as I know NeDB does not have an option to create initial data.
I think the easiest way to achieve this is to simply query whether there is data. If counting documents returns 0, obviously the initial data have not yet been saved, so you should do this now.
If you include this check in the startup code of your application, it will automatically initialize the data on first run, and afterwards simply do nothing.
I came across this question while looking for a similar solution. I thought I'd share what I ended up with (this is a module):
var fs = require("fs");
module.exports = function (app) {
var customizationService = app.service("customization");
fs.readFile("./db/customization", "utf8", function (err, data) {
if (err) {
return console.log(err);
}
if (data) {
// Sweet, carry on
} else {
var customOptions = {
SiteTitle: "VendoMarket",
SiteTagline: "The freshest eCommerce platform around"
};
// Save data to the locations service
customizationService.create(customOptions);
}
});
};
And then in my app.js file:
//--------------------------------------
// Initialize
//--------------------------------------
var vendoInit = require("./src/init");
vendoInit(app);
(My app.js file is at the base of my project, src is a folder next to it)

Meteor/Node writeFile crashes server

I have the following code:
Meteor.methods({
saveFile: function(blob, name, path, encoding) {
var path = cleanPath(path), fs = __meteor_bootstrap__.require('fs'),
name = cleanName(name || 'file'), encoding = encoding || 'binary',
chroot = Meteor.chroot || 'public';
// Clean up the path. Remove any initial and final '/' -we prefix them-,
// any sort of attempt to go to the parent directory '..' and any empty directories in
// between '/////' - which may happen after removing '..'
path = chroot + (path ? '/' + path + '/' : '/');
// TODO Add file existance checks, etc...
fs.writeFile(path + name, blob, encoding, function(err) {
if (err) {
throw (new Meteor.Error(500, 'Failed to save file.', err));
} else {
console.log('The file ' + name + ' (' + encoding + ') was saved to ' + path);
}
});
function cleanPath(str) {
if (str) {
return str.replace(/\.\./g,'').replace(/\/+/g,'').
replace(/^\/+/,'').replace(/\/+$/,'');
}
}
function cleanName(str) {
return str.replace(/\.\./g,'').replace(/\//g,'');
}
}
});
Which I took from this project
https://gist.github.com/dariocravero/3922137
The code works fine, and it saves the file, however it repeats the call several time and each time it causes meteor to reset using windows version 0.5.4. The F12 console ends up looking like this: . The meteor console loops over the startup code each time the 503 happens and repeats the console logs in the saveFile function.
Furthermore in the target directory the image thumbnail keeps displaying and then display as broken, then a valid thumbnail again, as if the fs is writing it multiple times.
Here is the code that calls the function:
"click .savePhoto":function(e, template){
e.preventDefault();
var MAX_WIDTH = 400;
var MAX_HEIGHT = 300;
var id = e.srcElement.id;
var item = Session.get("employeeItem");
var file = template.find('input[name='+id+']').files[0];
// $(template).append("Loading...");
var dataURL = '/.bgimages/'+file.name;
Meteor.saveFile(file, file.name, "/.bgimages/", function(){
if(id=="goodPhoto"){
EmployeeCollection.update(item._id, { $set: { good_photo: dataURL }});
}else{
EmployeeCollection.update(item._id, { $set: { bad_photo: dataURL }});
}
// Update an image on the page with the data
$(template.find('img.'+id)).delay(1000).attr('src', dataURL);
});
},
What's causing the server to reset?
My guess would be that since Meteor has a built-in "automatic directories scanning in search for file changes", in order to implement auto relaunching of the application to newest code-base, the file you are creating is actually causing the server reset.
Meteor doesn't scan directories beginning with a dot (so called "hidden" directories) such as .git for example, so you could use this behaviour to your advantage by setting the path of your files to a .directory of your own.
You should also consider using writeFileSync insofar as Meteor methods are intended to run synchronously (inside node fibers) contrary to the usual node way of asynchronous calls, in this code it's no big deal but for example you couldn't use any Meteor mechanics inside the writeFile callback.
asynchronousCall(function(error,result){
if(error){
// handle error
}
else{
// do something with result
Collection.update(id,result);// error ! Meteor code must run inside fiber
}
});
var result=synchronousCall();
Collection.update(id,result);// good to go !
Of course there is a way to turn any asynchronous call inside a synchronous one using fibers/future, but that's beyond the point of this question : I recommend reading this EventedMind episode on node future to understand this specific area.

Resources