I'm a front-end dev just venturing into the Node.js, particularly in using it to create small command line tools.
My question: how do you persist data with command line tools? For example, if I want to keep track of the size of certain files over time, I'd need to keep a running record of changes (additions and deletions) to those files, and relevant date/time information.
On the web, you store that sort of data in a database on a server, and then query the database when you need it again. But how do you do it when you're creating a Node module that's meant to be used as a command line tool?
Some generic direction is all I'm after. I don't even know what to Google at this point.
It really depends on what you're doing, but a simple approach is to just save the data that you want to persist to a file and, since we're talking node, store it in JSON format.
Let's say you have some data like:
var data = [ { file: 'foo.bar', size: 1234, date: '2014-07-31 00:00:00.000'}, ...]
(it actually doesn't matter what it is, as long as it can be JSON.stringifiy()d)
You can just save it with:
fs.writeFile(filename, JSON.stringify(data), {encoding: 'utf8'}, function(err) { ... });
And load it again with:
fs.readFile(filename, {encoding: 'utf8'}, function(err, contents) {
data = JSON.parse(contents);
});
You'll probably want to give the user the ability to specify the name of the file you're going to persist the data to via an argument like:
node myscript.js <data_file>
You can get that passed in parameter with process.argv:
var filename = process.argv[2]; // Be sure to check process.argv.length and have a default
Using something like minimist can be really helpful if you want to get more complex like:
node myscript.js --output <data_file>
You also can store files in temporary directory, for example /tmp directory on linux and give user the option to change the directory.
To get path to temporary directory you can use os module in nodejs:
const os = require('os');
const tmp = os.tmpdir();
Related
Is there a way where I can save a timestamp out of my application / object, so when I restart the nodeserver I can get that value?
I need this for my cronjob. I need to save the last synching even though I restart the server.
There are all sorts of ways to save this sort of information so you can load it when you restart your node process. One is to write it to a file in your file system, then read it when you start your program.
To write the current timestamp to a file do this.
const fs = require('fs')
...
fs.writeFile('timestamp.txt', Date.now().toString(), err => {console.error(err)})
To read it do this.
const fs = require('fs')
...
const timestamp = Number(fs.readFileSync('timestamp.txt'))
Obviously there's more programming to do to put the file in the correct directory, to handle errors, and to cope with the case where you attempt to read the file before writing it. But that's the idea.
You can also store it in some kind of database. But this should do you for now. Unless you're using a system like Heroku where the files don't always get saved from run to run.
When a process dies, all data stored in its working memory (such as variables and functions) die with it.
I recently wrote an npm package cashola that makes it easier to store this data across process restarts.
You can run this example script twice and see how the print statements differ each time.
import { rememberSync } from 'cashola';
const myState = rememberSync('timestamp-example');
console.log('Before:', myState);
// First run: {}
// Second run: { <timeString1>: 'hi! }
myState[new Date.getTime().toString()] = 'hi!';
console.log('After:', myState);
// First run: { <timeString1>: 'hi! }
// Second run: { <timeString1>: 'hi!, <timeString2>: 'hi! }
I am trying to create a zip file in node using the code provided from how to create a zip file in node given multiple downloadable links, as shown below:
var fs = require('fs');
var archiver = require('archiver');
var output = fs.createWriteStream('./example.zip');
var archive = archiver('zip', {
gzip: true,
zlib: { level: 9 } // Sets the compression level.
});
archive.on('error', function(err) {
throw err;
});
// pipe archive data to the output file
archive.pipe(output);
// append files
archive.file('/path/to/file0.txt', {name: 'file0-or-change-this-whatever.txt'});
archive.file('/path/to/README.md', {name: 'foobar.md'});
//
archive.finalize();
When I use this suggestion, the zip file is downloaded without any kind of prompt asking me where I would like to save the file - is there any way I can make it so that a prompt is created asking me where I would like to save the file, which is quite normal these days?
If this is absolutely not possible, would it be possible to always save the file in the downloads folder (regardless of whether on mac or windows or any other operating system)?
So there's a couple of things here. In terms of a 'prompt' or 'pop-up' you won't find anything along the lines of WinForms out of the box, there are options for the command line such as prompts You can use that as your user input.
https://www.npmjs.com/package/prompts
You'll want to use path and more specifically path.join() to combat the mac/windows/linux issue.
Do you need to use path.join in node.js?
You can run an express server and create a route that uses res.download() in which you would provide the zipped file.
https://expressjs.com/en/api.html#res.download
Hey actually i am doing some project in Nodejs. I need a configuration file in order to store the data to file system but i do not know how to write a configuration file to store data to file. please help me with this. thanks in advance
Sounds to me that you are looking for the following NPM module/library - dotenv. You simply require('dotenv').config(); which is probably best placed at the top (after use strict;) and create a text file which would read as an example:
url_prefix='mongodb://'
url_ip='#localhost'
port=':27017/'
dbase='NameofDB'
Of course you can add anything you like to this file. Just remember it is a text file and should not contain spaces etc.
Though the default for the .env file is in the root of your project you can actually place it wherever you like, and simply put:
require('dotenv').config({path: '/custom/path/to/your/env/vars'});
(Above was taken from the dotenv documentation and it works as I use it in projects.)
To acquire any Global variable you would simply type:
process.env.url_prefix
Obviously from there you can build the needed entry code to your DB from process.env statements such as:
process.env.url_prefix+process.env.url_ip etc. OR
${process.env.url_prefix}${process.env.url_ip}
Using dotenv allows you to keep sane control over those process.env globals.
Be aware there is a gotcha! Be careful not to overwrite any of those globals in your code. As they will remain overwritten as long as your Node process is running.
If you mean you need some constants and business logic/data file to read from, you can simply include the file in your script using the require module.
Ex: Your file name is test.json, then:
var test = require('test.json');
Further, you can use a CONSTANT in the file as 'test.CONSTANT'
Note: Please make sure you use module.exports wherever needed. Details are here
Usually people use JSON to store configurations and stuff, since it is very javascripty.. You can simply make a JSON config file. In case you need to store some special data like SECRET URL, just use environment variables. FYI I found your question unclear. Does this answer your question.
const fs = require("fs");
// Example Config
let config = {
DB: "mongodb://blahblah:idhdiw#jsjsdi",
secret: "thisandthat",
someScript: "blah.js"
};
// Write to file.
fs.writeFile('config.cfg', JSON.stringify(config), err => {
if (err) throw err;
console.log("[+] Config file saved!");
// Retrieve
let confData = JSON.parse(fs.readFileSync('config.cfg'));
console.log(confData.secret);
});
// To save variables in environment variables
// Generally You will not set environment variables like this
// You will have access to setting environment variables incase
// you are using heroku or AWS from dash board. Incase of a machine
// you can use ** export SOME_ENV_VAR="value" ** in your bash profile
process.env.IP = "10.10.10.10";
// Too risky to put else where.
process.env.API_KEY = "2ke9u82hde82h8";
// Get Data
console.log(process.env.IP);
console.log(process.env.API_KEY);
I have a UDP client that grabs some data from another source and writes it to a file on the server. Since this is large amount of data, I dont want the end user to wait until they its full written to the server so that they can download it. So I made a NodeJS server that grabs the latest data from the file and sends it to the user.
Here is the code:
var stream = fs.readFileSync(filename)
.on("data", function(data) {
response.write(data)
});
The problem here is, if the download starts when the file was only for example 10mb.. the fs.readFileSync will only read my file up to 10mb. Even if 2 mins later the file increased to 100mb. fs.readFileSync will never know about the new updated data. How can I do this in Node? I would like somehow refresh the fs state or maybe perpaps wait for new data using fs file system. Or is there some kind of fs fileContent watcher?
EDIT:
I think the code below describes better what I would like to achieve, however in this code it keeps reading forever and I dont have any variable from fs.read that can help me stop it:
fs.open(filename, 'r', function(err, fd) {
var bufferSize=1000,
chunkSize=512,
buffer=new Buffer(bufferSize),
bytesRead = 0;
while(true){ //check if file has new content inside
fs.read(fd, buffer, 0, chunkSize, bytesRead);
bytesRead+= buffer.length;
}
});
Node has built-in methods in the fs module. It is tagged as unstable, so it can change in the future.
Its called: fs.watchFile(filename[, options], listener)
You can read more about it here: https://nodejs.org/api/fs.html#fs_fs_watchfile_filename_options_listener
But i highly suggest you to use one of the good modules mantained actively like
watchr:
From his readme:
Better file system watching for Node.js. Provides a normalised API the
file watching APIs of different node versions, nested/recursive file
and directory watching, and accurate detailed events for
file/directory changes, deletions and creations.
The module page is here: https://github.com/bevry/watchr
(Used the module in a couple of proyects and working great, im not related to it in other way)
you need store in some data base last size of file.
read filesize first.
load your file.
then make a script to check if file was change.
you can consult the size with jquery.post to obtain your result and decide if need to reload in javascript
I like how console.log(object) will output object's structure in json format. How do I make my app output the same stuff to a file?
As Golo said there is nothing built-in in Node for that but you can easily write your own (or use Winston) :)
fs = require('fs');
logToFile = function(fileName, objectToLog) {
jsonText = JSON.stringify(objectToLog, null, '\t');
fs.writeFileSync(fileName, jsonText, 'utf8');
}
sampleData = { name: 'Batman', city: 'Gotham' };
logToFile('log.txt', sampleData);
There is not out of the box support for file logging in Node.js.
Basically, you have two options:
You can redirect any output of the Node.js process to a file by using the mechanisms of your operating system to redirect streams.
Use a dedicated logging library, such as Winston.
I'd go with the second option as it's the more flexible one and you'll need it sooner or later, at least if your project gets slightly bigger.