Is there a way where I can save a timestamp out of my application / object, so when I restart the nodeserver I can get that value?
I need this for my cronjob. I need to save the last synching even though I restart the server.
There are all sorts of ways to save this sort of information so you can load it when you restart your node process. One is to write it to a file in your file system, then read it when you start your program.
To write the current timestamp to a file do this.
const fs = require('fs')
...
fs.writeFile('timestamp.txt', Date.now().toString(), err => {console.error(err)})
To read it do this.
const fs = require('fs')
...
const timestamp = Number(fs.readFileSync('timestamp.txt'))
Obviously there's more programming to do to put the file in the correct directory, to handle errors, and to cope with the case where you attempt to read the file before writing it. But that's the idea.
You can also store it in some kind of database. But this should do you for now. Unless you're using a system like Heroku where the files don't always get saved from run to run.
When a process dies, all data stored in its working memory (such as variables and functions) die with it.
I recently wrote an npm package cashola that makes it easier to store this data across process restarts.
You can run this example script twice and see how the print statements differ each time.
import { rememberSync } from 'cashola';
const myState = rememberSync('timestamp-example');
console.log('Before:', myState);
// First run: {}
// Second run: { <timeString1>: 'hi! }
myState[new Date.getTime().toString()] = 'hi!';
console.log('After:', myState);
// First run: { <timeString1>: 'hi! }
// Second run: { <timeString1>: 'hi!, <timeString2>: 'hi! }
Related
I have a JavaScript object with many different properties, and it might look something like this:
var myObj = {
prop1: "val1",
prop2: [...],
...
}
The values in this object keep updating very frequently (several times every second) and there could be thousands of them. New values could be added, existing ones could be changed or removed.
I want to have a file that always has the updated version of this object. The simple approach for doing this would be just writing the entire object to the file all over again after each time that it changes like so:
fs.writeFileSync("file.json", JSON.stringify(myObj));
This doesn't seem very efficient for big objects that need to be written very frequently. Is there a better way of doing this?
You should use a database. Something simple like sqlite3 would be a good option. Have a table with just two columns 'Key' 'Value' and use it as a key value store. You will gain advantages like transactions and better performance than a file as well as simplifying your access.
Maintaining a file (on the filesystem) containing the current state of a rapidly changing object is surprisingly difficult. Specifically, setting things up so some other program can read the file at any time is the hard part. Why? At any time the file may be in the process of being written, so the reader can get inconsistent results.
Here's an outline of a good way to do this.
1) write the file less often than each time the state changes. Whenever the state changes call updateFile (myObj). It sets a timer for, let's say, 500ms, then writes the very latest state to the file when the timer expires. Something like this: not debugged:
let latestObj
let updateFileTimer = 0
function updateFile (myObj) {
latestObj = myObj
if (updateFileTimer === 0) {
updateFileTimer = setTimeout (
function () {
/* write latestObj to the file */
updateFileTimer = 0
}, 500)
}
}
This writes the latest state of your object to the file, but no more than every 500ms.
Inside that timeout function, write out a temporary file. When it's written delete the existing file and rename the temp file to have the existing file's name. Do all this asynchronously so the rest of your program won't have to wait for the filesystem to work. Your timeout function will look like this
updateFileTimer = setTimeout (
function () {
/* write latestObj to the file */
fs.writeFile("file.json.tmp",
JSON.stringify(myObj),
function (err) {
if (err) throw err;
fs.unlink ( "file.json",
function (err) {
if (!err)
fs.renameSync( "file.json.tmp", "file.json")
} )
} )
updateFileTimer = 0
}, 500)
There's one more thing to worry about. There's a brief period of time between the unlink and the renameSync operation where the "file.json" file does not exist in the file system. So, any program you write that READs "file.json" needs to try again if the file isn't found.
If you use a Linux, MacOs, FreeBSD, or other UNIX-derived operating system for this code it will work well. Those operating systems' file systems allow one program to unlink a file while another program is reading it. If you're running it on a DOS-derived operating system like Windows, the unlink operation will fail when another program is reading the file.
I am trying to build a logging mechanism, to log changes done to a record. I am currently logging previous and new record. However, as the site is very busy, I expect the logfile to grow seriously huge. To avoid this, I plan to only capture the modified fields only.
Is there a way to capture only the modifications done to a record (in REACT), so my {request.body} will have fewer fields?
My Server-side is build with NODE.JS and the client-side is REACT.
One approach you might want to consider is to add an onChange(universal) or onTextChanged(native) listener to the text field and store the form update in a local state/variables.
Finally, when a user makes an action (submit, etc.) you can send the updated data to the logging module.
The best way I found and works for me is …
on the api server-side, where I handle the update request, before hitting the database, I do a difference between the previous record and {request.body} using lodash and use the result to send to my update database function
var _ = require('lodash');
const difference = (object, base) => {
function changes(object, base) {
return _.transform(object, function (result, value, key) {
if (!_.isEqual(value, base[key])) {
result[key] = (_.isObject(value) && _.isObject(base[key])) ? changes(value, base[key]) : value;
}
});
}
return changes(object, base);
}
module.exports = difference
I saved the above code in a file named diff.js and included it in my server-side file.
It worked good.
Thanks for giving the idea...
I have a UDP client that grabs some data from another source and writes it to a file on the server. Since this is large amount of data, I dont want the end user to wait until they its full written to the server so that they can download it. So I made a NodeJS server that grabs the latest data from the file and sends it to the user.
Here is the code:
var stream = fs.readFileSync(filename)
.on("data", function(data) {
response.write(data)
});
The problem here is, if the download starts when the file was only for example 10mb.. the fs.readFileSync will only read my file up to 10mb. Even if 2 mins later the file increased to 100mb. fs.readFileSync will never know about the new updated data. How can I do this in Node? I would like somehow refresh the fs state or maybe perpaps wait for new data using fs file system. Or is there some kind of fs fileContent watcher?
EDIT:
I think the code below describes better what I would like to achieve, however in this code it keeps reading forever and I dont have any variable from fs.read that can help me stop it:
fs.open(filename, 'r', function(err, fd) {
var bufferSize=1000,
chunkSize=512,
buffer=new Buffer(bufferSize),
bytesRead = 0;
while(true){ //check if file has new content inside
fs.read(fd, buffer, 0, chunkSize, bytesRead);
bytesRead+= buffer.length;
}
});
Node has built-in methods in the fs module. It is tagged as unstable, so it can change in the future.
Its called: fs.watchFile(filename[, options], listener)
You can read more about it here: https://nodejs.org/api/fs.html#fs_fs_watchfile_filename_options_listener
But i highly suggest you to use one of the good modules mantained actively like
watchr:
From his readme:
Better file system watching for Node.js. Provides a normalised API the
file watching APIs of different node versions, nested/recursive file
and directory watching, and accurate detailed events for
file/directory changes, deletions and creations.
The module page is here: https://github.com/bevry/watchr
(Used the module in a couple of proyects and working great, im not related to it in other way)
you need store in some data base last size of file.
read filesize first.
load your file.
then make a script to check if file was change.
you can consult the size with jquery.post to obtain your result and decide if need to reload in javascript
I am not very much familiar with nodejs but, I need some guidance in my task. Any help would be appreciated.
I have nodejs file which runs from command line.
filename arguments and that do some operation whatever arguments I have passed.
Now, I have html page and different options to select different operation. Based on selection, I can pass my parameters to any file. that can be any local node js file which calls my another nodejs file internally. Is that possible ? I am not sure about what would be my approach !
I always have to run different command from terminal to execute different task. so, my goal is to reduce that overhead. I can select options from UI and do operations through nodejs file.
I was bored so I decided to try to answer this even though I'm not totally sure it's what you're asking. If you mean you just need to run a node script from a node web app and you normally run that script from the terminal, just require your script and run it programmatically.
Let's pretend this script you run looks like this:
// myscript.js
var task = process.argv[2];
if (!task) {
console.log('Please provide a task.');
return;
}
switch (task.toLowerCase()) {
case 'task1':
console.log('Performed Task 1');
break;
case 'task2':
console.log('Performed Task 2');
break;
default:
console.log('Unrecognized task.');
break;
}
With that you'd normally do something like:
$ node myscript task1
Instead you could modify the script to look like this:
// Define our task logic as functions attached to exports.
// This allows our script to be required by other node apps.
exports.task1 = function () {
console.log('Performed Task 1');
};
exports.task2 = function () {
console.log('Performed Task 2');
};
// If process.argv has more than 2 items then we know
// this is running from the terminal and the third item
// is the task we want to run :)
if (process.argv.length > 2) {
var task = process.argv[2];
if (!task) {
console.error('Please provide a task.');
return;
}
// Check the 3rd command line argument. If it matches a
// task name, invoke the related task function.
if (exports.hasOwnProperty(task)) {
exports[task]();
} else {
console.error('Unrecognized task.');
}
}
Now you can run it from the terminal the same way:
$ node myscript task1
Or you can require it from an application, including a web application:
// app.js
var taskScript = require('./myscript.js');
taskScript.task1();
taskScript.task2();
Click the animated gif for a larger smoother version. Just remember that if a user invokes your task script from your web app via a button or something, the script will be running on the web server and not the user's local machine. That should be obvious but I thought I'd remind you anyway :)
EDIT
I already did the video so I'm not going to redo it, but I just discovered module.parent. The parent property is only populated if your script was loaded from another script via require. This is a better way to test if your script is being run directly from the terminal or not. The way I did it might have problems if you pass an argument in when you start your app.js file, such as --debug. It would try to run a task called "--debug" and then print out "Unrecognized task." to the console when you start your app.
I suggest changing this:
if (process.argv.length > 2) {
To this:
if (!module.parent) {
Reference: Can I know, in node.js, if my script is being run directly or being loaded by another script?
I'm a front-end dev just venturing into the Node.js, particularly in using it to create small command line tools.
My question: how do you persist data with command line tools? For example, if I want to keep track of the size of certain files over time, I'd need to keep a running record of changes (additions and deletions) to those files, and relevant date/time information.
On the web, you store that sort of data in a database on a server, and then query the database when you need it again. But how do you do it when you're creating a Node module that's meant to be used as a command line tool?
Some generic direction is all I'm after. I don't even know what to Google at this point.
It really depends on what you're doing, but a simple approach is to just save the data that you want to persist to a file and, since we're talking node, store it in JSON format.
Let's say you have some data like:
var data = [ { file: 'foo.bar', size: 1234, date: '2014-07-31 00:00:00.000'}, ...]
(it actually doesn't matter what it is, as long as it can be JSON.stringifiy()d)
You can just save it with:
fs.writeFile(filename, JSON.stringify(data), {encoding: 'utf8'}, function(err) { ... });
And load it again with:
fs.readFile(filename, {encoding: 'utf8'}, function(err, contents) {
data = JSON.parse(contents);
});
You'll probably want to give the user the ability to specify the name of the file you're going to persist the data to via an argument like:
node myscript.js <data_file>
You can get that passed in parameter with process.argv:
var filename = process.argv[2]; // Be sure to check process.argv.length and have a default
Using something like minimist can be really helpful if you want to get more complex like:
node myscript.js --output <data_file>
You also can store files in temporary directory, for example /tmp directory on linux and give user the option to change the directory.
To get path to temporary directory you can use os module in nodejs:
const os = require('os');
const tmp = os.tmpdir();