NodeJS large directory file changes - node.js

I am working on more of a security dashboard, it watches for changes in files in the entire home directory with hundreds of sites (all Joomla, so a lot of files).
In order to keep on top of potential security issues we want to watch for file changes in an efficient way without creating unnecessary CPU/Memory overhead. We want to watch it at a faster interval but I know its more of a balancing act when you do want to keep it from using more cpu then a side process should.
I have tried to use "watch" with the following code, running in the home directory:
var watch, fs;
watch = require('watch');
fs = require('fs');
watch.createMonitor(__dirname,{interval:500,filter:function(file,stat){
if(file.indexOf('index.php')!=-1){
return true;
}else{
return false;
}
}},function(monitor){
monitor.filter(function(file){
console.log(file);
})
monitor.on('created',function(file,stat){
console.log(file + ' new');
});
monitor.on('changed',function(file,stat){
console.log(file + ' changed');
});
monitor.on('removed',function(file,stat){
console.log(file + ' deleted');
});
});
However this spikes the CPU to over 100% of a single core (sometimes 2) out of 8. Memory also takes up about 20% of 8gb pretty quickly as well. This is all just to create the watch event on all the files, so its before it can actually detect any file changes.
I know the issue with this is it goes through each file individually, and only does not track it if you filter that sort of file. Typically all I need to watch is the index.php in every directory, down to a point that it could be consistent (with some exceptions).
Is there a module already built to do this? Or is this something new? All modules I find assume its a smaller directory (like watching LESS or something) So not built for this sort of application at all.
Any ideas? I know this code will need to be scrapped as there is no way I can see to stop the CPU overhead.

Do not use package 'watch', just use fs.watch(...)
package 'watch':
consistent APIs
very slow because implement mostly in node, look source to see how it work
souce code: https://github.com/mikeal/watch/blob/master/main.js
fs.watch(..)
non-consistent APIs, not all OSs are supported.
very fast because it reused OS features
document: http://nodejs.org/docs/latest/api/fs.html#fs_fs_watch_filename_options_listener

Related

Problems using Tail (Node.js module) to read file as it is updated

I'm trying to use Tail (https://www.npmjs.com/package/tail) to export Minecraft server log data to Discord (The discord bot part works, so I have excluded it from here).
If I say something in the game and then check "latest.log", it has been changed accordingly. However, using this script, the bot only sees a change if I open "latest.log" in notepad, it doesn't work otherwise. The bot will recognize changes as long as "latest.log" is open in the background, which is an annoyance but not too big of a deal.
However, my friend is the one who I was making this for, and for him Tail only updates the moment he opens "latest.log". Which means he would need to keep opening up that file for Tail to see it, instead of just letting it run in the background.
Tail = require('tail').Tail;
var fileToTail = "C:/Users/user/Downloads/logs/latest.log";
tail = new Tail(fileToTail);
tail.on("line", function(data) {
//Working code that sends data
});
tail.on("error", function(error) {
console.log('ERROR: ', error);
});
What could be causing the discrepancy between the two of us, and what can I do so that the bot can see the file changes without the user opening the file? Thanks in advance!
If you are using chokidar, you should pay attention to whether you are using fs.watch Vs. fs.watchFile. If using fs.watch you not successfully catch changes (which might be what you are experiencing).
See below from official docs for chokidar options:
usePolling (default: false). Whether to use fs.watchFile (backed by
polling), or fs.watch. If polling leads to high CPU utilization,
consider setting this to false. It is typically necessary to set this
to true to successfully watch files over a network, and it may be
necessary to successfully watch files in other non-standard
situations. Setting to true explicitly on MacOS overrides the
useFsEvents default. You may also set the CHOKIDAR_USEPOLLING env
variable to true (1) or false (0) in order to override this option.

Nightmare doesn't run twice in a row - NodeJS

EDIT
I have noticed the removal of the .end() function appears to solve the issue, but after reading the Nightmare docs on the use of .end() it says: Completes any queue operations, disconnect and close the electron process.
Now while this does solve the problem, am I now just opening more and more electron processes each time the route is called, which will eventually cause the server to run out of memory, or is this a safe way to fix the issue?
ORIGINAL TEXT
Please consider the following problem:
I am developing a Node based service that will allow the user to request screenshot of a particular URL.
For this I am using Nightmare to visit the URL, wait 2 seconds, take a screenshot, which is saved to the disk, convert it to base64, delete the image and then return the base64 string.
console.log('Nightmare starts');
nightmare
.goto(url)
.wait(2000)
.screenshot(filename)
.end()
.then(function (result)
{
fs.exists(filename, function(exists)
{
if (exists)
{
data = fs.readFileSync(filename);
var base64 = data.toString('base64')
fs.unlink(filename);
var output = {'message':'success','map_image':base64};
res.send(output);
}
});
})
.catch(function (error)
{
console.error('Search failed:', error);
});
console.log("Nightmare Finished");
The above code works just fine, the first time it runs. However any subsequent calls to this just consoles "Nightmare starts" and "Nightmare Finished" instantly with the actual code in-between not running. I don't appear to have any errors display, nothing is caught if I wrap it in a try/catch. The node requires a reboot to allow it to happen again.
Something worth noting is that I am running on a headless ubuntu machine, as electron (one of the nightmare dependencies) appears to need a GUI, I am using xvfb to launch the node using the following command:
xvfb-run --auto-servernum --server-num=1 node server.js
I'm assuming this may be an issue with some resource not being released correctly on the first run, but any assistance would be appreciated.
Also open to any constructive criticism of my code, very new to Node and i'm sure i'm not writing in the most optimal way (sync file loading etc)
It appears that you are simply misplacing where you are creating the nightmare instances. Cannot help much without some more code snippet and information.
Way 1
Create nightmare instance every time and close them after you are done with your task. It will require some time to boot up the instance, but it will also lessen the memory load. Not to mention you can have multiple nightmare instances for different users.
Way 2
Don't end and re-use same nightmare instance. Have multiple nightmare instances and queue the call for screenshot. The websites will load fast and it won't take time to boot up an instance, but you will have longer wait time for longer queue.

Logrotation for a Nodejs Application

I am working on a very old Nodejs application which creates a new child process using forever-monitor. The logs of this child process are taken care by forever-monitor only. This is how the configuration looks like:
var child = new (forever.Monitor)(__dirname + '/../lib/childprocess.js', {
max: 3,
silent: true,
options: [program.port],
'errFile': __dirname + '/../childprocess_error.log',
'outFile': __dirname + '/../childprocess_output.log'
}
);
Everything is working fine in this setup. The new requirement is to rotate these logs every 12 hours. That is every 12 hours a new file will be created which will have all the content of this file childprocess_output.log and should be stored in some other directory. The new log file will obviously have the timestamp appended at the end of the name (eg: childprocess_output_1239484034.log).
And the original file childprocess_output.log should be reset, that is all its content should be deleted and it should start logging from fresh.
I am trying to understand which npm library should I used for this purpose. I googled a bit and found a few of the npm libraries which matches my requirement, but the number of downloads for these libraries was really small, so I doubt the reliability of those libraries.
Which library NodeJs developers use for log rotation?
Also, my last resort would be to use the Linux tool Logrotate if I couldn't find any appropriate library in Node. I am avoiding using Logroate because I want my application to handle the scenario and not depend on the instance configuration.
you can use :
fs (the file system library) handled with methods like statSync and renameSync coupled with try-catches block-codes.

Efficient media transfers in Node and Electron

As a toy project for Electron, I'm trying to rewrite a WPF media management application and would like to get up to speed on what would be the best means of loading media (namely images, but later video) through node's fs and get it into a webpage for Electron to render.
Currently I've got a basic working setup using Base64 encoding:
function getImage(imageID, success) {
fs.readFile('f:/pictures/2.jpg', function(err, data) {
if (err) throw err;
success(Buffer.from(data).toString('base64'));
});
}
and rendering it like so in my html:
api.getImage(0, (result) => {
$('#mainContent').append('<img src="data:image/jpeg;base64,' + result + '" />');
});
This doesn't strike me as the most efficient method to do so. Base64 conversions are helpful for cross technology boundaries, but not the most efficient way to do things.
Some other considerations I had:
Host the file directory via node so I can link to the file directly from my webpage (this is not ideal as files can come from anywhere on the local filesystem)
Use node to copy the file from the filesystem to the served directory so I can link to it via html (this would be slow: disk IO and whatnot)
Pass the raw file buffer back to the webpage somehow (this seems like it should be the way to go, but not sure how to render it in "traditional" html)
Considering that I may want to process 50 or 100 images or so at a time (including possible resizing for thumbnail views: something I was able to do in WPF effectively by optimizing the decoding for the BitmapImage), what would be the general approach to follow?
To clarify, I'm not asking so much about async/process blocking as I am the data transfer itself.

How to optimize my heroku webapp

I'm new to web development and I'm currently in the process of building my own website for my portfolio. My app uses node, express, and Heroku to launch it online. However, the page isn't as smooth as I'd like it to be. It drops a lot of frames when scrolling and viewing animations. I'm wondering what I can do to make my app feel buttery smooth. I've looked all around for solutions but I'm too new to development to really understand what to do. I've attempted to use the inspector to see if the css or javascript files are slowing down the processes via the waterfall insepctor, but came up dry. So far, I've compressed the files and lowered the scale of all images. Also, the entire website is static content so I can't imagine why its running so slowly. Any help would be much appreciated.
The website in question
The "lag" could be caused by the ScrollFire plugin. Every time you call Materialize.scrollFire(..) you actually add a JavaScript listener for the "scroll" event. The way you use it, you call Materialize.scrollFire for each of your target objects, so you actually create multiple "scroll" listeners. But the scrollFire options is actually an array of targets, so you could get away with only initializing it once. Like so:
var options = [
{selector: '.iphone1', offset: 300, callback: function(el) {
$('.iphone1').css('visibility', 'visible');
$('.iphone1').addClass('animated slideInLeft');
}},
{selector: '#paragraph_intro', offset: 300, callback: function(el) {
$('#paragraph_intro').css('visibility', 'visible');
$('#paragraph_intro').addClass('animated slideInRight');
}},
// ... And so on
];
Materialize.scrollFire(options);
It could also be caused by your own scroll listener at:
$(window).on('scroll', function(){
updateNavigation();
changeNavColor();
changeHeaderColor();
});
I would consider adding some kind of throttling, so these functions are called less frequent while the user is scrolling.
These may not be the problem, or the whole problem, as I cannot see how the page would behave without it, but it could have an impact, so it's worth investigating.

Resources