suppose i want to monitor mysql, as far as i know, all mysql runtime info is stored in /proc/{mysql_pid}/stat. so is it possible to read and parse mysql stat info via node.js and client display the chart real time ?
nagios and alternative is so heavy, and sometimes i just want to monitor some progress info. so i want a lightweight solution.
I tried using Node Inotify, which is an excellent library. Yet, it seems like the proc filesystem doesn't event to inotify when the stat files are changed. If you're watching a file on a normal filesystem, though, this is how you can do it using that library:
sys = require('sys');
fs = require('fs');
Inotify = require('inotify').Inotify;
var inotify = new Inotify();
function callback(ev) {
console.log(sys.inspect(ev));
}
var home_dir = {
path: '/proc/5499/stat'
, watch_for: Inotify.IN_ALL_EVENTS
, callback: callback
};
var home_watch_descriptor = inotify.addWatch(home_dir);
Just change Inotify.IN_ALL_EVENTS to whatever action you want to watch for, which is documented on the github page I linked.
Sorry this doesn't solve your particular problem, but I thought I'd post it informationally.
I'm assuming you haven't tried implementing a solution to your problem. What exactly do you mean by "real time"? What sort of client are you talking about?
If you're talking about a web browser client, there's no reason you couldn't update some sort of display every half-second with loads of clients (or much faster, if the charts aren't too intricate).
You should be more specific if you want a more specific answer than that.
Related
This is an update to a question I had asked previously but wasn't thinking straight when I asked the question (I was taking a very backwards approach to my solution). I'm currently working with Puppeteer and I'm trying to create an association between a particular task and a puppeteer browser instance. Right now I am currently getting the browser's context id using:
const {browserContextId} = await browser._connection.send('Target.createBrowserContext');
and I am storing it in a file along with other details related to the task. My goal is to somehow be able to close that browser instance using the stored context id. I've given this issue a read on the Puppeteer's GitHub hoping that it would help in some way but it seems that it's not super helpful to me as its not really related to what I'm doing.
The real issue is that I am going to be spawning browser instances in one file and attempting to close them in another, otherwise this wouldn't be an issue at all. Right now the only thing I've been able to do is just spawn another browser instance using the context id (pretty much useless for my task) and have had no luck in closing it or disposing it.
Any help would be greatly appreciated, thanks!
P.S. If there is a better approach to solving this association issue I'm all ears!
Turns out I was thinking about it way too much and trying to make it too complex. For anyone trying to do something similar I'll leave my solution here for you.
Instead of using the browser's context id I found it much easier to just grab the browser's process id (pid). From there I could kill the process using different strategies based on where I was running the close command.
For Node.js:
// Lets say for example we're instantiating our browser like this
const browser = await puppeteer.launch({ headless: false });
// You can simply run this to get the browser's pid
const browserPID = browser.process().pid
// Then you can either store it for use later
fs.writeFile(file, JSON.stringify(jsondata, null, 4), (err) => {
if (err) console.log(err);
})
// Or just kill the process
process.kill(browserPID);
Keep in mind that if you are storing the PID you need to read the file and parse the data to pass into the process.kill command.
For React via Electron
// Require process from the electron file
const process = window.require('process');
// Then same process as before
process.kill(yourbrowserPID);
Hopefully my stupidity can help someone in the future if they are trying to do something similar. It was way easier than I was making it out to be.
I'm trying to use Tail (https://www.npmjs.com/package/tail) to export Minecraft server log data to Discord (The discord bot part works, so I have excluded it from here).
If I say something in the game and then check "latest.log", it has been changed accordingly. However, using this script, the bot only sees a change if I open "latest.log" in notepad, it doesn't work otherwise. The bot will recognize changes as long as "latest.log" is open in the background, which is an annoyance but not too big of a deal.
However, my friend is the one who I was making this for, and for him Tail only updates the moment he opens "latest.log". Which means he would need to keep opening up that file for Tail to see it, instead of just letting it run in the background.
Tail = require('tail').Tail;
var fileToTail = "C:/Users/user/Downloads/logs/latest.log";
tail = new Tail(fileToTail);
tail.on("line", function(data) {
//Working code that sends data
});
tail.on("error", function(error) {
console.log('ERROR: ', error);
});
What could be causing the discrepancy between the two of us, and what can I do so that the bot can see the file changes without the user opening the file? Thanks in advance!
If you are using chokidar, you should pay attention to whether you are using fs.watch Vs. fs.watchFile. If using fs.watch you not successfully catch changes (which might be what you are experiencing).
See below from official docs for chokidar options:
usePolling (default: false). Whether to use fs.watchFile (backed by
polling), or fs.watch. If polling leads to high CPU utilization,
consider setting this to false. It is typically necessary to set this
to true to successfully watch files over a network, and it may be
necessary to successfully watch files in other non-standard
situations. Setting to true explicitly on MacOS overrides the
useFsEvents default. You may also set the CHOKIDAR_USEPOLLING env
variable to true (1) or false (0) in order to override this option.
I have a UDP client that grabs some data from another source and writes it to a file on the server. Since this is large amount of data, I dont want the end user to wait until they its full written to the server so that they can download it. So I made a NodeJS server that grabs the latest data from the file and sends it to the user.
Here is the code:
var stream = fs.readFileSync(filename)
.on("data", function(data) {
response.write(data)
});
The problem here is, if the download starts when the file was only for example 10mb.. the fs.readFileSync will only read my file up to 10mb. Even if 2 mins later the file increased to 100mb. fs.readFileSync will never know about the new updated data. How can I do this in Node? I would like somehow refresh the fs state or maybe perpaps wait for new data using fs file system. Or is there some kind of fs fileContent watcher?
EDIT:
I think the code below describes better what I would like to achieve, however in this code it keeps reading forever and I dont have any variable from fs.read that can help me stop it:
fs.open(filename, 'r', function(err, fd) {
var bufferSize=1000,
chunkSize=512,
buffer=new Buffer(bufferSize),
bytesRead = 0;
while(true){ //check if file has new content inside
fs.read(fd, buffer, 0, chunkSize, bytesRead);
bytesRead+= buffer.length;
}
});
Node has built-in methods in the fs module. It is tagged as unstable, so it can change in the future.
Its called: fs.watchFile(filename[, options], listener)
You can read more about it here: https://nodejs.org/api/fs.html#fs_fs_watchfile_filename_options_listener
But i highly suggest you to use one of the good modules mantained actively like
watchr:
From his readme:
Better file system watching for Node.js. Provides a normalised API the
file watching APIs of different node versions, nested/recursive file
and directory watching, and accurate detailed events for
file/directory changes, deletions and creations.
The module page is here: https://github.com/bevry/watchr
(Used the module in a couple of proyects and working great, im not related to it in other way)
you need store in some data base last size of file.
read filesize first.
load your file.
then make a script to check if file was change.
you can consult the size with jquery.post to obtain your result and decide if need to reload in javascript
So this is the setup I'm working with:
I am on an express server which must stream an archived binary payload to a browser (does not matter if it is zip, tar or tar.gz - although zip would be nice).
On this server, I have a websocket open that connects to another server which is sending me binary payloads of individual files in a directory. I get these payloads streamed, piece-by-piece, as buffers, and I'm doing this serially (that is - file-by-file - there aren't multiple websockets open at one time, and there is one websocket per file). This is the websocket library I'm using: https://github.com/einaros/ws
I would like to go through each file, open a websocket, and then append the buffers to an archiver as they come through the websockets. When data is appended to the archiver, it would be nice if I could stream the ouput of the archiver to the browser (via the response object with response.write). So, basically, as I'm getting the payload from the websocket, I would like that payload streamed through an archiver and then to the response. :-)
Some things I have looked into:
node-zipstream - This is nice because it gives me an output stream I can pipe directly to response.write. However, it doesn't appear to support nested files/folders, and, more importantly, it only accepts an input stream. I have looked at the source code (which is quite terse and readable), and it seems as though, if I were able to have access to the update function within ZipStream.prototype.addFile, I could just call that each time on the message event when I get a binary buffer from the websocket. This is quite messy/hacky though, and, given that this library already doesn't seem to support nested files/folders, I'm not sure I will be going with it.
node-archiver - This suffers from the same issue as node-zipstream (probably because it was inspired by it) where it allows me to pipe the output, but I cannot append multiple buffers for the same file within the archive (and then manually signal when the last buffer has been appended for a given file). However, it does allow me to have nested folders, which is a clear win over node-zipstream.
Is there something I'm not aware of, or is this just a really crazy thing that I want to do?
The only alternative I see at this point is to wait for the entire payload to be streamed through a websocket and then append with node-archiver, but I really would like to reap the benefit of true streaming/archiving on-the-fly.
I've also thought about the possibility of creating a read stream of sorts just to serve as a proxy object that I can pass into node-archiver and then just append the buffers I get from the websocket to this read stream. Looking at various read streams, I'm not sure how to do this though. The only way I could think of was creating a writestream, piping buffers to it, and having a readstream read from that writestream. Am I on the correct thought process here?
As always, thanks for any help/direction you can offer SO community.
EDIT:
Since I just opened this question, and I'm new to node, there may be a better answer than the one I provided. I will keep this question open and accept a better answer if one presents itself within a few days. As always, I will upvote any other answers, even if they're ridiculous, as long as they're correct and allow me to stream on-the-fly as mine does.
I figured out a way to get this working with node-archiver. :-)
It was based off my hunch of creating a temporary "proxy stream" of sorts, inspired by this SO question: How to create streams from string in Node.Js?
The basic gist is (coffeescript syntax):
archive = archiver 'zip'
archive.pipe response // where response is the http response
// and then for each file...
fileName = ... // known file name
fileSize = ... // known file size
ws = .... // create websocket
proxyStream = new Stream()
numBytesStreamed = 0
archive.append proxyStream, name: fileName
ws.on 'message', (dataBuffer) ->
numBytesStreamed += dataBuffer.length
proxyStream.emit 'data', dataBuffer
if numBytesStreamed is fileSize
proxyStream.emit 'end'
// function/indicator to do this for the next file in the folder
// and then when you're completely done...
archive.finalize (err, bytesOfArchive) ->
if err?
// do whatever
else
// unless you somehow knew this ahead of time
res.addTrailers
'Content-Length': bytesOfArchive
res.end()
Note that this is not the complete solution I implemented. There is still a lot of logic dealing with getting the files, their paths, etc. Not to mention error-handling.
EDIT:
Since I just opened this question, and I'm new to node, there may be a better answer. I will keep this question open and accept a better answer if one presents itself within a few days. As always, I will upvote any other answers, even if they're ridiculous, as long as they're correct and allow me to stream on-the-fly as mine does.
Did anyone set up something like this for himself using the existing
node.js REPL? I didn't think of a quick way to do it.
The way I do it today is using emacs and this:
https://github.com/ivan4th/swank-js
This module is composed of:
A SLIME-js addon to emacs which, in combination with js2-mode, lets
you simply issue a C-M-x somewhere in the body of a function def - and
off goes the function's string to the ..
Swank-js server (yes, you could eval from your local-machine
directly to a remote process) written in Node.js - It receives the
string of the function you eval'ed and actually evals it
A whole part that lets you connect to another port on that server
with your BROWSER and then lets you manipulate the DOM on that browser
(which is pretty amazing but not relevant)
My solution uses SLIME-js on the emacs side AND I require('swank-
js') on my app.js file
Now.. I have several issues and questions regarding my solution or
other possible ones:
Q1: Is this overdoing it? Does someone have a secret way to eval stuff
from nano into his live process?
Q2: I had to change the way swank-js is EVALing.. it used some
kind of black magic like this:
var Script = process.binding('evals').Script;
var evalcx = Script.runInContext;
....
this.context = Script.createContext();
for (var i in global) this.context[i] = global[i];
this.context.module = module;
this.context.require = require;
...
r = evalcx("CODECODE", this.context, "repl");
which, as far I understand, just copies the global variables to the
new context, and upon eval, doesn't change the original function
definitions - SOOO.. I am just using plain "eval" and IT
WORKS.
Do you have any comments regarding this?
Q3: In order to re-eval a function, it needs to be a GLOBAL function -
Is it bad practice to have all function definitions as global (clojure-like) ? Do you think there is another way to do this?
Actually, swank.js is getting much better, and it is now much easier to set up swank js with your project using NPM. I'm in the process of writing the documentation right now, but the functionality is there!
Check this out http://nodejs.org/api/vm.html
var util = require('util'),
vm = require('vm'),
sandbox = {
animal: 'cat',
count: 2
};
vm.runInNewContext('count += 1; name = "kitty"', sandbox, 'myfile.vm');
console.log(util.inspect(sandbox));
// { animal: 'cat', count: 3, name: 'kitty' }
Should help you a lot, all of the sandbox things for node uses it :) but you can use it directly :)
You might take a look at jsapp.us, which runs JS in a sandbox, and then exposes that to the world as a quick little test server. Here's the jsapp.us github repo.
Also, stop into #node.js and ask questions for a quicker response :)