I've created a single-page system and disk status node app dashboard in javascript using the Chokidar package in the server to watch for system and disk status file updates placed into a "new" directory from other enterprise applications. The "watcher.add" event causes the file to be processed by the server with the data ultimately sent to the web client dashboard via a websocket in JSON format. When it finishes processing the "new" file and sends out the websocket message, the event handler moves the "new" file to a processed directory in prep for next update.
The code below shows that the watcher process is declared inside a function in the websocket definition. This was only done because the message processing code needs access to the "conn" object defined with the websocket. Because of that the watcher process is only created in the server when a web client comes up and connects to the server. Until that happens none of the file updates get processed.
const express = require('express');
let ws = require('nodejs-websocket');
const fs = require('fs-extra');
const chokidar = require('chokidar');
.
.
.
let server = ws.createServer(function (conn) {
console.log("WebSocket Server turned on, port number:" + ws_port);
console.log("Server side connection created.");
//*******************************************
// Set up file Watcher processes
//*******************************************
// Define Watcher process for the file - <somefile>
let watcherSystemNew = chokidar.watch(
systemStatusBasePath+'\\new',{
persistent: true});
watcherSystemNew.on('ready', function onReady() {
console.log('Watcher function SystemNew ready!');
watcherSystemNew.on('add', function onAdd() {
console.log('Found file in \'new\'directory');
handleNewFileUpdate(<filetype>, "system",conn); //Event Handler
});
});
I would like to move the watcher declarations outside of this websocket definition so that the server processes the updates into its internal storage whether or not any web dashboard clients are connected to it. I tried that already but I have been unable to access the "conn" object from the event handler once I do that and have not figured out a way around it. There may be a way rewrite the websocket definition statement to enable access to the "conn" object outside the declaration but my javascript knowledge is not yet that deep and so far it has eluded me.
Can anyone suggest a possible solution? All my web searches just return the "standard" declaration above.
Related
I am running into a problem while using socket.io to do some event handling. For some reason, the following code snippet does not handle the event 'update', or any event for that matter. Let me explain the situation.
I have created a file named updates.js to create a socket.io socket variable named socket_8888 that is bound to port 8888. I then use module.exports to make that socket variable available to any other file that imports updates.js using require('updates.js'). I structured my application this way because I need to emit events from several different files.
In app.js:
var updates = require('updates.js');
setTimeout(function() {
updates.regular.on("update", function () {
console.log("Updated.");
})
}, 1000);
setTimeout(
function () {
console.log(updates.regular.eventNames()); // Verifying that there is actually a listener bound to the socket -> prints ['update']
updates.regular.emit("update", 100)
}, 1500);
In updates.js:
var io = require("socket.io");
var socket_8888 = io(8888);
var updates = {
regular: socket_8888
};
module.exports = updates;
However, a few simple tests have uncovered that events are not being handled, and I really cannot figure out why. The word "Updated" should print a second and a half after I run the application using "node www", but it does not.
The reason I started doing this simple testing was because I am trying to revive an old website of mine, but after a couple years, API updates have rendered a lot of my code useless. So I am trying to rebuild. I am not trying to send events between different files on the server. I am only testing the events locally because the events were not firing to the browser client. For this reason, I decided to investigate using this simple test, and it turns out the events can not even be emitted/listened to on the actual server, let alone be handled on a client that is on a whole different network.
I have already verified that the listener is actually binding to the socket. However, I do not know how to check whether or not the socket is actually emitting the event "update".
I have written the listener to bind only after one second because attempting to bind the moment the application starts does not give Express enough time to set everything up. Otherwise, the socket would still be undefined.
I do not get any error messages. The code just does not work as I expected.
I would really appreciate it if the community can tell me why the event 'update' is not being handled.
To include update module (update.js)
Try this
It work's Perfectly
module.exports = updates
var updates = require('./updates');
I need to code a REST API that can check if a PDF file exist in a specific folder in the server.
The client send GET request and server should wait before send response, until the PDF file exist.
When the PDF file appears in the folder, the server need to response filename to client.
I think using node.js with express and socket.io to do this.
Do you think it's the right way ?
Have you got a code example for sync wait and file check response ?
Thanks
Before coding REST API routes, i prefer in a first step to code file checking function.
I tested fs.existsSync not really good
const fs = require('fs')
const path = './*.pdf'
if (fs.existsSync(path)) {
//file exists
}
and i am going to test maybe with glob.sync or glob-fs
I don't know what the good way for this first step
Update :
Glob-fs seems to be ok, but I need a wait time until .PDF file arrived on the server fs.
var glob = require('glob-fs')({ gitignore: true });
glob.readdir('**/*.pdf', function(err, files) {
console.log(files);
});
REST API is not what you are looking for. You should not stall your node.js server.
You should use Websocket: You can register your application as interested to know when a file appears in a directory. Then, when that event occurs, the server sends you a notification. No waiting.
Check https://www.tutorialspoint.com/websockets/index.htm for more info about Websockets.
Check https://nodejs.org/api/fs.html#fs_fs_watchfile_filename_options_listener for watching file modifications
Here a code using Chokidar to watch PDF file creation :
var fileWatcher = require("chokidar");
// Initialize watcher.
var watcher = fileWatcher.watch("./*.pdf", {
ignored: /[\/\\]\./,
persistent: true
});
// Add event listeners.
watcher
.on('add', function(path) {
console.log('File', path, 'has been added');
})
I want to emit an event to the client when a long fucntion comes to an end.
This will show a hidden div with a link - on the client side.
This is the approach i tested:
//server.js
var express = require('express');
var app = express();
var http = require('http').Server(app);
var io = require('socket.io')(http);
require('./app/routes.js')(app, io);
//routes.js
app.post('/pst', function(req, res) {
var url = req.body.convo;
res.render('processing.ejs');
myAsyncFunction(url).then(result => {
console.log('Async Function completed');
socket.emit('dlReady', { description: 'Your file is ready!'});
//do some other stuff here
}).catch(err => {
console.log(err);
res.render('error.ejs');
})
});
I get this
ERROR: ReferenceError: socket is not defined
If i change the socket.emit() line to this:
io.on('connection', function(socket) {
socket.emit('dlReady', { description: 'Your file is ready!'});
});
Then i don't receive an error, but nothing happens at the client.
This is the client code:
<script>
document.querySelector('.container2').style.display = "none";
var socket = io();
socket.on('dlReady', function(data) { //When you receive dlReady event from socket.io, show the link part
document.querySelector('.container1').style.display = "none";
document.querySelector('.container2').style.display = "block";
});
</script>
This whole concept is likely a bit flawed. Let me state some facts about this environment that you must fully understand before you can follow what needs to happen:
When the browser does a POST, there's an existing page in the browser that issues the post.
If that POST is issued from a form post (not a post from Javascript in the page), then when you send back the response with res.render(), the browser will close down the previous page and render the new page.
Any socket.io connection from the previous page will be closed. If the new page from the res.render() has Javascript in it, when that Javascript runs, it may or may not create a new socket.io connection to your server. In any case, that won't happen until some time AFTER the res.render() is called as the browser has to receive the new page, parse it, then run the Javascript in it which has to then connect socket.io to your server again.
Remember that servers handle lots of clients. They are a one-to-many environment. So, you could easily have hundreds or thousands of clients that all have a socket.io connection to your server. So, your server can never assume there is ONE socket.io connection and sending to that one connection will go to a particular page. The server must keep track of N socket.io connections.
If the server ever wants to emit to a particular page, it has to create a means of figuring out which exact socket.io connect belongs to the page that it is trying to emit to, get that particular socket and call socket.emit() only on that particular socket. The server can never do this by creating some server-wide variable named socket and using that. A multi-user server can never do that.
The usual way to "track" a given client as it returns time after time to a server is by setting a unique cookie when the client first connects to your server. From then on, every connection from that client to your server (until the cookie expires or is somehow deleted by the browser) whether the client is connection for an http request or is making a socket.io connection (which also starts with an http request) will present the cookie and you can then tell which client it is from that cookie.
So, my understanding of your problem is that you'd like to get a form POST from the client, return back to the client a rendered processing.ejs and then sometime later, you'd like to communicate with that rendered page in the client via socket.io. To do that, the following steps must occur.
Whenever the client makes the POST to your server, you must make sure there is a unique cookie sent back to that client. If the cookie already exists, you can leave it. If it does not exist, you must create a new one. This can be done manually, or you can use express-session to do it for you. I'd suggest using express-session because it will make the following steps easier and I will outline steps assuming you are using express-session.
Your processing.ejs page must have Javascript in it that makes a socket.io connection to your server and registers a message listener for your "dlready" message that your server will emit.
You will need a top-level io.on('connection', ...) on your server that puts the socket into the session object. Because the client can connect from multiple tabs, if you don't want that to cause trouble, you probably have to maintain an array of sockets in the session object.
You will need a socket.on('disconnect', ...) handler on your server that can remove a socket from the session object it's been stored in when it disconnects.
In your app.post() handler, when you are ready to send the dlready message, you will have to find the appropriate socket for that browser in the session object for that page and emit to that socket(s). If there are none because the page you rendered has not yet connected, you will have to wait for it to connect (this is tricky to do efficiently).
If the POST request comes in from Javascript in the page rather than from a form post, then things are slightly simpler because the browser won't close the current page and start a new page and thus the current socket.io connection will stay connected. You could still completely change the page visuals using client-side Javascript if you wanted. I would recommend this option.
Introduction
Say that on the same local network we have two Node JS servers set up with Express: Server A for API and Server F for form.
Server A is an API server where it takes the request and saves it to MongoDB database (files are stored as Buffer and their details as other fields)
Server F serves up a form, handles the form post and sends the form's data to Server A.
What is the most efficient way to send files between two NodeJS servers where the receiving server is Express API? Where does the file size matter?
1. HTTP Way
If the files I'm sending are PDF files (that won't exceed 50mb) is it efficient to send the whole contents as a string over HTTP?
Algorithm is as follows:
Server F handles the file request using https://www.npmjs.com/package/multer and saves the file
then Server F reads this file and makes an HTTP request via https://github.com/request/request along with some details on the file
Server A receives this request and turns the file contents from string to Buffer and saves a record in MongoDB along with the file details.
In this algorithm, both Server A (when storing into MongoDB) and Server F (when it was sending it over to Server A) have read the file into the memory, and the request between the two servers was about the same size as the file. (Are 50Mb requests alright?)
However, one thing to consider is that -with this method- I would be using the ExpressJS style of API for the whole process and it would be consistent with the rest of the app where the /list, /details requests are also defined in the routes. I like consistency.
2. Socket.IO Way
In contrast to this algorithm, I've explored https://github.com/nkzawa/socket.io-stream way which broke away from the consistency of the HTTP API on Server A (as the handler for socket.io events are defined not in the routes but the file that has var server = http.createServer(app);).
Server F handles the form data as such in routes/some_route.js:
router.post('/', multer({dest: './uploads/'}).single('file'), function (req, res) {
var api_request = {};
api_request.name = req.body.name;
//add other fields to api_request ...
var has_file = req.hasOwnProperty('file');
var io = require('socket.io-client');
var transaction_sent = false;
var socket = io.connect('http://localhost:3000');
socket.on('connect', function () {
console.log("socket connected to 3000");
if (transaction_sent === false) {
var ss = require('socket.io-stream');
var stream = ss.createStream();
ss(socket).emit('transaction new', stream, api_request);
if (has_file) {
var fs = require('fs');
var filename = req.file.destination + req.file.filename;
console.log('sending with file: ', filename);
fs.createReadStream(filename).pipe(stream);
}
if (!has_file) {
console.log('sending without file.');
}
transaction_sent = true;
//get the response via socket
socket.on('transaction new sent', function (data) {
console.log('response from 3000:', data);
//there might be a better way to close socket. But this works.
socket.close();
console.log('Closed socket to 3000');
});
}
});
});
I said I'd be dealing with PDF files that are < 50Mb. However, if I use this program to send larger files in the future, is socket.io a better way to handle 1GB files as it's using stream?
This method does send the file and the details across but I'm new to this library and don't know if it should be used for this purpose or if there is a better way of utilizing it.
Final thoughts
What alternative methods should I explore?
Should I send the file over SCP and make an HTTP request with file details including where I've sent it- thus, separating the protocols of files and API requests?
Should I always use streams because they don't store the whole file into memory? (that's how they work, right?)
This https://github.com/liamks/Delivery.js ?
References:
File/Data transfer between two node.js servers this got me to try socket-stream way.
transfer files between two node.js servers over http for HTTP way
There are plenty of ways to achieve this , but not so much to do it right !
socket io and wesockets are efficient when you use them with a browser , but since you don't , there is no need for it.
The first method you can try is to use the builtin Net module of nodejs, basically it will make a tcp connection between the servers and pass the data.
you should also keep in mind that you need to send chunks of data not the entire file , the socket.write method of the net module seems to be a good fit for your case check it : https://nodejs.org/api/net.html
But depending on the size of your files and concurrency , memory consumption can be quite large.
if you are running linux on both servers you could even send the files at ground zero with a simple linux command called scp
nohup scp -rpC /var/www/httpdocs/* remote_user#remote_domain.com:/var/www/httpdocs &
You can even do this with windows to linux or the other way.
http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
the client scp for windows is pscp.exe
Hope this helps !
For logged in users only, I want to somehow notify them if they have any e.g. new notifications.
For example, say a member has sent them a private message, I want to tell the user that they have a new message to view (assuming they have not refreshed the page).
With Nodejs and redis, how would I go about doing this?
Note: I only need nodejs to send a small json to the user saying they have a new message.
The workflow is as follows that I was thinking:
1. user is logged in, a new message is sent to them.
2. somehow using nodejs and redis and long-polling, nodejs communicates back to the logged in users browser they have a pending message.
3. when nodejs sends this push notification, I then call another javascript function that will call a rest service to pull down additional json with the message.
I am integrating nodejs into an existing application, so I want to keep it as simple as possible with nodejs responsible for only notifying and not doing any additional logic.
Can someone outline how I should get going with this?
Should I be using redis http://redis.io/topics/pubsub somehow?
I'm not really sure how that works even after reading the page about it.
If you are integrating your node service into an existing application I would rather use some sort of messaging system to communicate messages from that application to node instead of a DB, even an in-memory DB. For clarity, I will assume you can use rabbitmq. If you do need to use redis, you will just need to find a way to use its publishing instead of rabbitmq publishing and corresponding node-side subscription, but I would imagine that the overall solution would be identical.
You need the following modules:
rabbitmq server (installation complexity about the same as for redis)
rabbitmq library in your external application to send messages, most languages are supported
rabit.js module for node to subscribe to messages or to communicate back to the external application
socket.io module for node to establish real-time connection between the node server and clients
I will also assume that both your external application and your node server have access to some shared DB (which can be redis), where node client session information is stored (e.g. redis-session-store for node). This would allow to use sessionId to validate who the message is for, if the user in the session is logged in and if certain users need to be sent notifications at all (by an external app).
This is how your stack might look like (unpolished):
Define a publisher in node to notify your external application that it needs to start/stop sending messages for a given sessionId. I will assume that for a given sessionId the user information can be recovered on either side (node or external application) from the shared DB and the user can be validated (here for simplicity by checking session.authenticated_user). Also define a subscriber to listen to incoming messages for the users:
var context = require('rabbit.js').createContext();
var pub = context.socket('PUB');
var sub = context.socket('SUB');
Define a socket.io connection(s) from your node server to the clients. As soon the client's web page is (re)loaded and io.connect() is called the below code will be executed (see clinet side at the end of the answer). As a new connection is established, validate the user is logged in (meaning its credentials are in the session), register the socket handler and publish a notification to the external application to start sending messages for this sessionId. The code here assumes a page reload on login/logout (and thus new socket.io session). If this is not the case, just emit a corresponding socket.io message from the client to node and register a handler in the method below in the same way as it is done for a new connection (this is beyond the scope of this example):
var sessionStore = undefined; // out-of-scope: define redis-session-store or any other store
var cookie = require("cookie"),
parseSignedCookie = require('connect').utils.parseSignedCookie;
// will store a map of all active sessionIds to sockets
var sockets = {};
// bind socket.io to the node http server
var io = require('socket.io').listen(httpServer);
// assumes some config object with session secrect and cookie sid
io.sockets.on("connection", function(socket) {
if (socket.handshake.headers.cookie) {
var cks = cookie.parse(socket.handshake.headers.cookie);
var sessionId = parseSignedCookie(cks[config.connectSid], config.sessionSecret);
// retrieve session from session store for sessionId
sessionStore.get(sessionId, function(err, session) {
// check if user of this session is logged in,
// define your elaborate method here
if (!err && session.authenticated_user) {
// define cleanup first for the case when user leaves the page
socket.on("disconnect", function() {
delete sockets[sessionId];
// notify external app that it should STOP publishing
pub.connect('user_exchange', function() {
pub.write(JSON.stringify({sessionId: sessionId, action: 'stop', reason: 'user disconnected'}), 'utf8');
});
});
// store client-specific socket for emits to the client
sockets[sessionId] = socket;
// notify external app that it should START publishing
pub.connect('user_exchange', function() {
pub.write(JSON.stringify({sessionId: sessionId, action: 'start'}), 'utf8');
});
}
});
}
});
Connect subscriber to the rabbitmq exchange to catch messages and emit them to clients:
sub.connect('messages_exchange', function() {
sub.on("readable", function() {
// parse incoming message, we need at least sessionId
var data = JSON.parse(sub.read());
// get socket to emit for this sessionId
var socket = sockets[data.sessionId];
if (socket) {
socket.emit("message", data.message);
} else {
// notify external app that it should STOP publishing
pub.connect('user_exchange', function() {
pub.write(JSON.stringify({sessionId: sessionId, action: 'stop', reason: 'user disconnected'}), 'utf8');
});
// further error handling if no socket found
}
});
});
Finally your client will look roughly like this (here in Jade, but that's just because I already have this whole stack along these lines):
script(src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js")
script(src="/socket.io/socket.io.js")
script(type='text/javascript').
$(function(){
var iosocket = io.connect();
iosocket.on('connect', function () {
// do whatever you like on connect (re-loading the page)
iosocket.on('message', function(message) {
// this is where your client finally gets the message
// do whatever you like with your new message
});
});
// if you want to communicate back to node, e.g. that user was logged in,
// do it roughly like this
$('#btnSend').click(function(event) {
iosocket.send('a message back to the node server if you need one');
});
});
Here is also a really nice explanation from Flickr on how they created a highly available and scalable push notification system with NodeJS and Redis.
http://code.flickr.net/2012/12/12/highly-available-real-time-notifications/