I'm using Express with node-amqp. My goal is to create amqpConnection and save it to global object before server starts and in Express routes use previously created globals.amqp_connection.
### server.coffee
app.use( ... )
...
# create RabbitMQ connection before server starts
connection = require("amqp").createConnection
host: "localhost"
connection.on "ready", ->
console.log "Got connection.on(\"ready\") event from node-amqp..."
# make amqp_connection accessible from any point of application
globals.amqp_connection = connection
server = globals.http.createServer(app).listen(8082)
console.log "Express server is listening 8082..."
The problem is connection.on "ready"-event is fired eveytime I call a route. I may suppose that's because of Express way of serving http-requests — executing server.js for every route called. So, for every request, a new instance of connection is created and on it's "ready" app tries to create one more instance of Express server.
How to make amqp_connection accessible from any point of my app, but no doubling require("amqp").createConnection() in every point where I need to push something to RabbitMQ?
UPD: or maybe there is no problem with Express. node-amqp seems to fire ready event every second after creation. Don't know if it's correct behavior
Thank you.
Ready event fired multiple 'cause of connection error:
https://github.com/postwait/node-amqp/issues/255
Related
I am running a nodeJS application using forever npm module.
Node application also connects to Redis DB for cache check. Quite often the API stops working with the following error on the forever log.
{ ReplyError: Ready check failed: ERR max number of clients reached
at parseError (/home/myapp/core/node_modules/redis/node_modules/redis-parser/lib/parser.js:193:12)
at parseType (/home/myapp/core/node_modules/redis/node_modules/redis-parser/lib/parser.js:303:14)
at JavascriptRedisParser.execute (/home/myapp/ecore/node_modules/redis/node_modules/redis-parser/lib/parser.js:563:20) command: 'INFO', code: 'ERR' }
when I execute the client list command on the redis server it shows too many open connections. I have also set the timeout = 3600 in my Redis configuration.
I do not have any unclosed Redis connection object on my application code.
This happens once or twice in a week depending on the application load, as a stop gap solution I am restarting the node server( it works ).
What could be the permanent solution in this case?
I have figured out why. This has nothing to do with Redis. Increasing the OS file descriptor limit was just a temporary solution. I was using Redis in a web application and the connection was created for every new request.
When the server was restarted occasionally, all the held-up connections by the express server were released.
I solved this by creating a global connection object and re-using the same. The new connection is created only when necessary.
You could do so by creating a global connection object, make a connection once, and make sure it is connected before every time you use that. Check if there is an already coded solution depending on your programming language. In my case it was perl with dancer framework and I used a module called Dancer2::Plugin::Redis
redis_plugin
Returns a Dancer2::Plugin::Redis instance. You can use redis_plugin to
pass the plugin instance to 3rd party modules (backend api) so you can
access the existing Redis connection there. You will need to access
the actual methods of the the plugin instance.
In case if you are not running a web-server and you are running a worker process or any background job process, you could do this simple helper function to re-use the connection.
perl example
sub get_redis_connection {
my $redis = Redis->new(server => "www.example.com:6372" , debug => 0);
$redis->auth('abcdefghijklmnop');
return $redis;
}
...
## when required
unless($redisclient->ping) {
warn "creating new redis connection";
$redisclient = get_redis_connection();
}
I was running into this issue in my chat app because I was creating a new Redis instance each time something connected rather than just creating it once.
// THE WRONG WAY
export const getRedisPubSub = () => new RedisPubSub({
subscriber: new Redis(REDIS_CONNECTION_CONFIG),
publisher: new Redis(REDIS_CONNECTION_CONFIG),
});
and where I wanted to use the connection I was calling
// THE WRONG WAY
getNewRedisPubsub();
I fixed it by just creating the connection once when my app loaded.
export const redisPubSub = new RedisPubSub({
subscriber: new Redis(REDIS_CONNECTION_CONFIG),
publisher: new Redis(REDIS_CONNECTION_CONFIG),
});
and then I passed the one-time initialized redisPubSub object to my createServer function.
It was this article here that helped me see my error: https://docs.upstash.com/troubleshooting/max_concurrent_connections
I want to emit an event to the client when a long fucntion comes to an end.
This will show a hidden div with a link - on the client side.
This is the approach i tested:
//server.js
var express = require('express');
var app = express();
var http = require('http').Server(app);
var io = require('socket.io')(http);
require('./app/routes.js')(app, io);
//routes.js
app.post('/pst', function(req, res) {
var url = req.body.convo;
res.render('processing.ejs');
myAsyncFunction(url).then(result => {
console.log('Async Function completed');
socket.emit('dlReady', { description: 'Your file is ready!'});
//do some other stuff here
}).catch(err => {
console.log(err);
res.render('error.ejs');
})
});
I get this
ERROR: ReferenceError: socket is not defined
If i change the socket.emit() line to this:
io.on('connection', function(socket) {
socket.emit('dlReady', { description: 'Your file is ready!'});
});
Then i don't receive an error, but nothing happens at the client.
This is the client code:
<script>
document.querySelector('.container2').style.display = "none";
var socket = io();
socket.on('dlReady', function(data) { //When you receive dlReady event from socket.io, show the link part
document.querySelector('.container1').style.display = "none";
document.querySelector('.container2').style.display = "block";
});
</script>
This whole concept is likely a bit flawed. Let me state some facts about this environment that you must fully understand before you can follow what needs to happen:
When the browser does a POST, there's an existing page in the browser that issues the post.
If that POST is issued from a form post (not a post from Javascript in the page), then when you send back the response with res.render(), the browser will close down the previous page and render the new page.
Any socket.io connection from the previous page will be closed. If the new page from the res.render() has Javascript in it, when that Javascript runs, it may or may not create a new socket.io connection to your server. In any case, that won't happen until some time AFTER the res.render() is called as the browser has to receive the new page, parse it, then run the Javascript in it which has to then connect socket.io to your server again.
Remember that servers handle lots of clients. They are a one-to-many environment. So, you could easily have hundreds or thousands of clients that all have a socket.io connection to your server. So, your server can never assume there is ONE socket.io connection and sending to that one connection will go to a particular page. The server must keep track of N socket.io connections.
If the server ever wants to emit to a particular page, it has to create a means of figuring out which exact socket.io connect belongs to the page that it is trying to emit to, get that particular socket and call socket.emit() only on that particular socket. The server can never do this by creating some server-wide variable named socket and using that. A multi-user server can never do that.
The usual way to "track" a given client as it returns time after time to a server is by setting a unique cookie when the client first connects to your server. From then on, every connection from that client to your server (until the cookie expires or is somehow deleted by the browser) whether the client is connection for an http request or is making a socket.io connection (which also starts with an http request) will present the cookie and you can then tell which client it is from that cookie.
So, my understanding of your problem is that you'd like to get a form POST from the client, return back to the client a rendered processing.ejs and then sometime later, you'd like to communicate with that rendered page in the client via socket.io. To do that, the following steps must occur.
Whenever the client makes the POST to your server, you must make sure there is a unique cookie sent back to that client. If the cookie already exists, you can leave it. If it does not exist, you must create a new one. This can be done manually, or you can use express-session to do it for you. I'd suggest using express-session because it will make the following steps easier and I will outline steps assuming you are using express-session.
Your processing.ejs page must have Javascript in it that makes a socket.io connection to your server and registers a message listener for your "dlready" message that your server will emit.
You will need a top-level io.on('connection', ...) on your server that puts the socket into the session object. Because the client can connect from multiple tabs, if you don't want that to cause trouble, you probably have to maintain an array of sockets in the session object.
You will need a socket.on('disconnect', ...) handler on your server that can remove a socket from the session object it's been stored in when it disconnects.
In your app.post() handler, when you are ready to send the dlready message, you will have to find the appropriate socket for that browser in the session object for that page and emit to that socket(s). If there are none because the page you rendered has not yet connected, you will have to wait for it to connect (this is tricky to do efficiently).
If the POST request comes in from Javascript in the page rather than from a form post, then things are slightly simpler because the browser won't close the current page and start a new page and thus the current socket.io connection will stay connected. You could still completely change the page visuals using client-side Javascript if you wanted. I would recommend this option.
I am trying to use Socket.io and Sequelize to create a chat app. Socket.io will handle the socket to allow for instant messaging. Sequelize will handle storing the messages so when you refresh the screen you still have your messages.
What is happening is that on localhost my socket works, but it does not send the messages to the database. When I put it onto Heroku, my database worked, but it does not use the sockets.
My socket is located in app.js and my database route is located in routes/messages.js.
I have been working on this bug for a while now and I have been trying to get help with it. I think the best way to share this is with my markdown I created detailing my efforts to fix my bug that can be found at here. My repo for this can be found here.
There are a few different parts that you need to distinguish:
the HTTP server, in your code represented by the variable http
the Express app, represented by app
the Socket.IO server, represented by io
a Socket.IO (client) connection (see below)
The HTTP server directs "normal" HTTP requests to the Express app, which will handle them according to the middleware and routes that are set up. A router handler gets called with (at least) two arguments, generally called req and res, to represent the (incoming) HTTP request and the (outgoing) HTTP response.
The Socket.IO server gets to handle specific Socket.IO requests, which get sent to the server by the Socket.IO client (running in the browser). When such a client sets up a connection with the server, the connection event gets triggered on the server. Any handlers for this event will get passed an argument, generally called socket, that represents the (bidirectional) connection with that client.
That Socket.IO connection can receive messages (sent from the client running in the browser), which trigger events on the socket. You can install a handler to listen for particular messages (like "chat message"), which will receive, as argument, the data that was sent to it by the client.
The issue in your code seems to be with setting up everything to handle those chat messages. The correct setup order would be:
listen on the Socket.IO server for connection events
when such an event is received, add a listener for the chat message event on the connection
when such an event is received, write the data to the database.
In code:
// Listen for new client connections.
io.on('connection', function(socket) {
// Listen for the client to send a _"chat message"_ message.
socket.on('chat message', function(data) {
// Store the data in the database.
models.Messages.create({
message : data.message,
username : data.username
});
});
});
As you can see, req and res aren't available inside of those Socket.IO event handlers, because those are only used for normal HTTP requests.
Also, as opposed to HTTP, you don't necessarily have to send anything back to the client when you have received a message, so I left that part out. The handler above only writes the message data to the database (it also doesn't check for, or handle, errors, which eventually you should add).
I have developed an application with ReactJS, ExpressJS, MongoDB and SocketIO.
I have two servers:- Server A || Server B
Socket Server is hosted on the Server A and application is hosted on the Server B
I am using Server A socket on Server B as a client.
Mainly work of Server A socket is to emit the data after fetching from the MongoDB database of Server A
Everything is working as expected but after 4-5-6 hours stop emitting the data but the socket connection will work.
I have checked using
socket.on('connection',function(){
console.log("Connected")
)
I am not getting whats the wrong with the code.
My code : https://jsfiddle.net/ymqxo31d/
Can anyone help me out on this
I have some programming errors.
I am getting data from MongoDB inside the setInterval() so after a little while exhausts resources and database connection starts failing every time.
Firstly i have created a Single MongoDB connection and used every place where i needed.
2ndly i removed setInterval and used setTimeout as below. (NOTE: If i continue using setInterval it execute on defined interval. It doesn't have any status that the data is emitted or not [this also cause the heavy resource usages] but i need to emit the data to socket when successfully fetched.)
setTimeout(emitData,1000);
function emitData(){
db.collection.find({}).toArray(function(data){
socket.emit('updateData',data);
setTimeout(emitData,1000);
})
}
I've ran into a fairly difficult to debug error with my node web server.
Background
I'm creating a node server with socket.io to provide a restful service, connected to mongodb which use web sockets(socket.io) for server-client messages.
Issue
In my node app, I've used an npm package called node-scheduler, in which I do some processing at set times(these are very dynamic times but work fairly well to date).
So I'll set off a job, using node-scheduler and when it ends you can provide a function.
In this function I emit a web socket message, exactly how I emit messages in the rest of the application but my client side never receives the message.
Checking the logs the client disconnections then re connections after the function has completed.
I've debugged a little further, and I send two messages to the client in this function. Only one of them is processed by the client. May be a client issue not a server issue.
Any ideas for solutions or suggestions would greatly be appreciated.
Well generally socket.io is only meant to be used as a "channel". You should have the Client exist as a separate entity in memory or something, and update the socket if and when it reconnects. Otherwise you're just sending to the past (disconnected) sockets.
Using passport you can identify a client as a user.
app.get('/', function(req, res){
// req.user;
});
Using passport.socketio you can get the same user in your socket
io.on('connection', function(socket){
// socket.request.user;
socket.request.user.socket = socket;
// this will be updated with the latest socket in case of a future reconnection
// So now you can be sure that user object will always have the latest socket
nodeScheduler(function(){
carryOutJobs(function callback(){
socket.request.user.socket.emit('done');
// will always emit to the "latest" socket.
});
});
});