I'm using Redis in my nodejs server, communicating with a browser html/js application. Under conditions I can't identify, my server gets stuck. It throws no errors, keep to be alive and responding, but can't get values from Redis. This is (the simplified version) how the connection with Redis is established.
var Stash = require('redis'),
Sockjs = require('sockjs')
stash = Stash.createClient(some_port,some_host);
sockjs = Sockjs.createServer();
var Instances = {
get: function (id, cb) {
stash.get('something_' + id, cb)
}
sockjs = Sockjs.createServer();
sockjs.on('connection', function(socket) {
socket.on('data', function(data) {
var id = somecorrectid;
Instances.get(id, function(err, instance) { /*Here the magic happens*/ }
}
}
Now, this works perfectly in normal conditions.
When the issue happens, the code between the last brackets isn't executed anymore, no matter how many times I fire again the event triggering the piece of code (that is a post data throught a socket) and from what client. Redis, in the meanwhile, is alive and rocking. Redis-cli is responding. The key searched with the 'get' exist and it is correct in its form. My server is still able to write keys to Redis correctly, but seems it can't read them anymore. This have no sense to me, and it is difficult to investigate or have more informations for I can't get how to replicate the issue. Any clue?
Related
I googled a lot but still have no clear solution to my issue.
Connecting to MongoDB, usually you establish a connection and after the job is done you close it.
Since next.js (and probably node.js) is single threaded. Sometimes it happens that there are two requests processed async while one request established the connection to the database, the otherone is closing the exact same connection. So the first request runs into an Topology closed exception. I have the feeling that the mongodb driver client is shared.
Is there something I did not understood correct in this?
try {
await client.connect()
const database = client.db("test")
const collection = database.collection("test")
const newDataset = await collection.insertOne({})
return newDataset.insertedId.toString()
} finally {
await client.close();
}
As in the comments stated, ive seen a lot of examples & questions here on stackoverflow where in each received request (example below) a database connection is established. This has no benefits and is "bad" because it just takes time and makes no sense. E.g:
app.get("/", (req, res) => {
MongoClient.connect("...", (err, client) => {
// do what ever you want here
client.close();
});
});
If you application needs a database connection, establish the connection "in the startup phase" and keep the connection open. There is no reason to open and close the database connection for each request.
const mongodb = require("monogdb");
const express = require("express");
const app = express();
// some custom init stuff
// e.g. require your route handler etc.
mongodb.MongoClient("...", (err, client) => {
// do what ever you want with the db connection now
// e.g. monkey patch it, so you can use it in other files
// (There are better ways to handle that)
mongodb.client = client;
// or the better way
// pass it as function parameter
require("./routes")(app, client);
app.listen(8080, () => {
console.log("http server listening");
});
});
As you can see in the code above, we first create a database connection and then do other stuff. This has some advantages:
If your credentials are invalid, your application is not externeal reachable because the http server is not started
You have a single connection for all requests
Database queries are potential faster because you dont have to wait to establish first a db connection
NOTE: the code above was "inline coded" here and is not tested.
But i think its illustrated the concept behind my statement.
I have a Node.js script which is supposed to regularly access a SailsJS application via a socket connection. Client and server run on physically different machines on different networks. The SailsJS application is proxied behind nginx. That works in general. However, at random times, the connection is established but the first post request within the websocket connection never reaches its destination.
The code looks basically like this:
var socketIOClient = require('socket.io-client');
var sailsIOClient = require('sails.io.js');
var io = sailsIOClient(socketIOClient);
io.sails.url = 'https://foo.foo:443';
io.sails.rejectUnauthorized = false;
io.socket.on('connect', function() {
console.log("Connected!")
io.socket.post('/someroute', { someOptions: "foo" } ,
function(data) {
console.log(data);
});
});
io.socket.on('disconnect', function(){
console.log("Disconnected!");
});
io.socket.on('connect_error',function () {
console.log("connect_error!");
});
In case of a failure, simply nothing happens after console.log("Connected!"). Nothing appears in nginx's logs (in contrast to successful cases), the callback of io.socket.post never gets executed.
The most important question for me is: At which side is the problem? Client or server?
How can I debug this and narrow down the problem? Could it be a networking issue? Or something wrong the configuration, implementation or with the script itself?
I've started writing a node.js websocket solution using socket.io.
The browsers connects to the node server successfully and I get see the socket.id and all config associated with console.log(socket). I also pass a userid back with the initial connection and can see this on the server side to.
Question: I'm not sure the best way to associate a user with a connection. I can see the socket.id changes every page change and when a tab is opened up. How can I track a user and send 'a message' to all required sockets. (Could be one page or could be 3 tabs etc).
I tried to have a look at 'express-socket.io-session' but I'm unsure how to code for it and this situation.
Question: I have 'io' and 'app' variables below. Is it possible to use the 2 together? app.use(io);
Essentially I want to be able to track users (I guess by session - but unsure of how to handle different socket id's for tabs etc) and know how to reply to user or one or more sockets.
thankyou
The best way to handle the situation is rely on SocketIO's rooms. Name the room after the user's unique ID. This will support multiple connections out of the box. Then, whenever you need to communicate with a particular user, simply call the message function and pass in their id, the event, and any relevant data. You don't need to worry about explicitly leaving a room, SocketIO does that for you whenever their session times out or they close their browser tab. (We do explicitly leave a room whenever they log out though obviously)
On the server:
var express = require('express');
var socketio = require('socket.io');
var app = express();
var server = http.createServer(app);
var io = socketio(server);
io.on('connect', function (socket) {
socket.on('userConnected', socket.join); // Client sends userId
socket.on('userDisconnected', socket.leave); // Cliend sends userId
});
// Export this function to be used throughout the server
function message (userId, event, data) {
io.sockets.to(userId).emit(event, data);
}
On the client:
var socket = io('http://localhost:9000'); // Server endpoint
socket.on('connect', connectUser);
socket.on('message', function (data) {
console.log(data);
});
// Call whenever a user logs in or is already authenticated
function connectUser () {
var userId = ... // Retrieve userId somehow
if (!userId) return;
socket.emit('userConnected', userId);
}
// Call whenever a user disconnects
function disconnectUser () {
var userId = ... // Retrieve userId somehow
if (!userId) return;
socket.emit('userDisconnected', userId);
}
I'm building a realtime visualization using redis as pubsub messenger between python and node. There's a python script always running which sets a redis hash with hmset. That side of the app is working fine, if I enter the following example command: "HGETALL 'sellers-80183917'" in a redis client I end up getting the proper data.
The problem is in the js side. I'm using socketio and redis nodejs libraries to listen to the redis instance and publish the results online through a d3js viz.
I run the following code with node:
var express = require('express');
var app = express();
var redis = require('redis');
app.use(express.static(__dirname + '/public'));
var http = require('http').Server(app);
var io = require('socket.io')(http);
var sredis = require('socket.io-redis');
io.adapter(sredis({ host: 'localhost', port: 6379 }));
redisSubscriber = redis.createClient(6379, 'localhost', {});
redisSubscriber.on('message', function(channel, message) {
io.emit(channel, message);
});
app.get('/sellers/:seller_id', function(req, res){
var seller_id = req.params.seller_id;
redisSubscriber.subscribe('sellers-'.concat(seller_id));
res.render( 'seller.ejs', { seller:seller_id } );
});
http.listen(3000, '127.0.0.1', function(){
console.log('listening on *:3000');
});
And this is the relevant part of the seller.ejs file that's receiving the user requests and outputting the viz:
var socket = io('http://localhost:3000');
var stats;
var seller_key = 'sellers-'.concat(<%= seller %>);
socket.on(seller_key, function(msg){
stats = [];
console.log('Im in');
var seller = $.parseJSON(msg);
var items = seller['items'];
for(item in items) {
var item_data = items[item];
stats.push({'title': item_data['title'], 'today_visits': item_data['today_visits'], 'sold_today': item_data['sold_today'], 'conversion_rate': item_data['conversion_rate']});
}
setupData(stats);
});
The problem is that the socket_on() method never receives anything and I don't see where the problem is as everything seems to be working fine besides this.
I think that you might be confused as to what Pub/Sub in Redis actually is. It's not a way to listen to changes on hashes; you can have a Pub/Sub channel called sellers-1, and you can have a hash with the key sellers-1, but those are unrelated to each other.
As documented here:
Pub/Sub has no relation to the key space.
There is a thing called keyspace notifications that can be used to listen to changes in the key space (through Pub/Sub channels); however, this feature isn't enabled by default because it'll take up more resources.
Perhaps an easier method would be to publish a message after the HMSET, so any subscribers would know that the hash got changed (they would then retrieve the hash contents themselves, or the published message would contain the relevant data).
This brings us to the next possible issue: you only have one subscriber connection, redisSubscriber.
From what I understand from the Node.js Redis driver, calling .subscribe() on such a connection would remove any previous subscriptions in favor of the new one. So if you were previously subscribed to the sellers-1 channel and subscribe to sellers-2, you wouldn't be receiving messages from the sellers-1 channel anymore.
You can listen on multiple channels by either passing an array of channels, or by passing them as a arguments:
redisSubscriber.subscribe([ 'sellers-1', 'sellers-2', ... ])
// Or:
redisSubscriber.subscribe('sellers-1', 'sellers-2', ... )
You would obviously have to track each "active" seller subscription. Either that, or create a new connection for each subscription, which also isn't ideal.
It's probably a better idea to have a single Pub/Sub channel on which all changes would get published, instead of a separate channel for each seller.
Finally: if your seller id's aren't hard to guess (for instance, if it's based on an incremental integer value), it would be trivial for someone to write a client that would make it possible to listen in on any seller channel they'd like. It might not be a problem, but it is something to be aware of.
Now, after some hours of playing around with nodejs and socket.io, I'm getting a couple more problems - one being, that I need to get the sessionID inside node.js, whitout using app.get('/' ... - since that event doesnt seem to fire when socket.io connects, it only fires .on('connection', function( ...
var express = require('express')()
express.set('port', process.env.PORT || 8080)
var server = require('http').createServer(express)
var socket = require('socket.io').listen(server)
server.listen(express.get('port'))
// this event is fired, get('/', ... isnt!
server.on('connection', function(stream) {
// ??
} )
The Session is initially created by the PHP application, as the user logs in. Session data is stored in the database, and the key I need to access that data is the SESSION ID. What's the easiest way to get to it? Like mentioned, I found a couple examples that used app.get('/' ... but I couldnt get that event to fire at all.
Thanks.
If the session data is being stored as a cookie (most likely), then you should be able to re-parse that data during the socket handshake. I posted code for that on this answer, but copied the code here as well:
io.configure(function () {
io.set('authorization', function (handshakeData, callback) {
var cookie = handshakeData.headers.cookie;
// parse the cookie to get user data...
// second argument to the callback decides whether to authorize the client
callback(null, true);
});
});
If the session data is being propagated in the URL, then you may be able to gather this information from handshakeData.url or handshakeData.query. You'll probably have to do your own experimentation.