I recently learned node.js, and I'm having fun rewriting perl scripts into non-blocking scripts. I am currently writing a script to go out and connect to some hosts and pull some data. I'm pulling the connection data from a database, and would like to limit it to x connections (probably 50) at a time, with new connections starting up when one is complete.
Here's the code I'm playing with:
var net = require('net');
var mysql = require('mysql');
var mysql_conn = mysql.createConnection({
//connection info...
});
var connect_hash ={};
function make_connection (id, ip, port, username, password, cb) {
var conn = net.createConnection(port, ip);
var completeData = '';
connect_hash[id] = {};
connect_hash.id.ip = ip;
// stuff happens here..
}
mysql_conn.query("select id, IP, Port, Username, Password from Hosts",
function (err, ne_records, fields){
if (err) throw err;
ne_records.forEach(function(host){
make_connection(host.id, host.IP, host.Port, host.Username, host.Password, function(attempt) {
delete connection_hash[id];
db_save (attempt);
});
});
});
As written right now, it'll just open connections to every host in the table and do stuff on them. I'd really like it to open a specified number at a time, and only start new connections when the old ones are done and deleted out of connection_hash. Is there a way to do this?
This is simple – keep a queue of hosts to be connected to, and only open n connections at a time. When a connection completes (or fails), start the next one in the queue.
You might want to take a look at how the HTTP Agent class is implemented. It does what you're trying to accomplish for HTTP requests.
Related
I'm new to websockets, and am wondering how best to go about this.
My scenario: I have a server that handles different classes of users. For this example, let's say the classes are "mice", "cats", and "dogs"
Each of those classes should have their own channels to listen to for changes e.g. "mice-feed", "cat-feed", and "dog-feed"
My question is: after the server authenticates and determines the class of the current user, what's the best way to have them subscribed to a specific channel, or channel(s), so that when I broadcast messages to said channel(s), I can make sure that only members of particular classes get them (as against everyone currently connected to that server)?
My current code setup looks like this:
var ws = require('ws');
var redis = require('redis');
/* LOCATION 1 */
// prep redis, for websocket channels
var pub = redis.createClient();
var sub = redis.createClient();
// subscribe to our channels
sub.subscribe('mice-feed');
sub.subscribe('cat-feed');
sub.subscribe('dog-feed');
// declare the server
const wsServer = new ws.Server({
noServer: true,
path: "/",
});
/* ... removing some code for brevity... */
wsServer.on("connection", function connection(websocketConnection, connectionRequest) {
/* LOCATION 2 */
});
Do I put the redis declarations in LOCATION 1 (where it currently is), or in LOCATION 2 (when a successful connection is established)? Or neither of the above?
(also: I know it's possible to do this on the websocket end directly i.e. iterate through every client and ws.send if some criterion is matched, but iteration can become costly, and I'm wondering if I can do it on a redis-channel wide operation instead)
If I were building this, my first approach would be this:
// connect to Redis
const client = createClient();
client.on('error', (err) => console.log('Redis Client Error', err));
await client.connect();
// declare the server
const wsServer = new ws.Server(...elided...);
// handle connection
wsServer.on('connection', async (websocketConnection, connectionRequest) => {
const sub = client.duplicate()
// figure out the feed
const feed = 'animal-feed';
await sub.subscribe(feed, message => {
...do stuff...
});
});
It's pretty straightforward but would result in ever user having a dedicated connect to Redis. That may or may not matter depending on how many users you anticipate having.
I googled a lot but still have no clear solution to my issue.
Connecting to MongoDB, usually you establish a connection and after the job is done you close it.
Since next.js (and probably node.js) is single threaded. Sometimes it happens that there are two requests processed async while one request established the connection to the database, the otherone is closing the exact same connection. So the first request runs into an Topology closed exception. I have the feeling that the mongodb driver client is shared.
Is there something I did not understood correct in this?
try {
await client.connect()
const database = client.db("test")
const collection = database.collection("test")
const newDataset = await collection.insertOne({})
return newDataset.insertedId.toString()
} finally {
await client.close();
}
As in the comments stated, ive seen a lot of examples & questions here on stackoverflow where in each received request (example below) a database connection is established. This has no benefits and is "bad" because it just takes time and makes no sense. E.g:
app.get("/", (req, res) => {
MongoClient.connect("...", (err, client) => {
// do what ever you want here
client.close();
});
});
If you application needs a database connection, establish the connection "in the startup phase" and keep the connection open. There is no reason to open and close the database connection for each request.
const mongodb = require("monogdb");
const express = require("express");
const app = express();
// some custom init stuff
// e.g. require your route handler etc.
mongodb.MongoClient("...", (err, client) => {
// do what ever you want with the db connection now
// e.g. monkey patch it, so you can use it in other files
// (There are better ways to handle that)
mongodb.client = client;
// or the better way
// pass it as function parameter
require("./routes")(app, client);
app.listen(8080, () => {
console.log("http server listening");
});
});
As you can see in the code above, we first create a database connection and then do other stuff. This has some advantages:
If your credentials are invalid, your application is not externeal reachable because the http server is not started
You have a single connection for all requests
Database queries are potential faster because you dont have to wait to establish first a db connection
NOTE: the code above was "inline coded" here and is not tested.
But i think its illustrated the concept behind my statement.
I'm building a realtime visualization using redis as pubsub messenger between python and node. There's a python script always running which sets a redis hash with hmset. That side of the app is working fine, if I enter the following example command: "HGETALL 'sellers-80183917'" in a redis client I end up getting the proper data.
The problem is in the js side. I'm using socketio and redis nodejs libraries to listen to the redis instance and publish the results online through a d3js viz.
I run the following code with node:
var express = require('express');
var app = express();
var redis = require('redis');
app.use(express.static(__dirname + '/public'));
var http = require('http').Server(app);
var io = require('socket.io')(http);
var sredis = require('socket.io-redis');
io.adapter(sredis({ host: 'localhost', port: 6379 }));
redisSubscriber = redis.createClient(6379, 'localhost', {});
redisSubscriber.on('message', function(channel, message) {
io.emit(channel, message);
});
app.get('/sellers/:seller_id', function(req, res){
var seller_id = req.params.seller_id;
redisSubscriber.subscribe('sellers-'.concat(seller_id));
res.render( 'seller.ejs', { seller:seller_id } );
});
http.listen(3000, '127.0.0.1', function(){
console.log('listening on *:3000');
});
And this is the relevant part of the seller.ejs file that's receiving the user requests and outputting the viz:
var socket = io('http://localhost:3000');
var stats;
var seller_key = 'sellers-'.concat(<%= seller %>);
socket.on(seller_key, function(msg){
stats = [];
console.log('Im in');
var seller = $.parseJSON(msg);
var items = seller['items'];
for(item in items) {
var item_data = items[item];
stats.push({'title': item_data['title'], 'today_visits': item_data['today_visits'], 'sold_today': item_data['sold_today'], 'conversion_rate': item_data['conversion_rate']});
}
setupData(stats);
});
The problem is that the socket_on() method never receives anything and I don't see where the problem is as everything seems to be working fine besides this.
I think that you might be confused as to what Pub/Sub in Redis actually is. It's not a way to listen to changes on hashes; you can have a Pub/Sub channel called sellers-1, and you can have a hash with the key sellers-1, but those are unrelated to each other.
As documented here:
Pub/Sub has no relation to the key space.
There is a thing called keyspace notifications that can be used to listen to changes in the key space (through Pub/Sub channels); however, this feature isn't enabled by default because it'll take up more resources.
Perhaps an easier method would be to publish a message after the HMSET, so any subscribers would know that the hash got changed (they would then retrieve the hash contents themselves, or the published message would contain the relevant data).
This brings us to the next possible issue: you only have one subscriber connection, redisSubscriber.
From what I understand from the Node.js Redis driver, calling .subscribe() on such a connection would remove any previous subscriptions in favor of the new one. So if you were previously subscribed to the sellers-1 channel and subscribe to sellers-2, you wouldn't be receiving messages from the sellers-1 channel anymore.
You can listen on multiple channels by either passing an array of channels, or by passing them as a arguments:
redisSubscriber.subscribe([ 'sellers-1', 'sellers-2', ... ])
// Or:
redisSubscriber.subscribe('sellers-1', 'sellers-2', ... )
You would obviously have to track each "active" seller subscription. Either that, or create a new connection for each subscription, which also isn't ideal.
It's probably a better idea to have a single Pub/Sub channel on which all changes would get published, instead of a separate channel for each seller.
Finally: if your seller id's aren't hard to guess (for instance, if it's based on an incremental integer value), it would be trivial for someone to write a client that would make it possible to listen in on any seller channel they'd like. It might not be a problem, but it is something to be aware of.
I'm using Redis in my nodejs server, communicating with a browser html/js application. Under conditions I can't identify, my server gets stuck. It throws no errors, keep to be alive and responding, but can't get values from Redis. This is (the simplified version) how the connection with Redis is established.
var Stash = require('redis'),
Sockjs = require('sockjs')
stash = Stash.createClient(some_port,some_host);
sockjs = Sockjs.createServer();
var Instances = {
get: function (id, cb) {
stash.get('something_' + id, cb)
}
sockjs = Sockjs.createServer();
sockjs.on('connection', function(socket) {
socket.on('data', function(data) {
var id = somecorrectid;
Instances.get(id, function(err, instance) { /*Here the magic happens*/ }
}
}
Now, this works perfectly in normal conditions.
When the issue happens, the code between the last brackets isn't executed anymore, no matter how many times I fire again the event triggering the piece of code (that is a post data throught a socket) and from what client. Redis, in the meanwhile, is alive and rocking. Redis-cli is responding. The key searched with the 'get' exist and it is correct in its form. My server is still able to write keys to Redis correctly, but seems it can't read them anymore. This have no sense to me, and it is difficult to investigate or have more informations for I can't get how to replicate the issue. Any clue?
I'm looking to build a node app that will accomplish the following:
Open several websocket connections, almost as if each of them were a
thread
Allow each websocket to have a unique/dynamic URL
Create a pool of websocket connections in an object based off some kind of DB query (so I can dynamically add/remove connections)
I've decided to use the ws library (https://github.com/websockets/ws) since its the fastest and least bloated option available. I currently have the following function, which only supports a single ws connection:
chat.prototype.connect = function() {
var self = this;
self.ws = new ws(url);
self.ws.on('message', function(data, flags) {
var message = JSON.parse(data);
self.handle(message);
});
};
This code listens to a single websocket URL and passes the message(s) to my handler to process the message. Instead, I want to make this function listen to multiple (potentially hundreds) of websocket URL's.
Does anyone have some ideas on how to accomplish this?
Say that you have the list of url's you need to connect to stored in an instance property called urls. You could set up the connections like this:
chat.prototype.connect = function() {
urls.forEach(this.connectOne.bind(this));
};
chat.prototype.connectOne = function(url) {
var handle = this.handle.bind(this);
var conn = this.connections[url] = new ws(url);
conn.on('message', function(data, flags) {
var message = JSON.parse(data);
handle(message);
});
};
To implement adding new connections, periodically query your database and check if each URL is already present in this.connections; if not, you can use this.connectOne() to add it. You'd do something similar to remove a connection.