I am creating a Node.js application through Express.js foundation and my knowledge lead me to create a models folder and a file for each model. Ok, so far, so good — but the point is: each model of mine has the following fragment of code on its top:
var mongoose = require ('mongoose')
, database = mongoose.createConnection('mongodb://localhost:27017/test');
Am I repeating myself doing this? I'm not searching for "the best approach ever", but doing this is not a bad design thing?
If this is something normal, could someone explain me why this behavior happens? Couldn't I create an organic mechanism to establish a connection and just distribute its properties for the entire application?
I already worked on Rails, C# and PHP and I've never see something like this before. Probably because database connections were for me something very abstract — I touched just a few times on the connection bridge.
DISCLAIMER: My question is really very specific. I just want to know if there's a way to announce the database connection from one file to the rest of the application.
What you're doing is not typical.
Unless you have a specific need to do something else, call mongoose.connect once during app startup and register all models using mongoose.model.
This lets your app share a common connection pool. You can tweak the size of the connection pool via the options provided in the mongoose.connect call.
Related
We have architecture problem on our project. This project requires sharding, as soon as we need almost unlimited scalability for the part of services.
Сurrently we use Node.js + MongoDb (Mongoose) and MySQL (TypeORM). Data is separated by databases through the simple 'DB Locator'. So node process needs connections to a lot of DBs (up to 1000).
Requests example:
HTTP request from client with Shop ID;
Get DB IP address/credentials in 'DB Locator' service by Shop ID;
Create connection to specific database with shop data;
Perform db queries.
We tried to implement it in two ways:
Create connection for each request, close it on response.
Problems:
we can't use connection after response (it's the main problem, because sometimes we need some asynchronous actions);
it works slower;
Keep all connections opened.
Problems:
reach simultaneous connections limit or some another limits;
memory leaks.
Which way is better? How to avoid described problems? Maybe there is a better solution?
Solution #1 perfectly worked for us on php as it runs single process on request and easily drops connections on process end. As we know, Express is pure JS code running in v8 and is not process based.
It would be great to close non-used connections automatically but can't find options to do that.
The short answer: stop using of MongoDB with Mongoose 😏
Longer answer:
MongoDB is document-oriented DBMS. The main usage case is when you have some not pretty structured data that you have to store, but you don't need to use too much. There is lazy indexing, dynamic typing and many more things that not allow you to use it as RDBMS, but it is great as a storage of logs or any serialized data.
The worth part here is Mongoose. This is the library that makes you feel like your trashbox is wonderful world with relations, virtual fields and many things that should not to be in DODBMS. Also, there is a lot of legacy code from previous versions that also make some troubles with connections management.
You already use TypeORM that may works instead Mongoose. With some restrictions, for sure.
It works exactly same way as MySQL connection management.
Here is some more data: https://github.com/typeorm/typeorm/blob/master/docs/mongodb.md#defining-entities-and-columns
In this case you may use you TypeORM Repository as transparent client that will init connections and close it or keep it alive on demand.
My application is using Net API (https://nodejs.org/api/net.html) sockets and accept connections from clients that cannot use any other way to communicate than raw TCP sockets.
I'm trying to use pm2 with my app and clusterize it on all my cpus. Unfortunately this does not work since one process cannot use the in-memory stored sockets from any of the other processes.
I'm looking for a way to save each connections to a database, even an in-memory store would fit my needs for as long as i can call it and use it again from any other process.
Somebody else asked pretty much the same question here he was told that he should just use Redis to store the sockets. But the guy who answered apparently had no idea how to do that.
Question was asked here: Will PM2 work with Node.js net API?
My question is how could i do that ?
Actually i think this is just impossible. What i really need is a way to "recreate" the socket object through it's file descriptor number that could be save as an integer in the database, but again, this is getting really tricky. I'm kind of stuck here. Documentation says nothing about it.
Maybe there's another way to keep my app stateless ?
Thanks a lot for reading, i'll be glad if someone can help.
There are no methods to save a special type that is a resource.
In the database you can save the way how to re-created it (re-opene).
But I think that in your situation, a better solution will be to add an extra layer that will provide communication with a descriptor and all elements of the cluster. In the option you have: exchange events between node process or WebSockets.
What you can do is sharing file descriptors by passing them between the server processes using a Unix domain socket. Have a look at the usocket package which implements passing file descriptors for Node.js.
I'm trying to create a Flash app with some real time functionality, and would like to use Redis' pubsub functionality which is a perfect fit for what I need.
I know that connecting to a data store directly from client is almost always bad. What are the security implications of this (since I'm not an expert on Redis), and are there ways to work around them? From what I read, there is a possible exploit of doing config sets and changing the rdb file location and be able to arbitrary overwrite files. Is there anything else? (If I don't use that particular redis instance for anything at all, i.e. no data being stored)
I understand the alternative is to write some custom socket server program and have it act as the mediating layer for connecting to redis and issuing commands -- that's the work I'd like to avoid having to write, if possible.
** Edit **
Just learned about the rename-command configuration to disable commands. If I disable every single command on the redis instance and leave only SUBSCRIBE and PUBLISH open, would this be good enough to run on production?
I think it would be a bad idea to connect directly your client to Redis. Redis offers an authentication system for a unique user only. It expects this user to be your server app.
From my point of view, directly exposing Redis is always a bad idea. It would allow anybody to access all of your data. This is confirmed by the Redis doc.
So you won't avoid adding or developing the server side of your app.
I've got problem connecting Flash client to Node.js server.
Short story:
For a first time I'm building a Node.js server that should be used by both web client (WebSocket) as well as a Flash client (Socket). The web client, of course, works like a charm, but I can't get over the Flash one. I get SECURITY_ERROR. After a day of research I think it's because of the policy file not being loaded. Ideas (primus on top of engine.io) ?
Long story:
I'm using Primus as I thought I'll need it because I have both web sockets and flash sockets to handle. Not sure if this is accurate? :)
I'm using Engine.io as a 'transformer/transporter' - the main framework that the layer uses. I won't discuss the standard web client (using Chrome and primus-client), as it's easy to setup.
I'm using simple and standard Sockets in AS3:
_socket = new Socket();
_socket.addEventListener(Event.CONNECT, onSocketConnect);
//...
_socket.addEventListener(SecurityErrorEvent.SECURITY_ERROR, onSecurityError);
_socket.addEventListener(IOErrorEvent.IO_ERROR, onIOError);
_socket.connect('localhost', '1337);
When building it within Flash IDE, it goes to the onSocketConnect function, but if I try to write anything to the socked - I get disconnected. If I run this from the web browser, I get into the onSecurityError method.
I must say that I don't get any traces in the node console!
primus.on('connection', function connection(spark) {
console.log('new connection'); // never gets logged!
As I know, security error is thrown when there is error with the policy file, so I started searching for a solution for that.
I've read a lot of things online, and most common solution was simple usage of socket.io and so called FlashSocket.IO. I tried implementing it, but it's so old, that some of the code is a kind of missing and I finally got some errors from the hurlant library - I couldn't get it working.
I also saw some node package called policy, which runs separate server to server the policy file.
I tried adding a transport array with flashsocket in it - no change. I also can't understand why all of the samples are using transports - I've searched and both index.js and primus.js are using transport (why there are two separate files, Jesus?!)
I could try using only engine.io without primus, but I don't know if this would be of any help. All the posts and samples I've found are pretty old - please help me with any up to date solution or at least some explanation what needs to be done - seems like a whole new universe to me :)
Thanks in advance!
Edit:
Thanks to the The_asMan, I figured out it has something to do with the handshake. I've tried this simple example (despite the fact it's so old) - it worked perfectly for the Flash client! Of course I cannot connect web sockets to it, as the handshake is not proper - it has some kind of protocol for it.
So I guess I just have to understand how to get the <policy-file-request/> in node - I'll be able to return the policy file. But I don't know how to get it - I don't receive any kind of data nor connect handler...
You have a cross domain policy issue.
I answered it all here.
AS3 - Flash/AIR Socket Communication writeUTFBytes only works once
just an idea:
On some operating systems, flush() is called automatically between execution frames, but on other operating systems, such as Windows, the data is never sent unless you call flush() explicitly. To ensure your application behaves reliably across all operating systems, it is a good practice to call the flush() method after writing each message (or related group of data) to the socket.
Brief Description:
Well, since many days I've been looking for an answer to this question but there seems to be answers for 'How to create a Push Notification Server' and like questions. I am using node.js and it's quite easy to 'create' a push notification server using sock.js (I've heard socket.io isn't good as compared to sock.js). No problem till here. But what I want is how to model such a server.
Details:
OK, so, let's say I've an application where there's a chat service (just an example this is, actual thing is big as you might have guessed). A person sends a message in a room and all the people in the room get notified. But what I want is a 'stateful' chat - that is, I want to store the messages in a data store. Here's where the trouble comes. Storing the message in the database and later telling everyone that "Hey, there's a message for you". This seems easy when we need the real-time activity for just one part of the app. What to do when the whole app is based on real-time communication? Besides this, I also want to have a RESTful api.
My solution (with which I am not really happy)
What I thought of doing was this: (on the server side of course)
Data Store
||
Data Layer (which talks to data store)
||
------------------
| |
Real-Time Server Restful server
And here, the Real-time server listens to interesting events that the data-layer publishes. Whenever something interesting happens, the server notifies the client. But which client? - This is the problem with my method
Hope you can be of help. :)
UPDATE:
I think I forgot to emphasize an important part of my question. How to implement a pub-sub system? (NOTE: I don't want the actual code, I'll manage that myself; just how to go about doing it is where I need some help). The problem is that I get quite boggled when writing the code - what to do how (my confusion is quite apparent from this question itself). Could please provide some references to read or some advice as to how to begin with this thing?
I am not sure if I understood you correctly; but I will summarize how I read it:
We have a real-time chat server that uses socket connections to publish new messages to all connected clients.
We have a database where we want to keep chat logs.
We have also a restful interface to access the realtime server to get current chats in a lazier manner.
And you want to architect your system this way:
In the above diagram, the components I circled with purple curve wants to be updated like all other clients. Am I right? I don't know what you meant with "Data Layer" but I thought it is a daemon that will be writing to database and also interfacing database for other components.
In this architecture, everything is okay in the direction you meant. I mean DataStore is connected by servers to access data, maybe to query client credentials for authentication, maybe to read user preferences etc.
For your other expectation from these components, I mean to allow these components to be updated like connected clients, why don't you allow them to be clients, too?
Your realtime server is a server for clients; but it is also a client for data layer, or database server, if we prefer a more common naming. So we already know that there is nothing that stops a server from being a client. Then, why can't our database system and restful system also be clients? Connect them to realtime server the same way you connect browsers and other clients. Let them enjoy being one of the people. :)
I hope I did not understand everything completely wrong and this makes sense for the question.