I'm looking into building some signalR applications in .NET hosted in Azure (Self hosting workers).
I want to scaleOut and setup a Backplane using Azure Redis, however when i've gone to setup a new Redis Cache i've got confused to what the 'Up to X connections' actually means.
For example, the 'CO Basic 250MB Cache' has 'Up to 256 connections' and the 'C1 Standard 1GB Cache' has 'Up to 1,000 connections'
To confirm, can i take 'Up to 256 connections' to mean that i could (In theory) have up to 256 worker threads all pushing SignalR messages around at once ... Or does this mean the total amount of connections (users) from my website that are connected to my SignalR and in turn, pushing messages around the Redis Cache?
Obviously, if it means 256 workers thats fine - But if it means the total number of different connections from my website, then that is a deal breaker
Thanks and sorry if this is a silly question!
From a SignalR backplane perspective, the SignalR websocket connections do not correlate with the number of connections to a Redis Cache server.
A user's SignalR connection is with the SignalR Hub server, which in turn acts as a Redis client in case of a scaleout.
The Redis client in signalr connects using the standard ConnectionMultiplexer which handles the connections to redis internally. And the guidance is to use a single multiplexer for the whole application, or a minimal number.
The Redis client is to send/receive messages and not create/access keys for each operation, so it makes sense to have a single channel open and to have all the messages being exchanged on that single channel.
I am not sure how that connectionmultiplexer manages Redis connections, but we do use Redis backplane for SignalR scaleout on Azure for our application.
We load tested the app and with around 200 thousand always active Signalr websocket connections scaled out over 10 servers, the Azure Redis cache connections count hovered around 50 on average, almost never going above even 60..
I think it is safe to say that the connection limits on Azure Redis cache are not a limiting factor for SignalR, unless you scale out to hundreds of servers.
The connection limit is not about the number of worker threads that are reading from or writing to Redis. It is about physical TCP connections. Redis supports pipelining. Many of the client libraries are thread safe so that you can use the same physical connection object to write from multiple threads at the same time.
Related
Can somebody explains how the connections are calculated in Azure Redis Cache?
For example if I have an MVC app and I use a Basic Redis Cache with 256 connections, and I have 256 users accessing my websites will there be 256 connections made? How exactly does this work?
How many connections are made depends on the application you implement.
If you follow best practices, your application will be able to handle many users with a very low amount of connections.
E.g. Stackexchange.Redis should be able to handle thousands of users without exhausting your 256 connections if you reuse the connection multiplexer object.
Some more information:
https://gist.github.com/JonCole/925630df72be1351b21440625ff2671f
https://stackexchange.github.io/StackExchange.Redis/Basics
the key idea in StackExchange.Redis is that it aggressively shares the connection between concurrent callers
I am working on a nodejs application and the requirement is to send around 10k requests per second per connection. The client application has to open one websocket connection to send these requets and at the server side it has to just receive and send the data to a queue. The number of socket connections at the server side isn't that much, may be around 1k. I have few questions regarding this and any help is greatly appreciated.
First, is it possible to achieve this setup with a single master process? Since I cannot share the web socket connections with the child processes I need to get the bandwith from master process.
When I tried benchmarking nodejs ws library, I was only able to send approximately 1k requests per second of 9kb each. How can I increase the throughput?
Are there any examples on how to achieve max throughput since I can only find posts with how to achieve max connections?
Thanks.
You will eventually need to scale
First, is it possible to achieve this setup with a single master process?
I don't really think its possible to achieve this with a single thread.
(you should consider scaling and never design restricting yourself from that option)
Since I cannot share the web socket connections with the
child processes I need to get the bandwith from master process.
Im sure you will be happy to know about the existance of socket.io-redis.
With it you will be able to send/receive events (share clients) between multiple instances of your code (processes or servers). Read more : socket.io-redis (Github)
I know you are speaking about ws but maybe its worth the change to socket.io. Image Source Article
Specially knowing you can scale both vertically (increase the number of threads per machine)
and horizontally (deploy more "masters" accross more machines) with relative ease. (and I repeat myself : sharing your socket clients & communications accross all instances)
When I tried benchmarking nodejs ws library, I was only able to send
approximately 1k requests per second of 9kb each. How can I increase
the throughput?
I would suggest trying socket.io + socket.io-redis
spawn a master with a number of workers equal to the number of CPU
cores. (vertical scaling)
deploy your master accross 2 or more machines (horizontal scaling)
learn about load-balancing & perform benchmarks.
Are there any examples on how to achieve max throughput since I can
only find posts with how to achieve max connections?
You will increase the total throughput if you increase the number of instances communicating with clients. (Horizontal + Vertical Scaling)
socket.io-redis Application Example (github)
Using Redis with Node.js and Socket.IO (tutorial)
This might also be an interesting read :
SocketCluster 100k messages / sec (ycombinator)
SocketCluster (github)
Hope it helped.
I feel like this question would have been asked before, but I can't find one. Pardon me if this is a repeat.
I'm building a service on Node.js hosted in Heroku and using MongoDB hosted by Compose. Under heavy load, the latency is most likely to come from the database, as there is nothing very CPU-heavy in the service layer. Thus, when MongoDB is overloaded, I want to return an HTTP 503 promptly instead of waiting for a timeout.
I'm also using REDIS, and REDIS has a feature where you can check the number of queued commands (redisClient.command_queue.length). With this feature, I can know right away if REDIS is backed up. Is there something similar for MongoDB?
The best option I have found so far is polling the server for status via this command, but (1) I'm hoping for something client side, as there could be spikes within the polling interval that cause problems, and (2) I'm not actually sure what part of the status response I want to act on. That second part brings me to a follow up question...
I don't fully understand how the MondoDB client works with the server. Is one connection shared per client instance (and in my case, per process)? Are queries and writes queued locally or on the server? Or, is one connection opened for each query/write, until the database's connection pool is exhausted? If the latter is the case, it seems like I might want to keep an eye on the open connections. Does the MongoDB server return such information at other times, besides when polled for status?
Thanks!
MongoDB connection pool workflow-
Every MongoClient instance has a built-in connection pool. The client opens sockets on demand to support the number of concurrent MongoDB operations your application requires. There is no thread-affinity for sockets.
The client instance, opens one additional socket per server in your MongoDB topology for monitoring the server’s state.
The size of each connection pool is capped at maxPoolSize, which defaults to 100.
When a thread in your application begins an operation on MongoDB, if all other sockets are in use and the pool has reached its maximum, the thread pauses, waiting for a socket to be returned to the pool by another thread.
You can increase maxPoolSize:
client = MongoClient(host, port, maxPoolSize=200)
By default, any number of threads are allowed to wait for sockets to become available, and they can wait any length of time. Override waitQueueMultiple to cap the number of waiting threads. E.g., to keep the number of waiters less than or equal to 500:
client = MongoClient(host, port, maxPoolSize=50, waitQueueMultiple=10)
Once the pool reaches its max size, additional threads are allowed to wait indefinitely for sockets to become available, unless you set waitQueueTimeoutMS:
client = MongoClient(host, port, waitQueueTimeoutMS=100)
Reference for connection pooling-
http://blog.mongolab.com/2013/11/deep-dive-into-connection-pooling/
I've recently read a lot about best practices with JMS, Spring (and TIBCO EMS) around connections, sessions, consumers & producers
When working within the Spring world, the prevailing wisdom seems to be
for consuming/incoming flows - to use an AbstractMessageListenerContainer with a number of consumers/threads.
for producing/publishing flows - to use a CachingConnectionFactory underneath a JmsTemplate to maintain a single connection to the broker and then cache sessions and producers.
For producing/publishing, this is what my (largeish) server application is now doing, where previously it was creating a new connection/session/producer for every single message it was publishing (bad!) due to use of the raw connection factory under JmsTemplate. The old behaviour would sometimes lead to 1,000s of connections being created and closed on the broker in a short period of time in high peak periods and even hitting socket/file handle limits as a result.
However, when switching to this model I am having trouble understanding what the performance limitations/considerations are with the use of a single TCP connection to the broker. I understand that the JMS provider is expected to ensure it can be used in the multi-threaded way etc - but from a practical perspective
it's just a single TCP connection
the JMS provider to some degree needs to co-ordinate writes down the pipe so they don't end up an interleaved jumble, even if it has some chunking in its internal protocol
surely this involves some contention between threads/sessions using the single connection
with certain network semantics (high latency to broker? unstable throughput?) surely a single connection will not be ideal?
On the assumption that I'm somewhat on the right track
Am I off base here and misunderstanding how the underlying connections work and are shared by a JMS provider?
is any contention a problem mitigated by having more connections or does it just move the contention to the broker?
Does anyone have any practical experience of hitting such a limit they could share? Either with particular message or network throughput, or even caused by # of threads/sessions sharing a connection in parallel
Should one be concerned in a single-connection scenario about sessions that write very large messages blocking other sessions that write small messages?
Would appreciate any thoughts or pointers to more reading on the subject or experience even with other brokers.
When thinking about the bottleneck, keep in mind two facts:
TCP is a streaming protocol, almost all JMS providers use a TCP based protocol
lots of the actions from TIBCO EMS client to EMS server are in the form of request/reply. For example, when you publish a message / acknowledge a receive message / commit a transactional session, what's happening under the hood is that some TCP packets are sent out from client and the server will respond with some packets as well. Because of the nature of TCP streaming, those actions have to be serialised if they are initiated from the same connection -- otherwise say if from one thread you publish a message and in the exact same time from another thread you commit a session, the packets will be mixed on the wire and there is no way server can interpret the right message from the packets. [ Note: the synchronisation is done from the EMS client library level, hence user can feel free to share one connection with multiple threads/sessions/consumers/producers ]
My own experience is multiple connections always output perform single connection. In a lossy network situation, it is definitely a must to use multiple connections. Under best network condition, with multiple connections, a single client can nearly saturate the network bandwidth between client and server.
That said, it really depends on what is your clients' performance requirement, a single connection under good network can already provides good enough performance.
Even if you use one connection and 100 sessions it means finally you
are using 100threads, it is same as using 10connections* 10 sessions =
100threads.
You are good until you reach your system resource limits
Trying to build a TCP server using Spring Integration in which keeps connections may run into thousands at any point in time. Key concerns are regarding
Max no. of concurrent client connections that can be managed as session would be live for a long period of time.
What is advise in case connections exceed limit specified in (1).
Something along the lines of a cluster of servers would be helpful.
There's no mechanism to limit the number of connections allowed. You can, however, limit the workload by using fixed thread pools. You could also use an ApplicationListener to get TcpConnectionOpenEvents and immediately close the socket if your limit is exceeded (perhaps sending some error to the client first).
Of course you can have a cluster, together with some kind of load balancer.