as I seen topic which not recommending more than 200 threads for server machine,
I am trying to implement Listener class which listens 1000 devices, I mean 1000 devices sending different type of messages to that application,
I tried 2 different way 1. create thread for each device runtime and dynamic list which hold the messages for that device and start thread for processing those messages from the list
but my machine not creating thread more than 50 :), and I agree its bad idea...
I created 10 different lists which holds the messages 10 different type of messages
and I created 10 processor thread for those list, which go to its relevant list and process the message and then delete it.
but here is the problem, let say I received 50 messages from 50 devices in List 1
by the time its list1's processor thread will go to last message (50th) its time will be expired which is 10 second
any idea to for best architecture that talk to more than 500 devices and process their different type of messages with in 10 seconds.
I am working in C#, my application connected with the server as a client using tcp/ip,
that server further connects with online devices, sending messages to server with device id and message data and message typ thn further I receiving messages from that server and then reply back through that server using device id,
I think you need to partition the system differently. The listeners should be high priority but only enqueue the requests. The queue should then be processed using a pool of workers. You could add prioritisation and other optimisations on the dequeuing side. In terms of getting every process done in 10s you will really be getting the second half of the system optimised.
Think of the traditional queuing system. You have a queue of work requests to process. Each request has a series of attributes. Lets same Name (string) and Priority (int). Once a the work request has been queued, other worker (thread/processes etc) can interrogate the queue to pull out items based on priority and process them.
To get the 10s I'd say as soon a worker has started processing the request a timer comes in to play and will mark that request as timed out in 10s unless the worker completes the task. Other workers can watch for results of the work in the queue and then handle the response behaviours.
use other Highly-concurrent programming models other than threaded, though threaded is one of the highly-concurrent models too.
if socket/tcpip/network messaging, please use epoll on Linux 2.6x and completion port on win/msvc.
see the docoument named EffoNetMsg.pdf at http://code.google.com/p/effonetmsg/downloads/list to learn more about highly-concurrent programming models. we only use 2 or 3 threads for multi-listeners and >1000 clients.
Related
Context:
We have micro service which consumes(subscribes)messages from 50+ RabbitMQ queues.
Producing message for this queue happens in two places
The application process when encounter short delayed execution business logic ( like send emails OR notify another service), the application directly sends the message to exchange ( which in turn it is sent to the queue ).
When we encounter long/delayed execution business logic We have messages table which has entries of messages which has to be executed after some time.
Now we have cron worker which runs every 10 mins which scans the messages table and pushes the messages to RabbitMQ.
Scenario:
Let's say the messages table has 10,000 messages which will be queued in next cron run,
9.00 AM - Cron worker runs and it queues 10,000 messages to RabbitMQ queue.
We do have subscribers which are listening to the queue and start consuming the messages, but due to some issue in the system or 3rd party response time delay it takes each message to complete 1 Min.
9.10 AM - Now cron worker once again runs next 10 Mins and see there are yet 9000+ messages yet to get completed and time is also crossed so once again it pushes 9000+ duplicates messages to Queue.
Note: The subscribers which consumes the messages are idempotent, so there is no issue in duplicate processing
Design Idea I had in my mind but not best logic
I can have 4 status ( RequiresQueuing, Queued, Completed, Failed )
Whenever a message is inserted i can set the status to RequiresQueuing
Next when cron worker picks and pushes the messages successfully to Queue i can set it to Queued
When subscribers completes it mark the queue status as Completed / Failed.
There is an issue with above logic, let's say RabbitMQ somehow goes down OR in some use we have purge the queue for maintenance.
Now the messages which are marked as Queued is in wrong state, because they have to be once again identified and status needs to be changed manually.
Another Example
Let say I have RabbitMQ Queue named ( events )
This events queue has 5 subscribers, each subscribers gets 1 message from the queue and post this event using REST API to another micro service ( event-aggregator ). Each API Call usually takes 50ms.
Use Case:
Due to high load the numbers events produced becomes 3x.
Also the micro service ( event-aggregator ) which accepts the event also became slow in processing, the response time increased from 50ms to 1 Min.
Cron workers follows your design mentioned above and queues the message for each min. Now the queue is becoming too large, but i cannot also increase the number of subscribers because the dependent micro service ( event-aggregator ) is also lagging.
Now the question is, If keep sending the messages to events queue, it is just bloating the queue.
https://www.rabbitmq.com/memory.html - While reading this page, i found out that rabbitmq won't even accept the connection if it reaches high watermark fraction (default is 40%). Of course this can be changed, but this requires manual intervention.
So if the queue length increases it affects the rabbitmq memory, that is reason i thought of throttling at producer level.
Questions
How can i throttle my cron worker to skip that particular run or somehow inspect the queue and identify it already being heavily loaded so don't push the messages ?
How can i handle the use cases i said above ? Is there design which solves my problem ? Is anyone faced the same issue ?
Thanks in advance.
Answer
Check the accepted answer Comments for the throttling using queueCount
You can combine QoS - (Quality of service) and Manual ACK to get around this problem.
Your exact scenario is documented in https://www.rabbitmq.com/tutorials/tutorial-two-python.html. This example is for python, you can refer other examples as well.
Let says you have 1 publisher and 5 worker scripts. Lets say these read from the same queue. Each worker script takes 1 min to process a message. You can set QoS at channel level. If you set it to 1, then in this case each worker script will be allocated only 1 message. So we are processing 5 messages at a time. No new messages will be delivered until one of the 5 worker scripts does a MANUAL ACK.
If you want to increase the throughput of message processing, you can increase the worker nodes count.
The idea of updating the tables based on message status is not a good option, DB polling is the main reason that system uses queues and it would cause a scaling issue. At one point you have to update the tables and you would bottleneck because of locking and isolations levels.
I've got a basic knowledge from here about epoll. I know that epoll can monitor multiple FDs and handle them.
My question is: can a heavy event block the server so I must use multithreading?
For example, the epoll of a server is monitoring 2 sockets A and B. Now A starts to send lot of messages to the server so the server starts to read them. One second later, B starts to send messages too while A is still sending. In this case, Need I create a thread for these read actions? If I don't, does it mean that the server has no chance to get the messages from B until A finishes its sending?
If you can process incoming messages fast enough (no blocking calls, no heavy computations), you don't need a separate thread. Otherwise, you would benefit from going multi-threaded.
In any case, it helps to understand what happens when you have only one thread and you can't process messages fast enough. If you are working with TCP protocol, the machines sending you the data will simply reduce their transmission rate. When using UDP, some incoming packets will get dropped.
So I have this API endpoint called www.example.com/endpoint on which many devices post(I work in an IOT firm). We have implemented our whole backed in NodeJS and are stuck while scaling from 1 device to 'n' number of devices. The devices post their packets at this API endpoint, from where I execute a complex bit of code(arnd 1000 lines) and save the state of the device in the database(mongoDB). Now the issue is. Whenever I receive a packet from device 1 and I am executing it and in the middle I get a packet from device 2, NodeJS leaves the device 1 execution as it is and starts serving the packet 2 from device 2, I saw this when I put extensive console.log() statements
Now in an ideal world. I would want Node to save the context of my current progress with packet 1. then leave. and go on to save the packet 2 in a queue to be processed later. Once I am done with packet 1 I shall take up packet 2 and process it.
I know libraries like RabbitMQ and kue for storing it in queue and processing it later, but how do I context switch from one execution to another?
This is my way of thinking. There could be other solutions as well. Would like to hear your thoughts on the matter.
Q: How to implement concurrency or context-switching in NodeJS.
A: Short answer: Not possible. Because Javascript is single threaded.
Q: Now the issue is. Whenever I receive a packet from device 1 and I am executing it and in the middle I get a packet from device 2, NodeJS leaves the device 1 execution as it is and starts serving the packet 2 from device 2, I saw this when I put extensive console.log() statements
A: As you might have already read in numerous places that NodeJS is based on an event-driven model that is non-blocking for I/O.
The reason why Node seems to have ditched device1 midway to serve device2 was because the code for device1 has already been processed up till a point where it is just waiting on an asynchronous function to callback. E.g. performing a database write. So meantime while it is available, it went on to service device2
Similar case for device2 - once it hits an async function where an event gets pushed into the event queue, pending for a return. Node might go back to device1 if a response has come back. Or it could be other devices, deviceN.
We say NodeJS is non-blocking because the node process does not lock the entire web application down for a sole response. Instead it move on and pick the next event (essentially a block of code) from the queue to run it. Hence it is constantly busy, unless there is really nothing available on the event queue.
Q: I know libraries like RabbitMQ and kue for storing it in queue and processing it later, but how do I context switch from one execution to another?
A:
As said earlier. as of 2016 - it is still not possible for Javascript to do threading. NodeJS is not designed for heavy computation work, it should only be focused on serving requests therefore the code should preferably be light and non-blocking. Basically you will want to leave those heavy I/O duties like writing to file or databases or making HTTP requests (network) to other processes by wrapping the calls with async functions.
NodeJS is not a silver bullet technology. If your application is expected to do a lot of computational work on the event thread then Node is probably not a good choice of technology but it is not the end of the world - as you can fork your own child process for the heavy computational jobs.
See:
https://nodejs.org/api/child_process.html
You might also want to consider alternative like Java which has NIO and Threading capabilities.
I'm using Managed Executor Service to implement a process manager which will process tasks in the background upon receiving an JMS message event. Normally, there will be a small number of tasks running (maybe 10 max) but what if something happens and my application starts getting hundred of JMS message events. How do I handle such event?
My thought is to limit the number of threads if possible and save all the other messages to database and will be run when thread available. Thanks in advance.
My thought is to limit the number of threads if possible and save all the other messages to database and will be run when thread available.
The detailed answer to this question depends on which Java EE app server you choose to run on, since they all have slightly different configuration.
Any Java EE app server will allow you to configure the thread pool size of your Managed Executor Service (MES), this is the number of worker threads for your thread pool.
Say you have a 10 worker threads, and you get flooded with 100 requests all at once, the MES will keep a queue of requests that are backlogged, and the worker threads will take work off the queue whenever they finish work until the queue is empty.
Now, it's fine if work goes to the queue sometimes but if overall your work queue increases more quickly than your worker threads can take work off the queue, you will run into problems. The solution to this is to increase your thread pool size otherwise the backlog will get overrun and your server will run out of memory.
what if something happens and my application starts getting hundred of JMS message events. How do I handle such event?
If the load on your server will be so sporadic that tasks need to be saved to a database, it seems that the best approach would be to either:
increase thread pool size
have the server immediately reject incoming tasks when the task backlog queue is full
have clients do a blocking wait for the server task queue to be not full (I would only advise this option if client task submission is in no way connected to user experience)
I'm working on a server architecture for sending/receiving messages from remote embedded devices, which will be hosted on Windows Azure. The front-facing servers are going to be maintaining persistent TCP connections with these devices, and I need a way to communicate with them on the backend.
Problem facts:
Devices: ~10,000
Frequency of messages device is sending up to servers: 1/min
Frequency of messages originating server side (e.g. from user actions, scheduled triggers, etc.): 100/day
Average size of message payload: 64 bytes
Upward communication
The devices send up messages very frequently (sensor readings). The constraints for that data are not very strong, due to the fact that we can aggregate/insert those sensor readings in a batched manner, and that they don't require in-order guarantees. I think the best way of handling them is to put them in a Storage Queue, and have a worker process poll the queue at intervals and dump that data. Of course, I'll have to be careful about making sure the worker process does this frequently enough so that the queue doesn't infinitely back up. The max batch size of Azure Storage Queues is 32, but I'm thinking of potentially pulling in more than that: something like publishing to the data store every 1,000 readings or 30 seconds, whichever comes first.
Downward communication
The server sends down updates and notifications much less frequently. This is a slightly harder problem, as I can see two viable paradigms here (with some blending in between). Could either:
Create a Service Bus Queue for each device (or one queue with thousands of subscriptions - limit is for number of queues is 10,000)
Have a state table housed in a DB that contains the latest "state" of a specific message type that the devices will get sent to them
With option 1, the application server simply enqueues a message in a fire-and-forget manner. On the front-end servers, however, there's quite a bit of things that have to happen. Concerns I can see include:
Monitoring 10k queues (or many subscriptions off of a queue - the
Azure SDK apparently reuses connections for subscriptions to the same
queue)
Connection Management
Should no longer monitor a queue if device disconnects.
Need to expire messages if device is disconnected for an extended period of time (so that queue isn't backed up)
Need to enable some type of "refresh" mechanism to update device's complete state when it goes back online
The good news is that service bus queues are durable, and with sessions can arrange messages to come in a FIFO manner.
With option 2, the DB would house a table that would maintain state for all of the devices. This table would be checked periodically by the front-facing servers (every few seconds or so) for state changes written to it by the application server. The front-facing servers would then dispatch to the devices. This removes the requirement for queueing of FIFO, the reasoning being that this message contains the latest state, and doesn't have to compete with other messages destined for the same device. The message is ephemeral: if it fails, then it will be resent when the device reconnects and requests to be refreshed, or at the next check interval of the front-facing server.
In this scenario, the need for queues seems to be removed, but the DB becomes the bottleneck here, and I fear it's not as scalable.
These are both viable approaches, and I feel this question is already becoming too large (although I can provide more descriptions if necessary). Just wanted to get a feel for what's possible, what's usually done, if there's something fundamental I'm missing, and what things in the cloud can I take advantage of to not reinvent the wheel.
If you can identify the device (may be device id/IMEI/Mac address) by the the message it sends then you can reduce the number of queues from 10,000 to 1 queue and not have 10000 subscriptions too. This could also help you in the downward communication as you will be able to identify the device and send the message to the appropriate socket.
As you mentioned the connections last longer you could deliver the command to the device that is connected and decide what to do with the commands to the device that are not connected.
Hope it helps