I need a variant of Crossbeam's zero-capacity channel, crossbeam_channel::bounded(0), which does not block on send() if there is no receive operation. In my case, messages that are sent while there is no receive operation in progress can be discarded. A receiver will receive all the messages sent after it started listening. This is similar to what happens with Redis channels, but between threads.
Does something like this exist already or do I need to implement it myself? At the moment it's not clear to me how such a functionality would be implemented, but I can probably start by looking at the implementation of the bounded zero-capacity channel and possibly replace the blocking send operation with a non-blocking version.
All crossbeam channels offer a non-blocking version of the send() method as well. It's called try_send(), and it returns an error in case the message could not be sent. For a zero-capacity channel, this result in exactly the behaviour you ask for – the message will only be sent "if there happens to be a receive operation on the other side of the channel at the same time" (quote from the docs).
Related
I am new to Rust and trying to understand the Dining Philosopher code here :
https://google.github.io/comprehensive-rust/exercises/day-4/solutions-morning.html
By the time the execution reaches the following lines in the main thread, isn't it possible that none of the spawned threads have started executing their logic, resulting in nothing in 'rx' and the program simply quitting?
for thought in rx {
println!("{}", thought);
}
When iterating over a channel, it internally calls Receiver::recv, where the documentation specifies
This function will always block the current thread if there is no data available and it’s possible for more data to be sent (at least one sender still exists). Once a message is sent to the corresponding Sender (or SyncSender), this receiver will wake up and return that message.
So the receiver will block until it has data avalible, or all the senders have been dropped.
Yes, execution can reach for thought in rx { ... } before the threads have even started. However, this will still work because iterating over a Receiver will wait until there is a message and will only stop if all Senders have been destroyed (ergo it is no longer possible to receive any messages).
I'm looking at Flume's unbounded channel (mpmc), but i'm confused why it's named as it is, because the consumer channel needs to be wrapped in a mutex in order to receive messages from it from multiple threads? What would the difference between an mpmc and mpsc be then?
Someone on the Rust Community Discord gave me an answer:
I was told that flume::Receiver implements Clone + Send + Sync so it needn't be wrapped in a Mutex. And I simply just need to clone a reference the receiver and send it off to threads.
A zmq subscriber fails to subscribe to the message if socket.send() is used only once in publisher.
Subscriber was able to subscribe to the messages when the following code is used in publisher:
var zmq = require('zmq')
, sock = zmq.socket('pub')
sock.bindSync('tcp://127.0.0.1:3000');
var message = {"test" : true};
setInterval(function(){
sock.send(['notify_anomaly', JSON.stringify(message)]);
},1000);
But it doesn't work if setInterval is removed in the publisher code, as follows:
var zmq = require('zmq')
, sock = zmq.socket('pub')
sock.bindSync('tcp://127.0.0.1:3000');
var message = {"test" : true};
sock.send(['notify_anomaly', JSON.stringify(message)]);
This is a result of the "slow joiner" problem.
Here is a quote from the guide:
There is one more important thing to know about PUB-SUB sockets: you
do not know precisely when a subscriber starts to get messages. Even
if you start a subscriber, wait a while, and then start the publisher,
the subscriber will always miss the first messages that the publisher
sends. This is because as the subscriber connects to the publisher
(something that takes a small but non-zero time), the publisher may
already be sending messages out.
Basically, when the publisher runs, it has a handshake with the subscribers. Since that is asynchronous, the publisher may finish sending the messages before the handshake is complete. In that case the subscribers will miss the messages. To get the subscriber to get the first message, the publisher needs to wait to send until it is sure the subscriber is connected.
Here is a further quote:
In Chapter 2 - Sockets and Patterns we'll explain how to synchronize a
publisher and subscribers so that you don't start to publish data
until the subscribers really are connected and ready.
It shows how you can use another socket pair using REQ-REP to signal when the subscriber is ready to receive on the subscibe socket here.
Well, not exactly, Sir.
Historically, ZeroMQ used a SUB-side subscription ( topic-filtering ). That means two things. PUB-lisher has zero knowledge of who SUB'ed to what & spends Zero-effort on topic-filter processing. Plus, it pours and has to do so, all messages onto all diverse transport-class channels down the road, towards ( only down-there topic-filtering ) SUB-scribers ( which principally creates some level of inefficiency on the transport-layers resources ).
So, if your code uses "old" ZeroMQ wrapper / language-binding, your "PUB-lisher" is out of question not the root-cause of the problem, as it by-design does not care about any, including "late-SUBscriber", counterparties. The time-delay helps ( not the PUB.send(), but the
// unknown SUB code, a default SUB-state is TOPIC-FILTER throws everything, YES !
// THROWS EVERYTHING AWAY
SUB.setsockopt( zmq.SUBSCRIBE,
"<some_TOPIC_FILTER_string_to_be_used_after_this_happens_in_time"
);
So this has nothing to do with the code per-se on the PUB side, Q.E.D., but designers have to bear the timing-coincidences in mind, if designing robust app-architectures.
Next, the newer ZeroMQ versions have switched to PUB-side filtering. This seems as a major change, but it does not have any great implications into your example.
PUB-side filtering has just removed the transport-layer congestions by centralised topic-filtering on the PUB-side, at a cost of sum-of-(so-far-cheap-'cause-distributed)-workloads, that now reside, concentrated, on the PUB-side.
So, your observation still shows not receiving any message on SUB-side, so why going into such details? Well, now, in the case of the newer version, if the SUB did not manage to "tell & deliver" it's SUB-scription preferences to the PUB-side, before that one had already dispatched the PUB.send( aFirstMESSAGE_to_Those_whom_I_know_they_SUBed_to_this_TOPICFILTER )
there would be no music down the road at all, again, due to timing-coincidence of the distributed system events' propagation & delivery in time and not due to a PUB-side ( only ) code tweaking.
Epilogue:
In either case, ZeroMQ is a broker-less messaging framework. That means, persistence of messages is not even intended to create, nor provided. Each of the Scaleable Formal Communication Pattern's behavioural archetype node has clearly specified in API-documentation, buffer-management and message-{ retain | discard } and other rules. Be sure to check also the different versions of ZeroMQ low-level protocol, that is being used on all nodes of your distributed system realm ( typically one cannot control that, but may design version-aware behavioral policy enforcements to handle this sort of production ecosystem uncertainties ).
TheBestNextStep:
If one strives to remain some time in an area of a professional design of distributed systems, the best thing one can do is to read the fabulous book from one of the ZeroMQ fathers, Pieter HINTJENS ( may check other post on ZeroMQ and follow the direct link to the book's PDF-version ).
This is more of a conceptual question and doesn't apply to any particular programming language.
I have two entities communicating with each other, with three types of messages allowed:
Command Message: An unsolicited message commanding the other entity to do something.
Query Message: An unsolicited message asking the other entity for information.
Response Message: A solicited message answering a query message from the other entity.
Now each entity has two threads:
Reader Thread: Reads messages.
Worker Thread: Sends messages and does useful things
The two possible communication scenarios are:
Entity A sends a command to Entity B, and Entity A doesn't care what happens after.
Entity A sends a query to Entity B, and Entity A must wait until Entity B responds with the answer.
So the question is, how does the reader thread handle both solicited and unsolicited messages?
Unsolicited messages are easy to handle through events. The reader thread can just fire an event on the worker thread saying it received a query or a command, and the worker thread can react accordingly.
Solicited messages are hard to handle though. The worker thread sends a query, and must block until it receives a response or times out. How does the worker thread let the reader thread know it is waiting for a response, and how does the reader thread tie a response back to a specific query from the worker thread and deliver that response back to the worker thread's execution?
I know this has been done a million times in other programs, so whats the standard practice?
[I used Windows Azure Service Bus messaging entities as I am familiar with it, but in general this should be true with any Messaging system.]
Lets say your entity names are A and B.
Have 1 Topic (pub-sub entities) and 1 Queue for communication between A and B (as you need bidirectional communication) : Topic-A2B & Queue-B2A. A2B is for Commands from A to B or Queries from A to B and B2A, as the name says, is for Responses from B to A.
Typical Messaging Systems will offer MessageType property - for you to be able to set it and the later distinguish which type of messages you are reading and route it accordingly : Example from Windows Azure ServiceBus Brokered Message. Use that Property - to set whether its a Query or Command or Response.
The idea here is - while receiving a message in B - you will receive using Subscriptions. You will have 2 threads reading - (one) reads only Commands (theSecondOne) reads only Queries
For UnSolicited messages - as you said, its easy to handle. All you need to do is
A should send message to B with BrokeredMsg.ContentType="Cmd" and B should create a Subscription with a filter and read and process
For Solicited Messages - like Queries (a feature called Sessions will come handy here).
A should send Message to B with something like: BrokeredMessage.ContentType = "Query"
A also sets a correlation Id on the Message it sends to B: BrokeredMessage.SessionId = "ABC456" <-- The Correlation Id for A to be able to correlate this message with
Now A will wait for response and expects B to also set
BrokeredMessage.SessionId="ABC456" <--- The exact same value it had set earlier.
using the AcceptMessageSession API - with the Session Id and a Timeout. Ex: Q_B2A_QClient.AcceptMessageSession("ABC456", 2 mins)
At the receiving end B should Create a Subscription with a filter to be able to Receive these messages.
Once B receives the query - it processes and puts back the result in the Q-BToA
If B succeeds to put back the message in the Q-B2A in less than 2 Mins - then A will receive it and then you can orchestrate it further with a Callback method (as all of these are async methods - you will not need to use any Reader or Writer thread as you mentioned above - which will be a huge performance booster).
HTH!
Sree
To my current understanding, after calling MPI_Send, the calling thread should block until the variable is received, so my code below shouldn't work. However, I tried sending several variables in a row and receiving them gradually while doing operations on them and this still worked... See below. Can someone clarify step by step what is going on here?
matlab code: (because I am using a matlab mex wrapper for MPI functions)
%send
if mpirank==0
%arguments to MPI_Send_variable are (variable, destination, tag)
MPI_Send_variable(x,0,'A_22')%thread 0 should block here!
MPI_Send_variable(y,0,'A_12')
MPI_Send_variable(z,1,'A_11')
MPI_Send_variable(w,1,'A_21')
end
%recieve
if mpirank==0
%arguments to MPI_Recv_variable are (source, tag)
a=MPI_Recv_variable(0,'A_12')*MPI_Recv_variable(0,'A_22');
end
if mpirank==1
c=MPI_Recv_variable(0,'A_21')*MPI_Recv_variable(0,'A_22');
end
MPI_SEND is a blocking call only in the sense that it blocks until it is safe for the user to use the buffer provided to it. The important text to read here is in Section 3.4:
The send call described in Section 3.2.1 uses the standard communication mode. In this mode, it is up to MPI to decide whether outgoing messages will be buffered. MPI may buffer outgoing messages. In such a case, the send call may complete before a matching receive is invoked. On the other hand, buffer space may be unavailable, or MPI may choose not to buffer outgoing messages, for performance reasons. In this case, the send call will not complete until a matching receive has been posted, and the data has been moved to the receiver.
I highlighted the part that you're running up against in bold there. If your message is sufficiently small (and there are sufficiently few of them), MPI will copy your send buffers to an internal buffer and keep track of things internally until the message has been received remotely. There's no guarantee that when MPI_SEND is done, the message has been received.
On the other hand, if you do want to know that the message was actually received, you can use MPI_SSEND. That function will synchronize (hence the extra S both sides before allowing them to return from the MPI_SSEND and the matching receive call on the other end.
In a correct MPI program, you cannot do a blocking send to yourself without first posting a nonblocking receive. So a correct version of your program would look something like this:
Irecv(..., &req1);
Irecv(..., &req2);
Send(... to self ...);
Send(.... to self ...);
Wait(&req1, ...);
/* do work */
Wait(&req2, ...);
/* do more work */
Your code is technically incorrect, but the reason it is working correctly is because the MPI implementation is using internal buffers to buffer your send data before it is transmitted to the receiver (or matched to the later receive operation in the case of self sends). An MPI implementation is not required to have such buffers (generally called "eager buffers"), but most implementations do.
Since the data you are sending is small, the eager buffers are generally sufficient to buffer them temporarily. If you send large enough data, the MPI implementation will not have enough eager buffer space and your program will deadlock. Try sending, for example, 10 MB instead of a double in your program to notice the deadlock.
I assume that there is just a MPI_Send() behind MPI_Send_variable() and MPI_Receive() behind MPI_Receive_variable().
How do a process can ever receive a message that he sent to himself if both the send and receive operations are blocking ? Either send to self or receive to self are non-blocking or you will get a deadlock, and sending to self is forbidden.
Following answer of #Greginozemtsev Is the behavior of MPI communication of a rank with itself well-defined? , the MPI standard states that send to self and receive to self are allowed. I guess it implies that it's non blocking in this particular case.
In MPI 3.0, in section 3.2.4 Blocking Receive here, page 59, the words have not changed since MPI 1.1 :
Source = destination is allowed, that is, a process can send a message to itself.
(However, it is unsafe to do so with the blocking send
and receive operations described above, since this may lead to deadlock.
See Section 3.5.)
I rode section 3.5, but it's not clear enough for me...
I guess that the parenthesis are here to tell us that talking to oneself is not a good practice, at least for MPI communications !