My Python3 server code:
import zmq
context = zmq.Context()
router = context.socket(zmq.ROUTER)
router.bind("tcp://*:1337")
router.send_multipart([b'build_0', b''])
while True:
pass
My C# client code:
/// <summary>
/// Receive data from server.
/// </summary>
private void RecvFromServer()
{
// Init the socket.
recvFromServerSocket = new DealerSocket();
recvFromServerSocket.Options.Identity = StringToByteArray("build_0");
recvFromServerSocket.Connect(NETWORK_PREFIX + serverAddress + PORT_INFIX + recvFromServerPort);
NetMQMessage frames = new NetMQMessage();
// When the socket is ready, receive a multipart message.
recvFromServerSocket.ReceiveReady += (o, s) =>
{
onBeginListenToServer?.Invoke(ref BenchmarkData.benchmark.began_to_listen_to_server);
frames = s.Socket.ReceiveMultipartMessage();
foreach (NetMQFrame frame in frames)
{
Debug.Log(frame.ConvertToString());
messagesFromServer.Enqueue(frame.ConvertToString());
}
frames.Clear();
onReceivedMessage?.Invoke(ref BenchmarkData.benchmark.received_from_server);
};
// Init the poller.
recvFromServerPoller = new NetMQPoller { recvFromServerSocket };
// Run the poller.
recvFromServerPoller.Run();
}
Right now, the message will not be received by the C# client.
However, if I set the Python socket to zmq.DEALER, the client will receive the message, but will treat build_0 as part of the multipart message, rather than the sender ID. I need the ID to actually be accepted, as there will be multiple clients. Python dealer sockets with an identity of build_0 can communicate with the server example.
How do I set up a NetMQ DealerSocket to accept any messages from a pyzmq router socket while using identities properly?
Related
I created a simple Azure Service bus (Queue) and a client that is sending message to service bus. Using below code to send message:
using Microsoft.Azure.ServiceBus;
using Microsoft.Extensions.Configuration;
public async Task SendMessageAsync<T>(T message, string queueName)
{
try
{
var queueClient = new QueueClient(_config.GetConnectionString("AzureServiceBus"), queueName);
string messageBody = JsonSerializer.Serialize(message);
var byteMessage = new Message(Encoding.UTF8.GetBytes(messageBody));
queueClient.SendAsync(byteMessage);
Console.WriteLine((message as Employee).FirstName);
}
catch (Exception ex)
{
var c = ex;
}
}
Sending message using:
using SenderApp;
Console.WriteLine("Hello, World!");
QueueService service = new QueueService();
for (int i = 0; i < 100; i++)
{
Employee e = new Employee();
e.FirstName = "1 " + i.ToString();
e.LastName = "2 " + i.ToString();
service.SendMessageAsync<Employee>(e, "employeequeue");
}
When I try to see active messages, There is nothing in the queue:
However I do see some traffic. But the number of message I sent (over 100) is not equal to number of incoming request show (62) at the bottom of the image. I am not sure what is happening to my messages? This defeats the purpose of the queue.
Please guide me why I am not seeing any messages. What is the best way to handle this ?
I am using following nuget packages:
<PackageReference Include="Microsoft.Azure.ServiceBus" Version="5.2.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
A message sent to an Azure Service Bus queue will be delivered to the queue unless operation is failing. In that case, an exception will be thrown. Check the following:
Exception handling doesn't swollow exceptions
Await asynchronous send operations to ensure messages are dispatched
Namespace/queue used for sending is what you use to receive
There are no competing consumers, actively receiving messages from the queue.
Validate TCP ports needed for AMQP are not blocked. If those ports are blocked, you could configure your client to use WebSockets.
So I still dont know what caused this issue. But I realized Microsoft.Azure.ServiceBus package was deprecated and later I started using Azure.Messaging.ServiceBus package to connect to service bus and things started to work.
I used following code to send message to queue:
string connectionString = "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=f3f+qMYTyVwE18YNl5J6ygJFi30v6J/Smph5HZvyQyE=";
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create the sender
ServiceBusSender sender = client.CreateSender(queueName);
// create a message that we can send. UTF-8 encoding is used when providing a string.
ServiceBusMessage message = new ServiceBusMessage("Hello world! " + id);
// send the message
await sender.SendMessageAsync(message);
return "Sent";
Used following code to receive message:
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create a receiver that we can use to receive and settle the message
ServiceBusReceiver receiver = client.CreateReceiver(queueName);
// the received message is a different type as it contains some service set properties
ServiceBusReceivedMessage receivedMessage = await receiver.ReceiveMessageAsync();
string body = receivedMessage.Body.ToString();
// complete the message, thereby deleting it from the service
await receiver.CompleteMessageAsync(receivedMessage);
More info is available # https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.7.0/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
I have an asp.net MVC 5 application. Now I want to have a new feature allow me to receive GPS signal from devices and update devices real time in web client side.
My plan is to use RabbitMQ to handle message queue and SignalR to notify clients for postion update
My code in Rabbit consumer console application like this
var client = GlobalHost.ConnectionManager.GetHubContext<MyHub>().Clients;
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
var queueName = channel.QueueDeclare().QueueName;
channel.ExchangeDeclare("mychannel", "fanout");
channel.QueueBind(queueName, "mychannel", "");
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine("[x] Receive {0}", message);
client.All.hello(message);
};
channel.BasicConsume(queue: queueName,
autoAck: true,
consumer: consumer);
Console.WriteLine("Press Enter to quit");
Console.ReadLine();
}
}
But the code is not fired when I produce a message to RabbitMQ. If I remove code of signalR client:
var client = GlobalHost.ConnectionManager.GetHubContext<MyHub>().Clients;
things work as expected and I can receive message as usual
My question is:
Is my approach correct
if the approach correct then how to make it work?
Many thanks
Given this snippet of code:
//DirectApi
mqServer.RegisterHandler<LeadInformationInfo>(m =>
{
repository.SaveMessage(m as Message);
LeadInformationInfoResponse response = new LeadInformationInfoResponse();
try
{
var client = new JsonServiceClient(settingsFactory.GetMasterSetting("ProcessorApi:baseUri"));
response = client.Post(m.GetBody());
}
catch (WebServiceException webServiceException)
{
_log.Error("RegisterHandler<LeadInformationInfo>", webServiceException);
response = ((LeadInformationInfoResponse) webServiceException.ResponseDto);
response.CorrelationId = m.Id;
}
// Log response message here
return response;
}, 1);
I've gone to great lengths to make sure that a correlationId based off the original message Id property is propagated through the life of this message and any child messages spawned from this action. How do I get a handle on the response message so that I may log it in the handler? I only have access to the ResponseDto and not the message.
One of the reasons for this request is that the message queue client does not have access to the database, only the process that has the handler registered does. Hope that explains the situation better.
Just to clarify, this question is about persisting a MQ Response Message in the handler, the correlation Id is something that all messages in 1 request/response workflow will share. I'm also using ServiceStack ORMlite to persist the Message object, so querying this table by ID for troubleshooting is paramount.
Thank you,
Stephen
You're calling a WebService from within your MQ Handler:
var client = new JsonServiceClient(...);
response = client.Post(m.GetBody());
So there is no MQ Response which is only available in MQ Services. Although the WebService will return the response for the request that's sent so you can either use the CorrelationId on the MQ Request, otherwise you can have your Response DTO implement an interface like IHasCorrelationId and get it that way, e.g:
var correlationResponse = response as IHasCorrelationId;
if (correlationResponse != null)
{
var correlationId = correlationResponse.CorrelationId;
}
Create your own Instance of Message
As the Message<T> class is just a POCO if you wanted to create your own you can intialize your own instance:
var mqResponse = new Message<Response>(response);
If you only had the runtime late-bound type info, you can create one with:
var mqResponse = MessageFactory.Create(response);
Use RabbitMQ Message Filters
If you just wanted to log incoming and outgoing messages you can use the RabbitMQ Message Filters, e.g:
var mqServer = new RabbitMqServer("localhost")
{
PublishMessageFilter = (queueName, properties, msg) => {
properties.AppId = "app:{0}".Fmt(queueName);
},
GetMessageFilter = (queueName, basicMsg) => {
var props = basicMsg.BasicProperties;
receivedMsgType = props.Type; //automatically added by RabbitMqProducer
receivedMsgApp = props.AppId;
}
};
Is there any way to add the type of the response dto to the rabbitmq response message's headers collection?
(My consumer is using spring's rabbitmq handler which seems to depend on explicit type information inside the mq header when deserializing.)
Currently servicestack's mq producer already returns serveral headers, such as "content_type='application/json".
I am in need of an additional header, e.g. "typeId"="HelloResponse", so that the consuming web app knows how to deserialize the message, even in RPC cases where the response queue name is some kind of GUID.
Is there some kind of configuration which would enable me to archieve such an behaviour? Or some hook before the message gets published so that I can add the header myself?
I've added support for automatically populating the Message Body Type in RabbitMQ's IBasicProperties.Type as well as adding support for both Publish and GetMessage Filters in this commit.
Here's an example of configuring a RabbitMqServer with custom handlers where you can modify the message and its metadata properties when its published and received:
string receivedMsgApp = null;
string receivedMsgType = null;
var mqServer = new RabbitMqServer("localhost")
{
PublishMessageFilter = (queueName, properties, msg) => {
properties.AppId = "app:{0}".Fmt(queueName);
},
GetMessageFilter = (queueName, basicMsg) => {
var props = basicMsg.BasicProperties;
receivedMsgType = props.Type; //automatically added by RabbitMqProducer
receivedMsgApp = props.AppId;
}
};
mqServer.RegisterHandler<Hello>(m =>
new HelloResponse { Result = "Hello, {0}!".Fmt(m.GetBody().Name) });
mqServer.Start();
Once Configured any message published or received will go through the above handlers, e.g:
using (var mqClient = mqServer.CreateMessageQueueClient())
{
mqClient.Publish(new Hello { Name = "Bugs Bunny" });
}
receivedMsgApp.Print(); // app:mq:Hello.In
receivedMsgType.Print(); // Hello
using (IConnection connection = mqServer.ConnectionFactory.CreateConnection())
using (IModel channel = connection.CreateModel())
{
var queueName = QueueNames<HelloResponse>.In;
channel.RegisterQueue(queueName);
var basicMsg = channel.BasicGet(queueName, noAck: true);
var props = basicMsg.BasicProperties;
props.Type.Print(); // HelloResponse
props.AppId.Print(); // app:mq:HelloResponse.Inq
var msg = basicMsg.ToMessage<HelloResponse>();
msg.GetBody().Result.Print(); // Hello, Bugs Bunny!
}
This change is available from ServiceStack v4.0.33+ that's now available on MyGet.
My requirements are stated below:
I have to develop a wrapper service on top a queue,so i was just going through some message Queue like (ActiveMQ,Apollo,Kafka). But decided to proceed with ActiveMQ to match our usecases.Now the requirement are as follows:
1) A restful api through which different publisher will publish to queue,based on clientId queue will be selected.
2) Consumer will consume message through restful api and will consume message in batches. say consumer as for something like give me 10 message from queue.
Now the service should provide 10 message if there is 10 message or if message number is less or zero it will send accordingly. After receiving the message the client will process with the message and send back acknowledgement through different res-full uri. upon receiving that acknowledgement,the MQService should commit or rollback message from the queue.
In order to this in the MQService layer, i have used a cached,where im keeping the JMS connection and session object till acknowledgemnt is received or ttl expire.
In-order to retrieve message in batches and send back to client, i have created a multi-threaded consumer,so that for 5 batch message request,the service layer will create 5 thread each having different connection and session object( as stated in ActiveMQ multiple consumer http://activemq.apache.org/multiple-consumers-on-a-queue.html)
Basic use-case:
MQ(BROKER)[A] --> Wrapper(MQService)[B]-->Client [C]
Note:[B] is a restfull service having JMS consumer implemented in it.It keeps the connection and session object in cache.
[C] request to [B] to give 3 message
[B] must fetch 3 message if available in queue,wrap it in batchmsgFormat and send it to [C]
[C] process the message and send acknowledgemnt suces/failed to [B] through /send-ack uri.
Upon receiving Ack from [C], [B] will commit the Jms session and close the session and connection object. Also it will evict those from the cache.
The above work-flow is working fine with single message fetching.
But the queue hungs up on JMS MesageConsumer.receive() when try to fetch message with mutilple consumer using multithreading. ...
Here the JMS Consumer code in MQService layer:
----------------------------------------------
public BatchMessageFormat getConsumeMsg(final String clientId, final Integer batchSize) throws Exception {
BatchMessageFormat batchmsgFormat = new BatchMessageFormat();
List<MessageFormat> msgdetails = new ArrayList<MessageFormat>();
List<Future<MessageFormat>> futuremsgdetails = new ArrayList<Future<MessageFormat>>();
if (batchSize != null) {
Integer msgCount = getMsgCount(clientId, batchSize);
for (int batchconnect = 0; batchconnect <msgCount; batchconnect++) {
FutureTask<MessageFormat> task = new FutureTask<MessageFormat>(new Callable<MessageFormat>() {
#Override
public MessageFormat call() throws Exception {
MessageFormat msg=consumeBatchMsg(clientId,batchSize);
return msg;
}
});
futuremsgdetails.add(task);
Thread t = new Thread(task);
t.start();
}
for(Future<MessageFormat> msg:futuremsgdetails){
msgdetails.add(msg.get());
}
batchmsgFormat.setMsgDetails(msgdetails);
return batchmsgFormat
}
Message fetching:
private MessageFormat consumeBatchMsg(String clientId, Integer batchSize) throws JMSException, IOException{
MessageFormat msgFormat= new MessageFormat();
Connection qC = ConnectionUtil.getConnection();
qC.start();
Session session = qC.createSession(true, -1);
Destination destination = createQueue(clientId, session);
MessageConsumer consumer = session.createConsumer(destination);
Message message = consumer.receive(2000);
if (message!=null || message instanceof TextMessage) {
TextMessage textMessage = (TextMessage) message;
msgFormat.setMessageID(textMessage.getJMSMessageID());
msgFormat.setMessage(textMessage.getText());
CacheObject cacheValue = new CacheObject();
cacheValue.setConnection(qC);
cacheValue.setSession(session);
cacheValue.setJmsQueue(destination);
MQCache.instance().add(textMessage.getJMSMessageID(),cacheValue);
}
consumer.close();
return msgFormat;
}
Acknowledgement and session closing:
public String getACK(String clientId,String msgId,String ack)throws JMSException{
if (MQCache.instance().get(msgId) != null) {
Connection connection = MQCache.instance().get(msgId).getConnection();
Session session = MQCache.instance().get(msgId).getSession();
Destination destination = MQCache.instance().get(msgId).getJmsQueue();
MessageConsumer consumer = session.createConsumer(destination);
if (ack.equalsIgnoreCase("SUCCESS")) {
session.commit();
} else {
session.rollback();
}
session.close();
connection.close();
MQCache.instance().evictCache(msgId);
return "Accepted";
} else {
return "Rejected";
}
}
Does anyone worked on similar scenario or can you pls throw some light? Is there any other way to implement this batch mesage fetching as well as client failure handling?
Try after setting the prefetch limit to 0 as below:
ConnectionFactory connectionFactory
= new ActiveMQConnectionFactory("tcp://localhost:61616?jms.prefetchPolicy.queuePrefetch=0");
I'll give a few pointers to help to code this logic better.
I'm assuming you are using pure JMS 1.1 as much as possible. Ensure that you have one place where you get the connection from the pool or create a connection. You need not do that inside a thread. You can do that outside. Sessions must be created inside a thread and shouldn't be shared. This will impact the logic in the function consumeBatchMsg().
Secondly, its simpler to use one thread to consume all the messages of the given batchSize. I see that you are using transacted session. So you can do one commit after getting all the messages of the batchSize.
If you really want to take the complicated route of having multiple consumers on a queue (probably little better performance), you can using CountDownLatch or CyclicBarrier of Java and set it to batchSize to trigger. Once all the threads have received the messages, it can commit and close the sessions in the respective threads. Never let the session instance go out of the context of the thread that created it.