just a rapid question: why the MessageEvent returned by NestJs (https://docs.nestjs.com/techniques/server-sent-events#usage) it's not compliant with the standard (https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format)?
In particular, nestjs uses type instead of event for the event type.
Thank you
Related
Let me give you a bigger picture of the problem... I am designing a ROS2-based system with multiple ROS2 nodes each containing a wrapper part (ROS2 layer) and driver/module part where my low-level logic is implemented. The wrapper part is using some ROS2-specific communication mechanisms (topics, services, actions...) to exchange the data/commands between the nodes.
Now, one of the nodes in the system should establish an MQTT connection with the Google Cloud Platform, keep the connectivity alive and allow data exchange between the Cloud and ROS2 system. For that purpose, I am using iot-device-sdk-embedded-c SDK from Google.
It has iotc_connect() blocking function for establishing and keeping connection with the Cloud so the challenge I am facing with is to simultaneously keep the ROS2 node spinning while keeping MQTT connectivity alive.
My idea was to launch a thread from ROS2 wrapper that will be used for establishing/keeping MQTT connectivity and use a callback function as an argument for the thread function that will enable me to forward the data received from the Cloud ithin the thread directly to ROS2 layer. Launching a separate thread for handling connectivity and data exchange would enable my ROS2 node to properly spin and rest synchronized with the rest of the ROS2 system.
ROS2_Wrapper.cpp
thread mqtt_thread(MqttConnHandler::ConnectToMqttServer, &MqttThreadCallback);
mqtt_thread.detach();
...
void MqttThreadCallback(void* data, size_t size){
}
MqttThreadCallback() should be called every time I receive the command/config data from the Cloud.
However, I am not sure how can I fire the callback function within the thread because I have two layers of nested callbacks within the thread:
my_thread.cpp
ConnectToMqttServer(void (*MqttThreadCallback)(void*, size_t)){
...
iotc_connect(...,&OnConnectionStateChanged);
...
}
OnConnectionStateChanged(...){
...
case IOTC_CONNECTION_STATE_OPENED:
iotc_subscribe(...,&iotc_mqttlogic_subscribe_callback,...);
...
}
iotc_mqttlogic_subscribe_callback(...){
//The place where data from the Cloud are received
}
iotc_connect() contains OnConnectionStateChanged() callback from where iotc_subscribe() function is called at the moment connection is established. iotc_subscribe() contains iotc_mqttlogic_subscribe_callback() where data from the Cloud are received.
I am not sure how can I mount the data from iotc_mqttlogic_subscribe_callback() up to the thread caller. Do you have any suggestions? Perhaps using the threads is not the best approach?
Usually C libraries provide an optional additional argument called user_data for this purpose:
extern iotc_state_t iotc_subscribe(iotc_context_handle_t iotc_h,
const char* topic, const iotc_mqtt_qos_t qos,
iotc_user_subscription_callback_t* callback,
void* user_data);
That way you can cast your callback function pointer to void when calling subscribe and catch it as argument in the iotc_mqttlogic_subscribe_callback function call. Where you should recast the data back to the function pointer type and use it.
In addition, you may find yourself in need to pass more data to the callback (mutex to protect the data, loggers from higher level code...). In that case, the best practice is to wrap all this info in a new class of your choice and pass a pointer to the instance in the callback.
How can I consume Server Sent Events with Spring Integration? I am aware Spring supports SSE with Webflux, but how to convert the incoming Flux into separate Message instances? And possibly wrap this code into some Spring-Integration-Lifecycle-aware component (MessageProducerSupport?)
WebClient client = WebClient.create("http://myhost:8080/sse");
ParameterizedTypeReference<ServerSentEvent<String>> type
= new ParameterizedTypeReference<ServerSentEvent<String>>() {};
Flux<ServerSentEvent<String>> eventStream = client.get()
.uri("/stream-sse")
.retrieve()
.bodyToFlux(type);
eventStream.subscribe(
content -> ;/* here I believe the message should be produced/sent to a channel */ );
See Spring Integration WebFlux Outbound Gateway: https://docs.spring.io/spring-integration/docs/current/reference/html/webflux.html#webflux-outbound:
The setExpectedResponseType(Class<?>) or setExpectedResponseTypeExpression(Expression) identifies the target type of the response body element conversion. If replyPayloadToFlux is set to true, the response body is converted to a Flux with the provided expectedResponseType for each element, and this Flux is sent as the payload downstream. Afterwards, you can use a splitter to iterate over this Flux in a reactive manner.
WebFlux.outboundGateway("http://myhost:8080/sse/stream-sse")
.httpMethod(HttpMethod.GET)
.replyPayloadToFlux(true)
.setExpectedResponseTypeExpression(new ParameterizedTypeReference<ServerSentEvent<String>>() {})
To make it start working just after an application is ready, yo can implement an ApplicationRunner to send a "void" message into a channel for the flow with that WebFlux.outboundGateway(). I don't think we need a special, dedicated component just for SSE requesting and producing. The combination of existing components is fully enough.
In normal grpc communications, grpc automatically creates a context to go with the request. How do I create a mock context to pass with my unit test call, so my servicing function has an object to manipulate?
Currently I have the code skip the context calls if it receives a NoneType as context, but that is not a good solution.
You can use the magic mock library unittest.mock. And mock over the client RPC multicallable object grpc.UnaryUnaryMultiCallable (see ref), so it returns your response and context. This object is generated by our ProtoBuf plugin, and sits in the foo_pb2_grpc.py file.
I have a single ServerEventsClient object that I use to dynamically subscribe and unsubscribe from channels as needed. I have some channels that are always open and that I pass in the constructor. I register to other channels by calling SubscribeToChannelsAsync(). The connection is actually established and I am able to communicate with the other side using it (I'm using SSE as chat), but none of our OnJoin registered methods get called. The same is true for UnsubscribeFromChannelsAsync() and OnLeave. I tried using the UpdateSubscriberAsync() and got the same results.
Worth noting is the fact that I have NotifyChannelOfSubscriptions set to true in my ServerEventsFeature.
Could the problem be in the fact that we are (un)subscribing after we initialize the ServerEventsClient object with initial channels?
When a subscribers channels subscription is updated after they've subscribed it fires an onUpdate event.
We have recently started working on Typescript language for one of the application where a queue'd communication is expected between a server and client/clients.
For achieving the queue'd communication, we are trying to use the ZeroMQ library version 4.6.0 as a npm package: npm install -g zeromq and npm install -g #types/zeromq.
The exact scenario :
The client is going to send thousands of messages to the server over ZeroMQ. The server in-turn will be responding with some acknowledgement message per incoming message from the client. Based on the acknowledgement message, the client will send next message.
ZeroMQ pattern used :
The ROUTER/DEALER pattern (we cannot use any other pattern).
Client side code :
import Zmq = require('zeromq');
let clientSocket : Zmq.Socket;
let messageQueue = [];
export class ZmqCommunicator
{
constructor(connString : string)
{
clientSocket = Zmq.socket('dealer');
clientSocket.connect(connString);
clientSocket.on('message', this.ReceiveMessage);
}
public ReceiveMessage = (msg) => {
var argl = arguments.length,
envelopes = Array.prototype.slice.call(arguments, 0, argl - 1),
payload = arguments[0];
var json = JSON.parse(msg.toString('utf8'));
if(json.type != "error" && json.type =='ack'){
if(messageQueue.length>0){
this.Dispatch(messageQueue.splice(0, 1)[0]);
}
}
public Dispatch(message) {
clientSocket.send(JSON.stringify(message));
}
public SendMessage(msg: Message, isHandshakeMessage : boolean){
// The if condition will be called only once for the first handshake message. For all other messages, the else condition will be called always.
if(isHandshakeMessage == true){
clientSocket.send(JSON.stringify(message));
}
else{
messageQueue.push(msg);
}
}
}
On the server side, we already have a ROUTER socket configured.
The above code is pretty straight forward. The SendMessage() function is essentially getting called for thousands of messages and the code works successfully but with load of memory consumption.
Problem :
Because the behavior of ZeroMQ is asynchronous, the client has to wait on the call back call ReceiveMessage() whenever it has to send a new message to ZeroMQ ROUTER (which is evident from the flow to the method Dispatch).
Based on our limited knowledge with TypeScript and usage of ZeroMQ with TypeScript, the problem is that because default thread running the typescript code (which creates the required 1000+ messages and sends to SendMessage()) continues its execution (creating and sending more messages) after sending the first message (handshake message essentially), unless all the 1000+ messages are created and sent to SendMessage() (which is not sending the data but queuing the data as we want to interpret the acknowledgement message sent by the router socket and only based on the acknowledgement we want to send the next message), the call does not come to the ReceiveMessage() call back method.
It is to say that the call comes to ReceiveMessage() only after the default thread creating and calling SendMessage() is done doing this for 1000+ message and now there is no other task for it to do any further.
Because ZeroMQ does not provide any synchronous mechanism of sending/receiving data using the ROUTER/DEALER, we had to utilize the queue as per the above code using a messageQueue object.
This mechanism will load a huge size messageQueue (with 1000+ messages) in memory and will dequeue only after the default thread gets to the ReceiveMessage() call at the end. The situation will only worsen if say we have 10000+ or even more messages to be sent.
Questions :
We have validated this behavior certainly. So we are sure of the understanding that we have explained above. Is there any gap in our understanding of either/or TypeScript or ZeroMQ usage?
Is there any concept like a blocking queue/limited size array in Typescript which would take limited entries on queue, and block any new additions to the queue until the existing ones are queues (which essentially applies that the default thread pauses its processing till the time the call back ReceiveMessage() is called which will de-queue entries from the queue)?
Is there any synchronous ZeroMQ methodology (We have used it in similar setup for C# where we pool on ZeroMQ and received the data synchronously)?.
Any leads on using multi-threading for such a scenario? Not sure if Typescript supports multi threading to a good extent.
Note : We have searched on many forums and have not got any leads any where. The above description may have multiple questions inside one question (against the rules of stackoverflow forum); but for us all of these questions are interlinked to using ZeroMQ effectively in Typescript.
Looking forward to getting some leads from the community.
Welcome to ZeroMQ
If this is your first read about ZeroMQ, feel free to first take a 5 seconds read - about the main conceptual differences in [ ZeroMQ hierarchy in less than a five seconds ] Section.
1 ) ... Is there any gap in our understanding of either/or TypeScript or ZeroMQ usage ?
Whereas I cannot serve for the TypeScript part, let me mention a few details, that may help you move forwards. While ZeroMQ is principally a broker-less, asynchronous signalling/messaging framework, it has many flavours of use and there are tools to enforce both a synchronous and asynchronous cooperation between the application code and the ZeroMQ Context()-instance, which is the cornerstone of all the services design.
The native API provides means to define, whether a respective call ought block, until a message processing across the Context()-instance's boundary was able to get completed, or, on the very contrary, if a call ought obey the ZMQ_DONTWAIT and asynchronously return the control back to the caller, irrespectively of the operation(s) (in-)completion.
As additional tricks, one may opt to configure ZMQ_SND_HWM + ZMQ_RCV_HWM and other related .setsockopt()-options, so as to meet a specific blocking / silent-dropping behaviours.
Because ZeroMQ does not provide any synchronous mechanism of sending/receiving data
Well, ZeroMQ API does provide means for a synchronous call to .send()/.recv() methods, where the caller is blocked until any feasible message could get delivered into / from a Context()-engine's domain of control.
Obviously, the TypeScript language binding/wrapper is responsible for exposing these native API services to your hands.
3 ) Is there any synchronous ZeroMQ methodology (We have used it in similar setup for C# where we pool on ZeroMQ and received the data synchronously) ?
Yes, there are several such :
- the native API, if not instructed by a ZMQ_DONTWAIT flag, blocks until a message can get served
- the native API provides a Poller()-object, that can .poll(), if given a -1 as a long duration specifier to wait for sought for events, blocking the caller until any such event comes and appears to the Poller()-instance.
Again, the TypeScript language binding/wrapper is responsible for exposing these native API services to your hands.
... Large memory consumption ...
Well, this may signal a poor resources management care. ZeroMQ messages, once got allocated, ought become also free-d, where appropriate. Check your TypeScript code and the TypeScript language binding/wrapper sources, if the resources systematically get disposed off and free-d from memory.