Is there any known problem regarding input message receive order in graylog? I have problems on graylog (using both gelf udp and syslog udp), messages are displayed in wrong order. Several messages are displayed with the same timestamp which is not possible, so we have lot of problems reading logs in messages section because messsages are received in same milisecond and not ordered as sent from client. I suppose there is a problem with input message buffering on graylog server side. Receive buffer is set to 262144.
Related
Intro
We're developing a system to support multiple real-time messages (chat) and updates (activity notifications).
That is, user A can receive via Web Socket messages for :
receiving new chat messages
receiving updates for some activity, for example if someone like their photo.
and more.
We use one single WebSocket connection to send all these different messages to the client.
However, we also support multiple applications/clients to be open by the user at the same time.
(i.e - user A connect on their web browser, and also from their mobile app, at the same time).
Architecture
We have a "Hub" that stores a map of UserId to a list of active websocket sessions.
(user:123 -> listOf(session#1, session#2))
Each client, once websocket connection is established, has its own Consumer which subscribes to a pulsar topic "userId" (e.g - user:123 topic).
If user A connected on both mobile and web, each client has its own Consumer to topic user:A.
When user A sends a new message from session #1 to user B, the flow is :
user makes a REST POST request to send a message.
service stores a new message to DB.
service sends a Pulsar message to topic user:B and user:A.
return 200 status code + created Message response.
Problem
If user A has two sessions open (two clients/websockets), and they send a message from session #1, how can we make sure only session #2 gets the message ?
Since user A has already received the 200 response with the created message in session #1, there's no need to send the message to him again by sending a message to his Consumer.
I'm not sure if it's a Pulsar configuration, or perhaps our architecture is wrong.
how can we make sure only session #2 gets the message ?
I'm going to address this at the app level.
Prepend a unique nonce (e.g. a guid) to each message sent.
Maintain a short list of recently sent nonces,
aging them out so we never have more than, say, half a dozen.
Upon receiving a message,
check to see if we sent it.
That is, check to see if its nonce is in the list.
If so, silently discard it.
Equivalently, name each connection.
You could roll a guid just once when a new websocket is opened.
Or you could incorporate some of the websocket's addressing
bits into the name.
Prepend the connection name to each outbound message.
Discard any received message which has "sender" of "self".
With this de-dup'ing approach
there's still some wasted network bandwidth.
We can quibble about it if you wish.
When the K-th websocket is created,
we could create K topics,
each excluding a different endpoint.
Sounds like more work than it's worth!
I'm using the stomp-client library and i want to know if it is possible to know if the message was delivered to the queue. Because im implementing a java service to do the dequeue of the messages and an node js to send the messages to the queue. the code bellow shows how I send the message to the queue.
this._stompClient.publish('/queue/MessagesQueue', messageToPublish, { })
When you send a SEND frame (i.e. publish a message) you can add a receipt header and then when you receive the RECEIPT frame from the broker you know it has successfully received the message. The STOMP specification says this about the receipt header:
Any client frame other than CONNECT MAY specify a receipt header with an arbitrary value. This will cause the server to acknowledge receipt of the frame with a RECEIPT frame which contains the value of this header as the value of the receipt-id header in the RECEIPT frame.
However, looking at the documentation for stomp-client I don't see any mention of how to receive RECEIPT frames. I actually would expect the ability to specify a callback on the publish method which was called when the RECEIPT frame is received. It doesn't appear that stomp-client supports working with receipts. Unfortunately that means there's no real way to confirm the message was received by the broker.
I recommend you find a more mature STOMP client implementation that supports receipts. For example stomp-js supports receipts.
I am using MQTT.js library as a MQTT client in my node app.
I am using Qos 2.
I just need to know if there is any property that controls the number of messages to send at once to the Emq broker.
In low band width like 10kb/s my application not sending data to Emq (The data size for each message is 25kb and there will be only 10 packets every time in the outgoing store).
I am building chat app.My server code:
socket.on('sendMessage',function(data){
var to=data.To;
var from=data.From;
var message=data.Message;
var sendData={
to:to,
from:from,
message:message,
messageId:randomId
}
io.to(to).emit("processmessage",sendData);
});
It is working well when receiver is online.But when I plugged out the ethernet cable and when someone sends a message to me the message doesn't come after I reconnected. (I am reconnecting within 10 seconds.)
Ps:After I reconnected new messages are coming without any problem.I lost the messages when I reconnecting.
How can I resolve this ?
You can implement a retry system that uses each messages unique id and timestamp to check whether it has reached the other side.
You would require that the receiver send back an acknowledgement that it has received a message of a particular ID.
Set a timeout (T) where you will retry a message after (T) seconds if it hasn't been acknowledged.
You can keep a buffer of messages that you send, then for every message that you confirm has been received (via an acknowledgement from the receiver), you can remove it from the buffer. After receiving an acknowledgement, you can also check the oldest message in your buffer to see if it is older than (T). If it is, you can retry the message and reset the timestamp. Continue checking the next oldest until you find one that is not older than (T). Then move on to sending next message.
This procedure will help your system recover messages that may have not made it to the receiver during downtime in the network.
I want to send group messages but when I am sending it first time its working fine but after when I received it, it occurs duplication of the messages. Number of duplication of sending messages is equal to number of received messages.
I tried to use Message Connection for sending message.
I have faced a similar issue.
The best solution for such is to give your messages an id value before sending them. You could encode in JSON.
Then check for the existence of the message id (if it doesnt exist) before storing/rendering the message.