I am trying the find out what RPC semantics the gRPC library provides? Is it at-most once? Does it guarantee that an RPC call made by a client is not executed more than once on a server? I couldn't find this explicitly mentioned anywhere in the docs.
From what I understand, gRPC channels have an exponential back-off based retry mechanism of re-initiating TCP connections after transient failures. So, if a server fails after executing an RPC call but before responding, and later comes back up, a client RPC may end up getting executed twice.
Can someone throw more light on this?
gRPC provides at-most-once guarantee by default. Things like retries and similar can change that, but they are opt-in. Another example of this is marking your RPC idempotent or no-side-effects; you've explicitly told gRPC it is okay to replay the RPC.
gRPC may also replay when it is provided assurances elsewhere the RPC was not processed. We call this transparent retries, which we're implementing now, and is based on HTTP/2 semantics where a server tells us the RPC was not seen by the service application.
gRPC's exponential back-off for reconnecting (not retrying RPCs) doesn't change the behavior. Normal RPCs fail outright when the connection is known-bad, but wait-for-ready RPCs that have not been sent are simply delayed until the connection is ready. There is no "replay" of the RPC involved; it is only sent once.
From what I understand, gRPC channels have an exponential back-off based retry mechanism of re-initiating TCP connections after transient failures.
I think, that GRPC clients are pretty flexible, and you can try to disable retry attempts with https://godoc.org/google.golang.org/grpc#FailOnNonTempDialError
Though you'll have to care about retries on your own.
Related
When I connect to the socket server from the client side, which is considered react, every few seconds a repeated request is sent by the socket client. Generally, the requests are of get type and most of the time they are in pending mode. Sometimes the result of requests is 2.
What do you think is the problem of sending repeated requests after connecting or doing anything with the socket?
UPDATE
This problem occurs when I use namespace . I tried all the solutions but this problem was not solved.
image
This is expected behavior when the option used for transport is polling (long-polling).
What happens is, by default, the transport parameter is ["polling", "websocket"] (client, server), where the sequence of elements matters. So, the first connection attempt is made via polling (which is faster to start compared to websocket), and then (or in parallel, I don't know the working details) there is a connection attempt by websocket (this takes a little longer to establish but is faster for later communication).
If the websocket connection is successfully established, the communication will be carried in this way. But if an error occurs, or the connection takes a long time to be established, or this transport option is not present in the instance's parameters, then the communication will continue being carried out through polling, which are the various requests that remain pending. It is normal for them to remain pending, so they receive an update and are able to inform the requester immediately, without the need for several quick requests consulting the application's status.
Check the instance parameters you set for this connection to find out if transport via websocket is enabled. Be careful when using the socket server behind a reverse proxy, as this reverse proxy needs to be properly configured to accept websocket connections, otherwise it won't work.
You can check the websocket requests in the browser inspection, Network tab, by enabling the WS filter.
Here are some additional links for you to read more about:
https://socket.io/docs/v4/how-it-works/
https://socket.io/docs/v4/using-multiple-nodes/
https://socket.io/docs/v4/reverse-proxy/
https://ably.com/blog/websockets-vs-long-polling
Currently I am using stomp protocol to send messages to activeMQ and to listen to messages. This is done in Nodejs using stompit library.
When the application is having high CPU or Memory usage, it stops sending heartbeat to broker. So the broker redelivers the message which is currently being processed, leading to repetitive processing of the same message
On disabling heartbeat, the application seems to work fine but I am unsure of the further issues disabling heartbeat might cause. Even when the broker is stopped while sending messages, behaviour seems to be same with or without heartbeat
I have read that it is an optional parameter but I am unable to find out it's exact use cases
Can anyone mention scenarios where no heart beat can cause issues to the application?
Regarding the purpose of heart-beating the STOMP 1.2 specification just says:
Heart-beating can optionally be used to test the healthiness of the underlying TCP connection and to make sure that the remote end is alive and kicking.
Heart-beats potentially flow both from the client to the server and from the server to the client so the "remote end" referenced in the spec here could be the client or the server.
For the server, heart-beating is useful to ensure that server-side resources are cleaned up in a timely manner to avoid excessive strain. Server-side resources are maintained for all client connections and it helps the broker to be able to detect quickly when those connections fail (i.e. heart-beats aren't received) so it can clean up those resources. If heart-beating is disabled then it's possible that a dead connection would not be detected and the server would have to maintain its resources for that dead connection in vain.
For a client, heart-beating is useful to avoid message loss when performing asynchronous sends. Messages are often sent asynchronously by clients (i.e. fire and forget). If there was no mechanism to detect connection loss the client could continue sending messages async on a dead connection. Those messages would be lost since they would never reach the broker. Heart-beating mitigates this situation.
I'm connecting to a websocket endpoint using aiohttp's default WebsocketProtocol and out of the blue (after some time and multiple infinite loop iterations) I always get WSMsgType.ERROR with EofStream as data. To my understanding, this should not happen. I tried researching how to deal with this but have been relatively unsuccessful. Should I just close and reconnect to the endpoint? Is there a way to ensure this doesn't happen? Should I implement a specific handling algorithm?
It means connection closed by peer.
Internet is unstable transport, you should always be prepared to such situations.
Usually reconnection helps in cases like this.
Before I dive into the code, can someone tell me if there is any documentation available for confirmed delivery in Socket.IO?
Here's what I've been able to glean so far:
A callback can be provided to be invoked when and if a message is acknowledged
There is a special mode "volatile" that does not guarantee delivery
There is a default mode that is not "volatile"
This leaves me with some questions:
If a message is not volatile, how is it handled? Will it be buffered indefinitely?
Is there any way to be notified if a message can't be delivered within a reasonable amount of time?
Is there any way to unbuffer a message if I want to give up?
I'm at a bit of a loss as to how Socket.IO can be used in a time sensitive application without falling back to volatile mode and using an external ACK layer that can provide failure events and some level of configurability. Or am I missing something?
TL;DR You can't have reliable confirmed delivery unless you're willing to wait until the universe dies.
The delivery confirmation you seek is related to the theoretical Two Generals Problem, which is also discussed in this SO answer.
TCP manages the reliability problem by guaranteeing delivery after infinite retries. We live in a finite universe, so the word "guarantee" is theoretically dubious :-)
Theory aside, consider this: engine.io, the underpinnings of socket.io 1.x, uses the following transports:
WebSocket
FlashSocket
XHR polling
JSONP polling
Each of those transports is based upon TCP, and TCP is reliable. So as long as connections stay connected and transports don't change, each individual socket.io message or event should be reliable. However, two things can happen on the fly:
engine.io can change transports
socket.io can reconnect in case the underlying transport disconnects
So what happens when a client or your server squirts off a few messages while the plumbing is being fiddled with like that? It doesn't say in either the engine.io protocol or the socket.io protocol (at versions 3 and 4, respectively, as of this writing).
As you suggest in your comments, there is some acknowledgement logic in the implementation. But even simple digital communications has notrivial behavior, so I do not trust an unsupervised socket.io connection for reliable delivery for mission- or safety-critical operations. That won't change until reliable delivery is part of their protocol and their methods have been independently and formally verified.
You're welcome to adopt my policies:
Number my messages
Ask for a resend when in doubt
Do not mutate my state - client or server - unless I know I'm ready
In Short:
Guaranteed message delivery acknowledgement is proven impossible, but TCP guarantees delivery and order given "infinite" retries. I'm less confident about socket.io messages, but they're really powerful and easy to use so I just use them with care.
I ensured delivery using different strategies
I send data using socket including nonce in the message to prevent repeated message errors
The other party sends a confirmation of recived meassage or i resend after x seconds
I used a REST call by the client every 30 seconds to request all new messages sent by server to catch any dropped messages during transport
I got an app that web server re-direct some requests to backend servers, and the backend servers(Linux) will do complicated computations and response to web server.
For the tcp socket connection management between web server and backend server, i think there are two basic strategy:
"short" connection: that is, one connection per request. This seems very easy for socket management and simplify the whole program structure. After accept, we just get some thread to process the request and finally close this socket.
"long" connection: that is, for one tcp connection, there could be multi request one by one. It seems this strategy could make better use of socket resource and bring some performance improvement(i am not quite sure). BUT it seems this brings a lot of complexity than "short" connection. For example, since now socket fd may be used by multi-threads, synchronization must be involved. and there are more, socket failure process, message sequence...
Is there any suggestions for these two strategies?
UPDATE:, #SargeATM 's answer remind me that i should tell more about the backend service.
Each request is kind of context-free. Backend service can do calculation based on one single request message. It seems to be sth. stateless.
Without getting into the architecture of the backend which I think heavily influences this decision, I prefer short connections for stateless "quick" request/response type traffic and long connections for stateful protocols like a synchronization or file transfer.
I know there is some tcp overhead for establishing a new connection (if it isn't local host) but that has never been anything I have had to optimize in my applications.
Ok I will get a little into architecture since this is important. I always use threads not per request but by function. So I would have a thread that listened on the socket. Another thread that read packets off of all the active connections and another thread doing the backend calculations and a last thread saving to a database if needed. This keep things clean and simple. Easy to measure slow spots, maintain, and to optimize later when needed if needed.
What about a third option... no connection!
If your job description and job results are both of small size, UDP sockets may be a good idea. You have even less resources to manage, as there's no need to bound the request/response to a file descriptor, which give you some flexibility for the future. Imagine you have more backend services and would like to do some load balancing – a busy service can send the job to another one with UDP address of job submitter. The latter just waits for the result and doesn't care where you performed the task.
Obviously you'd have to deal with lost, duplicated and out of order packets, but as a reward you don't have to deal with broken connections. Out of order packets are probably not a big deal if you can fit the request and response in one UDP message, duplication can be taken care of by some job ids, and lost packet... well, they can be simply resent ;-)
Consider this!
Well, you are right.
The biggest problem with persistent connections will be making sure that app got "clean" connection from pool. Without any garbage left of data from another request.
There are a lot of ways to deal with that problem, but at the end it is better to close() tainted connection and open new one than trying to clean it...