I'm new to Spring Integration. I have a simple flow which send request to external resource with several attemps.
IntegrationFlows.from(MY_CHANNEL)
.handle(myOutboundGateway, e -> e.advice(myRetryAdvice))
.wireTap(logResponse())
.get();
What I need to do is to take some action (saving data to a database) when calling external resource (after retrying) is not succesful (http status code is not 200 OK). How can I achieve that in my flow?
When all the attempts of the retry are exhausted, the RecoveryCallback is called.
See some sample here: How to get HTTP status in advice recovery callback. In that RecoveryCallback you can just return null and send a message to some channel for that storing to DB logic.
Another way is to have extra advice on top of that retry instead of RecoveryCallback. See its docs: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#expression-advice. This way when all the attempts are done, the exception is going to be bubbled and caught by that ExpressionEvaluatingRequestHandlerAdvice and its failureChannel. Pay attention to the trapException = true, so the error doesn't go back to the flow. Only to that failureChannel for your DB logic.
Related
I have an event-driven system with my application implemented in nodejs using cosmosdb (azure-cosmosdb-sqlapi) to store the events. I have planning data coming via various events from kafka broker, to complete a planning documnet I need to combine data from 5 different events, I combine the events using planning id. In such a system for upsert operation we encounter 412 Precondition failure error very often, as we receive many events for a planning id.
The official Microsoft link says to retry. I am confused about which approach to take to retry
Handle the error code using a try catch and call the method handling the event from catch block for n number of times. If the retry fails after n times throw the exception back to broker, in that case the event is send again by the broker. The issue with this is I am not able to add test for the same. Secondly I need to manage all the retry logic in my code base. The advantage here is that I know that an event is failed and I can retry directly without sending the event back to broker. Below is the the snippet from planningservice.ts handlePlanningEvents method
try {
await repository.upsert(planningEntry, etag)
} catch (e: any) {
if (e.code === 412 and retries) {
this.handlePlanningEvents(event, retries-1)
} else {
throw e // throws exception back to broker
}
}
Not using try catch to handle the error in service code but propagating the error to controller where it sends a 500 error response to broker and the broker sends the event again. The issue with this case is that it's longer path as compared to using try catch where I can retry directly. But the advantage here is that I don't to worry about retry logic anymore its handled by broker, less and cleaner code.
Not sure which approach to take, also open to other suggestions.
I want to treat 4xx HTTP responses from a function app (e.g. a 400 response after sending a HTTP request to my function app) as failures in application insights. The function app is being called by another service I control so a 4xx response probably means an implementation error and so I'd like to capture that to ultimately run an alert on it (so I can get an email instead of checking into Azure everytime).
If possible, I'd like it to appear here:
If not, are there any alternative approaches that might fit my use case?
Unless an unhandled exception occurs the function runtime will mark the invocation as succesful, whether the status code is actually denoting an error or not. Since this behavior is defined by the runtime there are 2 things you can do: throw an exception in the code of the function and/or remove exception handling so the invocation is marked as not succesful.
Since you ultimately want to create an alert, you better alert on this specific http status code using a "Custom log search" alert
requests
| where toint(resultCode) >= 400
I am new to spring integration.
I have a flow on which I need to perform an http or a tcp call depending on some conditions.
The problem I am focused on is related to the http call.
The rest endpoint called needs an accessToken as header parameter for authentication that is provided by a spring service that has 2 methods getCurrentAccessToken() and refreshAccessToken(). I want to call the method refresh accessToken only when the currentAccessToken is expired.
What I would like to do is to add the following logic when performing the call to the rest api:
If the token is expired the rest endpoint returns a 401 and I would like to intercept in the flow this error and retry the request by adding a refreshed access token.
#Bean
public IntegrationFlow clientIn(AbstractServerConnectionFactory server,
AbstractClientConnectionFactory client, LogService logService) {
return IntegrationFlows.from(Tcp.inboundAdapter(client)
.enrichHeaders(t -> t.headerFunction(IpHeaders.CONNECTION_ID, message -> this.client, true))
.log(msg -> "client: " + logService.log(msg))
.<byte[], Boolean>route(this::shouldForwardToHttp,
mapping -> mapping.subFlowMapping(true, sf -> sf
.enrichHeaders(t -> t.header("Content-Type", MimeTypeUtils.APPLICATION_JSON_VALUE))
.<byte[], RequestMessage>transform(this::buildRequestFromMessage)
.<RequestMessage, HttpEntity>transform(this::getHttpEntity)
.handle(Http.outboundGateway(restUrl).httpMethod(HttpMethod.POST)
.expectedResponseType(ResponseMessage.class))
.<ResponseMessage, byte[]>transform(p -> this.transformResponse(p))
.handle(Tcp.outboundAdapter(client))).subFlowMapping(false,
t -> t.handle(Tcp.outboundAdapter(server).retryInterval(1000))))
.get();
}
HttpEntity getHttpEntity(RequestMessage request) {
MultiValueMap<String, String> mv = new HttpHeaders();
mv.add("accessToken", tokenProvider.getCurrentAccessToken());
HttpEntity entity = new HttpEntity(request, mv);
return entity;
}
I have tried by adding a requestHandlerRetry advice and redirecting it to a recoveryChannel, but I was not able to return something to the caller flow in order to get the response with the status code and retry the call with the new accessToken.
Any idea on how I can implement this?
I don't think you need a retry advice since you definitely are simulating it via catching that 401 exception and calling a service back with refreshed token. Sounds more like recursion. To achieve it properly I would suggest to take a look into an ExpressionEvaluatingRequestHandlerAdvice: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#message-handler-advice-chain. Yes, it is similar to the retry one and it also has that failureChannel, but there is no built-in retry since we are going to simulate it calling the same endpoint again and again when necessary.
To simplify a recursion logic, I would extract that .handle(Http.outboundGateway(restUrl).httpMethod(HttpMethod.POST) .expectedResponseType(ResponseMessage.class)) into a separate flow and use a gateway() with an input channel for that flow in the main flow instead.
A failureChannel sub-flow should re-route its message back to the input of the gateway flow.
What is the most important part in this logic is to carry on all the original request message headers which includes a required for the gateway logic replyChannel.
See more docs about gateways: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#gateway.
When an ExpressionEvaluatingRequestHandlerAdvice sends a message to the failureChannel, it comes as an ErrorMessage with a MessageHandlingExpressionEvaluatingAdviceException as a payload. The message which causes a failure and has all the required headers is there in the getFailedMessage() property. So, you take that message, request for fresh token, add it into headers of a new message based on that original. In the end you send this new message to the input channel of the IntegrationFlow for an HTTP request. When all is good, the result of the HTTP call is going to be forwarded to the mentioned replyChannel from headers and in therefore to the main flow for next steps.
I have a simple IntegrationFlow sending SOAP messages with MarshallingWebServiceOutboundGateway. The deal is that, normally, the target service responds with empty message. It can of course return soap fault, but, on success, we expect an empty response. Now the problem is that if we get an empty response, SI recognizes it as "no response" and does not execute steps that are after the gateway in the IntegrationFlow. This happens even if gateway is set with setIgnoreEmptyResponses(false). According to documentation, it only works if it received an empty String, but response returned from the service is represented as null in my case.
I want to be able to continue the flow no matter if external service responds with empty or non-empty response. What would be the best way to achieve that?
EDIT1: It feels especially odd, because it stops execution of the IntegrationFlow even if both handlers are subscribed to publishSubscribeChannel (instead of being directly on the main flow). I would expect all subscribers to be executed unless the previous one produced an exception, but this is not what happens.
MarshallingWebServiceOutboundGateway outboundGateway = new MarshallingWebServiceOutboundGateway(
gatewaysUtils.buildOutboundUriToX(webServiceProperties.getServicePath()),
jaxb2Marshaller, messageFactory);
outboundGateway.setIgnoreEmptyResponses(false);
IntegrationFlows.from(channelName())
.transform(getKafkaMarshaller()::deserialize)
.handle(getOutboundGateway())
.handle(messageStatusRegistrator.registerMessageStatus())
.get();
EDIT2: Having tried the publishSubscribeChannel again, it worked this time around. It feels a bit hacky to me, but apparently it does work without any side-effects (for now). Fixed flow below. One thing to be careful about is scenario, when the service returns a non-empty, non-error response, which can potentially cause "output-channel or replychannel header available" if you do not provide the header or replyChannel explicitly.
IntegrationFlows.from(channelName())
.transform(getKafkaMarshaller()::deserialize)
.publishSubscribeChannel(channel -> channel
.subscribe(subflow -> subflow
.handle(getOutboundGateway()))
.subscribe(subflow -> subflow
.handle(messageStatusRegistrator::registerMessageStatus)))
.get();
I want to do something when/if an insert operation on Azure Table Storage fails. Assume that I want to return false from the below code when I receive an error. _table is of type CloudTable and the code below works.
public bool InsertEntity(TableEntity entity)
{
var insertOperation = TableOperation.Insert(entity);
var result = _table.Execute(insertOperation);
return (result.HttpStatusCode == (int)System.Net.HttpStatusCode.OK);
}
I get the result 203 when the operation succeeds. But there are other possible results like "200 OK".
How can I write a piece of code that will allow me to understand from the status code that something went wrong?
Using the .NET SDK, any situation that needs to be handled will throw an exception. i.e. Any status code that is not 2xx will cause an exception.
To handle situations where something went wrong, I don't have to manually check the status code of the result for every request. All I have to do is to write exception handling code. Like below:
try
{
var result = _table.Execute(insertOperation);
}
catch (Exception)
{
Log("Something went wrong in table operation.");
}
From this page:
REST API operations for Azure storage services return standard HTTP
status codes, as defined in the HTTP/1.1 Status Code Definitions.
So every successful operation against table service will return 2XX status code. To find out about the exact code returned, I would recommend checking out each operation on the REST API Documentation page. For example, Create Table operation returns 201 status code if the operation is successful.
Similarly, for errors in table service you will get error code in 400 range (that would mean you provided incorrect data e.g. 409 (Conflict) error if you're trying to create a table which already exists) or in 500 range (for example, table service is unavailable). You can find the list of all Table Service Error Codes here: https://msdn.microsoft.com/en-us/library/azure/dd179438.aspx.
Basically, any return in 2xx is "OK". In this example:
https://msdn.microsoft.com/en-us/library/system.net.httpstatuscode%28v=vs.110%29.aspx
203 Non-Authoritative Information:
Indicates that the returned metainformation is from a cached copy
instead of the
origin server and therefore may be incorrect.
This Azure white paper elaborates further:
http://go.microsoft.com/fwlink/?LinkId=153401
9.6.5 Error handling and reporting
The REST API is designed to look like a standard HTTP server interacting with existing HTTP clients
(e.g., browsers, HTTP client libraries, proxies, caches, and so on).
To ensure the HTTP clients handle errors properly, we map each Windows
Azure Table error to an HTTP status code.
HTTP status codes are less expressive than Windows Azure Table error
codes and contain less information about the error. Although the HTTP
status codes contain less information about the error, clients that
understand HTTP will usually handle the error correctly.
Therefore, when handling errors or reporting Windows Azure Table
errors to end users, use the Windows Azure Table error code along with
the HTTP status code as it contains more information about the error.
Additionally, when debugging your application, you should also consult
the human readable element of the XML error
response.
These links are also useful:
Microsoft Azure: Status and Error Codes
Clean way to catch errors from Azure Table (other than string match?)
If you are using Azure Storage SDK accessing Azure Table Storage, the SDK would throw a StorageException on the client side for unexpected Http Status Codes returned from the table storage service. To extract the actual HttpStatusCode you would need to wrap your code in a try {} catch(StorageException ex){} block. And then parse the actual exception object to extract the HttpStatusCode embedded in it.
Have a look at Azure Storage Exception parser I implemented in Nuget:
https://www.nuget.org/packages/AzureStorageExceptionParser/
This extracts HttpStatusCode and many other useful fields from Azure StorageExceptions. You can use the same library accross table, blob, queue clients etc. as they all follow the same StorageException pattern.
Note that there will be some exceptions thrown by the Azure Storage SDK that are not StorageExceptions, those are mostly client side request validation type of exceptions and naturally they do not contain any HttpStatusCode. (Hence you would need to have a catch for specifically StorageExceptions to extract HttpStatusCode s).
As a separate note, Azure Storage SDK has a fairly robust retry mechanism for failed requests. Below is the snippet from SDK source code where they decide if the failed response is retrieable or not.
https://github.com/Azure/azure-storage-net/blob/master/Lib/Common/RetryPolicies/ExponentialRetry.cs
if ((statusCode >= 300 && statusCode < 500 && statusCode != 408)
|| statusCode == 501 // Not Implemented
|| statusCode == 505 // Version Not Supported
|| lastException.Message == SR.BlobTypeMismatch)
{
return false; //aka. do not Retry if w are here otherwise Retry if within max retry count..
}