I have an application which calls external REST API.
It reads pending data from DB, after API call updates the status and response msg in the DB.
Application Reads from DB -> AWS-ActiveMQ -> JMS Listener -> Service Class -> Call the external API -> Update the status and response in DB.
The APIs are getting called (order by msgId) in milliseconds gap which causing problem for dependent calls.
Table Structure:
msg_id
group_id
data
status
response
NB: Here calls belong to same group_id are dependent and should be sequential
(2nd api call should start after receiving 1st api call response).
Multiple group calls can be made as they are not dependent on each other.
Any suggestion/better-approach this scenario.
Approach-1:
Application Reads group_id from DB -> AWS-ActiveMQ (receives group_id) -> JMS Listener -> Service Class (#Async) -> Fetch Data based on group_id -> Call the external API -> Wait to get the response of previous call before making another call -> Update the status and response in DB.
Service Class (#Async): Multiple group requests can be processed parallel.
Any better approach for "Wait to get the response of previous call before making another call"
Related
I am trying to build a Spring Integration Flow with DSL where a part of the flow calls an existing service that will process data asynchronously and return the response on a channel. The service call returns a task ID that can be used as a correlation ID to obtain the correct response message on the channel.
I am unsure how to build a flow (which components to use) that will call the service (I assume with a service activator), then take the returned ID and then wait for a message on a different channel that has that ID in the correlation ID header (maybe some sort of aggregator?). I have googled and cannot find anything that seems similar.
Also, my flow will receive a request object that I would like to pair up with the response object to pass along the flow after the response is received.
Request -> service call -> returns task ID ---->|
| |---- (Request+Response) --> More processing
| (async) |
---------> Response ----------->|
for task ID
on task complete channel
You are correct. The best way to solve your task is really an aggregator pattern:
https://www.enterpriseintegrationpatterns.com/Aggregator.html
https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#aggregator
So, you probably need to use a header enricher to instead of a service activator to obtain that task ID in the reply and store it in a header for future correlation of this request and some reply later with the same task ID. Or if you have a property on the request object for this task ID, you can use a content enricher instead: https://docs.spring.io/spring-integration/docs/current/reference/html/message-transformation.html#content-enricher
Then you send this request object with the task ID to an aggregator where this task ID must be used as a correlation key. The group size of course is just 2 - request and reply.
Your async service must send a completion to the same aggregator's input channel with. When aggregator encounters a proper correlation key, it will complete the group of two messages and send a single one to its output channel.
UPDATE
The aggregator we are talking about must have its own input channel, so you can send a request with task ID for correlation and then from your async service a reply must be sent to the same channel. With Java DSL it is a matter of exposing that input channel for your convenience:
#Bean
IntegrationFlow aggregatorFlow() {
return f -> f
.aggregate(...)
.channel("correlatedReplyChannel")
}
This flow implicitly starts with a channel like aggregatorFlow.input. So, you use this name in a channel() definition of request and reply flows as the last EIP-method in their definitions.
I have Azure durable function run by timer trigger, which runs another function (UploadActivity) that does some http call to the external to Azure REST service. We know for sure that small percentage of all UploadActivity invocations end up in http error and exception risen, the rest are exception-free and upload some data to the remote http resource. Interesting finding I got is that Azure Insight's 'requests' collection contains only failed requests, and no successful one recorded
// gives no results
requests
| where success == "True"
// gives no results
requests
| where success <> "False"
// gives results
requests
| where success == "False"
I can't realize why. Here are some attributes of one of returned request with success=='False' if it helps to find why
operation_Name:
UploadActivity
appName:
/subscriptions/1b3e7d9e-e73b-4061-bde1-628b728b43b7/resourcegroups/myazuretest-rg/providers/microsoft.insights/components/myazuretest-ai
sdkVersion:
azurefunctions: 4.0.1.16815
'resource' is defined in Azure as http call to http-triggered function, but I have no http triggered functions in my app which makes things even more confusing, I think maybe these requests belong to Azure Insights calls, that could be also built based on Azure Functions
For a timer triggered function it is normal that there are no records in the requests collection of Application Insights. If it would be an http triggered function you would have 1. Only the request that triggers the function is recorded as a request in Application Insights. A timer trigger does not respond to a request.
Once the function is triggered all http requests (and all kind of other communication like calls to service busses etc.) executed by that function will be recorded as a dependency in the dependencies collection. This is by design and is how Application Insight works.
I am trying to make parallel queues for video processing.
However, I've faced problem in doing so. Below is the diagram of what i am trying to achieve.
Flow is:
User sends GET request to the /process endpoint (actually, GET is only for testing, it rather uses #MessagePattern to receive data from other service)
This request contains ModelDTO as well as sequence which is used for internal tracking
Controller imports private readonly _queueService: QueueService via constructor
It then calls this._queueService.process({ model, sequence })
QueueService imports #InjectQueue('video_processor') private readonly _processorQueue: Queue via constructor
QueueService simply calls this._processorQueue.add('process', data);
VideoProcessor imports private readonly _videoService: VideoService via constructor
Inside VideoProcessor there is a method with #Process('process') decorator
Inside this method I am awaiting for the result from the service with await this._videoService.configure(job.data).process()
And here is the problem:
Whenever I am trying to run 1 job at a time (sending single request and actually waiting for job to complete) everything works just fine
If I am queueing two jobs at the same time, for some reason, the console.log(this._videoData.id) inside VideoService will now return the ID of the second model rather than actual ID.
So far I have tried adding scope: Scope.TRANSIENT to almost all services with no luck. Seems like i just can't figure out where this scope should be added.
I am expecting for 10 jobs to be able to run in parallel, however, if I add more than 1 job to the queue, they start mixing in data from the other jobs.
I have a NodeJS endpoint that receives requests to gather data from a reporting engine.
To keep the request endpoint light and because some of the reports generated have a few steps (Gather data -> assemble report -> convert to PDF -> Email to relevant person) I want to separate the inbound request from the job itself.
Using AWS.SQS I can accept the request, put the variables into SQS and the respond with a 200 / 201.
What are some of the better practices around picking this job up on the other end?
If I were to trigger a lambda function would I have to wait for that function to complete before 200 / 201 can be sent? or can I:
Accept Request ->
Job to SQS ->
Initiate Lamba function ->
200 Response.
Alternatively what other options would be available to decouple the inbound request from the processing itself?
Here are a few options:
Insert the request in your SQS queue and return a 200 response immediately. Have a process on an EC2 server polling the SQS queue and performing the query when it gets a message out of SQS.
Invoke a Lambda function asynchronously, passing it the properties needed to perform the query, and return a 200 response immediately. Since you invoked the Lambda function asynchronously your NodeJS code that invoked the Lambda function doesn't wait for the function to complete.
An alternative to #2 is to send the request to an SNS topic, and have the SNS topic configured to invoke the Lambda function. This is probably the best method if you are using Lambda, because SNS will retry if the Lambda function fails for some reason.
I don't recommend combining SQS with Lambda because those two services don't integrate very well. SNS on the other hand does integrate very well with Lambda.
Also, you need to make sure your Lambda function invocations can be completed in under 5 minutes since that's currently the maximum time a Lambda function can execute. If you need individual steps to run for longer than 5 minutes you will need to use EC2 or ECS.
I think AWS Step Functions may be a good fit for your use case.
I have a flow that is similar to
IntegrationFlows.from(
Http.inboundGateway("/events")
.requestMapping(requestMappingSpec -> {
requestMappingSpec.methods(HttpMethod.POST);
requestMappingSpec.consumes(MediaType.APPLICATION_JSON_VALUE);
requestMappingSpec.produces(MediaType.APPLICATION_JSON_VALUE);
})
.requestPayloadType(PushEvent.class)
.errorChannel(ERROR_CHANNEL))
.channel(ReleaseFlow.REQUEST_CHANNEL)
.enrichHeaders(h -> h
.header(HttpHeaders.STATUS_CODE, HttpStatus.ACCEPTED))
.get();
When submitting multiple requests, a request will be processed by the flow attached to the REQUEST_CHANNEL and the following request will be processed by just the enrichedHeaders. My understanding is that the endpoints in this example should be processed serially ...
A request arrives at the /events endpoint
The request is processed by the flow listening to REQUEST_CHANNEL
The response from the previous flow will then have its headers enriched
The flow ends and the response is returned to the remote requestor
I appreciate your help in understanding why request n is processed by the channel (and not enrichHeaders()), request n + 1 is being processed by enrichHeaders() (and not the flow listening to the REQUEST_CHANNEL), request n + 2 processed by the channel (and not enrichHeaders()), ...
UPDATE 1
I am new to Spring Integration, but thought it was appropriate to collect events from a GitHub server and then create a release using an external service. The integration service would be responsible for determining the appropriate workflow based upon the data associated to the commit. The endpoint in question would receive a push event and forward it to the flow attached to the subscribable request channel (REQUEST_CHANNEL). This second flow will make a number of outbound requests to collect the appropriate release template and construct and start the pipeline.
UPDATE 2
I have not developed the second flow completely at this point, but here is a first version that simply performs a transformation based upon data associated with the commit.
return f -> f
.route(branchRouter(), mapping -> mapping
.subFlowMapping(true, t -> t
.transform(pushEventToEventTransformer()))
.subFlowMapping(false, n -> n
.transform(skip -> "")));
When the code has been submitted to a "monitored" branch the actions described in the first update will be performed. I am attempting to build the flows incrementally given my limited knowledge of the framework.
Subscribable channels are point-to-point by default, which means if there are two subscribers, messages will be distributed in round-robin fashion.
If you have another flow " ... attached to the REQUEST_CHANNEL " then you have two subscribers - that flow and the header-enricher.
Perhaps if you can describe what you are trying to do we can help.
With the header enricher after the channel, all that happens is the headers are enriched and the inbound message is returned to the gateway.
Perhaps you want this... ?
IntegrationFlows.from(
Http.inboundGateway("/events")
.requestMapping(requestMappingSpec -> {
requestMappingSpec.methods(HttpMethod.POST);
requestMappingSpec.consumes(MediaType.APPLICATION_JSON_VALUE);
requestMappingSpec.produces(MediaType.APPLICATION_JSON_VALUE);
})
.requestPayloadType(PushEvent.class)
.errorChannel(ERROR_CHANNEL))
.enrichHeaders(h -> h
.header(HttpHeaders.STATUS_CODE, HttpStatus.ACCEPTED))
.channel(ReleaseFlow.REQUEST_CHANNEL)
.get();
Which means all messages with the enriched headers will be sent to the channel.
EDIT
If you want the message to go to both places, you need to use a publish-subscribe channel.