I am trying to read about 3.9 million records from an upstream odata service using GetAllRequestBuilder. The code sets the pageSize to 5000 and uses the instance of HttpDestination to read from the upstream service.
The exact snippet that is executed :
CompletableFuture.runAsync(() ->{
// initial setup code for destination
Stream<Product> stream = new DefaultProductService()
.getAllProduct()
.withPreferredPageSize(5000)
.streamingEntities()
.execute(sourceDestination);
// ...
stream.peek(entry -> {log.info("entry: " + entry.toString());})
.forEach(entry -> publishOnEventBus(entry));
});
After a while, I receive an ODATADeserializationException. I cannot share the exact stack trace, but in the stack trace I see Caused by: com.google.gson.JsonIOException: java.io.IOException: Stream closed.
The exception is thrown in the class HttpEntityReader.
"com.sap.cloud.sdk.datamodel.odata.client.exception.ODataDeserializationException: A problem occurred while streaming the OData response.","tat com.sap.cloud.sdk.datamodel.odata.client.request.HttpEntityReader.lambda$stream$0(HttpEntityReader.java:121)"
This occurs abruptly, during multiple runs, sometimes 66k records are read, sometimes 400k, sometimes 14k.
Note : We use com.sap.cloud.sdk.datamodel to generate the service classes. We use getAllProduct that returns a GetAllRequestBuilder. The execute method itself returns deserialized entities.
Is there something that I can do to ensure that this exception does not occur and stream continues reading to the end? We are using sap-cloud-sdk version 3.48.0.
Related
Something strange is happening with my app, I am using SailsJs with official PostgreSQL driver and my data gets deleted. I don't have any pattern or list of specific events which deletes the data but I have following observations.
Few days back i was writing a function to destroy data and when I
executed that function it gave me an error I fixed the error and ran
my web app again and whoa data from one of my table was all gone.
Yesterday i wrote a function and I tried to get the HTTP call to that
function but it was giving me 500 server error, I started debugging it
and after executing my program 3 to 4 times with this error partial
data was deleted from one of my database table. Later the error was i
had a typo in URL.
If any of you guys had any experience with what is happening to me please let me know how to fix it? or at least help me on how to reproduce this issue ?
EDIT
I activated the logs and was waiting for it to happen again and it happened again and here is the log from sailsjs
In the logs I saw that its talking about alter.js sync strategy but i have selected it to be the safe strategy
It has happened to me quite a few times, when lifting the app and it is in the process of making changes to the db and it fails, sometimes due to ORM timeout.
What sails do when its lifting and needs to update the data structure is controlled in config/models.js migrate: 'alter', usually commented out, you get a prompt for what to do 1... 2... 3... (writing from the top of my head, i dont remember the actual messages) and a warning about using alter on a production system.
Changing
config/orm.js to have this
// config/orm.js
module.exports.orm = {
_hookTimeout: 60000 // I used 60 seconds as my new timeout
};
And for reasons I don't know changing config/pubsub.js
// config/pubsub.js
module.exports.pubsub = {
_hookTimeout: 60000 // I used 60 seconds as my new timeout
};
has helped me, avoid data loss.
The Azure Service Bus supports a built-in retry mechanism which makes an abandoned message immediately visible for another read attempt. I'm trying to use this mechanism to handle some transient errors, but the message is made available immediately after being abandoned.
What I would like to do is make the message invisible for a period of time after it is abandoned, preferably based on an exponentially incrementing policy.
I've tried to set the ScheduledEnqueueTimeUtc property when abandoning the message, but it doesn't seem to have an effect:
var messagingFactory = MessagingFactory.CreateFromConnectionString(...);
var receiver = messagingFactory.CreateMessageReceiver("test-queue");
receiver.OnMessageAsync(async brokeredMessage =>
{
await brokeredMessage.AbandonAsync(
new Dictionary<string, object>
{
{ "ScheduledEnqueueTimeUtc", DateTime.UtcNow.AddSeconds(30) }
});
}
});
I've considered not abandoning the message at all and just letting the lock expire, but this would require having some way to influence how the MessageReceiver specifies the lock duration on a message, and I can't find anything in the API to let me change this value. In addition, it wouldn't be possible to read the delivery count of the message (and therefore make a decision for how long to wait for the next retry) until after the lock is already required.
Can the retry policy in the Message Bus be influenced in some way, or can a delay be artificially introduced in some other way?
Careful here because I think you are confusing the retry feature with the automatic Complete/Abandon mechanism for the OnMessage event-driven message handling. The built in retry mechanism comes into play when a call to the Service Bus fails. For example, if you call to set a message as complete and that fails, then the retry mechanism would kick in. If you are processing a message an exception occurs in your own code that will NOT trigger a retry through the retry feature. Your question doesn't get explicit on if the error is from your code or when attempting to contact the service bus.
If you are indeed after modifying the retry policy that occurs when an error occurs attempting to communicate with the service bus you can modify the RetryPolicy that is set on the MessageReciver itself. There is an RetryExponitial which is used by default, as well as an abstract RetryPolicy you can create your own from.
What I think you are after is more control over what happens when you get an exception doing your processing, and you want to push off working on that message. There are a few options:
When you create your message handler you can set up OnMessageOptions. One of the properties is "AutoComplete". By default this is set to true, which means as soon as processing for the message is completed the Complete method is called automatically. If an exception occurs then abandon is automatically called, which is what you are seeing. By setting the AutoComplete to false you required to call Complete on your own from within the message handler. Failing to do so will cause the message lock to eventually run out, which is one of the behaviors you are looking for.
So, you could write your handler so that if an exception occurs during your processing you simply do not call Complete. The message would then remain on the queue until it's lock runs out and then would become available again. The standard dead lettering mechanism applies and after x number of tries it will be put into the deadletter queue automatically.
A caution of handling this way is that any type of exception will be treated this way. You really need to think about what types of exceptions are doing this and if you really want to push off processing or not. For example, if you are calling a third party system during your processing and it gives you an exception you know is transient, great. If, however, it gives you an error that you know will be a big problem then you may decide to do something else in the system besides just bailing on the message.
You could also look at the "Defer" method. This method actually will then not allow that message to be processed off the queue unless it is specifically pulled by its sequence number. You're code would have to remember the sequence number value and pull it. This isn't quite what you described though.
Another option is you can move away from the OnMessage, Event-driven style of processing messages. While this is very helpful you don't get a lot of control over things. Instead hook up your own processing loop and handle the abandon/complete on your own. You'll also need to deal some of the threading/concurrent call management that the OnMessage pattern gives you. This can be more work but you have the ultimate in flexibility.
Finally, I believe the reason the call you made to AbandonAsync passing the properties you wanted to modify didn't work is that those properties are referring to Metadata properties on the method, not standard properties on BrokeredMessage.
I actually asked this same question last year (implementation aside) with the three approaches I could think of looking at the API. #ClemensVasters, who works on the SB team, responded that using Defer with some kind of re-receive is really the only way to control this precisely.
You can read my comment to his answer for a specific approach to doing it where I suggest using a secondary queue to store messages that indicate which primary messages have been deferred and need to be re-received from the main queue. Then you can control how long you wait by setting the ScheduledEnqueueTimeUtc on those secondary messages to control exactly how long you wait before you retry.
I ran into a similar issue where our order picking system is legacy and goes into maintenance mode each night.
Using the ideas in this article(https://markheath.net/post/defer-processing-azure-service-bus-message) I created a custom property to track how many times a message has been resubmitted and manually dead lettering the message after 10 tries. If the message is under 10 retries it clones the message increments the custom property and sets the en queue of the new message.
using Microsoft.Azure.ServiceBus;
public PickQueue()
{
queueClient = new QueueClient(QUEUE_CONN_STRING, QUEUE_NAME);
}
public async Task QueueMessageAsync(int OrderId)
{
string body = JsonConvert.SerializeObject(OrderId);
var message = new Message(Encoding.UTF8.GetBytes(body));
await queueClient.SendAsync(message);
}
public async Task ReQueueMessageAsync(Message message, DateTime utcEnqueueTime)
{
int resubmitCount = (int)(message.UserProperties["ResubmitCount"] ?? 0) + 1;
if (resubmitCount > 10)
{
await queueClient.DeadLetterAsync(message.SystemProperties.LockToken);
}
else
{
Message clone = message.Clone();
clone.UserProperties["ResubmitCount"] = ++resubmitCount;
await queueClient.ScheduleMessageAsync(message, utcEnqueueTime);
}
}
This question asks how to implement exponential backoff in Azure Functions. If you do not want to use the built-in RetryPolicy (only available when autoComplete = false), here's the solution I've been using:
public static async Task ExceptionHandler(IMessageSession MessageSession, string LockToken, int DeliveryCount)
{
if (DeliveryCount < Globals.MaxDeliveryCount)
{
var DelaySeconds = Math.Pow(Globals.ExponentialBackoff, DeliveryCount);
await Task.Delay(TimeSpan.FromSeconds(DelaySeconds));
await MessageSession.AbandonAsync(LockToken);
}
else
{
await MessageSession.DeadLetterAsync(LockToken);
}
}
I am building a system with Spring Integration that processes all lines in a file as records. Because some of the String records are malformed I have multiple paths through the application via a Splitter and Aggregator combination (I'm building the Aggregator as we speak).
Further, some of the records are so malformed that they are effectively errors. However I have a requirement that all records must be processed therefore I must identify and log gross malformation errors separately and finish processing the file. In other words, I can not fail to process the file but instead must only log errors.
Aggregator
I intend to do achieve the goal of processing grossly malformed records by modifying the headers on the incoming message and passing the message on-ward to the Aggregator which can search for the existence of such a parameter. I'll effectively be hand coding in some error handling situations to my processors and aggregator.
My Release Strategy for the Aggregator will be when all messages are processed.
Code Extract
This code comes from a blog entry by Matt Vickery. He constructs an entirely new message (using MessageBuilder and transferring headers) whereas I will just add something to the Message headers. He includes this code in a gateway which subsequently transfers the Message onto the Aggregator.
public Message<AvsResponse> service(Message<AvsRequest> message) {
Assert.notNull(message, MISSING_MANDATORY_ARG);
Assert.notNull(message.getPayload(), MISSING_MANDATORY_ARG);
MessageHeaders requestMessageHeaders = message.getHeaders();
Message<AvsResponse> responseMessage = null;
try {
logger.debug("Entering AVS Gateway");
responseMessage = avsGateway.send(message);
if (responseMessage == null)
responseMessage = buildNewResponse(requestMessageHeaders,
AvsResponseType.NULL_RESULT);
logger.debug("Exited AVS Gateway");
return responseMessage;
}
catch (Exception e) {
return buildNewResponse(responseMessage, requestMessageHeaders,
AvsResponseType.EXCEPTION_RESULT, e);
}
}
Confusion (...at least, that which I know about)
My questions are as follows:
When I have such a release strategy (all messages processed), is that the best way to ensure all messages get through to the Aggregator?
When using an Aggregator it seems like in practical cases, it would be very common to need access to the Message in some previous step, as opposed to just passing and processing simple POJOs. Would that be true or is there something I should be doing to simplify my design so I can avoid Message
I came across a blog entry by Matt Vickery showing how he achieves what seems to be similar with an Aggregator. I'm using his work as a guide.
P.S. Per Artem Bilan's advice, I'm avoiding creating my own messages and letting SI turn them into Messages
There is no difference for Aggregator if payload is valid or not. Its general purpose is to build a List (by default) of payloads to one Message. And it does it via some sequenceDetails from MessageHeaders. It is first.
If you use Splitter, it is responsible to enrich each produced Message with default sequenceDetails. So, if you have this configuration:
<splitter/>
<aggregator/>
And if your inbound payload is List, you end up with List after aggregator as well.
I assume, that your Splitter just produces String payloads from File lines.
Then you pass each Message to some service/transformer.
The result of that you may pass to the Aggregator.
But as you say some of payloads are not valid and your processor fails with an Exception.
So, how about just try...catch within that POJO method and return some payload with error indicator, e.g. simple String "Oops!".
As I described before: the result of POJO method will be pushed to payload of the Message by Framework. And what is magic, that sequenceDetails will be there in the MessageHeaders too.
I don't see reason to write some custom ReleaseStrategy for this task, or even any other Aggregator's strategies...
Let me know, what you don't understand.
UPDATE
To add some error-indicator to message headers and don't throw Exception, it really will be simpler to build a new Message from code, not via some error-channel flow:
try {
return [GOOD_RESULT];
}
catch(Exception e) {
return MessageBuilder.withPayload(payload).setHeader("ERROR", e.getMessage()).build();
}
But in this case you should use <service-activator> instead of <transformer>, because the last one doesn't copy headers from inbound Message. And you really need them - setHeader for aggregator.
I am using the following transaction
var transactionScopeOptions = new TransactionOptions() { IsolationLevel = IsolationLevel.ReadUncommitted, Timeout = Timeout = new TimeSpan(0, 10, 0) };
using (TransactionScope transactionScope = new TransactionScope(TransactionScopeOption.Required, transactionScopeOptions))
{
/* update query here with data context execute command */
}
And I keep getting The operation is not valid for the state of the transaction exception with the inner exception of Transaction Timeout.
Locally I only get in 1 in 100,000 chances, but in the server it happens like every now and then. The application is running on MSMQ and WCF.
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single,
ConcurrencyMode = ConcurrencyMode.Multiple)]
I have tried the following:
<system.transactions>
<machineSettings maxTimeout="02:00:00" />
</system.transactions>
and setting the dataContext.CommandTimeout to 1 hour or 0 (infinite).
Changed the Connection Timeout=3600 in the app.config
I have tried almost everything I've read on Google but still no luck. Hope I can remove this problem in the server.
NOTE: update query usually last between 0 to 20 seconds (max) but since it's multithreading, it causes the error. If I ran the ones in the exception, it doesn't seem to get the error anymore either because the transaction doesn't time out.
EDIT:
All my queries have dirty read (nolock for reads and rowlock for updates)
EDIT:
Component Services > My Computer > Options > Transaction Timeout (seconds): 600
Then restarted "Distributed Transaction Coordinator" in services.
Still no luck
EDIT:
Timestamp was added when entering the transactionScope and when getting the exception (Transaction TimeOut) it seems it is not really time out related because it is kicked out less than a minute from entering the transaction scope (when I already specified the transaction to have a time out in 10 minutes).
This means it just gives out Transaction TimeOut exception even if it really didn't timed out.
EDIT:
Based on the last error, I tried adding the following, based from this website to my connection string:
Transaction Binding=Explicit Unbind
Though I am using SQL Server 2012 and latest Visual Studio Framework, so not sure it helps at all.
EDIT:
I also have the following:
In the app config:
<system.transactions>
<defaultSettings timeout="2:00:00" />
</system.transactions>
binding services:
<binding name="FooBinding"
deadLetterQueue="System"
maxReceivedMessageSize="524288"
exactlyOnce="true"
receiveErrorHandling="Move"
retryCycleDelay="00:00:30"
maxRetryCycles="120"
receiveRetryCount="0"
sendTimeout="00:01:00">
and
<serviceTimeouts transactionTimeout="00:10:00" />
EDIT:
I tried changing the config to the following, hoping it was a WCF timeout (default is 1 minute) but still get those exceptions: (extend timeout to 10 minutes)
<binding name="FooBinding"
deadLetterQueue="System"
maxReceivedMessageSize="524288"
exactlyOnce="true"
receiveErrorHandling="Move"
retryCycleDelay="00:00:30"
maxRetryCycles="120"
receiveRetryCount="0"
closeTimeout="00:10:00"
openTimeout="00:10:00"
receiveTimeout="00:10:00"
sendTimeout="00:10:00" />
I asked a related question with some code logic in another link.
We have an application where we use an H2 embedded database to store the data. We have a synchronized write method which does DB inserts. Since the H2 DB is a small Java embedded DB, we use "synchronized" on the write method to handle the transaction management in embedded DB rather than in DB.
But during heavy load, we could see that the write thread is getting hung. We are not sure for which resource, this thread is getting hung.
Please look at this snippet of code:
public synchronized int write(IEvent event) {
String methodName = "write";
Connection conn = null;
PreparedStatement updtStmt = null;
Statement stmt = null;
ResultSet rSet = null;
int status = 0;
try {
dbConnect.checkDBSizeExceed();
conn = dbConnect.getConnection();
updtStmt = conn.prepareStatement(insertQuery);
updtStmt.setString(1, (String) event.getAttributeValue());
......
updtStmt.setString(30, (String) event.getAttributeValue());
updtStmt.setBoolean(31, false);
status = updtStmt.executeUpdate();
}catch(SQLException ex){
logger.log(methodName,logger.print(ex),Logger.ERROR);
} catch(Exception ex){
logger.log(methodName,logger.print(ex),Logger.ERROR);
} finally {
try {
if (updtStmt != null)
updtStmt.close();
if (conn != null)
conn.close();
}catch(SQLException ex) {
logger.log(methodName,logger.print(ex),Logger.ERROR);
return status;
}
return status;
}
}
We have multiple write methods which can access this DB. From the code we could see that the code is straightforward. But we are not sure where the resource is locked.
Another problem is in the thread dump in the (Websphere) system.out, we could see the thread stacktrace as below.
[6/15/12 3:13:38:225 EDT] 00000032 ThreadMonitor W WSVR0605W: Thread "WebContainer : 3" (00000066) has been active for 632062 milliseconds and may be hung. There is/are 2
thread(s) in total in the server that may be hung.
at com.xxxx.eaws.di.agent.handlers.AuditEmbeddedDBHandler.store(Unknown Source)
at com.xxxx.eaws.di.agent.eventlogger.2LoggerImpl.logEvent(Unknown Source)
at com.xxxx.eecs.eventlogger.EventLoggerAdapter.logAuditEvent(EventLoggerAdapter.java:682)
at com.xxxx.eecs.eventlogger.EventLoggerAdapter.logEvent(EventLoggerAdapter.java:320)
at com.xxxx.eecs.eventlogger.EventLogger.logEventInternal(EventLogger.java:330)
at com.xxxx.eecs.eventlogger.EventLogger.logEvent(EventLogger.java:283)
at com.ibm.wps.auth.impl.ImplicitLoginFilterChainImpl.login(ImplicitLoginFilterChainImpl.java:55)
at com.ibm.wps.auth.impl.AuthenticationFilterChainHandlerImpl.invokeImplicitLoginFilterChain(AuthenticationFilterChainHandlerImpl.java:393)
at com.ibm.wps.auth.impl.InitialAuthenticationHandlerImpl.checkAuthentication(InitialAuthenticationHandlerImpl.java:204)
at com.ibm.wps.state.phases.PhaseManagerImpl.callInitialAuthenticationHandler(PhaseManagerImpl.java:240)
In the above stacktrace, I need to know the reason why I am getting "Unknown Source" in the stack trace. Those jars are available in the class path and we also have the H2.jar in the classpath. We are not sure why, if the thread gets in hung in H2, we are not able to get the thread stacktrace.
If not, I also need to know why the thread stack trace is showing "Unknown Source".
Appreciate your help.
Thanks in advance.
Are you using ejbs? How do you get the connection? Is it injected by the aop server? Do you retrieve it from jndi? You should Not synchronize the method.
Even if it is an embedded db you should rely on the app server facilities.
You need to configure the connection as a datasource, even if your db is in memory. If you want a serialized write on the db you need to configure the connection pool to the serialized ansi isolation level (there are 4 ansi isolation levels). In this way you should obtain the same effect in a managed environment (the app server) without the synchronized, that should be avoided inside an app server.
Unknown Source typically implies that the line numbers are available.
When you compile, the compiler can add in debug information like line numbers. If they are not present in the JAR or .class files, the JVM can't provide you that information.
HTH
It looks like the conn = dbConnect.getConnection(); is waiting for more than 60000ms.
The error thrown by WAS is because resource adapter has one mechanism poll period. It is considered the rate (in milliseconds) at which to poll the enterprise information system (EIS) event store for new inbound events. The poll cycle is established at a fixed rate, meaning that if execution of the poll cycle is delayed for any reason, the next cycle will occur immediately to “catch up”. During the poll period, the polling thread will be sleeping.
Once the time is calculated to 60000 milliseconds, the WebSphere Application Server thread monitor regards this polling thread as hung and throws the exception.