I am using a groovy.sql.Sql.rows(sql, parameters) inside a quartz job to call a stored procedure. At times the stored proc times out but it doesn't propagate the exception to the calling code. For ex in the following code the exception never makes it to the try/catch block-
try{
res = db.rows(sql, parameter) //db is of type groovy.sql.Sql
}
catch(Exception e) {
log.error("error in calling the proc", e)
}
The only thing I see in the logs is a warning from sql.SQL which looks like this -
[quartzScheduler_Worker-1] WARN sql.Sql - Failed to execute: { call stored_prod_name(?, ?, ?, ?) } because: JZ006: Caught IOException: java.net.SocketTimeoutException: Read timed out
After this my entire JVM hangs! I would like to recover from a timeout but I can't seem to catch the exception. What am I doing wrong?
Related
I'm using NiFi ExecuteScript to call a Groovy script that extracts text from PDFs. When it fails to extract, an Exception is supposed to be thrown and the flowfile is redirected to REL_FAILURE. Some PDFs go through fine, and some give the error:
ExecuteScript[id=9a39e0cb-ebcc-31e4-a169-575e367046e9] Failed to process session due to javax.script.ScriptException: javax.script.ScriptException: java.lang.IllegalStateException: StandardFlowFileRecord[uuid=2d6540f7-b7a2-48c7-8978-6b90bbfb0ff5,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1538596326047-12, container=default, section=12], offset=2134, length=930225],offset=0,name=1 i-9 INS rev 87 05-07-87.pdf,size=930225] already in use for an active callback or an OutputStream created by ProcessSession.write(FlowFile) has not been closed: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: javax.script.ScriptException: java.lang.IllegalStateException: StandardFlowFileRecord[uuid=2d6540f7-b7a2-48c7-8978-6b90bbfb0ff5,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1538596326047-12, container=default, section=12], offset=2134, length=930225],offset=0,name=1 i-9 INS rev 87 05-07-87.pdf,size=930225] already in use for an active callback or an OutputStream created by ProcessSession.write(FlowFile) has not been closed
My (simplified) code is below:
def flowFile = session.get()
if(!flowFile) return
flowFile = session.write(flowFile, { inputStream, outputStream ->
try {
// Load PDF from inputStream and parses text into a JSON string
// If nothing can be extracted, throw an exception so the flowfile
// can be routed to REL_FAILURE and processed further down the NiFi pipeline
if(outputLength < 15) {
throw new Exception('No output, send to REL_FAILURE')
}
// Write the string to the flowFile to be transferred
outputStream.write(json.getBytes(StandardCharsets.UTF_8))
} catch (Exception e){
System.out.println(e.getMessage())
session.transfer(flowFile, REL_FAILURE)
}
} as StreamCallback)
session.transfer(flowFile, REL_SUCCESS)
It pretty closely follows the cookbook posted in the Hortonworks community forums, and the author even mentions that closing is handled automatically.
I think that the error is caused when a PDF fails to process. This throws an exception, which should be caught in the try{}catch{} and then be transferred to REL_FAILURE. Instead it appears that the catch{} is never getting called, so the outputStream object is never closed. It works as expected and gets caught just fine when I run the same Groovy code outside of NiFi.
If you want to try running it on your own server
NiFi template
full Groovy code.
Sample PDF
The try/catch should be outside the session.write() call rather than in the callback. Inside the callback, throw an IOException rather than an Exception, that should be propagated up through the session.write() and should enter your catch clause outside. Then you can transfer the flow file to failure (you shouldn't be allowed to transfer the flow file while you're writing to it).
I am developing a scheduled job to send message to Message queue using Quartz.net. The Execute method of IJob is not async. so I can't use async Task. But I want to call a method with await keyword.
Please find below my code. Not sure whether I am doing correct. Can anyone please help me with this?
private async Task PublishToQueue(ChangeDetected changeDetected)
{
_logProvider.Info("Publish to Queue started");
try
{
await _busControl.Publish(changeDetected);
_logProvider.Info($"ChangeDetected message published to RabbitMq. Message");
}
catch (Exception ex)
{
_logProvider.Error("Error publishing message to queue: ", ex);
throw;
}
}
public class ChangedNotificatonJob : IJob
{
public void Execute(IJobExecutionContext context)
{
//Publish message to queue
Policy
.Handle<Exception>()
.RetryAsync(3, (exception, count) =>
{
//Do something for each retry
})
.ExecuteAsync(async () =>
{
await PublishToQueue(message);
});
}
}
Is this correct way? I have used .GetAwaiter();
Policy
.Handle<Exception>()
.RetryAsync(_configReader.RetryLimit, (exception, count) =>
{
//Do something for each retry
})
.ExecuteAsync(async () =>
{
await PublishToQueue(message);
}).GetAwaiter()
Polly's .ExecuteAsync() returns a Task. With any Task, you can just call .Wait() on it (or other blocking methods) to block synchronously until it completes, or throws an exception.
As you have observed, since IJob.Execute(...) isn't async, you can't use await, so you have no choice but to block synchronously on the task, if you want to discover the success-or-otherwise of publishing before IJob.Execute(...) returns.
.Wait() will cause any exception from the task to be rethrown, wrapped in an AggregateException. This will occur if all Polly-orchestrated retries fail.
You'll need to decide what to do with that exception:
If you want the caller to handle it, rethrow it or don't catch it and let it cascade outside the Quartz job.
If you want to handle it before returning from IJob.Execute(...), you'll need a try {} catch {} around the whole .ExecuteAsync(...).Wait(). Or consider Polly's .ExecuteAndCaptureAsync(...) syntax: it avoids you having to provide that outer try-catch, by instead placing the final outcome of the execution into a PolicyResult instance. See the Polly doco.
There is a further alternative if your only intention is to log somewhere that message publishing failed, and you don't care whether that logging happens before IJob.Execute(...) returns or not. In that case, instead of using .Wait(), you could chain a continuation task on to ExecuteAsync() using .ContinueWith(...), and handle any logging in there. We adopt this approach, and capture failed message publishing to a special 'message hospital' - capturing enough information so that we can choose whether to republish that message again later, if appropriate. Whether this approach is valuable depends on how important it is to you never to lose a message.
EDIT: GetAwaiter() is irrelevant. It won't magically let you start using await inside a non-async method.
I want to verify exception throwing in one of my running threads. This is piece of my test code:
then:
def e = thrown(RequestFormatException)
e.message == "Incorrect first line: INCORRECT LINE"
When I run this I get next messages:
Exception in thread "Thread-1" by.westside.staircase.core.exception.RequestFormatException: Incorrect first line: INCORRECT LINE
at by.westside.staircase.core.util.HttpUtil.parseHttpRequest(HttpUtil.kt:19)
at by.westside.staircase.core.server.ServerThread.run(ServerThread.kt:26)
at java.lang.Thread.run(Thread.java:745)
Expected exception of type 'by.westside.staircase.core.exception.RequestFormatException', but no exception was thrown
at org.spockframework.lang.SpecInternals.checkExceptionThrown(SpecInternals.java:79)
at org.spockframework.lang.SpecInternals.thrownImpl(SpecInternals.java:66)
at by.westside.staircase.core.server.SyncServerSpec.should throw exception in incorrect first line case(SyncServerSpec.groovy:26)
Spock, like JUnit, can only assert on exceptions thrown from the thread executing the test, not "any thread in the application". Your exceptions are not caught by spock, and can't be asserted.
You can play with Thread.uncaughtExceptionHandler but you should probably unit-test the runnable executed in your thread - or implement some error handling in your business logic, and test this part of the code.
I think another option is to actually catch the exception in your test case and assert on that. here is a snippet of my code (written in Groovy Spock):
def exceptionThrown = false
def exceptionMessage
def thread = new Thread( {_ ->
try{
//code you are testing
} catch(Exception e) { // change this to the exception you want to catch
exceptionThrown = true
exceptionMessage = e.getMessage()
}
})
then: "the right exception should be thrown"
exceptionThrown
exceptionMessage = "I am thrown" //this should be your error message
I ran into the same issue and took a simple, hokey route. In the spirit of "good software is testable software" I added a flag and asserted on that, labeling it: // only for testing. Which, of course, will be ignored down the road.
thrown(RequestFormatException)
this should be in your first line after then: as this is the constraint imposed by spock.
Whenever thrown or notThrown is called in it should be the first statement.
Note: thrown and notThrown both return true and hence there should be no comparison operator as well.
Hence, In your case , it should be like below:
then:
thrown(RequestFormatException)
I'm constantly (since years) wondering the most senseful way to implement the following (it's kind of paradoxic for me):
Imagine a function:
DoSomethingWith(value)
{
if (value == null) { // Robust: Check parameter(s) first
throw new ArgumentNullException(value);
}
// Some code ...
}
It's called like:
SomeFunction()
{
if (value == null) { // Fail early
InformUser();
return;
}
DoSomethingWith(value);
}
But, to catch the ArgumentNullException, should I do:
SomeFunction()
{
if (value == null) { // Fail early
InformUser();
return;
}
try { // If throwing an Exception, why not *not* check for it (even if you checked already)?
DoSomethingWith(value);
} catch (ArgumentNullException) {
InformUser();
return;
}
}
or just:
SomeFunction()
{
try { // No fail early anymore IMHO, because you could fail before calling DoSomethingWith(value)
DoSomethingWith(value);
} catch (ArgumentNullException) {
InformUser();
return;
}
}
?
This is a very general question and the right solution depends on the specific code and architecture.
Generally regarding error handling
The main focus should be to catch the exception on the level where you can handle it.
Handling the exceptions at the right place makes the code robust, so the exception doesn't make the application fail and the exception can be handled accordingly.
Failing early makes the application robust, because this helps avoiding inconsistent states.
This also means that there should be a more general try catch block at the root of the execution to catch any non fatal application error which couldn't be handled. Which often means that you as a programmer didn't think of this error source. Later you can extend your code to also handle this error. But the execution root shouldn't be the only place where you think of exception handling.
Your example
In your example regarding ArgumentNullException:
Yes, you should fail early. Whenever your method is invoked with an invalid null argument, you should throw this exception.
But you should never catch this exception, cause it should be possible to avoid it. See this post related to the topic: If catching null pointer exception is not a good practice, is catching exception a good one?
If you are working with user input or input from other systems, then you should validate the input. E.g. you can display validation error on the UI after null checking without throwing an exception. It is always a critical part of error handling how to show the issues to users, so define a proper strategy for your application. You should try to avoid throwing exceptions in the expected program execution flow. See this article: https://msdn.microsoft.com/en-us/library/ms173163.aspx
See general thoughts about exception handling below:
Handling exceptions in your method
If an exception is thrown in the DoSomethingWith method and you can handle it and continue the flow of execution without any issue, then of course you should do those.
This is a pseudo code example for retrying a database operation:
void DoSomethingAndRetry(value)
{
try
{
SaveToDatabase(value);
}
catch(DeadlockException ex)
{
//deadlock happened, we are retrying the SQL statement
SaveToDatabase(value);
}
}
Letting the exception bubble up
Let's assume your method is public. If an exception happens which implies that the method failed and you can't continue execution, then the exception should bubble up, so that the calling code can handle it accordingly. It depends one the use-case how the calling code would react on the exception.
Before letting the exception bubble up you may wrap it into another application specific exception as inner exception to add additional context information. You may also process the exception somehow, E.g log it , or leave the logging to the calling code, depending on your logging architecture.
public bool SaveSomething(value)
{
try
{
SaveToFile(value);
}
catch(FileNotFoundException ex)
{
//process exception if needed, E.g. log it
ProcessException(ex);
//you may want to wrap this exception into another one to add context info
throw WrapIntoNewExceptionWithSomeDetails(ex);
}
}
Documenting possible exceptions
In .NET it is also helpful to define which exceptions your method is throwing and reasons why it might throw it. So that the calling code can take this into consideration. See https://msdn.microsoft.com/en-us/library/w1htk11d.aspx
Example:
/// <exception cref="System.Exception">Thrown when something happens..</exception>
DoSomethingWith(value)
{
...
}
Ignoring exceptions
For methods where you are OK with a failing method and don't want to add a try catch block around it all the time, you could create a method with similar signature:
public bool TryDoSomethingWith(value)
{
try
{
DoSomethingWith(value);
return true;
}
catch(Exception ex)
{
//process exception if needed, e.g. log it
ProcessException(ex);
return false;
}
}
I am seeing assertion error when I include a Throwable while invoking Log4j.error method. I do have Logger.class, PrintWriter.class, AuthenticationException.class in the #PreparateForTest block. I do not see the error if I do not pass the Throwable as an argument.
What am I missing in setting up mocks correctly?
Caused by: java.lang.AssertionError:
Unexpected method call AuthenticationException.printStackTrace(java.io.PrintWriter#2c64e8ad):
at org.junit.Assert.fail(Assert.java:93)
at com.xxx.yy.security.client.ClientTest.authenticateFail(ClientTest.java:282)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:66)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:310)
... 23 more
JUnit test code snippet is as below:
AuthenticationException mockAuthException = PowerMock
.createMock(AuthenticationException.class);
PrintWriter mockPrintWriter = PowerMock
.createMock(PrintWriter.class);
Logger mockLogger = PowerMock.createMock(Logger.class);
String message = "blah";
mockLogger.error(message, mockAuthException);
EasyMock.expectLastCall();
mockAuthException.printStackTrace(mockPrintWriter);
EasyMock.expectLastCall();
Code snippet that is causing the issue is as below:
try{
.
.
}catch (AuthenticationException ex) {
LOGGER.error("SOME MESSAGE HERE", ex);
throw ex;
}
you are getting an Unexpected method call error which can be solved as follows:
AuthenticationException mockAuthException = EasyMock.createNiceMock(AuthenticationException.class);
PrintWriter mockPrintWriter = EasyMock.createNiceMock(PrintWriter.class);
Logger mockLogger = EasyMock.createNiceMock(Logger.class);
String message = "blah";
mockLogger.error(message, mockAuthException);
EasyMock.expectLastCall();
mockAuthException.printStackTrace(mockPrintWriter);
EasyMock.expectLastCall();
The changes over here are using easymock instead of powermock,and creating a niceMock instead of normal mock.
The createMock(..) method is strict and can not identify if a method is internally called or not,but when you use createNiceMock(..) this check is ignored and you do not get a UnexpectedMethodCall error
Hope it helps!
Good luck!
LOGGER.error("SOME MESSAGE HERE", ex);
This line appears to be calling the printStacktrace method of the ex, i.e the AuthenticationException type. Since you've mocked it, you have to tell the behavior the object (the mock, technically) should behave in, when this call is encountered.
You have 3 choices, apart from relaxing the verification of course thru the use of nice mocks, assuming you really expect this exception to be thrown (your question is a bit vague about it):
See if you can expect the LOGGER.error("SOME MESSAGE HERE", ex);, i.e mock the LOGGER. The LOGGER appears to be a static variable, but you seem to be using Powermock, so I don't think you'll have any problems mocking it, but if you do have concerns mocking the LOGGER, read on
Tell Easymock how it's supposed to respond when it encounters the printStacktrace method call on the mock object mockAuthException by placing an expectation on it like this (printStacktrace being a void method) before you replay the mock:
mockAuthException.printStacktrace(); expectLastCall().andDoSomethingIfNeeded();
Don't mock the AuthenticationException type at all unless it's absolutely necessary - it does look like you can do without mocking it, say, by creating a Stub of the type.