I am trying to execute Jdbctemplate queries in parallel using completableFuture. Queries would run perfectly for first few times but after sometime spring fails to create bean if I execute queries. I have to restart program to make it work again.
Here's my code -
List<CompletableFuture<JSONObject>> completableFutureList = new ArrayList<>();
for(String query : queryList){
CompletableFuture<JSONObject> completableFuture = CompletableFuture
.supplyAsync(() -> jdbcTemplate.queryForList(query))
.thenApply(data -> createJson(data));
completableFutureList.add(completableFuture);
}
//collecting all the jsonObjects and putting in JSONArray tableList
CompletableFuture<Void> combinedFuture = CompletableFuture
.allOf(completableFutureList.toArray(new CompletableFuture<?>[0])).thenAccept(justVoid -> {
for (CompletableFuture<JSONObject> a : completableFutureList) {
JSONObject b = a.join();
tableList.put(b);
}
});
try {
combinedFuture.get();
} catch (InterruptedException e || ExecutionException e) {
//catch exception
}
The exception I'm getting is -
java.util.concurrent.CompletionException: org.springframework.boot.context.properties.ConfigurationPropertiesBindException: Error creating bean with name 'spring.datasource-org.springframework.boot.autoconfigure.jdbc.DataSourceProperties': Could not bind properties to 'DataSourceProperties' : prefix=spring.datasource, ignoreInvalidFields=false, ignoreUnknownFields=true; nested exception is java.lang.IllegalStateException: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext#d7d7ea8 has been closed already
Related
We are using spring integration DSL to call downstream services. But we are facing issues when 4XX series error code is returned by downstream service.
Below is the code snippet that we are using to call downstream services
#Bean
public IntegrationFlow getDataChannelFlow() {
return IntegrationFlows.from(MessageChannels.executor("getDataChannel", Executors.newCachedThreadPool()))
.enrichHeaders(h -> h.headerExpression(USER_REF, SRC_USER_REF)
.handle(Http.outboundGateway(endPoint1, auth2RestTemplate)
.uriVariable("data", PAYLOAD)
.httpMethod(HttpMethod.GET)
.transferCookies(true)
.expectedResponseType(String.class), e->e.advice(integrationAdvice).id("docIDAdvice"))
.transform(this::responseTransformer)
.enrichHeaders(header -> header.headerExpression("DOCUMENTS", PAYLOAD))
}
In case of 200 and 500 response from downstream services, our code is working fine but when we get 4XX series errors we are getting below exception in logs and control does not return back to transformer method
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8080/fetchUser": Attempted read from closed stream.; nested exception is java.io.IOException: Attempted read from closed stream.
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785)
at org.springframework.security.oauth2.client.OAuth2RestTemplate.doExecute(OAuth2RestTemplate.java:138)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:732)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:612)
at org.springframework.integration.http.outbound.HttpRequestExecutingMessageHandler.exchange(HttpRequestExecutingMessageHandler.java:196)
... 42 more
Caused by: java.io.IOException: Attempted read from closed stream.
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:141)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:118)
Few things that we noticed while debugging -
Spring's OAuth2ErrorHandler.java class differentiates between 4XX and 5XX series of errors
public boolean hasError(ClientHttpResponse response) throws IOException
{
return HttpStatus.Series.CLIENT_ERROR.equals(response.getStatusCode().series())
|| this.errorHandler.hasError(response);
}
In above code snippet hasError() method returns true for 4XX series of codes and due to this we are getting IOException when below code snippet is executed
protected <T> T doExecute(URI url, #Nullable HttpMethod method, #Nullable RequestCallback requestCallback,
#Nullable ResponseExtractor<T> responseExtractor) throws RestClientException {
Assert.notNull(url, "URI is required");
Assert.notNull(method, "HttpMethod is required");
ClientHttpResponse response = null;
try {
ClientHttpRequest request = createRequest(url, method);
if (requestCallback != null) {
requestCallback.doWithRequest(request);
}
response = request.execute();
handleResponse(url, method, response);
return (responseExtractor != null ? responseExtractor.extractData(response) : null);
}
catch (IOException ex) {
String resource = url.toString();
String query = url.getRawQuery();
resource = (query != null ? resource.substring(0, resource.indexOf('?')) : resource);
throw new ResourceAccessException("I/O error on " + method.name() +
" request for \"" + resource + "\": " + ex.getMessage(), ex);
}
finally {
if (response != null) {
response.close();
}
}
}
Our expectation is that control should return back to transformer method so that we will have the control over response processing.
Any suggestions on this issue would be much appreciated.
Is it possible to abort a Task in JavaFX? My Task could run into situations where I want to cancel the rest of the operations within it.
I would need to return a value, somehow, so I can handle the cause of the abort in the JFX Application Thread.
Most of the related answers I've seen refer to handling an already-canceled Task, but now how to manually cancel it from within the Task itself.
The cancel() method seems to have no effect as both messages below are displayed:
public class LoadingTask<Void> extends Task {
#Override
protected Object call() throws Exception {
Connection connection;
// ** Connect to server ** //
updateMessage("Contacting server ...");
try {
connection = DataFiles.getConnection();
} catch (SQLException ex) {
updateMessage("ERROR: " + ex.getMessage());
ex.printStackTrace();
cancel();
return null;
}
// ** Check user access ** //
updateMessage("Verifying user access ...");
try {
String username = System.getProperty("user.name");
ResultSet resultSet = connection.createStatement().executeQuery(
SqlQueries.SELECT_USER.replace("%USERNAME%", username));
// If user doesn't exist, block access
if (!resultSet.next()) {
}
} catch (SQLException ex) {
}
return null;
}
}
And example would be greatly appreciated.
Why not just let the task go into a FAILED state if it fails? All you need (I also corrected the errors with the type of the task and return type of the call method) is
public class LoadingTask extends Task<Void> {
#Override
protected Void call() throws Exception {
Connection connection;
// ** Connect to server ** //
updateMessage("Contacting server ...");
connection = DataFiles.getConnection();
// ** Check user access ** //
updateMessage("Verifying user access ...");
String username = System.getProperty("user.name");
ResultSet resultSet = connection.createStatement().executeQuery(
SqlQueries.SELECT_USER.replace("%USERNAME%", username));
// I am not at all sure what this is supposed to do....
// If user doesn't exist, block access
if (!resultSet.next()) {
}
return null;
}
}
Now if an exception is thrown by DataFiles.getConnection(), the call method terminates immediately with an exception (the remained is not executed) and the task enters a FAILED state. If you need access to the exception in the case that something goes wrong, you can do:
LoadingTask loadingTask = new LoadingTask();
loadingTask.setOnFailed(e -> {
Throwable exc = loadingTask.getException();
// do whatever you need with exc, e.g. log it, inform user, etc
});
loadingTask.setOnSucceeded(e -> {
// whatever you need to do when the user logs in...
});
myExecutor.execute(loadingTask);
I have a class like this
import java.util.concurrent.*;
public class TestClass<V> {
private final ConcurrentMap<String, Future<V>> requests;
private final ExecutorService executorService;
public TestClass(final ExecutorService executorService) {
this.executorService = executorService;
this.requests = new ConcurrentHashMap<>();
}
public V submitRequest(String cacheKey, Callable<V> request) throws Exception {
final Future<V> task = getOrCreateTask(cacheKey, request);
final V results;
try {
results = task.get();
} catch (InterruptedException | ExecutionException e) {
throw new IllegalStateException(String.format("Exception while executing request for key '%s'", cacheKey),
e);
} finally {
//Nullpointer here
requests.remove(cacheKey);
}
return results;
}
private synchronized Future<V> getOrCreateTask(String key, Callable<V> request) {
if (requests.containsKey(key)) {
return requests.get(key);
} else {
final Future<V> newTask = executorService.submit(request);
requests.put(key, newTask);
return newTask;
}
}
}
but sometimes under heavy load server throws nullpointer on requests.remove(cacheKey). I have read final when not escaped by this in the constructor is write guaranteed. i.e. other threads can see what is going on with my requests map.
Not sure how do i fix efficiently? Does not like that idea of adding synchronised on the whole parent level method
I'm not actually sure the NPE is where you're identifying it is unless cacheKey is null, which you could check for. The concurrentmap is set correctly so the requests field should never be null. Nevertheless, this code is not correctly synchronized. You are attempting to perform two operations in getOrCreateTask() that while under the synchronized keyword are not correctly synchronized with the map because the map is interacted with in submitRequest when you remove the values.
What is likely happening is that between the check ConcurrentMap#containsKey and ConcurrentMap#get that another thread has removed the value from the cache (ConcurrentMap#remove).
Thread A: Check Contains "foobar" => true
Thread B: Remove "foobar"
Thread A: Call get("foobar") => null
Thread A: Calls Future#get on a null pointer, which then throws a NPE.
Since you control the concurrentmap you can know you'll never have null values. In that case you should instead just call the #get method and check if the returned value is null. This will prevent another thread from removing the value between a contains/get pair since you'll be only accessing the map through one atomic operation.
I have created a POST endpoint using DropWizard.
#POST
#Timed
public String runPageSpeed(#RequestParam String request) {
try {
JSONObject requestJSON = new JSONObject(request);
JSONArray urls = requestJSON.getJSONArray("urls");
process(urls); // this takes around 10 minutes to complete
return "done";
} catch (Exception e) {
throw new WebApplicationException("failed", Response.Status.INTERNAL_SERVER_ERROR);
}
}
process(urls); takes around 10 minutes to complete, so if we call this endpoint, it takes more than 10 minutes to get the response.
I want process(urls); to run in the background after receiving the URLs from the request and immediately return a response to the user.
I tried the following code using threads:
#POST
#Timed
public String runPageSpeed(#RequestParam String request) {
try {
JSONObject requestJSON = new JSONObject(request);
JSONArray urls = requestJSON.getJSONArray("urls");
Thread thread = new Thread() {
public void run() {
process(urls); // this takes around 10 minutes to complete
}
};
thread.start();
return "done";
} catch (Exception e) {
throw new WebApplicationException("failed", Response.Status.INTERNAL_SERVER_ERROR);
}
}
This works, but are there any issues if I use this approach, especially at a high volume?
DropWizard users should promote using CompletableFuture for async handling as it is the safest for handling background processing. With CompletableFuture you can move the heavyweight task to a background thread and simultaneously continue with the lightweight task thus can also send back a response to the client.
#POST
#Timed
public String runPageSpeed(#RequestParam String request) {
try {
JSONObject requestJSON = new JSONObject(request);
JSONArray urls = requestJSON.getJSONArray("urls");
CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
try {
// perform heavyweight task
process(urls); // this takes around 10 minutes to complete
} catch (InterruptedException e) {
e.printStackTrace();
}
});
// perform lightweight task
return "done";
} catch (Exception e) {
throw new WebApplicationException("failed",
Response.Status.INTERNAL_SERVER_ERROR);
}
}
CompletableFuture helps in every aspects whether its using the return value of first complex task into second function or notifying on failure with the vast variety of methods it provides
runAsync()
supplyAsync()
thenApply()
thenAccept()
thenRun()
exceptionally()
handle()
You can also chain the CompletableFuture using thenCompose() and thenCombine() which is used when one task is dependent upon others.
I have a panel with a JTabbedpane and in every tab you can set parameters to execute a query. When one query is busy retrieving his data from the database, you can already open a new tab to set the new parameters. To avoid overload on the database only one query may be executed at once. But when you click execute the program must remember which queries to execute in the right order. During the execution a loader icon is shown and the GUI may not be frozen, because there is a stop button you can click to stop the execution.
I used a swingworker to avoid the GUI from blocking while executing the query and that works fine. But now I want to prevent the next query to start before the previous has finished. In a model, common for the whole panel, I initialized a semaphore: private final Semaphore semaphore = new Semaphore(1, true);
This is the code which starts the swingworker (I've added println commands to see which is started, stopped or finished)
private void doStoredQuery() {
try {
semaphore.acquire();
System.out.println(queryName + "started");
worker.execute();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
And this is my swingworker (initializeWorker() is called from the constructor of the main class):
private SwingWorker<StoredQueryDataModel, Integer> initializeWorker() {
worker = new SwingWorker<StoredQueryDataModel, Integer>() {
#Override
protected StoredQueryDataModel doInBackground() throws Exception {
try {
StoredQueryDataModel dataModel = null;
publish(0);
try {
dataModel = new StoredQueryDataModel(queryRunner, ldbName, queryName, params);
} catch (S9SQLException e) {
//
} catch (Throwable e) {
showErrorMessage(e);
}
return dataModel;
}
finally {
semaphore.release();
System.out.println(queryName + "finished");
}
}
#Override
protected void process(List<Integer> chunks) {
//ignore chunks, just reload loader icon
panel.repaint();
}
#Override
protected void done() {
String error;
try {
result = get();
error = null;
} catch (Exception e) {
error = e.getMessage();
}
if(result == null) {
semaphore.release();
System.out.println(queryName + " stopped");
}
if(error == null) {
// process result
}
else {
showErrorMessage(new Throwable(error));
}
}
};
return worker;
}
I've tried putting the acquire and release on other positions in the code, but nothing seems to work. I am bot in Swingworker and sempahores quite new... Can someone help?
I have found the problem: the semaphore had to be a static variable. In my code there were as many semaphores as there are tabs, which caused them to run at the same time instead of sequentially.