I'm using asyncTask to download some files over the internet. This is the code I've written which works
downloadUrl task = new downloadUrl(url1,"jsonData1","/sdcard/appData/LocalJson/jsonData1",context);
task.execute();
downloadUrl task1 = new downloadUrl(url2,"jsonData2","/sdcard/appData/LocalJson/jsonData2",context);
task1.execute();
downloadUrl task2 = new downloadUrl(url3,"jsonData3","/sdcard/appData/LocalJson/jsonData3",context);
task2.execute();
downloadUrl task3 = new downloadUrl(url4,"jsonData4","/sdcard/appData/LocalJson/jsonData4",context);
task3.execute();
Now, the tasks run in parallel considering the UI-Thread but they run serialized between one another, which is time consuming. So instead I've tried to execute them on the executor But the thing is that this way I'm missing some files, meaning that when they run serialized I end up with 38 files downloaded while the run on the Executor I end up with 20. I'm pretty sure that is, because I messed up something in the multi-threading code So I'll post it that to:
#Override
protected String doInBackground(String... args) {
downloadAndStoreJson(url,targetFolder);
JSONObject jsonObj = loadJSONObject(pathForLoad);
try {
processJsonData(jsonObj);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return "done";
}
#Override
protected void onPostExecute(String result) {
s(targetFolder+" Finished!");
++mutex;
progressBar.setProgress(25*mutex);
if(mutex==4){
mutex=0;
progressBar.setProgress(100);
progressBar.dismiss();
s(monuments.size());
Intent intent = new Intent (getApplicationContext(),NextClass.class);
intent.putExtra("monuments", monuments);
startActivity(intent);
}
}
private void downloadAndStoreJson(String url,String tag){
JSONParser jParser = new JSONParser();
JSONObject json = jParser.getJSONFromUrl(url);
String jsonString = json.toString();
byte[] jsonArray = jsonString.getBytes();
File fileToSaveJson = new File("/sdcard/appData/LocalJson/",tag);
BufferedOutputStream bos;
try {
bos = new BufferedOutputStream(new FileOutputStream(fileToSaveJson));
bos.write(jsonArray);
bos.flush();
bos.close();
} catch (FileNotFoundException e4) {
// TODO Auto-generated catch block
e4.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
finally {
jsonArray=null;
jParser=null;
System.gc();
}
}
private JSONObject loadJSONObject(String path){
JSONObject jsonObj = null;
File readFromJson = new File(path);
byte[] lala;
try {
lala= org.apache.commons.io.FileUtils.readFileToByteArray(readFromJson);
s("---------------"+lala.length);
String decoded = new String(lala, "UTF-8");
jsonObj = new JSONObject(decoded);
} catch (IOException e5) {
// TODO Auto-generated catch block
e5.printStackTrace();
} catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return jsonObj;
}
and processJsonData is a long method which parses the json files, creates objects and then stores them in an ArrayList, that's where a problem might exist.
You need to make sure your code is Reentrant, meaning it must be possible to run it by several threads at the same time. Or if some code is used to syncronize the execution between your threads you need to make sure it is synchronized.
Looking at your code I see that the mutex is a static variable, which you use to keep track of your threads. Make sure that the operation on the mutex is synchronized, just to keep it clean. But that will not cause you problem...
I dont see your error in this code-snippet, either I fail to see the problem or it might be located in some other methods? Can you please share "downloadAndStoreJson"?
Related
My purpose is to upload the file on primary location synchronously and same time upload same file on multiple secondary locations asynchronously. So can you please help me to implement multi threading for secondary locations writing(e.g. same file to write multiple paths simultaneously while reading it)
Below is a code to read from input stream
List<FileUploadMultiLocator> fileUploadList = new ArrayList<>();
FileUploadMultiLocator fum1 = new FileUploadMultiLocator("fileserver0", new BufferedOutputStream(new FileOutputStream(new File(reposMap.get("D:\\Harisingh\\FileUpload\\Destination1.txt")))));
FileUploadMultiLocator fum2 = new FileUploadMultiLocator("fileserver0", new BufferedOutputStream(new FileOutputStream(new File(reposMap.get("D:\\Harisingh\\FileUpload\\Destination2.txt")))));
fileUploadList.add(fum1);
fileUploadList.add(fum2);
while((len=channel.read(byteBuffer))>=0)
{
itrCount++;
totalBytes+=len;
baOS.write(byteBuffer.array(),0,len);
if(itrCount>=maxIterateCnt)
{
//primary location writing
bFout.write(baOS.toByteArray(),0,totalBytes);
// this is for secondary location writing
for (FileUploadMultiLocator fileUploadMultiLocator : fileUploadList) {
fileUploadMultiLocator.baOS = baOS;
fileUploadMultiLocator.totalBytes = totalBytes;
new Thread(fileUploadMultiLocator).start();
}
totalBytes=0;
baOS.reset();
itrCount=0;
}
byteBuffer.clear();
}
This is my runnable class to write multiple BufferedOutputStream same time
public class FileUploadMultiLocator implements Runnable{
public FileUploadMultiLocator(String fileserver,BufferedOutputStream bFout) {
// TODO Auto-generated constructor stub
this.fileserver = fileserver;
this.bFout = bFout;
}
#Override
public void run() {
// TODO Auto-generated method stub
try
{
bFout.write(baOS.toByteArray(),0,totalBytes);
bFout.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I have created a POST endpoint using DropWizard.
#POST
#Timed
public String runPageSpeed(#RequestParam String request) {
try {
JSONObject requestJSON = new JSONObject(request);
JSONArray urls = requestJSON.getJSONArray("urls");
process(urls); // this takes around 10 minutes to complete
return "done";
} catch (Exception e) {
throw new WebApplicationException("failed", Response.Status.INTERNAL_SERVER_ERROR);
}
}
process(urls); takes around 10 minutes to complete, so if we call this endpoint, it takes more than 10 minutes to get the response.
I want process(urls); to run in the background after receiving the URLs from the request and immediately return a response to the user.
I tried the following code using threads:
#POST
#Timed
public String runPageSpeed(#RequestParam String request) {
try {
JSONObject requestJSON = new JSONObject(request);
JSONArray urls = requestJSON.getJSONArray("urls");
Thread thread = new Thread() {
public void run() {
process(urls); // this takes around 10 minutes to complete
}
};
thread.start();
return "done";
} catch (Exception e) {
throw new WebApplicationException("failed", Response.Status.INTERNAL_SERVER_ERROR);
}
}
This works, but are there any issues if I use this approach, especially at a high volume?
DropWizard users should promote using CompletableFuture for async handling as it is the safest for handling background processing. With CompletableFuture you can move the heavyweight task to a background thread and simultaneously continue with the lightweight task thus can also send back a response to the client.
#POST
#Timed
public String runPageSpeed(#RequestParam String request) {
try {
JSONObject requestJSON = new JSONObject(request);
JSONArray urls = requestJSON.getJSONArray("urls");
CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
try {
// perform heavyweight task
process(urls); // this takes around 10 minutes to complete
} catch (InterruptedException e) {
e.printStackTrace();
}
});
// perform lightweight task
return "done";
} catch (Exception e) {
throw new WebApplicationException("failed",
Response.Status.INTERNAL_SERVER_ERROR);
}
}
CompletableFuture helps in every aspects whether its using the return value of first complex task into second function or notifying on failure with the vast variety of methods it provides
runAsync()
supplyAsync()
thenApply()
thenAccept()
thenRun()
exceptionally()
handle()
You can also chain the CompletableFuture using thenCompose() and thenCombine() which is used when one task is dependent upon others.
I need to create the Structure and Template progrmatically through java code.I used following code snippets.
Structure:
public void createStructure(String userName,long userId){
log_.info("Inside create structure ");
long structureId=115203;
DDMStructure ddmStructure=DDMStructureLocalServiceUtil.createDDMStructure(structureId);
ddmStructure.setName("MigrationStructure");
ddmStructure.setDescription("This Structure created programatically");
ddmStructure.setUserId(userId);
ddmStructure.setUserName(userName);
File fXmlFile = new File("D:/FilesDataMigration/structure.xml");
try {
Document document = SAXReaderUtil.read(fXmlFile);
ddmStructure.setDocument(document);
DDMStructureLocalServiceUtil.addDDMStructure(ddmStructure);
}catch (DocumentException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SystemException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
log_.info("Inside create structure done");
}
Template:
public void createTemplate(String userName,long userId){
log_.info("Inside create template ");
long templateId=12504;
DDMTemplate ddmTemplate=DDMTemplateLocalServiceUtil.createDDMTemplate(templateId);
ddmTemplate.setName("MigrationTemplate");
ddmTemplate.setDescription("This Template created programatically");
ddmTemplate.setUserId(userId);
ddmTemplate.setUserName(userName);
try {
BufferedReader br = new BufferedReader(new FileReader("D:/FilesDataMigration/template.txt"));
StringBuilder sb = new StringBuilder();
String line = br.readLine();
while (line != null) {
sb.append(line);
sb.append(System.lineSeparator());
line = br.readLine();
}
String script = sb.toString();
ddmTemplate.setScript(script);
DDMTemplateLocalServiceUtil.addDDMTemplate(ddmTemplate);
}catch(IOException e){
e.printStackTrace();
} catch (SystemException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
log_.info("Inside create template done");
}
The above snippets are executing properly with out any exceptions But unable to see in the content section of Control Panel.Suggest me if anything wrong
There are couple of issues with your code:
You are not setting all the required properties, like groupId, companyId, classNameId, structureKey, dates etc.
There isn't any setName and setDescription method for DDMStructure or DDMTemplate accepting String argument (Liferay 6.2 GA2). Instead, there are only setNameMap and setDescriptionMap methods for both accepting Map<Locale, String>.
Use dynamic ids (structureId and templateId) in place of hard-coded ids, as following:
DDMStructure ddmStructure = DDMStructureUtil.create(CounterLocalServiceUtil.increment());and
DDMTemplate ddmTemplate = DDMTemplateUtil.create(CounterLocalServiceUtil.increment());
For classNameId, you can get it using it's value, like:
ClassName className = ClassNameLocalServiceUtil.getClassName("com.liferay.portlet.journal.model.JournalArticle");
long classNameId = className.getClassNameId();
Also, better to use update over populated object in place of adding:
DDMStructureUtil.update(ddmStructure);
and
DDMTemplateUtil.update(ddmTemplate);
Additionally, if you have access to the ThemeDisplay object, you can get groupId, companyId, userId, userFullName from it. Also, set new Date() for createDate and modifiedDate properties.
I have a panel with a JTabbedpane and in every tab you can set parameters to execute a query. When one query is busy retrieving his data from the database, you can already open a new tab to set the new parameters. To avoid overload on the database only one query may be executed at once. But when you click execute the program must remember which queries to execute in the right order. During the execution a loader icon is shown and the GUI may not be frozen, because there is a stop button you can click to stop the execution.
I used a swingworker to avoid the GUI from blocking while executing the query and that works fine. But now I want to prevent the next query to start before the previous has finished. In a model, common for the whole panel, I initialized a semaphore: private final Semaphore semaphore = new Semaphore(1, true);
This is the code which starts the swingworker (I've added println commands to see which is started, stopped or finished)
private void doStoredQuery() {
try {
semaphore.acquire();
System.out.println(queryName + "started");
worker.execute();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
And this is my swingworker (initializeWorker() is called from the constructor of the main class):
private SwingWorker<StoredQueryDataModel, Integer> initializeWorker() {
worker = new SwingWorker<StoredQueryDataModel, Integer>() {
#Override
protected StoredQueryDataModel doInBackground() throws Exception {
try {
StoredQueryDataModel dataModel = null;
publish(0);
try {
dataModel = new StoredQueryDataModel(queryRunner, ldbName, queryName, params);
} catch (S9SQLException e) {
//
} catch (Throwable e) {
showErrorMessage(e);
}
return dataModel;
}
finally {
semaphore.release();
System.out.println(queryName + "finished");
}
}
#Override
protected void process(List<Integer> chunks) {
//ignore chunks, just reload loader icon
panel.repaint();
}
#Override
protected void done() {
String error;
try {
result = get();
error = null;
} catch (Exception e) {
error = e.getMessage();
}
if(result == null) {
semaphore.release();
System.out.println(queryName + " stopped");
}
if(error == null) {
// process result
}
else {
showErrorMessage(new Throwable(error));
}
}
};
return worker;
}
I've tried putting the acquire and release on other positions in the code, but nothing seems to work. I am bot in Swingworker and sempahores quite new... Can someone help?
I have found the problem: the semaphore had to be a static variable. In my code there were as many semaphores as there are tabs, which caused them to run at the same time instead of sequentially.
I am developping a BlackBerry application which communicates with the server via HTTP requests(javax.microedition.io.HttpConnection). On device, user clicks some UI items, and device sends the requests to server, when the response comes, UI changes. Communication takes place under new thread, while UI thread pushes and pops ProgressDialogScreen.
The problem is sometimes, when response comes and ProgressDialogScreen is popped, UI does not change but after couple seconds UI changes. If you have requested in between when ProgressDialogScreen is popped and when new Screen is pushed, there comes the mess. First oldest new Screen is pushed, and the newest new Screen is pushed. And this situation can be observed like server responsing wrong requests. This problems occur on simulator and device.
The other problem is, sometimes two same response returns for one request. I was able to see these two problems on simulator at the logs, but i have not able to see this issue on device since i can not see the logs.
EDIT:
String utf8Response;
HttpConnection httpConn = null;
try{
httpConn = (HttpConnection) Connector.open(url);
httpConn.setRequestMethod(HttpConnection.GET);
httpConn.setRequestProperty("Content-Type", "text/html; charset=UTF8");
if(sessionIdCookie != null){
//may throw IOException, if the connection is in the connected state.
httpConn.setRequestProperty("Cookie", sessionIdCookie);
}
}catch (Exception e) {
//...
}
try{
httpConn.getResponseCode();
return httpConn;
}catch (IOException e) {
// ...
}
byte[] responseStr = new byte[(int)httpConn.getLength()];
DataInputStream strm = httpConn.openDataInputStream();
strm.readFully(responseStr);
try{
strm.close();
}catch (IOException e) {
// ....
}
utf8Response = new String(responseStr, "UTF-8");
If this code successfully run, this piece of code runs and new screen is pushed:
UiApplication.getUiApplication().invokeLater(new Runnable() {
public void run() {
Vector accounts = Parser.parse(utf8Response,Parser.ACCOUNTS);
if (accounts.size() == 0){
DialogBox.inform(Account.NO_DEPOSIT);
return;
}
currentScreen = new AccountListScreen(accounts);
changeScreen(null,currentScreen);
}
});
public void changeScreen(final AbstractScreen currentScreen,final AbstractScreen nextScreen) {
if (currentScreen != null)
UiApplication.getUiApplication().popScreen(currentScreen);
if (nextScreen != null)
UiApplication.getUiApplication().pushScreen(nextScreen);
}
EDITv2:
private static void progress(final Stoppable runThis, String text,boolean cancelable) {
progress = new ProgressBar(runThis, text,cancelable);
Thread threadToRun = new Thread() {
public void run() {
UiApplication.getUiApplication().invokeLater(new Runnable() {
public void run() {
try{
UiApplication.getUiApplication().pushScreen(progress);
}catch(Exception e){
Logger.log(e);
}
}
});
try {
runThis.run();
} catch (Throwable t) {
t.printStackTrace();
}
UiApplication.getUiApplication().invokeLater(new Runnable() {
public void run() {
try {
UiApplication.getUiApplication().popScreen(progress);
} catch (Exception e) { }
}
});
}
};
threadToRun.start();
}
By the way ProgressBar is extended from net.rim.device.api.ui.container.PopupScreen and Stoppable is extended from Runnable
I preferred to pop progress bar after new Screen is prepared and pushed. This way there will be no new request between request and response.
Why not do:
private static void progress(final Stoppable runThis, String text,boolean cancelable) {
progress = new ProgressBar(runThis, text,cancelable);
UiApplication.getUiApplication().pushScreen(progress);
[...]
Seems like you are parsing on the UI Thread. Please remove Vector accounts = Parser.parse(utf8Response,Parser.ACCOUNTS); from ui thread and do it in a separate thread.