ExecutorService for list of object - multithreading

Question

I'm trying add to set some objects that I getting from every link. But objects aren't adding. I receive empty list. Document - jsoup object. What I am doing wrong? Thank you.
public void spinOffThreadsToGetPages() {
Set<WebPage> webPages = new HashSet<>();
ExecutorService executor = Executors.newFixedThreadPool(10);
for (String link : links) {
executor.submit(() -> addWebPagesToSet(webPages, link));
}
executor.shutdown();
}
private void addWebPagesToSet(Set<WebPage> webPages, String link) {
Document document = getDocument(link);
if (document == null) {
return;
}
WebPage webPage = new WebPage(document.title(), link, document.body().text());
webPages.add(webPage);
}

Answer 1

You're not doing anything wrong. The program fires off the threads, then you call ExecutorService::shutdown. The shutdown only occurs once all threads are complete but execution continueS, without blocking the threads execution. If you print out the results as they come back, you'll get output. If you try to print out the results as soon as the threads are submitted, but not yet complete, you'll get nothing back.
Try this:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.io.IOException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class MultiThreadedGet {
public static void main(String[] args) throws InterruptedException {
Set<WebPage> webPages = new HashSet<>();
ExecutorService executor = Executors.newFixedThreadPool(10);
Map<String, Document> results = new HashMap<>();
List<String> links = Arrays.asList("https://jsoup.org/cookbook/extracting-data/working-with-urls",
"https://news.ycombinator.com/",
"https://www.spectacleapp.com/",
"https://gradle.org/releases/");
for (String link : links) {
executor.submit(() -> {
try {
addWebPagesToSet(webPages, link);
} catch (IOException e) {
e.printStackTrace();
}
});
}
// executor.awaitTermination(30, TimeUnit.SECONDS);
executor.shutdown();
// premature call to print incompleted thread results
for (WebPage webPage : webPages) {
// You won't see this as the threads take some time to complete.
System.out.println("Premature call: " + webPage.toString());
}
}
private static void addWebPagesToSet(Set<WebPage> webPages, String link) throws IOException {
Document document = Jsoup.connect(link).get();
if (document == null) {
return;
}
WebPage webPage = new WebPage(document.title(), link, document.body().text());
webPages.add(webPage);
System.out.println("WebPage Result: " + webPage.toString());
}
}
class WebPage {
private String title;
private String link;
private String text;
public WebPage(String title, String link, String text) {
this.title = title;
this.link = link;
this.text = text;
}
public String getTitle() {
return title;
}
public String getLink() {
return link;
}
public String getText() {
return text;
}
#Override
public String toString() {
return "title='" + title + '\'' +
", link='" + link + '\'' +
", text='" + text.substring(0, 25) + '\'';
}
}
Output
WebPage Result: title='Gradle | Releases', link='https://gradle.org/releases/', text='Have you checked out the '
WebPage Result: title='Spectacle', link='https://www.spectacleapp.com/', text='Move and resize windows w'
WebPage Result: title='Hacker News', link='https://news.ycombinator.com/', text='Hacker News new | past | '
WebPage Result: title='Working with URLs: jsoup Java HTML parser', link='https://jsoup.org/cookbook/extracting-data/working-with-urls', text='jsoup News Bugs Discussio'
If you need to wait for threads to complete before continuing you can use:
executor.awaitTermination(30, TimeUnit.SECONDS);
which will block the current thread's execution so you can continue with completed results.