Java 8 CompletedFuture web crawler doesn't crawl past one URL - multithreading

Question

I'm playing with the newly introduced concurrency features in Java 8, working exercises from the book "Java SE 8 for the Really Impatient" by Cay S. Horstmann. I created the following web crawler using the new CompletedFuture and jsoup. The basic idea is given a URL, it'll find first m URLs on that page and repeat the process n times. m and n are parameters, of course. Problem is the program fetches the URLs for the initial page but doesn't recurse. What am I missing?
static class WebCrawler {
CompletableFuture<Void> crawl(final String startingUrl,
final int depth, final int breadth) {
if (depth <= 0) {
return completedFuture(startingUrl, depth);
}
final CompletableFuture<Void> allDoneFuture = allOf((CompletableFuture[]) of(
startingUrl)
.map(url -> supplyAsync(getContent(url)))
.map(docFuture -> docFuture.thenApply(getURLs(breadth)))
.map(urlsFuture -> urlsFuture.thenApply(doForEach(
depth, breadth)))
.toArray(size -> new CompletableFuture[size]));
allDoneFuture.join();
return allDoneFuture;
}
private CompletableFuture<Void> completedFuture(
final String startingUrl, final int depth) {
LOGGER.info("Link: {}, depth: {}.", startingUrl, depth);
CompletableFuture<Void> future = new CompletableFuture<>();
future.complete(null);
return future;
}
private Supplier<Document> getContent(final String url) {
return () -> {
try {
return connect(url).get();
} catch (IOException e) {
throw new UncheckedIOException(
" Something went wrong trying to fetch the contents of the URL: "
+ url, e);
}
};
}
private Function<Document, Set<String>> getURLs(final int limit) {
return doc -> {
LOGGER.info("Getting URLs for document: {}.", doc.baseUri());
return doc.select("a[href]").stream()
.map(link -> link.attr("abs:href")).limit(limit)
.peek(LOGGER::info).collect(toSet());
};
}
private Function<Set<String>, Stream<CompletableFuture<Void>>> doForEach(
final int depth, final int breadth) {
return urls -> urls.stream().map(
url -> crawl(url, depth - 1, breadth));
}
}
Test case:
#Test
public void testCrawl() {
new WebCrawler().crawl(
"http://en.wikipedia.org/wiki/Java_%28programming_language%29",
2, 10);
}

Answer 1

The problem is in the following code:
final CompletableFuture<Void> allDoneFuture = allOf(
(CompletableFuture[]) of(startingUrl)
.map(url -> supplyAsync(getContent(url)))
.map(docFuture -> docFuture.thenApply(getURLs(breadth)))
.map(urlsFuture -> urlsFuture.thenApply(doForEach(depth, breadth)))
.toArray(size -> new CompletableFuture[size]));
For some reason you are doing all this inside a stream of one element (is that a part of the exercise?). The result is that allDoneFuture is not tracking the completion of the sub-tasks. It's tracking the completion of the Stream<CompletableFuture> that comes from doForEach. But that stream is ready right away and the futures inside of it are never asked to complete.
Fix it by removing the stream that doesn't do anything helpful:
final CompletableFuture<Void> allDoneFuture=supplyAsync(getContent(startingUrl))
.thenApply(getURLs(breadth))
.thenApply(doForEach(depth,breadth))
.thenApply(futures -> futures.toArray(CompletableFuture[]::new))
.thenCompose(CompletableFuture::allOf);