I have two filters regexFilter and lastModified.
return IntegrationFlows.from(Sftp.inboundAdapter(inboundSftp)
.localDirectory(this.getlocalDirectory(config.getId()))
.deleteRemoteFiles(true)
.autoCreateLocalDirectory(true)
.regexFilter(config.getRegexFilter())
.filter(new LastModifiedLsEntryFileListFilter())
.remoteDirectory(config.getInboundDirectory())
, e -> e.poller(Pollers.fixedDelay(60_000)
.errorChannel(MessageHeaders.ERROR_CHANNEL).errorHandler((ex) -> {
})))
By googling I understand I have to use CompositeFileListFilter for regex so change my code to
.filter(new CompositeFileListFilter().addFilter(new RegexPatternFileListFilter(config.getRegexFilter())))
Its compiled but on run time throws error and channel stooped and same error goes for
.filter(ftpPersistantFilter(config.getRegexFilter()))
.
.
.
public CompositeFileListFilter ftpPersistantFilter(String regexFilter) {
CompositeFileListFilter filters = new CompositeFileListFilter();
filters.addFilter(new FtpRegexPatternFileListFilter(regexFilter));
return filters;
}
I just want to filter on the basis of file name. There are 2 flows for same remote folder and both are polling with same cron but should pick their relevant file.
EDIT
adding last LastModifiedLsEntryFileListFilter. Its working fine but adding upon request.
public class LastModifiedLsEntryFileListFilter implements FileListFilter<LsEntry> {
private final Logger log = LoggerFactory.getLogger(LastModifiedLsEntryFileListFilter.class);
private static final long DEFAULT_AGE = 60;
private volatile long age = DEFAULT_AGE;
private volatile Map<String, Long> sizeMap = new HashMap<String, Long>();
public long getAge() {
return this.age;
}
public void setAge(long age) {
setAge(age, TimeUnit.SECONDS);
}
public void setAge(long age, TimeUnit unit) {
this.age = unit.toSeconds(age);
}
#Override
public List<LsEntry> filterFiles(LsEntry[] files) {
List<LsEntry> list = new ArrayList<LsEntry>();
long now = System.currentTimeMillis() / 1000;
for (LsEntry file : files) {
if (file.getAttrs()
.isDir()) {
continue;
}
String fileName = file.getFilename();
Long currentSize = file.getAttrs().getSize();
Long oldSize = sizeMap.get(fileName);
if(oldSize == null || currentSize.longValue() != oldSize.longValue() ) {
// putting size in map, will verify in next iteration of scheduler
sizeMap.put(fileName, currentSize);
log.info("[{}] old size [{}] increased to [{}]...", file.getFilename(), oldSize, currentSize);
continue;
}
int lastModifiedTime = file.getAttrs()
.getMTime();
if (lastModifiedTime + this.age <= now ) {
list.add(file);
sizeMap.remove(fileName);
} else {
log.info("File [{}] is still being uploaded...", file.getFilename());
}
}
return list;
}
}
PS : When I am testing filter for regex I have removed LastModifiedLsEntryFileListFilter just for simplicity. So my final Flow is
return IntegrationFlows.from(Sftp.inboundAdapter(inboundSftp)
.localDirectory(this.getlocalDirectory(config.getId()))
.deleteRemoteFiles(true)
.autoCreateLocalDirectory(true)
.filter(new CompositeFileListFilter().addFilter(new RegexPatternFileListFilter(config.getRegexFilter())))
//.filter(new LastModifiedLsEntryFileListFilter())
.remoteDirectory(config.getInboundDirectory()),
e -> e.poller(Pollers.fixedDelay(60_000)
.errorChannel(MessageHeaders.ERROR_CHANNEL).errorHandler((ex) -> {
try {
this.destroy(String.valueOf(config.getId()));
configurationService.removeConfigurationChannelById(config.getId());
// // logging here
} catch (Exception ex1) {
}
}))).publishSubscribeChannel(s -> s
.subscribe(f -> {
f.handle(Sftp.outboundAdapter(outboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getOutboundDirectory()), c -> c.advice(startup.deleteFileAdvice()));
})
.subscribe(f -> {
if (doArchive) {
f.handle(Sftp.outboundAdapter(inboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getInboundArchiveDirectory()));
} else {
f.handle(m -> {
});
}
})
.subscribe(f -> f
.handle(m -> {
// I am handling exception here
})
))
.get();
and here are exceptions
2020-01-27 21:36:55,731 INFO o.s.i.c.PublishSubscribeChannel - Channel
'application.2.subFlow#0.channel#0' has 0 subscriber(s).
2020-01-27 21:36:55,731 INFO o.s.i.e.EventDrivenConsumer - stopped 2.subFlow#2.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2020-01-27 21:36:55,731 INFO o.s.i.c.DirectChannel - Channel 'application.2.subFlow#2.channel#0' has 0 subscriber(s).
2020-01-27 21:36:55,731 INFO o.s.i.e.EventDrivenConsumer - stopped 2.subFlow#2.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
EDIT
After passing regex to LastModifiedLsEntryFileListFilter and handle there works for me. When I use any other RegexFilter inside CompositeFileListFilter it thorws error.
.filter(new CompositeFileListFilter().addFilter(new LastModifiedLsEntryFileListFilter(config.getRegexFilter())))
Show, please, your final flow. I don't see that you use LastModifiedLsEntryFileListFilter in your CompositeFileListFilter... You definitely can't use regexFilter() and filter() together - the last one wins. To avoid confusion we suggest to use a filter() and compose all those with CompositeFileListFilter or ChainFileListFilter.
Also what is an error you are mentioning, please.
Related
Here is, what I'm trying to do:
A Switch is turned on, starting a service in another thread (works fine so far)
When this service is successful, it should then start another function within the main thread
I don't mind whether the function is called directly by the service or the service is returning a "success"-value to the main thread, what then starts the next function from there.
Here is, what the important parts of the code looks like:
Main thread:
class SendNotif : AppCompatActivity() {
val context = this
private lateinit var Switch: Switch
// Start LocationService when the switch is on
Switch.setOnCheckedChangeListener { buttonView, isChecked ->
if (isChecked) {
Toast.makeText(context, "Starting LocationService", Toast.LENGTH_SHORT).show()
Intent(applicationContext, LocationService::class.java).apply {
action = LocationService.ACTION_START
startService(this)
}
} else {
Toast.makeText(context, "Stopping LocationService", Toast.LENGTH_SHORT).show()
Intent(applicationContext, LocationService::class.java).apply {
action = LocationService.ACTION_STOP
startService(this)
}
}
}
}
fun InitiateMessage() {
// This is the function, that is supposed to start after the LocationService
}
}
This is the LocationService. After being successful, the function InitiateMessage() should start.
class LocationService: Service() {
private val serviceScope = CoroutineScope(SupervisorJob() + Dispatchers.IO)
private lateinit var locationClient: LocationClient
var lat = 0.0F
var long = 0.0F
override fun onBind(p0: Intent?): IBinder? {
return null
}
override fun onCreate() {
super.onCreate()
locationClient = DefaultLocationClient(
applicationContext,
LocationServices.getFusedLocationProviderClient(applicationContext)
)
}
// Start or stop the service
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
when(intent?.action) {
ACTION_START -> start()
ACTION_STOP -> stop()
}
return super.onStartCommand(intent, flags, startId)
}
private fun start() {
// Starting notification
val notification = NotificationCompat.Builder(this, "location")
.setContentTitle("Tracking location...")
.setContentText("Location: null")
.setSmallIcon(R.drawable.ic_launcher_background)
// Can't swipe this notification away
.setOngoing(true)
val notificationManager = getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager
// Starting the location updates
locationClient
// Every 10 seconds
.getLocationUpdates(10000L)
.catch { e -> e.printStackTrace() }
.onEach { location ->
lat = location.latitude.toString().toFloat() // .takeLast(3) // taking only the last 3 digits
long = location.longitude.toString().toFloat() // .takeLast(3)
val updatedNotification = notification.setContentText(
"Location: ($lat, $long)"
)
// notificationManager.notify(1, updatedNotification.build())
// Geofence
MyGeofence(lat, long)
}
.launchIn(serviceScope)
// startForeground(1, notification.build())
}
private fun stop() {
// Stopping the notification
stopForeground(true)
// Stopping the location service
stopSelf()
}
override fun onDestroy() {
super.onDestroy()
serviceScope.cancel()
}
companion object {
const val ACTION_START = "ACTION_START"
const val ACTION_STOP = "ACTION_STOP"
}
fun MyGeofence(lat : Float, long : Float){
val context = this
var db = DataBaseHandler(context)
var data = db.readData()
// Setting the accuracy of the geofence
val acc = 2
val safelat : Double = data.get(0).LocLat.toFloat().round(acc)
val safelong = data.get(0).LocLong.toFloat().round(acc) // .take(acc).take(acc)
val h = Handler(context.mainLooper)
if(safelat == lat.toFloat().round(acc) && safelong == long.toFloat().round(acc)){
h.post(Runnable { Toast.makeText(context, "You have reached your safe refuge! " + lat.toFloat().round(acc) + " " + long.toFloat().round(acc), Toast.LENGTH_LONG).show() })
// ToDo: Right hereafter the function InitiateMessage() should start
}
else{
h.post(Runnable { Toast.makeText(context, "You are still in great danger! " + lat.toFloat().round(acc) + " " + long.toFloat().round(acc), Toast.LENGTH_LONG).show() })
}
}
fun Float.round(decimals: Int): Double {
var multiplier = 1.0
repeat(decimals) { multiplier *= 10 }
return round(this * multiplier) / multiplier
}
}
So far, I tried it with a Looper, which did not work.
java.lang.RuntimeException: Can't create handler inside thread Thread[DefaultDispatcher-worker-1,5,main] that has not called Looper.prepare()
But I guess the far easier way would be a returned value by the service. How do I implement this, and how do I start the next function through this returned value?
I solved my problem with an observe-function and a companion object, that is a MutableLiveData.
The companion object is placed inside the main thread:
companion object {
// var iamsafe: Boolean = false
val iamsafe: MutableLiveData<Boolean> by lazy {
MutableLiveData<Boolean>()
}
}
The observe-function is placed within onCreate:
val safeObserver = Observer<Boolean> { newState ->
Toast.makeText(context, "Initiating message to my mate.", Toast.LENGTH_SHORT).show()
InitiateMessage()
}
iamsafe.observe(this, safeObserver)
The companion is changed in the second thread like this:
SendNotif.iamsafe.postValue (true)
In my android project, I use objectbox as database, if I insert with lock and query without lock, is there any side effect ? such as crash and so on.
fun query(uniqueId: String = ""): MutableList<T> {
if (box.store.isClosed) return mutableListOf()
val query = box.query()
withQueryBuilder(query, uniqueId)
//开始
return query.build().find()
}
private fun putInner(entity: T): Long {
synchronized(box.store) {
if (box.store.isClosed) return -1
if (entity.unique.isBlank()) {
entity.unique = entity.providerUnique()
}
entity.timestamp = System.currentTimeMillis()
return try {
box.put(entity).let { id -> entity.id = id }
entity.id
} catch (ex: Exception) {
-1
}
}
}
I have a remote server from where I want to fetch 20 items(Job) per api call and show them in RecyclerView using paging library.
For that, I want to show a loading indicator at the beginning of the first api call when list of items is being fetched from the server. Everything is okay if data is fetched successfully. That means the loading indicator got invisible if data loaded successfully. The code is given bellow.
JobService.KT
#GET(Constants.API_JOB_LIST)
fun getJobPost(
#Query("page") pageNumber: Int
): Observable<Response<JobResponse>>
JobResponse.kt
data class JobResponse(
#SerializedName("status") val status: Int? = null,
#SerializedName("message") val message: Any? = null,
#SerializedName("data") val jobData: JobData? = null
)
JobData.kt
data class JobData(
#SerializedName("jobs") val jobs: List<Job?>? = null,
#SerializedName("total") val totalJob: Int? = null,
#SerializedName("page") val currentPage: Int? = null,
#SerializedName("showing") val currentlyShowing: Int? = null,
#SerializedName("has_more") val hasMore: Boolean? = null
)
NetworkState.kt
sealed class NetworkState {
data class Progress(val isLoading: Boolean) : NetworkState()
data class Failure(val errorMessage: String?) : NetworkState()
companion object {
fun loading(isLoading: Boolean): NetworkState = Progress(isLoading)
fun failure(errorMessage: String?): NetworkState = Failure(errorMessage)
}
}
Event.kt
open class Event<out T>(private val content: T) {
private var hasBeenHandled = false
fun getContentIfNotHandled() = if (hasBeenHandled) {
null
} else {
hasBeenHandled = true
content
}
fun peekContent() = content
}
JobDataSource.kt
class JobDataSource(
private val jobService: JobService,
private val compositeDisposable: CompositeDisposable
) : PageKeyedDataSource<Int, Job>() {
val paginationState: MutableLiveData<Event<NetworkState>> = MutableLiveData()
val initialLoadingState: MutableLiveData<Event<NetworkState>> = MutableLiveData()
val totalJob: MutableLiveData<Event<Int>> = MutableLiveData()
companion object {
private const val FIRST_PAGE = 1
}
override fun loadInitial(params: LoadInitialParams<Int>, callback: LoadInitialCallback<Int, Job>) {
compositeDisposable += jobService.getJobPost(FIRST_PAGE)
.performOnBackgroundOutputOnMain()
.doOnSubscribe { initialLoadingState.postValue(Event(loading(true))) }
.doOnTerminate { initialLoadingState.postValue(Event(loading(false))) }
.subscribe({
if (it.isSuccessful) {
val jobData = it.body()?.jobData
totalJob.postValue(Event(jobData?.totalJob!!))
jobData.jobs?.let { jobs -> callback.onResult(jobs, null, FIRST_PAGE+1) }
} else {
val error = Gson().fromJson(it.errorBody()?.charStream(), ApiError::class.java)
when (it.code()) {
CUSTOM_STATUS_CODE -> initialLoadingState.postValue(Event(failure(error.message!!)))
else -> initialLoadingState.postValue(Event(failure("Something went wrong")))
}
}
}, {
if (it is IOException) {
initialLoadingState.postValue(Event(failure("Check Internet Connectivity")))
} else {
initialLoadingState.postValue(Event(failure("Json Parsing error")))
}
})
}
override fun loadAfter(params: LoadParams<Int>, callback: LoadCallback<Int, Job>) {
compositeDisposable += jobService.getJobPost(params.key)
.performOnBackgroundOutputOnMain()
.doOnSubscribe { if (params.key != 2) paginationState.postValue(Event(loading(true))) }
.doOnTerminate { paginationState.postValue(Event(loading(false))) }
.subscribe({
if (it.isSuccessful) {
val jobData = it.body()?.jobData
totalJob.postValue(Event(jobData?.totalJob!!))
jobData.jobs?.let { jobs -> callback.onResult(jobs, if (jobData.hasMore!!) params.key+1 else null) }
} else {
val error = Gson().fromJson(it.errorBody()?.charStream(), ApiError::class.java)
when (it.code()) {
CUSTOM_STATUS_CODE -> initialLoadingState.postValue(Event(failure(error.message!!)))
else -> initialLoadingState.postValue(Event(failure("Something went wrong")))
}
}
}, {
if (it is IOException) {
paginationState.postValue(Event(failure("Check Internet Connectivity")))
} else {
paginationState.postValue(Event(failure("Json Parsing error")))
}
})
}
override fun loadBefore(params: LoadParams<Int>, callback: LoadCallback<Int, Job>) {}
}
JobDataSourceFactory.kt
class JobDataSourceFactory(
private val jobService: JobService,
private val compositeDisposable: CompositeDisposable
): DataSource.Factory<Int, Job>() {
val jobDataSourceLiveData = MutableLiveData<JobDataSource>()
override fun create(): DataSource<Int, Job> {
val jobDataSource = JobDataSource(jobService, compositeDisposable)
jobDataSourceLiveData.postValue(jobDataSource)
return jobDataSource
}
}
JobBoardViewModel.kt
class JobBoardViewModel(
private val jobService: JobService
) : BaseViewModel() {
companion object {
private const val PAGE_SIZE = 20
private const val PREFETCH_DISTANCE = 20
}
private val jobDataSourceFactory: JobDataSourceFactory = JobDataSourceFactory(jobService, compositeDisposable)
var jobList: LiveData<PagedList<Job>>
init {
val config = PagedList.Config.Builder()
.setPageSize(PAGE_SIZE)
.setInitialLoadSizeHint(PAGE_SIZE)
.setPrefetchDistance(PREFETCH_DISTANCE)
.setEnablePlaceholders(false)
.build()
jobList = LivePagedListBuilder(jobDataSourceFactory, config).build()
}
fun getPaginationState(): LiveData<Event<NetworkState>> = Transformations.switchMap<JobDataSource, Event<NetworkState>>(
jobDataSourceFactory.jobDataSourceLiveData,
JobDataSource::paginationState
)
fun getInitialLoadingState(): LiveData<Event<NetworkState>> = Transformations.switchMap<JobDataSource, Event<NetworkState>>(
jobDataSourceFactory.jobDataSourceLiveData,
JobDataSource::initialLoadingState
)
fun getTotalJob(): LiveData<Event<Int>> = Transformations.switchMap<JobDataSource, Event<Int>>(
jobDataSourceFactory.jobDataSourceLiveData,
JobDataSource::totalJob
)
}
JobBoardFragment.kt
class JobBoardFragment : BaseFragment() {
private val viewModel: JobBoardViewModel by lazy {
getViewModel { JobBoardViewModel(ApiFactory.jobListApi) }
}
private val jobAdapter by lazy {
JobAdapter {
val bundle = Bundle()
bundle.putInt(CLICKED_JOB_ID, it.jobId!!)
navigateTo(R.id.jobBoard_to_jobView, R.id.home_navigation_fragment, bundle)
}
}
override fun getLayoutResId() = R.layout.fragment_job_board
override fun initWidget() {
job_list_recycler_view.adapter = jobAdapter
back_to_main_image_view.setOnClickListener { onBackPressed() }
}
override fun observeLiveData() {
with(viewModel) {
jobList.observe(this#JobBoardFragment, Observer {
jobAdapter.submitList(it)
})
getInitialLoadingState().observe(this#JobBoardFragment, Observer {
it.getContentIfNotHandled()?.let { state ->
when (state) {
is Progress -> {
if (state == loading(true)) {
network_loading_indicator.visible()
} else {
network_loading_indicator.visibilityGone()
}
}
is Failure -> context?.showToast(state.errorMessage.toString())
}
}
})
getPaginationState().observe(this#JobBoardFragment, Observer {
it.getContentIfNotHandled()?.let { state ->
when (state) {
is Progress -> {
if (state == loading(true)) {
pagination_loading_indicator.visible()
} else {
pagination_loading_indicator.visibilityGone()
}
}
is Failure -> context?.showToast(state.errorMessage.toString())
}
}
})
getTotalJob().observe(this#JobBoardFragment, Observer {
it.getContentIfNotHandled()?.let { state ->
job_board_text_view.visible()
with(profile_completed_image_view) {
visible()
text = state.toString()
}
}
})
}
}
}
But the problem is if data fetching failed due to internet connectivity or any other server related problem loading indicator does not invisible that means it still loading though I make the loadingStatus false and error message is shown. it means .doOnTerminate { initialLoadingState.postValue(Event(loading(false))) } is not called if error occured. This is the first problem. Another problem is loadInitial() and loadAfter() is being called simultaneously at the first call. But I just want the loadInitial() method is called at the beginning. after scrolling loadAfter() method will be called.
Try replacing all your LiveData's postValue() methods by setValue() or simply .value =.
The problem is that the postValue() method is for updating the value from a background thread to observers in the main thread. In this case you are always changing the values from the main thread itself, so you should use .value =.
Hope it's not too late.
Need a quick help. I am trying to write a test class and getting below error
"can not resolve the method .thenreturn(org.apache.kafka.clients.producer)
#Test
public void testPublishData_Success() throws java.lang.Exception {
when(GetPropValues.getPropValue(PublisherConstants.ATMID)).thenReturn("ATM");
when(GetPropValues.getPropValue(PublisherConstants.DATA_SOURCE)).thenReturn("PCE");
ReadAndWriteFiles mockFiles = Mockito.mock(ReadAndWriteFiles.class);
PowerMockito.whenNew(ReadAndWriteFiles.class).withNoArguments().thenReturn(mockFiles);
Mockito.when(mockFiles.getAllFiles()).thenReturn("someValue");
KafkaProducer mockProducer = Mockito.mock(KafkaProducer.class);
PowerMockito.whenNew(KafkaProducer.class).withAnyArguments().thenReturn(mockProducer);
producer.publishData(null, "Test", "Data1");
}
Powermockito is fine in returning ReadAndWriteFiles.class object but it is throwing an error for KafkaProducer.class. on line
PowerMockito.whenNew(KafkaProducer.class).withAnyArguments().thenReturn(mockProducer);
Is there any other way to for this work around? Any suggestion will be appreciated.
Note: KafkaProducer.class is in not a custom class but its inside from apache spark kafka libraries
Main code is as per below
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
InputData inputMessage;
try {
inputMessage = populateData(timeStamp, dataCategory, data, atmId, topic);
ReadAndWriteFiles readerWriter = new ReadAndWriteFiles();
File[] directory = readerWriter.getAllFiles();
if (directory != null && directory.length > 0) {
if (connectionSet && !publishingData) {
sendDataFromFiles(producer, directory);
publishingData = false;
}
} else {
producer.send(keyedMsg, new KafkaResponseHandler(inputMessage));
}
} catch (IOException e) {
}
I think the error is
KafkaProducer mockProducer = Mockito.mock(KafkaProducer.class);
PowerMockito.whenNew(ReadAndWriteFiles.class).withAnyArguments().thenReturn(mockProducer)
I think the returned value should be a mock for ReadAndWriteFiles class not a KafkaProducer
ReadAndWriteFiles readMock = Mockito.mock(ReadAndWriteFiles.class)
PowerMockito.whenNew(ReadAndWriteFiles.class).withAnyArguments().thenReturn(readMock)
Mockito.when(readMock.getAllFiles()).thenReturn(anArrayOfFiles);
The signature of the thenReturn method is as follow
OngoingStubbing<T> [More ...] thenReturn(T value);
So you are using to return a ReadAndWriteFiles you shouls return an object of the same class
I'm playing with the newly introduced concurrency features in Java 8, working exercises from the book "Java SE 8 for the Really Impatient" by Cay S. Horstmann. I created the following web crawler using the new CompletedFuture and jsoup. The basic idea is given a URL, it'll find first m URLs on that page and repeat the process n times. m and n are parameters, of course. Problem is the program fetches the URLs for the initial page but doesn't recurse. What am I missing?
static class WebCrawler {
CompletableFuture<Void> crawl(final String startingUrl,
final int depth, final int breadth) {
if (depth <= 0) {
return completedFuture(startingUrl, depth);
}
final CompletableFuture<Void> allDoneFuture = allOf((CompletableFuture[]) of(
startingUrl)
.map(url -> supplyAsync(getContent(url)))
.map(docFuture -> docFuture.thenApply(getURLs(breadth)))
.map(urlsFuture -> urlsFuture.thenApply(doForEach(
depth, breadth)))
.toArray(size -> new CompletableFuture[size]));
allDoneFuture.join();
return allDoneFuture;
}
private CompletableFuture<Void> completedFuture(
final String startingUrl, final int depth) {
LOGGER.info("Link: {}, depth: {}.", startingUrl, depth);
CompletableFuture<Void> future = new CompletableFuture<>();
future.complete(null);
return future;
}
private Supplier<Document> getContent(final String url) {
return () -> {
try {
return connect(url).get();
} catch (IOException e) {
throw new UncheckedIOException(
" Something went wrong trying to fetch the contents of the URL: "
+ url, e);
}
};
}
private Function<Document, Set<String>> getURLs(final int limit) {
return doc -> {
LOGGER.info("Getting URLs for document: {}.", doc.baseUri());
return doc.select("a[href]").stream()
.map(link -> link.attr("abs:href")).limit(limit)
.peek(LOGGER::info).collect(toSet());
};
}
private Function<Set<String>, Stream<CompletableFuture<Void>>> doForEach(
final int depth, final int breadth) {
return urls -> urls.stream().map(
url -> crawl(url, depth - 1, breadth));
}
}
Test case:
#Test
public void testCrawl() {
new WebCrawler().crawl(
"http://en.wikipedia.org/wiki/Java_%28programming_language%29",
2, 10);
}
The problem is in the following code:
final CompletableFuture<Void> allDoneFuture = allOf(
(CompletableFuture[]) of(startingUrl)
.map(url -> supplyAsync(getContent(url)))
.map(docFuture -> docFuture.thenApply(getURLs(breadth)))
.map(urlsFuture -> urlsFuture.thenApply(doForEach(depth, breadth)))
.toArray(size -> new CompletableFuture[size]));
For some reason you are doing all this inside a stream of one element (is that a part of the exercise?). The result is that allDoneFuture is not tracking the completion of the sub-tasks. It's tracking the completion of the Stream<CompletableFuture> that comes from doForEach. But that stream is ready right away and the futures inside of it are never asked to complete.
Fix it by removing the stream that doesn't do anything helpful:
final CompletableFuture<Void> allDoneFuture=supplyAsync(getContent(startingUrl))
.thenApply(getURLs(breadth))
.thenApply(doForEach(depth,breadth))
.thenApply(futures -> futures.toArray(CompletableFuture[]::new))
.thenCompose(CompletableFuture::allOf);