groovy multithreading - multithreading

Question

I'm newbie to groovy/grails.
How to implement thread for this code . Had 2500 urls and this was taking hours of time for checking each url.
so i decided to implement multi-thread for this :
Here is my sample code :
def urls = [
"http://www.wordpress.com",
"http://67.192.103.225/QRA.Public/" ,
"http://www.subaru.com",
"http://baldwinfilter.com/products/start.html"
]
def up = urls.collect { ur ->
try {
def url = new URL(ur)
def connection = url.openConnection()
if (connection.responseCode == 200) {
return true
} else {
return false
}
} catch (Exception e) {
return false
}
}
For this code i need to implement multi-threading .
Could any one please suggest me the code.
thanks in advance,
sri.

Answer 1

I would take a look at the Groovy Parallel Systems library. In particular I think that the Parallel collections section would be useful.
Looking at the docs, I believe that collectParallel is a direct drop-in replacement for collect (bearing in mind the obvious caveats about side-effects). The following works fine for me:
def urls = [
"http://www.wordpress.com",
"http://www.subaru.com",
"http://baldwinfilter.com/products/start.html"
]
Parallelizer.doParallel {
def up = urls.collectParallel { ur ->
try {
def url = new URL(ur)
def connection = url.openConnection()
if (connection.responseCode == 200) {
return true
} else {
return false
}
} catch (Exception e) {
return false
}
}
println up
}

Answer 2

See the Groovy docs for an example how to use an ExecutorService to do what you want.

Answer 3

You can use this to check the URL in a separate thread.
class URLReader implements Runnable
{
def valid
def url
URLReader( url ) {
this.url = url
}
void run() {
try {
def connection = url.toURL().openConnection()
valid = ( connection.responseCode == 200 ) as Boolean
} catch ( Exception e ) {
println e.message
valid = Boolean.FALSE
}
}
}
def reader = new URLReader( "http://www.google.com" )
new Thread( reader ).start()
while ( reader.valid == null )
{
Thread.sleep( 500 )
}
println "valid: ${reader.valid}"
Notes: The valid attribute will be either null, Boolean.TRUE or Boolean.FALSE. You'll need to wait for a while to give all the threads a chance to open the connection. Depending on the number of URLs you're checking you will eventually hit a limit of the number of threads / connections you can realistically handle, so should check URLs in batches of the appropriate size.

Answer 4

I think this way is very simple to achieve.
import java.util.concurrent.*
//Thread number
THREADS = 100
pool = Executors.newFixedThreadPool(THREADS)
defer = { c -> pool.submit(c as Callable) }
def urls = [
"http://www.wordpress.com",
"http://www.subaru.com",
]
def getUrl = { url ->
def connection = url.openConnection()
if (connection.responseCode == 200) {
return true
} else {
return false
}
}
def up = urls.collect { ur ->
try {
def url = new URL(ur)
defer{ getUrl(url) }.get()
} catch (Exception e) {
return false
}
}
println up
pool.shutdown()

Answer 5

This is how I implemented:
class ValidateLinks extends Thread{
def valid
def url
ValidateLinks( url ) {
this.url = url
}
void run() {
try {
def connection = url.toURL().openConnection()
connection.setConnectTimeout(5000)
valid = ( connection.responseCode == 200 ) as Boolean
} catch ( Exception e ) {
println url + "-" + e.message
valid = Boolean.FALSE
}
}
}
def threads = [];
urls.each { ur ->
def reader = new ValidateLinks(ur.site_url)
reader.start()
threads.add(reader);
}
while (threads.size() > 0) {
for(int i =0; i < threads.size();i++) {
def tr = threads.get(i);
if (!tr.isAlive()) {
println "URL : " + tr.url + "Valid " + tr.valid
threads.remove(i);
i--;
}
}
}