Twisted Reactor not Restartable - Using asynchronous threading repeatedly - multithreading

Question

I have a list of URLs. I want to get their content asynchronously every 10 seconds.
urls = [
'http://www.python.org',
'http://stackoverflow.com',
'http://www.twistedmatrix.com',
'http://www.google.com',
'http://launchpad.net',
'http://github.com',
'http://bitbucket.org',
]
waiting = [client.getPage(url) for url in urls]
defer.gatherResults(waiting).addCallback(saveResults)
reactor.run()
How do I do this? This code allows me to only get the urls content once. Calling it again throws error.ReactorNotRestartable()
Thanks :)

Answer 1

This is definitely possible with Twisted.
First off, although this is somewhat unrelated to your question, don't use getPage. It's a very limited API, with poor defaults for security on HTTPS. Instead, use Treq.
Now, onto your main question.
The important thing to understand about reactor.run() is that it doesn't mean "run this code here". It means "run the whole program". When reactor.run() exits, it's time for your program to exit.
Lucky for you, Twisted has a nice built-in way to do things on a regular schedule: LoopingCall.
Here's a working example, using treq and LoopingCall:
urls = [
'http://www.python.org',
'http://stackoverflow.com',
'http://www.twistedmatrix.com',
'http://www.google.com',
'http://launchpad.net',
'http://github.com',
'http://bitbucket.org',
]
from twisted.internet.task import LoopingCall
from twisted.internet.defer import gatherResults
from treq import get, content
def fetchWebPages():
return (gatherResults([get(url).addCallback(content) for url in urls])
.addCallback(saveResults))
def saveResults(responses):
print("total: {} bytes"
.format(sum(len(response) for response in responses)))
repeatedly = LoopingCall(fetchWebPages)
repeatedly.start(10.0)
from twisted.internet import reactor
reactor.run()
As a bonus, this handles the case where fetchWebPages takes longer than 10 seconds, and will react intelligently rather than letting too many outstanding requests pile up, or delaying longer and longer as the requests take longer.