I am making an app which will return one random line from the .txt file. I made a class to implement this behaviour. The idea was to use one method to open file (which will remain open) and the other method which will close it after the app exits. I do not have much experience in working with files hence the following behaviour is strange to me:
In __init__ I called self.open_file() in order to just open it. And it works fine to get self.len. Now I thought that I do not need to call self.open_file() again, but when I call file.get_term()(returns random line) it raises IndexError (like the file is empty), But, if I call file.open_file() method again, everything works as expected.
In addition to this close_file() method raises AttributeError - object has no attribute 'close', so I assumed the file closes automatically somehow, even if I did not use with open.
import random
import os
class Pictionary_file:
def __init__(self, file):
self.file = file
self.open_file()
self.len = self.get_number_of_lines()
def open_file(self):
self.opened = open(self.file, "r", encoding="utf8")
def get_number_of_lines(self):
i = -1
for i, line in enumerate(self.opened):
pass
return i + 1
def get_term_index(self):
term_line = random.randint(0, self.len-1)
return term_line
def get_term(self):
term_line = self.get_term_index()
term = self.opened.read().splitlines()[term_line]
def close_file(self):
self.opened.close()
if __name__ == "__main__":
print(os.getcwd())
file = Pictionary_file("pictionary.txt")
file.open_file() #WITHOUT THIS -> IndexError
file.get_term()
file.close() #AttributeError
Where is my mistake and how can I correct it?
Here in __init__:
self.open_file()
self.len = self.get_number_of_lines()
self.get_number_of_lines() actually consumes the whole file because it iterates over it:
def get_number_of_lines(self):
i = -1
for i, line in enumerate(self.opened):
# real all lines of the file
pass
# at this point, `self.opened` is empty
return i + 1
So when get_term calls self.opened.read(), it gets an empty string, so self.opened.read().splitlines() is an empty list.
file.close() is an AttributeError, because the Pictionary_file class doesn't have the close method. It does have close_file, though.
Related
I am really new to multiprocessing!
What I was trying to do:
Run a particular instance method i.e. ( wait_n_secs() which was slow!) as a separate process so that other processes can run on the side.
Once instance method is done processing we retrieve its output and use it using shared array provided by multiprocessing module.
Here is the code I was trying to run.
import cv2
import time
from multiprocessing import Array
import concurrent.futures
import copyreg as copy_reg
import types
def _pickle_method(m):
if m.im_self is None:
return getattr, (m.im_class, m.im_func.func_name)
else:
return getattr, (m.im_self, m.im_func.func_name)
copy_reg.pickle(types.MethodType, _pickle_method)
class Testing():
def __init__(self):
self.executor = concurrent.futures.ProcessPoolExecutor()
self.futures = None
self.shared_array = Array('i', 4)
def wait_n_secs(self,n):
print(f"I wait for {n} sec")
cv2.waitKey(n*1000)
wait_array = (n,n,n,n)
return wait_array
def function(waittime):
bbox = Testing().wait_n_secs(waittime)
return bbox
if __name__ =="__main__":
testing = Testing()
waittime = 5
# Not working!
testing.futures = testing.executor.submit(testing.wait_n_secs,waittime)
# Working!
#testing.futures = testing.executor.submit(function,waittime)
stime = time.time()
while 1:
if not testing.futures.running():
print("Checking for results")
testing.shared_array = testing.futures.result()
print("Shared_array received = ",testing.shared_array)
break
time_elapsed = time.time()-stime
if (( time_elapsed % 1 ) < 0.001):
print(f"Time elapsed since some time = {time_elapsed:.2f} sec")
Problems I faced:
1) Error on Python 3.6:
Traceback (most recent call last):
File "C:\Users\haide\AppData\Local\Programs\Python\Python36\lib\multiprocessing\queues.py", line 234, in _feed
obj = _ForkingPickler.dumps(obj)
File "C:\Users\haide\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
File "C:\Users\haide\AppData\Local\Programs\Python\Python36\lib\multiprocessing\queues.py", line 58, in __getstate__
context.assert_spawning(self)
File "C:\Users\haide\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 356, in assert_spawning
' through inheritance' % type(obj).__name__
RuntimeError: Queue objects should only be shared between processes through inheritance
2) Error on Python 3.8:
testing.shared_array = testing.futures.result()
File "C:\Users\haide\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\_base.py", line 437, in result
return self.__get_result()
File "C:\Users\haide\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\_base.py", line 389, in __get_result
raise self._exception
File "C:\Users\haide\AppData\Local\Programs\Python\Python38\lib\multiprocessing\queues.py", line 239, in _feed
obj = _ForkingPickler.dumps(obj)
File "C:\Users\haide\AppData\Local\Programs\Python\Python38\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'weakref' object
As others like Amby, falviussn has previously asked.
Problem:
We get a pickling error specifically for instance methods in multiprocessing as they are unpickable.
Solution I tried (Partially):
The solution most mentioned is to use copy_reg to pickle the instance method.
I don't fully understand copy_reg. I have tried adding the lines of code to the top of mp.py provided by Nabeel. But I haven't got it to work.
(Important consideration): I am on Python 3 using copyreg and solutions seem to be using python 2 as they imported copy_reg (Python 2)
(I haven't tried):
Using dill because they were either not the case of multiprocessing or even if they were. They were not using concurrent.futures module.
Workaround:
Passing function that calls the instance method ( instead of the instance method directly ) to submit() method.
testing.futures = testing.executor.submit(function,waittime)
This does work. But does not seem like an elegant solution.
What I want:
Please guide me on how to correctly use copyreg as I clearly don't understand its workings.
Or
If it's a python3 issue, Suggest another solution where i can pass instance methods to conccurent.futurs.ProcessPoolExecutor.submit() for multi-processing. :)
Update #1:
#Aaron Can you share an example code of your solution? "passing a module level function that takes instance as an argument"
or
Correct my mistake here:
This was my attempt. :(
Passing the instance to the module level function along with the arguments
inp_args = [waittime]
testing.futures = testing.executor.submit(wrapper_func,testing,inp_args)
And this was the module wrapper function I created,
def wrapper_func(ins,*args):
ins.wait_n_secs(args)
This got me back to...
TypeError: cannot pickle 'weakref' object
We get a pickling error specifically for instance methods in multiprocessing as they are unpickable.
This is not true, instance methods are very much picklable in python 3 (unless they contain local attributes, like factory functions). You get the error because some other instance attributes (specific to your code) are not picklable.
Please guide me on how to correctly use copyreg as I clearly don't understand its workings.
It's not required here
If it's a python3 issue, Suggest another solution where i can pass instance methods to conccurent.futurs.ProcessPoolExecutor.submit() for multi-processing. :)
It's not really a python issue, it's to do with what data your sending to be pickled. Specifically, all three attributes (after they are populated), self.executor, self.futures and self.shared_array cannot be put on a multiprocessing.Queue (which ProcessPoolExecutor internally uses) and pickled.
So, the problem happens because you are passing an instance method as the target function, which means that all instance attributes are also implicitly pickled and sent to the other process. Since, some of these attributes are not picklable, this error is raised. This is also the reason why your workaround works, since the instance attributes are not pickled there as the target function is not an instance method. There are a couple of things you can do, the best way depends on if there are other attributes that you need to send as well.
Method #1
Judging from the sample code, your wait_n_secs function is not really using any instance attributes. Therefore, you can convert it into a staticmethod and pass that as the target function directly instead:
import time
from multiprocessing import Array
import concurrent.futures
class Testing():
def __init__(self):
self.executor = concurrent.futures.ProcessPoolExecutor()
self.futures = None
self.shared_array = Array('i', 4)
#staticmethod
def wait_n_secs(n):
print(f"I wait for {n} sec")
# Have your own implementation here
time.sleep(n)
wait_array = (n, n, n, n)
return wait_array
if __name__ == "__main__":
testing = Testing()
waittime = 5
testing.futures = testing.executor.submit(type(testing).wait_n_secs, waittime) # Notice the type(testing)
stime = time.time()
while 1:
if not testing.futures.running():
print("Checking for results")
testing.shared_array = testing.futures.result()
print("Shared_array received = ", testing.shared_array)
break
time_elapsed = time.time() - stime
if ((time_elapsed % 1) < 0.001):
print(f"Time elapsed since some time = {time_elapsed:.2f} sec")
Method #2
If your instance contains attributes which would be used by the target functions (so they can't be converted to staticmethods), then you can also explicitly not pass the unpicklable attributes of the instance when pickling using the __getstate__ method. This would mean that the instance recreated inside other processes would not have all these attributes either (since we did not pass them), so do keep that in mind:
import time
from multiprocessing import Array
import concurrent.futures
class Testing():
def __init__(self):
self.executor = concurrent.futures.ProcessPoolExecutor()
self.futures = None
self.shared_array = Array('i', 4)
def wait_n_secs(self, n):
print(f"I wait for {n} sec")
# Have your own implementation here
time.sleep(n)
wait_array = (n, n, n, n)
return wait_array
def __getstate__(self):
d = self.__dict__.copy()
# Delete all unpicklable attributes.
del d['executor']
del d['futures']
del d['shared_array']
return d
if __name__ == "__main__":
testing = Testing()
waittime = 5
testing.futures = testing.executor.submit(testing.wait_n_secs, waittime)
stime = time.time()
while 1:
if not testing.futures.running():
print("Checking for results")
testing.shared_array = testing.futures.result()
print("Shared_array received = ", testing.shared_array)
break
time_elapsed = time.time() - stime
if ((time_elapsed % 1) < 0.001):
print(f"Time elapsed since some time = {time_elapsed:.2f} sec")
I've been fighting with this problem for some time now and I've finally managed to narrow down the issue and create a minimum working example.
The summary of the problem is that I have a class that inherits from a dict to facilitate parsing of misc. input files. I've overridden the the __setitem__ call to support recursive indexing of sections in our input file (e.g. parser['some.section.variable'] is equivalent to parser['some']['section']['variable']). This has been working great for us for over a year now, but we just ran into an issue when passing these Parser classes through a multiprocessing.apply_async call.
Show below is the minimum working example - obviously the __setitem__ call isn't doing anything special, but it's important that it accesses some class attribute like self.section_delimiter - this is where it breaks. It doesn't break in the initial call or in the serial function call. But when you call the some_function (which doesn't do anything either) using apply_async, it crashes.
import multiprocessing as mp
import numpy as np
class Parser(dict):
def __init__(self, file_name : str = None):
print('\t__init__')
super().__init__()
self.section_delimiter = "."
def __setitem__(self, key, value):
print('\t__setitem__')
self.section_delimiter
dict.__setitem__(self, key, value)
def some_function(parser):
pass
if __name__ == "__main__":
print("Initialize creation/setting")
parser = Parser()
parser['x'] = 1
print("Single serial call works fine")
some_function(parser)
print("Parallel async call breaks on line 16?")
pool = mp.Pool(1)
for i in range(1):
pool.apply_async(some_function, (parser,))
pool.close()
pool.join()
If you run the code below, you'll get the following output
Initialize creation/setting
__init__
__setitem__
Single serial call works fine
Parallel async call breaks on line 16?
__setitem__
Process ForkPoolWorker-1:
Traceback (most recent call last):
File "/home/ijw/miniconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/ijw/miniconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/ijw/miniconda3/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/home/ijw/miniconda3/lib/python3.7/multiprocessing/queues.py", line 354, in get
return _ForkingPickler.loads(res)
File "test_apply_async.py", line 13, in __setitem__
self.section_delimiter
AttributeError: 'Parser' object has no attribute 'section_delimiter'
Any help is greatly appreciated. I spent considerable time tracking down this bug and reproducing a minimal example. I would love to not only fix it, but clearly fill some gap in my understanding on how these apply_async and inheritance/overridden methods interact.
Let me know if you need any more information.
Thank you very much!
Isaac
Cause
The cause of the problem is that multiprocessing serializes and deserializes your Parser object to move its data across process boundaries. This is done using pickle. By default pickle does not call __init__() when deserializing classes. Because of this self.section_delimiter is not set when the deserializer calls __setitem__() to restore the items in your dictionary and you get the error:
AttributeError: 'Parser' object has no attribute 'section_delimiter'
Using just pickle and no multiprocessing gives the same error:
import pickle
parser = Parser()
parser['x'] = 1
data = pickle.dumps(parser)
copy = pickle.loads(data) # Same AttributeError here
Deserialization will work for an object with no items and the value of section_delimiter will be restored:
import pickle
parser = Parser()
parser.section_delimiter = "|"
data = pickle.dumps(parser)
copy = pickle.loads(data)
print(copy.section_delimiter) # Prints "|"
So in a sense you are just unlucky that pickle calls __setitem__() before it restores the rest of the state of your Parser.
Workaround
You can work around this by setting section_delimiter in __new__() and telling pickle what arguments to pass to __new__() by implementing __getnewargs__():
def __new__(cls, *args):
self = super(Parser, cls).__new__(cls)
self.section_delimiter = args[0] if args else "."
return self
def __getnewargs__(self):
return (self.section_delimiter,)
__getnewargs__() returns a tuple of arguments. Because section_delimiter is set in __new__(), it is no longer necessary to set it in __init__().
This is the code of your Parser class after the change:
class Parser(dict):
def __init__(self, file_name : str = None):
print('\t__init__')
super().__init__()
def __new__(cls, *args):
self = super(Parser, cls).__new__(cls)
self.section_delimiter = args[0] if args else "."
return self
def __getnewargs__(self):
return (self.section_delimiter,)
def __setitem__(self, key, value):
print('\t__setitem__')
self.section_delimiter
dict.__setitem__(self, key, value)
Simpler solution
The reason pickle calls __setitem__() on your Parser object is because it is a dictionary. If your Parser is just a class that happens to implement __setitem__() and __getitem__() and has a dictionary to implement those calls then pickle will not call __setitem__() and serialization will work with no extra code:
class Parser:
def __init__(self, file_name : str = None):
print('\t__init__')
self.dict = { }
self.section_delimiter = "."
def __setitem__(self, key, value):
print('\t__setitem__')
self.section_delimiter
self.dict[key] = value
def __getitem__(self, key):
return self.dict[key]
So if there is no other reason for your Parser to be a dictionary, I would just not use inheritance here.
I am new to object-oriented programming and I have some trouble with declaring global variables that are set to a method inside a class. I declared them in the __init__ method (I want them to be inside the class, not outside) with self.censored_transcript = self.apply_filter(), and within this file, everything works fine. But when I import the class in a different .py file and instantiate the Profanity class, the following error occurs: UnboundLocalError: local variable 'censored_transcript' referenced before assignment. I have done quite some research on this topic but can't find anything that fits my case. The global statement does not work. Can anyone please let me know how I can fix this? Thank you very much in advance!
File 1 (file to be imported), called profanity_filter.py:
from better_profanity import profanity
class Profanity(object):
def __init__(self, transcript_filename, wordlist_filename):
self.__transcript_filename = transcript_filename
self.__wordlist_filename = wordlist_filename
self.__transcript = self.__load_transcript()
self.censored_transcript = self.apply_filter()
def __load_transcript(self):
f = open(self.__transcript_filename, "r")
file = f.read()
f.close()
return file
def apply_filter(self):
if __name__ == "__main__":
profanity.load_censor_words_from_file(self.__wordlist_filename)
censored_transcript = profanity.censor(self.__transcript)
return censored_transcript
File 2 (test file):
from profanity_filter import Profanity
# Instantiate the Profanity class:
p = Profanity("sample_transcript.txt", "profanity_wordlist.txt")
# Apply the profanity filter.
censored_transcript = p.apply_filter()
print(censored_transcript)
You can use self.censored_transcript in all places inside your class - self refers to current object of class. Each created object of your class will store self self set of variables available.
censored_transcript without self treated as regular local variable which will be newly created for each call of method.
Error cause:
When method def apply_filter(self) gets called from outer .py file, if __name__ == "__main__" check will be false and local variable censored_transcript will not be created inside this method. So, return censored_transcript tries to return local variable with unassigned value and fails.
Solution 1: in apply_filter(self) you can use object's censored_transcript, not local, to save result. Just append self. before it. And you do not need to return something from this method:
from better_profanity import profanity
class Profanity(object):
def __init__(self, transcript_filename, wordlist_filename):
self.__transcript_filename = transcript_filename
self.__wordlist_filename = wordlist_filename
self.__transcript = self.__load_transcript()
self.apply_filter() # changed
def __load_transcript(self):
f = open(self.__transcript_filename, "r")
file = f.read()
f.close()
return file
# changed
def apply_filter(self):
profanity.load_censor_words_from_file(self.__wordlist_filename)
self.censored_transcript = profanity.censor(self.__transcript)
Solution 2: use a local variable, but assign value to it without if. Or even omit this local variable:
from better_profanity import profanity
class Profanity(object):
def __init__(self, transcript_filename, wordlist_filename):
self.__transcript_filename = transcript_filename
self.__wordlist_filename = wordlist_filename
self.__transcript = self.__load_transcript()
self.censored_transcript = self.apply_filter()
def __load_transcript(self):
f = open(self.__transcript_filename, "r")
file = f.read()
f.close()
return file
# changed
def apply_filter(self):
profanity.load_censor_words_from_file(self.__wordlist_filename)
return profanity.censor(self.__transcript)
I'm having some trouble with ProcessPoolExecutor.
The following code is trying to find the shortest path in a WikiRace game, it gets 2 titles and navigates between one to another.
Here is my code:
class AsyncSearch:
def __init__(self, start, end):
self.start = start
self.end = end
# self.manager = multiprocessing.Manager()
self.q = multiprocessing.Queue()
# self.q = self.manager.Queue()
def _add_starting_node_page_to_queue(self):
start_page = WikiGateway().page(self.start)
return self._check_page(start_page)
def _is_direct_path_to_end(self, page):
return (page.title == self.end) or (page.links.get(self.end) is not None)
def _add_tasks_to_queue(self, pages):
for page in pages:
self.q.put(page)
def _check_page(self, page):
global PATH_WAS_FOUND_FLAG
logger.info('Checking page "{}"'.format(page.title))
if self._is_direct_path_to_end(page):
logger.info('##########\n\tFound a path!!!\n##########')
PATH_WAS_FOUND_FLAG = True
return True
else:
links = page.links
logger.info("Couldn't find a direct path form \"{}\", "
"adding {} pages to the queue.".format(page.title, len(links)))
self._add_tasks_to_queue(links.values())
return "Couldn't find a direct path form " + page.title
def start_search(self):
global PATH_WAS_FOUND_FLAG
threads = []
logger.debug(f'Running with concurrent processes!')
if self._add_starting_node_page_to_queue() is True:
return True
with concurrent.futures.ProcessPoolExecutor(max_workers=AsyncConsts.PROCESSES) as executor:
threads.append(executor.submit(self._check_page, self.q.get()))
I'm getting the following exception:
Traceback (most recent call last):
File "c:\users\tomer smadja\appdata\local\programs\python\python36-32\lib\multiprocessing\queues.py", line 241, in _feed
obj = _ForkingPickler.dumps(obj)
File "c:\users\tomer smadja\appdata\local\programs\python\python36-32\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
File "c:\users\tomer smadja\appdata\local\programs\python\python36-32\lib\multiprocessing\queues.py", line 58, in __getstate__
context.assert_spawning(self)
File "c:\users\tomer smadja\appdata\local\programs\python\python36-32\lib\multiprocessing\context.py", line 356, in assert_spawning
' through inheritance' % type(obj).__name__
RuntimeError: Queue objects should only be shared between processes through inheritance
It's weird since I'm using multiprocessing.Queue() that should be shared between the processes as mentioned by the exception.
I found this similar question but couldn't found the answer there.
I tried to use self.q = multiprocessing.Manager().Queue() instead of self.q = multiprocessing.Queue(), I'm not sure if this takes me anywhere but the exception I'm getting is different:
Traceback (most recent call last):
File "c:\users\tomer smadja\appdata\local\programs\python\python36-32\lib\multiprocessing\queues.py", line 241, in _feed
obj = _ForkingPickler.dumps(obj)
File "c:\users\tomer smadja\appdata\local\programs\python\python36-32\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
File "c:\users\tomer smadja\appdata\local\programs\python\python36-32\lib\multiprocessing\process.py", line 282, in __reduce__
'Pickling an AuthenticationString object is '
TypeError: Pickling an AuthenticationString object is disallowed for security reasons
Also, when I'm trying to use multiprocessing.Process() instead of ProcessPoolExecutor, I'm unable to finish the process once I do find a path. I set up a global variable to stop PATH_WAS_FOUND_FLAG to stop the process initiation but still with no success. What I'm missing here?
ProcessPoolExecutor.submit(...) will not pickle multiprocessing.Queue instances as well other shared multiprocessing.* class instances. You can do two things: One is to use SyncManager, or you can initialize the worker with the multiprocessing.Queue instance at ProcessPoolExecutor construction time. Both are shown below.
Following is your original variation with a couple of fixes applied (see note at end)... with this variation, multiprocessing.Queue operations are slightly faster than below SyncManager variation...
global_page_queue = multiprocessing.Queue()
def set_global_queue(q):
global global_page_queue
global_page_queue = q
class AsyncSearch:
def __init__(self, start, end):
self.start = start
self.end = end
#self.q = multiprocessing.Queue()
...
def _add_tasks_to_queue(self, pages):
for page in pages:
#self.q.put(page)
global_page_queue.put(page)
#staticmethod
def _check_page(self, page):
...
def start_search(self):
...
print(f'Running with concurrent processes!')
with concurrent.futures.ProcessPoolExecutor(
max_workers=5,
initializer=set_global_queue,
initargs=(global_page_queue,)) as executor:
f = executor.submit(AsyncSearch._check_page, self, global_page_queue.get())
r = f.result()
print(f"result={r}")
Following is SyncManager variation where queue operations are slightly slower than above multiprocessing.Queue variation...
import multiprocessing
import concurrent.futures
class AsyncSearch:
def __init__(self, start, end):
self.start = start
self.end = end
self.q = multiprocessing.Manager().Queue()
...
#staticmethod
def _check_page(self, page):
...
def start_search(self):
global PATH_WAS_FOUND_FLAG
worker_process_futures = []
print(f'Running with concurrent processes!')
with concurrent.futures.ProcessPoolExecutor(max_workers=5) as executor:
worker_process_futures.append(executor.submit(AsyncSearch._check_page, self, self.q.get()))
r = worker_process_futures[0].result()
print(f"result={r}")
Note, for some shared objects, SyncManager can be anywhere from slightly to noticeably slower compared to multiprocessing.* variations. For example, a multiprocessing.Value is in shared memory whereas a SyncManager.Value is in the sync manager processes, requiring overhead to interact with it.
An aside, unrelated to your ask, your original code was passing _check_page with incorrect parameters, where you were passing dequeued item to self, leaving the page parameter None. I resolved this by changing _check_page to a static method and passing self.
I have a script that downloads images from urls, but I would like to parallelise it otherwise it will take hours. With this code:
import requests
from math import floor, log10
import urllib
import time
import multiprocessing
with open('images.csv', 'r') as f:
images = f.readlines()
num_position = floor(log10(len(images)) + 1)
a = time.time()
for i, image in enumerate(images[1:10]):
if (i+1) % 1000 == 0:
print('Downloading {} image'.format(i+1) )
# a = time.time()
with open(str(i).zfill(num_position)+'a.jpg', 'wb') as file:
try:
writing = file.write(requests.get(image.split(',')[2]).content)
p = multiprocessing.Process(target=writing, args=(image,))
p.start()
p.join()
except:
print('Skipping an image!')
pass
b = time.time()
print('multiple process -- {}'.format(b-a))
I get an error :
Process Process-9:
Traceback (most recent call last):
File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
self.run()
File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
TypeError: 'int' object is not callable
Why am I getting an error but the task is still completed and the code doesn't break? (and by that I mean the piece in try: )
What would be the easiest way to include some kind of paralleling here?
You get the error because AFAIK this line
writing = file.write(requests.get(image.split(',')[2]).content)
has the output of integer type. write returns the number of written characters which is equal to the length of the string-representation of your image. Now you assign that to the variable writing -> writing becomes a number.
p = multiprocessing.Process(target=writing, args=(image,))
calls writing as target function, which raises the error since your are not calling a function but integer-type writing (not callable). The code works since your workers do not have anything to do and close immediatly and the file is already written.
To get things working, your would have to define a function that takes your image as argument and maybe the file name. This function you later call in the setup of your workers. Something like that:
def write_file(image, filename):
filestream = open(filename, mode="w")
filestream.write(requests.get(image.split(',')[2]).content)
filestream.close()
And in your application
p = multiprocessing.Process(target=write_file, args=(image, filename,))
However, that is just the writing part. If you want to do the downloads in separate task too then you have to put the code for that into your separate function.
def download_write(urls):
for url in iter(urls.get, 'STOP'):
#download code here#
filestream = open(filename, mode="w")
filestream.write(requests.get(image.split(',')[2]).content)
filestream.close()
And your main application:
list_urls = [] # your list of urls to download
urls = Queue()
for element in list_urls:
urls.put(element)
p = multiprocessing.Process(target=download_write, args=(urls,))
urls.put("STOP") #signals end of tasks for your workers
p.start() #start worker
p.join() #wait for worker to finish