I am trying to use python built in logging with multiprocessing.
Goal -- is to have errors logged to a file called "error.log"
Issue -- The errors are printed in the console instead of the log file. see code below
import concurrent.futures
from itertools import repeat
import logging
def data_logging():
error_logger = logging.getLogger("error.log")
error_logger.setLevel(logging.ERROR)
formatter = logging.Formatter('%(asctime)-12s %(levelname)-8s %(message)s')
file_handler = logging.FileHandler('error.log')
file_handler.setLevel(logging.ERROR)
file_handler.setFormatter(formatter)
error_logger.addHandler(file_handler)
return error_logger
def check_number(error_logger, key):
if key == 1:
print ("yes")
else:
error_logger.error(f"{key} is not = 1")
def main():
key_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 4, 5, 4, 3, 4, 5, 4, 3, 4, 5, 4, 3, 4, 3]
error_logger = data_logging()
with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
executor.map(check_number, repeat(error_logger), key_list)
if __name__ == '__main__':
main()
function check_number checks if numbers in list key_list is 1 or not
if key = 1, prints yes to the console, if not i would like the program to add {key} is not = 1 to the log file.
instead with the code above it prints it to the console. please help if u can. this is a mini example to my program so don't change the logic
Be able to pass a logger instance to the child processes, you must have been using python 3.7+. So here is a little about how things work.
The basic
Only serializable objects can be passed to the child process, or in the other way of speaking, pickleable. This includes all primitive types, such as int, float, str. Why? because python knows how to reconstruct (or unpickle) them back to an object in the child process.
For any other complex class instance, it is unpickable because the lack of information about the class to reconstruct its instance from serialized bytes.
So if we provide the class information, our instance can be unpickable, right?
To a certain degree, yes. By calling a ClassName(*parameters) it certainly can reconstruct the instance from scratch. So, what if you have modified your instance before it gets pickled, like adding some attributes that not in the __init__ method, such as error_logger.addHandler(file_handler) ? The pickle module is not that smart to know every other thing that you added to your instance afterward.
The why
Then, how does python 3.7+ can pickle a Logger instance? It doesn't do anything much. It just saves the logger's name which is a pure str. Next, to unpickle, it just calls getLogger(name) to reconstruct the instance. So now you understand your first complicated problem: the logger that the child process reconstructs, is a default logger without any handler attached to it and a default level of WARNING.
The how
Long story short: use logger-tt. It supports multiprocessing out of the box.
import concurrent.futures
from itertools import repeat
from logger_tt import setup_logging, logger
setup_logging(use_multiprocessing=True)
def check_number(key):
if key == 1:
print ("yes")
else:
logger.error(f"{key} is not = 1")
def main():
key_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 4, 5, 4, 3, 4, 5, 4, 3, 4, 5, 4, 3, 4, 3]
error_logger = data_logging()
with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
executor.map(check_number, key_list)
if __name__ == '__main__':
main()
If you want to start from the beginning, there are some more problems that you need to solve:
interprocess communication: multiprocessing.Queue or socket
logging using QueueHandler and QueueListener
offload the logging to a different thread or child process
The above things are needed to avoid duplicating log entries or missing a part of the log or no log at all.
Related
So I can use an iterable with side_effect in python mock to produce changing values returned by my calls to the mock:
some_mock.side_effect = [1, 2, 3]
return_value provides the same value every time
some_mock.return_value = 8
Is there a way I can use one or both of these methods so that a mock produces some scheduled values to begin and then an infinite response of one particular value when the first set is exhausted? i.e.:
[1, 2, 3, 8, 8, 8, 8, 8, 8, etc. etc etc.]
There is no specific build-in feature that does that, but you can achieve this by adding a side effect that does this.
In the cases I can think of, it would be sufficient to just add some highest needed number of values instead of an infinite number, and use the side_effect version that takes a list:
side_effect = [1, 2, 3] + [8] * 100
my_mock.side_effect = side_effect
If you really need that infinite number of responses, you can use the other version of side_effect instead that uses a function object instead of a list. You need some generator function that creates your infinite list, and iterate over that in your function, remembering the current position. Here is an example implementation for that (using a class to avoid global variables):
from itertools import repeat
class SideEffect:
def __init__(self):
self.it = self.generator() # holds the current iterator
#staticmethod
def generator():
yield from range(1, 4) # yields 1, 2, 3
yield from repeat(8) # yields 8 infinitely
def side_effect(self, *args, **kwargs):
return next(self.it)
...
my_mock.side_effect = SideEffect().side_effect
This should have the wanted effect.
I am having difficulty implementing parallelisation for generating a list of arrays. In this case, each array is generated independently, and then appended to a list. Somehow multiprocessing.apply_asynch() is outputting an empty array when I feed it with complicated arguments.
More specifically, just to give the context, I am attempting implement a machine learning algorithm using parallelisation . The idea is the following: I have an 'system', and an 'agent' which performs actions on the system. To teach the agent (in this case a neural net) how to behave optimally (with respect to a certain reward scheme that I have omitted here), the agent needs to generate trajectories of the system by applying actions on it. From the obtained reward obtained upon performing the actions, the agent then learns what to do and what not to do. Note importantly that the possible actions in the code are referred to as integers with:
possible_actions = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
So here I am attempting to generate many such trajectories using multiprocessing (sorry the code is not runnable here as it requires many other files, but I'm hoping somebody can spot the issue):
from quantum_simulator_EC import system
from reinforce_keras_EC import Agent
import multiprocessing as mp
s = system(1200, N=3)
s.set_initial_state([0,0,1])
agent = Agent(alpha=0.0003, gamma=0.95, n_actions=len( s.actions ))
def get_result(result):
global action_batch
action_batch.append(result)
def generate_trajectory(s, agent):
sequence_of_actions = []
for k in range( 5 ):
net_input = s.generate_net_input_FULL(6)
action = agent.choose_action( net_input )
sequence_of_actions.append(action)
return sequence_of_actions
action_batch = []
pool = mp.Pool(2)
for i in range(0, batch_size):
pool.apply_async(generate_trajectory, args=(s,agent), callback=get_result)
pool.close()
pool.join()
print(action_batch)
The problem is the code returns an empty array []. Can somebody explain to me what the issue is? Are there restrictions on the kind of arguments that I can pass to apply_asynch? In this example I am passing my system 's' and my 'agent', both complicated objects. I am mentioning this because when I test my code with simple arguments like integers or matrices, instead of agent and system, it works fine. If there is no obvious reason why it's not working, if somebody has some tips to debug the code that would also be helpful.
Note that there is no problem if I do not use multiprocessing by replacing the last part by:
action_batch = []
for i in range(0, batch_size):
get_result( generate_sequence(s,agent) )
print(action_batch)
And in this case, the output here is as expected, a list of sequences of 5 actions:
[[4, 2, 1, 1, 7], [8, 2, 2, 12, 1], [8, 1, 9, 11, 9], [7, 10, 6, 1, 0]]
The final results can directly be appended to a list in the main process, no need to create a callback function. Then you can close and join the pool, and finally retrieve all the results using get.
See the following two examples, using apply_async and starmap_async, (see this post for the difference).
Solution apply
import multiprocessing as mp
import time
def func(s, agent):
print(f"Working on task {agent}")
time.sleep(0.1) # some task
return (s, s, s)
if __name__ == '__main__':
agent = "My awesome agent"
with mp.Pool(2) as pool:
results = []
for s in range(5):
results.append(pool.apply_async(func, args=(s, agent)))
pool.close()
pool.join()
print([result.get() for result in results])
Solution starmap
import multiprocessing as mp
import time
def func(s, agent):
print(f"Working on task {agent}")
time.sleep(0.1) # some task
return (s, s, s)
if __name__ == '__main__':
agent = "My awesome agent"
with mp.Pool(2) as pool:
result = pool.starmap_async(func, [(s, agent) for s in range(5)])
pool.close()
pool.join()
print(result.get())
Output
Working on task My awesome agent
Working on task My awesome agent
Working on task My awesome agent
Working on task My awesome agent
Working on task My awesome agent
[(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4)]
I am trying to implement a program in python which reads from 4 different files, changes the data and writes it to another file.
Currently I am attempting to read and change the data with 4 different processes to speed up the runtime.
I have already tried to use manager.list, but this makes the script slower than sequential.
Is it possible to share a List between processes or to make each process return a list and extend a list in the main process with those lists?
Thanks
The code looks like this (currently myLists stays empty, so nothing is written to the output.csv):
from multiprocessing import Process
import queue
import time
myLists=[[],[],[],[]]
myProcesses = []
def readAndList(filename,myList):
with open(filename,"r") as file:
content = file.read().split(":")
file.close()
j=1
filmid=content[0]
while j<len(content):
for entry in content[j].split("\n"):
if len(entry)>10:
print(entry)
myList.append(filmid+","+entry+"\n")
else:
if len(entry)>0:
filmid=entry
j+=1
if __name__ == '__main__':
start=time.time()
endList=[]
i=1
for loopList in myLists:
myProcesses.append(Process(target=readAndList,args=("combined_data_"+str(i)+".txt",loopList)))
i+=1
for process in myProcesses:
process.start()
for process in myProcesses:
process.join()
k=0
while k<4:
endList.extend(myLists[k])
k+=1
with open("output.csv","w") as outputFile:
outputFile.write(''.join(endList))
outputFile.flush()
outputFile.close()
end = time.time()
print(end-start)
Try to use a modern way.
As an example..
Assuming that we have several files that store numbers, like
1
2
3
4
and so on.
This example greps them from the file and combines into the list of lists, so you can merge them as you want. I use python 3.7.2, it should work in 3.7 as well, but I'm not sure.
Code
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from asyncio import ensure_future, gather, run
async def read(file_list):
tasks = list()
result = None
for file in file_list:
task = ensure_future(read_one(file))
tasks.append(task)
result = await gather(*tasks)
return result
async def read_one(file):
result = list()
with open(file, 'r+') as f:
for line in f.readlines():
result.append(int(line[:-1]))
return result
if __name__ == '__main__':
files = ['1', '2', '3', '4']
res = run(read(files))
print(res)
Output
[[1, 2, 3, 4, 5], [6, 7, 8, 9, 10], [11, 12, 13, 14, 15], [16, 17, 18, 19, 20]]
You can find this source code here
I think this approach is much faster and easier to read. This example can be extended to grepping data from WEB, you can read the info about it here
Asyncio and aiohttp are really great tools. I recommend trying them.
I wanted to run my program on all the cores that I have. Here is the code below which I used in my program(which is a part of my full program. somehow, managed to write the working flow).
def ssmake(data):
sslist=[]
for cols in data.columns:
sslist.append(cols)
return sslist
def scorecal(slisted):
subspaceScoresList=[]
if __name__ == '__main__':
pool = mp.Pool(4)
feature,FinalsubSpaceScore = pool.map(performDBScan, ssList)
subspaceScoresList.append([feature, FinalsubSpaceScore])
#for feature in ssList:
#FinalsubSpaceScore = performDBScan(feature)
#subspaceScoresList.append([feature,FinalsubSpaceScore])
return subspaceScoresList
def performDBScan(subspace):
minpoi=2
Epsj=2
final_data = df[subspace]
db = DBSCAN(eps=Epsj, min_samples=minpoi, metric='euclidean').fit(final_data)
labels = db.labels_
FScore = calculateSScore(labels)
return subspace, FScore
def calculateSScore(cluresult):
score = random.randint(1,21)*5
return score
def StartingFunction(prvscore,curscore,fe_select,df):
while prvscore<=curscore:
featurelist=ssmake(df)
scorelist=scorecal(featurelist)
a = {'a' : [1,2,3,1,2,3], 'b' : [5,6,7,4,6,5], 'c' : ['dog', 'cat', 'tree','slow','fast','hurry']}
df2 = pd.DataFrame(a)
previous=0
current=0
dim=[]
StartingFunction(previous,current,dim,df2)
I had a for loop in scorecal(slisted) method which was commented, takes each column to perform DBSCAN and has to calculate the score for that particular column based on the result(but I tried using random score here in example). This looping is making my code to run for a longer time. So I tried to parallelize each column of the DataFrame to perform DBSCAN on the cores that i had on my system and wrote the code in the above fashion which is not giving the result that i need. I was new to this multiprocessing library. I was not sure with the placement of '__main__' in my program. I also would like to know if there is any other way in python to run in a parallel fashion. Any help is appreciated.
Your code has all what is needed to run on multi-core processor using more than one core. But it is a mess. I don't know what problem you trying to solve with the code. Also I cannot run it since I don't know what is DBSCAN. To fix your code you should do several steps.
Function scorecal():
def scorecal(feature_list):
pool = mp.Pool(4)
result = pool.map(performDBScan, feature_list)
return result
result is a list containing all the results returned by performDBSCAN(). You don't have to populate the list manually.
Main body of the program:
# imports
# functions
if __name__ == '__main__':
# your code after functions' definition where you call StartingFunction()
I created very simplified version of your code (pool with 4 processes to handle 8 columns of my data) with dummy for loops (to achieve cpu-bound operation) and tried it. I got 100% cpu load (I have 4-core i5 processor) that naturally resulted in approx x4 faster computation (20 seconds vs 74 seconds) in comparison with single process implementation through for loop.
EDIT.
The complete code I used to try multiprocessing (I use Anaconda (Spyder) / Python 3.6.5 / Win10):
import multiprocessing as mp
import pandas as pd
import time
def ssmake():
pass
def score_cal(data):
if True:
pool = mp.Pool(4)
result = pool.map(
perform_dbscan,
(data.loc[:, col] for col in data.columns))
else:
result = list()
for col in data.columns:
result.append(perform_dbscan(data.loc[:, col]))
return result
def perform_dbscan(data):
assert isinstance(data, pd.Series)
for dummy in range(5 * 10 ** 8):
dummy += 0
return data.name, 101
def calculate_score():
pass
def starting_function(data):
print(score_cal(data))
if __name__ == '__main__':
data = {
'a': [1, 2, 3, 1, 2, 3],
'b': [5, 6, 7, 4, 6, 5],
'c': ['dog', 'cat', 'tree', 'slow', 'fast', 'hurry'],
'd': [1, 1, 1, 1, 1, 1]}
data = pd.DataFrame(data)
start = time.time()
starting_function(data)
print(
'running time = {:.2f} s'
.format(time.time() - start))
To get to know threading concept better I tried to use threads in a simple program. I want to call a function 3 times which does random selection.
def func(arg):
lst = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
num = random.choice(lst)
arg.append(num)
return arg
def search(arg):
a = func(arg)
a = func(a)
threads_list = []
que = queue.Queue()
for i in range(3):
t = threading.Thread(target=lambda q, arg1: q.put(func(arg1)), args=(que, a))
t.start()
threads_list.append(t)
for t in threads_list:
t.join()
while not que.empty():
result = que.get()
print (result)
if __name__ == '__main__':
lst = []
search(lst)
As you can see In the third part, I used threads but I expected to get different lists ( different for the third part).
but all the threads return the same answer.
Can anyone help me to get different lists from different threads?
I think I misunderstood the concept of multiprocessing and multithreading.
Possibly, the pseudo-random number generator which random.choice is using is using three instances - one for each thread - and in the absence of a unique seed will produce the same pseudo-random sequence. Since no seed is provided, it may be using the system time which, depending on the precision, may be the same for all three threads.
You might try seeding the PRNG with something that will differ from thread to thread, inside the thread that invokes the PRNG. This should cause the three threads to use different seeds and give you different pseudo-random sequences.