I have a celery task like this
#app.task(bind=True,max_retries=3, default_retry_delay=1 * 60)
def doTargetprefilter(self,*args,**kwargs ):
I am calling this as
args = [None,sourcedns, targetdnlist]
kwargs= {'workername':workername,}
result = r.doTargetprefilter.apply_async(*args,**kwargs)
However I am getting a strange error
File "/usr/local/lib/python3.4/dist-packages/celery/app/amqp.py", line 254, in publish_task
raise ValueError('task kwargs must be a dictionary')
ValueError: task kwargs must be a dictionary
A small unit test of the invocation works fine;
def test_doTargetprefilter(self):
from ltecpxx.mrosimpleexecutor import doTargetprefilter
s=[1,2,3]
t=[1,2,2]
workername="ltecpxx.mrosimpleexecutor.SimplePrefilter"
args =[None,s,t]
kwargs={'wokername':workername}
doTargetprefilter(*args,**kwargs)
I have tried all sorts of combinaiton and also seen the apply_async documentation. It works if I make it as a normal method (without *args and **kwargs); What am I doing wrong
The bind annotation supplies the self; So we need to remove that from the args list, and all the arguments must be in a tuple when we call apply_async. Changing these two will give
args = [sourcedns, targetdnlist]
kwargs= {'workername':workername}
result = r.doTargetprefilter.apply_async((args),kwargs)
And function signature
#app.task(bind=True,max_retries=3, default_retry_delay=1 * 60) # retry in 1 minutes.
def doTargetprefilter(self,*args,workername=""):
Related
The following code snippets were all run with python 3.7.11.
I came across some unexpected behavior with the unittest.mock Mock class. I wrote a unit test where a Mock was called with specific arguments to cover the case that the the real object, being mocked for, was expected to be called with during real runtime. The tests passed so I pushed a build to a real running device to discover a bug where not all of the arguments were being passed to the real object method. I quickly found my bug however to me it initially looked like my unit test should have failed when it instead succeeded. Below are some simplified examples of that situation. What I am curious about is if this behavior should be considered a bug or an error in understanding on my part.
from `unittest.mock` import Mock
mock = Mock()
l = [1]
mock(l)
l.append(2)
mock.call_args
# Output: call([1,2])
# rather than call([1])
id(l) == id(mock.call_args[0][0])
# Output: True
# This means the l object and latest call_args reference occupy the same space in memory
This copy by reference behavior is confusing because when a function is called in the same procedure, it is not expected that the function would be called with arguments appended to the object after the call.
def print_var(x):
print(x)
l = [1]
print_var(l)
# Output: 1
l.append(2)
# print_var never be called with [1,2]
Would it make sense for call_args to use deepcopy to emulate the behavior I was expecting?
Found a solution on the unittest.mock examples
from unittest.mock import Mock
from copy import deepcopy
class CopyingMock(MagicMock):
def __call__(self, *args, **kwargs):
args = deepcopy(args)
kwargs = deepcopy(kwargs)
return super().__call__(*args, **kwargs)
c = CopyingMock(return_value=None)
arg = set()
c(arg)
arg.add(1)
c.assert_called_with(set())
c.assert_called_with(arg)
# Traceback (most recent call last):
# ...
# AssertionError: Expected call: mock({1})
# Actual call: mock(set())
I'm writing a code to simplify a graph. In this case I've needed to remove a node of degree 2 and connect its two neighbors each other. Here is the simplified code
class node():
def __init__(self,ind):
#some code
self.neighbors=queue() #queue is another class defined by me
self.distances=queue()
#some code
def addngh(self,nd,distance):
#some code
def remngh(self,nd): #remove node "nd" from neighbors queue
#some code
def d2noderem(self): #removing self node from its left,right neighbors' "neighbors" queue by passing self to left and right's "remngh" function
left,right = self.neighbors[0:2]
#some code
left.remngh(self) #======= Error occurs at here ==========
right.remngh(self)
#some code
when I call that d2noderem function the following error occurs
File "/path/to/file/simplifygraphs.py", line 51, in d2noderem left.remngh(self)
TypeError: remngh() missing 1 required positional argument: 'nd'
Then I tried with
left.remngh(self,self)
and this is the result
File "/path/to/file/simplifygraphs.py", line 51, in d2noderem left.remngh(self,self)
TypeError: remngh() takes 2 positional arguments but 3 were given
I can't understand how did the no of args increased from 0 to 3 by adding 1 more argument.
And I couldn't find a solution for this problem yet.
How to overcome this type of problem?
I appreciate your help very much
The method 'remng' expects an argument as defined by the parameter 'nd' in def remngh(self,nd): Since you're calling it without supplying the expected argument, it's throwing an error.
You should either provide the expected argument or rewrite the function entirely.
I wrote a decorated along these lines:
from functools import wraps
def mark_something(f):
#wraps(f)
def decorated(*args, **kwargs):
# How do I find the pd.DataFrame parameter of the decorated function?
# df = args[<pos_1>] or kwargs['df']
df = df.apply(lambda x: ...)
return f(*args, **kwargs)
return decorated
The wrapped method receives a DatFrame parameter as the 2nd parameter.
The problem is that the caller can do either foo(something, my_df) and then I need to look in args or foo(something, df=my_df) and I need to look in kwargs.
Is there a "nice" way to find a parameter in the wrapped function without having to explicitly check both dictionary and list?
EDIT
I tried kwargs.get('df', args[1]) but that throws an error since args has a single item ... I thought that optional part is evaluated only if get fails ...
Since your decorator is very specific, ie it has a dataframe and a single non-keyword argument, you could use the correct argument signature in your decorator method.
def wrapper(fun):
def working(a, kwd=None):
print("arg: ", a)
print("kwarg: ", kwd)
return fun(a, kwd)
return working
#wrapper
def testme(a, kwd=None):
print("running: ", a, kwd)
testme(1, 2)
testme(1, kwd=3)
The output is then:
arg: 1
kwarg: 2
running: 1 2
arg: 1
kwarg: 3
running: 1 3
Of course this could be improved by add #wraps from functools, but I think the idea is there.
I am trying to add a custom field in my logging using LogRecordFactory. I am repeatedly calling a class and every time I do that, I want to set the custom_attribute in the init module so the remainder of the code within the class will have this attribute. But I cannot get this to work. I found the following which works, but its static.
import logging
old_factory = logging.getLogRecordFactory()
def record_factory(*args, **kwargs):
record = old_factory(*args, **kwargs)
record.custom_attribute = "whatever"
return record
logging.basicConfig(format="%(custom_attribute)s - %(message)s")
logging.setLogRecordFactory(record_factory)
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logging.debug("test")
This will output correctly:
whatever - test
However, my use case is that the custom_attribute will vary. Every time I call a specific function, I want to change this. So it seems like record_factory needs another parameter passed to it so it can then return the correct record with the new parameter. But I cant figure it out. I have tried adding a parameter to the function, but when I make the call I get:
TypeError: __init__() missing 7 required positional arguments: 'name', 'level', 'pathname', 'lineno', 'msg', 'args', and 'exc_info'
I think this has something to do with the *args and **kwargs but I don't really know. Also, why are there no parenthesis after record_factory when its called by logging.setLogRecordFactory? I have never seen a function work like this.
You can try to use closure:
import logging
old_factory = logging.getLogRecordFactory()
def record_factory_factory(context_id):
def record_factory(*args, **kwargs):
record = old_factory(*args, **kwargs)
record.custom_attribute = context_id
return record
return record_factory
logging.basicConfig(format="%(custom_attribute)s - %(message)s")
logging.setLogRecordFactory(record_factory_factory("whatever"))
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logging.debug("test")
logging.setLogRecordFactory(record_factory_factory("whatever2"))
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logging.debug("test")
result:
$ python3 log_test.py
whatever - test
whatever2 - test
I stumbled upon this question while I was trying to do something similar. This is how I solved it, assuming that you want to add something called xyz to every log line (further explanation below):
import logging
import threading
thread_local = threading.local()
def add_xyz_to_logrecords(xyz):
factory = logging.getLogRecordFactory()
if isinstance(factory, XYZLogFactory):
factory.set_xyz(xyz)
else:
logging.setLogRecordFactory(XYZLogFactory(factory, xyz))
class XYZLogFactory():
def __init__(self, original_factory, xyz):
self.original_factory = original_factory
thread_local.xyz = xyz
def __call__(self, *args, **kwargs):
record = self.original_factory(*args, **kwargs)
try:
record.xyz = thread_local.xyz
except AttributeError:
pass
return record
def set_xyz(self, xyz):
thread_local.xyz = xyz
Here I've created a callable class XYZLogFactory, that remembers what the current value of xyz is, and also remembers what the original LogRecordFactory was. When called as a function, it creates a record using the original LogRecordFactory, and adds an xyz attribute with the current value.
The thread_local is to make it thread-safe, but for an easier version, you could just use an attribute on the XYZLogFactory:
class XYZLogFactory():
def __init__(self, original_factory, xyz):
self.original_factory = original_factory
self.xyz = xyz
def __call__(self, *args, **kwargs):
record = self.original_factory(*args, **kwargs)
record.xyz = self.xyz
return record
def set_xyz(self, xyz):
self.xyz = xyz
In my very first attempt (not shown here), I did not store the original factory, but stored it implicitly in the new LogRecordFactury using a closure. However, after a while that led to a RecursionError, because it kept calling the previous factory, which called the previous factory, etc.
Regarding your last question: there are no parentheses because the function is not called here. Instead it's passed to the logging.setLogRecordFactory, which saves it in a variable somewhere, and then calls that someplace else. If you want more information you can google something like 'functions as first class citizens'.
Easy example:
x = str # Assign to x the function that gives string representation of object
x(1) # outputs the string representation of 1, same as if you'd called str(1)
> '1'
for some reason I am having trouble passing *args and **kwargs to both a threading.Timer() and a function...here is the code I am having trouble with;
from threading import Timer
def print_me(text='foo'):
print(text)
def repeat_task(delay, action, *args, **kwargs):
Timer(delay, repeat_task, (delay, action, [*args], {**kwargs}).start()
action(*args, **kwargs)
repeat_task( 5, print_me, text='bar' )
if anyone could point me in the right direction I would be very appreciative. :)
[*args] and {**kwargs} are both invalid syntax. kwargs is already a dict, so if you want it as a dict, just use it directly. args is a tuple, so if you want its value as a tuple, just use it directly, and if you want to turn it into a list for some reason, you can use list(args).
You're passing these arguments to Timer(), whose parameters are defined as:
Timer(interval, function, args=None, kwargs=None)
Notice that there are no *s there: the args and kwargs parameters for this constructor are normal parameters.
So your third argument needs to be all your positional arguments: delay and action, then all the contents of args. One way to do this is to add lists together with +:
[delay, action] + list(args)
And your fourth argument needs to be the keyword arguments, which is just kwargs.
So this should work:
Timer(delay, repeat_task, [delay, action] + list(args), kwargs).start()