Python Passing Function Handles To Threads [duplicate] - multithreading

This question already has answers here:
Local variables in nested functions
(4 answers)
Closed 4 years ago.
I'm trying to call 2 functions in parallel using the python 2.7 threading module, and, to make things easier for myself, I've written my own AsycTask class.
class AsyncTask:
def __init__(self, task, name="async-task", callback=None):
self.task = task
self.t = None
if self.task is not None:
if callback is not None:
self.t = threading.Thread(target=lambda: [task(), callback()], name=name)
else:
self.t = threading.Thread(target=self.task, name=name)
def start(self):
if self.t is not None:
self.t.start()
else:
Log.warn("Can't start async task: thread is None")
def join(self):
if self.t is not None:
self.t.join()
else:
Log.warn("Can't join async task: thread is None")
But I get some strange results when I pass it function handles.
Elsewhere I have this class:
class Foo:
def __init__(self, id):
self.id = id
def bar(self, args):
result = None
# do some stuff that takes a while
time.sleep(10)
Log.debug("bar() called in object %s" % self.id)
return result
Then I create a list of foo's
foos = []
foos.append(Foo("1"))
foos.append(Foo("2"))
And call bar asynchronously
results = []
tasks = []
for foo in foos:
args = "some stuff"
fn = foo.bar
Log.debug("before async: " + str(foo))
task = AsyncTask(lambda: [Log.debug("in async: " + str(fn)), results.append(fn(args))])
tasks.append(task)
task.start()
for task in tasks:
task.join()
# process results
When I run this I get:
before async: <__main__.Foo instance at 0x7f9caef7e200>
before async: <__main__.Foo instance at 0x7f9caef7e248>
in async: <bound method Foo.bar of <__main__.Foo instance at 0x7f9caef7e248>>
in async: <bound method Foo.bar of <__main__.Foo instance at 0x7f9caef7e248>>
bar() called in object 2
bar() called in object 2
Note that bar() on the first Foo instance is never called, while it is called twice on the second instance.
I don't have a ton of python experience, so clearly, I am doing something wrong and must not properly understand how threading and function handles work in python.
What would a more pythonic way be to accomplish this?

Your setup classes are fine, the reason you are getting the odd behavior which we can't see is most likely how you set up your logger. I got the same behavior you did until I setup my logger as follows:
import logging
logging.basicConfig()
log = logging.getLogger('log')
Afterwards I get the following:
before async: <__main__.Foo instance at 0x00000000028C9FC8>
before async: <__main__.Foo instance at 0x00000000028D5048>
<__main__.Foo instance at 0x00000000028C9FC8>
<__main__.Foo instance at 0x00000000028D5048>

Related

two issues on the lazy instantiation in the Singleton pattern

My python version:
python3 --version
Python 3.9.2
Issue 1:
What does isinstance function mean?
class Singleton1(object):
__instance = None
def __init__(self):
if not hasattr(Singleton1, '__instance'):
print("__init__ method called, but no instance created")
else:
print("instance already created:", self.__instance)
#classmethod
def get_instance(cls):
if not cls.__instance:
cls.__instance = Singleton1()
return cls.__instance
Initialize it :
x = Singleton1()
__init__ method called, but no instance created
Have a check with isinstance function:
isinstance(x,Singleton1)
True
If x is not an instance,why does isinstance(x,Singleton1) say it is an instance of Singleton1?
Issue2:
Why __init__ method can't be called anyway?
Now repalce all __instance (double underscores) with _instance(single underscore) in the class Singleton1 and replace all Singleton1 with Singleton2:
class Singleton2(object):
_instance = None
def __init__(self):
if not hasattr(Singleton2, '_instance'):
print("__init__ method called, but no instance created")
else:
print("instance already created:", self._instance)
#classmethod
def get_instance(cls):
if not cls._instance:
cls._instance = Singleton2()
return cls._instance
Initialize it:
y = Singleton2()
instance already created: None
Why __init__ method can't be called anyway in this status?
#snakecharmerb on issue1,Why someone say it is lazy instantiation ,if isinstance(x,Singleton1) is true,it is no need to call with Singleton1.get_instance() ,because the instance is already created during instantiation.
The hasattr check does not do what you think it does. Using Singleton2*, hasattr(Singleton2, '_instance') is always True, because the class has an attribute named _instance. You want to check the value of the instance, so use getattr instead; then the expected output will be printed.
The isinstance checks succeed because Singleton2() will return a new instance each time - there is nothing to prevent this. You can add a __new__ method to create _instance and return it every time Singleton2() is called. Note that this will mean that _instance will always exist by the time __init__ is called.
class Singleton2:
_instance = None
def __new__(cls):
if cls._instance is not None:
return cls._instance
instance = super().__new__(cls)
cls._instance = instance
return instance
* The hasattr check in Singleton1 is complicated by the name-mangling performed on __instance. In general, avoid using double-underscored variable names, except for avoiding name clashes in class hierarchies.

How to pass self to function instance when it gets assigned in a decorator?

I am trying to assign dictionary keys to object functions but for some reason it won't work inside of decorators. When I try to call a.run(), self doesn't seem to be passed into the dictionary func. I also don't have access to f.self in decorator so I know it has to be something wrong in there. I have written a simple example of my code. I want it to be something similar to app.route in flask being that it init the mapping between endpoints and functions.
ERROR:
Traceback (most recent call last):
File "main.py", line 27, in <module>
a.run()
File "main.py", line 14, in run
self.rmap[k](data)
TypeError: one_way() missing 1 required positional argument: 'data'
CODE:
class A (object):
def __init__(self):
self.rmap = {}
def route(self, r):
def decorator(f):
self.rmap[r] = f
return f
return decorator
def run(self):
data = [1,2,3]
for k in self.rmap.keys():
self.rmap[k](data)
a = A()
class B (object):
def __init__(self):
pass
#a.route('/one/way')
def one_way (self, data):
print('A WAY:{}'.format(self))
b = B()
a.run()
At the time it's being decorated, one_way() is a plain function, not a method - it only becomes a method when looked up on a B instance. IOW, you need to explicitely provide a B instance when calling it from A().run() (the fact you have a global b instance in your code is irrelevant - the function object stored in a.rmap knows absolutely nothing about it, nor even about the B class FWIW.
To make a long story short, your current design cannot work as is. If you only ever intend to decorate methods (well, functions) from one single class and call them on one single instance of this class, you could pass an instance of this class to a.run() ie:
class A():
# ...
def run(self, obj):
data = [1,2,3]
for k in self.rmap.keys():
self.rmap[k](obj, data)
b = B()
a.run(b)
but this would be of very limited use.
Or you could just use the decorator to "mark" functions to be used for routing (together with the effective route), add some register() methdo to A and explicitely pass B or whatever else instance to this method ie
def route(r):
def decorator(f):
f._A_route = r
return f
return decorator
class A (object):
def __init__(self):
self.rmap = {}
def register(self, *objects):
for obj in objects:
self._register(obj)
def _register(self, obj):
for name in dir(obj):
if name.startswith("_"):
continue
attr = getattr(obj, name)
if callable(attr) and hasattr(attr, "_A_route"):
self.rmap[attr._A_route] = attr
def run(self):
data = [1,2,3]
for k in self.rmap.keys():
self.rmap[k](data)
class B (object):
def __init__(self):
pass
#route('/one/way')
def one_way (self, data):
print('A WAY:{}'.format(self))
if __name__ == "__main__":
a = A()
b = B()
a.register(b)
a.run()
Now there might be better solutions for your concrete use case, but it's impossible to tell without knowing about the whole context etc.
When calling self.rmap[k](data) you are not passing in the self parameter. This has to be an instance of class B in order to work.
Normally you'd just pass on the parameters with which the decorated function was called, but you seem to want to use your decorated function differently. In your case what would work is:
def run(self):
data = [1,2,3]
b = B()
for k in self.rmap.keys():
self.rmap[k](b, data)
You could of course also instantiate the B instance somewhere else if you want to reuse it between calls.

How to create a class share methods in memory and not replicate for each object in python3

is more than anything a doubt but, I want to create a class that put in memory methods a single time and then instances that believes it can share these methods without these have to be put in memory for each one of them.
Methods are "put in memory" only once. Methods are sequences of bytecode instructions and will be stored only once. However, properties (methods operate on) are not.
I think, you are confusing the difference of class properties and object properties. Use them as shown in the following examples:
class Foo:
def __init__(self):
self.object_prop = None
def set(self, val):
self.object_prop = val
def get(self):
return self.object_prop
Then you can use it like this:
a = Foo()
a.set(3)
print(a.get()) # prints 3
b = Foo()
b.set(4)
print(a.get()) # prints 3
And the class properties example looks like this:
class Foo:
class_prop = None
#classmethod
def set(cls, val):
cls.class_prop = val
#classmethod
def get(cls):
return cls.class_prop
a = Foo()
a.set(3)
print(a.get()) # prints 3
b = Foo()
b.set(4)
print(a.get()) # prints 4
Maybe this is, what you are looking for.
In the comments, you pointed out that you are looking for an implementation of singletons in python. In python, singletons are an anti-pattern and need to be avoided at all costs. However, a simple search on StackOverflow gives the following result:
class Singleton(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
return cls._instances[cls]
class Logger(metaclass=Singleton):
pass
With this implementation, all objects will have the same memory address:
>>> id(Logger())
140202032740616
>>> id(Logger())
140202032740616

Pytests with context manager

I am trying to understand how to test Context managert with pytests.
I created some Class and need to count how much times was called static method do_dome_stuff
class Iterator():
def __init__(self):
pass
#staticmethod
def do_some_stuff():
pass
def __enter__(self):
return [i for i in range(10)]
def __exit__(self, *args):
return True
iterator = Iterator()
def f(iterator):
with iterator as i:
for _ in i:
iterator.do_some_stuff()
I have created py.test file and need to check if function was called 10 times. But my solution isn't working:
#pytest.fixture
def iterator():
return MagicMock(spec=Iterator)
def test_f(iterator):
f(iterator)
assert (iterator.do_some_stuff.call_count == 10)
Thanks in advance
The reason your code doesn't work, is that MagicMock(spec=Iterator) replaces the __enter__ method of your Iterator class by a MagicMock object, see the MagicMock documentation. This means that in your test, the value of i in function f is a MagicMock object instead of list(range(10)), so the code inside the for loop is never executed.
To make it work, you will probably only want to mock the do_some_stuff method:
#pytest.fixture
def iterator():
it = Iterator()
it.do_some_stuff = Mock()
return it
def test_f(iterator):
f(iterator)
assert (iterator.do_some_stuff.call_count == 10)

class instance from nowhere [duplicate]

If I have a class ...
class MyClass:
def method(arg):
print(arg)
... which I use to create an object ...
my_object = MyClass()
... on which I call method("foo") like so ...
>>> my_object.method("foo")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method() takes exactly 1 positional argument (2 given)
... why does Python tell me I gave it two arguments, when I only gave one?
In Python, this:
my_object.method("foo")
... is syntactic sugar, which the interpreter translates behind the scenes into:
MyClass.method(my_object, "foo")
... which, as you can see, does indeed have two arguments - it's just that the first one is implicit, from the point of view of the caller.
This is because most methods do some work with the object they're called on, so there needs to be some way for that object to be referred to inside the method. By convention, this first argument is called self inside the method definition:
class MyNewClass:
def method(self, arg):
print(self)
print(arg)
If you call method("foo") on an instance of MyNewClass, it works as expected:
>>> my_new_object = MyNewClass()
>>> my_new_object.method("foo")
<__main__.MyNewClass object at 0x29045d0>
foo
Occasionally (but not often), you really don't care about the object that your method is bound to, and in that circumstance, you can decorate the method with the builtin staticmethod() function to say so:
class MyOtherClass:
#staticmethod
def method(arg):
print(arg)
... in which case you don't need to add a self argument to the method definition, and it still works:
>>> my_other_object = MyOtherClass()
>>> my_other_object.method("foo")
foo
In simple words
In Python you should add self as the first parameter to all defined methods in classes:
class MyClass:
def method(self, arg):
print(arg)
Then you can use your method according to your intuition:
>>> my_object = MyClass()
>>> my_object.method("foo")
foo
For a better understanding, you can also read the answers to this question: What is the purpose of self?
Something else to consider when this type of error is encountered:
I was running into this error message and found this post helpful. Turns out in my case I had overridden an __init__() where there was object inheritance.
The inherited example is rather long, so I'll skip to a more simple example that doesn't use inheritance:
class MyBadInitClass:
def ___init__(self, name):
self.name = name
def name_foo(self, arg):
print(self)
print(arg)
print("My name is", self.name)
class MyNewClass:
def new_foo(self, arg):
print(self)
print(arg)
my_new_object = MyNewClass()
my_new_object.new_foo("NewFoo")
my_bad_init_object = MyBadInitClass(name="Test Name")
my_bad_init_object.name_foo("name foo")
Result is:
<__main__.MyNewClass object at 0x033C48D0>
NewFoo
Traceback (most recent call last):
File "C:/Users/Orange/PycharmProjects/Chapter9/bad_init_example.py", line 41, in <module>
my_bad_init_object = MyBadInitClass(name="Test Name")
TypeError: object() takes no parameters
PyCharm didn't catch this typo. Nor did Notepad++ (other editors/IDE's might).
Granted, this is a "takes no parameters" TypeError, it isn't much different than "got two" when expecting one, in terms of object initialization in Python.
Addressing the topic: An overloading initializer will be used if syntactically correct, but if not it will be ignored and the built-in used instead. The object won't expect/handle this and the error is thrown.
In the case of the sytax error: The fix is simple, just edit the custom init statement:
def __init__(self, name):
self.name = name
Newcomer to Python, I had this issue when I was using the Python's ** feature in a wrong way. Trying to call this definition from somewhere:
def create_properties_frame(self, parent, **kwargs):
using a call without a double star was causing the problem:
self.create_properties_frame(frame, kw_gsp)
TypeError: create_properties_frame() takes 2 positional arguments but 3 were given
The solution is to add ** to the argument:
self.create_properties_frame(frame, **kw_gsp)
As mentioned in other answers - when you use an instance method you need to pass self as the first argument - this is the source of the error.
With addition to that,it is important to understand that only instance methods take self as the first argument in order to refer to the instance.
In case the method is Static you don't pass self, but a cls argument instead (or class_).
Please see an example below.
class City:
country = "USA" # This is a class level attribute which will be shared across all instances (and not created PER instance)
def __init__(self, name, location, population):
self.name = name
self.location = location
self.population = population
# This is an instance method which takes self as the first argument to refer to the instance
def print_population(self, some_nice_sentence_prefix):
print(some_nice_sentence_prefix +" In " +self.name + " lives " +self.population + " people!")
# This is a static (class) method which is marked with the #classmethod attribute
# All class methods must take a class argument as first param. The convention is to name is "cls" but class_ is also ok
#classmethod
def change_country(cls, new_country):
cls.country = new_country
Some tests just to make things more clear:
# Populate objects
city1 = City("New York", "East", "18,804,000")
city2 = City("Los Angeles", "West", "10,118,800")
#1) Use the instance method: No need to pass "self" - it is passed as the city1 instance
city1.print_population("Did You Know?") # Prints: Did You Know? In New York lives 18,804,000 people!
#2.A) Use the static method in the object
city2.change_country("Canada")
#2.B) Will be reflected in all objects
print("city1.country=",city1.country) # Prints Canada
print("city2.country=",city2.country) # Prints Canada
It occurs when you don't specify the no of parameters the __init__() or any other method looking for.
For example:
class Dog:
def __init__(self):
print("IN INIT METHOD")
def __unicode__(self,):
print("IN UNICODE METHOD")
def __str__(self):
print("IN STR METHOD")
obj = Dog("JIMMY", 1, 2, 3, "WOOF")
When you run the above programme, it gives you an error like that:
TypeError: __init__() takes 1 positional argument but 6 were given
How we can get rid of this thing?
Just pass the parameters, what __init__() method looking for
class Dog:
def __init__(self, dogname, dob_d, dob_m, dob_y, dogSpeakText):
self.name_of_dog = dogname
self.date_of_birth = dob_d
self.month_of_birth = dob_m
self.year_of_birth = dob_y
self.sound_it_make = dogSpeakText
def __unicode__(self, ):
print("IN UNICODE METHOD")
def __str__(self):
print("IN STR METHOD")
obj = Dog("JIMMY", 1, 2, 3, "WOOF")
print(id(obj))
If you want to call method without creating object, you can change method to static method.
class MyClass:
#staticmethod
def method(arg):
print(arg)
MyClass.method("i am a static method")
I get this error when I'm sleep-deprived, and create a class using def instead of class:
def MyClass():
def __init__(self, x):
self.x = x
a = MyClass(3)
-> TypeError: MyClass() takes 0 positional arguments but 1 was given
You should actually create a class:
class accum:
def __init__(self):
self.acc = 0
def accumulator(self, var2add, end):
if not end:
self.acc+=var2add
return self.acc
In my case, I forgot to add the ()
I was calling the method like this
obj = className.myMethod
But it should be is like this
obj = className.myMethod()

Resources