I am providing a minimal example of what I want to solve. I have defined a class and in that there are some variables defined across different functions. I want to know how to track those variables across functions to get the gradient. I think I have to use tf.GradientTape but I have tried some variants without success.
class A():
def __init__(self):
self.alpha = tf.Variable(2.0)
def f1(self):
wt = self.alpha * 5.0
return wt
def f2(self):
wt_f1 = f1()
with tf.GradientTape() as tape:
wt_f2 = wt_f1 * 10.0
print(tape.gradient(wt_f2, self.alpha))
a = A()
print(a.f2())
The last line returns None. Clearly the derivative of wt_f2 with respect to alpha is 50.0. However, I get None. Any idea? I tried initializing a persistent gradient tape in the __init__ function and use that to watch variables such as wt and self.alpha but that didn't help. Any idea?
Update 1:
Putting wt_f1 call under tape does not work .
class A():
def __init__(self):
self.alpha = tf.Variable(2.0)
def f1(self):
wt = self.alpha * 5.0
return wt
def f2(self):
with tf.GradientTape() as tape:
wt_f1 = f1()
wt_f2 = wt_f1 * 10.0
print(tape.gradient(wt_f2, self.alpha))
This also returns None.
You are printing None. Because f2() returns nothing, so you get None.
Remove print:
a = A()
a.f2()
Furthermore some edits may be good for your written code.
You missed the self before f1() function and this works because you have defined f1 function somewhere else. Anyway add self.f1().
Move print statement outside of tape scope. Because it's better to get gradient where recording is finished.
Add tape.watch() to ensure it is being traced by tape.
class A():
def __init__(self):
self.alpha = tf.Variable(2.0)
def f1(self):
wt = self.alpha * 5.0
return wt
def f2(self):
with tf.GradientTape() as tape:
tape.watch(self.alpha)
wt_f1 = self.f1()
wt_f2 = wt_f1 * 10.0
print(tape.gradient(wt_f2, self.alpha))
Related
I usually use a custom collate_fn and use it as an argument when defining my DataLoader. It usually looks something like:
def collate_fn(batch):
max_len = max([len(b['input_ids']) for b in batch])
input_ids = [b['input_ids'] + ([0] * (max_len - len(b['input_ids'])))]
labels = [b['label'] for b in batch]
return input_ids
As you can see, I'm using 0 for my padding sequence. What I'm wondering is, since language models and their tokenizers use different IDs for padding tokens, is there a way that I can make the collate_fn flexible to take that into account?
I was able to make a workaround by making a Trainer class and making the collate_fn a method. After that I was able to do something like self.pad_token_id = tokenizer.pad_token_id and modify the original collate_fn to use self.pad_token_id rather than a hardcoded value.
I'm still curious if there's any way to do this while keeping collate_fn a top-level function though. For example if there would be any way to pass an argument or something.
<Original>
def collate_fn(batch):
max_len = max([len(b['input_ids']) for b in batch])
input_ids = [b['input_ids'] + ([0] * (max_len - len(b['input_ids']))) for b in batch]
return input_ids
class Trainer():
def __init__(self, tokenizer, ...):
...
def train(self):
train_dataloader = DataLoader(features, collate_fn=collate_fn, ...)
...
<Workaround>
class Trainer():
def __init__(self, tokenizer, ...):
self.pad_token_id = tokenizer.pad_token_id
...
def collate_fn(self, batch):
max_len = max([len(b['input_ids']) for b in batch])
input_ids = [b['input_ids'] + ([self.pad_token_id] * (max_len - len(b['input_ids']))) for b in batch]
return input_ids
def train(self):
train_dataloader = DataLoader(features, collate_fn=self.collate_fn, ...)
...
I am trying to refer to variables (x1,x2,x3,x4) of a member function (distance) inside another method (slope) within the same class. However, it's throwing an error. What am I doing wrong?
class Line():
def __init__ (self,cor1,cor2):
self.cor1 = cor1
self.cor2 = cor2
def distance(self):
x1 = self.cor1[0]
y1 = self.cor1[1]
x2 = self.cor2[0]
y2 = self.cor2[1]
return math.sqrt((x2-x1) ** 2 + (y2-y1) ** 2)
def slope(self):
return (self.y1-self.y2)/(self.x1-self.x2)
I tried moving it to the init method and it works. The following code works for me. However, is there a way to call x1,x2,y1,y2 which are variables of one method from another?
class Line():
def __init__ (self,cor1,cor2):
self.cor1 = cor1
self.cor2 = cor2
self.x1 = self.cor1[0]
self.y1 = self.cor1[1]
self.x2 = self.cor2[0]
self.y2 = self.cor2[1]
def distance(self):
return math.sqrt((self.x2-self.x1) ** 2 + (self.y2-self.y1) ** 2)
def slope(self):
return (self.y1-self.y2)/(self.x1-self.x2)
No, you cannot do that; further, the use of global variables is not recommended inside classes.
Your OOP design could use a class Point to represent a pair of coordinates; that class should also hold the distance method.
In the following example, I added a length method to the class LineSegment, as a property, so you can access it like an attribute:
class Point:
"""represents a pair of coordinates
"""
def __init__(self, x, y):
self.x = x
self.y = y
def distance(self, other):
return ((other.x - self.x)**2 + (other.y - self.y)**2)**.5
class LineSegment:
"""represents a line segment with its start and end Point
"""
def __init__ (self, start, end):
self.start = start
self.end = end
def slope(self):
denom = (self.start.x - self.end.x)
return (self.start.y - self.end.y) / (self.start.x - self.end.x) if denom else float('inf')
#property
def length(self):
return self.start.distance(self.end)
cor1 = Point(1, 2)
cor2 = Point(3, 4)
line = LineSegment(cor1, cor2)
print(line.slope(), line.length)
Consider the two classes below.
class Alpha:
def __init__(self):
pass
def Bar(self, x):
def Foo(mult):
return x * mult
self._Foo = Foo
def Foo(self, mult):
return self._Foo(mult)
class Beta:
def __init__(self):
pass
def Bar(self, x):
self._x = x
def Foo(self, mult):
return self._x * mult
For Alpha with a deferred function _Foo, I believe it is more efficient memory-wise since it only evaluates x when the function is called. For Beta on the other hand, x is stored explicitly as a class attribute.
The question is, where exactly is x stored in Alpha? How efficient is it compared to Beta?
x is not stored in alpha as a class attribute, it is only created when you call the function, Foo. This eliminates the need unnecessary data storage.
I am trying to create a classmethod which can be called again and again, however it only works once and stops. Here is the code:
class NewBytes(bytes):
def __init__(self, var):
self.var= var
#classmethod
def rip(cls):
return cls(var[2:])
a = b"12asd5789"
x = NewBytes(a)
print(x, x.rip(), x.rip().rip(), x.rip().rip().rip())
Here is what I got from this:
b'12asd5789' b'asd5789' b'asd5789' b'asd5789'
However, what I want to have is:
b'12asd5789' b'asd5789' b'd5789' b'789'
Thanks in advance.
Probably you don't actually want a class method, since you need access to instance state here.
class NewBytes(bytes):
def __init__(self, x):
self.x = x
def rip(self):
return type(self)(self.x[2:])
My previous answer of using self.x doesnt make sense since this is a class method (too quick to answer). I think this is a case of the XY problem, see the below example of how to use a class method.
class Test(object):
x = "hey there whats up this is a long string"
#classmethod
def TestFunction(cls):
cls.x = cls.x[3:]
print(cls.x)
print(Test().x)
Test().TestFunction()
Test().TestFunction()
Test().TestFunction()
Test().TestFunction()
Test().TestFunction()
First, let's consider this working example using get and set methods for the variable x
class Foo:
def __init__(self):
self._x = 0
def set_x(self, x):
self._x = x
def get_x(self):
return self._x
class Bar:
def __init__(self, set_method):
self._set_method = set_method
def set_x(self, x):
self._set_method(x)
f = Foo()
f.set_x(5)
print(f.get_x())
# Prints 5
b = Bar(f.set_x)
b.set_x(10)
print(f.get_x())
# Prints 10
As you can see I pass the possibility to set the variable x of the instance f of class Foo, to the instance b of class Bar.
Now, I would like to do the same, but with property decorators instead, roughly like this
class Foo:
def __init__(self):
self._x = 0
#property
def x(self):
return self._x
#x.setter
def x(self, x):
self._x = x
class Bar:
def __init__(self, x_property):
self._x_property = x_property
def set_x(self, x):
self.x_property = x
f = Foo()
f.x = 5
print(f.x)
# Prints 5
b = Bar(f.x)
b.set_x(10)
print(f.x)
# Prints 5
What happens is that the value 5, instead of the property, gets passed to instance b, meaning that b can't access x in instance f. Is there a nice way to solve this?
I would then also like to do the same thing for the get method. In the first code that requires me to pass both methods, but if there is a way to get the second code to work I would hopefully only have to pass on the property which I then can set and get as a normal variable.
I would really want to use the property decorators or similar as it cleans up my code a lot. I use python 3.5.2.
Thanks,
Andreas
You can accomplish this by accessing the fset attribute of Foo.x. Note the use of class-dot notation rather than instance-dot. fset takes two arguments: the instance to access and the value to write. Here is a working example
class Foo:
#property
def x(self):
return self._x
#x.setter
def x(self, x):
self._x = x
class Bar:
def __init__(self, x_property):
self.x_property = x_property
def set_x(self, foo, value):
self.x_property(foo, value)
f = Foo()
f.x = 5
print(f.x)
b = Bar(Foo.x.fset)
b.set_x(f, 10)
print(f.x)
Notice that we had to pass f to set_x because we need it to invoke the setter. You could eliminate the f param by using partial from the functools module to bind f to the property setter. Pass the partial binding in to the constructor of Bar.
class Bar:
def __init__(self, x_property):
self.x_property = x_property
def set_x(self, value):
self.x_property(value)
f = Foo()
b = Bar(partial(Foo.x.fset, f))
b.set_x(10)
print(f.x)
It might be wise to rename x_property and this point. It is really just a function as far as Bar is concerned. It wouldn't have to be a property.