I was reading and trying to work with below documentation from tensorflow
https://www.tensorflow.org/recommenders/examples/basic_retrieval?hl=sl
In this we have implementation of MovielenseModel class. Let me provide snippet of same code below
class MovielensModel(tfrs.Model):
def __init__(self, user_model, movie_model):
super().__init__()
self.movie_model: tf.keras.Model = movie_model
self.user_model: tf.keras.Model = user_model
self.task: tf.keras.layers.Layer = task
def compute_loss(self, features: Dict[Text, tf.Tensor], training=False) -> tf.Tensor:
# We pick out the user features and pass them into the user model.
user_embeddings = self.user_model(features["user_id"])
# And pick out the movie features and pass them into the movie model,
# getting embeddings back.
positive_movie_embeddings = self.movie_model(features["movie_title"])
# The task computes the loss and the metrics.
return self.task(user_embeddings, positive_movie_embeddings)
In this one usages are not clear and could not find much help in any online documentations
Usage of self.movie_model: tf.keras.Model = movie_model . Looks like its first class object implementation of function but how does this work? When I simply tried d:c=3, just to replicate it worked fine d gets value 3 and c its saying as undefined.
Its an type annotation, check this link here https://docs.python.org/3/library/typing.html. Here self.movie_model is supposed to be an instance of tf.keras.Model it is very useful and helpful as python is dynamically typed language especially in function / method signatures
you can annotate types of inpuit params and the type of return value
Related
I have a BaseClass and two classes (Volume and testing) which inherits from the BaseClass. The class "Volume" use a method "driving_style" from another python module. I am trying to write another method "test_Score" which wants to access variables computed in the method "driving_style" which I want to use to compute further. These results will be accessed to the class "testing" as shown.
from training import Accuracy
import ComputeData
import model
class BaseClass(object):
def __init__(self, connections):
self.Type = 'Stock'
self.A = connections.A
self.log = self.B.log
def getIDs(self, assets):
ids = pandas.Series(assets.ids, index=assets.B)
return ids
class Volume(BaseClass):
def __init__(self, connections):
BaseClass.__init__(self, connections)
self.daystrade = 30
self.high_low = True
def learning(self, data, rootClass):
params.daystrade = self.daystrade
params.high_low = self.high_low
style = Accuracy.driving_style()
return self.Object(data.universe, style)
class testing(BaseClass):
def __init__(self, connections):
BaseClass.__init__(self, connections)
def learning(self, data, rootClass):
test_score = Accuracy.test_score()
return self.Object(data.universe, test_score)
def driving_style(date, modelDays, params):
daystrade = params.daystrade
high_low = params.high_low
DriveDays = model.DateRange(date, params.daystrade)
StopBy = ComputeData.instability(DriveDays)
if high_low:
style = ma.average(StopBy)
else:
style = ma.mean(StopBy)
return style
def test_score(date, modelDays, params):
"want to access the following from the method driving_style:"
DriveDays =
StopBy =
return test_score ("which i compute using values DriveDays and StopBy and use test_score in the method learning inside
the 'class - testing' which inherits some params from the BaseClass")
You can't use locals from a call to a function that was made elsewhere and has already returned.
A bad solution is to store them as globals that you can read from later (but that get replaced on every new call). A better solution might to return the relevant info to the caller along with the existing return values (return style, DriveDays, StopBy) and somehow get it to where it needs to go. If necessary, you could wrap the function into a class and store the computed values as attributes on an instance of the class, while keeping the return type the same.
But the best solution is probably to refactor, so the stuff you want is computed by dedicated methods that you can call directly from test_score and driving_style independently, without duplicating code or creating complicated state dependencies.
In short, basically any time you think you need to access locals from another function, you're almost certainly experiencing an XY problem.
I had a problem with saving weights of TFBertModel wrapped in Keras. the problem is described here in GitHub issue and here in Stack Overflow.The solution proposed in both cases is to use
config = BertConfig.from_pretrained(transformer_model_name)
bert = TFBertMainLayer(config=config,trainable=False)
instead of
bert = TFBertModel.from_pretrained(transformer_model_name, trainable=False)
The problem is that when I change my model to the former code, the accuracy decreases by 10 percent.While the parameters count in both cases are the same. I wonder what is the reason and how can be prevented?
It seems like the performance regression in the code snippet that instantiates MainLayer directly occurs because the pre-trained weights are not being loaded. You can load the weights by either:
Calling TFBertModel.from_pretrained and grabbing the MainLayer from the loaded TFBertModel
Creating the MainLayer directly, then loading the weights in a similar way to from_pretrained
Why This Happens
When you call TFBertModel.from_pretrained, it uses the function TFPreTrainedModel.from_pretrained (via inheritance) which handles a few things, including downloading, caching, and loading the model weights.
class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin, TFGenerationMixin):
...
#classmethod
def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
...
# Load model
if pretrained_model_name_or_path is not None:
if os.path.isfile(os.path.join(pretrained_model_name_or_path, TF2_WEIGHTS_NAME)):
# Load from a TF 2.0 checkpoint
archive_file = os.path.join(pretrained_model_name_or_path, TF2_WEIGHTS_NAME)
...
resolved_archive_file = cached_path(
archive_file,
cache_dir=cache_dir,
force_download=force_download,
proxies=proxies,
resume_download=resume_download,
local_files_only=local_files_only,
)
...
model.load_weights(resolved_archive_file, by_name=True)
(If you read the actual code, a lot has been ...'ed out above).
However, when you instantiate TFBertMainLayer directly, it doesn't do any of this set up work.
#keras_serializable
class TFBertMainLayer(tf.keras.layers.Layer):
config_class = BertConfig
def __init__(self, config, **kwargs):
super().__init__(**kwargs)
self.num_hidden_layers = config.num_hidden_layers
self.initializer_range = config.initializer_range
self.output_attentions = config.output_attentions
self.output_hidden_states = config.output_hidden_states
self.return_dict = config.use_return_dict
self.embeddings = TFBertEmbeddings(config, name="embeddings")
self.encoder = TFBertEncoder(config, name="encoder")
self.pooler = TFBertPooler(config, name="pooler")
... rest of the class
Essentially, you need to make sure these weights are being loaded.
Solutions
(1) Using TFAutoModel.from_pretrained
You can rely on transformers.TFAutoModel.from_pretrained to load the model, then just grab the MainLayer field from the specific subclass of TFPreTrainedModel. For example, if you wanted to access a distilbert main layer, it would look like:
model = transformers.TFAutoModel.from_pretrained(`distilbert-base-uncased`)
assert isinstance(model, TFDistilBertModel)
main_layer = transformer_model.distilbert
You can see in modeling_tf_distilbert.html
that the MainLayer is a field of the model.
This is less code and less duplication, but has a few disadvantages. It's less easy to change the pre-trained model you're going to use, because now you're depending on the fieldname, if you change the model type, you'll have to change the field name (for example in TFAlbertModel the MainLayer field is called albert). In addition, this doesn't seem to be the intended way to use huggingface, so this could change under your nose, and your code could break with huggingface updates.
class TFDistilBertModel(TFDistilBertPreTrainedModel):
def __init__(self, config, *inputs, **kwargs):
super().__init__(config, *inputs, **kwargs)
self.distilbert = TFDistilBertMainLayer(config, name="distilbert") # Embeddings
[DOCS] #add_start_docstrings_to_callable(DISTILBERT_INPUTS_DOCSTRING)
#add_code_sample_docstrings(
tokenizer_class=_TOKENIZER_FOR_DOC,
checkpoint="distilbert-base-uncased",
output_type=TFBaseModelOutput,
config_class=_CONFIG_FOR_DOC,
)
def call(self, inputs, **kwargs):
outputs = self.distilbert(inputs, **kwargs)
return outputs
(2) Re-implementing the weight loading logic from from_pretrained
You can do this by essentially copy/pasting the parts of from_pretrained that are relevant to loading weights. This also has some serious disadvantages, you'll be duplicating logic that that can fall out of sync with the huggingface libraries. Though you could likely write it in a way that is more flexible and robust to underlying model name changes.
Conclusion
Ideally this is something that will get fixed internally by the huggingface team, either by providing a standard function to create a MainLayer, wrapping the weight loading logic into its own function that can be called, or by supporting serialization on the model class.
New to OOP in Python. Was going through the chapter on classes and inheritance in the book Python Crash Course by Eric Matthes. In the following excerpt from the inheritance section of the chapter...:
...the author:
first defines a new __init__() method for the child class ElectricCar (which is a subclass to Car as per definition), and which takes the exact same parameters as it's parent's __init__() method
Within ElectricCar's __init__() method the author used super() to access the Car superclass's __init__() method.
The Author then initiates the my_tesla object which is an instance of the child ElectricCar class and prints the return value from the superclass's get_descriptive_name() method to generate output:
My question is was it necessary to go through all these hoops to achieve this end? When he could have just relied upon the default implicit inheritance as such...
#!/usr/bin/python3
class Car(object):
"""A simple attempt to represent a car."""
def __init__(self, manufacturer, model, year):
"""Initialize attributes to describe a car."""
self.manufacturer = manufacturer
self.model = model
self.year = year
self.odometer_reading = 0
def get_descriptive_name(self):
"""Return a neatly formatted descriptive name."""
long_name = f"{self.year} {self.manufacturer} {self.model}"
return long_name.title()
.
.
.
class ElectricCar(Car):
"""Represent aspects of a car, specific to electric vehicles."""
pass
my_tesla = ElectricCar('tesla', 'model s', 2019)
print(my_tesla.get_descriptive_name())
...to get the same output?
I'm learning PyTorch recently, and this question comes up.
For example, if I have a net inheriting the "torch.nn.Module".
class Net(torch.nn.Module):
def __init__(self, something):
super(net, self).__init__()
self.p1=something
def forward():
pass
net1=Net(123)
net1.cuda() ##Here I can't see what is changed.
Then how can I know whether net1 (and that something) is stored on GPU.
I've read how the *.cuda() works, seems like let all the "children" run the *.cuda(). I tried to see what the "children" are. It seems the net1 above has no children.
To check a simple tensor, you can check the is_cuda attribute. For example:
x = torch.zeros(100).cuda()
y = torch.zeros(100)
print(x.is_cuda) # True
print(y.is_cuda) # False
To check a model, a think the easiest way is using the parameters() method, which returns all trainable parameters of your model.
next(model.parameters()).is_cuda
method 1:
class subnet1():
def __init__()
class subnet2():
def __init__()
class father_net():
def __init__():
self.n1 = subnet1()
self.n2 = subnet2()
def forward():
x = self.n1()
x = self.n2(x)
method 2:
class father_net():
def subnet1():
def subnet2():
def forward():
x = self.subnet1()
x = self.subnet2()
It seems nothing different when I train father_net.
However I still want to get some detail explanations. Any idea?
From object-oriented perspective, method 1 implements the two sub-nets as objects and the forward pass through the nets as a behavior. While in method 2, the two sub-nets are defined as the behavior of father-net object, and hence defined as methods.
From execution perspective, both are same, but it makes more sense to consider sub-nets as entities (objects) and have their respective forward pass (and backward pass) as their respective behavior.
One scenario where method-2 might more meaningful is when you have to explicitly constrain your sub-nets to be only used by father_net.