To facilitate managing trainable hyperparameters, I am looking for a way to create a class Hyperparameter that acts both as a nn.Parameter and a nn.Module. In particular, I would like to use Hyperparameter objects both as a nn.Parameter (e.g. for tensor operations) but still have access to the interface provided by nn.Module to for example store the objects in a nn.ModuleDict along with other modules or use methods like zero_grad(), parameters().
I tried to accomplish this through multiple inheritance but it didn't quite work out:
import torch
class Hyperparameter(torch.nn.Parameter, torch.nn.Module):
def __new__(cls, tensor, name):
return torch.nn.Parameter.__new__(cls, data=tensor)
def __init__(self, tensor, name):
torch.nn.Parameter.__init__(self)
torch.nn.Module.__init__(self)
self.register_parameter(name, self)
hp1 = Hyperparameter(torch.ones(5), "test1")
hp2 = Hyperparameter(torch.ones(8), "test2")
# Examples of what I want to do, which already work
tmp = hp1 * hp2
hp_dict = torch.nn.ModuleDict({"hp1": hp1, "hp2": hp2})
# What does not work with this solution
hp_dict.to(torch.device("cpu"))
# KeyError: "attribute 'data' already exists"
This works for the things I described (can be added to ModuleDict, can perform algebraic operations, ...), but calling to() throws an error. I think something is no longer as nn.Module expects it to be, but I do not understand what it is.
EDIT: Here is the full stack trace as requested:
Traceback (most recent call last):
File "tmp.py", line 16, in <module>
hp_dict.to(torch.device("cpu"))
File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 612, in to
return self._apply(convert)
File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 359, in _apply
module._apply(fn)
File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 384, in _apply
param.data = param_applied
File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 796, in __setattr__
self.register_parameter(name, value)
File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 316, in register_parameter
raise KeyError("attribute '{}' already exists".format(name))
KeyError: "attribute 'data' already exists"
I'm not sure why you need both, nn.Module and nn.Parameter at the same object.
You can have a nn.Module that is basically the parameter:
class Hyperparameter(torch.nn.Module):
def __init__(self, tensor, name):
super(Hyperparameter, self).__init__()
self.register_parameter(name=name, param=nn.Parameter(tensor))
self._name = name
def forward(self):
return getattr(self, self._mame) # expose the parameter via forward
Now you can have a Hyperparameter Module:
my_hp = Hyperparameter(name='hyper', data=torch.arange(3.).requires_grad_(True))
Now you can access the hyper parameter hyper in several ways:
In [1]: list(my_hp.named_parameters())
Out[1]:
[('hyper', Parameter containing:
tensor([0., 1., 2.], requires_grad=True))]
In [2]: my_hp.hyper
Out[2]:
Parameter containing:
tensor([0., 1., 2.], requires_grad=True)
In [3]: my_hp()
Out[3]:
Parameter containing:
tensor([0., 1., 2.], requires_grad=True)
Related
I am working on a class speedup with Numba. When you want to use Numba inside classes you have to define/preallocate your class variables. In this respect my issue is declaring a 2D array before the jitclass. The following MWE directly shows my issue:
import numpy as np
from numba import int32, float32
from numba.experimental import jitclass # import the decorator
spec = [
('value', int32), # a simple scalar field
('array', float32[:]), # an array field
('foo_matrix',int32[:,:]),
]
#jitclass(spec)
class Bag(object):
def __init__(self, value):
self.value = value
self.array = np.zeros(value)
self.foo_matrix = np.zeros((value, value))
#property
def size(self):
return self.array.size
def increment(self, val):
for i in range(self.size):
self.array[i] = val
return self.array
my_class = Bag(3)
When I execute this code I get the following error:
Traceback (most recent call last):
File "/home/acer/codici/tech/numba_prototype.py", line 38, in <module>
my_class = Bag(3)
File "/usr/lib/python3/dist-packages/numba/experimental/jitclass/base.py", line 122, in __call__
return cls._ctor(*bind.args[1:], **bind.kwargs)
File "/usr/lib/python3/dist-packages/numba/core/dispatcher.py", line 414, in _compile_for_args
error_rewrite(e, 'typing')
File "/usr/lib/python3/dist-packages/numba/core/dispatcher.py", line 357, in error_rewrite
raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Internal error at <numba.core.typeinfer.CallConstraint object at 0x7fc6d4945c40>.
Failed in nopython mode pipeline (step: nopython mode backend)
Can only insert float* at [4] in {i8*, i8*, i64, i64, float*, [1 x i64], [1 x i64]}: got double*
File "numba_prototype.py", line 19:
def __init__(self, value):
<source elided>
self.value = value
self.array = np.zeros(value)
^
During: lowering "(self).array = $14call_method.5" at /home/acer/codici/tech/numba_prototype.py (19)
During: resolving callee type: jitclass.Bag#7fc6d5a2afa0<value:int32,array:array(float32, 1d, A),foo_matrix:array(int32, 2d, A)>
During: typing of call at <string> (3)
Enable logging at debug level for details.
File "<string>", line 3:
<source missing, REPL/exec in use?>
which is related to the declaration of the matrix foo_matrix.
Concerning the types definition I followed this.
Of course if I comment out the lines about array declaration and population the code works fine.
How should I modify/do the declaration of the matrix with respect to a jitclass object?
EDIT: inside the class, I have changed the declaration of foo_matrix from np.zeros([value, value]) to np.zeros((value, value)) since defining a numpy array using a list instead of a tuple may be a source of error for numba functions. However the problem persists even with this modification.
The error is due to the fact that in the zeros array and matrix there is no the type specification i.e. : self.array = np.zeros(value, dtype=np.float32).
The working code is the following:
import numpy as np
from numba import int32, float32
from numba.experimental import jitclass
spec = [
('value', int32),
('array', float32[:]),
('foo_matrix',int32[:,:]),
]
#jitclass(spec)
class Bag(object):
def __init__(self, value):
self.value = value
self.array = np.zeros(value, dtype=np.float32)
self.foo_matrix = np.zeros((value, value), dtype=np.int32)
alice = value +1
bob = np.sum(self.array)
#property
def size(self):
return self.array.size
def increment(self, val):
for i in range(self.size):
self.array[i] = val
return self.array
my_class = Bag(3)
Hi I have a function called
tfnet.return_predict()
which when run on an image outputs certain set o values such as the class of object confidence and coordinates of bounding box. What i want to do is make a wrapper which returns only the confidence value.
So my code is as follows. I am using Darkflow to perform Prediction of classes on images.
#Initialise Libraries
# Load the YOLO Neural Network
tfnet = TFNet(options) #call the YOLO network
image = cv2.imread('C:/darkflow/Car.jpg', cv2.IMREAD_COLOR) #Load image
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
print(tfnet.return_predict(image)) #function to run predictions
The output of print is
[{'label': 'Car', 'confidence': 0.32647023, 'topleft': {'x': 98, 'y': 249}, 'bottomright': {'x': 311, 'y': 455}}]
So from this i want to create a wrapper which just returns the 'confidence' value.
I know how to create wrappers and define functions for it but how to do it for already defined functions.
Any suggestion is of great help to mee
EDIT: I tried:
def log_calls(tfnet.return_predict):
def wrapper(*args, **kwargs):
#name = func.__name__
print('before {name} was called')
r = func(*args, **kwargs)
print('after {name} was called')
return r
return wrapper
But the 'tfnet.return_predict' is returning error
SyntaxError: invalid syntax
Do you need to redefine the tfnet.return_predict function to only return confidence? Or is having a separate function okay? If it's the latter, then it seems like you can just do this:
def conf_only(*args, **kwargs):
out = tfnet.return_predict(*args, **kwargs)
return out[0]["confidence"]
and calling conf_only returns just that part of the dict.
If you need to have tfnet.return_predict redefined and want that to only return confidence, then you can make a decorator:
def conf_deco(func):
def wrapper(*args, **kwargs):
return func(*args, **kwargs)[0]["confidence"]
return wrapper
For example, pretending dummy_function is already predefined
def dummy_function(*args, **kwargs):
print(args, kwargs)
return [{"confidence": .32, "other": "asdf"}]
In [4]: dummy_function("something", kw='else')
('something',) {'kw': 'else'}
Out[4]: [{'confidence': 0.32, 'other': 'asdf'}]
Now redefine it with:
In [6]: dummy_function = conf_deco(dummy_function)
and it'll only return the confidence value
In [7]: dummy_function("something", kw='else')
('something',) {'kw': 'else'}
Out[7]: 0.32
I would like to create a class that uses sklearn transformation methods. I found this article and I am using it as an example.
from sklearn import preprocessing
from sklearn.base import TransformerMixin
def minmax(dataframe):
minmax_transformer = preprocessing.MinMaxScaler()
return minmax_tranformer
class FunctionFeaturizer(TransformerMixin):
def __init__(self, scaler):
self.scaler = scaler
def fit(self, X, y=None):
return self
def transform(self, X):
fv = self.scaler(X)
return fv
if __name__=="__main__":
scaling = FunctionFeaturizer(minmax)
df = pd.DataFrame({'feature': np.arange(10)})
df_scaled = scaling.fit(df).transform(df)
print(df_scaled)
The output is StandardScaler(copy=True, with_mean=True, with_std=True) which is actually the result of the preprocessing.StandardScaler().fit(df) if I use it out of the class.
What I am expecting is:
array([[0. ],
[0.11111111],
[0.22222222],
[0.33333333],
[0.44444444],
[0.55555556],
[0.66666667],
[0.77777778],
[0.88888889],
[1. ]])
I am feeling that I am mixing few things here but I do not know what.
Update
I did some modifications:
def minmax():
return preprocessing.MinMaxScaler()
class FunctionFeaturizer(TransformerMixin):
def __init__(self, scaler):
self.scaler = scaler
def fit(self, X, y=None):
return self
def fit_transform(self, X):
self.scaler.fit(X)
return self.scaler.transform(X)
if __name__=="__main__":
scaling = FunctionFeaturizer(minmax)
df = pd.DataFrame({'feature': np.arange(10)})
df_scaled = scaling.fit_transform(df)
print(df_scaled)
But now I am receiving the following error:
Traceback (most recent call last):
File "C:/my_file.py", line 33, in <module>
test_scale = scaling.fit_transform(df)
File "C:/my_file.py", line 26, in fit_transform
self.scaler.fit(X)
AttributeError: 'function' object has no attribute 'fit'
Solving your error
in your code you have:
if __name__=="__main__":
scaling = FunctionFeaturizer(minmax)
df = pd.DataFrame({'feature': np.arange(10)})
df_scaled = scaling.fit_transform(df)
print(df_scaled)
change the line
scaling = FunctionFeaturizer(minmax)
to
scaling = FunctionFeaturizer(minmax())
you need to call the function to get the instantiation of MinMaxScaler returned to you.
Suggestion
Instead of implementing fit and fit_transform, implement fit and transform unless you can optimize both process into fit_tranform. This way, it is clearer what you are doing.
If you implement only fit and transform, you can still call fit_transform because you extend the TransformerMixin class. It will just call both functions in a row.
Getting your expected results
Your transformer is looking at every column of your dataset and distributing the values linearly between 0 and 1.
So, to get your expected results, it will really depend on what your df looks like. However, you did not share that with us, so it is difficult to tell if you will get it.
However, if you have df = [[0],[1],[2],[3],[4],[5],[6],[7],[8],[9]], you will see your expected result.
if __name__=="__main__":
scaling = FunctionFeaturizer(minmax())
df = [[0], [1], [2], [3], [4], [5], [6], [7], [8], [9]]
df_scaled = scaling.fit_transform(df)
print(df_scaled)
> [[0. ]
> [0.11111111]
> [0.22222222]
> [0.33333333]
> [0.44444444]
> [0.55555556]
> [0.66666667]
> [0.77777778]
> [0.88888889]
> [1. ]]
I started learning Machine Learning and came across Neural Networks. while implementing a program i got this error. i have tried checking for every solution but no luck. here's my code:
from numpy import exp, array, random, dot
class neural_network:
def _init_(self):
random.seed(1)
self.weights = 2 * random.random((2, 1)) - 1
def train(self, inputs, outputs, num):
for iteration in range(num):
output = self.think(inputs)
error = outputs - output
adjustment = 0.01*dot(inputs.T, error)
self.weights += adjustment
def think(self, inputs):
return (dot(inputs, self.weights))
neural = neural_network()
# The training set
inputs = array([[2, 3], [1, 1], [5, 2], [12, 3]])
outputs = array([[10, 4, 14, 30]]).T
# Training the neural network using the training set.
neural.train(inputs, outputs, 10000)
# Ask the neural network the output
print(neural.think(array([15, 2])))
this is the error which i'm getting when running neural.train:
Traceback (most recent call last):
File "neural.py", line 27, in <module>
neural.train(inputs, outputs, 10000)
File "neural.py", line 10, in train
output = self.think(inputs)
File "neural.py", line 16, in think
return (dot(inputs, self.weights))
AttributeError: 'neural_network' object has no attribute 'weights'
Though its has a self attribute self.weights() still it says no such attribute.
Well, it turns out that your initialization method should be named __init__ (two underscores), not _init_...
So, changing the method to
def __init__(self):
random.seed(1)
self.weights = 2 * random.random((2, 1)) - 1
your code works OK:
neural.train(inputs, outputs, 10000)
print(neural.think(array([15, 2])))
# [ 34.]
Your initializing method is written wrong, its two underscores __init__(self): not one underscore_init_(self):
Otherwise, nice code!
Recently I built a customized deep neural net model using TFLearn, which claims to bring deep learning to the scikit-learn estimator API. I could train models and make predictions, but I couldn't get the scoring (evaluate) function to work, so I couldn't do cross-validation. I tried to ask questions about TFLearn in various places, but I got no responses.
It appears that TensorFlow itself has an estimator class. So I am putting TFLearn aside, and I'm trying to follow the guide at https://www.tensorflow.org/extend/estimators. Somehow I'm managing to get variables where they don't belong. Can anyone spot my problem? I will post code and the output.
Note: Of course, I can see the RuntimeWarning at the top of the output. I have found references to this warning online, but so far everyone claims it's harmless. Maybe it is not...
CODE:
import tensorflow as tf
from my_library import Database, l2_angle_distance
def my_model_function(topology, params):
# This function will eventually be a function factory. This should
# allow easy exploration of hyperparameters. For now, this just
# returns a single, fixed model_fn.
def model_fn(features, labels, mode):
# Input layer
net = tf.layers.conv1d(features["x"], topology[0], 3, activation=tf.nn.relu)
net = tf.layers.dropout(net, 0.25)
# The core of the network is here (convolutional layers only for now).
for nodes in topology[1:]:
net = tf.layers.conv1d(net, nodes, 3, activation=tf.nn.relu)
net = tf.layers.dropout(net, 0.25)
sh = tf.shape(features["x"])
net = tf.reshape(net, [sh[0], sh[1], 3, 2])
predictions = tf.nn.l2_normalize(net, dim=3)
# PREDICT EstimatorSpec
if mode == tf.estimator.ModeKeys.PREDICT:
return tf.estimator.EstimatorSpec(mode=mode,
predictions={"vectors": predictions})
# TRAIN or EVAL EstimatorSpec
loss = l2_angle_distance(labels, predictions)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=params["learning_rate"])
train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
return tf.estimator.EstimatorSpec(mode, predictions, loss, train_op)
return model_fn
##===================================================================
window = "whole"
encoding = "one_hot"
db = Database("/home/bwllc/Documents/Files for ML/compact")
traindb, testdb = db.train_test_split()
train_features, train_labels = traindb.values(window, encoding)
test_features, test_labels = testdb.values(window, encoding)
# Create the model.
tf.logging.set_verbosity(tf.logging.INFO)
LEARNING_RATE = 0.01
topology = (60,40,20)
model_params = {"learning_rate": LEARNING_RATE}
model_fn = my_model_function(topology, model_params)
model = tf.estimator.Estimator(model_fn, model_params)
print("\nmodel_dir? No? Why not? ", model.model_dir, "\n") # This documents the error
# Input function.
my_input_fn = tf.estimator.inputs.numpy_input_fn({"x" : train_features}, train_labels, shuffle=True)
# Train the model.
model.train(input_fn=my_input_fn, steps=20)
OUTPUT
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': {'learning_rate': 0.01}, '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f0b55279048>, '_task_type': 'worker', '_task_id': 0, '_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
model_dir? No? Why not? {'learning_rate': 0.01}
INFO:tensorflow:Create CheckpointSaverHook.
Traceback (most recent call last):
File "minimal_estimator_bug_example.py", line 81, in <module>
model.train(input_fn=my_input_fn, steps=20)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 302, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 756, in _train_model
scaffold=estimator_spec.scaffold)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 411, in __init__
self._save_path = os.path.join(checkpoint_dir, checkpoint_basename)
File "/usr/lib/python3.6/posixpath.py", line 78, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not dict
------------------
(program exited with code: 1)
Press return to continue
I can see exactly what went wrong, model_dir (which I left as the default) somehow bound to the value I intended for model_params. How did this happen in my code? I can't see it.
If anyone has advice or suggestions, I would greatly appreciate them. Thanks!
Simply because you're feeding your model_param as a model_dir when you construct your Estimator.
From the tensorflow documentation :
Estimator __init__ function :
__init__(
model_fn,
model_dir=None,
config=None,
params=None
)
Notice how the second argument is the model_dir one. If you want to specify only the params one, you need to pass it as a keyword argument.
model = tf.estimator.Estimator(model_fn, params=model_params)
Or specify all the previous positional arguments :
model = tf.estimator.Estimator(model_fn, None, None, model_params)