I have a pytorch tensor that has shape [n1, n2, n3]. I need to make the shape [n1, n2, n3, 1].
So I know I can use either unsqueeze or view. Is there a difference in what each one would do in this case?
You can achieve this with four different solutions. There are slight differences between those. More precisely you can:
insert a new singleton dimension with torch.Tensor.unqueeze:
>>> x.unsqueeze(-1) # grad_fn=<UnsqueezeBackward0>
use fancy indexing to add a new dimension which is identical:
>>> x[..., None] # grad_fn=<UnsqueezeBackward0>
or similarly with torch.Tensor.view:
>>> x.view(*x.shape, 1) # grad_fn=<ViewBackward0>
add a new dimension with torch.Tensor.reshape:
>>> x.reshape(*x.shape, 1) # grad_fn=<ReshapeAliasBackward0>
I have added the backward gradient function name as a line comment next to each method. You can see how indexing and unsqueezing are the same, while view and reshape rely on two different methods.
All three methods: indexing, unsqueeze, and view will return a view of the tensor while reshape can return a copy of the tensor if needed (i.e. when the data is not contiguous).
You can read more about the differences between torch.view and torch.reshape on this thread.
Related
I want to transpose the data I have using transpose, but I am encountering such an error. My data and related process uploaded to github.
https://github.com/nurkbts/error/blob/main/error.ipynb
When using torch.bmm (batch matrix multiplication), both tensors must have three dimensions (first one being the batch). Please read the documentation for the details.
Since you were trying to use bmm, you should just use the # operator (equivalent to applying torch.matmul). Also, don't forget to transpose. This will give you a shape (64, 64).
_scores = queries#keys.T / np.sqrt(64)
Currently I have the following code using TimeseriesGenerator from Keras:
TimeseriesGenerator(train, prediction, length=TIME_STEPS, batch_size=1)
Currently this shifts prediction one value backwards, so the train data for t will have the output of t+1. Which makes sense, but I want to predict t+2, thus train data for t will have the output of t+2.
Is there any way to do it using TimeseriesGenerator?
The quickest solution is to just shift your predictions by 1, ie.:
TimeseriesGenerator(train[:-1], prediction[1:], length=TIME_STEPS, batch_size=1)
Note that you have to trim the train set, so both datasets have equal lengths.
You can also use the timeseries_dataset_from_array function where you can align the data and targets according to your needs as you can read in the documentation:
data: Numpy array or eager tensor containing consecutive data points
(timesteps). Axis 0 is expected to be the time dimension.
targets:
Targets corresponding to timesteps in data. It should have same length
as data. targets[i] should be the target corresponding to the window
that starts at index i (see example 2 below). Pass None if you don't
have target data (in this case the dataset will only yield the input
data).
So in your case it would be something like this:
tf.keras.preprocessing.timeseries_dataset_from_array(
train[:-TIME_STEPS-2],
prediction[TIME_STEPS+2:],
length=TIME_STEPS,
batch_size=1
)
As you know, in order to utilize tf.estimator, one needs to implement the model function builds a pipeline that yields batches of (features, labels) pairs, therefore the signature should be as following:
model_fn(features, labels, mode, params, config):
These features and labels should be returned from the input_fn. We assume that features -> X, and labels-> y, I am having a problem here because I have two type of labels.(targets, labels)
Features = X : [None, 2048]
Labels = targets: [None, 2048]
labels: [None, 1]
In order to provide targets and labels as separate arguments instead of just one label argument, what would be the alternative?
Note: I tried to concatenate targets and labels, then slice them where it needs but it created an additional problem during execution of the model. Therefore I am wondering whether you guys have any other better ideas or not?
Thank you.
In your input_fn, you can simply return a dictionary instead of a tensor as labels. That is, your input function likely returns an iterator over a tuple (features, labels). Both features and labels can either be a single tensor or a dict. This dict should map from strings to tensors.
You can prepare the dataset as one returning three elements (features, targets, labels), and then include a mapping to pack the targets into a dict (there might be better ways but this works):
data = ... # prepare dataset of 3-tuples
def pack_in_dict(features, targets, labels):
return features, {"targets": targets, "labels": labels}
data = data.map(pack_in_dict)
Now, if one of the elements is a dict (say, labels), then the corresponding input to model_fn will also be a dict. You can then simply use labels["targets"] and labels["labels"] in your model_fn.
I am currently working with multilayer LSTM using tensorflow and python.
I am passing the previous state of the cells to the next step using initial_state in tf.nn.dynamic_rnn.
Creating the layers like that:
cells = []
for c in range(0, num_layers):
cells.append(tf.nn.rnn_cell.BasicLSTMCell(num_units = num_units, forget_bias = 1.0, activation = tf.nn.tanh))
basic_cell = tf.nn.rnn_cell.MultiRNNCell(cells)
state_series, current_state = tf.nn.dynamic_rnn(basic_cell, x, dtype=tf.float32, initial_state = rnn_tuple_state)
With types and dimensions like that:
rnn_tuple_state: <class 'tuple'> len:num_layers
Layer 0 : <class 'tensorflow.python.ops.rnn_cell_impl.LSTMStateTuple'> len:2
cell: <class 'tensorflow.python.framework.ops.Tensor'> dimensions:(outputs, truncated_backprop_len)
hidden: <class 'tensorflow.python.framework.ops.Tensor'> dimensions:(outputs, truncated_backprop_len)
Layer 1 : ...
...
Layer num_layers : ...
state_series: <class 'tensorflow.python.framework.ops.Tensor'> dimension:(outputs, truncated_backprop_len, num_units)
current_state: <class 'tuple'> len:num_layers
Layer 0 : <class 'tensorflow.python.ops.rnn_cell_impl.LSTMStateTuple'> len:2
cell_state: <class 'tensorflow.python.framework.ops.Tensor'> (7, 360) dimensions:(outputs, truncated_backprop_len)
hidden_state: <class 'tensorflow.python.framework.ops.Tensor'> (7, 360) dimensions:(outputs, truncated_backprop_len)
Layer 1 : ...
...
Layer num_layers: ...
Using this I was able to realize state-space LSTM looking like that:
simple_initial-state
In the picture in blue are the zero_state, in green the 2 LSTM-layers, the columns are all the same cells and should only show the recurrences, the arrows shows the passing of the states from one step to the next.
Now I want to use a more complex initial_state, passing states not only from one step to the next but also from one layer to the other like that:
wanted complex initial_state
And that is where I got stuck now.
I played around a bit with just adding additional tuples to the initial_state but that leads me just to errors like ValueError: too many values to unpack (expected 2).
I was also looking at other types of LSTM-cells but was not able to identify the needed.
So my question is, how can I realize this more complex initial_state in tensorflow, e.g. which cell types to use or how to shape the initial_state?
Thank you in advance.
Your cell state must be a single vector of numbers which has to be of the size defined in the LSTM cell. The LSTM cell will output the same size as it expects for its input. When you have multiple layers you are generally passing information from one layer to the next via the output of the first layer to the input of the second layer. You don't generally pass the state up. However, if you want to experiment in this way, I suppose the way to do it would be to add the two states together, as you can't really change the shape. You might consider the precedence of residual networks as justification for such an idea (though I have no idea if it's a good idea or not).
If you want to output a different cell state size than you input so that you can actually concatenate the state of the layer before, within the input cell state, you are going to have to code the RNN by hand, no LSTM cell wrapper is going to allow this kind of configuration.
To get a better sense of the internals of the LSTM cell I'd read up here:
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
I'm trying to perform feature selection by evaluating my regressions coefficient outputs, and select the features with the highest magnitude coefficients. The problem is, I don't know how to get the respective features, as only coefficients are returned form the coef._ attribute. The documentation says:
Estimated coefficients for the linear regression problem. If multiple
targets are passed during the fit (y 2D), this is a 2D array of
shape (n_targets, n_features), while if only one target is passed,
this is a 1D array of length n_features.
I am passing into my regression.fit(A,B), where A is a 2-D array, with tfidf value for each feature in a document. Example format:
"feature1" "feature2"
"Doc1" .44 .22
"Doc2" .11 .6
"Doc3" .22 .2
B are my target values for the data, which are just numbers 1-100 associated with each document:
"Doc1" 50
"Doc2" 11
"Doc3" 99
Using regression.coef_, I get a list of coefficients, but not their corresponding features! How can I get the features? I'm guessing I need to modfy the structure of my B targets, but I don't know how.
What I found to work was:
X = your independent variables
coefficients = pd.concat([pd.DataFrame(X.columns),pd.DataFrame(np.transpose(logistic.coef_))], axis = 1)
The assumption you stated: that the order of regression.coef_ is the same as in the TRAIN set holds true in my experiences. (works with the underlying data and also checks out with correlations between X and y)
You can do that by creating a data frame:
cdf = pd.DataFrame(regression.coef_, X.columns, columns=['Coefficients'])
print(cdf)
coefficients = pd.DataFrame({"Feature":X.columns,"Coefficients":np.transpose(logistic.coef_)})
I suppose you are working on some feature selection task. Well using regression.coef_ does get the corresponding coefficients to the features, i.e. regression.coef_[0] corresponds to "feature1" and regression.coef_[1] corresponds to "feature2". This should be what you desire.
Well I in its turn recommend tree model from sklearn, which could also be used for feature selection. To be specific, check out here.
Coefficients and features in zip
print(list(zip(X_train.columns.tolist(),logreg.coef_[0])))
Coefficients and features in DataFrame
pd.DataFrame({"Feature":X_train.columns.tolist(),"Coefficients":logreg.coef_[0]})
This is the easiest and most intuitive way:
pd.DataFrame(logisticRegr.coef_, columns=x_train.columns)
or the same but transposing index and columns
pd.DataFrame(logisticRegr.coef_, columns=x_train.columns).T
Suppose your train data X variable is 'df_X' then you can map into a dictionary and feed into pandas dataframe to get the mapping:
pd.DataFrame(dict(zip(df_X.columns,model.coef_[0])),index=[0]).T
Try putting them in a series with the data columns names as index:
coeffs = pd.Series(model.coef_[0], index=X.columns.values)
coeffs.sort_values(ascending = False)