I have a prediction model that predicts several future states of a system. I calculate metrics per future timestep. So after the test run is finished I return a dictionary like this:
metrics = {'mae_0': some_scalar, 'mse_0': some_scalar,
'mae_1': some_scalar, 'mse_1:' some_scalar,
'mae_2': ...
.
.
.
'mae_n': some_scalar, 'mse_n': some_scalar}
I'd like to plot these in tensorboard with [0, 1, 2, ..., n] as the x-values.
However from what I found so far I can only use step, batch or epoch as x-value, which is identical so I get a vertical line of values instead of a plot.
Is there a way to provide custom x-values to the tensorboard logger?
Related
I am pretty new to ML and I am studying the k-nearest neighbors classifier using python documentation
I am somehow confused about the training part. Let's say my training data is some points in 1d
training = [[1], [4], [3]]
and I want to use k-nearest neighbor classifier to label them into to "teams" :
labels = [[0], [1]]
Why does that doesn't make sense ?
I get an error that target values size does not match the input.
If I put instead labels = [[0], [1], [1]] or labels = [[0], [0], [1]]
it will compile .
Also a side-note question : does the permutation of labels matter?
It looks like you're trying to clusterize your data, which is done by sklearn.neighbors.NearestNeighbors(). sklearn.neighbors.KNeighborsClassifier() is a supervized model which requires supplying the actual class for every observation for training and can predict the class for the previously unseen data after that.
However, NearestNeighbors() method does not allow to limit the amount of clusters iirc, you should probably try something like sklearn.cluster.KMeans(n_clusters=2).
https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly
The like above is the documentation regarding the custom keras data generator.
I have doubt in the "NOTATION" heading in the above link which says the following:-
Before getting started, let's go through a few organizational tips that are particularly useful when dealing with large datasets.
Let ID be the Python string that identifies a given sample of the dataset. A good way to keep track of samples and their labels is to adopt the following framework:
1. Create a dictionary called partition where you gather:
a) in partition['train'] a list of training IDs
b) in partition['validation'] a list of validation IDs
2. Create a dictionary called labels where for each ID of the dataset, the associated label is given by labels[ID]
For example, let's say that our training set contains id-1, id-2 and id-3 with respective labels 0, 1 and 2, with a validation set containing id-4 with label 1. In that case, the Python variables partition and labels look like
>>> partition
{'train': ['id-1', 'id-2', 'id-3'], 'validation': ['id-4']}
and
>>> labels
{'id-1': 0, 'id-2': 1, 'id-3': 2, 'id-4': 1}
I'm really not able to understand what does labels and id's mean.
For example:- Say, I have a data frame, where there are 1000 columns. Each row corresponds to id's i.e., each ID meant to be just a "DATA POINT".
OR
Say, I have multiple data frame. Each data frame represents different id's?
It seems labels meant not to be the number of class-variable.
I would like to have a clear understanding regarding id's and labels WITH SOME EXAMPLES.
The mentioned article provides a good practice to better organize your data between training and validation. To do so, it's relevant to store line indexes from your dataframe (named IDs here) and corresponding target values (named label here) in an independent object so that in case of transformation on the input, you don't lose track of things.
Here is a basic example using a train/test split
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.DataFrame([[0.1, 1, 'label_a'], [0.2, 2, 'label_a'], [0.3, 3, 'label_a'], [0.4, 4, 'label_b']], columns=['feature_a', 'feature_b', 'target'])
# df.index.tolist() results in [0, 1, 2, 3] (4 rows)
partitions = dict()
labels = dict()
X_train, X_test, y_train, y_test = train_test_split(df[['feature_a', 'feature_b']], df['target'], test_size=0.25, random_state=42)
partitions['train'] = X_train.index.tolist()
partitions['validation'] = X_test.index.tolist()
# partitions['train'] results in [3, 0, 2]
# partitions['validation'] results in [1]
labels = df['target'].to_dict()
# labels is {0: 'label_a', 1: 'label_a', 2: 'label_a', 3: 'label_b'}```
I am using multiple linear regression to predict the temperature in every region of a field where wireless sensors are deployed, the sensors are as follows : 42 sensors deployed in a 1000x600 m² surface and collecting the temperature in each of these 42 locations per hour, see picture:
Sensors placement
We have here two features ( the location aka : x and y ), and the output which is the temperature, so i fit my model according to 70% of the dataset, for the sake of later accuracy computations, however after fitting my model I want to have the temperatures prediction over all the surface, specifically a heat map that gives me the temperature as a function of x and y ( see picture : Heatmap)
I am stuck in the part of visualization, as my dataset contains the 42 known locations and their respective temperatures, how can i plot predictions for every x in [0,1000] and y in [0,600]
Do i have to make an nx2 matrix iterating over all the values of x and y and then feeding it my fitted model ? or is there a simpler way
You can use np.meshgrid to create a grid of points, then use your model to predict on this grid of points.
import numpy as np
import matplotlib.pyplot as plt
grid_x, grid_y = np.meshgrid(np.linspace(0, 1000, 100),
np.linspace(0, 600, 60))
X = np.stack([grid_x.ravel(), grid_y.ravel()]).T
y_pred = model.predict(X) # use your scikit-learn model here
image = np.reshape(y_pred, grid_x.shape)
plt.imshow(image, origin="lower")
plt.colorbar()
plt.show()
Convolution for a grayscale image is straightforward. You have a filter of shape nxnx1and convolve the input image to extract whatever features you desire.
I also understand how convolution would work for a RGB image. The filter would have a shape of nxnx3. However, would all 3 'layers' in the filter hold the same kernel? For example, if the 0th layer a map as shown below, would layer 1 and 2 also hold the exact values? I am asking in regards to Convolutional Neural Networks and not conventional image processing. I understand the weights of each filter are learned and are randomized initially, am I correct in thinking that each layer would have different randomized values?
Would all 3 'layers' in the filter hold the same kernel?
The short answer is no. The longer answer is, there isn't a kernel per layer, but instead just one kernel which handles all input and output layer at once.
The code below shows step by step how one would calculate each convolution manually, and from this we can see that at a high level the calculation goes like this:
take a patch from the batch of images (BatchSize x 3x3x3 in your case)
flatten it [BatchSize, 27]
matrix multiply it by the reshaped kernel [27, output_filters]
add in the bias of shape [output_filters]
All the colors are processed at once using matrix multiplication with the kernel matrix. If we think about the kernel matrix, we can see that the values in the kernel matrix that are used to generate the first filter are in the first column, and the values to generate the second filter are in the second column. So, indeed, the values are different and not reused, but they are not stored or applied separately.
The code walkthrough
import tensorflow as tf
import numpy as np
# Define a 3x3 kernel that after convolution will create an image with 2 filters (channels)
conv_layer = tf.keras.layers.Conv2D(filters=2, kernel_size=3)
# Lets create a random input image
starting_image = np.array( np.random.rand(1,4,4,3), dtype=np.float32)
# and process it
result = conv_layer(starting_image)
weight, bias = conv_layer.get_weights()
print('size of weight', weight.shape)
print('size of bias', bias.shape)
size of weight (3, 3, 3, 2)
size of bias (2,)
# The output of the convolution of the 4x4x3 image input
# is a 2x2x2 output (because we don't have padding)
result.numpy()
array([[[[-0.34940776, -0.6426925 ],
[-0.81834394, -0.16166998]],
[[-0.37515935, -0.28143463],
[-0.60084903, -0.5310158 ]]]], dtype=float32)
# Now let's see how we can recreate this using the weights
# The way convolution is done is to extract a patch
# the size of the kernel (3x3 in this case)
# We will use the first patch, the first three rows and columns and all the colors
patch = starting_image[0,:3,:3,:]
print('patch.shape' , patch.shape)
# Then we flatten the patch
flat_patch = np.reshape( patch, [1,-1] )
print('New shape is', flat_patch.shape)
patch.shape (3, 3, 3)
New shape is (1, 27)
# next we take the weight and reshape it to be [-1,filters]
flat_weight = np.reshape( weight, [-1,2] )
print('flat_weight shape is ',flat_weight.shape)
flat_weight shape is (27, 2)
# we have the patch of shape [1,27] and the weight of [27,2]
# doing a matric multiplication of the two shapes [1,27]*[27,2] = a shape of [1,2]
# which is the output we want, 2 filter outputs for this patch
output_for_patch = np.matmul(flat_patch,flat_weight)
# but we haven't added the bias yet, so lets do that
output_for_patch = output_for_patch + bias
# Finally, we can see that our manual calculation matches
# what Conv2D does exactly for the first patch
output_for_patch
array([[-0.34940773, -0.64269245]], dtype=float32)
If we compare this to the full convolution above, we can see that this is exactly the first patch
array([[[[-0.34940776, -0.6426925 ],
[-0.81834394, -0.16166998]],
[[-0.37515935, -0.28143463],
[-0.60084903, -0.5310158 ]]]], dtype=float32)
We would repeat this process for each patch. If we want to optimize this code some more, instead of passing only one image patch at a time [1,27] we can pass [batch_number,27] patches at a time and the kernel will process them all at once returning [batch_number,filter_size].
I am using ImageDataGenerator().flow_from_directory(...) to generate batches of data from directories.
After the model builds successfully I'd like to get a two column array of True and Predicted class labels. With model.predict_generator(validation_generator, steps=NUM_STEPS) I can get a numpy array of predicted classes. Is it possible to have the predict_generator output the corresponding True class labels?
To add: validation_generator.classes does print the True labels but in the order that they are retrieved from the directory, it doesn't take into account the batching or sample expansion by augmentation.
You can get the prediction labels by:
y_pred = numpy.rint(predictions)
and you can get the true labels by:
y_true = validation_generator.classes
You should set shuffle=False in the validation generator before this.
Finally, you can print confusion matrix by
print confusion_matrix(y_true, y_pred)
There's another, slightly "hackier" way, of retrieving the true labels as well. Note that this approach can handle when setting shuffle=True in your generator (it's generally speaking a good idea to shuffle your data - either if you do this manually where you've stored the data, or through the generator, which is probably easier). You will need your model for this approach though.
# Create lists for storing the predictions and labels
predictions = []
labels = []
# Get the total number of labels in generator
# (i.e. the length of the dataset where the generator generates batches from)
n = len(generator.labels)
# Loop over the generator
for data, label in generator:
# Make predictions on data using the model. Store the results.
predictions.extend(model.predict(data).flatten())
# Store corresponding labels
labels.extend(label)
# We have to break out from the generator when we've processed
# the entire once (otherwise we would end up with duplicates).
if (len(label) < generator.batch_size) and (len(predictions) == n):
break
Your predictions and corresponding labels should now be stored in predictions and labels, respectively.
Lastly, remember that we should NOT add data augmentation on the validation and test sets/generators.
Using np.rint() method will get one hot coding result like [1., 0., 0.] which once I've tried creating a confusion matrix with confusion_matrix(y_true, y_pred) it caused error. Because validation_generator.classes returns class labels as a single number.
In order to get a class number for example 0, 1, 2 as class label specified, I have found the selected answer in this topic useful. here
You should try this to resolve the class probabilities and convert it to a single class based on the score.
if Y_preds.ndim !=1:
Y_preds = np.argmax(Y_preds, axis=1)