I use custom loss function in keras. Now, I want to use sample weights in Keras.
I've searched in google and some article suggest model.fit(X,y,sample_weight= custom_weights)
But I want to use sample weight directly in custom loss function. My custom loss function quite complex and for some reason i need to process sample weight directly.
for example:
custom_weights = np.array([1,2,3,4,5,6,7,8,9,10])
#my failed attempt
def custom_loss_function(y_true, y_pred , custom_weights):
return K.mean(K.abs(y_pred - y_true) * custom_weights), axis=-1)
note: my real custom_loss_function is very complex. In this question, I use "MAE" as example to simplify the problem so we can focus to answer "how to use sample weights in custom_loss_function "
how to do this task correctly ?
Related
I have a highly imbalanced data set from which I want to get both classification (binary) as well as probabilities. I have managed to use logistic regression as well as random forest to obtain results from cross_val_predict using class weights.
I am aware that RandomForestClassifier and LogisiticRegression can take class weight as an argument while KNeighborsRegressor and GaussianNB do not. However, for KNN and NB in the documentation it says that for that I can use fit which incorporates sample weights:
fit(self, X, y, sample_weight=None)
So I was thinking of working around it by calculating class weights and using these to create an array of sample weights depending on the classification of the sample. Here is the code for that:
c_w = class_weight.compute_class_weight('balanced', np.unique(y), y)
sw=[]
for i in range(len(y)):
if y[i]==False:
sw.append(c_w[0])
else:
sw.append(c_w[1])
Not sure if this workaround makes sense, however I managed to fit the model using this method and I seem to get better results in terms of my smaller class.
The issue now is that I want to use this method in sklearn's
cross_val_predict()
however I am not managing to pass sample weights through cross validation.
I have 2 questions:
Does my workaround to use sample weights to substitute class weights make sense?
Is there a way to pass sample weights through cross_val_predict just like you would when you use fit without cross validation?
please see the response for this post for the description of sample and class weights difference. Ingeneral if you use class weights, you "make your model aware" of class imbalance. If you use sample weights you make your model aware that some samples must be "considered more carefully" or not taken into account at all.
fit_params argument should do the job, see here:
fit_params : dict, defualt=None - parameters to pass to the fit method of the estimator.
In Keras I often see people compile a model with mean square error function and "acc" as metrics.
model.compile(optimizer=opt, loss='mse', metrics=['acc'])
I have been reading about acc and I can not find an algorithm for it?
What if I would change my loss function to binary crossentropy for an example and use 'acc' as metrics? Would this be the same metrics as in first case or Keras changes this acc based on loss function - so binary crossentropy in this case?
Check the source code from line 375. The metric_fn change dependent on loss function, so it is automatically handled by keras.
If you want to compare models using different loss function it could in some cases be necessary to specify what accuracy method you want to grade your model with, such that the models actually are tested with the same tests.
I am playing around with Keras and try to predict a word from within a context e.g. from a sentence "I have to say the food was tasty!" I hope to get something like this:
[say the ? was tasty] -> food, meals, spaghetti, drinks
However, my problem currently is that the network I am training appears to learn just the probabilities of the single words, and not the probabilities they have in a particular context.
Since the frequency of words is not balanced I thought I might/could/should apply weights to my loss function - which is currently the binary-cross entropy function.
I simply multiply the converse probability of each word with the error:
def weighted_binary_crossentropy(y_true, y_pred):
return K.mean(K.binary_crossentropy(y_pred, y_true) * (1-word_weights), axis=1)
This function is being used by the model as loss function:
model.compile(optimizer='adam', loss=weighted_binary_crossentropy)
However, my results are the exact same and I am not sure if just my model is broken or if I am using the loss paramter/function wrong.
is my weighted_binary_crossentropy() function doing what I just described? I asked because for some reason this works similar:
word_weights), axis=1)
Actually, as one may read in a documentation of a fit function, one may provide sample_weights which seem to be exactly what you want use.
How do I get the sample loss while training instead of the total loss? The loss history is available which gives the total batch loss but it doesn't provide the loss for individual samples.
If possible I would like to have something like this:
on_batch_end(batch, logs, **sample_losses**)
Is something like this available and if not can you provide some hints how to change the code to support this?
To the best of my knowledge it is not possible to get this information via callbacks since the loss is already computed once the callbacks are called (have a look at keras/engine/training.py). To simply inspect the losses you may override the loss function, e.g.:
def myloss(ytrue, ypred):
x = keras.objectives.mean_squared_error(ytrue, ypred)
return theano.printing.Print('loss for each sample')(x)
model.compile(loss=myloss)
Actually this can be done using a callback. This is now included in the keras documentation on callbacks. Define your own callback like this
class LossHistory(keras.callbacks.Callback):
def on_train_begin(self, logs={}):
self.losses = []
def on_batch_end(self, batch, logs={}):
self.losses.append(logs.get('loss'))
And then pass in this callback to your model. You should get per batch losses appended to the history ojbect.
I have also not found any existing functions in the Keras API that can return individual sample losses, while still computing on a minibatch. It seems you have to hack keras, or maybe access the tensorflow graph directly.
set batch size to 1 and use callbacks in model.evaluate OR manually calculate the loss between prediction (model.predict) and ground truth.
How to write a categorization accuracy loss function for keras (deep learning library)?
Categorization accuracy loss is the percentage of predictions that are wrong, i.e. #wrong/#data points.
Is it possible to write a custom loss function for that?
Thanks.
EDIT
Although Keras allows you to use custom loss function, I am not convinced anymore that using accuracy as loss makes sense. First, the network's last layer will typically be soft-max, so that you obtain a vector of class probabilities rather than the single most likely class. Second, I fear that there will be issues with gradient computation due to lack of smoothness of accuracy.
OLD POST
Keras offers you the possibility to use custom loss functions. To get the accuracy loss, you can take inspiration from the examples that are already implemented. For binary classification, I would suggest the following implementation
def mean_accuracy_error(y_true, y_pred):
return K.mean(K.abs(K.sign(y_true - y_pred)), axis=-1)