New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Custom loss function y_true y_pred shape mismatch #4781
Comments
Short of hacking into Keras internals, easiest solution is to pad the output to match the shape of the target. Add a lambda layer to pad 0s or use RepeatVector. Alternatively, add a dummy output that matches the target shape, so there are two outputs. Train with a dummy target and the real target so Keras doesn't complain about the shapes. You will need to directly get the tensors from within the loss function and ignore the ytrue and ypred. This is a common issue in Keras but you can usually get around it by dummy outputs and targets. Cheers, |
Is there any plan to relax this restriction? It seems like when you're writing a custom loss function it's not uncommon that you're doing some complicated comparison, not just seeing how close your model output is to some target. |
Not sure if any better solutions out there or any plans. It is easy enough to add custom losses by just adding them to the model. The problem is if this ends up meaning you don't need any targets, there is nothing to pass for the outputs. @ssfrr let's open a feature request for keras-contrib and continue the discussion there. Need some subclass of Model that supports dummy outputs. It can directly interpret an output tensor as a loss, in which case the corresponding target is not required. Shouldn't be too hard to put together. If it looks good we can always try to push it back into keras. Cheers |
After a little more reading, it looks like setting loss weight to None will drop the tensor. Did not know that was a feature. Something like this might work but haven't tested yet. Set the loss weight to None, then separately add the loss to the model and add the loss as a metric. Then it will still be used as a loss but it will not require a target. There is some |
Wow! @ssfrr @RishabGargeya so this is a little weird architecturally and I didn't think it would work but try the below code. It trains a model where the @fchollet do you have any thoughts on how to approach this type of problem? In some situations, like sequence learning, you need your output sequence to also be an I had no idea about how to skip outputs. Maybe need more examples or docs about that feature. The below approach works for passing your target as an input but it is verbose and you have to add the losses and the metrics in the right order. If there isn't something significantly better, I can abstract it into a custom model. import keras.backend as K
from keras.callbacks import CSVLogger
from keras.datasets import mnist
from keras.layers import Input, Lambda, Dense, Flatten, BatchNormalization, Activation
from keras.models import Model
def main():
# Both inputs and targets are `Input` tensors
input_x = Input((28, 28), name='input_x', dtype='uint8') # uint8 [0-255]
y_true = Input((1,), name='y_true', dtype='uint8') # uint8 [0-9]
# Build prediction network as usual
h = Flatten()(input_x)
h = Lambda(lambda _x: K.cast(_x, 'float32'),
output_shape=lambda _x: _x,
name='cast')(h) # cast uint8 to float32
h = BatchNormalization()(h) # normalize pixels
for i in range(3): # hidden relu and batchnorm layers
h = Dense(256)(h)
h = BatchNormalization()(h)
h = Activation('relu')(h)
y_pred = Dense(10, activation='softmax', name='y_pred')(h) # softmax output layer
# Lambda layer performs loss calculation (negative log likelihood)
loss = Lambda(lambda (_yt, _yp): -K.log(_yp[K.reshape(K.arange(K.shape(_yt)[0]), (-1, 1)), _yt] + K.epsilon()),
output_shape=lambda (_yt, _yp): _yt,
name='loss')([y_true, y_pred])
# Model `inputs` are both x and y. `outputs` is the loss.
model = Model(inputs=[input_x, y_true], outputs=[loss])
# Manually add the loss to the model. Required because the loss_weight will be None.
model.add_loss(K.sum(loss, axis=None))
# Compile with the loss weight set to None, so it will be omitted
model.compile('adam', loss=[None], loss_weights=[None])
# Add accuracy to the metrics
# Cannot add as a metric to compile, because metrics for skipped outputs are skipped
accuracy = K.mean(K.equal(K.argmax(y_pred, axis=1), K.flatten(y_true)))
model.metrics_names.append('accuracy')
model.metrics_tensors.append(accuracy)
# Model summary
model.summary()
# Train model
train, test = mnist.load_data()
cb = CSVLogger("mnist_training.csv")
model.fit(list(train), [None], epochs=300, batch_size=64, callbacks=[cb], validation_data=(list(test), [None]))
if __name__ == "__main__":
main() Cheers |
For now I think just using dummy targets where your loss is |
@bstriner Thanks! I've been looking for this as well and this saved me a lot of time. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed. |
Is there any chance this will get supported in a more natural way? This is quite a hack |
I am trying to use a custom loss function that gets two tensor of different shapes and returns a single value. When compiling the model, I tell keras to use the identity function as the loss function. The actual loss function is inside the model, which has two inputs: one for the data and one for the labels. It seems to work fine, but the model does not converge properly. I am guessing it has something to do with the model outputting a single scalar and keras somehow trying to match that to my dummy label vector that has 'batch_size' many entries. When I use tensorflow to train my model directly, everything works and converges just fine. There must be something happening behind the scenes that messes with the gradient and I can't figure out a solution. |
@bstriner, Any updates on this issue? Another use case is quantile regression when predicting multiple quantiles from one y_true. |
@bstriner Taking advantage of your idea of the above algorithm, I would like to know if it is possible to have one classifier inside the other, for example, use the last layer to obtain an SVM. I tried to do this using a custom cost function, but unfortunately I always get an error with the tensor format. Can you help me? I present the algorithm below. # Classification block
x = GlobalAveragePooling2D()(x)
x = Dense(4096, kernel_regularizer=l2(1e-4), name='Dense_1')(x)
x = Activation('relu', name='relu1')(x)
x = Dropout(DROPOUT)(x)
x = Dense(4096, kernel_regularizer=l2(1e-4), name='Dense_2')(x)
x = Activation('relu', name='relu2')(x)
model_output = Dropout(DROPOUT)(x)
model = Model(model_input, model_output)
model.summary()
import tensorflow as tf
from keras import backend as K
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from keras.losses import categorical_hinge
def custom_loss_value(y_true, y_pred):
X = K.eval(y_pred)
print(X)
Y = np.ravel(K.eval(y_true))
Predict = []
Prob = []
scaler = StandardScaler()
X = scaler.fit_transform(X)
param_grid = {'C': [0.1, 1, 8, 10], 'gamma': [0.001, 0.01, 0.1, 1]}
SVM = GridSearchCV(SVC(kernel='rbf',probability=True), cv=3, param_grid=param_grid, scoring='auc', verbose=1)
SVM.fit(X, Y)
Final_Model = SVM.best_estimator_
Predict = Final_Model.predict(X)
Prob = Final_Model.predict_proba(X)
return categorical_hinge(tf.convert_to_tensor(Y, dtype=tf.float32), tf.convert_to_tensor(Predict, dtype=tf.float32))
sgd = SGD(lr=0.001, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss=custom_loss_value, optimizer=sgd, metrics=['accuracy']) |
Seems there is an even better solution to this : https://towardsdatascience.com/advanced-keras-constructing-complex-custom-losses-and-metrics-c07ca130a618 |
Hello,
I am trying to create a custom loss function in Keras, where the target values for my network and the output of my network are of different shapes. Here is the custom loss function I have defined:
y_pred is of shape (batch_size, 1) and y_true is of shape (batch_size,3), and I aim to calculate a single error value using the above code. However, Keras gives me the following error:
Does Keras not allow you to have different y_true and y_pred shapes? My cost function requires a singular output of my network and must calculate the cost against a y_true matrix of shape (batch_size,3).
Here is the output of model.summary():
Thank you for the help!
The text was updated successfully, but these errors were encountered: