Skip to content

Custom loss function y_true y_pred shape mismatch #4781

New issue
New issue
Closed
Closed
Custom loss function y_true y_pred shape mismatch#4781
@RishabGargeya

Description

@RishabGargeya
RishabGargeya
opened on Dec 21, 2016

Hello,

I am trying to create a custom loss function in Keras, where the target values for my network and the output of my network are of different shapes. Here is the custom loss function I have defined:

def custom_loss(y_true, y_pred):
    sml= T.nnet.sigmoid( - y_pred )
    s1ml= T.nnet.sigmoid( 1.0 -y_pred )
    a = sml
    b = s1ml - sml
    c = 1.0 - s1ml
    p = T.stack((a,b,c), axis=1)
    part1 =  np.log(p + 1.0e-20)
    part2 = y_true * part1
    cost = -(part2).sum()
    return cost

y_pred is of shape (batch_size, 1) and y_true is of shape (batch_size,3), and I aim to calculate a single error value using the above code. However, Keras gives me the following error:

ValueError: Input dimension mis-match. (input[0].shape[1] = 3, input[1].shape[1] = 1)
Apply node that caused the error: Elemwise{Composite{EQ(i0, RoundHalfAwayFromZero(i1))}}(dense_3_target, Elemwise{Add}[(0, 0)].0)
Toposort index: 83
Inputs types: [TensorType(float32, matrix), TensorType(float32, matrix)]
Inputs shapes: [(1001, 3), (1001, 1)]
Inputs strides: [(12, 4), (4, 4)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Sum{acc_dtype=int64}(Elemwise{Composite{EQ(i0, RoundHalfAwayFromZero(i1))}}.0)]]

Does Keras not allow you to have different y_true and y_pred shapes? My cost function requires a singular output of my network and must calculate the cost against a y_true matrix of shape (batch_size,3).

Here is the output of model.summary():

____________________________________________________________________________________________________
Layer (type)                       Output Shape        Param #     Connected to                     
===================================================================================================
convolution2d_1 (Convolution2D)    (None, 30, 1, 591)  1830        convolution2d_input_1[0][0]      
____________________________________________________________________________________________________
maxpooling2d_1 (MaxPooling2D)      (None, 30, 1, 147)  0           convolution2d_1[0][0]            
____________________________________________________________________________________________________
convolution2d_2 (Convolution2D)    (None, 30, 1, 138)  9030        maxpooling2d_1[0][0]             
____________________________________________________________________________________________________
maxpooling2d_2 (MaxPooling2D)      (None, 30, 1, 34)   0           convolution2d_2[0][0]            
____________________________________________________________________________________________________
convolution2d_3 (Convolution2D)    (None, 30, 1, 25)   9030        maxpooling2d_2[0][0]             
____________________________________________________________________________________________________
maxpooling2d_3 (MaxPooling2D)      (None, 30, 1, 6)    0           convolution2d_3[0][0]            
____________________________________________________________________________________________________
flatten_1 (Flatten)                (None, 180)         0           maxpooling2d_3[0][0]             
____________________________________________________________________________________________________
dense_1 (Dense)                    (None, 20)          3620        flatten_1[0][0]                  
____________________________________________________________________________________________________
activation_1 (Activation)          (None, 20)          0           dense_1[0][0]                    
____________________________________________________________________________________________________
dense_2 (Dense)                    (None, 20)          420         activation_1[0][0]               
____________________________________________________________________________________________________
activation_2 (Activation)          (None, 20)          0           dense_2[0][0]                    
____________________________________________________________________________________________________
dense_3 (Dense)                    (None, 1)           21          activation_2[0][0]               
====================================================================================================
Total params: 23951

Thank you for the help!

Activity

RishabGargeya
changed the title [-]Custom objective function shape mismatch[/-] [+]Custom loss function y_true y_pred shape mismatch[/+] on Dec 21, 2016
bstriner

bstriner commented on Dec 24, 2016

@bstriner
bstriner
on Dec 24, 2016
Contributor

Short of hacking into Keras internals, easiest solution is to pad the output to match the shape of the target. Add a lambda layer to pad 0s or use RepeatVector.

Alternatively, add a dummy output that matches the target shape, so there are two outputs. Train with a dummy target and the real target so Keras doesn't complain about the shapes. You will need to directly get the tensors from within the loss function and ignore the ytrue and ypred.

This is a common issue in Keras but you can usually get around it by dummy outputs and targets.

Cheers,
Ben

stale
added
stale
on May 23, 2017
ssfrr

ssfrr commented on May 31, 2017

@ssfrr
ssfrr
on May 31, 2017

Is there any plan to relax this restriction? It seems like when you're writing a custom loss function it's not uncommon that you're doing some complicated comparison, not just seeing how close your model output is to some target.

stale
removed
stale
on May 31, 2017
bstriner

bstriner commented on May 31, 2017

@bstriner
bstriner
on May 31, 2017
Contributor

Not sure if any better solutions out there or any plans. It is easy enough to add custom losses by just adding them to the model. The problem is if this ends up meaning you don't need any targets, there is nothing to pass for the outputs.

@ssfrr let's open a feature request for keras-contrib and continue the discussion there. Need some subclass of Model that supports dummy outputs. It can directly interpret an output tensor as a loss, in which case the corresponding target is not required. Shouldn't be too hard to put together.

If it looks good we can always try to push it back into keras.

Cheers

bstriner

bstriner commented on Jun 1, 2017

@bstriner
bstriner
on Jun 1, 2017
Contributor

After a little more reading, it looks like setting loss weight to None will drop the tensor. Did not know that was a feature.

Something like this might work but haven't tested yet. Set the loss weight to None, then separately add the loss to the model and add the loss as a metric. Then it will still be used as a loss but it will not require a target. There is some skip_indices logic in training that I am reading through.

bstriner

bstriner commented on Jun 1, 2017

@bstriner
bstriner
on Jun 1, 2017
Contributor

Wow! @ssfrr @RishabGargeya so this is a little weird architecturally and I didn't think it would work but try the below code. It trains a model where the inputs are x and y (not one-hot), and the targets are None.

@fchollet do you have any thoughts on how to approach this type of problem? In some situations, like sequence learning, you need your output sequence to also be an Input so you can use it in an RNN, and you don't want the redundancy of it being both an input and a target. I had been using dummy targets, but that still meant I had to pass zeros or something to train, which is kind of awkward. This is also the kind of thing you might do if you don't want to one-hot encode your targets.

I had no idea about how to skip outputs. Maybe need more examples or docs about that feature.

The below approach works for passing your target as an input but it is verbose and you have to add the losses and the metrics in the right order. If there isn't something significantly better, I can abstract it into a custom model.

import keras.backend as K
from keras.callbacks import CSVLogger
from keras.datasets import mnist
from keras.layers import Input, Lambda, Dense, Flatten, BatchNormalization, Activation
from keras.models import Model


def main():
    # Both inputs and targets are `Input` tensors
    input_x = Input((28, 28), name='input_x', dtype='uint8')  # uint8 [0-255]
    y_true = Input((1,), name='y_true', dtype='uint8')  # uint8 [0-9]
    # Build prediction network as usual
    h = Flatten()(input_x)
    h = Lambda(lambda _x: K.cast(_x, 'float32'),
               output_shape=lambda _x: _x,
               name='cast')(h)  # cast uint8 to float32
    h = BatchNormalization()(h)  # normalize pixels
    for i in range(3):  # hidden relu and batchnorm layers
        h = Dense(256)(h)
        h = BatchNormalization()(h)
        h = Activation('relu')(h)
    y_pred = Dense(10, activation='softmax', name='y_pred')(h)  # softmax output layer
    # Lambda layer performs loss calculation (negative log likelihood)
    loss = Lambda(lambda (_yt, _yp): -K.log(_yp[K.reshape(K.arange(K.shape(_yt)[0]), (-1, 1)), _yt] + K.epsilon()),
                  output_shape=lambda (_yt, _yp): _yt,
                  name='loss')([y_true, y_pred])

    # Model `inputs` are both x and y. `outputs` is the loss.
    model = Model(inputs=[input_x, y_true], outputs=[loss])
    # Manually add the loss to the model. Required because the loss_weight will be None.
    model.add_loss(K.sum(loss, axis=None))
    # Compile with the loss weight set to None, so it will be omitted
    model.compile('adam', loss=[None], loss_weights=[None])
    # Add accuracy to the metrics
    # Cannot add as a metric to compile, because metrics for skipped outputs are skipped
    accuracy = K.mean(K.equal(K.argmax(y_pred, axis=1), K.flatten(y_true)))
    model.metrics_names.append('accuracy')
    model.metrics_tensors.append(accuracy)
    # Model summary
    model.summary()

    # Train model
    train, test = mnist.load_data()
    cb = CSVLogger("mnist_training.csv")
    model.fit(list(train), [None], epochs=300, batch_size=64, callbacks=[cb], validation_data=(list(test), [None]))


if __name__ == "__main__":
    main()

Cheers

bstriner

bstriner commented on Jun 1, 2017

@bstriner
bstriner
on Jun 1, 2017
Contributor

For now I think just using dummy targets where your loss is lambda _yt, _yp: _yp is the easiest for anyone who doesn't want to play with internals. Just pass whatever as the target as long as it is the right shape.

waleedka

waleedka commented on Jun 4, 2017

@waleedka
waleedka
on Jun 4, 2017
Contributor

@bstriner Thanks! I've been looking for this as well and this saved me a lot of time.

jhung0
mentioned this on Jul 11, 2017
hgaiser
mentioned this on Jul 21, 2017
stale
added
stale
on Sep 2, 2017
stale

stale commented on Sep 2, 2017

@stale
stalebot
on Sep 2, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

stale
closed this as completedon Oct 2, 2017

7 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @waleedka@ssfrr@RishabGargeya@edmondja@bstriner

        Issue actions
          Custom loss function y_true y_pred shape mismatch · Issue #4781 · keras-team/keras