Add CNTK as keras backend #6800

souptc · 2017-05-30T18:14:16Z

This is the Beta version of CNTK as keras backend. Those changes are suppose to work with CNTK v2.0 GA. Because the CNTKv2.0 is not officially published, you could use the wheel below to have a first try:

http://cntk.ai/PythonWheel/ForKeras/cntk-2.0rc3-cp35-cp35m-linux_x86_64.whl
http://cntk.ai/PythonWheel/ForKeras/cntk-2.0rc3-cp27-cp27mu-linux_x86_64.whl

Most of the keras features are supported, except:

Performance optimization on CPU device.
CNTK performance optimization on cpu device is not finished, so if you want better performance, please try to run CNTK_Keras with GPU device.
Gradient as symbolic ops.
This is not include in CNTK_Keras beta release, and we will support it in CNTK_Keras GA release.
Stateful recurrent layer.
This is not include in CNTK_Keras beta release.
Masking on recurrent layer.
This is not include in CNTK_Keras beta release, and we will support it in CNTK_Keras GA release.
Padding with non-specified shape.
Padding with non-speicfied shape is not supported now. To using cntk backend in keras with padding, please specified the concrete input shape.
Convolution with dilation.
This is not include in CNTK_Keras beta release, and we will support it in CNTK_Keras GA release.
Randomness op across batch axis.
This is not include in CNTK_Keras beta release, and we will support it in CNTK_Keras GA release.
Some Keras backend apis: reverse / top_k / ctc / map / foldl / foldr.
These are not include in CNTK_Keras beta release, and we will support it in CNTK_Keras GA release.

fix cross entropy break; add more ut fix pep8 issue

support inferred dim in rnn and fix pep8

recover pytest

…eras

fix cross entropy break; add more ut fix pep8 issue

souptc · 2017-06-03T03:47:15Z

@fchollet , I just update the implementation which resolve the most comments. For the "bias_shape", I explained the purpose of the change, do you have any suggestion about a better solution?

souptc · 2017-06-03T03:52:13Z

And for the maintainance, yes, CNTK team will definitely commit to maintain the cntk backend in the future.

* Add top_k_sparse_categorical_accuracy and test_top_k_sparse_categorical_accuracy * Rename top_k_sparse_categorical_accuracy and sparse_top_k_categorical_accuracy

fchollet

The file cntk_backend will require some cleanups and a style normalization. E.g. it mixes different quote characters for string delimitation, and other small style issues.

fchollet · 2017-06-05T21:51:47Z

LICENSE

@@ -8,6 +8,10 @@ All contributions by Google:
 Copyright (c) 2015, Google, Inc.
 All rights reserved.

+All contributions by Microsoft:
+Copyright (c) 2015, Microsoft, Inc.


This should say "2017", I believe

thanks, fixed.

fchollet · 2017-06-05T22:15:25Z

keras/backend/tensorflow_backend.py

@@ -3334,13 +3334,17 @@ def pool3d(x, pool_size, strides=(1, 1, 1), padding='valid',
    return _postprocess_conv3d_output(x, data_format)


-def bias_add(x, bias, data_format=None):
+def bias_add(x, bias, data_format=None, bias_shape=None):


Wouldn't the bias argument already have a shape, that you could just read?

fix theano issue fix bias shape issue in thano

souptc · 2017-06-06T07:22:08Z

clean the cntk_backend file with style issue.

souptc · 2017-06-06T07:22:50Z

fix the bias_shape issue.

fchollet

Some style nitpicks. Almost ready to merge!

fchollet · 2017-06-06T18:56:07Z

keras/backend/cntk_backend.py

+NAME_SCOPE_STACK = []
+
+
+@contextmanager


fchollet · 2017-06-06T18:58:01Z

keras/backend/tensorflow_backend.py

@@ -3346,28 +3346,41 @@ def bias_add(x, bias, data_format=None):
        Output tensor.

    # Raises
-        ValueError: In case of invalid `data_format` argument.
+        ValueError: In case of invalid `data_format` argument, or the input bias dimension is not expect


Can you rephrase this? The meaning is unclear. Additionally, please make sure to introduce line breaks to keep line length reasonable.

fchollet · 2017-06-06T18:58:51Z

keras/backend/tensorflow_backend.py

    if ndim(x) == 5:
        if data_format == 'channels_first':
-            x += reshape(bias, (1, int_shape(bias)[0], 1, 1, 1))
+            shape = (bias_shape[0], 1, 1, 1) if len(bias_shape) == 1 else (bias_shape[3],) + bias_shape[:3]


Introduce an if block to avoid a very long line.

fchollet · 2017-06-06T18:58:56Z

keras/backend/tensorflow_backend.py

        elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, 1, 1, int_shape(bias)[0]))
+            shape = (1, 1, 1, bias_shape[0]) if len(bias_shape) == 1 else bias_shape


fchollet · 2017-06-06T18:59:00Z

keras/backend/tensorflow_backend.py

    elif ndim(x) == 4:
        if data_format == 'channels_first':
-            x += reshape(bias, (1, int_shape(bias)[0], 1, 1))
+            shape = (bias_shape[0], 1, 1) if len(bias_shape) == 1 else (bias_shape[2],) + bias_shape[:2]


fchollet · 2017-06-06T19:01:26Z

keras/backend/theano_backend.py

    elif ndim(x) == 4:
        if data_format == 'channels_first':
-            x += reshape(bias, (1, bias.shape[0], 1, 1))
+            shape = (bias_shape[0], 1, 1) if ndim(bias) == 1 else (bias_shape[2],) + bias_shape[:2]


Line too long

fchollet · 2017-06-06T19:01:29Z

keras/backend/theano_backend.py

        elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, 1, bias.shape[0]))
+            shape = (1, 1, bias_shape[0]) if ndim(bias) == 1 else bias_shape


Line too long

fchollet · 2017-06-06T19:01:33Z

keras/backend/theano_backend.py

    elif ndim(x) == 3:
        if data_format == 'channels_first':
-            x += reshape(bias, (1, bias.shape[0], 1))
+            shape = (bias_shape[0], 1) if ndim(bias) == 1 else (bias_shape[1],) + bias_shape[:1]


Line too long

fchollet · 2017-06-06T19:01:36Z

keras/backend/theano_backend.py

        elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, bias.shape[0]))
+            shape = (1, bias_shape[0]) if ndim(bias) == 1 else bias_shape


Line too long

fchollet · 2017-06-06T19:01:47Z

keras/layers/local.py

-            output = K.reshape(output,
-                               (self.output_row, self.output_col, -1, filters))
-            output = K.permute_dimensions(output, (2, 0, 1, 3))
+        output = K.local_conv2d(inputs, self.kernel, self.kernel_size, self.strides, (self.output_row, self.output_col), self.data_format)


Line too long

fchollet · 2017-06-07T03:50:08Z

The failing test is a flake.

fchollet · 2017-06-07T05:01:48Z

I have applied a few style fixes to the backend source and merged. Thanks a lot! 👍

While going through the backend code, I have noticed that a lot of the error messages raised were not as helpful and as clear as they should be. I suggest you go over the error messages and improve them. Typically you want to tell the user: 1) what they did (e.g. print the arguments they passed), 2) why that was wrong (e.g. something not supported), and 3) what they should do instead. Currently you are mostly doing 2) only.

This will greatly improve the user experience for CNTK Keras users, and improve the usability / ease of debugging of Keras models on CNTK.

souptc · 2017-06-07T05:57:25Z

Thanks François! Sure, I will go though the message tomorrow and improve them.

ebarsoumMS · 2017-06-07T06:13:35Z

Thanks, everybody...

fchollet · 2017-06-07T06:15:44Z

Huge thanks to everyone who contributed to this project! It's a big milestone.

chentaMS and others added 30 commits May 16, 2017 14:39

merge CNTK support

f2dc253

fix merge bugs

95d8e19

fix cross entropy break; add more ut fix pep8 issue

fix rebase error

703be00

fix merge error

fdba0ea

fix unit test failure

5e6ca8b

fix comments in CR

3eee73f

move recurrent modification to cntk wrapper

4354069

fix recurrent issue and cntk identical issue

85db861

reshape batch

5ce4a0a

move out special handle in recurrent layer code

a07b9c5

support inferred dim in rnn and fix pep8

add flattern; remove useless code

93a2b1e

remove useless code

f1c1a3c

recover pytest

fix merge error

2b1302e

use broadcast_as; and fix perf issue

c887817

Updated backend.md to include CNTK as available backend

a3784ab

update according to cnkt latest master intferface

f7f90da

Updated index.md to remove toolkit names from title

48eb428

Updated README.md to include CNTK

04bd2e5

Updated README.md installation instructions for CNTK

8b5fbe2

Updated faq.md for CNTK

f5e7689

fix channel_first issue

5550250

fix recurrent layer issue

db740ba

batch learner

0137f87

Merge remote-tracking branch 'origin/CNTKDocs' into chenta/keras-ci

bdee5c9

add docs for cpu warning and examples

3b80376

Merge remote-tracking branch 'cntk_keras/chenta/keras-ci' into cntk_k…

db1a340

…eras

merge CNTK support

1416e1b

fix merge bugs

bb44aa4

fix cross entropy break; add more ut fix pep8 issue

fix rebase error

cd8b70e

fix merge error

36ea1fb

Merge branch 'fix_issues'

94d1616

chentaMS and others added 5 commits June 4, 2017 17:52

fix the equal issue

e16848a

add the list type check

48c83e0

Add sparse_top_k_categorical_accuracy and test code (#6840)

3f84890

* Add top_k_sparse_categorical_accuracy and test_top_k_sparse_categorical_accuracy * Rename top_k_sparse_categorical_accuracy and sparse_top_k_categorical_accuracy

skip sparse top k test for cntk since cntk not support it now

4349d29

Merge branch 'master' into master

8a93935

fchollet reviewed Jun 5, 2017

View reviewed changes

chentaMS added 4 commits June 5, 2017 22:47

avoid extra agument bias_shape

c93e8c1

fix theano issue fix bias shape issue in thano

fix style issue

461c43c

update the ut of bias_add

37db64d

update the year

59bda65

fchollet reviewed Jun 6, 2017

View reviewed changes

chentaMS added 6 commits June 6, 2017 14:33

fix too long lines and doc strings

d69553e

fix line endings

8884013

Merge branch 'test_cr' into fix_style

1e1ebbd

Merge remote-tracking branch 'keras-master/master'

7fb38b1

skip stateful recurrent test for cntk

1eca798

fix reshape issue with inferred dimension

82832a3

fchollet merged commit 82832a3 into keras-team:master Jun 7, 2017

minimaxir mentioned this pull request Jun 7, 2017

1bit-SGD + Keras microsoft/CNTK#1975

Closed

gabrieldemarmiesse mentioned this pull request Dec 28, 2017

backend cntk: TypeError: '<' not supported between instances of 'Function' and 'float' #8821

Closed

		NAME_SCOPE_STACK = []


		@contextmanager

Add CNTK as keras backend #6800

Add CNTK as keras backend #6800

Conversation

souptc commented May 30, 2017

Uh oh!

souptc commented Jun 3, 2017

Uh oh!

souptc commented Jun 3, 2017

Uh oh!

fchollet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

souptc commented Jun 6, 2017

Uh oh!

souptc commented Jun 6, 2017

Uh oh!

fchollet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fchollet commented Jun 7, 2017

Uh oh!

fchollet commented Jun 7, 2017

Uh oh!

souptc commented Jun 7, 2017

Uh oh!

ebarsoumMS commented Jun 7, 2017

Uh oh!

fchollet commented Jun 7, 2017

Uh oh!