I would appreciate any comments on this issue, as I want to deploy the model asap. And I need to know if I can use it or code something else.

Do you have a code snippet to reproduce this issue? I can guarantee you that predict does in fact work, including with TensorFlow.

It appears this bug had nothing to do with either keras or tensorflow, but rather how async events were handled by the webserver I am using.

@Froskekongen could you describe how you fixed this in more detail? I'm having an exactly the same error however in a different program.

It seems to work when I do it manually in a REPL, however when I deploy it as a webservice it breaks.

I also have the same error under the tensorflow backend, however, it works using the theano backend.
@jstypka @Froskekongen Have you found a solution to fix it?

@pxlong it also works on Theano for me, I think it's exactly the same problem. I didn't manage to solve it though, was hoping for some hints from @Froskekongen

same here, same issue! Works fine in REPL, issues running it behind a webservice.

Running the webservice with gunicorn in sync mode solved the issue.

Hey everybody, I'm still not sure what's wrong with this combination.
But I use meinheld instead and it workes even better than gevent.
Hope this help.

Same problem (model.predtict breaking) for me too, but it worked when i switched to theano backend from tensflow.

Same problem here.
Seems to work fine normally. When deployed as a webservice using Flask, get this error.

Works when using Theano as backend, doesn't work with tensorflow.

I had this problem when doing inference in a different thread than where I loaded my model. Here's how I fixed the problem:

Right after loading or constructing your model, save the TensorFlow graph:

graph = tf.get_default_graph()

In the other thread (or perhaps in an asynchronous event handler), do:

global graph
with graph.as_default():
    (... do inference here ...)

I learned about this from https://www.tensorflow.org/versions/r0.11/api_docs/python/framework.html#get_default_graph

Thanks a lot.
it worked for me.

Amazing solution!!
I also encounter this problem.
In my way, I define the model in the model.py.
On the other hand, I create the model instance in main.py
Just do the revision in the main.py!!!

could you give the code snippet

lazy-apps = true

How to use this ?

For me the is only solvable if i load the model inside the flask @app.route POST method.
Which means I reload the model on every request which is very inefficient.
Loading the model as a global either in the beginning of the flask app script or as global in the main(), does not work.

Any ideas on how to solve this?

avital's solution works!

keras with tensorflow backend.

Details:
global thread:

        self.model = load_model(model_path)
        self.model._make_predict_function()
        self.graph = tf.get_default_graph()

another thread:

        with self.graph.as_default():
            labels = self.model.predict(data)

thanks a million!!

I had this problem when doing inference in a different thread than where I loaded my model. Here's how I fixed the problem:
Right after loading or constructing your model, save the TensorFlow graph:

graph = tf.get_default_graph()

In the other thread (or perhaps in an asynchronous event handler), do:

global graph
with graph.as_default():
    (... do inference here ...)

I learned about this from https://www.tensorflow.org/versions/r0.11/api_docs/python/framework.html#get_default_graph

I think I have fixed this bug, ths.

If multithreaded performance is not a necessity, you can also run tensorflow on a single thread

(as in my case , I came across the same issue and none of the mentioned method worked, it would be great if someone could help in figuring out but just in case)

To run tensorflow on a single thread-

global session
session_conf = tf.ConfigProto(
      intra_op_parallelism_threads=1,
      inter_op_parallelism_threads=1)
session = tf.Session(config=session_conf)

# for inference
global session
with session.graph.as_default():
      (...do infernece here ... )

I solved this upgrading to tensorflow2 and implementing a singleton pattern in my model, like so:

class MyModel(object, metaclass=Singleton):
...

class Singleton(type):
    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]

Should work for both keras alone and tensorflow2.

Well. Currently my tf is 2.2.0 and keras is 2.3.1. I add "_make_predict_function()" after loading the model.

However, I get this error "RuntimeError: Attempting to capture an EagerTensor without building a function." and the command "tf.compat.v1.disable_eager_execution()" seems not work and generate new error.

Can anybody help me solve this problem? Thanks a lot!

I am not able to follow avitals instructions. Could someone share an example code of using one of the pretrained models like Resnet50 or VGGnet? My flask app only works on the development server and stop running as soon as I use nginx with uwsgi on production with multi threading. I am currently using this image https://hub.docker.com/r/tiangolo/uwsgi-nginx/ in production and here is my code for making prediction.

I am importing the predict_class function to make predictions on uploaded files from the main.py file

from tensorflow.keras.applications import ResNet50
from tensorflow.keras.applications.resnet50 import preprocess_input as resnet_preprocess_input
from PIL import Image
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.preprocessing.image import load_img
import pickle
import numpy as np
import os

folder_path = os.path.dirname(os.path.abspath(__file__))
model_path = os.path.join(folder_path, 'Logistic_model_resnet.sav')
res_model_path = os.path.join(folder_path, 'resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5')
resnet_base_model = ResNet50(weights=res_model_path, include_top=False, pooling='max')
logistic_model = pickle.load(open(model_path, 'rb'))

def predict_class(img_path):
    img = load_img(img_path, target_size=(224, 224), color_mode='grayscale')
    img_data = img_to_array(img)
    img_data = np.repeat(img_data, repeats = 3, axis = -1)
    img_data = np.expand_dims(img_data, axis=0)
    img_data = resnet_preprocess_input(img_data)
    feature_vector = resnet_base_model.predict(img_data)[0]
    prediction_result = logistic_model.predict([feature_vector])[0]
    return prediction_result