I would appreciate any comments on this issue, as I want to deploy the model asap. And I need to know if I can use it or code something else.

Do you have a code snippet to reproduce this issue? I can guarantee you that predict does in fact work, including with TensorFlow.

It appears this bug had nothing to do with either keras or tensorflow, but rather how async events were handled by the webserver I am using.

@Froskekongen could you describe how you fixed this in more detail? I'm having an exactly the same error however in a different program.

It seems to work when I do it manually in a REPL, however when I deploy it as a webservice it breaks.

I also have the same error under the tensorflow backend, however, it works using the theano backend.
@jstypka @Froskekongen Have you found a solution to fix it?

@pxlong it also works on Theano for me, I think it's exactly the same problem. I didn't manage to solve it though, was hoping for some hints from @Froskekongen

same here, same issue! Works fine in REPL, issues running it behind a webservice.

Running the webservice with gunicorn in sync mode solved the issue.

Hey everybody, I'm still not sure what's wrong with this combination.
But I use meinheld instead and it workes even better than gevent.
Hope this help.

Same problem (model.predtict breaking) for me too, but it worked when i switched to theano backend from tensflow.

Same problem here.
Seems to work fine normally. When deployed as a webservice using Flask, get this error.

Works when using Theano as backend, doesn't work with tensorflow.

I had this problem when doing inference in a different thread than where I loaded my model. Here's how I fixed the problem:

Right after loading or constructing your model, save the TensorFlow graph:

graph = tf.get_default_graph()

In the other thread (or perhaps in an asynchronous event handler), do:

global graph
with graph.as_default():
    (... do inference here ...)

I learned about this from https://www.tensorflow.org/versions/r0.11/api_docs/python/framework.html#get_default_graph

Thanks a lot.
it worked for me.

Thanks so much! @avital
Works like a charm!

Thanks a lot! @avital It worked.
env: keras with tensorflow on flask

Thank you @avital, that did the trick! This issue really should not be closed. This should be fixed by the Keras library.

You are savior! Thanks a lot. @avital

Hi all, I followed @avital codes but got 'AttributeError: exit' after the with statement, does anyone know how to deal with this? Thanks!!

Thanks a million, it works!!!!@avital

@avital
Thanks, worked for me!

@avital
Thanks! It works well with tensorflow, but my model was built on keras. How to fix this problem on keras?

Worked like a charm!! :) @avital

the same problem and i solve it. thanks @avital
my solution is https://justttry.github.io/justttry.github.io/not-an-element-of-Tensor-graph/

Amazing solution!!
I also encounter this problem.
In my way, I define the model in the model.py.
On the other hand, I create the model instance in main.py
Just do the revision in the main.py!!!

avital's solution works!

keras with tensorflow backend.

Details:
global thread:

        self.model = load_model(model_path)
        self.model._make_predict_function()
        self.graph = tf.get_default_graph()

another thread:

        with self.graph.as_default():
            labels = self.model.predict(data)

@shaoeChen but gunicorn also does the same thing right? it loads the model for each process.Anyways good for you.

@SiddhardhaSaran hi dear.
i know what you mean.
but in my case, on uwsgi even i setting one process one thread it still no work, so i use gunicorn.
anyway thanks your good advice.

@jglwiz
Thank you , it solved my problem

I faced the same issue recently when deploying the model as a webservice using django. I ended up creating a singleton class that would have the model and the tf.graph() i.e it would be instatiated only once. it solved the problem.

hi @shaoeChen , you might want to try my approach: #2397 (comment)

It both save session and graph

Not sure if it's relevant to the original question, but maybe it'll be useful to others:

Based on this answer, the following resolved tf's multithreading compatibility for me:

# on thread 1
session = tf.Session(graph=tf.Graph())
with session.graph.as_default():
    k.backend.set_session(session)
    model = k.models.load_model(filepath)

# on thread 2
with session.graph.as_default():
    k.backend.set_session(session)
    model.predict(x, **kwargs)

The novelty here is keeping both the Session and the Graph for other threads.
The model is loaded in their "context" (instead of the default ones) and kept for other threads to use.
(By default the model is loaded to the default Session and the default Graph)
Another plus is that they're kept in the same object - easier to handle.

@eliadl @emesha92 how about multi-model prediction and multiprocessing under the one main process? would you mind giving me some advice.
Looking forward to any replies.
Thx

@Ai-is-light by "multiprocessing under one main process" what exactly do you mean?

Amazing solution!!
I also encounter this problem.
In my way, I define the model in the model.py.
On the other hand, I create the model instance in main.py
Just do the revision in the main.py!!!

could you give the code snippet

lazy-apps = true

How to use this ?

For me the is only solvable if i load the model inside the flask @app.route POST method.
Which means I reload the model on every request which is very inefficient.
Loading the model as a global either in the beginning of the flask app script or as global in the main(), does not work.

Any ideas on how to solve this?

avital's solution works!

keras with tensorflow backend.

Details:
global thread:

        self.model = load_model(model_path)
        self.model._make_predict_function()
        self.graph = tf.get_default_graph()

another thread:

        with self.graph.as_default():
            labels = self.model.predict(data)

thanks a million!!

I had this problem when doing inference in a different thread than where I loaded my model. Here's how I fixed the problem:
Right after loading or constructing your model, save the TensorFlow graph:

graph = tf.get_default_graph()

In the other thread (or perhaps in an asynchronous event handler), do:

global graph
with graph.as_default():
    (... do inference here ...)

I learned about this from https://www.tensorflow.org/versions/r0.11/api_docs/python/framework.html#get_default_graph

I think I have fixed this bug, ths.

If multithreaded performance is not a necessity, you can also run tensorflow on a single thread

(as in my case , I came across the same issue and none of the mentioned method worked, it would be great if someone could help in figuring out but just in case)

To run tensorflow on a single thread-

global session
session_conf = tf.ConfigProto(
      intra_op_parallelism_threads=1,
      inter_op_parallelism_threads=1)
session = tf.Session(config=session_conf)

# for inference
global session
with session.graph.as_default():
      (...do infernece here ... )

I solved this upgrading to tensorflow2 and implementing a singleton pattern in my model, like so:

class MyModel(object, metaclass=Singleton):
...

class Singleton(type):
    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]

Should work for both keras alone and tensorflow2.

Well. Currently my tf is 2.2.0 and keras is 2.3.1. I add "_make_predict_function()" after loading the model.

However, I get this error "RuntimeError: Attempting to capture an EagerTensor without building a function." and the command "tf.compat.v1.disable_eager_execution()" seems not work and generate new error.

Can anybody help me solve this problem? Thanks a lot!

I am not able to follow avitals instructions. Could someone share an example code of using one of the pretrained models like Resnet50 or VGGnet? My flask app only works on the development server and stop running as soon as I use nginx with uwsgi on production with multi threading. I am currently using this image https://hub.docker.com/r/tiangolo/uwsgi-nginx/ in production and here is my code for making prediction.

I am importing the predict_class function to make predictions on uploaded files from the main.py file

from tensorflow.keras.applications import ResNet50
from tensorflow.keras.applications.resnet50 import preprocess_input as resnet_preprocess_input
from PIL import Image
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.preprocessing.image import load_img
import pickle
import numpy as np
import os

folder_path = os.path.dirname(os.path.abspath(__file__))
model_path = os.path.join(folder_path, 'Logistic_model_resnet.sav')
res_model_path = os.path.join(folder_path, 'resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5')
resnet_base_model = ResNet50(weights=res_model_path, include_top=False, pooling='max')
logistic_model = pickle.load(open(model_path, 'rb'))

def predict_class(img_path):
    img = load_img(img_path, target_size=(224, 224), color_mode='grayscale')
    img_data = img_to_array(img)
    img_data = np.repeat(img_data, repeats = 3, axis = -1)
    img_data = np.expand_dims(img_data, axis=0)
    img_data = resnet_preprocess_input(img_data)
    feature_vector = resnet_base_model.predict(img_data)[0]
    prediction_result = logistic_model.predict([feature_vector])[0]
    return prediction_result