Skip to content

Can python client be independent of tensorflow ? #271

@drcege

Description

@drcege

In the original version of client https://github.com/tensorflow/serving/blob/e1c16e4859260fab02eab3af5414bf4b75a0b34f/tensorflow_serving/example/inception_client.py, tf was only used to handle command-line arguments. So I could write a python program that doesn't depend on tensoflow.

While, the new client example relies on tf to process image data:
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/example/inception_client.py#49

request.inputs['images'].CopyFrom(
        tf.contrib.util.make_tensor_proto(data, shape=[1]))

And it imports predict_pb2, which also depends on tensorflow.

So how could I write a client that doesn't depend on tensorflow? Any example codes?

Activity

nubbel

nubbel commented on Dec 9, 2016

@nubbel
Contributor

Hi @cegedr!

I built a ruby client for tensorflow/serving just by compiling the protos for ruby (i.e. no custom code).
You can find it here: https://github.com/nubbel/tensorflow_serving_client-ruby#usage

You should be able to do the same for python.

kirilg

kirilg commented on Dec 9, 2016

@kirilg
Contributor

+1 to @nubbel's reply. The only TF Serving code that needs to be linked in is the predict API (proto files). This will require the proto definition of TensorProto as well which is in the TensorFlow repo. Other than that, you don't need any other TF code.

With a more involved change, you can define your own API to the model server (your own proto instead of predict_pb2) that doesn't depend on TF at all, and the client can use that.

drcege

drcege commented on Dec 10, 2016

@drcege
Author

@nubbel @kirilg Thanks! I'm not familiar with google protobuf, but I will try.

sebastian-schlecht

sebastian-schlecht commented on Dec 11, 2016

@sebastian-schlecht

@cegedr I started working on this in Python - my implementation is yet very basic, and the code still uses the tf contrib stuff but in case you come up with a solution to this please let me know as well.

https://github.com/sebastian-schlecht/tensorflow-serving-python

drcege

drcege commented on Dec 11, 2016

@drcege
Author

@sebastian-schlecht If you follow the code of tf.contrib.util.make_tensor_proto, you can see that it actually uses numpy and some proto files (tensor_pb2, tensor_shape_pb2, ... ). So in fact, the client example can be independent of tensorflow, only that it's not so clear as before.

It would be better if the client example can be more concise and independent.

simsicon

simsicon commented on Dec 19, 2016

@simsicon

I would love to see it's independent to request tensorflow models as clear and intuitive as possible.

Bazel, protobuf and grpc are nice, but I think a more light weight serving solution would be nice as well.

Any ideas? Thanks

tobegit3hub

tobegit3hub commented on Jan 10, 2017

@tobegit3hub
Contributor

We have implemented the dependent Python library for this in deep_recommend_system.

I'm not sure if this is contributions welcome. We may contribute the clients for TensorFlow Serving in different programming languages in this repository. Providing batteries-included clients is really helpful for our TensorFlow users.

We may submit the PR for this and hope it would help 😃

tobegit3hub

tobegit3hub commented on Jan 10, 2017

@tobegit3hub
Contributor

Hi @cegedr , I have submit the PR about how to compile the independent python client in #290 .

You can follow the instructions to include the generated python files without bazel. This library have been used in our production environment and may need your requirements.

drcege

drcege commented on Jan 10, 2017

@drcege
Author

@tobegit3hub
Your implementation still used tensorflow? What I actually mean is no import tensorflow as tf and no tf.contrib.util.make_tensor_proto. In some cases, we do not want to install the entire tensorflow library.

tobegit3hub

tobegit3hub commented on Jan 11, 2017

@tobegit3hub
Contributor

The only place to use tensorflow or tf is calling tf.contrib.util.make_tensor_proto. You can definitely construct TensorProto without this function, but you need to compile most of the proto files in TensorFlow repository.

Actually, we have to construct TensorProto by ourselves in Java. You can refer to this example code in InceptionPredictClient.java.

We can also generate all the proto files with protoc or grpcio. Anyway, I don't think it's necessary for Python.

CrowbarKZ

CrowbarKZ commented on Apr 5, 2017

@CrowbarKZ

+1 to @simsicon 's idea of a lightweight client which could be used without tf dependencies. Would be great to have such an option

stianlp

stianlp commented on Jul 4, 2017

@stianlp

Trying to write a lightweight client myself. Cloned this repo https://www.npmjs.com/package/tf-serving-nodejs-client and created the *_pb2_grpc.py and *_pb2.py files with
python -m grpc.tools.protoc -I./protos --python_out=. --grpc_python_out=. protos/*

I am getting an error after using tf.contrib.util.make_tensor_proto and calling MergeFrom with the return value as parameter.

Here's the stack trace:

Traceback (most recent call last): File "client.py", line 87, in <module> tf.app.run() File "/Users/slp/anaconda/envs/ml/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "client.py", line 83, in main FLAGS.concurrency, FLAGS.num_tests) File "client.py", line 68, in do_inference request.inputs['images'].CopyFrom(tp) File "/Users/slp/anaconda/envs/ml/lib/python3.5/site-packages/google/protobuf/message.py", line 118, in CopyFrom self.MergeFrom(other_msg) File "/Users/slp/anaconda/envs/ml/lib/python3.5/site-packages/google/protobuf/internal/python_message.py", line 1209, in MergeFrom 'expected %s got %s.' % (cls.__name__, msg.__class__.__name__)) TypeError: Parameter to MergeFrom() must be instance of same class: expected TensorProto got TensorProto.

The problem is that MergeFrom expects <class 'tensorflow.core.framework.tensor_pb2.TensorProto'>, but gets the TensorProto I generated which is <class 'tensor_pb2.TensorProto'>

I have checked my .proto files, they don't differ from those used in tfserving and tensorflow. So the only issue seems to be that tf.contrib.util.make_tensor_proto returns a <class 'tensorflow.core.framework.tensor_pb2.TensorProto'> which is not the same type as <class 'tensor_pb2.TensorProto'>.

@tobegit3hub Haven't tried your implementation yet, but looks like your repo has a lot of tensorflow stuff in it. It's getting late here, I checking out your implementation tomorrow. Would be cool if we could implement a client without tf.

tobegit3hub

tobegit3hub commented on Jul 5, 2017

@tobegit3hub
Contributor

Hi @stianlp , you can checkout the python_predict_client. There are generated gRPC files which are not dependent on TensorFlow.

Actually we use tf.contrib.util.make_tensor_proto to construct TensorProto objects easily. You can checkout its source code if you don't want to use it directly.

stianlp

stianlp commented on Jul 6, 2017

@stianlp

Tested it last night, works like a charm :) Thank you 🥇

stianlp

stianlp commented on Jan 28, 2018

@stianlp

I just wanted to check in here and tell you that I wrote a client that is not dependent on TensorFlow.

I also wrote up two blog posts on medium, one on how to save a model for tfserving and one about the client.

https://medium.com/@stianlindpetlund/tensorflow-serving-101-pt-1-a79726f7c103
https://medium.com/@stianlindpetlund/tensorflow-serving-101-pt-2-682eaf7469e7

Feel free to use the client :)

nuchi

nuchi commented on Mar 25, 2018

@nuchi

Chiming in for anyone else following: I wrote a tool to generate a package of tf and tf-serving python protobuf interfaces. https://github.com/nuchi/tf_pb_without_tf

It generates a wheel which contains things like TensorProto, PredictionServiceStub, etc. It downloads TF and TF Serving to generate the wheel, but then you can install the result anywhere and don't need TF or TF serving installed.

gautamvasudevan

gautamvasudevan commented on Jun 25, 2018

@gautamvasudevan
Collaborator

TF Serving now also supports a REST API, for which it's fairly easy to build a client around given the plethora of libraries supporting REST APIs across various languages.

alexeckert

alexeckert commented on Jul 26, 2021

@alexeckert

Hi, I wouldn't say that the REST api is a good subsitute as grpc with tensors often offers performance characteristcs. It would be very convenient if tf.contrib.util.make_tensor_proto could live in its own package to avoid tf and all its dependencies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @nubbel@kirilg@simsicon@stianlp@tobegit3hub

        Issue actions

          Can python client be independent of tensorflow ? · Issue #271 · tensorflow/serving