You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
TensorframeOnSpark support the real distribute trainning on tensorflow . Maybe it's a good idea to port TensorframeOnSpark to spark deep learning project.
The text was updated successfully, but these errors were encountered:
Thanks, @allwefantasy, that makes a lot of sense. In fact, we've been collaborating with the TensorFlowOnSpark (TFoS) team to see how we can bring all these ideas together...! They have implemented a Spark MLlib Estimator in the TFoS project (https://github.com/yahoo/TensorFlowOnSpark/blob/master/tensorflowonspark/pipeline.py#L214) which is very exciting. The two projects are quite complementary currently, and it could make sense to combine them together. What are your thoughts?
Excellent, that's really what I want. Also TFEstimator implemented by TFoS proves it is easy to be a distributed runtime for spark-deep-learning. That means we can just treat TFoS as pip dependency.
I check the last PR in spark-deep-learning is KerasImageFileEstimator, we can add a new param eg. distribute_mode to make it run in TFoS.
spark-deep-learning depends on tensorframes which makes spark can use tf's power of numerical calculation and connects tf / spark strongly, and we can provide users with a perfect experience like MLlib. we don't encourage people to write complex TFcode . People should just define their network structure or tell us their model file and input/output,then everything should been done by spark-deep-learning.
TensorframeOnSpark support the real distribute trainning on tensorflow . Maybe it's a good idea to port TensorframeOnSpark to spark deep learning project.
The text was updated successfully, but these errors were encountered: