Description
Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.
Thank you!
Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/fchollet/keras.git --upgrade --no-depsIf running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.
This is a short and simple issue. Following upgrading to Keras 2.0.9 I have been using the multi_gpu_model utility but I can seem to save my models or best weights using model.save.
The error I get is
TypeError: can’t pickle module objects
I suspect there is some problem gaining access to the model object. Is there a work around this issue?
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Participants
Activity
datumbox commentedon Nov 10, 2017
Are you trying to save directly the model which is parallelized? That can be the problem. Could you provide a snippet that shows the steps you are following?
SimonWalsh1000 commentedon Nov 10, 2017
Yes - thats exactly what I am doing - I'm building the model normally, then calling the multi gpu model method function on it and after training, trying to save it with model.save. Does this need to be converted back to a non-parallelized version of the model? If so, how is this achieved?
fchollet commentedon Nov 10, 2017
For now we recommend saving the original (template) model instead of the parallel model. I.e. call
saveon the model you passed tomulti_gpu_model, not the model returned by it.Both models share the same weights.
SimonWalsh1000 commentedon Nov 10, 2017
SimonWalsh1000 commentedon Nov 10, 2017
Just to clarify - do you mean call model.save or model.save_weights on the template model at the end of training?
fchollet commentedon Nov 10, 2017
Yes. Either method should work fine.
DingkunLiu commentedon Jan 9, 2018
But how can I save the optimizer state by just saving the template model?
s-zk commentedon Jan 25, 2018
I am on keras 2.1.2 and encountered same problem. I tried follow this answer on StackOverflow, and it works for me. Hope it helps.
wt-huang commentedon Nov 13, 2018
Closing as this is resolved
taalbrecht commentedon Nov 28, 2018
Has this been resolved by a commit and can models and weights be saved as expected when using multi_gpu_model?
Heisenberg0391 commentedon Dec 8, 2018
So, as mentioned above, i should train with parallel_model but save the origin model. But what if i want save weights on every epoch as checkpoints using a callback, what should i do?
jashshah commentedon Jun 19, 2019
Are there any updates regarding this either on keras or on tf.keras?