Tensorflow Object detection training job fails on Google cloud - linux

Question

I have my Google Storage Bucket in the following manner:
-data
--labels.pbtxt
--train.record
--test.record
-training
--config file
--packages
And my local machine has the data in /tensorflow/models/research/object_detection in the same manner, additionally
-training
--cloud.yml
And I'm running the following command to start job on google cloud ML engine
gcloud ml-engine jobs submit training object_detection_0.1 --job-
dir=gs://{BUCKET NAME}/training --packages dist/object_detection-
0.1.tar.gz,slim/dist/slim-0.1.tar.gz --module-name object_detection.train --
region us-central1 --config /##/##/models/research/object_detection/training
-- --train_dir=gs://{BUCKET NAME}/training --
pipeline_config_path=gs://{BUCKET NAME}/training/config_file.config
Google cloud logs show me the following error.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/.local/lib/python2.7/site-packages/object_detection/train.py",
line 49, in <module>
from object_detection import trainer
File "/root/.local/lib/python2.7/site-
packages/object_detection/trainer.py", line 33, in <module>
from deployment import model_deploy
ImportError: No module named deployment
replica worker 0,1,2,3 - same error
The replica worker 4 exited with a non-zero status of 1. Termination reason:
Error.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/.local/lib/python2.7/site-packages/object_detection/train.py",
line 49, in <module>
from object_detection import trainer
File "/root/.local/lib/python2.7/site-
packages/object_detection/trainer.py", line 33, in <module>
from deployment import model_deploy
ImportError: No module named deployment
replica ps 0,1 -same error
The replica ps 2 exited with a non-zero status of 1. Termination reason:
Error.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/.local/lib/python2.7/site-packages/object_detection/train.py",
line 49, in <module>
from object_detection import trainer
File "/root/.local/lib/python2.7/site-
packages/object_detection/trainer.py", line 33, in <module>
from deployment import model_deploy
ImportError: No module named deployment

Answer 1

I am having the same problem with the deeplab model. It seems they refer to this folder, because it works for me if I placed were it should to be called properly
By the way...I let me know how you solved it.