when ever i run # kubectl run ubuntu --image=ubuntu or centos
i gt containercrashoff , when checked in kubectl describe pod below error is observed
Warning Failed 4s (x3 over 22s) kubelet Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "ping": executable file not found in $PATH: unknown
pl suggest to solve this issue
While running the dataflow template in python using the following command
gcloud dataflow flex-template run "streaming-beam-`date +%Y%m%d-%H%M%S`" \
--template-file-gcs-location "$TEMPLATE_PATH" \
--region "$REGION"
The job gets triggered on the console but the DAG is not formed and the following error is reported after a couple of minutes
/usr/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/opt/google/dataflow/python_template_launcher\": stat /opt/google/dataflow/python_template_launcher: no such file or directory": unknown.
Can someone figure out what the problem could be?
I have already checked the following
The template file is present in the location
The registry has the image
Not sure what the error means?
Anyone has any idea how I can correct this error? Seems templates on GitLab don't work.
ERROR: Job failed (system failure): Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/volumes/runner-mh5zgyxe-project-119-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data\\\" to rootfs \\\"/var/lib/docker/overlay2/19e6d863d0d048b030aa4702ee47667085a8e571c266b9623294c4ac68e7c8da/merged\\\" at \\\"/var/lib/docker/overlay2/19e6d863d0d048b030aa4702ee47667085a8e571c266b9623294c4ac68e7c8da/merged/builds\\\" caused \\\"mkdir /var/lib/docker/overlay2/19e6d863d0d048b030aa4702ee47667085a8e571c266b9623294c4ac68e7c8da/merged/builds: no space left on device\\\"\"": unknown (docker.go:817:0s)
I upload a new version of my app as a zipfile and click deploy. Then the status changes to severe.
This is the error trace:
WARN
Environment health has transitioned from Info to Degraded. Command failed on all instances. Incorrect application version found on all instances. Expected version "Sample" (deployment 2). Application update failed 10 seconds ago and took 4 minutes.
ERROR
During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
ERROR
Failed to deploy application.
ERROR
Unsuccessful command execution on instance id(s) 'i------'. Aborting the operation.
ERROR
[Instance: i-002326d7ceeba0ea9] Command failed on instance. Return code:
1 Output: nginx: [emerg] no host in upstream ":80" in /etc/nginx/conf.d/elasticbeanstalk-nginx-docker-upstream.conf:
2 nginx: configuration file /etc/nginx/nginx.conf test failed Failed to start nginx, abort deployment.
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh failed.
For more detail, check /var/log/eb-activity.log using console or EB CLI.
ERROR
Failed to start nginx, abort deployment
/var/log/eb-activity.log
here are errors in this log:
[0mInstalling dependencies from Pipfile.lock (5e00f3)…
Failed to load paths: /bin/sh: 1: /root/.local/share/virtualenvs/app-lp47FrbD/bin/python: not found
...
[2020-05-29T01:51:24.746Z] INFO [11395] - [Application update v1.3.3-1#3/AppDeployStage1/AppDeployEnactHook/00run.sh] : Completed activity. Result:
jq: error (at <stdin>:1): Cannot iterate over null (null)
a2f568b1c255eb9e0fdc6ceebdd29b9ec64b9ab4481a3e1c5bcb11828b0ac526
[2020-05-29T01:51:24.747Z] INFO [11395] - [Application update v1.3.3-1#3/AppDeployStage1/AppDeployEnactHook/01flip.sh] : Starting activity...
[2020-05-29T01:51:26.099Z] INFO [11395] - [Application update v1.3.3-1#3/AppDeployStage1/AppDeployEnactHook/01flip.sh] : Activity execution failed, because: nginx: [emerg] no host in upstream ":80" in /etc/nginx/conf.d/elasticbeanstalk-nginx-docker-upstream.conf:2
nginx: configuration file /etc/nginx/nginx.conf test failed
Failed to start nginx, abort deployment (ElasticBeanstalk::ExternalInvocationError)
caused by: nginx: [emerg] no host in upstream ":80" in /etc/nginx/conf.d/elasticbeanstalk-nginx-docker-upstream.conf:2
nginx: configuration file /etc/nginx/nginx.conf test failed
Failed to start nginx, abort deployment (Executor::NonZeroExitStatus)
...
[2020-05-29T01:51:26.099Z] INFO [11395] - [Application update v1.3.3-1#3/AppDeployStage1/AppDeployEnactHook/01flip.sh] : Activity failed.
[2020-05-29T01:51:26.099Z] INFO [11395] - [Application update v1.3.3-1#3/AppDeployStage1/AppDeployEnactHook] : Activity failed.
[2020-05-29T01:51:26.099Z] INFO [11395] - [Application update v1.3.3-1#3/AppDeployStage1] : Activity failed.
[2020-05-29T01:51:26.100Z] INFO [11395] - [Application update v1.3.3-1#3] : Completed activity. Result:
Application update - Command CMD-AppDeploy failed
The inability to deploy has been consistent for this environment, after several attempts, even reverting to an older version.
Afterwards, I resolved this by isolating the code and error messages using a local docker image with the zipfile. Running the code on my machine outside of docker did NOT reveal any problems, because the pip / pipenv part was missing some depdendency.
Steps for local docker testing:
WITHIN a docker container:
docker system prune
Go to the folder with Dockerfile
docker image build -t <app_name>:<version_number> .
TO run locally:
(docker rm <app_name> first, if you've already got a stopped container with the same name from prior testing)
docker container run --publish 80:80 --name <app_name> myapp:1.0
NOTE:
this won't let you test AWS functions that require environment variables, such as ~.aws credentials because they're not inside the image.
(but you could add them with your Dockerfile)
Once the docker container is running, you'll see (I saw) error messages that were not there when testing locally, because they were caused by a missing package dependency and a pipenv error.
I was trying to re-implement this code from Github and it requires me to install nvidia-docker and run it. The installation of nvidia-docker seemed successful. However, when I run the command nvidia-docker run -it --ipc=host deep-colorization, it throws the following error::
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request\\\\n\\\"\"": unknown.
ERRO[0002] error waiting for container: context canceled
I am not sure what the error means as I don't have any previous experience with the docker ecosystem. Any kind of assistance is appreciated. I am running Ubuntu 18 by the way.
Thanking you in advance.