Ubuntu 16.04 Cron to run Docker container - azure

I have a virtual machine running Ubuntu 16.04 that I want to run cron-scheduled Docker containers on. I have configured the Docker host and my image repository such that they are accessible by the VM, and images run fine on the machine.
My issue is that when I create a cron-schedule using crontab -e in /var/spool/cron, the containers do not show up. I should be able to view any stopped containers using $ docker ps -a, but it does not show them. Running sudo grep CRON -i /var/log/syslog shows the output:
Mar 20 16:22:01 SpacyVM CRON[121879]: (bdsadmin) CMD (docker run -d
bdsdev.azurecr.io/crawler-public)
Mar 20 16:22:05 SpacyVM CRON[121878]: (CRON) info (No MTA installed,
discarding output)
Mar 20 16:24:01 SpacyVM CRON[124254]: (bdsadmin) CMD (docker run -d
bdsdev.azurecr.io/crawler-public)
Mar 20 16:24:02 SpacyVM CRON[124253]: (CRON) info (No MTA installed,
discarding output)
Now, this does run every 2 minutes, per the schedule for debugging purposes, but docker ps -a shows nothing, even though the containers should have exited and thus should be idle until I use docker rm $(docker ps -aq). Has anyone seen this problem before?
Here is my job configured in crontab:
*/2 * * * * docker run -d bdsdev.azurecr.io/crawler-public
Note: I have not created a mailing setup for the cronjobs to output to, hence the No MTA... error, and I am running this in an Ubuntu VM, not on an Ubuntu docker base image.

Found my problem. When I was originally troubleshooting using grep, my search was omitting a fairly important piece of the log stream. So by using vim to view /var/log/syslog:
Mar 20 06:34:01 SpacyVM dockerd[1469]: time="2018-03-20T06:34:01.634861659Z" level=info msg="Attempting next endpoint for pull after error: manifest unknown: manifest unknown"
I had made a spelling error in my crontab file, so the image is not able to be located by the docker daemon, causing this problem.
Chalk it up to amateur hour

Related

Crontab failing to restart systemctl process

Hi im trying to setup a cron job to restart a game server daily but its failing to work
i have used crontab -e to install a job
* 3 * * * systemctl --user restart <servicename>
this failed to restart my process at 3 am as expected.
i also tried moving the command to a bash file and running the bash file through cron (as i have seen alot of people use .sh files to run commands)
additionally i have tried installing to job as root with zero success
this had similar results
my syslog did show a command execute as expected yesterday though the service didnt restart, since then cron isnt showing any new attempts.
i found some info on troubleshooting and following the guidance shows that my cron is working, i managed write the date/time to a file every minute and the service is clearly running and this showed in the logs.
is there a problem with cron running systemctl commands or is there further steps i can use to see why its failing?
Edit:
i tried running my command as root also and nothing happened
Jan 31 09:00:01 ns509515 CRON[108307]: (root) CMD (<user> systemctl --user restart EcoServer)
Jan 31 09:01:01 ns509515 CRON[108330]: (root) CMD (<user> systemctl --user restart EcoServer)
Jan 31 09:02:01 ns509515 CRON[108356]: (root) CMD (<user> systemctl --user restart EcoServer)

Docker service exits on deploy-over-SSH from Jenkins but manually succeeds

I have a strange issue. I have configured a SSH_USER on Jenkins and trying to deploy a simple docker-service with "deploy-over-SSH".
Every time I deploy it Exits as below, and logs just says "Terminated"
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bea48e1ee755 localhost/my-image:latest /bin/sh -c npm ru... 13 seconds ago Exited (143) 13 seconds ago 0.0.0.0:6007->3000/tcp my-cont
$ docker logs my-cont
Terminated
But if I try running manually on the same server with same SSH_USER, I am able to run docker container successfully.
docker run -d -it -p 6007:3000 --name my-cont my-image
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
nce48e1ee721 localhost/my-image:latest /bin/sh -c npm ru... 21 minutes ago 21 minutes ago 0.0.0.0:6007->3000/tcp my-cont
The script I am running over-SSH is very simple from Jenkins, I am passing the Port from Jenkins,
$ bash ./run.sh $Port
docker run -d -it -p $1:3000 --name my-cont my-image
Not sure , what is causing the issue.
As I mentioned in the comment, upgrading the version 'podman version 2.0.5' worked for me.
The error or docker logs never gave me any hint, clue on version.
But that was the solution. Thanks for your comments.

Dockerized nginx shuts down after a few seconds

I'm working with Ubuntu 18 and I´m trying to run a dockerized nginx with a shared file between the host machine and the container: /home/ric/wrkspc/djangodocker/djangodocker/nginx.conf
I do so by running the following command, after which I'm prompted with container's ID:
$ sudo docker container run -v /home/ric/wrkspc/djangodocker/djangodocker/nginx.conf:/etc/nginx/nginx.conf:ro -d nginx
facc4d32db31de85d6f360e581bc7d36f257ff66953814e985ce6bdf708c3ad0
Now, if I try to list all the running containers, the nginx one doesn't appear listed:
(env) ric#x:~/wrkspc/djangodocker/djangodocker$ sudo docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
36090ff0759c mysql "docker-entrypoint.s…" 3 days ago Up 3 days 0.0.0.0:3306->3306/tcp, 33060/tcp boring_panini
Sometimes, if I run the docker ls command fast enough, I can see the nginx container listed for just a few seconds and then it disappears.
Why is the nginx container not being listed?
I think container immediately exits after started.
can you troubleshoot by looking into docker logs using the command
docker logs containerID
Also, you can try running the container interactively to identify the error without using -d option

Docker command can't connect to Docker daemon

I want to make a move to Docker, so I've just started to mess around with it. I've installed Docker on a VirtualBox Ubuntu 15.10 (Wily Werewolf) installation and as suggested here I then tried running a basic nginx Docker image:
$ docker run --name mynginx1 -P -d nginx
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
So I checked out whether Docker was running:
$ sudo service docker status
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since vr 2015-11-06 08:41:48 CET; 15min ago
Docs: https://docs.docker.com
Main PID: 7542 (docker)
CGroup: /system.slice/docker.service
└─7542 /usr/bin/docker daemon -H fd://
nov 06 08:41:47 kramer65-VirtualBox systemd[1]: Starting Docker Application Container Engine...
nov 06 08:41:47 kramer65-VirtualBox docker[7542]: time="2015-11-06T08:41:47.900410966+01:00" level=info msg="API ...ock"
nov 06 08:41:48 kramer65-VirtualBox docker[7542]: time="2015-11-06T08:41:48.033514149+01:00" level=info msg="Fire...lse"
nov 06 08:41:48 kramer65-VirtualBox docker[7542]: time="2015-11-06T08:41:48.141594321+01:00" level=info msg="Defa...ess"
nov 06 08:41:48 kramer65-VirtualBox docker[7542]: time="2015-11-06T08:41:48.416294436+01:00" level=warning msg="Y...it."
nov 06 08:41:48 kramer65-VirtualBox docker[7542]: time="2015-11-06T08:41:48.565507576+01:00" level=info msg="Load...rt."
nov 06 08:41:48 kramer65-VirtualBox docker[7542]: time="2015-11-06T08:41:48.567907022+01:00" level=info msg="Load...ne."
nov 06 08:41:48 kramer65-VirtualBox docker[7542]: time="2015-11-06T08:41:48.567945214+01:00" level=info msg="Daem...ion"
nov 06 08:41:48 kramer65-VirtualBox docker[7542]: time="2015-11-06T08:41:48.567969891+01:00" level=info msg="Dock....9.0
nov 06 08:41:48 kramer65-VirtualBox systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
This suggests that the Docker daemon is actually already running, but to be sure I just started the Docker daemon manually:
$ sudo docker daemon
INFO[0000] API listen on /var/run/docker.sock
INFO[0000] [graphdriver] using prior storage driver "aufs"
INFO[0000] Firewalld running: false
INFO[0000] Default bridge (docker0) is assigned with an IP address XXX.XX.X.X/XX. Daemon option --bip can be used to set a preferred IP address
WARN[0000] Your kernel does not support swap memory limit.
INFO[0000] Loading containers: start.
INFO[0000] Loading containers: done.
INFO[0000] Daemon has completed initialization
INFO[0000] Docker daemon commit=76d6bc9 execdriver=native-0.2 graphdriver=aufs version=1.9.0
I then tried running the image again, but with the same result:
$ docker run --name mynginx1 -P -d nginx
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
I tried sudo'ing the command, but to no avail. What am I doing wrong here?
You need to add your current user to the docker group as follows:
sudo usermod -aG docker $(whoami)
then logout & login again into the system or restart the system.
test by docker version
for further info how to install docker-engine follow docker documentation
Add the user to the docker group
Add the docker group if it doesn't already exist:
sudo groupadd docker
Add the connected user "${USER}" to the docker group:
sudo gpasswd -a ${USER} docker
Restart the Docker daemon:
sudo service docker restart
Either do a newgrp docker or log out/in to activate the changes to
groups.
Usually, the following command does the trick:
sudo service docker restart
This, instead of docker start for the cases where Docker seems to already be running.
If that works then, as suggested and in another answer and on this GitHub issue, if you haven't added yourself in the docker group do it by running:
sudo usermod -aG docker <your-username>
And you're most likely good to go.
As for anybody else bumping into this, in some OS's docker doesn't start right after you install it and, as a result, the same can't connect to daemon message appears. In this case you can first verify that Docker is indeed not running by checking the status of your docker service by executing:
sudo service docker status
If the output looks something like: docker stop/waiting instead of docker start/running, process 15378 then it obviously means Docker is not active. In this case make sure you start it with:
sudo service docker start
And, as before, you'll most likely be good to go.
note to self: I get the error from the question's title when I forget to run docker command with sudo:
sudo docker run ...
[Ubuntu 15.10]
Had the same issue and what worked for me was:
Checking the ownership of /var/run/docker.sock
ls -l /var/run/docker.sock
If you're not the owner then change ownership with the command
sudo chown *your-username* /var/run/docker.sock
Then you can go ahead and try executing the docker commands hassle-free :D
After installing docker on Ubuntu, I ran the following command:
sudo service docker start
Have you tried it?
After install everything and start the service, try close your terminal and open it again, then try pull your image
Edit
I also had this issue again, if the solution above won't worked, try this solution that is the command bellow
sudo mv /var/lib/docker/network/files/ /tmp/dn-bak
Considerations
If command above works you probably are with network docker problems, anyway this resolves it, to confirm that, see the log with the command bellow
tail -5f /var/log/upstart/docker.log
If the output have something like that
FATA[0000] Error starting daemon: Error initializing network controller: could not delete the default bridge network: network bridge has active endpoints
/var/run/docker.sock is up
You really are with network problems, however I do not know yet if the next time you restart(update, 2 months no issue again) your OS will get this problem again and if it is a bug or installation problem
My docker version
Client:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64
I had the same problem. Been struggling for two days to solve it.
It only worked when I did:
According to Docker's Tutorial, you need to add the Docker key if not already added using:
$ sudo wget -qO- https://get.docker.com/gpg | sudo apt-key add -
Then make sure you grant docker privileges to yourself using:
$ sudo usermod -aG docker $USER
Hope this helps you too.
enter as root (sudo su) and try this:
unset DOCKER_HOST
docker run --name mynginx1 -P -d nginx
I've the same problem here, and the docker command only worked running as root, and also with this DOCKER_HOST empty
PS: also beware that the correct and official way to install on Ubuntu is to use their apt repositories (even on 15.10), not with that "wget" thing.
For OSX:
After opening docker and starting the 'default' machine via the Quickstart Terminal (https://docs.docker.com/engine/installation/mac/), you try docker commands and get this "can't connect to docker daemon" message, it turns out you need some env variables set:
eval "$(docker-machine env default)"
Then try it out with docker run hello-world to see if everything is peachy.
For the ones who already tried restarting your machine, unsetting the environment variable DOCKER_HOST as told in the docker env documentation and all the rest just try to go with the
sudo service docker restart
Only this did the trick for me even after restarting the machine.
Giving non-root access - from docker
Add the docker group if it doesn't already exist.
$ sudo groupadd docker
Add the connected user "${USER}" to the docker group.
Change the user name to match your preferred user.
You may have to logout and log back in again for
this to take effect.
$ sudo gpasswd -a ${USER} docker
Restart the Docker daemon.
$ sudo service docker restart
This question is currently number 3 on a Google search. After doing some research into solving this problem on my Linux system I thought I would write this answer. The original post states the problem is on Ubuntu but I also experienced the issue using Fedora. With that in mind, here is what I did to fix the problem.
On Fedora 22
Installing Docker:
$> curl -fsSL https://get.docker.com/ | sh
After installing Docker:
A user needs to be added to the docker group.
$> sudo usermod -aG docker
The docker daemon needs to be started
$> sudo service docker start
You can set the daemon to start at boot
$> sudo chkconfig docker on
You can verify the docker service is running
$> service docker status
And one last final check
$> docker run hello-world
Tested in Ubuntu 16.04
# Create the docker group and add your user to the docker group
groupadd docker
usermod -aG docker $USER
newgrp docker
# Configure docker service to be exposed
mkdir -p /etc/systemd/system/docker.service.d
echo -e '[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376' >> /etc/systemd/system/docker.service.d/override.conf
# restart service
systemctl daemon-reload
service docker restart
Try to use "sudo" with the command you are running.
I have same issue while running docker.
you can run commands as sudo user:
sudo docker ***your command here***
For Ubuntu:
Happened with me when I updated docker.
You need to unmask the service and socket and then restart the service.
Following worked for me:
systemctl unmask docker.service
systemctl unmask docker.socket
systemctl start docker.service
What happend behind the scenes
systemd also has the ability to mark a unit as completely unstartable, automatically or manually, by linking it to /dev/null. This is called masking the unit, and is possible with the mask command.
sudo systemctl mask docker.service
You can check the list of masked services using:
sudo systemctl list-unit-files
To enable auto/manual start of service you need to unmask it using:
sudo sytemctl unmask docker.service
Now the service will be enabled as shown below
As docker binds to a unix socket which is owned by root while starting up, using 'sudo' along with the docker commands will work.
I also had the same issue. The problem was in sockets allocated to docker-daemon and docker-client.
First, permission was not set for the docker-client on docker.sock You can set it using "sudo usermod -aG docker $USER"
Then check your bash file where the docker-client is running, For me it was on 0.0.0.0:2375, while docker-daemon was running on unix socket.(It was set in the configuration file of dockerd).
Just comment the bash-line and it'll work fine.
But if you want to make it work on TCP port instead of unix socket, change the configuration file of dockerd and set it on 0.0.0.0.2375 and keep the line in bash as it is if present or set it to 0.0.0.0:2375.
Perhaps this will help someone, as the error message is extremely unhelpful, and I had gone through all of the standard permission steps numerous times to no avail.
Docker occasionally leaves ghost environment variables in place that block access, despite your system otherwise being correctly set up. The following shell commands may make it accessible again, if you have had it running at one point and it just stopped cooperating after a reboot:
unset DOCKER_HOST
unset DOCKER_TLS_VERIFY
unset DOCKER_TLS_PATH
docker ps
I had a previously working docker install, and after rebooting my laptop it simply refused to work. Was correctly added to the docker user group, had the correct permissions on the socket, etc, but could still not run docker login, docker run ..., etc. This fixed it for me. Unfortunately I have to run this on each reboot. This is mentioned on a couple of github issues also as a workaround, although it seems like a bug that this is a persistent barrier to correct operation of Docker (note: I am on Arch Linux, not OSX, but this was the same issue for me).
I was able to fix that by running the following command:
sudo mv /var/lib/dpkg/info/docker-ce* /tmp
I have faced same error on Amazon EC2 instance. The issue got fixed after restarting the instance.
Add current user to docker group:
sudo usermod -aG docker $(whoami)
For Ubuntu 16.04
Inside file /lib/systemd/system/docker.service change:
ExecStart=/usr/bin/dockerd fd://
with:
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375
Inside file /etc/init.d/docker change:
DOCKER_OPTS=
with:
DOCKER_OPTS="-H tcp://0.0.0.0:2375"
and then restart your computer.

Docker: Are you trying to connect to a TLS-enabled daemon without TLS?

On Linux Mint 17.1 x86_64 with kernel 3.13.0-48-generic and OpenSSL version 1.0.1f-1ubuntu2.11; whenever I try to execute any docker command (like docker login or docker run hello-world), I get the following error:
FATA[0000] Get http:///var/run/docker.sock/v1.18/info: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?
I have made a group called docker using sudo usermod -aG docker username, I have tried running the commands both as root and normally, added $(boot2docker shellinit 2> /dev/null) to ~/.profile as instructed here, restarted my PC and reinstalled OpenSSL.
Any idea what am I missing? Can this be a hardware issue?
You can check if it has started by running the command ps -ef. You can also grep it to docker if you want to reduce the number of results(using | grep docker). If its of not running, execute
sudo service docker start
OR if it still doesn't work then
You can refer this link
docker docs
You can run docker run -d or docker run -d & so that you can use the same terminal or even close it if. It will set the value to true, so your container will run in "detached" mode, in the background.
You can also auto start it when your OS starts using update-rc.d servicename defaults or you can also refer to the links below, where you have to give docker as service name and your defaults.
Some more links to refer -
auto start,
upstart
These are different ways of doing it.
I deployed docker on CentOS 7, and my friend first helped me check whether the process/service docker is actually running or not. He used the command ps -ef | grep docker. It turned out that the process wasn't running at all. Finally, he started this process by using the command service docker start. In the end, the problem was solved.
Try installing apparmor
sudo apt install apparmor
There are other things to look for in this answer too.
I had this same problem when I tried to follow along some online resource; I was able to resolve this by running docker as the super user, try adding sudo before your docker commands:
sudo docker ps -a
sudo docker run hello-world
Hope it helps.
I faced the same issue when I was creating docker image from Jenkins simply add
the user to docker group and then restart docker services and in my case I have to restart Jenkins services
This was the error which I got
http:///var/run/docker.sock/v1.19/build?cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&t=59aec062a8dd8b579ee1b61b299e1d9d340a1340: dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?
FATAL: Failed to build docker image from project Dockerfile
java.lang.RuntimeException: Failed to build docker image from project Dockerfile
Solution:
[root#Jenkins ssh]# groupadd docker
[root#Jenkins ssh]# gpasswd -a jenkins docker
Adding user jenkins to group docker
[root#Jenkins ssh]# /etc/init.d/docker restart
Stopping docker: [ OK ]
Starting docker: [ OK ]
[root#Jenkins ssh]# /etc/init.d/jenkins restart
Shutting down Jenkins [ OK ]
Starting Jenkins [ OK ]
[root#Jenkins ssh]#
Ran into the same problem this morning. You may just want to do service docker start. It works for centos (Could work for all), and its one of those small nagging things i forget all the time.
Technically when you first install a service such as docker or httpd (apache) it needs actually be started otherwise you will get that error. You can also see the status's of other services by doing service service_name statusbut again, that's how it is for my centos 7.
This can happen when the docker service is unable to start.
sudo service docker start or restart returning no output does not mean that it started successfully.
You can use sudo /etc/init.d/docker status to find out if, and why, it failed to start.
In my case, it was due to lack of disk space.
username#computer:~$ sudo service docker restart
username#computer:~$ sudo /etc/init.d/docker status
? docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit) since Mon 2015-12-21 15:11:59 PST; 21s ago
Docs: http://docs.docker.com
Process: 26463 ExecStart=/usr/bin/docker -d -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
Main PID: 26463 (code=exited, status=1/FAILURE)
Dec 21 15:11:59 computer docker[26463]: time="2015-12-21T15:11:59-08:00" level=info msg="Listening for HTTP on fd ()"
Dec 21 15:11:59 computer docker[26463]: time="2015-12-21T15:11:59-08:00" level=info msg="+job init_networkdriver()"
Dec 21 15:11:59 computer docker[26463]: time="2015-12-21T15:11:59-08:00" level=info msg="-job init_networkdriver() = OK (0)"
Dec 21 15:11:59 computer docker[26463]: time="2015-12-21T15:11:59-08:00" level=fatal msg="Shutting down daemon due to errors: Insertion failed because database is full: database or disk is full"
Dec 21 15:11:59 computer systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Dec 21 15:11:59 computer systemd[1]: Unit docker.service entered failed state.
Dec 21 15:11:59 computer systemd[1]: docker.service failed.
Dec 21 15:11:59 computer systemd[1]: start request repeated too quickly for docker.service
Dec 21 15:11:59 computer systemd[1]: Failed to start Docker Application Container Engine.
Dec 21 15:11:59 computer systemd[1]: docker.service failed.
After deleting some files, I was able to restart the service and run docker containers normally.
I solved it by running that command in sudo mode, e.g. "sudo docker images"
Ran into the same problem after following the steps at https://docs.docker.com/linux/step_one/. docker service was running and I had also added the user to docker group. docker commands were not working without sudo.
What solved it for me was restarting the PC.
Note- Before PC restart, I had also tried stopping and starting the docker daemon. I'm on Ubuntu 12.04.
In my case it was because of file /Users/user/.ssh/config has permissions 777.
To check it run in terminal ssh docker#localhost
Ran into the same problem after following the steps at https://docs.docker.com/linux/step_one/. docker service was running and I had also added the user to docker group. docker commands were not working without sudo.
What solved it for me was restarting the PC. Note- Before PC restart, I had also tried stopping and starting the docker daemon. I'm on Ubuntu 12.04.
sudo chmod 777 /var/run/docker.sock

Resources