docker swarm create fails when tar can't find container file system - linux

I'm running docker on a CentOS VM. Some version information:
Linux cmodqa.lab.c-cor.com 3.10.0-229.4.2.el7.x86_64 #1 SMP Wed May 13 10:06:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root#xxx ~]# docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 8aae715/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 8aae715/1.6.0
OS/Arch (server): linux/amd64
Ran this command, as root:
TOKEN=$(docker run --rm swarm create)
Returns:
Timestamp: 2015-05-29 09:26:25.967347311 -0700 PDT
Code: System error
Message: [/usr/bin/tar -cf /var/lib/docker/tmp/c77446605e81944d4fb0d09a68339d2026db2b2af100/_tmp.tar -C /var/lib/docker/devicemapper/mnt/c77446605e81944d4fb0d09a68339d2026db2b2afs/tmp .] failed: /usr/bin/tar: /var/lib/docker/devicemapper/mnt/c77446605e81944d4fb0d09a6cb119e60ff/rootfs/tmp: Cannot chdir: No such file or directory
/usr/bin/tar: Error is not recoverable: exiting now
: exit status 2
Frames:
0: setupRootfs
Package: github.com/docker/libcontainer
File: rootfs_linux.go#30
1: Init
Package: github.com/docker/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go#52
2: StartInitialization
Package: github.com/docker/libcontainer.(*LinuxFactory)
File: factory_linux.go#223
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go#35
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go#26
5: main
Package: main
File: docker.go#29
6: main
Package: runtime
File: proc.go#63
7: goexit
Package: runtime
File: asm_amd64.s#2232
time="2015-05-29T09:26:27-07:00" level=fatal msg="Error response from daemon: : exit stat
The file system location the tar command is trying to read from doesn't exist:
[root#cmodqa system]# ls -l /var/lib/docker/devicemapper/mnt/c77446605e81944d4fb0d09a68339d2026db2b2af1335a8a6395b1cb119e60ff/rootfs/tmp
ls: cannot access /var/lib/docker/devicemapper/mnt/c77446605e81944d4fb0d09a68339d2026db2b2af1335a8a6395b1cb119e60ff/rootfs/tmp: No such file or directory
In fact:
ls -l /var/lib/docker/devicemapper/mnt/c77446605e81944d4fb0d09a68339d2026db2b2af1335a8a6395b1cb119e60ff
total 0
The rootfs for the container doesn't seem to be there. (Does it disappear after the container stops?)
I've run this a few times. Same result.

I did some further digging, on docker's repository on Git.
This is a known issue, apparently based in Red Hat's packaging of Docker, and it effects more than swarm.
A bug was filed with Redhat:
https://bugzilla.redhat.com/show_bug.cgi?id=1213258
Use Docker 1.5.0 to work around this situation.

Related

RHEL7 docker-compose will not work "transport: Error while dialing unable to upgrade to h2c, received 404"

Looking to fix this issue. I'm running RHEL7 with docker-compose 2.9.0. I've boiled things down to these files:
[<user>]$ ls -la
-rw-rw-r--. 1 glaisne glaisne 166 Aug 4 14:22 docker-compose.cloud-build.yml
-rw-rw-r--. 1 glaisne glaisne 76 Aug 4 14:04 .env.azure
drwxrwxr-x. 2 glaisne glaisne 24 Aug 4 14:12 helloWorld
[<user>]$ ls ./helloWorld
Dockerfile
[<user>]$ cat ./docker-compose.cloud-build.yml
version: '3.4'
services:
hello-world:
image: ${AZURE_DOCKER_REGISTRY}<servername>.hello-world:${DOCKER_IMAGE_TAG}
build:
context: ./helloWorld
[<user>]$ cat .env.azure
AZURE_DOCKER_REGISTRY=<server>.azurecr.io/
DOCKER_IMAGE_TAG=dev
[<user>]$ cat ./helloWorld/Dockerfile
FROM alpine
CMD ["echo", "Hello World"]
If I run
sudo docker-compose -f ./docker-compose.cloud-build.yml --env-file=./.env.azure build
I get this error:
[+] Building 0.0s (0/0)
listing workers for Build: failed to list workers: Unavailable: connection error: desc = "transport: Error while dialing unable to upgrade to h2c, received 404"
I am no Linux or docker guru, but I've done about all I can to fix this with no luck. The Docker service is running on the system (Docker version info below). Any insight on what this could be would be helpful. I've rebooted, Docker service has been given enough time to run, I don't see any network/firewall issues that I can tell ( from 404 ). I've even looked at the source (I don't speak go), and can't fine any reference to 'dial', 'h2c', 'workers.'
TIA
Docker version info:
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-209.git7d71120.el7_9.x86_64
Go version: go1.10.3
Git commit: 7d71120/1.13.1
Built: Fri Jan 7 13:15:46 2022
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-209.git7d71120.el7_9.x86_64
Go version: go1.10.3
Git commit: 7d71120/1.13.1
Built: Fri Jan 7 13:15:46 2022
OS/Arch: linux/amd64
Experimental: false
It seems that this problem has not yet been completely solved, but for the moment this may help you, try to set DOCKER_BUILDKIT=0 env var to disable the use of the buildkit API.
Reference: https://github.com/containers/podman/issues/13889
Note: If you use the sudo command for execute docker-compose please use the -E flag for preserve the env variables.

access denied in docker when mounting volumes while userns is enabled

I am trying to use docker's user namespaces feature using the official documentation here
I have added the configuration to my daemon.json file like
{
"debug":true,
"experimental": false,
"features":{"buildkit": false},
"userns-remap":"default"
}
I also verified that both subuid and subguid in /etc contain the following entries
dhost:100000:65536
dockremap:165536:65536
I built my image to verify the functionality using an alpine:latest like so
FROM alpine:latest
RUN mkdir -p /root/.cache
WORKDIR /app
command used in building the image docker image build -t myimage:1 .
Then I run a container from this image using
docker container run -it --rm --name mycontainer -v "$(pwd)/test:/app" myimage:1 sh
I get access to the workdir inside the container (app) but I cannot touch/create any file without getting permission denied. Do I need to change the owner of the test directory I used to mount? if yes, who should own it?
docker version
docker version
Client: Docker Engine - Community
Version: 20.10.14
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 24 01:47:57 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.14
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 87a90dc
Built: Thu Mar 24 01:45:46 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Host OS info
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic

How to change the default directory docker uses to build an image

I am trying to set up a gitlab ci.
Because I for some reasons I do not have "gitlab-runner" user and I do not have permission writin on "/home/user_1", this is my installation
/usr/local/bin/gitlab-runner install --user=user_1 --working-directory=/data/external/tmp/gitlab-runner
And this is how I register
/usr/local/bin/gitlab-runner register --url GITLAB_URL --registration-token TOKEN
By the way, I create this gitlab-ci.yml file:
stages:
- deploy
deploy:
stage: deploy
# only:
# - 3.0.x
script:
- echo "deploying"
- sudo docker build -t my_image:v1 .
- echo "********Docker Images********"
- sudo docker image list
- echo "********End of Docker Images********"
- sudo docker run -d -p 3000:5000 --rm --name my_container my_image:v1
tags:
- deploy
I get this error:
Error: error creating build container: Error committing the finished image:
error adding layer with blob "sha256:bb7d5a84853b217ac05783963f12b034243070c1c9c8d2e60ada47444f3cce04":
Error processing tar file(exit status 1):
Error setting up pivot dir: mkdir
/home/user_1/.local/share/containers/storage/overlay/62a747bf1719d2d37fff5670ed40de6900a95743172de1b4434cb019b56f30b4/diff/.pivot_root436648414:
permission denied
I would like to replace /home/user_1/.local/share/containers/storage/overlay/
with another address so that I do not get permission error.
Any advice on how to do so?
I am using Redhat Linux
docker --version is podman version 3.2.3
docker info:
server_name:/home/my_user[ 52 ] --> docker info
host:
arch: amd64
buildahVersion: 1.21.3
cgroupControllers: []
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.0.29-1.module+el8.4.0+11822+6cc1e7d7.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.29, commit: ae467a0c8001179d4d0adf4ada381108a893d7ec'
cpus: 8
distribution:
distribution: '"rhel"'
version: "8.4"
eventLogger: file
hostname: server_name
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
uidmap:
- container_id: 0
host_id: 67298
size: 1
kernel: 4.18.0-305.3.1.el8_4.x86_64
linkmode: dynamic
memFree: 1818484736
memTotal: 33444728832
ociRuntime:
name: runc
package: runc-1.0.0-74.rc95.module+el8.4.0+11822+6cc1e7d7.x86_64
path: /usr/bin/runc
version: |-
runc version spec: 1.0.2-dev
go: go1.15.13
libseccomp: 2.5.1
os: linux
remoteSocket:
path: /run/user/67298/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /bin/slirp4netns
package: slirp4netns-1.1.8-1.module+el8.4.0+11822+6cc1e7d7.x86_64
version: |-
slirp4netns version 1.1.8
commit: d361001f495417b880f20329121e3aa431a8f90f
libslirp: 4.3.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.1
swapFree: 67353165824
swapTotal: 67448598528
uptime: 789h 40m 40.57s (Approximately 32.88 days)
registries:
localhost:
Blocked: false
Insecure: true
Location: localhost
MirrorByDigestOnly: false
Mirrors: []
Prefix: localhost
mkdcvtmaapp01:
Blocked: false
Insecure: true
Location: server_name
MirrorByDigestOnly: false
Mirrors: []
Prefix: server_name
search:
- registry.access.redhat.com
- registry.redhat.io
- docker.io
store:
configFile: /home/my_user/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /bin/fuse-overlayfs
Package: fuse-overlayfs-1.6-1.module+el8.4.0+11822+6cc1e7d7.x86_64
Version: |-
fusermount3 version: 3.2.1
fuse-overlayfs: version 1.6
FUSE library version 3.2.1
using FUSE kernel interface version 7.26
graphRoot: /home/my_user/.local/share/containers/storage
graphStatus:
Backing Filesystem: nfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 0
runRoot: /run/user/67298/containers
volumePath: /home/my_user/.local/share/containers/storage/volumes
version:
APIVersion: 3.2.3
Built: 1627570963
BuiltTime: Thu Jul 29 11:02:43 2021
GitCommit: ""
GoVersion: go1.15.7
OsArch: linux/amd64
Version: 3.2.3
I also have tried these three in my gitlab ci but it did not work:
deploy:
variables:
DOCKER_DRIVER: overlay2
DOCKER_TMP: /data/external/tmp_docker_build
TMPDIR: /data/external/tmp_docker_build
I also did chmod 777 on .local, share, containers, storage, and overlay in this rout /home/user_1/.local/share/containers/storage/overlay/ but it is still not working.
I did not know about this before, either. Apparently you can set the data dir used by docker daemon by adding -g /path/to/dir to the docker daemon command.
For example by adding -g to the DOCKER_OPTS in /etc/default/docker on Ubuntu or Debian systems:
DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4 -g /data/external/docker"
My source is https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169 - there is also a note about how this is done on Fedora or CentOS:
edit /etc/sysconfig/docker, and add the -g option in the other_args variable: ex. other_args="-g /var/lib/testdir". If there’s more than one option, make sure you enclose them in " ". After a restart, (service docker restart) Docker should use the new directory.

Cannot start minikube in Windows 10 with Docker as driver

I tried to run Minikube v1.19.0 on my laptop first time, using Docker as driver, but no luck though. Here are the execution logs:
PS C:\Users\______\kubernetes> minikube start --driver=docker --alsologtostderr
I0916 19:01:44.059272 20464 out.go:278] Setting OutFile to fd 96 ...
I0916 19:01:44.059272 20464 out.go:291] Setting ErrFile to fd 100...
W0916 19:01:44.091562 20464 root.go:292] Error reading config file at C:\Users\proskos\.minikube\config\config.json: open C:\Users\proskos\.minikube\config\config.json: The system cannot find the file specified.
I0916 19:01:44.097634 20464 out.go:285] Setting JSON to false
I0916 19:01:44.110479 20464 start.go:108] hostinfo: {"hostname":"--------","uptime":118298,"bootTime":1631689806,"procs":291,"os":"windows","platform":"Microsoft Windows 10 Enterprise","platformFamily":"Standalone Workstation","platformVersion":"10.0.18363 Build 18363","kernelVersion":"10.0.18363 Build 18363","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"---------------"}
W0916 19:01:44.110479 20464 start.go:116] gopshost.Virtualization returned error: not implemented yet
I0916 19:01:44.113153 20464 out.go:157] πŸ˜„ minikube v1.19.0 on Microsoft Windows 10 Enterprise 10.0.18363 Build 18363
πŸ˜„ minikube v1.19.0 on Microsoft Windows 10 Enterprise 10.0.18363 Build 18363
I0916 19:01:44.116296 20464 driver.go:322] Setting default libvirt URI to qemu:///system
W0916 19:01:50.237519 20464 docker.go:108] docker version returned error: deadline exceeded running "docker version --format {{.Server.Os}}-{{.Server.Version}}": exit status 1
I0916 19:01:50.244288 20464 out.go:157] ✨ Using the docker driver based on user configuration
✨ Using the docker driver based on user configuration
I0916 19:01:50.244288 20464 start.go:276] selected driver: docker
I0916 19:01:50.244810 20464 start.go:718] validating driver "docker" against <nil>
I0916 19:01:50.244810 20464 start.go:729] status for docker: {Installed:true Healthy:false Running:false NeedsImprovement:false Error:exit status 1
deadline exceeded running "docker version --format {{.Server.Os}}-{{.Server.Version}}"
k8s.io/minikube/pkg/minikube/registry/drvs/docker.status
/app/pkg/minikube/registry/drvs/docker/docker.go:104
k8s.io/minikube/pkg/minikube/registry.Status
/app/pkg/minikube/registry/global.go:140
k8s.io/minikube/pkg/minikube/driver.Status
/app/pkg/minikube/driver/driver.go:305
k8s.io/minikube/cmd/minikube/cmd.selectDriver
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra#v1.1.3/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra#v1.1.3/command.go:960
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra#v1.1.3/command.go:897
k8s.io/minikube/cmd/minikube/cmd.Execute
/app/cmd/minikube/cmd/root.go:156
main.main
/app/cmd/minikube/main.go:82
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371 Reason:PROVIDER_DOCKER_DEADLINE_EXCEEDED Fix:Restart the Docker service Doc:https://minikube.sigs.k8s.io/docs/drivers/docker/}
I0916 19:01:50.250032 20464 out.go:157]
W0916 19:01:50.250390 20464 out.go:222] πŸ’£ Exiting due to PROVIDER_DOCKER_NOT_RUNNING: deadline exceeded running "docker version --format -": exit status 1
πŸ’£ Exiting due to PROVIDER_DOCKER_NOT_RUNNING: deadline exceeded running "docker version --format -": exit status 1
W0916 19:01:50.250390 20464 out.go:222] πŸ’‘ Suggestion: Restart the Docker service
πŸ’‘ Suggestion: Restart the Docker service
W0916 19:01:50.250924 20464 out.go:222] πŸ“˜ Documentation: https://minikube.sigs.k8s.io/docs/drivers/docker/
πŸ“˜ Documentation: https://minikube.sigs.k8s.io/docs/drivers/docker/
I0916 19:01:50.252904 20464 out.go:157]
Docker Desktop Service is up and running, though. The problem wasn't resolved by restarting Docker. Here are the details of my Docker desktop installation:
PS C:\Users\______\kubernetes> docker version
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:58:50 2021
OS/Arch: windows/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:10 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Finally, my Docker Desktop installation is configured to run Linux containers:
docker info --format '{{.OSType}}'
linux
Could you help me find out why Minikube can't communicate with Docker?
Try these three commands sequentially:
minikube config set driver docker
minikube delete
minikube start --driver=docker
I might be wrong but in my personal opinion this is a docker version issue.
It is not a typical behavior to have such error in the situation when docker service is up&running.
status for docker: {Installed:true Healthy:false Running:false NeedsImprovement:false Error:exit status 1
deadline exceeded running "docker version --format {{.Server.Os}}-{{.Server.Version}}"
and
Exiting due to PROVIDER_DOCKER_NOT_RUNNING: deadline exceeded running "docker version --format -": exit status 1
πŸ’£ Exiting due to PROVIDER_DOCKER_NOT_RUNNING: deadline exceeded running "docker version --format -": exit status 1
My recommendations:
docker system prune to completeley delete docker cache, images, etc
remove completely minikube
now 2 options. Either try to again start minikube --driver-docker or uninstall docker and install 1-2 versions below. I would go in your case with last one.

Fabric ./byfn.sh up returns error "...container_linux.go:348..." [win10 home edition]

I am try to start an fabric network according to the doc "Building Your First Network" and the prerequisite docs.
However, when I execute the command ./byfn.sh up, it returns the error below:
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "no such file or directory": unknown ERROR !!!! Test failed
I have already tried to search for this error but with no luck.
I would appreciate if anyone can help me ...
System information:
$ docker version
Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24302
Built: Fri Mar 23 08:31:36 2018
OS/Arch: windows/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:13:39 2018
OS/Arch: linux/amd64
Experimental: false
37675#DESKTOP-JU1BJMT MINGW64 /c/Users/fabric-samples_120/first-network ((v1.2.0 ))
$ go version
go version go1.10.1 windows/amd64
The complete output of byfn.sh up is here.
Please change the docker exec command to below
docker exec cli //bin//bash scripts/script.sh $CHANNEL_NAME $CLI_DELAY $LANGUAGE $CLI_TIMEOUT $VERBOSE
Add //bin//bash to the command so it points to the bash part of the container
Later in case if it happens to throw EOL exception, because shell doesn't understand DOS/Windows-like line endings
Edit in notepad ++ Edit > EOL Conversion > select Unix/OSX
Then it woks out
I guess it is because your docker version is too new for release-1.2,
As documented in
https://hyperledger-fabric.readthedocs.io/en/release-1.2/prereqs.html#docker-and-docker-compose
Try to pin to Docker version 17.06.2-ce, it could be better

Resources