Docker overlay2: error walking file system: OSError [Errno 40] Too many levels of symbolic links - linux

Question

Main app: uvicorn server on starlette (python) webapp
While I was trying to debug the error in the title (troubleshoot log is following below) running the below command at the host's FS (/var/lib/docker/overlay2/[IMAGE_HASH_FOLDER]
find -L ./ -mindepth 15
I find the files involved in the loop.
Locally is the /usr/bin/X11 and at server I'm getting the following:
error walking file system: OSError [Errno 40] Too many levels of symbolic links: '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr'
The owner of the conflicting files (FS is the one of the host), pending docker service restart after the pruning:
➜ overlay2 find -L ./ -mindepth 15
find: File system loop detected; ‘./d2dba43e7cdbdec81bac529bb85908a7f859f227cda0149389f164272cb372e8/diff/usr/bin/X11’ is part of the same file system loop as ‘./d2dba43e7cdbdec81bac529bb85908a7f859f227cda0149389f164272cb372e8/diff/usr/bin’.
find: File system loop detected; ‘./6ec18b03535c1dac329e05b2abdc68fb0eea742a06878d90f84c4de73ea6a4a9/merged/usr/bin/X11’ is part of the same file system loop as ‘./6ec18b03535c1dac329e05b2abdc68fb0eea742a06878d90f84c4de73ea6a4a9/merged/usr/bin’.
find: File system loop detected; ‘./l/GCDLBXTJXAL5PFTI4BE3MM3OE2/usr/bin/X11’ is part of the same file system loop as ‘./l/GCDLBXTJXAL5PFTI4BE3MM3OE2/usr/bin’.
➜ overlay2 ls -l ./d2dba43e7cdbdec81bac529bb85908a7f859f227cda0149389f164272cb372e8/diff/usr/bin/X11
lrwxrwxrwx 1 root root 1 May 3 2017 ./d2dba43e7cdbdec81bac529bb85908a7f859f227cda0149389f164272cb372e8/diff/usr/bin/X11 -> .
The dockerfile:
FROM python:3.8
COPY src/ ./
RUN /usr/local/bin/python -m pip install --upgrade pip || true
RUN pip install -r requirements.txt || true
ARG POSTGRES_USER
ENV POSTGRES_USER=$POSTGRES_USER
ARG POSTGRES_PASSWORD
ENV POSTGRES_PASSWORD=$POSTGRES_PASSWORD
ARG POSTGRES_SERVER
ENV POSTGRES_SERVER=$POSTGRES_SERVER
ARG POSTGRES_DB
ENV POSTGRES_DB=$POSTGRES_DB
ARG POSTGRES_PORT
ENV POSTGRES_PORT=$POSTGRES_PORT
ARG SESSION_SECRET
ENV SESSION_SECRET=$SESSION_SECRET
ARG DO_YOU_WANT_USERS
ENV DO_YOU_WANT_USERS=$DO_YOU_WANT_USERS
ARG WHERE_AM_I
ENV WHERE_AM_I=$WHERE_AM_I
# SSL
ARG FORWARDED_ALLOW_IPS
ENV FORWARDED_ALLOW_IPS=$FORWARDED_ALLOW_IPS
ARG SSL_CERTIFICATE
ENV SSL_CERTIFICATE=$SSL_CERTIFICATE
ARG SSL_KEYFILE
ENV SSL_KEYFILE=$SSL_KEYFILE
ARG UPLOADS_PATH
ENV UPLOADS_PATH=$UPLOADS_PATH
RUN echo "FINAL STAGE - RUN APP"
EXPOSE 7000
CMD ["python", "run.py"]
Either I run the container with the volume I usually bind:
UPLOADS_PATH=/var/opt/tmp
LOCAL_UPLOADS_PATH=/var/containers/TEST_UPLOADS
docker build --build-arg POSTGRES_USER --build-arg POSTGRES_PASSWORD --build-arg POSTGRES_SERVER --build-arg POSTGRES_DB --build-arg POSTGRES_PORT --build-arg UPLOADS_PATH --build-arg WHERE_AM_I --build-arg SESSION_SECRET --build-arg DO_YOU_WANT_USERS -t test .
docker run -d --name test_container -v ${LOCAL_UPLOADS_PATH}:${UPLOADS_PATH} -p 7000:7000 test
or without the binding, I still get the same error logs & the app is constantly restarting after every request.
How is possible to have such a loop (linked files?) inside the image?
UPDATE
The container was running smoothly until I've changed bcrypt library with pybcrypt and uvicorn with its cythonized version.
Much appreciate any suggestions on what to further explore.
P.S. I've also tried the docker system prune -a, and although there were deprecated stuff, nothing changed.
P. S. 2: #jordanvrtanoski I've separated the question as you've suggested.
UPDATE #2
Following #jordanvrtanoski inspect command:
➜ docker image inspect -f $'{{.RepoTags}}\t{{.GraphDriver.Data.LowerDir}}' $(docker images -q)
[test:latest] /var/lib/docker/overlay2/99e3b5db623ae543d045cc86c2d7d36400c8d1780ec4b86c297f5055bbdfe81a/diff:/var/lib/docker/overlay2/4ed6de1627ba5957c8fa9834c797a60d277c76e61f138d1b6909c55ef5475523/diff:/var/lib/docker/overlay2/7f790257bc4e6ed9e6ea6ef5bed0eb0cf3af213ea913484a40946a45639d8188/diff:/var/lib/docker/overlay2/c8e04185bdc7714e116615a3599a9832ebe2080b43f09b68331cca5d7c109371/diff:/var/lib/docker/overlay2/9ef94affd46bbcc11d62999ab0c59d6bf28cc6d51f13a7513b93bb209738940a/diff:/var/lib/docker/overlay2/62438cdccba1f312f34e8458e4ec695019e6af65107b2e16c3d7eaa53ca03c06/diff:/var/lib/docker/overlay2/9ec57b8b2680944690cdceae73c1c49b31716bd5efbed78bd3d54810bffdc7b6/diff:/var/lib/docker/overlay2/b2c4ce8d2b6764476a452489f58e615fcce939eaecb3d65466f81f5f115a5b5d/diff:/var/lib/docker/overlay2/f8609908601489fb7e3e28a32c423ee556ec041c69ba274a02de316ccbef5c48/diff:/var/lib/docker/overlay2/dcd13187b642277de35f299c1abb1d7d9695972e8b8893267a62f65338679080/diff:/var/lib/docker/overlay2/e2ed1696e3a34e69ed493da3a2c10b942f09384b1cebac54afebea6fef9c4521/diff
[python:3.8] /var/lib/docker/overlay2/c8e04185bdc7714e116615a3599a9832ebe2080b43f09b68331cca5d7c109371/diff:/var/lib/docker/overlay2/9ef94affd46bbcc11d62999ab0c59d6bf28cc6d51f13a7513b93bb209738940a/diff:/var/lib/docker/overlay2/62438cdccba1f312f34e8458e4ec695019e6af65107b2e16c3d7eaa53ca03c06/diff:/var/lib/docker/overlay2/9ec57b8b2680944690cdceae73c1c49b31716bd5efbed78bd3d54810bffdc7b6/diff:/var/lib/docker/overlay2/b2c4ce8d2b6764476a452489f58e615fcce939eaecb3d65466f81f5f115a5b5d/diff:/var/lib/docker/overlay2/f8609908601489fb7e3e28a32c423ee556ec041c69ba274a02de316ccbef5c48/diff:/var/lib/docker/overlay2/dcd13187b642277de35f299c1abb1d7d9695972e8b8893267a62f65338679080/diff:/var/lib/docker/overlay2/e2ed1696e3a34e69ed493da3a2c10b942f09384b1cebac54afebea6fef9c4521/diff
UPDATE #3
So after following both #jordanvrtanoski advices & this post (#Janith Shanilka): Docker overlay2 eating Disk Space
I was missing the following file:
nano /etc/docker/daemon.json
and populated with:
{
"storage-driver": "aufs"
}
then sudo systemctl restart docker
Now the app doesn't crash, but I'm still getting at logs the same loop message:
error walking file system: OSError [Errno 40] Too many levels of symbolic links: '/usr/bin/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/cc'
also #jordanvrtanoski
➜ docker image inspect -f $'{{.RepoTags}}\t{{.GraphDriver.Data.LowerDir}}' $(docker images -q)
[test:latest] <no value>
[python:3.8] <no value>
I've also noticed that df is a little weird, it looks like the docker's volume is like a 'clone' of the basic host's filesystem?
➜ df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 4046520 0 4046520 0% /dev
tmpfs 815676 3276 812400 1% /run
/dev/sda3 49014600 20123088 26798560 43% /
tmpfs 4078368 304 4078064 1% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 4078368 0 4078368 0% /sys/fs/cgroup
/dev/sda1 474730 148714 296986 34% /boot
tmpfs 815672 0 815672 0% /run/user/0
none 49014600 20123088 26798560 43% /var/lib/docker/aufs/mnt/0d98503bd3ea82e353f6776c2d813a642536ad6dd4300299a8fc22b5d6348bc8
UPDATE #4
So after #jordanvrtanoski 's suggestion I returned docker to overlay, from 'aufs'.
The below results are from the host:
➜ cd /var/lib/docker/overlay2
➜ find -L ./ -mindepth 15
find: File system loop detected; ‘./2ecf467259235c8c4605b058bff4f80100790ee7f5010d4954d6aab1a7f28686/merged/usr/bin/X11’ is part of the same file system loop as ‘./2ecf467259235c8c4605b058bff4f80100790ee7f5010d4954d6aab1a7f28686/merged/usr/bin’.
find: File system loop detected; ‘./6f39e8e2089c99f636da9a534e2ccbe7e41202eeb2ce645efa9387dd0ef0b908/diff/usr/bin/X11’ is part of the same file system loop as ‘./6f39e8e2089c99f636da9a534e2ccbe7e41202eeb2ce645efa9387dd0ef0b908/diff/usr/bin’.
find: File system loop detected; ‘./l/5AOADDMRCAKLG2FQDDJEYC6CY2/usr/bin/X11’ is part of the same file system loop as ‘./l/5AOADDMRCAKLG2FQDDJEYC6CY2/usr/bin’.

Answer 1

UPDATE #5
Found the cause: lib uvicorn[standard] is the cythonized version of itself. Once I removed it all errors were gone. So I'll move this to uvicorn's github.
#jordanvrtanoski Thank you once again for your help!

Answer 2

This problem is caused by an self-referencing symbolic linl in the pyhton:3.8 image.
~# docker run -ti --rm python:3.8 bash
root#ef6c6f4e18ff:/# ls -l /usr/bin/X11/X11
lrwxrwxrwx 1 root root 1 May 3 2017 /usr/bin/X11/X11 -> .
The fix the circular reference is caused by python:3.8 image you can simply delete the /usr/bin/X11/X11 symbolic link
root#ef6c6f4e18ff:/# rm /usr/bin/X11/X11
You can add this to your build file as follwos:
FROM python:3.8
COPY src/ ./
RUN rm /usr/bin/X11/X11
RUN /usr/local/bin/python -m pip install --upgrade pip || true
RUN pip install -r requirements.txt || true
ARG POSTGRES_USER
ENV POSTGRES_USER=$POSTGRES_USER
ARG POSTGRES_PASSWORD
ENV POSTGRES_PASSWORD=$POSTGRES_PASSWORD
ARG POSTGRES_SERVER
ENV POSTGRES_SERVER=$POSTGRES_SERVER
ARG POSTGRES_DB
ENV POSTGRES_DB=$POSTGRES_DB
ARG POSTGRES_PORT
ENV POSTGRES_PORT=$POSTGRES_PORT
ARG SESSION_SECRET
ENV SESSION_SECRET=$SESSION_SECRET
ARG DO_YOU_WANT_USERS
ENV DO_YOU_WANT_USERS=$DO_YOU_WANT_USERS
ARG WHERE_AM_I
ENV WHERE_AM_I=$WHERE_AM_I
# SSL
ARG FORWARDED_ALLOW_IPS
ENV FORWARDED_ALLOW_IPS=$FORWARDED_ALLOW_IPS
ARG SSL_CERTIFICATE
ENV SSL_CERTIFICATE=$SSL_CERTIFICATE
ARG SSL_KEYFILE
ENV SSL_KEYFILE=$SSL_KEYFILE
ARG UPLOADS_PATH
ENV UPLOADS_PATH=$UPLOADS_PATH
RUN echo "FINAL STAGE - RUN APP"
EXPOSE 7000
CMD ["python", "run.py"]

Answer 3

In case someone else stumbles across this later - it looks like uvicorn accesses all subfolders of the path you invoke it from. If you're not explicitly setting a working directory in your dockerfile/compose.yaml, this will be the file system root, which gets into all the bind mount infrastructure in proc that you probably don't care about for running an ASGI server.
WORKDIR /home in a dockerfile or working_dir: /home in a compose.yaml should generally be a fine workaround for most Docker use cases for this error, or to your app directory if you're volume mounting in code.