I would appreciate your suggestions and advise on the following please :
I am using a SLURM cluster and my colleagues have advised to run a singularity container on the cluster, and re-direct the output of the singularity container to a folder that is hosted in the /scratch folder of each computing node.
for example :
singularity exec --bind /local/scratch/bt:/output \
singularity_latest.sif run \
-o /output
i would like to ask please : how can i access the "output" folder in the "scratch" of the computing node ? Thanks a lot !
bogdan
You can think of --bind as a bit like a symlink. Running ls /local/scratch/bt on the host OS is equivalent to running ls /output inside the exec process.
mkdir scratch
touch scratch/file1
ls -l scratch
# total 0
# -rw-rw-r-- 1 tsnowlan tsnowlan 0 Jun 8 09:13 file1
singularity exec -B $PWD/scratch:/output my_image.sif ls -l /output
# total 0
# -rw-rw-r-- 1 tsnowlan tsnowlan 0 Jun 8 09:13 file1
# singularity also accepts relative paths
singularity exec -B scratch:/output my_image.sif touch /output/file2
ls -l scratch
# total 0
# -rw-rw-r-- 1 tsnowlan tsnowlan 0 Jun 8 09:13 file1
# -rw-rw-r-- 1 tsnowlan tsnowlan 0 Jun 8 09:16 file2
Related
So I have a docker container running which has a couple of folder/files inside.
When I go inside the container I can see that they are:
hostmachine$> docker exec -it testContainer bash
testContainer#testContainer:~$ ls -l
total 40200
-rw-r--r-- 1 root root 41156267 Apr 11 20:10 herp-4.2.2-SNAPSHOT-ap3.tar.gz
drwxr-xr-x 2 root root 4096 Apr 11 20:10 var
drwxr-xr-x 5 root root 4096 Apr 11 20:10 herp-4.2.2-SNAPSHOT
lrwxrwxrwx 1 root root 39 Apr 11 20:10 current -> /home/testContainer/herp-4.2.2-SNAPSHOT
My question is:
What script, how can I use it on my hostmachine that will:
1) Echo back (return) the following info from the container:
"current -> /home/testContainer/herp-4.2.2-SNAPSHOT"
2) And then taking it further, just echo back the last bit:
herp-4.2.2-SNAPSHOT
So essentially it could be like:
hostmachine$> ./getSnapshotName.sh
herp-4.2.2-SNAPSHOT
Just build it up in steps. First, no need to use -it and bash to run an interactive shell, just run the command directly:
$ docker exec testContainer ls -l current
lrwxrwxrwx 1 root root 39 Apr 11 20:10 current -> /home/testContainer/herp-4.2.2-SNAPSHOT
Next, use readlink to save you from parsing the ls output:
$ docker exec testContainer readlink current
/home/testContainer/herp-4.2.2-SNAPSHOT
Finally, run that through basename to strip out the directory parts:
$ basename $(docker exec testContainer readlink current)
herp-4.2.2-SNAPSHOT
I start mesos master container with the following command:
ExecStart=/usr/bin/docker run --rm \
-e MESOS_HOSTNAME={{ ansible_hostname }} \
-e MESOS_IP={{mesos.external_ip}} \
-e MESOS_QUORUM={{mesos.quorum}} \
-e MESOS_ZK={{mesos.zk}} \
-e MESOS_LOG_DIR=/var/log/mesos -v /var/log/mesos:/var/log/mesos \
-e MESOS_WORK_DIR=/var/lib/mesos -v {{mesos_work_dir}}:/var/lib/mesos \
--name mesos-master --net host {{docker.master}}
And I get a huge amount of mesos logs:
mesos logs
lrwxrwxrwx. 1 root root 60 Apr 19 17:14 mesos-master.INFO -> mesos-master.vdorct1.invalid-user.log.INFO.20170419-141458.6
-rw-r--r--. 1 root root 23983 Apr 19 16:46 mesos-master.vdorct1.invalid-user.log.INFO.20170419-134210.6
-rw-r--r--. 1 root root 48678 Apr 19 17:14 mesos-master.vdorct1.invalid-user.log.INFO.20170419-135536.6
-rw-r--r--. 1 root root 6595 Apr 19 17:14 mesos-master.vdorct1.invalid-user.log.INFO.20170419-141436.6
-rw-r--r--. 1 root root 8240 Apr 19 17:14 mesos-master.vdorct1.invalid-user.log.INFO.20170419-141444.6
-rw-r--r--. 1 root root 122324 Apr 19 18:14 mesos-master.vdorct1.invalid-user.log.INFO.20170419-141458.6
-rw-r--r--. 1 root root 253 Apr 19 16:42 mesos-master.vdorct1.invalid-user.log.WARNING.20170419-134210.
I need to manage this logs defining how many files to keep and/or for how many days to keep logs.
I can reach it using bash + cron. But this is not acceptable because of company policy: I have to use linux tools.
I tried to configure logrotate. But it is not usable only for deleting files: it can delete only rotated files or all files.
May be I can force mesos write logs in a single file: for example only
mesos-master.vdorct1.invalid-user.log.INFO
instead of
mesos-master.vdorct1.invalid-user.log.INFO.20170419-134210.6
mesos-master.vdorct1.invalid-user.log.INFO.20170419-135536.6
...
In that case I could configure logrotate.
So the question is: how I can automatically clean mesos logs using standart linux tools? Or how I can force mesos to write logs in single files?
Thanks!
Something like this can be implemented using logrotate with olddir parameter:
/var/log/mesos/*.*.* {
olddir /var/log/mesos/backup
maxsize 2000k
daily
maxage 7
copytruncate
postrotate
find /var/log/mesos /var/log/mesos/backup -mtime +7 -delete
endscript
}
It is necessary for folder from olddir parameter have been created before logrotate run.
You could try something like this in /var/log/mesos:
find /var/log/mesos -mtime +7 -name '*log.INFO.*' | xargs --no-run-if-empty rm
Adjust to match INFO|WARNING|ERROR, etc
When I try to do a manual source install of some software on NixOS 15.09, I get (sh -x is to get an exec log):
[nix-shell:/tmp/nix-shell-es/EventStore-oss-v3.5.0-src]$ sh -x scripts/build-js1/build-js1-linux.sh werror=no
...
+ CXXFLAGS=-fPIC
+ make x64.release werror=no
PYTHONPATH="/tmp/nix-shell-es/EventStore-oss-v3.5.0-src/scripts/build-js1/v8/tools/generate_shim_headers:" \
GYP_GENERATORS=make \
build/gyp/gyp --generator-output="out" build/all.gyp \
-Ibuild/standalone.gypi --depth=. \
-Dv8_target_arch=x64 \
-Dv8_optimized_debug=0 \
-S.x64.release -Dv8_enable_backtrace=1 -Dwerror='' -Darm_fpu=default -Darm_float_abi=default
/bin/sh: build/gyp/gyp: /bin/bash: bad interpreter: No such file or directory
Makefile:389: recipe for target 'out/Makefile.x64.release' failed
make: *** [out/Makefile.x64.release] Error 126
And indeed there is no bash program in /bin/
[nix-shell:/tmp/nix-shell-es/EventStore-oss-v3.5.0-src]$ ls -la /bin/
total 12
drwxr-xr-x 2 root root 4096 Feb 24 12:25 .
drwxr-xr-x 18 root root 4096 Feb 24 12:16 ..
lrwxrwxrwx 1 root root 63 Feb 24 12:25 sh -> /nix/store/l80ddf18bbig2icv6cmgjfws9a2vm3jj-bash-4.3-p42/bin/sh
However when I inspect the script I dont see any mention of it so it must be some subprogram looking for it.
As an aside, I imagine that's probably why the script wisely starts with #!/usr/bin/env bash and not the direct location of bash.
Anyone knows a blessed way to deal with this ?
one has to run patchShebangs ./scripts/build-js1/build-js1-linux.sh on files coming from source when they are not in the store and we want to execute them.
I'm trying to provide my docker container a volume of encrypted file system for internal use.
The idea is that the container will write to the volume as usual, but in fact the host will be encrypting the data before writing it to the filesystem.
I'm trying to use EncFS - it works well on the host, e.g:
encfs /encrypted /visible
I can write files to /visible, and those get encrypted.
However, when trying to run a container with /visible as the volume, e.g.:
docker run -i -t --privileged -v /visible:/myvolume imagename bash
I do get a volume in the container, but it's on the original /encrypted folder, not going through the EncFS. If I unmount the EncFS from /visible, I can see the files written by the container. Needless to say /encrypted is empty.
Is there a way to have docker mount the volume through EncFS, and not write directly to the folder?
In contrast, docker works fine when I use an NFS mount as a volume. It writes to the network device, and not to the local folder on which I mounted the device.
Thanks
I am unable to duplicate your problem locally. If I try to expose an encfs filesystem as a Docker volume, I get an error trying to start the container:
FATA[0003] Error response from daemon: Cannot start container <cid>:
setup mount namespace stat /visible: permission denied
So it's possible you have something different going on. In any case, this is what solved my problem:
By default, FUSE only permits the user who mounted a filesystem to have access to that filesystem. When you are running a Docker container, that container is initially running as root.
You can use the allow_root or allow_other mount options when you mount the FUSE filesystem. For example:
$ encfs -o allow_root /encrypted /other
Here, allow_root will permit the root user to have acces to the mountpoint, while allow_other will permit anyone to have access to the mountpoint (provided that the Unix permissions on the directory allow them access).
If I mounted by encfs filesytem using allow_root, I can then expose that filesystem as a Docker volume and the contents of that filesystem are correctly visible from inside the container.
This is definitely because you started the docker daemon before the host mounted the mountpoint. In this case the inode for the directory name is still pointing at the hosts local disk:
ls -i /mounts/
1048579 s3-data-mnt
then if you mount using a fuse daemon like s3fs:
/usr/local/bin/s3fs -o rw -o allow_other -o iam_role=ecsInstanceRole /mounts/s3-data-mnt
ls -i
1 s3-data-mnt
My guess is that docker does some bootstrap caching of the directory names to inodes (someone who has more knowledge of this than can fill in this blank).
Your comment is correct. If you simply restart docker after the mounting has finished your volume will be correctly shared from host to your containers. (Or you can simply delay starting docker until after all your mounts have finished mounting)
What is interesting (but makes complete since to me now) is that upon exiting the container and un-mounting the mountpoint on the host all of my writes from within the container to the shared volume magically appeared (they were being stored at the inode on the host machines local disk):
[root#host s3-data-mnt]# echo foo > bar
[root#host s3-data-mnt]# ls /mounts/s3-data-mnt
total 6
1 drwxrwxrwx 1 root root 0 Jan 1 1970 .
4 dr-xr-xr-x 28 root root 4096 Sep 16 17:06 ..
1 -rw-r--r-- 1 root root 4 Sep 16 17:11 bar
[root#host s3-data-mnt]# docker run -ti -v /mounts/s3-data-mnt:/s3-data busybox /bin/bash
root#5592454f9f4d:/mounts/s3-data# ls -als
total 8
4 drwxr-xr-x 3 root root 4096 Sep 16 16:05 .
4 drwxr-xr-x 12 root root 4096 Sep 16 16:45 ..
root#5592454f9f4d:/s3-data# echo baz > beef
root#5592454f9f4d:/s3-data# ls -als
total 9
4 drwxr-xr-x 3 root root 4096 Sep 16 16:05 .
4 drwxr-xr-x 12 root root 4096 Sep 16 16:45 ..
1 -rw-r--r-- 1 root root 4 Sep 16 17:11 beef
root#5592454f9f4d:/s3-data# exit
exit
[root#host s3-data-mnt]# ls /mounts/s3-data-mnt
total 6
1 drwxrwxrwx 1 root root 0 Jan 1 1970 .
4 dr-xr-xr-x 28 root root 4096 Sep 16 17:06 ..
1 -rw-r--r-- 1 root root 4 Sep 16 17:11 bar
[root#host /]# umount -l s3-data-mnt
[root#host /]# ls -als
[root#ip-10-0-3-233 /]# ls -als /s3-stn-jira-data-mnt/
total 8
4 drwxr-xr-x 2 root root 4096 Sep 16 17:28 .
4 dr-xr-xr-x 28 root root 4096 Sep 16 17:06 ..
1 -rw-r--r-- 1 root root 4 Sep 16 17:11 bar
You might be able to work around this by wrapping the mount call in nsenter to mount it in the same Linux mount namespace as the docker daemon, eg.
nsenter -t "$PID_OF_DOCKER_DAEMON" encfs ...
The question is whether this approach will survive a daemon restart itself. ;-)
Good day!
I have a linux sftp server located in VM. This VM has access to a GlusterFS storage, where sftp directories are located. Sftp works via OpenSSH server and chroots sftpusers group to sftp directories on GlusterFS storage. All worked well... After one moment I had got an issue...
Trying to create user:
# useradd -d /mnt/cluster-data/repositories/masters/test-user -G masters,sftpusers -m -s /bin/nologin test-user
Checking:
# cat /etc/passwd | grep test-user
test-user:x:1029:1032::/mnt/cluster-data/repositories/masters/test-user:/bin/nologin
# cat /etc/group | grep test-user
masters:x:1000:test-user
sftpusers:x:1005:test-user
test-user:x:1032:
Doing chown and chmod for home dir by hand:
# chown -R test-user:test-user /mnt/cluster-data/repositories/masters/test-user
# chmod -R 770 /mnt/cluster-data/repositories/masters/test-user
Checking:
# ls -la /mnt/cluster-data/repositories/masters/test-user
итого 16
drwxrwx--- 2 test-user test-user 4096 Окт 27 2013 .
drwxr-xr-x 13 root masters 4096 Окт 27 2013 ..
Adding another user to test-user's group:
# usermod -G test-user -a tarasov-af
# cat /etc/passwd | grep tarasov-af
tarasov-af:x:1028:1006::/mnt/cluster-data/repositories/lecturers/tarasov-af/:/bin/nologin
# cat /etc/group | grep tarasov-af
masters:x:1000:tarasov-af,test-user
sftpusers:x:1005:tarasov-af,test-user
lecturers:x:1006:tarasov-af
specialists:x:1008:tarasov-af
test-user:x:1032:tarasov-af
Login as tarasov-af:
sftp> cd masters/test-user
sftp> ls
remote readdir("/masters/test-user"): Permission denied
sftp> ls -la ..
drwxr-xr-x 13 0 1000 4096 Oct 26 21:30 .
drwxr-xr-x 6 0 0 4096 Oct 2 15:53 ..
drwxrwx--- 2 1029 1032 4096 Oct 26 21:53 test-user
I tried to login as tarasov-af into bash (usermod -s /bin/bash tarasov-af):
$ id
uid=1028 gid=1006
groups=1000,1005,1006,1008,1032
p.s. I guess this issue began after VM disk failed and I've got /etc/passwd and /etc/group broken, I've restored them from backups and all previous accounts works well, I have this issue only with new accounts.
I've found the reason of this issue: user tarasov-af has more than 16 secondary groups, first 15 groups work good, other -- don't work. I've set kernel.ngroups_max = 65535 in sysctl.conf on every computer in cluster (GlusterFS) and on sftp VM but nothing changed.
This issue goes to glusterfs client, it can't manipulate with more than 15 secondary groups.
# glusterfs --version
glusterfs 3.2.7 built on Sep 29 2013 03:28:05