Compressing a list of files contain spaces in their file-names - linux

I’ve been trying to compress a set of files contain spaces in thier file-names using a bash file. The bash file is:
#!/bin/bash
tar -cPf 'myconfigs.tar' `cat myconfigs.list`
The content of myconfigs.list file is:
/home/anas/.config/chromium/Default/Bookmarks
/home/anas/.config/chromium/Default/Login Data
/home/anas/.config/chromium/Default/Login Data-journal
The problem is that files contain spaces in their file-names don’t be included in the result TAR archive.
I tried '', "", %20... but didn’t work.
Thanks for your help in advance.

Anas,
I think you need to use:
tar -cPf myconfigs.tar -T myconfigs.list
instead of your "cat" . cat should work too if you properly escape the filenames inside, but -T is better.
UPDATED (to address your question in question's comments):
I cannot comment in your question (don't have enough reputation), so I decided to improve my answer instead.
The tilde (~) expansion is a shell thing, tar does not support it. However, to achieve what you want to achieve, you can use the following trick:
use relative paths in your myconfigs.list file (relative to the home directory):
.config/chromium/Default/Bookmarks
.config/chromium/Default/Login Data
.config/chromium/Default/Login Data-journal
run tar in such way that it changes directory to your home on startup:
tar -cPf myconfigs.tar -C ~/ -T myconfigs.list
OK, I went ahead and created a sample session illustrating it:
root#web:~ # useradd -m galaxy
root#web:~ # su - galaxy
galaxy#web:~ $ mkdir -p {1,2}/{3,4}/{5,6,7}
galaxy#web:~ $ find . -xdev -type d -exec touch '{}/file.txt' \;
galaxy#web:~ $ cat << EOF > include.lst
> 1/3
> 1/4/5/file.txt
> 1/4/7
> 2/file.txt
> EOF
galaxy#web:~ $ cd 2/3/6
galaxy#web:~/2/3/6 $ tar cjSpf ~/sample.tar.bz2 -C ~/ -T ~/include.lst
galaxy#web:~/2/3/6 $ cd
galaxy#web:~ $ tar tjvf sample.tar.bz2
drwx------ galaxy/galaxy 0 2014-02-19 04:10 1/3/
-rw------- galaxy/galaxy 0 2014-02-19 04:10 1/3/file.txt
drwx------ galaxy/galaxy 0 2014-02-19 04:10 1/3/7/
-rw------- galaxy/galaxy 0 2014-02-19 04:10 1/3/7/file.txt
drwx------ galaxy/galaxy 0 2014-02-19 04:10 1/3/6/
-rw------- galaxy/galaxy 0 2014-02-19 04:10 1/3/6/file.txt
drwx------ galaxy/galaxy 0 2014-02-19 04:10 1/3/5/
-rw------- galaxy/galaxy 0 2014-02-19 04:10 1/3/5/file.txt
-rw------- galaxy/galaxy 0 2014-02-19 04:10 1/4/5/file.txt
drwx------ galaxy/galaxy 0 2014-02-19 04:10 1/4/7/
-rw------- galaxy/galaxy 0 2014-02-19 04:10 1/4/7/file.txt
-rw------- galaxy/galaxy 0 2014-02-19 04:10 2/file.txt
galaxy#web:~ $
This should give you a start :)

Don't use cat's output like that otherwise space is considered a delimiter and a separete argument to the tar command.
You can use -T tar option:
tar -cPv -T myconfigs.list -f myconfigs.tar

Related

`ls -l` for all parent directories

I want to get a list of all directory permissions from current folder to /. For example, for the directory: /var/lib/program/subfolder, I want an output such as:
$ pwd
/var/lib/program/subfolder
$ magic_ls_-l_command somefile
drwxr-xr-x 10 root root 4096 May 15 20:20 var
drwxr-xr-x 10 root root 4096 May 15 20:20 lib
drwxrwxr-x 10 root user 4096 May 16 20:21 program
drwxrwxr-x 10 root user 4096 May 16 20:21 subfolder
-rwxrwxr-- 1 root user 4096 May 16 20:22 somefile
I don't care about the order (from /var to /subfolder or the other way around), the number of hard links or even the date. I just wrote them down to emulate the ls -l output. Also, I don't care how each filename in printed (/var and /lib, var and lib, or /var and /var/lib). I'm just interested in the ownership of each file/directory in the path from the choosen file or pwd to /.
In case I should install some program, I'm under Ubuntu 20.04.
This question has already been answered in superuser.com (I don't know if I can mark a question from one site as duplicate from another). The solution is as simple as writing (assuming I am in the same directory as the target filename):
$ namei -l $(pwd)/somefile ## or `namei -l $(realpath -s somefile)`
Because of -l, it lists basic permissions in long format for each parent directory.
I have to use pwd/realpath because namei doesn't resolve relative paths. If I'm not in the target directory, just write the full path.
I made this small script that does this. I use cd "$1"; pwd to get the current directory so that paths are not canonicalized (say, if you try magic-ls . and your current directory is /var/lib/postgres, but that is a symlink to /mnt/postgres, you will get /var, /var/lib and /var/lib/postgres, while using realpath you would get /mnt and /mnt/postgres)
magic-ls() {
local current=$(cd "$1"; pwd)
while [[ $current != '/' ]]; do
ls -ld "$current"
current=$(dirname "$current")
done
}
Here's an example output:
[leodag#desk ~]$ magic-ls
drwx------ 1 leodag leodag 2722 jun 21 13:49 /home/leodag
drwxr-xr-x 1 root root 18 mai 2 2019 /home
By the way it will also work with no argument since cd "" does not change your directory.
Edit: removed realpath from the while check, since that could lead to unexpected results if there was a link to / in the path, and was unneeded.
I wrote a bash script for you. It'll have some bugs, if you have space in names. If it bothers you, I'm happy for changes recommendations in the comments.
#!/bin/bash
if [ ! -z "$1" ] && [ -e "$1" ]
then
path=`realpath -s "$1"` # read argument as absolute path
else
path="$PWD" # No valid argument, so we take pwd
fi
paths=""
while [ "$path" != / ];do
paths+=" $path"
path=`dirname "$path"`
done
paths+=" $path" # Adding / to pathlist too
ls -ld $paths
With realpath -s you can catch the absolute path, but you wont follow the link. If no argument is given, we will use pwd as the file/directory to list.
We append each path to a list. This gives us the advantage of a better layout in the end, so that we get a nice table because we run ls only once.
Output:
bobafit:~$ magic_ls_-l_command /usr/bin/python3
drwxr-xr-x 21 root root 4096 Jun 20 10:07 /
drwxr-xr-x 14 root root 4096 Sep 5 2019 /usr
drwxr-xr-x 2 root root 110592 Jun 20 10:07 /usr/bin
lrwxrwxrwx 1 root root 9 Apr 7 12:43 /usr/bin/python3 -> python3.8
Just using parameter expansion:
#!/usr/bin/env bash
path="$1"
while test -n "$path"; do
ls -lLd "$path"
path="${path%/*}"
done
calling method :
bash test.sh /var/lib/program/subfolder/somefile
giving
-rw-r--r-- 1 root root 0 Jun 21 18:49 /var/lib/program/subfolder/somefile
drwxr-xr-x 1 root root 4096 Jun 21 18:49 /var/lib/program/subfolder
drwxr-xr-x 1 root root 4096 Jun 21 18:49 /var/lib/program
drwxr-xr-x 1 root root 4096 Jun 21 18:49 /var/lib
drwxr-xr-x 1 root root 4096 Jun 13 19:24 /var
#! /bin/bash
cur=""
IFS="/"
path=`pwd`
for dir in ${path:1}
do
cur=$cur/$dir
ls -lhd "$cur"
done
cur=$cur/$1
ls -lhd "$cur"
Terminal Session:
$ pwd
/tmp/dir_underscore/dir space/dir special #!)
$ ls
bash.sh test.txt
$ ./bash.sh test.txt
drwxrwxrwt 28 root root 36K Jun 21 22:45 /tmp
drwxr-xr-x 3 root root 4.0K Jun 21 22:27 /tmp/dir_underscore
drwxr-xr-x 3 root root 4.0K Jun 21 22:28 '/tmp/dir_underscore/dir space'
drwxr-xr-x 2 root root 4.0K Jun 21 22:54 '/tmp/dir_underscore/dir space/dir special #!)'
-rw-r--r-- 1 root root 0 Jun 21 22:29 '/tmp/dir_underscore/dir space/dir special #!)/test.txt'
This should possibly work:
pwd ; ls -lh ; while true ; do cd .. ; pwd ; ls -lh ; [[ "$PWD" == "/" ]] && break ; done
EDIT: I misunderstood the question at first. Try this:
(pwd ; ls -ldh ; while true ; do cd .. ; pwd ; ls -ldh ; [[ "$PWD" == "/" ]] &&
break ; done ; cd "$START")
EDIT2: fillipe's answer is probably the best, but here's my third and last attempt, which works on both files and directories:
magic_ls() {
fname="$1"
while true ; do
ls -lhd "$fname"
[[ "$fname" == "/" ]] && break ;
fname=$(dirname $(readlink -f "$fname"))
done
}
Just my 2 cents. My mac doesn't have the namei command (perhaps homebrew has a copy), but wanted to whip up a quick version that aligned the output in top-down order
#!/usr/bin/env bash
path="${1%/}"
DIRS=()
while test -n "$path"; do
DIRS=( "$path" "${DIRS[#]}" )
path="${path%/*}"
done
ls -ld "${DIRS[#]}"
Example output:
$ lspath $TMPDIR
lrwxr-xr-x# 1 root wheel 11 Oct 5 2018 /var -> private/var
drwxr-xr-x 7 root wheel 224 Jul 16 2020 /var/folders
drwxr-xr-x# 3 root wheel 96 Apr 5 2018 /var/folders/0c
drwxr-xr-x# 5 me staff 160 Apr 5 2018 /var/folders/0c/2_s_qxd11m3d1smzqdrs3qg40000gp
drwx------# 255 me staff 8160 Oct 7 09:18 /var/folders/0c/2_s_qxd11m3d1smzqdrs3qg40000gp/T

Path ambiguity through symbolic links

I have noticed a strange behavior in UNIX systems:
I'm standing in /noob/
I have a symbolic link to a folder (A# -> /B/C/D/A)
I enter the folder via my symlink (cd A)
pwd says /noob/A/
In /B/C/D/A/ i have a file abc which I can see now.
I want to copy it to /noob/
I type cp abc ..
I type cd ..
I end up in /noob/ which is empty - but the file ended up in /B/C/D/ ???
How come this ambiguity as to where cp and cd points when given .. as argument? I find it confusing. Can anyone explain it in terms I'll understand? (=simple)
All the best, and please forgive a UNIX-noob a stupid question. Lasse
First let's have a look at how cd command does behave by looking at the help menu. What we are looking for is option -L (the default behavior) and option -P
$ help cd cd: cd [-L|[-P [-e]] [-#]] [dir]
Change the shell working directory.
...
...
Options:
-L force symbolic links to be followed: resolve symbolic links in
DIR after processing instances of `..'
-P use the physical directory structure without following symbolic
links: resolve symbolic links in DIR before processing instances
of `..'
...
...
Important section
The default is to follow symbolic links, as if `-L' were specified.
`..' is processed by removing the immediately previous pathname component
back to a slash or the beginning of DIR.
Exit Status:
...
As you can see the default behavior of cd is not what you think it is since he will manipulate the $PWD variable accessed by pwd command in his own way, at each step you can run pwd command or do an echo $PWD to see how it reacts with the different cd commands hereunder.
Let's play with cd command:
We start from the following folder, with a sym link:
[/home/arobert/test/noob] >
ls -ltra
total 8
drwxrwxr-x 5 arobert arobert 4096 5月 11 09:48 ..
lrwxrwxrwx 1 arobert arobert 26 5月 11 09:48 A -> /home/arobert/link/B/C/D/A
drwxrwxr-x 2 arobert arobert 4096 5月 11 10:03 .
USAGE EXAMPLES:
[/home/arobert/test/noob] >
cd A
[/home/arobert/test/noob/A] >
cd ..
[/home/arobert/test/noob] >
cd -L A
[/home/arobert/test/noob/A] >
cd ..
[/home/arobert/test/noob] >
cd -P A
[/home/arobert/link/B/C/D/A] >
cd -P ..
[/home/arobert/link/B/C/D] >
cd /home/arobert/test/noob/
[/home/arobert/test/noob] >
cd A
[/home/arobert/test/noob/A] >
cd -P ..
[/home/arobert/link/B/C/D] >
Now let's play with readlink and cp command:
Let's say we have entered the symlink that points to A -> /home/arobert/link/B/C/D/A in which we have a file a
[/home/arobert/test/noob/A] >
ls -ltra
total 8
drwxrwxr-x 3 arobert arobert 4096 5月 11 09:55 ..
-rw-rw-r-- 1 arobert arobert 0 5月 11 10:10 a
drwxrwxr-x 2 arobert arobert 4096 5月 11 10:10 .
from this folder let's look at where does point . and .. by using readlink -f command:
[/home/arobert/test/noob/A] >
readlink -f .
/home/arobert/link/B/C/D/A
[/home/arobert/test/noob/A] >
readlink -f ..
/home/arobert/link/B/C/D
By consequence, when you run from the location /home/arobert/test/noob/A equivalent to /home/arobert/link/B/C/D/A the command cp a .. the file will be moved to /home/arobert/link/B/C/D as .. points to it.
What you can do now:
Use absolute path with your cp command to avoid bad surprise.
Call the command from /home/arobert/test/noob/ directory using
For example:
[/home/arobert/test/noob] >
cp A/a .
as readlink -f . points to the correct folder
[/home/arobert/test/noob] >
readlink -f .
/home/arobert/test/noob
Result:
[/home/arobert/test/noob] >
ls -ltra
total 8
drwxrwxr-x 5 arobert arobert 4096 5月 11 09:48 ..
lrwxrwxrwx 1 arobert arobert 26 5月 11 09:48 A -> /home/arobert/link/B/C/D/A
-rw-rw-r-- 1 arobert arobert 0 5月 11 10:13 a
drwxrwxr-x 2 arobert arobert 4096 5月 11 10:13 .

How to export directory created inside the Docker image to the host machine?

The program I'm running inside the Docker image, first creates a directory and writes some file into the directory.
To transfer the directory onto the host machine, I've mounted a datadir/ and then moved the directory created inside the image into the mounted directory, e.g.:
mkdir datadir
DATADIR=datadir/
docker run -i \
-v $(pwd)/$DATADIR:/$DATADIR/ ubuntu \
bash -c "mkdir /x1 && echo 'abc' > x1/test.txt && mv x1 $DATADIR"
But when I tried to access datadir/x1, it has root as the owner and it comes with read-only permissions:
$ mv datadir/x1/ .
mv: cannot move 'datadir/x1/' to './x1': Permission denied
$ ls -lah datadir/x1/
total 12K
drwxr-xr-x 2 root root 4.0K Jun 28 16:38 .
drwxrwxr-x 3 alvas alvas 4.0K Jun 28 16:38 ..
-rw-r--r-- 1 root root 4 Jun 28 16:38 test.txt
Is mounting the additional volume and copying the created directory inside the image the right approach to move files between the Docker image and the host machine? If not, what's the "canonical" way to perform the same operation?
About the directory permissions, what should be the correct way to assign the host machine permission to any files inside the mounted volume?
I've tried to chmod -R 777 inside the Docker image but I don't think that's the safe approach, i.e.:
$ docker run -i -v $(pwd)/$DATADIR:/$DATADIR/ -i ubuntu bash -c "mkdir /x1 && echo 'abc' > x1/test.txt && mv x1 $DATADIR && chmod -R 777 $DATADIR"
$ mv datadir/x1/ .
$ ls -lah x1
total 12K
drwxrwxrwx 2 root root 4.0K Jun 28 16:47 .
drwxrwxr-x 12 alvas alvas 4.0K Jun 28 16:47 ..
-rwxrwxrwx 1 root root 4 Jun 28 16:47 test.txt
To avoid permission issues use docker cp
For example:
# This is the directory you want to save the outputs
mkdir datadir
# We create a directory and file inside it, inside the Docker image.
# And we are naming the Docker image "thisinstance"
docker run -i --name thisinstance ubuntu \
bash -c "mkdir /x1 && echo 'abc' > x1/test.txt"
# Copies the new directory inside the Docker image to the host.
docker cp thisinstance:/x1 datadir/
# Destroy the temporary container
docker rm thisinstance
# Check the ownership of the directory and file
ls -lah datadir/x1/
[out]:
drwxr-xr-x 3 alvas 679754705 102B Jun 29 10:36 ./
drwxr-xr-x 3 alvas 679754705 102B Jun 29 10:36 ../
-rw-r--r-- 1 alvas 679754705 4B Jun 29 10:36 test.t

How to tar a branch of file tree?

I currently have some files and directories at this path:
/var/tmp/mydir/
I want to tar the whole path, excluding any other content in 'var' and 'tmp'.
Example:
$ ls /var
tmp
dir1 *(exclude)*
file1 *(exclude)*
$ ls /var/tmp
mydir
dir2 *(exclude)*
file2 *(exclude)*
$ ls /var/tmp/mydir
tarme1
tarme2
tarme3
In this case, I want to tar the directory tree /var/tmp/mydir and the content of 'mydir'.
Use tar -cf <archive_name>.tar /var/tmp/mydir which will give you what you need.
Use man tar to get more help (should be quite easy to understand).
If you want to modify your path some other way consider using -C switch. From man:
-C, --directory DIR
change to directory DIR
Do
tar -c --recursion --file backup.tar tmp/mydir
and
tar -tvf backup.tar
gives me :
drwxrwxr-x ssam/ssam 0 2016-05-02 12:02 tmp/mydir/
-rw-rw-r-- ssam/ssam 0 2016-05-02 12:02 tmp/mydir/tarme3
-rw-rw-r-- ssam/ssam 0 2016-05-02 12:02 tmp/mydir/tarme1
-rw-rw-r-- ssam/ssam 0 2016-05-02 12:02 tmp/mydir/tarme2
which is what you need. You can extract/restore it using
tar -xf backup.tar -C /var
Remember this will overwrite the files in mydir

realpath on Cygwin is not same as PWD

$ pwd
/home/Thomas/Utveckling/Cgreen/thoni56/cgreen
$ realpath .
/cygdrive/c/Users/Thomas/Utveckling/Cgreen/thoni56/cgreen
$ cygpath -a .
/cygdrive/c/Users/Thomas/Utveckling/Cgreen/thoni56/cgreen/
Why is this? Why is absolute path from the root /cygdrive and not /?
How do I get an absolute path that starts with /home from a relative path, such as .., if I'm in that tree?
It is the effect of the presence of a link in the path.
Eg:
ls -ld /pub /tmp
lrwxrwxrwx 1 marco Administrators 19 Jun 26 2014 /pub -> /cygdrive/e/cyg_pub
drwxrwxrwx+ 1 marco Administrators 0 May 2 17:49 /tmp
$ realpath /pub /tmp
/cygdrive/e/cyg_pub
/tmp

Resources