I am trying to create a script that foreach directoy in the folder folder, only the n most recent files are to be compressed.
However, I am having trouble with the multiple word files. I need a way to wrap them in quote marks so the tar command knows wich is each file.
Here is my script so far:
#!/bin/bash
if [ ! -d ~/backup ]; then
mkdir ~/backup
fi
cd ~/folder
for i in *; do
if [ -d "$i" ]; then
original=`pwd`
cd $i
echo tar zcf ~/backup/"$i".tar.gz "`ls -t | head -10`"
cd $original
fi
done
echo "Backup copied in $HOME/backup/"
exit 0
if [ ! -d ~/backup ]; then
mkdir ~/backup
fi
You can simplify by this :
[[ ! -d ~/backup ]] && mkdir ~/backup
Now to answer your question :
$ ls -t|head -10
file with spaces
file
test.txt
test
test.sh
$ lstFiles=""; while read; do lstFiles="$lstFiles \"$REPLY\""; done <<< "$(ls -t|head -10)"
$ echo $lstFiles
"file with spaces" "file" "test.txt" "test" "test.sh"
See how to read a command output or file content with a loop in Bash to read more details.
Several workarounds if you want to stick to one-liners - simplest is probably to use 'tr' and introduce wildcard for spaces:
echo tar zcf ~/backup/"$i".tar.gz "ls -t | head -10| tr ' ' '?'"
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 1_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 2_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 3_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 4_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 5_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 6_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 7_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 8_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 9_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 10_dummy.txt
-rw-rw-r-- 1 dale dale 35 Apr 6 09:11 test 11_dummy.txt
$ tar cvf TEST.tar $(ls -t | head -5 | tr ' ' '?')
test 11_dummy.txt
test 10_dummy.txt
test 9_dummy.txt
test 8_dummy.txt
test 7_dummy.txt
Another option might be to redirect to a file and then use '-T':
ls -t | head > /tmp/10tarfiles.txt
echo tar zcf ~/backup/"$i".tar.gz -T /tmp/10tarfiles.txt"
rm /tmp/10tarfiles.txt
Related
I am trying to create a bash command/script to remove all files in a directory older than X days that starts with a certain substring.
For example, if our directory contains the files
-rw-r--r-- 1 root root 0 Jun 30 10:22 foo_5
-rw-r--r-- 1 root root 0 Jun 29 10:22 bar_4
-rw-r--r-- 1 root root 0 Jun 29 10:22 foo_4
-rw-r--r-- 1 root root 0 Jun 28 10:22 bar_3
-rw-r--r-- 1 root root 0 Jun 28 10:22 foo_3
-rw-r--r-- 1 root root 0 Jun 27 10:22 bar_2
-rw-r--r-- 1 root root 0 Jun 27 10:22 foo_2
-rw-r--r-- 1 root root 0 Jun 26 10:22 foo_1
we want to delete all foo* files except the 2 most recent one. This will result in the directory
-rw-r--r-- 1 root root 0 Jun 30 10:22 foo_5
-rw-r--r-- 1 root root 0 Jun 29 10:22 bar_4
-rw-r--r-- 1 root root 0 Jun 29 10:22 foo_4
-rw-r--r-- 1 root root 0 Jun 28 10:22 bar_3
-rw-r--r-- 1 root root 0 Jun 27 10:22 bar_2
I am currently only able to delete all files except the 2 most recent, which will affect bar* files.
ls -t | tail -n +4 | xargs rm --
How can we also restrict our deletion to files that starts with a certain string?
Code to create test files
(
touch -d "6 days ago" foo_5
touch -d "7 days ago" foo_4
touch -d "7 days ago" bar_4
touch -d "8 days ago" foo_3
touch -d "8 days ago" bar_3
touch -d "9 days ago" foo_2
touch -d "9 days ago" bar_2
touch -d "10 days ago" foo_1
)
Parsing the output of ls is not a good idea. Using tools from GNU coreutils and findutils packages, a fail-safe program to achieve this task can be written as below.
n=2 # except the last two
find -maxdepth 1 -type f -name 'foo*' \
-printf '%T#\t%p\0' \
| sort -z -k 1n,1 \
| head -z -n -$n \
| cut -z -f 2- \
| xargs -0 rm
This is a job for stat
stat -c '%Y %n' foo* | sort -n | head -n -2 | cut -d " " -f 2- | xargs echo rm
rm foo_1 foo_2 foo_3
Remove "echo" if it is selecting the right files to delete.
Using perl and glob() (handle files with newlines or spaces as well) via only one process:
perl -e '
my #files = sort { -M $a <=> -M $b } grep -f, <./foo*>;
unlink #files[2..$#files]
'
I have a set of files including a date in their name:
MERRA2_400.tavg1_2d_slv_Nx.20151229.SUB.nc
MERRA2_400.tavg1_2d_slv_Nx.20151230.SUB.nc
MERRA2_400.tavg1_2d_slv_Nx.20151231.SUB.nc
I want to select the files matching a condition on this date. In this example: date > 20151230
I tried things like:
find . -regex ".*.SUB.nc" | cut -d "." -f 4 | while read a; do if [ $a -ge 20151201 ]; then echo $a; fi; done
BUT:
1) This is returning only a part of the filename, whereas I would like to return the entire filename.
2) There may be a more elegant way than using while read/do
thanks in advance!
Rearranging your code becomes:
#!/usr/bin/env bash
find . -regex ".*.SUB.nc" \
| rev | cut -d '.' -f 3 | rev \
| while read a; do
if [ $a -ge 20151201 ]; then
echo $a
fi
done
rev | cut -d '.' -f 3 | rev is used because
if you give absolute path or
the subdirectories have . in them
then it won't be the 4th field, but it will always be the 3rd last field.
This will give the output:
20151231
20151229
20151230
To show the complete file names change echo $a with ls *$a*. Output:
MERRA2_400.tavg1_2d_slv_Nx.20151231.SUB.nc
MERRA2_400.tavg1_2d_slv_Nx.20151229.SUB.nc
MERRA2_400.tavg1_2d_slv_Nx.20151230.SUB.nc
I tested this script with file names whose dates are less than 20151201. For example MERRA2_400.tavg1_2d_slv_Nx.20151200.SUB.nc. The results are consistent.
Perhaps a more efficient way to accomplish your task is using a grep regex like:
find . -regex ".*.SUB.nc" | grep -E "201512(0[1-9]|[1-9][0-9])|201[6-9][0-9][0-9][0-9]"
This will work just fine.
find . -regex ".*.SUB.nc" | rev | cut -d '.' -f 3 | rev | while read a; do if [ $a -ge 20151201 ]; then echo `ls -R | grep $a` ;fi ;done
rev | cut -d '.' -f 3 | rev is used because
if you give absolute path or
the subdirectories have . in them
then it won't be the 4th field now, but it will always be the 3rd last field always.
ls -R | grep $a so that you can recursively find out the name of the file.
Assume is the files and file structure is :
[root#localhost temp]# ls -lrt -R
.:
total 8
-rw-r--r--. 1 root root 0 Apr 25 16:15 MERRA2_400.tavg1_2d_slv_Nx.20151231.SUB.nc
-rw-r--r--. 1 root root 0 Apr 25 16:15 MERRA2_400.tavg1_2d_slv_Nx.20151230.SUB.nc
-rw-r--r--. 1 root root 0 Apr 25 16:15 MERRA2_400.tavg1_2d_slv_Nx.20151229.SUB.nc
drwxr-xr-x. 2 root root 4096 Apr 25 16:32 temp.3
drwxr-xr-x. 3 root root 4096 Apr 25 17:13 temp2
./temp.3:
total 0
./temp2:
total 4
-rw-r--r--. 1 root root 0 Apr 25 16:27 MERRA2_400.tavg1_2d_slv_Nx.20151111.SUB.nc
-rw-r--r--. 1 root root 0 Apr 25 16:27 MERRA2_400.tavg1_2d_slv_Nx.20151222.SUB.nc
drwxr-xr-x. 2 root root 4096 Apr 25 17:13 temp21
./temp2/temp21:
total 0
-rw-r--r--. 1 root root 0 Apr 25 17:13 MERRA2_400.tavg1_2d_slv_Nx.20151333.SUB.nc
Running above command gives :
MERRA2_400.tavg1_2d_slv_Nx.20151229.SUB.nc
MERRA2_400.tavg1_2d_slv_Nx.20151231.SUB.nc
MERRA2_400.tavg1_2d_slv_Nx.20151230.SUB.nc
MERRA2_400.tavg1_2d_slv_Nx.20151333.SUB.nc
MERRA2_400.tavg1_2d_slv_Nx.20151222.SUB.nc
I have a bunch of MP4 files that look like this:
-rw-rw-r-- 1 116M Apr 19 06:08 lULIqx9Akn4.mp4
These are youtube videos. When I try to do anything with all of them, I get a weird error. Every command I try says that I'm using invalid options (that I am not using). Here are some examples.
$ ls *.mp4
/bin/ls: invalid option -- '7'
Try '/bin/ls --help' for more information.
$ mv *.mp4 videos/
mv: invalid option -- 'L'
Try 'mv --help' for more information.
$ cp *.mp4 videos/.
cp: invalid option -- '7'
Try 'cp --help' for more information.
It doesn't do the same thing with a different extension (*.mp3, *.txt, *.sh).
What's going on? How do I fix this?
I used this as a cheap workaround,
find . -name "*.mp4" -exec mv {} videos/. \;
but I want to understand what's happening, not just get the job done.
One of your filenames starts with a hyphen, e.g,. -7 or -L. Try ls -- *.mp4 or cp -- *.mp4 videos. Also, allow me to suggest UNIX and Linux Stack Exchange for shell questions :) .
Solution:
Either move the files,
mv -- *.mp4 ./videos
or rename the files in situ...
for file in -*.mp4; do mv -- "$file" "${file:1}"; done
Explanation:
My sense is you have a file with a leading - in the directly... most commands stop you creating such files but if you copy them from another operating system it can occur. Thus, you need to rename any files with a leading - in their filename...
Let me explain with an example...
Let's try to create a file with a leading -:
touch "-7ULIqx9Akn4.mp4"
touch: illegal option -- 7
we can get around this as follows:
>touch -- "-7ULIqx9Akn4.mp4"
> ls -al -- -*.mp4
total 0
-rw-r--r--# 1 n staff 0 Apr 29 13:02 -7ULIqx9Akn4.mp4
ok, now lets set up an example and demonstrate a solution...
> ls -la
total 0
-rw-r--r--# 1 n staff 0 Apr 29 12:49 -75438752.mp4
-rw-r--r--# 1 n staff 0 Apr 29 12:49 -85438750.mp4
drwxr-xr-x# 7 n staff 238 Apr 29 12:49 .
drwxr-xr-x# 6 n staff 204 Apr 29 11:18 ..
-rw-r--r--# 1 n staff 0 Apr 29 12:36 75438750.mp4
-rw-r--r--# 1 n staff 0 Apr 29 12:33 7ULIqx9Akn4.mp4
-rw-rw-r--# 1 n staff 0 Apr 29 11:19 lULIqx9Akn4.mp4
next:
ls -- -*.mp4
-75438752.mp4 -85438750.mp4
ok, lets now rename these files...
A little explanation here, the following command uses mv to remove the leading character. i.e. Find files with a leading - and remove the leading character {$file:1} from the filename...
for file in -*.mp4; do mv -- "$file" "${file:1}"; done
Result:
> for file in -*.mp4; do mv -- "$file" "${file:1}"; done
> ll
total 0
drwxr-xr-x# 7 n 238 Apr 29 12:52 ./
drwxr-xr-x# 6 n 204 Apr 29 11:18 ../
-rw-r--r--# 1 n 0 Apr 29 12:36 75438750.mp4
-rw-r--r--# 1 n 0 Apr 29 12:49 75438752.mp4
-rw-r--r--# 1 n 0 Apr 29 12:33 7ULIqx9Akn4.mp4
-rw-r--r--# 1 n 0 Apr 29 12:49 85438750.mp4
-rw-rw-r--# 1 n 0 Apr 29 11:19 lULIqx9Akn4.mp4
Note
The above does not account for duplicate file names...
I am new in shell script.Will you please suggest how to write backup shell script. I am having following formated data in target directory.
StoreID_date_time.zip
Like:
-rw------- 1 rupesh ldapusers 8267310 Mar 22 12:00 44_22032014_115629.zip
-rw------- 1 rupesh ldapusers 8269938 Mar 22 12:07 44_22032014_120013.zip
-rw------- 1 rupesh ldapusers 8267110 Mar 22 12:14 44_22032014_120704.zip
-rw------- 1 rupesh ldapusers 8254223 Mar 22 14:25 45_22032014_142155.zip
-rw------- 1 rupesh ldapusers 7871060 Mar 22 12:11 48_22032014_120813.zip
-rw------- 1 rupesh ldapusers 8314418 Mar 22 12:22 48_22032014_121038.zip
-rw------- 1 rupesh ldapusers 8254699 Mar 24 12:13 49_22032014_145338.zip
Now I want to backup files with following way:
Backup directory : /backup/date/storeid/zip files of that store
like:
/backup/22032014/44/44_22032014_115629.zip,44_22032014_120013.zip...so on
/backup/22032014/45/45_22032014_142155.zip
/backup/22032014/48/48_22032014_120813.zip,48_22032014_121038.zip
/backup/22032014/49/49_22032014_145338.zip
for next day /backup/23032014/respective_storeIDfolder&files
Please give some hint or code example so I can move foreword.
I have coded in bare minimum steps without doing a real check but verified it. It works fine with some dummy files I created on my box :)
#!/bin/bash
for i in $(find * -type f -iname '*.zip' )
do
echo "Zip file : "$i
store_id=$(echo $i | cut -d "_" -f 1 );
timestamp=$(echo $i | cut -d "_" -f 2 );
echo Store id = ${store_id}
# I am assuming all these directories here will be of teh same pattern name. Else put a numeric check down.
mkdir -p /backup/${timestamp}/${store_id}
cp -f $i /backup/${timestamp}/${store_id}/
done;
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
I have two files in my home folder named "'?" and "'?;?" (without the double quotes). How can I delete them? I've tried to use escape, but it doesn't work.
Use single or double quotes to avoid wildcard expansion. A ? is a wildcard which indicates to the shell to match with any one single character. By placing it in quotes you are telling the shell not to perform wildcard expansion.
rm '?' '?;?'
rm "?" "?;?"
This will remove the two files named "?" and "?;?"
You can also use a backslash to quote the individual characters that have special meaning to the shell, so you could do this
rm \? \?\;\?
Notice you have to quote the '?' to prevent pathname expansion and you have to quote the ';' so the shell doesn't interpret that as separating commands.
If you leave out the quotes, then the shell parses it differently. Here's an experiment I ran.
$ for i in {1..4}; do for j in {a..c}; do touch "$i;$j" $j '?' '?;?';done;done
$ ls
1;a 1;b 1;c 2;a 2;b 2;c 3;a 3;b 3;c 4;a 4;b 4;c ? ?;? a b c
$ rm ? ?;?
rm: cannot remove `?': No such file or directory
rm: cannot remove `a': No such file or directory
rm: cannot remove `b': No such file or directory
rm: cannot remove `c': No such file or directory
bash: ?: command not found
$ rm `echo "?" "?;?"`
rm: cannot remove `?': No such file or directory
$
What happened here is the shell did pathname expansion, so
rm ? ?;?
became
rm ? a b c ? a b c;? a b c
The rm command removed files a b c ? then complained that the following files were not found (they had already been deleted). The semicolon separated commands, so it then tried to invoke the '?' command passing arguments "a" "b" "c" ... but there is no '?' command - the file named '?' had just been deleted, and it wasn't executable anyway - so the shell complains that the "?" command is not found.
If you want to remove all files matching "?" and "?;?" you need to trick the shell into expanding those, which I did like this
rm `echo "?" "?;?"`
This was expanded by the shell in two steps, first it runs echo "?" "?;?" which results in two strings, "?" and "?;?", then it does pathname expansion using those strings to produce the arguments for rm, which results in
rm ? 1;a 1;b 1;c 2;a 2;b 2;c 3;a 3;b 3;c 4;a 4;b 4;c ?;?
Notice that the wildcard expansion for '?' didn't produce any matching files this time (they had already been previously deleted), so the shell passes '?' as an argument to rm, which successfully removes all files passed as arguments except for '?' so it complains about that.
Here's another experiment
$ for i in {1..4}; do for j in {a..c}; do touch "$i;$j" $j '?' '?;?';done;done
$ ls
1;a 1;b 1;c 2;a 2;b 2;c 3;a 3;b 3;c 4;a 4;b 4;c ? ?;? a b c
$ rm "?" "?;?"
$ ls
1;a 1;b 1;c 2;a 2;b 2;c 3;a 3;b 3;c 4;a 4;b 4;c a b c
$ rm `echo "?" "?;?"`
$ ls
$
For more information consult the man page on globbing
man 7 glob
Wildcard Matching
A string is a wildcard pattern if it contains one of the characters '?', '*' or '['. Globbing is the operation that
expands a wildcard pattern into the list of pathnames matching the pattern. Matching is defined by:
A '?' (not between brackets) matches any single character.
A '*' (not between brackets) matches any string, including the empty string.
Note that ls can report a question mark for arbitrary non-printable characters, so there's a chance that what you've got as a file name does not contain a question mark.
You can spot this with the ls -b command, or with ls | cat.
As a convoluted example, complete with remedy, I created a script convolvulus like this:
set -x
mkdir convoluted &&
(
cd convoluted
cp /dev/null "$(ls -la | sed 1d)"
ls
ls -b
ls | cat
ls -la | cat
cp /dev/null $'\n'
cp /dev/null $'\n;\n'
ls -als | cat
ls -lab
ls
ls | cat
rm $'\n' $'\n;\n' d*
ls -a
)
rm -fr convoluted
When run, it yielded:
$ bash convolvulus 2>&1 | so
+ mkdir convoluted
+ cd convoluted
++ ls -la
++ sed 1d
+ cp /dev/null 'drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..'
+ ls
drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
+ ls -b
drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .\ndrwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
+ ls
+ cat
drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
+ ls -la
+ cat
total 0
drwxr-xr-x 3 jleffler staff 102 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
-rw-r--r-- 1 jleffler staff 0 Mar 9 11:58 drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
+ cp /dev/null '
'
+ cp /dev/null '
;
'
+ ls -als
+ cat
total 0
0 -rw-r--r-- 1 jleffler staff 0 Mar 9 11:58
0 -rw-r--r-- 1 jleffler staff 0 Mar 9 11:58
;
0 drwxr-xr-x 5 jleffler staff 170 Mar 9 11:58 .
0 drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
0 -rw-r--r-- 1 jleffler staff 0 Mar 9 11:58 drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
+ ls -lab
total 0
-rw-r--r-- 1 jleffler staff 0 Mar 9 11:58 \n
-rw-r--r-- 1 jleffler staff 0 Mar 9 11:58 \n;\n
drwxr-xr-x 5 jleffler staff 170 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
-rw-r--r-- 1 jleffler staff 0 Mar 9 11:58 drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .\ndrwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
+ ls
;
drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
+ ls
+ cat
;
drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..
+ rm '
' '
;
' 'drwxr-xr-x 2 jleffler staff 68 Mar 9 11:58 .
drwxr-xr-x 240 jleffler staff 8160 Mar 9 11:58 ..'
+ ls -a
.
..
+ rm -fr convoluted
$
Have fun!
The -b option to ls works for GNU ls and for Mac OS X and BSD ls (but is not defined by POSIX).
The '$'\n' notation is Bash ANSI-C Quoting.
Use quotes around the file names:
$ ls
? ?;?
$ rm '?'
$ ls
?;?
$ rm "?;?"
$ ls
$