Spaces in directory name Bash - linux

I'm new to bash and I'm working on script that traverses the tar.gz file archive and in each file changes a string specified to an another string. Args of script: name of archive, searched string, target word.
My problem is that when archive name contains a space (e.g. I run script with following args: > change_strings.sh "/tmp/tmp.m7xYn5EQ2y/work/data txt" a A) I have following error:
on line if [ ! -f $filename ] ; then [: data: binary operator expected, dirname: extra operand `txt'.
Here is my code:
#!/bin/bash
filename="${1##*/}"
VAR="$1"
DIR=$(dirname ${VAR})
cd "$DIR"
if [ ! -f $filename ] ; then
echo "no such archive" >&2
exit 1
fi
if ! tar tf $filename &> /dev/null; then
echo "this is not .tar.gz archive" >&2
exit 1
fi
dir=`mktemp -dt 'test.XXXXXX'`
tar -xf $filename -C $dir #extract archive to dir
cd $dir #go to argument directory
FILES=$dir"/*"
for f in $FILES
do
sed -i "s/$2/$3/g" "$f"
done
tar -czf $filename * #create tar gz archive with files in current directory
mv -f $filename $cdir"/"$filename #move archive
rm -r $dir #remove tmp directory

The proper way to handle this is to surround your variables with double quotes.
var=/foo/bar baz
CMD $var # CMD /foo/bar baz
The above code will execute CMD on /foo/bar and baz
CMD "$var"
This will execute CMD on "/foo/bar baz". It is a best practice to always surround your variables with double quotes in most places.

Welcome to stackoverflow!
For the convenience of current and future readers, here's a small, self contained example showing the problem:
filename="my file.txt"
if [ ! -f $filename ]
then
echo "file does not exist"
fi
Here's the output we get:
$ bash file
file: line 2: [: my: binary operator expected
And here's the output we expected to get:
file does not exist
Why are they not the same?
Here's what shellcheck has to say about it:
$ shellcheck file
In file line 2:
if [ -f $filename ]
^-- SC2086: Double quote to prevent globbing and word splitting.
and indeed, if we double quote it, we get the expected output:
$ cat file
filename="my file.txt"
if [ ! -f "$filename" ]
then
echo "file does not exist"
fi
$ bash file
file does not exist
You should be double quoting all your variables.
However, you have to take care with $FILES because it contains a glob/wildcards that you want to expand along with potential spaces that you don't want to wordsplit on. The easiest way is to just not put it in a variable and instead write it out:
for f in "$dir"/*
do
...

Related

Linux Script; For Loop to rename; New to Scripting

You will have to forgive me I have very little experience writing Linux Scripts. Ok What I'm trying to do is rename part of a file that has a specified name in, but the problem I'm coming across is I get the error during my For Loop is this 0403-011 The specified substitution is not valid for this command I'm not sure what I'm doing wrong in my for loop, can someone please assist?
#Creates Directory
echo "Name of New Directory"
read newdir
if [[ -n "$newdir" ]]
then
mkdir $newdir
fi
echo $userInput Directory Created
echo
echo "Directory you wish to Copy?"
read copydir
if [[ -n "$copydir" ]]
then
#Copies contents of Specified Directory
cp -R $copydir/!(*.UNC) $newdir;
#Searches through directory
for file in $newdir/$copydir*; do
mv -v -- "$file" "${file/old/new}";
done
fi
Which version of ksh are you using?
"${file//old/new}" and "${file/old/new}" are valid syntaxes in ksh93.
If your env is ksh88 "${file//old/new}" substitution is not supported.
You have to use sed/tr to replace pattern. Here is an example with sed.
mv -v -- "$file" "$(echo ${file}|sed 's/old/new/')"
The offending line:
mv -v -- "$file" "${file/old/new}";
should be:
mv -v -- "$file" "${file//old/new}";
If you want to replace $old with $new (as opposed to the literal string "old" with "new"), write:
mv -v -- "$file" "${file//$old/$new}";

Check that two file exists in UNIX Directory

Good Morning,
I am trying to write a korn shell script to look inside a directory that contains loads of files and check that each file also exists with .orig on the end.
For example if a file inside the directory is called 'mercury_1' there must also be a file called 'mercury_1.orig'
If there isn't, it needs to move the mercury_1 file to another location. However if the .orig file exists do nothing and move onto the next file.
I am sure it is really simple but I am not that experienced in writing Linux scripts and help would be greatly appreciated!!
Here's a small ksh snippet to check if a file exists in the current directory
fname=mercury_1
if [ -f $fname ]
then
echo "file exists"
else
echo "file doesn't exit"
fi
Edit:
The updated script that does the said functionality
#/usr/bin/ksh
if [ ! $# -eq 1 ]
then
echo "provide dir"
exit
fi
dir=$1
cd $dir
#process file names not ending with orig
for fname in `ls | grep -v ".orig$"`
do
echo processing file $fname
if [ -d $fname ] #skip directory
then
continue
fi
if [ -f "$fname.orig" ] #if equiv. orig file present
then
echo "file exist"
continue
else
echo "moving"
mv $fname /tmp
fi
done
Hope its of help!
You can use the below script
script.sh :
#!/bin/sh
if [ ! $# -eq 2 ]; then
echo "error";
exit;
fi
for File in $1/*
do
Tfile=${File%%.*}
if [ ! -f $Tfile.orig ]; then
echo "$File"
mv $File $2/
fi
done
Usage:
./script.sh <search directory> <destination dir if file not present>
Here, for each file with extension stripped check if "*.orig" is present, if not then move file to different directory, else do nothing.
Extension is stripped because you don't want to repeat the same steps for *.orig files.
I tested this on OSX (basically mv should not differ to much from linux). My test directory is zbar and destination is /tmp directory
#!/bin/bash
FILES=zbar
cd $FILES
array=$(ls -p |grep -v "/") # we search for file without extension so put them in array and ignore directory
echo $array
for f in $array #loop in array and find .orig file
do
#echo $f
if [ -e "$f.orig" ]
then
echo "found $f.orig"
else
mv -f "$f" "/tmp"
fi
done

linux zip and exclude dir via bash/shell script

I am trying to write a bash/shell script to zip up a specific folder and ignore certain sub-dirs in that folder.
This is the folder I am trying to zip "sync_test5":
My bash script generates an ignore list (based on) and calls the zip function like this:
#!/bin/bash
SYNC_WEB_ROOT_BASE_DIR="/home/www-data/public_html"
SYNC_WEB_ROOT_BACKUP_DIR="sync_test5"
SYNC_WEB_ROOT_IGNORE_DIR="dir_to_ignore dir2_to_ignore"
ignorelist=""
if [ "$SYNC_WEB_ROOT_IGNORE_DIR" != "" ];
then
for ignoredir in $SYNC_WEB_ROOT_IGNORE_DIR
do
ignorelist="$ignorelist $SYNC_WEB_ROOT_BACKUP_DIR/$ignoredir/**\*"
done
fi
FILE="$SYNC_BACKUP_DIR/$DATETIMENOW.website.zip"
cd $SYNC_WEB_ROOT_BASE_DIR;
zip -r $FILE $SYNC_WEB_ROOT_BACKUP_DIR -x $ignorelist >/dev/null
echo "Done"
Now this script runs without error, however it is not ignoring/excluding the dirs I've specified.
So, I had the shell script output the command it tried to run, which was:
zip -r 12-08-2014_072810.website.zip sync_test5 -x sync_test5/dir_to_ignore/**\* sync_test5/dir2_to_ignore/**\*
Now If I run the above command directly in putty like this, it works:
So, why doesn't my shell script exclude working as intended? the command that is being executed is identical (in shell and putty directly).
Because backslash quotings in a variable after word splitting are not evaluated.
If you have a='123\4', echo $a would give
123\4
But if you do it directly like echo 123\4, you'd get
1234
Clearly the arguments you pass with the variable and without the variables are different.
You probably just meant to not quote your argument with backslash:
ignorelist="$ignorelist $SYNC_WEB_ROOT_BACKUP_DIR/$ignoredir/***"
Btw, what actual works is a non-evaluated glob pattern:
zip -r 12-08-2014_072810.website.zip sync_test5 -x 'sync_test5/dir_to_ignore/***' 'sync_test5/dir2_to_ignore/***'
You can verify this with
echo zip -r 12-08-2014_072810.website.zip sync_test5 -x sync_test5/dir_to_ignore/**\* sync_test5/dir2_to_ignore/**\*
And this is my suggestion:
#!/bin/bash
SYNC_WEB_ROOT_BASE_DIR="/home/www-data/public_html"
SYNC_WEB_ROOT_BACKUP_DIR="sync_test5"
SYNC_WEB_ROOT_IGNORE_DIR=("dir_to_ignore" "dir2_to_ignore")
IGNORE_LIST=()
if [[ -n $SYNC_WEB_ROOT_IGNORE_DIR ]]; then
for IGNORE_DIR in "${SYNC_WEB_ROOT_IGNORE_DIR[#]}"; do
IGNORE_LIST+=("$SYNC_WEB_ROOT_BACKUP_DIR/$IGNORE_DIR/***") ## "$SYNC_WEB_ROOT_BACKUP_DIR/$IGNORE_DIR/*" perhaps is enough?
done
fi
FILE="$SYNC_BACKUP_DIR/$DATETIMENOW.website.zip" ## Where is $SYNC_BACKUP_DIR set?
cd "$SYNC_WEB_ROOT_BASE_DIR";
zip -r "$FILE" "$SYNC_WEB_ROOT_BACKUP_DIR" -x "${IGNORE_LIST[#]}" >/dev/null
echo "Done"
This is what I ended up with:
#!/bin/bash
# This script zips a directory, excluding specified files, types and subdirectories.
# while zipping the directory it excludes hidden directories and certain file types
[[ "`/usr/bin/tty`" == "not a tty" ]] && . ~/.bash_profile
DIRECTORY=$(cd `dirname $0` && pwd)
if [[ -z $1 ]]; then
echo "Usage: managed_directory_compressor /your-directory/ zip-file-name"
else
DIRECTORY_TO_COMPRESS=${1%/}
ZIPPED_FILE="$2.zip"
COMPRESS_IGNORE_FILE=("\.git" "*.zip" "*.csv" "*.json" "gulpfile.js" "*.rb" "*.bak" "*.swp" "*.back" "*.merge" "*.txt" "*.sh" "bower_components" "node_modules")
COMPRESS_IGNORE_DIR=("bower_components" "node_modules")
IGNORE_LIST=("*/\.*" "\.* "\/\.*"")
if [[ -n $COMPRESS_IGNORE_FILE ]]; then
for IGNORE_FILES in "${COMPRESS_IGNORE_FILE[#]}"; do
IGNORE_LIST+=("$DIRECTORY_TO_COMPRESS/$IGNORE_FILES/*")
done
for IGNORE_DIR in "${COMPRESS_IGNORE_DIR[#]}"; do
IGNORE_LIST+=("$DIRECTORY_TO_COMPRESS/$IGNORE_DIR/")
done
fi
zip -r "$ZIPPED_FILE" "$DIRECTORY_TO_COMPRESS" -x "${IGNORE_LIST[#]}" # >/dev/null
# echo zip -r "$ZIPPED_FILE" "$DIRECTORY_TO_COMPRESS" -x "${IGNORE_LIST[#]}" # >/dev/null
echo $DIRECTORY_TO_COMPRESS "compressed as" $ZIPPED_FILE.
fi
After a few trial and error, I have managed to fix this problem by changing this line:
ignorelist="$ignorelist $SYNC_WEB_ROOT_BACKUP_DIR/$ignoredir/**\*"
to:
ignorelist="$ignorelist $SYNC_WEB_ROOT_BACKUP_DIR/$ignoredir/***"
Not sure why this worked, but it does :)

Editing every file in a directory after opening it bash

Looking around I didn't see exactly what I was looking for. Some similar stuff, but for some reason what I tried so far hasn't worked.
My main goals:
run script in my current directory
open the picture to see what it is
rename the picture i just viewed
repeat the process without running the script again
These were the sources I attempted to follow:
Bash Shell Loop Over Set of Files
Bash loop through directory and rename every file
How to do something to every file in a directory using bash?
==================================================================================
echo "Rename pictures. Path"
read path
for f in $path
do
eog $path
echo "new name"
read newname
mv $path $newname
cat $f
done
You should pass the script an argument rather than trying to make it interactive. You also have numerous quoting problems. Try something like this instead (untested):
#!/usr/bin/env bash
moveFile() {
local newName=
until [[ $newName ]]; do
printf '%s ' 'new name:'
read -er newName # -e implies Bash with readline
echo
done
mv -i "$1" "${1%/*}/${newName}"
}
if [[ ! -d $1 ]]; then
echo 'Must specify a path' >&2
exit 1
fi
for f in "$1"/*; do
eog "$f"
moveFile "$f"
done
You might want to try something like this:
for f in $*; do
eog $f
echo "new name:"
read newname
mv $f $newname
done
If you name the script, say, rename.sh, you can call
./rename.sh *gif
to review all files with extention 'gif'.
Using find command allows you to search for image files in the specified directory recursively.
echo -n "Rename pictures. Input image directory: "
read path
for f in `find $path -type f`
do
eog $f
echo -n "Enter new name: "
read newname
mv $f $newname
echo "Renamed $f to $newname."
done

bash tar error doesn't create tar.gz

I have the following bash script:
#DIR is something like: /home/foo/foobar/test/ without any whitespace but can also include whitespace
DIR="$( cd "$( dirname "$0" )" && pwd )"
#backup_name is read from a file
backup_name=FOOBAR
date=`date +%Y%m%d_%H%M_%S`
#subdirs is also read from the same file
subdirs=etc/ sbin/ bin/
filename="$DIR/Backup_$backup_name"_"$date.tar.gz"
cd /
echo "filename: $filename"
echo "subdirs $subdirs"
cmd='tar czvf "'$filename'" '$subdirs
echo "cmd tar: $cmd"
$cmd
But I get following output:
filename: /home/foo/foobar/test/Backup_FOOBAR_20120322_1529_35.tar.gz
subdirs: etc/ sbin/ bin/
cmd tar: tar cfvz "/home/foo/foobar/test/Backup_FOOBAR_20120322_1529_35.tar.gz" etc/ sbin/ bin/
etc/
# ... list of files in etc
# but no files from sbin or bin directory
tar: "/home/foo/foobar/test/Backup_FOOBAR_20120322_1529_35.tar.gz": can open not execute: File or directory not found
tar: not recoverable error: abortion.
However, when I copy the echo output of the tar command, make a cd to / and paste it into the bash shell it is working:
tar cfvz "/home/foo/foobar/test/Backup_FOOBAR_20120322_1529_35.tar.gz" etc/ sbin/ bin/
etc/
Every variable is defined and there is no trailing newline
I also tried $cmd with backticks
the two variables: backup_name and subdirs are read from a file (I did not include the reading process in the code)
edit: I just copied my script to a dir with no whitespace and changed the line:
cmd='tar czvf "'$filename'" '$subdirs
#to
cmd="tar czvf $filename $subdirs"
and it's working now but when I do the same in a dir which also contents whitespaces I get still the same error.
edit2: reading from file (the file is read before anything else happens)
config="config.txt"
local line
while read line
do
#points to next free element and declares it
config_lines[${#config_lines[#]}]=$line
done <$config
backup_name=${config_line[0]}
subdirs=${config_line[1]}
What is wrong with my bash script?
Short answer: see BashFAQ #050: I'm trying to put a command in a variable, but the complex cases always fail!.
Long answer: embedding quotes in a variable doesn't do anything useful, because when you use it (i.e. $cmd), bash parses quotes before replacing variables; by the time the quotes are there, it's too late for them to do any good. You do, however, have several options:
Don't bother with putting the command in a variable in the first place, just use it directly:
echo "filename: $filename"
echo "subdirs $subdirs"
tar czvf "$filename" $subdirs
If you really need to put it in a variable first, use an array rather than a plain text variable (and ideally, do the same with the subdirs list):
subdirs=(etc/ sbin/ bin/)
...
echo "filename: $filename"
echo "subdirs ${subdirs[*]}"
cmd=(tar czvf "$filename" "${subdirs[#]}")
printf "cmd tar:"
printf " %q" "${cmd[#]}" # Have to do some trickery to get it printed right
printf "\n"
"${cmd[#]}"
Instead of mucking about with messy quoting issues you could get the results you want a different way and, perhaps, save some time. How about something like this?
#!/usr/bin/env bash
# abusing set -v for fun and profit
tar_output=/tmp/$$.tarout
tar_command=/tmp/$$.tarcmd
tmp_script=/tmp/$$.script
dir="$(cd "$(dirname "$0")"; pwd)"
cat>"${tmp_script}"<<-'END'
datestamp=$(date +%Y%m%d_%H%M_%S)
subdirs=(etc sbin bin)
backup_name=FOOBAR
filename="$1/Backup_${backup_name}_${date}.tar.gz"
printf 'tar cmd: '
set -v
tar czvf "$filename" "${subdirs[#]}" 2>"$2"
set +v
END
bash "${tmp_script}" "$dir" "${tar_output}" 2>"${tar_command}"
cat "${tar_command}" | head -n 1 | sed -e 's/2>"\$2"$//'
cat "${tar_output}"
rm -f "${tmp_script}" "${tar_command}" "${tar_output}"
I apologize for nothing, but in the real world note that you'd want to make proper temp files.
If you execute the string $cmd, it won't work if "filename" embeds spaces
You have to let bash creates the arguments.
like this:
tar czvf "${filename}" $subdirs
You don't even need to put '\' in filename
OK, your original script did not work because file/path determination happens before variable expansion, so the filename is wrong: tar thinks that it's supposed to write to a file in the current directory named "/home/foo/foobar/test/Backup_FOOBAR_20120322_1529_35.tar.gz" i.e. the file name contains slashes and double quotes!
tar cfz /this/file/does/nopt/exist .
tar: /this/file/does/nopt/exist: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
See the difference? There no double quotes around the file name/path in tar's error message.
It worked when you copy and paste the line because then, the doublequotes are intepreted by the shell.
Witness:
ls -l /tmp/screen-exchange
-rw-rw-rw- 1 aqn users 0 Mar 21 07:29 /tmp/screen-exchange
cmd='ls -l "'/tmp/screen-exchange'"'
$cmd
/bin/ls: "/tmp/screen-exchange": No such file or directory
eval $cmd
-rw-rw-rw- 1 aqn users 0 Mar 21 07:29 /tmp/screen-exchange
Of course, using eval won't guard against filenames with whitespaces in them. To guard against that, your tar command needs to be like so:
date>'file name with spaces'
file='file name with spaces' # this is the equivalent of your $filename
cmd='ls -l "$file"'
$cmd
ls: "$file": No such file or directory
eval $cmd
-rw-r--r-- 1 andyn SPICE\Domain Users 1083 Mar 22 15:28 a b
I would suggest you separate $cmd from $filename and $subdirs. I think the induced error comes from when you join these strings. Also, using multiple variables in one variable without proper quoting will also induce errors.
This should work for you:
cmd="tar -zcvf"
subdirs="etc/ sbin/ bin/"
filename="${DIR}/Backup_${backup_name}_${date}.tar.gz"
$cmd $filename $subdirs
#DIR is something like: /home/foo/foobar/test/ without any whitespace but can also include whitespace
DIR="$( cd "$( dirname "$0" )" && pwd )"
backup_name=FOOBAR
date=`date +%Y%m%d_%H%M_%S`
subdirs="etc/ sbin/ bin/"
filename="$DIR/Backup_$backup_name"_"$date.tar.gz"
cd /
echo "filename: $filename"
echo "subdirs $subdirs"
cmd="tar zcvf $filename $subdirs"
echo "cmd tar: $cmd"
$cmd

Resources