Linux Bash - excluding directories with find - linux

i'm having some trouble with a bash script i'm currently writing, and I've isolated the problematic code.
Find command when using argument from variable does not exclude directories.
Now i need to assign the '-not -path ./dir' argument to a variable as it's being looped over, however if i do, it doesn't seem to work.
I've included below the command that produces the desired output, along with my version which currently is broken. Please see my test below:
Variable assignment
findIgnorePaths="-not -path ./\.git\* -not -path ./administrator/cache\* -not -path ./images\* -not -path ./logs\* -not -path ./cache\* -not -path ./media\* -not -path ./plugins\* -not -path ./tmp\*"
Desired output.
find ./ -type d -not -path ./\.git\* -not -path ./administrator/cache\* -not -path ./images\* -not -path ./logs\* -not -path ./cache\* -not -path ./media\* -not -path ./plugins\* -not -path ./tmp\*
Broken output
find ./ -type d ${findIgnorePaths}
And for anyone who wants to see my script i'm working on which the problem above resides in, then please take a look below:
Original script (Not required to read to answer question, but... provided for those curious souls)
#!/bin/sh
declare -a ignoreDirs=()
findIgnorePaths=""
seperator="\e[33m**********************************\e[0m\n"
clear
printf "\n\n"
if [[ "${PWD##*/}" != "public_html" ]]
then
printf "\e[31mThis script will not run unless current location is; /home/*/public_html/*\e[0m\n"
exit
fi
echo "Site Locker! This will change all directory permissions to 555. And all file permissions to 444. Read only accces."
echo #Move to new line
echo "Are you sure you want to make changes to files/directories within the below location?"
printf "$seperator"
printf "\e[0;49;96m${PWD##/home/}\e[0m\n"
printf "$seperator"
echo #Move to new line
read -r -p "Y/n " response
echo #Move to new line
if [[ $response =~ ^([nN][oO]|[nN])$ ]]
then
printf "\n\e[31mNo changes made. Quitting.\e[0m\n\n"
exit
fi
printf "\e[95mIf you're working with a Joomla site, please select\nthe **JOOMLA** preset in the list below.\e[0m\n\nPlease select directory #) to add to the ignore list:\n\e[92m"
currentDir=( "**SAVE/SKIP**" "**JOOMLA**" )
currentDir+=( $(find . -maxdepth 1 -type d) )
select DIR in ${currentDir[#]};
do
case $DIR in
*SAVE*)
printf "\n\n\e[92mDone!!\e[0m\n\n"
break
;;
*JOOMLA*)
printf "\n\e[92mApplying Joomla preset.\n"
ignoreDirs+=("./.git" "./administrator/cache" "./cache" "./images" "./logs" "./media" "./tmp" "./plugins")
findIgnorePaths+=" -not -path ./\.git\* -not -path ./administrator/cache\* -not -path ./images\* -not -path ./logs\* -not -path ./cache\* -not -path ./media\* -not -path ./plugins\* -not -path ./tmp\*"
printf "\n\n\e[31mIgnore list:\n"
for item in "${ignoreDirs[#]}"
do
printf "$item\n"
done
;;
*)
ignoreDirs+=("$DIR")
findIgnorePaths+=" -not -path ${DIR}\*"
printf "\n\n\e[31mIgnore list:\n"
for item in "${ignoreDirs[#]}"
do
printf "$item\n"
done
printf "\e[92m"
;;
esac
done
findIgnorePaths=$(echo ${findIgnorePaths} | cut -c 1-)
printf "\e[0m\n"
echo #Move to new line
printf "$seperator"
echo #Move to new line
echo "Apply 555 permissions to the below directories & all sub directories? "
printf "$seperator"
printf "\e[0;49;96m"
echo
echo "$findIgnorePaths"
echo
find ./ -maxdepth 1 -type d -not -path ./\.git\* -not -path ./administrator/cache\* -not -path ./images\* -not -path ./logs\* -not -path ./cache\* -not -path ./media\* -not -path ./plugins\* -not -path ./tmp\*
echo
find ./ -maxdepth 1 -type d ${findIgnorePaths}
echo
printf "\e[0m\n"
printf "$seperator"
read -r -p "Y/n " response
echo #Move to new line
if [[ $response =~ ^([yY][eE][sS]|[yY])$ ]]
then
find ./ -type d $findIgnorePaths -exec chmod 555 {} +
printf "\e[92mChanged directory permissions to 555\e[0m\n\n"
else
printf "\e[31mSkipping this step. No directory permissions were set.\e[0m\n\n"
fi
read -r -p "Apply 444 permissions to all files in; ${PWD##*/}? Y/n " response
echo #Move to new line
if [[ $response =~ ^([yY][eE][sS]|[yY])$ ]]
then
find ./ -type f $findIgnorePaths -exec chmod 444 {} +
printf "\e[92mChanged file permissions to 444\e[0m\n\n"
else
printf "\e[31mSkipping this step. No file permissions were set.\e[0m\n\n"
fi
printf "\n\e[92mFinished!\e[0m\n\n"

This is Bash FAQ 50, which advises you to use an array variable to pass parameters to programs:
findIgnorePaths=(-not -path './.git*' -not -path './administrator/cache*' -not -path './images*' -not -path './logs*' -not -path './cache*' -not -path './media*' -not -path './plugins*' -not -path './tmp*')
find ./ -type d "${findIgnorePaths[#]}"
Note directory globs are single quoted so they aren't interpreted by the shell when the variable is created. Your backslash escapes were doing the same thing; it's a matter of personal preference which one you find easier to read.
Running scripts through shellcheck.net is always a good idea to find any problems that might be lurking.
Also, /bin/sh is not /bin/bash, so fix your shebang!

Related

Bash find command : No such file or directory

This one is stumping me quite well! Big points for a resolution!
I need to traverse a directory and all levels below it to discover any files or directories that do not match a pattern.
I'm using find command -
find -L /home/user/inbox \( ! -path /home/user/inbox -a ! -path '/home/user/inbox/*' -a ! -path /home/user/inbox/imaging -a ! -path '/home/user/inbox/imaging/*' \) -ls
The error -
find -L /home/user/inbox \( ! -path /home/user/inbox -a ! -path '/home/user/inbox/*' -a ! -path /home/user/inbox/imaging -a ! -path '/home/user/inbox/imaging/*' \) -ls: No such file or directory
Yes, each of those directories in the find command do indeed exist -
ls -ld /home/user/inbox /home/user/inbox/imaging/
drwxr-xr-x. 5 user grp-ftp-admin 8192 Jan 13 12:45 /home/user/inbox
drwxr-xr-x. 3 user grp-ftp-admin 152 Jan 13 12:21 /home/user/inbox/imaging/
The entire script source -
#!/usr/bin/env bash
function findInvalidArtifacts() {
unset inbox sshCommand exitCode cmdOutput
s ilocal -A VENDOR=(
['FTPBASE']="/home/user"
['FTPSOURCEDIR']="/home/user/inbox/imaging"
['FRIENDLYNAME']="user"
['ENVIRONMENT']="dev"
)
inbox="${VENDOR['FTPBASE']}/inbox"
printf -v sshCommand "find -L %s \( ! -path %s -a ! -path '%s/*' -a ! -path %s -a ! -path '%s/*' \) -ls" \
"${inbox}" "${inbox}" "${inbox}" "${VENDOR['FTPSOURCEDIR']}" "${VENDOR['FTPSOURCEDIR']}"
#cmdOutput=$(ssh ${SSHOPTIONS} ${SSHUSERHOST} "${sshCommand}" 2>&1)
cmdOutput=$("${sshCommand}")
exitCode="${?}"
if (( 0 != $exitCode )); then
printf "FAILED - sshCommand=[%s], exitCode=[%d], cmdOutput=[%s]\n" "${sshCommand}" "${exitCode}" "${cmdOutput}"
exit 1
fi
printf "SUCCEEDED - sshCommand=[%s] exitCode=[%d]\n" "${sshCommand}" "${exitCode}"
printf "%s\n\n" "${cmdOutput}"
} # end findInvalidArtifacts()
And, finally, running the script with bash -vx -
#!/usr/bin/env bash
function findInvalidArtifacts() {
unset inbox sshCommand exitCode cmdOutput
local -A VENDOR=(
['FTPBASE']="/home/user"
['FTPSOURCEDIR']="/home/user/inbox/imaging"
['FRIENDLYNAME']="user"
['ENVIRONMENT']="dev"
)
inbox="${VENDOR['FTPBASE']}/inbox"
printf -v sshCommand "find -L %s \( ! -path %s -a ! -path '%s/*' -a ! -path %s -a ! -path '%s/*' \) -ls" \
"${inbox}" "${inbox}" "${inbox}" "${VENDOR['FTPSOURCEDIR']}" "${VENDOR['FTPSOURCEDIR']}"
#cmdOutput=$(ssh ${SSHOPTIONS} ${SSHUSERHOST} "${sshCommand}" 2>&1)
cmdOutput=$("${sshCommand}")
exitCode="${?}"
if (( 0 != $exitCode )); then
printf "FAILED - sshCommand=[%s], exitCode=[%d], cmdOutput=[%s]\n" "${sshCommand}" "${exitCode}" "${cmdOutput}"
exit 1
fi
printf "SUCCEEDED - sshCommand=[%s] exitCode=[%d]\n" "${sshCommand}" "${exitCode}"
printf "%s\n\n" "${cmdOutput}"
} # end findInvalidArtifacts()
findInvalidArtifacts
+ findInvalidArtifacts
+ unset inbox sshCommand exitCode cmdOutput
+ VENDOR=(['FTPBASE']="/home/user" ['FTPSOURCEDIR']="/home/user/inbox/imaging" ['FRIENDLYNAME']="user" ['ENVIRONMENT']="dev")
+ local -A VENDOR
+ inbox=/home/user/inbox
+ printf -v sshCommand 'find -L %s \( ! -path %s -a ! -path '\''%s/*'\'' -a ! -path %s -a ! -path '\''%s/*'\'' \) -ls' /home/user/inbox /home/user/inbox /home/user/inbox /home/user/inbox/imaging /home/user/inbox/imaging
++ 'find -L /home/user/inbox \( ! -path /home/user/inbox -a ! -path '\''/home/user/inbox/*'\'' -a ! -path /home/user/inbox/imaging -a ! -path '\''/home/user/inbox/imaging/*'\'' \) -ls'
./try.sh: line 16: find -L /home/user/inbox \( ! -path /home/user/inbox -a ! -path '/home/user/inbox/*' -a ! -path /home/user/inbox/imaging -a ! -path '/home/user/inbox/imaging/*' \) -ls: No such file or directory
+ cmdOutput=
+ exitCode=127
+ (( 0 != 127 ))
+ printf 'FAILED - sshCommand=[%s], exitCode=[%d], cmdOutput=[%s]\n' 'find -L /home/user/inbox \( ! -path /home/user/inbox -a ! -path '\''/home/user/inbox/*'\'' -a ! -path /home/user/inbox/imaging -a ! -path '\''/home/user/inbox/imaging/*'\'' \) -ls' 127 ''
FAILED - sshCommand=[find -L /home/user/inbox \( ! -path /home/user/inbox -a ! -path '/home/user/inbox/*' -a ! -path /home/user/inbox/imaging -a ! -path '/home/user/inbox/imaging/*' \) -ls], exitCode=[127], cmdOutput=[]
+ exit 1
This is a stumper. I cannot see where why find is complaining about 'no such file or directory'.
Appreciate the valiant effort, however the constructed command still does not return the desired results. The contents of finding all existing artifacts in /home/user/inbox are:
/home/user/inbox
/home/user/inbox/invalidDir2
/home/user/inbox/invalidDir2/invalidSubdir2
/home/user/inbox/invalidDir2/invalidSubdir2/invalidFile1
/home/user/inbox/invalidDir2/invalidSubdir2/invalidSubSubDir2
/home/user/inbox/invalidDir2/invalidSubdir2/invalidSubSubDir2/invalidSubSubSubDir2
/home/user/inbox/invalidDir2/invalidSubdir2/invalidSubSubDir2/invalidSubSubSubDir2/invalidSubSubSubSubDir2
/home/user/inbox/invalidDir2/invalidSubdir2/invalidSubSubDir2/invalidSubSubSubDir2/invalidSubSubSubSubDir2/invalidFile
/home/user/inbox/invalidDir2/invalidSubdir2/invalidSubSubDir2/invalidSubSubSubDir2/invalidFile
/home/user/inbox/invalidDir2/invalidSubdir2/invalidSubSubDir2/invalidFile
/home/user/inbox/imaging
/home/user/inbox/imaging/validFile
/home/user/inbox/imaging/invalidFile
/home/user/inbox/imaging/invalidImagingDir1
/home/user/inbox/imaging/invalidImagingDir1/invalidSubdir1
/home/user/inbox/imaging/invalidImagingDir1/invalidSubdir1/invalidFile1
/home/user/inbox/imaging/invalidImagingDir1/invalidSubdir1/invalidSubSubDir1
/home/user/inbox/imaging/invalidImagingDir1/invalidSubdir1/invalidSubSubDir1/invalidSubSubSubDir1
/home/user/inbox/imaging/invalidImagingDir1/invalidSubdir1/invalidSubSubDir1/invalidSubSubSubDir1/invalidFile
/home/user/inbox/imaging/invalidImagingDir1/invalidSubdir1/invalidSubSubDir1/invalidFile
/home/user/inbox/imaging/invalidImagingDir1/invalildFile
/home/user/inbox/invalidFile
/home/user/inbox/invalidDir1
/home/user/inbox/invalidDir1/invalidSubdir1
/home/user/inbox/invalidDir1/invalidSubdir1/invalidFile1
/home/user/inbox/invalidDir1/invalidSubdir1/invalidSubSubDir1
/home/user/inbox/invalidDir1/invalidSubdir1/invalidSubSubDir1/invalidSubSubSubDir1
/home/user/inbox/invalidDir1/invalidSubdir1/invalidSubSubDir1/invalidSubSubSubDir1/invalidFile
/home/user/inbox/invalidDir1/invalidSubdir1/invalidSubSubDir1/invalidFile
The result set should NOT contain anything matching the directory to search in, or any of the parameters passed to your function that constructs the command. Results should NOT contain these lines -
/home/user/inbox
/home/user/inbox/imaging
/home/user/inbox/imaging/validFile
The command your function constructs looks like this -
find -L /home/user/inbox '(' '!' -path /home/user/inbox -a '!' -path '/home/user/inbox/*' -a '!' -path /home/user/inbox/imaging -a '!' -path '/home/user/inbox/imaging/*' ')' -ls
which returns nothing at all
I'm beginning to think that the find command is not what I need to obtain the desired results and that I'll need to do some more processing of the result set to disclude, probably awk or something.
Relevant snippet:
printf -v sshCommand "find -L %s \( ! -path %s -a ! -path '%s/*' -a ! -path %s -a ! -path '%s/*' \) -ls" [more args]
cmdOutput=$("${sshCommand}")
This is a classic pitfall of trying to shove a command into a variable and run it. The basic solution is to use a function, not a variable.
The reason for your error is that Bash determines the name of the command you're trying to run by selecting the first word of the command line after completing all of its parsing steps. You want to use the command find with some arguments, but since you have "${sshCommand}" in quotes, there's only one word in the entire command line so Bash thinks the command you want is called find -L /home/dir ( ! -path etc etc... ). I bet you don't have any executables with that name on your system.
A function equivalent of what you're trying to do might look like:
_findCmd() {
searchDir=$1 # first arg is the directory to search from
shift
# each extra argument is a directory to exclude from the search
# we can loop over the list and build the arguments that find needs
# in an array
path_args=()
if (( $# > 0 )) ; then
path_args+=( "(" "!" "-path" "$1" )
shift
while (( $# > 0 )) ; do
path_args+=( "-a" "!" "-path" "$1" )
shift
done
path_args+=( ")" )
fi
find -L "$searchDir" "${path_args[#]}" -ls
}
cmdOutput=$( _findCmd /home/user/inbox /home/user/inbox "/home/user/inbox/*" /home/user/inbox/imaging "/home/user/inbox/imaging/*" )

Move a file and rename it if the destination folder contains also a file with the same name

Sorry for my bad english. I want to move a file from the Desktop to user's input destination.
If destination has also a file with the same name, how can i ask user to rename the file so that he can have them both in destination folder?
I have this right now:
#!bin/bash
echo "path"
read path
echo "Do you want to move code files somewhere else?"
read -t 5 y
if [[ $y = 'yes' ]] ; then
echo "Tell me where:"
read mydest
find $path -type f \( -name "*.c" -or -name "*.cxx" -or -name "*.cpp" -or -name "*.cc" \) );do
mv ???
fi
Demonstrating using mv and switches -i (interactive) and --backup:
$ echo foo > foo # making test files
$ echo bar > bar
$ mv -i --backup foo bar # mving foo
mv: overwrite 'bar'? y # -i caused this
$ cat bar
foo # mv caused this
$ cat bar~ # --backup caused this
bar

Find files with certain date strings

I have a number of files with dates in their names:
file_19990101.txt
file_19990102.txt
...
file_20031231.txt
I want to generate an ASCII file with all files up to the date 20010320, and then a second file with files from 20010321 to 20031231. How can this be accomplished using bash commands?
My current solution is to do a bunch of find commands:
find . -name "file_1999*" -print > index.txt
find . -name "file_2000*" -print >> index.txt
find . -name "file_200101*" -print >> index.txt
find . -name "file_200102*" -print >> index.txt
find . -name "file_2001030*" -print >> index.txt
find . -name "file_2001031*" -print >> index.txt
find . -name "file_20010320*" -print >> index.txt
etc. But there must be an easier way to accomplish this task!
This might work:
for f in file_*
do
ymd=${f#file_}
ymd=${ymd%.txt}
[[ "${ymd}" < "20010321" ]] && echo "${f}" >>index1 || echo "${f}" >>index2
done
Loop over the glob, strip the prefix and suffix, compare with the cutoff and append to one file or the other.
Edit: I missed the second condition.
#!/bin/bash
for f in file_*
do
ymd=${f#file_}
ymd=${ymd%.txt}
if [[ "${ymd}" < "20010321" ]]
then
echo "${f}" >>index1
elif [[ "${ymd}" > "20031231" ]]
then
:
else
echo "${f}" >>index2
fi
done

Bash Script to process data containing input string

I am trying to create a script that will find all the files in a folder that contain, for example, the string 'J34567' and process them. Right now I can process all the files in the folder with my code, however, my script will not just process the contained string it will process all the files in the folder. In other words once I run the script even with the string name ./bashexample 'J37264' it will still process all the files even without that string name. Here is my code below:
#!/bin/bash
directory=$(cd `dirname .` && pwd)
tag=$1
echo find: $tag on $directory
find $directory . -type f -exec grep -sl "$tag" {} \;
for files in $directory/*$tag*
do
for i in *.std
do
/projects/OPSLIB/BCMTOOLS/sumfmt_linux < $i > $i.sum
done
for j in *.txt
do
egrep "device|Device|\(F\)" $i > $i.fail
done
echo $files
done
Kevin, you could try the following:
#!/bin/bash
directory='/home'
tag=$1
for files in $directory/*$tag*
do
if [ -f $files ]
then
#do your stuff
echo $files
fi
done
where directory is your directory name (you could pass it as a command-line argument too) and tag is the search term you are looking for in a filename.
Following script will give you the list of files that contain (inside the file, not in file name) the given pattern.
#!/bin/bash
directory=`pwd`
tag=$1
for file in $(find "$directory" -type f -exec grep -l "$tag" {} \;); do
echo $file
# use $file for further operations
done
What is the relevance of .std, .txt, .sum and .fail files to the files containing given pattern?
Its assumed there are no special characters, spaces, etc. in file names.
If that is the case following should help working around those.
How can I escape white space in a bash loop list?
Capturing output of find . -print0 into a bash array
There are multiple issues in your script.
Following is not required to set the operating directory to current directory.
directory=$(cd `dirname .` && pwd)
find is executed twice for the current directory due to $directory and ..
find $directory . -type f -exec grep -sl "$tag" {} \;
Also, result/output of above find is not used in for loop.
For loop is run for files in the $directory (sub directories not considered) with their file name having the given pattern.
for files in $directory/*$tag*
Following for loop will run for all .txt files in current directory, but will result in only one output file due to use of $i from previous loop.
for j in *.txt
do
egrep "device|Device|\(F\)" $i > $i.fail
done
This is my temporary solution. Please check if it follows your intention.
#!/bin/bash
directory=$(cd `dirname .` && pwd) ## Should this be just directory=$PWD ?
tag=$1
echo "find: $tag on $directory"
find "$directory" . -type f -exec grep -sl "$tag" {} \; ## Shouldn't you add -maxdepth 1 ? Are the files listed here the one that should be processed in the loop below instead?
for file in "$directory"/*"$tag"*; do
if [[ $file == *.std ]]; then
/projects/OPSLIB/BCMTOOLS/sumfmt_linux < "$file" > "${file}.sum"
fi
if [[ $file == *.txt ]]; then
egrep "device|Device|\(F\)" "$file" > "${file}.fail"
fi
echo "$file"
done
Update 1
#!/bin/bash
directory=$PWD ## Change this to another directory if needed.
tag=$1
echo "find: $tag on $directory"
while IFS= read -rd $'\0' file; do
echo "$file"
case "$file" in
*.std)
/projects/OPSLIB/BCMTOOLS/sumfmt_linux < "$file" > "${file}.sum"
;;
*.txt)
egrep "device|Device|\(F\)" "$file" > "${file}.fail"
;;
*)
echo "Unexpected match: $file"
;;
esac
done < <(exec find "$directory" -maxdepth 1 -type f -name "*${tag}*" \( -name '*.std' -or -name '*.txt' \) -print0) ## Change or remove the maxdepth option as wanted.
Update 2
#!/bin/bash
directory=$PWD
tag=$1
echo "find: $tag on $directory"
while IFS= read -rd $'\0' file; do
echo "$file"
/projects/OPSLIB/BCMTOOLS/sumfmt_linux < "$file" > "${file}.sum"
done < <(exec find "$directory" . -maxdepth 1 -type f -name "*${tag}*" -name '*.std' -print0)
while IFS= read -rd $'\0' file; do
echo "$file"
egrep "device|Device|\(F\)" "$file" > "${file}.fail"
done < <(exec find "$directory" -maxdepth 1 -type f -name "*${tag}*" -name '*.txt' -print0)

I am trying to force a script to accept user input, but it breaks on the second input

I am new to Power shell and I cant seem to wrap my head around this one.
The Code works when you take the input out and place actual values in for each $number and $Name as seen in the bottom example. I found that unlike specifying the values, line 7 char 13 cant be -$Name but rather has to be $Name. After doing this it tells me ln 15 char 13 "a positional character cannot be found that accepts argument "- "." where the "-" = $Name.
What do I need to change to get this to function? I have tired using [parameter(Mandatory=$true,name='What would you like your folder to be named: ')] but I cant seem to get that to work either.
Any help would be greatly appreciated.
$number = read-host "How Many Folders would you like to create: "
$Name = read-host "What would you like your folder to be named: "
$intFolders = $number
$intPad
$i = 1
New-Variable -$Name strPrefix -Value "$Name" -Option constant
do {
if ($i -lt $number)
{$intPad=0
new-item -path c:\mytest -$Name $strPrefix$intPad$i -type directory}
else
{new-item -path c:\mytest -$Name $strPrefix$i -type directory}
$i++}
until ($i -eq $intFolders+1)
This is the code below that I was able to get to work, but I want to make sure when I am gone people here can still use it for their needs as they do not understand coding.
$intFolders = 500
$intPad
$i = 1
New-Variable -Name strPrefix -Value "0001_" -Option constant
do {
if ($i -lt 500)
{$intPad=0
new-item -path c:\mytest -name $strPrefix$intPad$i -type directory}
else
{new-item -path c:\mytest -name $strPrefix$i -type directory}
$i++}
until ($i -eq $intFolders+1)
Thanks to Bruce here is the working Final code that does excatly what it should have :)
[int]$start = read-host "What number would you like your folder to start at"
[int]$number = read-host "How Many Folders would you like to create"
$Name = read-host "What would you like your folder to be named"
$intFolders = $number
$intPad
$i = $start
New-Variable -Name strPrefix -Value "$Name" -Option constant
do {
if ($i -lt 10)
{$intPad=""
new-item -path c:\mytest -Name $strPrefix$intPad$i -type directory}
else
{new-item -path c:\mytest -Name $strPrefix$i -type directory}
$i++}
until ($i -eq $intFolders+1)
Change New-Variable -$Name strPrefix -Value "$Name" -Option constant to
New-Variable -Name strPrefix -Value "$Name" -Option constant
Same for the -$name in the new-item lines. See Get-Help New-Variable and Get-Help New-Item.

Resources