I have the following four files in a directory:
TRAILBLAZER_107-10016_FTP_SCR_CT_CTAC
TRAILBLAZER_107-10016_FTP_SCR_CT_Recon
TRAILBLAZER_107-10016_FTP_SCR_PET_NAC
TRAILBLAZER_107-10016_FTP_SCR_PET_AC_Frames
And i've made a simple for loop to go through each of those files and change the name of the file based off a certain key word in the name. This essentially just changes the name of the bottom two files:
for file in TRAILBLAZER*
do
mv "$file" "${file/PET_AC/PET_TESTAC}"
mv "$file" "${file/PET_NAC/TESTNAC}"
done
How would I be able to echo the number of files that have been altered by that for loop, and the number of files in the directory that remain unchanged?
The Parameter Expansion (with substring replacement) does not return any value to let you know if it succeeded or failed. You will need to do that manually. Save the filename before you modify it and check if the file exists afterwards. If it doesn't, a change was made so increment a counter.
You could do something similar to:
changed=0 ## counter
for file in TRAILBLAZER*
do
fname="$file" ## save original file name
mv "$file" "${file/PET_AC/PET_TESTAC}"
mv "$file" "${file/PET_NAC/TESTNAC}"
[ -f "$fname" ] || ((changed++)) ## original exists or increment counter
done
printf "files changed: %s\n", "$changed"
Related
I have a couple of thousands CSV file. All of them have same structure and header. I would like to add a column at the end of the file. I found several solutions that add a column and value to that column but I didn't find anything that adds the header for that new column. For example, I have files like 1001.csv, 1002.csv, 1003.csv and so on.
Contents of 1001.csv
ID,URL
1,one.com
2,two.com
I want to modify it like this
ID,URL,FILE
1,one.com,1001
2,two.com,1001
Since I have tons of files like this, I don't want to mess up the data while adding a column. Also, I don't want to produce extra files if it's possible to do in place update.
I tested this on a huge number of files and it worked really fast. This code removes the header first then add a column plus value to the column and finally brings the header back.
#!/bin/bash
# How to run $ ./this-script.sh inputdir/
# here inputdir contains all csv files
# input argument is dir name
DIRNAME=`basename $1`
# go to target directory
cd $DIRNAME
# get list of all csv files
csvfiles=`ls *.csv`
for FILENAME in $csvfiles
do
echo $FILENAME
# filename without extension
CODE="${FILENAME%.*}"
echo $CODE
## remove header
tail -n +2 "$FILENAME" > "$FILENAME.tmp" && mv "$FILENAME.tmp" "$FILENAME"
## add new field at the end
sed "s/$/,$CODE/" "$FILENAME" > "$FILENAME.tmp2"
## add header with new column name
# keep filename.bak as a backup for safety
sed -i.bak 1i"id,url,file" "$FILENAME.tmp2"
# if all good then remove temp files
rm "$FILENAME"
rm "$FILENAME.tmp2.bak"
# rename output file to original name
mv "$FILENAME.tmp2" "$FILENAME"
done
# go back to parent directory
cd ..
I am still a newbie in shell scripting and trying to come up with a simple code. Could anyone give me some direction here. Here is what I need.
Files in path 1: /tmp
100abcd
200efgh
300ijkl
Files in path2: /home/storage
backupfile_100abcd_str1
backupfile_100abcd_str2
backupfile_200efgh_str1
backupfile_200efgh_str2
backupfile_200efgh_str3
Now I need to delete file 300ijkl in /tmp as the corresponding backup file is not present in /home/storage. The /tmp file contains more than 300 files. I need to delete the files in /tmp for which the corresponding backup files are not present and the file names in /tmp will match file names in /home/storage or directories under /home/storage.
Appreciate your time and response.
You can also approach the deletion using grep as well. You can loop though the files in /tmp checking with ls piped to grep, and deleting if there is not a match:
#!/bin/bash
[ -z "$1" -o -z "$2" ] && { ## validate input
printf "error: insufficient input. Usage: %s tmpfiles storage\n" ${0//*\//}
exit 1
}
for i in "$1"/*; do
fn=${i##*/} ## strip path, leaving filename only
## if file in backup matches filename, skip rest of loop
ls "${2}"* | grep -q "$fn" &>/dev/null && continue
printf "removing %s\n" "$i"
# rm "$i" ## remove file
done
Note: the actual removal is commented out above, test and insure there are no unintended consequences before preforming the actual delete. Call it passing the path to tmp (without trailing /) as the first argument and with /home/storage as the second argument:
$ bash scriptname /path/to/tmp /home/storage
You can solve this by
making a list of the files in /home/storage
testing each filename in /tmp to see if it is in the list from /home/storage
Given the linux+shell tags, one might use bash:
make the list of files from /home/storage an associative array
make the subscript of the array the filename
Here is a sample script to illustrate ($1 and $2 are the parameters to pass to the script, i.e., /home/storage and /tmp):
#!/bin/bash
declare -A InTarget
while read path
do
name=${path##*/}
InTarget[$name]=$path
done < <(find $1 -type f)
while read path
do
name=${path##*/}
[[ -z ${InTarget[$name]} ]] && rm -f $path
done < <(find $2 -type f)
It uses two interesting shell features:
name=${path##*/} is a POSIX shell feature which allows the script to perform the basename function without an extra process (per filename). That makes the script faster.
done < <(find $2 -type f) is a bash feature which lets the script read the list of filenames from find without making the assignments to the array run in a subprocess. Here the reason for using the feature is that if the array is updated in a subprocess, it would have no effect on the array value in the script which is passed to the second loop.
For related discussion:
Extract File Basename Without Path and Extension in Bash
Bash Script: While-Loop Subshell Dilemma
I spent some really nice time on this today because I needed to delete files which have same name but different extensions, so if anyone is looking for a quick implementation, here you go:
#!/bin/bash
# We need some reference to files which we want to keep and not delete,
# let's assume you want to keep files in first folder with jpeg, so you
# need to map it into the desired file extension first.
FILES_TO_KEEP=`ls -1 ${2} | sed 's/\.pdf$/.jpeg/g'`
#iterate through files in first argument path
for file in ${1}/*; do
# In my case, I did not want to do anything with directories, so let's continue cycle when hitting one.
if [[ -d $file ]]; then
continue
fi
# let's omit path from the iterated file with baseline so we can compare it to the files we want to keep
NAME_WITHOUT_PATH=`basename $file`
# I use mac which is equal to having poor quality clts
# when it comes to operating with strings,
# this should be safe check to see if FILES_TO_KEEP contain NAME_WITHOUT_PATH
if [[ $FILES_TO_KEEP == *"$NAME_WITHOUT_PATH"* ]];then
echo "Not deleting: $NAME_WITHOUT_PATH"
else
# If it does not contain file from the other directory, remove it.
echo "deleting: $NAME_WITHOUT_PATH"
rm -rf $file
fi
done
Usage: sh deleteDifferentFiles.sh path/from/where path/source/of/truth
I need to write a bash script that iterates through the files of a specified directory and replaces duplicates of files with hardlinks. Right now, my entire function looks like this:
#! /bin/bash
# sameln --- remove duplicate copies of files in specified directory
D=$1
cd $D #go to directory specified as default input
fileNum=0 #loop counter
DIR=".*|*"
for f in $DIR #for every file in the directory
do
files[$fileNum]=$f #save that file into the array
fileNum=$((fileNum+1)) #increment the counter
done
for((j=0; j<$fileNum; j++)) #for every file
do
if [ -f "$files[$j]" ] #access that file in the array
then
for((k=0; k<$fileNum; k++)) #for every other file
do
if [ -f "$files[$k]" ] #access other files in the array
then
test[cmp -s ${files[$j]} ${files[$k]}] #compare if the files are identical
[ln ${files[$j]} ${files[$k]}] #change second file to a hard link
fi
done
fi
done
Basically:
Loop through all files of depth 1 in specified directory
Put file contents into array
Compare each array item with every other array item and replace duplicates with hardlinks
The test directory has four files: a, b, c, d
a and b are different, but c and d are duplicates (they are empty). After running the script, ls -l shows that all of the files still only have 1 hardlink, so the script appears to have basically done nothing.
Where am I going wrong?
DIR=".*|*"
for f in $DIR #for every file in the directory
do
echo $f
done
This code outputs
.*|*
You should not loop over files like this. Look into the find command. As you see, your code doesn't work because the first loop is already faulty.
BTW, don't name your variables all uppercase, those are reserved for system variables, I believe.
You may be making this process a bit harder on yourself than necessary. There is already a Linux command fdupes that scans a directory conducting a byte-by-byte, md5sum, date & time comparison to determine whether files are duplicates of one another. It can easily find and return groups of files that are duplicates. Your are left with only using the results.
Below is a quick example of using this tool for the job. NOTE this quick example works only for filenames that do not contain spaces within them. You will have to modify it if you are dealing with filenames containing spaces. This is intended to show an approach to using a tool that already does what you want. Also note the actual ln command is commented out below. The program just prints what it would do. After testing you can remove the comment to the ln command once you are satisfied with the results.
#! /bin/bash
# sameln --- remove duplicate copies of files in specified directory using fdupes
[ -d "$1" ] || { # test valid directory supplied
printf "error: invalid directory '%s'. usage: %s <dir>\n" "$1" "${0//\//}"
exit 1
}
type fdupes &>/dev/null || { # verify fdupes is available in path
printf "error: 'fdupes' required. Program not found within your path\n"
exit 1
}
pushd "$1" &>/dev/null # go to directory specified as default input
declare -a files # declare files and dupes array
declare -a dupes
## read duplicate files into files array
IFS=$'\n' read -d '' -a files < <(fdupes --sameline .)
## for each list of duplicates
for ((i = 0; i < ${#files[#]}; i++)); do
printf "\n duplicate files %s\n\n" "${files[i]}"
## split into original files (no interal 'spaces' allowed in filenames)
dupes=( ${files[i]} )
## for the 1st duplicate on
for ((j = 1; j < ${#dupes[#]}; j++)); do
## create hardlink to original (actual command commented)
printf " ln -f %s %s\n" "${dupes[0]}" "${dupes[j]}"
# ln -f "${dupes[0]}" "${dupes[j]}"
done
done
exit 0
Output/Example
$ bash rmdupes.sh dat
duplicate files ./output.dat ./tmptest ./env4.dat.out
ln -f ./output.dat ./tmptest
ln -f ./output.dat ./env4.dat.out
duplicate files ./vh.conf ./vhawk.conf
ln -f ./vh.conf ./vhawk.conf
duplicate files ./outfile.txt ./newfile.txt
ln -f ./outfile.txt ./newfile.txt
duplicate files ./z1 ./z1cpy
ln -f ./z1 ./z1cpy
I'm taking a Unix class, and here's a part of my assignment:
For each file and subdirectory in the user’s ~/Documents directory, determine if the item is a file or directory, and display a message to that effect, using the file name in the statement.
So, what I have written is this:
docs=`ls ~/Documents`
for file in $docs ; do
if [ -f $file ] ; then
echo $file "is a file."
elif [ -d $file ] ; then
echo $file "is a directory."
else
echo $file "is not a file or directory."
fi
done
My Documents directory includes these files and directories:
DocList.txt (file)
Letter (file)
mypasswdfile (file)
samples (directory)
things (directory)
touchfile (file)
So I figured that the output should be this:
DocList.txt is a file.
Letter is a file.
mypasswdfile is a file.
samples is a directory.
things is a directory.
touchfile is a file.
However, this is the output:
DocList.txt is not a file or directory.
Letter is not a file or directory
mypasswdfile is not a file or directory
samples is not a file or directory
things is not a file or directory
touchfile is not a file or directory
I feel like I should mention that if I set the $docs variable to `ls ~' it will successfully display the contents of my home directory and whether the items are files or directories. This does not work with other paths I have tried.
Your problem is that ls only outputs the file names without path.
So your $file gets the values
DocList.txt
Letter
mypasswdfile
samples
things
touchfile
from loop run to loop run.
If your current directory is NOT ~/Documents, testing these file names is wrong, as this would search in the current directory and not in the intended one.
A much better way to accomplish your task is
for file in ~/Documents/* ; do
...
done
which will set $file to each of the full path names needed to find your file.
After doing so, it should work, but it is very error prone: once your path or one of your files starts having a space or other blank character in it, it will fall on your feet.
Putting " around variables which can potentially contain something with a space etc. is quite essential. There is almost no reason ever to use a variable without its surrounding ".
What is the difference here?
With [ -f $file ], and file='something with spaces', [ is called with the arguments -f, something, with, spaces and ]. This surely leads to wrong behaviour.
OTOH, with [ -f "$file" ], and file='something with spaces', [ is called with the arguments -f, something with spaces and ].
So quoting is very essential in shell programming.
Of course, the same holds for [ -d "$file" ].
The problem is your ls command - you're treating the output of ls as absolute e.g. /home/alex/Documents/DocList.txt, but when you do ls ~/Documents it prints out DocList.txt (a relative file path / name).
To get the expected absolute behaviour you can use the find command instead:
docs=`find ~/Documents`
As mentioned in the comments and in another answer, to also be able to handle whitespace in filenames you need to do something like:
docs=( ~/Documents/* )
for f in "${docs[#]}"; do
...
I want to delete all files from a folder that contain a not unique numerical string in the filename using linux terminal. E.g.:
werrt-110009.jpg => delete
asfff-110009.JPG => delete
asffa-123489.jpg => maintain
asffa-111122.JPG => maintain
Any suggestions?
I only now understand your question, I think. You want to remove all files that contain a numeric value that is not unique (in a particular folder). If a filename contains a value that is also found in another filename, you want to remove both files, right?
This is how I would do that (it may not be the fastest way):
# put all files in your folder in a list
# for array=(*) to work make sure you have enabled nullglob: shopt -s nullglob
array=(*)
delete=()
for elem in "${array[#]}"; do
# for each elem in your list extract the number
num_regex='([0-9]+)\.'
[[ "$elem" =~ $num_regex ]]
num="${BASH_REMATCH[1]}"
# use the extracted number to check if it is unique
dup_regex="[^0-9]($num)\..+?(\1)"
# if it is not unique, put the file in the files-to-delete list
if [[ "${array[#]}" =~ $dup_regex ]]; then
delete+=("$elem")
fi
done
# delete all found duplicates
for elem in "${delete[#]}"; do
rm "$elem"
done
In your example, array would be:
array=(werrt-110009.jpg asfff-110009.JPG asffa-123489.jpg asffa-111122.JPG)
And the result in delete would be:
delete=(werrt-110009.jpg asfff-110009.JPG)
Is this what you meant?
you can use the linux find command along with the -regex parameter and the -delete parameter
to do it in one command
Use "rm" command to delete all matching string files in directory
cd <path-to-directory>/ && rm *110009*
This command helps to delete all files with matching string and it doesn't depend on the position of string in file name.
I was mentioned rm command option as another option to delete files with matching string.
Below is the complete script to achieve your requirement,
#!/bin/sh -eu
#provide the destination fodler path
DEST_FOLDER_PATH="$1"
TEMP_BUILD_DIR="/tmp/$( date +%Y%m%d-%H%M%S)_clenup_duplicate_files"
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
clean_up()
{
if [ -d $TEMP_BUILD_DIR ]; then
rm -rf $TEMP_BUILD_DIR
fi
}
trap clean_up EXIT
[ ! -d $TEMP_BUILD_DIR ] && mkdir -p $TEMP_BUILD_DIR
TEMP_FILES_LIST_FILE="$TEMP_BUILD_DIR/folder_file_names.txt"
echo "$(ls $DEST_FOLDER_PATH)" > $TEMP_FILES_LIST_FILE
while read filename
do
#check files with number pattern
if [[ "$filename" =~ '([0-9]+)\.' ]]; then
#fetch the number to find files with similar number
matching_string="${BASH_REMATCH[1]}"
# use the extracted number to check if it is unique
#find the files count with matching_string
if [ $(ls -1 $DEST_FOLDER_PATH/*$matching_string* | wc -l) -gt 1 ]; then
rm $DEST_FOLDER_PATH/*$matching_string*
fi
fi
#reload remaining files in folder (this optimizes the loop and speeds up the operation
#(this helps lot when folder contains more files))
echo "$(ls $DEST_FOLDER_PATH)" > $TEMP_FILES_LIST_FILE
done < $TEMP_FILES_LIST_FILE
exit 0
How to execute this script,
Save this script into file as
path-to-script/delete_duplicate_files.sh (you can rename whatever
you want)
Make script executable
chmod +x {path-to-script}/delete_duplicate_files.sh
Execute script by providing directory path where duplicate
files(files with matching number pattern) needs to be deleted
{path-to-script}/delete_duplicate_files.sh "{path-to-directory}"