I have two folders each containing 351 text files and i want to copy the corresponding text from one folder to corresponding file in another folder?
when i am using cat command i am getting an empty file as a result? what could be the problem
my code is :
#!/bin/bash
DIR1=$(ls 2/)
DIR2=$(ls 3/)
for each $i in $DIR1; do
for each $j in $DIR2; do
if [[ $i == $j ]];then
sudo cat $i $j >> $j
fi
done
done
2/ and 3/ are the folders containing the data...
DIR1 and DIR2 contain the file names in directories 2 and 3 respectively.
Apart from possible problems with spaces or special characters in file names, you would have to use 2/$i and 3/$j. $i and $j alone would reference files with the same names in the current directory (parent of 2 and 3).
It's better not to parse the output of ls.
You don't need two nested loops.
#!/bin/bash
DIR1=2
DIR2=3
for source in $DIR1/*
do
dest="$DIR2/$(basename $source)"
if [ -f "$dest" ]
then
sudo cat "$source" >> "$dest"
fi
done
see also https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_.2A.mp3.29
Depending on your needs it may be better to run the whole script with sudo instead of running sudo for every file. The version above will only execute cat "$source" as root. When running the whole script as root this includes also >> "$dest".
Related
(Please see also the minimal example at bottom)
I have the following folder structure:
https://drive.google.com/open?id=1an6x1IRtNjOG6d9D5FlYwsUxmgaEegUY
Each folder shows at the end: the date, month and day:
e.g.
/HRIT/EPI/2004/01/14
Inside for each day, each of this sub-folders contains a lot of files (in total about 10 TB).
All available folders are:
/HRIT/EPI/YYYY/MM/DD/
/HRIT/HRV/YYYY/MM/DD/
/HRIT/VIS006/YYYY/MM/DD/
/HRIT/VIS008/YYYY/MM/DD/
/HRIT/WV_062/YYYY/MM/DD/
/HRIT/WV_073/YYYY/MM/DD/
/HRIT/IR_016/YYYY/MM/DD/
/HRIT/IR_039/YYYY/MM/DD/
/HRIT/IR_087/YYYY/MM/DD/
/HRIT/IR_097/YYYY/MM/DD/
/HRIT/IR_108/YYYY/MM/DD/
/HRIT/IR_120/YYYY/MM/DD/
/HRIT/IR_134/YYYY/MM/DD/
/HRIT/PRO/YYYY/MM/DD/
I would like to change the folder structure to:
/HRIT/YYYY/MM/DD/
All files from EPI, HRV, VIS006, VIS008, WV_062, WV_073, IR_016, IR_039, IR_087,IR_097, IR_108, IR_124, IR_134, PRO should not be copied physically but should be accessible by links in my home folder /home
-> /home/HRIT/YYYY/MM/DD/
Is something like that possible and how?
Here is an minimal example what I in principle have and what I wish as result:
I have:
/HRIT/EPI/2019/01/01/epi_1.txt
/HRIT/EPI/2019/01/01/epi_2.txt
/HRIT/HRV/2019/01/01/hrv_1.txt
/HRIT/HRV/2019/01/01/hrv_2.txt
/HRIT/VIS006/2019/01/01/vis006_1.txt
/HRIT/VIS006/2019/01/01/vis006_2.txt
As result I wish:
/home/HRIT/2019/01/01/epi_1.txt
/home/HRIT/2019/01/01/epi_2.txt
/home/HRIT/2019/01/01/hrv_1.txt
/home/HRIT/2019/01/01/hrv_2.txt
/home/HRIT/2019/01/01/vis006_1.txt
/home/HRIT/2019/01/01/vis006_2.txt
As mentioned above the files should not be copied to this new folder structure, but instead made accessible by links (because in reality I have too many files and not enough space to copy them).
!!! This is extremely simplified, since I have different years, month's and days (please see the link above).
Here's a small script to do this. I've made it just echo by default, you'd need to run it with --real to make it do the linking
This assumes you're running the script from within the /HRIT/ dir, and assumes that the date, then filename are the last part of the hierarchy. If there are further dirs below, this might not work for you.
DRYRUN=true
LINK="ln -s" # You can make this "ln" if you prefer hardlinks
if [[ $1 == '--real' ]]; then
DRYRUN=false
fi
for file in $(find "$PWD" -type f); do
dest=$( \
echo "$file" | awk -F/ '{ year = NF-3; mon=NF-2; day=NF-1; print "/home/HRIT/"$year"/"$mon"/"$day"/"$NF }' \
)
if [[ -f "$dest" ]]; then
echo "Skipping '$file' as destination link already exists at '$dest'"
else
if $DRYRUN; then
echo $LINK "$file" "$dest"
else
$LINK "$file" "$dest"
fi
fi
done
The simplest while read loop:
generate_list_of_files_for_example_with_find |
while IFS= read -r line; do
IFS='/' read -r _ hrit _ rest <<<"$line"
echo mkdir -p /home/"$hrit"/"$(dirname "$rest")"
echo ln -s /home/"$hrit/$rest" "$line"
done
Remove echos to really create links and directories.
I have the following bash script
#!/bin/bash
for i in `ls /file-directory/ | grep -v static-backup | grep -v fileGroup1 | grep -v fileGroup2`
do
echo $i
rsync --delete -avz --size-only --exclude "$i/stuff1" --exclude "$i/stuff2" --exclude "$i/stuff3" --exclude "$i/stuff4" --exclude "$i/stuff5" --exclude "$i/stuff6" /file-directory/$i otherServer:/file-directory/ && echo " exit code: " $?" $i" || echo " exit code: " $?" $i"
done
The script iterates through a file directory and rsyncs its subdirectories, excluding certain fileGroups and portions of those filegroup's directories. I would like this script to spawn two rsync jobs, one that starts at the top of the directory and another that starts at the bottom. They would iterate in opposite directions and meet in the middle.
This is relatively simple to do with normal counting for loops, and wouldn't bee to hard in something like python (you could just save the number of directories as a variable, then iterate using that var). How can I do something similar in bash?
You can make use of & in bash to fork a process. or even loop.
I am sure there are more than one way to achieve this, but here's one way.
Pseudo code.
Get listing of files in two separate arrays such that they each contain half of listing (for even).
For odd files , second array will contain one more file.
Loop through first array and fork a process using &
Loop through second array and fork second process using &
Code
#!/bin/bash
dir_len=$(ls /file-directory/ |wc -l|awk '{print $NF}')
midpoint=$(($dir_len / 2))
array1=()
array2=()
count=0
#Loop through files and divide contents into two arrays
for i in $(ls /file-directory/)
do
count=$(($count + 1))
if [ $count -le $midpoint ] ; then
array1+=($i)
elif [[ $count -gt $midpoint && $count -le $dir_len ]] ; then
array2+=($i)
fi
done
#Loop through first array and fork
for i in "${array1[#]}"
do
echo $i
#your rsync command here
done&
#Loop through second array and fork
for j in "${array2[#]}"
do
echo $j
#your rsync command here
done &
Note:
& is optional in second for loop as u can run it in foreground since there are only two processes.
Also, you may want harden the script for undesired files or if folder is empty.
Edit.
Based upon type of files u need to rsync, refer to this link for inspiration instead of using ‘ls /folder_name/‘.
https://unix.stackexchange.com/questions/9496/looping-through-files-with-spaces-in-the-names
My pictures are kept in the folder with the picture-date for folder name, for example the original path and file names:
.../Pics/2016_11_13/wedding/DSC0215.jpg
.../Pics/2016_11_13/afterparty/DSC0234.jpg
.../Pics/2016_11_13/afterparty/DSC0322.jpg
How do I rename the pictures into the format below, with continuous sequences and 4-digit padding?
.../Pics/2016_11_13_wedding.0001.jpg
.../Pics/2016_11_13_afterparty.0002.jpg
.../Pics/2016_11_13_afterparty.0003.jpg
I'm using Bash 4.1, so only mv command is available. Here is what I have now but it's not working
#!/bin/bash
p=0
for i in *.jpg;
do
mv "$i" "$dirname.%03d$p.JPG"
((p++))
done
exit 0
Let say you have something like .../Pics/2016_11_13/wedding/XXXXXX.jpg; then go in directory .../Pics/2016_11_13; from there, you should have a bunch of subdirectories like wedding, afterparty, and so on. Launch this script (disclaimer: I didn't test it):
#!/bin/sh
for subdir in *; do # scan directory
[ ! -d "$subdir" ] && continue; # skip non-directory
prognum=0; # progressive number
for file in $(ls "$dir"); do # scan subdirectory
(( prognum=$prognum+1 )) # increment progressive
newname=$(printf %4.4d $prognum) # format it
newname="$subdir.$newname.jpg" # compose the new name
if [ -f "$newname" ]; then # check to not overwrite anything
echo "error: $newname already exist."
exit
fi
# do the job, move or copy
cp "$subdir/$file" "$newname"
done
done
Please note that I skipped the "date" (2016_11_13) part - I am not sure about it. If you have a single date, then it is easy to add these digits in # compose the new name. If you have several dates, then you can add a nested for for scanning the "date" directories. One more reason I skipped this, is to let you develop something by yourself, something you can be proud of...
Using only mv and bash builtins:
#! /bin/bash
shopt -s globstar
cd Pics
p=1
# recursive glob for .jpg files
for i in **/*.jpg
do
# (date)/(event)/(filename).jpg
if [[ $i =~ (.*)/(.*)/(.*).jpg ]]
then
newname=$(printf "%s_%s.%04d.jpg" "${BASH_REMATCH[#]:1:2}" "$p")
echo mv "$i" "$newname"
((p++))
fi
done
globstar is a bash 4.0 feature, and regex matching is available even in OSX's anitque bash.
I need to create a text file in each subdirectory of all files in the list.
For example, subdirectory1 would contain a list of all of its files as a .txt and subdirectory2 would also contain a list of all of subdirectory2 files as a .txt.
I have tried
#!/bin/bash
for X in "$directory" *
do
if [ -d "$X" ];
then
cd "$X"
files="$(ls)"
echo "$files" >> filesNames.txt
fi
done
However this did not generate anything. I absolutely need it as a shell script because it will be part of a pipeline script, but I cannot seem to get it to work.
Here is the adjusted script giving me the no such file or directory comment. I know that the folder exists and have used it in commands that are run before this command.
#!/bin/bash
#Retrieve the base directory path
baseDir=$(dirname "$ini")
#Retrieve the reference genome path
ref=$(dirname "$genome")
#Create required directory structure
tested="$baseDir/tested"
MarkDups1="$baseDir/MarkDups1"
#don't create if already exists
[[ -d "tested" ]] || mkdir "$tested"
[[ -d "MarkDups1" ]] || mkdir "$MarkDups1"
#create a text file with all sorted and indexed bam files paths
#!/bin/bash
for x in $MarkDups1/*/;
do
(cd "$x"; ls > filesNames.txt)
done
The sequence to iterate over should be "$directory"/*/.
for x in "$directory"/*/; do
(cd "$x"
files=(*)
printf '%s\n' "${files[#]}" > filesNames.txt
)
done
I would like to write a linux script that will move or copy all files with the same filename (but different extensions) to a new filename for all those files, while maintaining their different extensions. In other words:
if I start with a directory listing:
file1.txt, file1.jpg, file1.doc, file12.txt, file12.jpg, file12.doc
I would like to write a script to change all the filenames without changing the extensions. For the same example, choosing file2 as the new filename the result would be:
file2.txt, file2.jpg and file2.doc, file12.txt, file12.jpg, file12.doc
So the files whose filename do not match the current criteria will not be changed.
Best wishes,
George
Note: If there's file1.doc in variable i, expression ${i##*.} extracts extension i.e. doc in this case.
One line solution:
for i in file1.*; do mv "$i" "file2.${i##*.}"; done
Script:
#!/bin/sh
# first argument - basename of files to be moved
# second arguments - basename of destination files
if [ $# -ne 2 ]; then
echo "Two arguments required."
exit;
fi
for i in $1.*; do
if [ -e "$i" ]; then
mv "$i" "$2.${i##*.}"
echo "$i to $2.${i##*.}";
fi
done
The util-linux-ng package (most of linux flavours have it installed by default) has the command 'rename'. See man rename for use instructions. Using it your task can be done simply as that rename file1 file2 file1.*
To handle input files whose basenames contain special characters, I would modify plesiv's script to the following:
if [ $# -ne 2 ]; then
echo "Two arguments required."
exit;
fi
for i in "$1".*; do
if [ -e "$i" ]; then
mv "$i" "$2.${i##*.}"
echo "$i to $2.${i##*.}";
fi
done
Note the extra quotes around $1.