I have a directory containing multiple directories. here is an example of the list of directories:
dir1_out
dir2_out
dir3_out
dir4_out
Each directory contains multiple files.
For example folder1_out contains the following files:
file1
file2
file3
In the same fashion other directories contain several folders.
I would like to add the name of each directory to file name in the corresponding directory.
I would like to have the following result in first directory(dir1_out):
dir1.file1
dir1.file2
dir1.file3
Since I have around 50 directories I would like to write a loop that takes the name of each directory and add the name to the beginning of all subfiles.
Do you have any idea how can I do that in linux.
A simple bash onliner if there aren't too many files is:
for p in */*; do [ -f "$p" ] && mv -i "$p" "${p%/*}/${p/\//.}"; done
This uses parameter expansions to generate new filenames, after checking that we are trying to rename an actual file - See bash manpage descriptions of ${parameter%word} and ${parameter/pattern/string}
If there may be too many files to safely expand them all into a single list:
#!/bin/bash
find . -maxdepth 2 -print |\
while read p; do
p="${p#./}"
mv -i "$p" "${p%/*}/${p/\//.}"
done
Related
I have directory structure like this.
From this I want to create different zip files such as
data-A-A_1-A_11.zip
data-A-A_1-A_12.zip
data-B-B_1-B_11.zip
data-C-C_1-C_11.zip
while read line;
do
echo "zip -r ${line//\//-}.zip $line";
# zip -r "${line//\//-}.zip" "$line"
done <<< "$(find data -maxdepth 3 -mindepth 2 -type d)"
Redirect the result of a find command into a while loop. The find command searches the directory data for directories only, searching 3 directories deep only. In the while loop with use bash expansion to convert all forward slashes to "-" and add ".zip" in such a way that we can build a zip command on each directory. Once you are happy that the zip command looks fine when echoed for each directory, comment in the actual zip command
I have many subdirectories and files in the folder mydata/files. I want to take files and copy them randomly into 3 folders:
train
test
dev
For example, mydata/files/ss/file1.wav could be copied into train folder:
train
file1.wav
And so on and so forth, until all files from mydata/files are copied.
How can I do it using Bash script?
Steps to solve this:
Need to gather all the files in the directory
Assign directories to a map
Generate random number for each file
Move the file to the corresponding directory
The script:
#!/bin/bash
original_dir=test/
## define 3 directories to copy into
# define an associative array (like a map)
declare -A target_dirs
target_dirs[0]="/path/to/train/"
target_dirs[1]="/path/to/test/"
target_dirs[2]="/path/to/dev/"
# recursively find all the files, and loop through them
find $original_dir -type f | while read -r file ; do
# find a random number 0 - (size of target_dirs - 1)
num=$(($RANDOM % ${#target_dirs[#]}))
# get that index in the associative array
target_dir=${target_dirs[$num]}
# copy the file to that directory
echo "Copying $file to $target_dir"
cp $file $target_dir
done
Things you'll need to change:
Change the destination of the directories to match the path in your system
Add executable priviledges to the file so that you can run it.
chmod 744 copy_script_name
./copy_script_name
Notes:
This script should easily be extendable to any number of directories if needed (just add the new directories, and the script will adjust the random numbers.
If you need to only get the files in the current directory (not recursively), you can add
-maxdepth 1 (see How to list only files and not directories of a directory Bash?).
Was able to leverage previous bash experience plus looking at bash documentation (it's generally pretty good). If you end up writing any scripts, be very careful about spaces
You can create a temp file, echo your destination folder to it, then use the shuf command.
dest=$(mktemp)
echo -e "test\ndev\ntrain" >> $dest
while IFS= read -r file; do
mv "$file" "$(shuf -n1 < $dest)/."
done < <(find mydata/files -type f 2>/dev/null)
rm -f "$dest"
I have created a file (search.txt) containing file names of .fasta files I want to copy from the current directory (which also contains many unwanted .fasta files). Is it possible to use this text file to find and copy the matching files in the current directory to a new location?
The search.txt file contains a list of names like this:
name_1
name_2
name_3
I tried to build the search term using find and grep, like this:
find . *.fasta | grep -f search.txt
which is returning output like this for each matching file:
./name_1.fasta
./name_2.fasta
./name_3.fasta
name_1.fasta
name_2.fasta
name_3.fasta
It's finding the correct files, but I'm not sure if this output is useful / can be used to copy these files?
To get only matching filenames from search.txt I would do this:
find . -type f -name '*.fasta' -print0 | grep -zf search.txt | xargs -r0 cp -t target-dir/
It will find all files with the extension .fasta, display only the ones with matching patterns in search.txt, and bulk cp them to target-dir, and each filename is terminated with a nullbyte in case filenames contain newlines.
Using Bash, you can read all the files from the list into an array:
$ mapfile -t files < search.txt
$ declare -p files
declare -a files=([0]="name_1" [1]="name_2" [2]="name_3")
Then, you can append the desired file extension to all array elements:
$ files=("${files[#]/%/.fasta}")
$ declare -p files
declare -a files=([0]="name_1.fasta" [1]="name_2.fasta" [2]="name_3.fasta")
And finally, move them to the desired location:
$ mv "${files[#]}" path/to/new/location
You don't actually need the intermediate step:
mapfile -t files < search.txt
mv "${files[#]/%/.fasta}" path/to/new/location
I'm trying to learn bash on Linux, just for fun. I thought it would be pretty useful to have a .sh that would group together similar files. For example, let's say we have the directory
/home/docs/
Inside the directory we have /mathdocs/, /codingdocs/, etc.
Inside those sub-directories we have doc.txt, in all of them. Same name for all the files on the subdirectories.
Let's say I want to group them together, and I want to move all the files to /home/allthedocs/ and rename them after the directories they were in. (mathdocs.txt, codingdocs.txt, etc.)
How could I do that?
I've tried to create a script based on the ls and cp commmands, but I don't know how I can take the name of the directories to rename the files in it after I moved them. I guess it has to be some sort of iterative sentence (for X on Y directories) but I don't know how to do it.
You can move and rename your file in one shot with mv, with a loop that grabs all your files through a glob:
#!/bin/bash
dest_dir=/home/allthedocs
cd /home/docs
for file in */doc.txt; do
[[ -f "$file" ]] || continue # skip if not a regular file
dir="${file%/*}" # get the dir name from path
mv "$file" "$dest_dir/$dir.txt"
done
See this post for more info:
Copying files from multiple directories into a single destination directory
Here is a one liner solution that treats whitespaces in filenames, just as #codeforester 's solution does with the glob.
Note that white spaces are treated with the "-print0" option passed to "find", the internal field separator (IFS) in while loop and the wrapping of file3 variable with quotes.
The parameter substitution from file2 into file3 gets rid of the leading "./".
The parameter substition inside the move command turns the path into a filename (run under /home/docs/):
find . -maxdepth 2 -mindepth 2 -print0 | while IFS= read -r -d '' file; \
do file2=$(printf '%s\n' "$file"); file3=${file2#*\/*}; \
mv "$file2" ../allsil/"${file3//\//}"; done
I am trying to take some directories that and transfer them from Linux to Windows. The problem is that the files on Linux have colons in them. And I need to copy these directories (I cannot alter them directly since they are needed as they are the server) over to files with a name that Windows can use. For example, the name of a directory on the server might be:
IAPLTR2b-ERVK-LTR_chr9:113137544-113137860_-
while I need it to be:
IAPLTR2b-ERVK-LTR_chr9-113137544-113137860_-
I have about sixty of these directories and I have collected the names of the files with their absolute paths in a file I call directories.txt. I need to walk through this file changing the colons to hyphens. Thus far, my attempt is this:
#!/bin/bash
$DIRECTORIES=`cat directories.txt`
for $i in $DIRECTORIES;
do
cp -r "$DIRECTORIES" "`echo $DIRECTORIES | sed 's/:/-/'`"
done
However I get the error:
./my_shellscript.sh: line 10: =/bigpartition1/JKim_Test/test_bs_1/129c-test-biq/IAPLTR1_Mm-ERVK-LTR_chr10:104272652-104273004_+.fasta: No such file or directory ./my_shellscript.sh: line 14: `$i': not a valid identifier
Can anyone here help me identify what I am doing wrong and maybe what I need to do?
Thanks in advance.
This monstrosity will rename the directories in situ:
find tmp -depth -type d -exec sh -c '[ -d "{}" ] && echo mv "{}" "$(echo "{}" | tr : _)"' \;
I use -depth so it descends down into the deepest subdirectories first.
The [ -d "{}" ] is necessary because as soon as the subdirectory is renamed, its parent directory (as found by find) may no longer exist (having been renamed).
Change "echo mv" to "mv" if you're satisfied it will do what you want.