I have more than 500 Mp4 files in my server 1
so i want half of them to send to server 2 and half of them to server 3
but i dont know how to make this
Is there a way to select files by alphabet or maybe date or something else
example videos that start with
a,c,e*.mp4
will send to server 2 and videos that start with
b,d,f*.mp4
will send to server 3
or is there any other way you think is better
rsync -avzP /home/user/public_html/domain.com/ ip:/home/user2/public_html/domain.com/
1) use find to make a list of all the files
find /opt/mymp3folder -print > /tmp/foo
2) find the count of lines and split the list in two
wc -l /tmp/foo
387
split -l 200 /tmp/foo
mv xaa xaa.txt
and then rsync like this
rsync -avzP -e ssh `cat xaa.txt` root#0.0.0.0:/var/www/
I think that is better to split files by size than for numbers (I assume that you have several file sizes in your mp4).
#!/bin/bash
FOLDER=$1
TMP_FILE=$(mktemp)
find $FOLDER -type f -exec stat -c "%s;%n" {} \; | sort -t ';' -k 2 | awk 'BEGIN{ sum=0; FS=";"} { sum += $1; print sum";"$1";"$2 }' > $TMP_FILE
TOTAL_SIZE=$(tail -n 1 $TMP_FILE | cut -f 1 -d ';')
HALF_SIZE=$(echo $TOTAL_SIZE / 2 | bc)
echo $TOTAL_SIZE $HALF_SIZE
# split part
IFS=';'
while read A B C ; do
[ $A -lt $HALF_SIZE ] && echo "$C" >> lst_files_1.txt || echo "$C" >> lst_files_2.txt
done < $TMP_FILE
rsync -avzP
rm $TMP_FILE
After execution you have list_files_1.txt and list_files_2.txt that contains half of files depending of size.
You can send this files to each server using rsync:
rsync -avzP $(cat list_files_1.txt) ip:/home/user2/public_html/domain.com/
1) use find to make a list of all the files
find /opt/mymp3folder -print > /tmp/foo
2) find the count of lines and split the list in two
cd /tmp
wc -l /tmp/foo
387
split -l 200 /tmp/foo
3) split by default makes a set of files called xaa xab xac etc. So use xaa to copy to one server and xab to copy to the other
rsync -av --files-from=/tmp/xaa . server1:/opt/newmp3folder/
rsync -av --files-from=/tmp/xab . server2:/opt/newmp3folder/
'.' in the above is the "source" path and allows the use of relative paths in the "files-from" You either need to be in the same path that the find command is run from and use . or set it to an absolute value
Obviously if you wanted to do this on a regular basis probably want to script it properly
Related
So guys,
I need your help trying to identify the fastest and the most "fault" tolerant solution to my problem.
I have a shell script which executes some functions, based on a txt file, in which I have a list of files.
The list can contain from 1 file to X files.
What I would like to do is iterate over the content of the file and execute my scripts for only 4 items out of the file.
Once the functions have been executed for these 4 files, go over to the next 4 .... and keep on doing so until all the files from the list have been "processed".
My code so far is as follows.
#!/bin/bash
number_of_files_in_folder=$(cat list.txt | wc -l)
max_number_of_files_to_process=4
Translated_files=/home/german_translated_files/
while IFS= read -r files
do
while [[ $number_of_files_in_folder -gt 0 ]]; do
i=1
while [[ $i -le $max_number_of_files_to_process ]]; do
my_first_function "$files" & # I execute my translation function for each file, as it can only perform 1 file per execution
find /home/german_translator/ -name '*.logs' -exec mv {} $Translated_files \; # As there will be several files generated, I have them copied to another folder
sed -i "/$files/d" list.txt # We remove the processed file from within our list.txt file.
my_second_function # Without parameters as it will process all the files copied at step 2.
done
# here, I want to have all the files processed and don't stop after the first iteration
done
done < list.txt
Unfortunately, as I am not quite good at shell scripting, I do not know how to structure it so that it won't waste any resources and mostly, to make sure that it "processes" everything from that file.
Do you have any advice on how to achieve what I am trying to achieve?
only 4 items out of the file. Once the functions have been executed for these 4 files, go over to the next 4
Seems to be quite easy with xargs.
your_function() {
echo "Do something with $1 $2 $3 $4"
}
export -f your_function
xargs -d '\n' -n 4 bash -c 'your_function "$#"' _ < list.txt
xargs -d '\n' for each line
-n 4 take for arguments
bash .... - run this command with 4 arguments
_ - the syntax is bash -c <script> $0 $1 $2 etc..., see man bash.
"$#" - forward arguments
export -f your_function - export your function to environment so child bash can pick it up.
I execute my translation function for each file
So you execute your translation function for each file, not for each 4 files. If the "translation function" is really for each file with no inter-file state, consider rather executing 4 processes in parallel with same code and just xargs -P 4.
If you have GNU Parallel it looks something like this:
doit() {
my_first_function "$1"
my_first_function "$2"
my_first_function "$3"
my_first_function "$4"
my_second_function "$1" "$2" "$3" "$4"
}
export -f doit
cat list.txt | parallel -n4 doit
I have a problem working with GNU Nano program code. This is my task:
Generate 100 files and in each one has to be one number(shuf -i1-1000 - n1). Then scan files and write numbers ascending order to a file named "output.txt".
My code:
#!/bin/bash
mkdir files
find /etc/ -name "*.txt"|xargs du -h >output.txt
for x in {1..100}
do
shuf -i 1-1000 -n 1 > files/$x.txt
done
for x in {1..100}
do
input=$(cat files/$x.txt)
done
I wanted to ask how to sort out numbers which are in files and write them all to output.txt file?
Thanks
Use sort to sort the numbers.
#! /bin/bash
mkdir files
shuf -i1-1000 -n100 | for i in {1..100} ; do
read n
echo $n > files/$i.txt
done
sort -n files/*.txt > files/output.txt
I have a script that queries the Twitter API for several queries, and then writes the raw data to a file with the query in the name, plus a timestamp. I'd like to have a script that, given the list of query strings (regexs?) and for all files in a folder, if one of the query strings is a substring in that file, move it to a specific folder. Right now I have just a script with just a few dozen mv commands, but I'd like a simpler and more maintainable version. Here's an example of what I'm doing now:
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*femin*/home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*patriarchy* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*yesallwomen* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*womanpower* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
I would use a for loop:
for i in femin patriarchy yesallwomen womanpower; do
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*$i* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
done
That way the list is in the first line so it is easy to amend.
I would isolate data (the words to be moved to feminism) and code.
When you have more keywords (feminism and so), you can make files with keywords and check these keywordfiles for the files you are considering to move.
With ${fromdir} where the files come from, ${todir} where you want them and ${keyfiledir} with the keywords, you get something like
for keyfile in ${keyfiledir}/*; do
key="${keyfile##*/}"
find $from -type f | sed 's#.*/##' | while read -r file; do
echo "${file}" | grep -q -f "${keyfiledir}"/"${key}" && mv "${from}"/"${file}" "${to}"/"${key}"
done
done
How does that work? I tested the solution above with the following script.
from=fromdir
to=todir
keyfiledir=keyfiledir
rm -rf ${from} ${to} ${keyfiledir}
mkdir ${from} ${to} ${keyfiledir}
mkdir ${to}/feminism ${to}/so
touch ${from}/yesallwomen ${from}/women ${from}/some_femin ${from}/"help move"
cat <<# > ${keyfiledir}/feminism
femin
patriarchy
yesallwomen
womanpower
#
touch ${from}/yesallwomen ${from}/women ${from}/some_femin
cat <<# > ${keyfiledir}/so
stack
exchange
help
#
test ! -d "${from}" && echo " Wrong dir ${from}" && exit 1
test ! -d "${to}" && echo " Wrong dir ${to}" && exit 1
test ! -d "${keyfiledir}" && echo " Wrong dir ${keyfiledir}" && exit 1
for keyfile in ${keyfiledir}/*; do
key="${keyfile##*/}"
find $from -type f | sed 's#.*/##' | while read -r file; do
echo "${file}" | grep -q -f "${keyfiledir}"/"${key}" && mv "${from}"/"${file}" "${to}"/"${key}"
done
done
echo "Not moved"
ls ${from}
echo "Moved"
ls -R ${to}
A simple combination of mv and egrep should suffice. egrep can take a pattern list from a file (and then you get to use full regexp syntax, not just glob syntax.) Make sure to exclude the name of the target folder.
cd /home/nick/TwitterSearchToDatabase/queries_for_amita
mv $(ls | egrep -f patterns.txt | grep -v '^feminism$') feminism
I am working on a project for one of my professors and he asked me to sort a couple hundred .fits images based on their header files (specifically what star they are images of) I think that grep would be the best way to do this however I can't seam to figure out how to use grep based on the header.
I am entering:
ls | imhead *.fits | grep -E -r "PG\ 1104+243" *
to just list them out for now, once they are listed I know how to copy them into a directory.
I am new to using grep so I am unsure as to where my error lies? any help would be greatly appreciated! Thanks!
Assuming that imghead will extract the headers of the .fits as txt, you can use a simple shell script to do it:
script.sh
#!/bin/bash
grep "$1" "$2" > /dev/null 2>&1 && echo "$2"
Note that the + is a special character if you use extended regular expression, meaning if you pass the -E as in the question. A simple grep without any options should do the trick here.
Use find to exec the script on every *.fits file in the current folder:
find -maxdepth 1 -name '*.fits' -exec ./script.sh 'PG 1104+243' {} \;
If you are going to copy/move/alter or do something with the files you find, you might be better off, in terms of complexity and ease of quoting, using a loop like this:
#!/bin/bash
find . -name \*.fits -print0 | while read -d '' -r file; do
echo Checking file: $file
imhead "$file" | grep -q 'PG 1104+243'
if [ $? -eq 0 ]; then
echo Object matches: $file
fi
done
Is there a one-line command/script to copy one file to many files on Linux?
cp file1 file2 file3
copies the first two files into the third. Is there a way to copy the first file into the rest?
Does
cp file1 file2 ; cp file1 file3
count as a "one-line command/script"? How about
for file in file2 file3 ; do cp file1 "$file" ; done
?
Or, for a slightly looser sense of "copy":
tee <file1 file2 file3 >/dev/null
just for fun, if you need a big list of files:
tee <sourcefile.jpg targetfiles{01-50}.jpg >/dev/null- Kelvin Feb 12 at 19:52
But there's a little typo. Should be:
tee <sourcefile.jpg targetfiles{01..50}.jpg >/dev/null
And as mentioned above, that doesn't copy permissions.
You can improve/simplify the for approach (answered by #ruakh) of copying by using ranges from bash brace expansion:
for f in file{1..10}; do cp file $f; done
This copies file into file1, file2, ..., file10.
Resource to check:
http://wiki.bash-hackers.org/syntax/expansion/brace#ranges
for FILE in "file2" "file3"; do cp file1 $FILE; done
You can use shift:
file=$1
shift
for dest in "$#" ; do
cp -r $file $dest
done
cat file1 | tee file2 | tee file3 | tee file4 | tee file5 >/dev/null
(no loops used)
To copy the content of one file (fileA.txt) to many files (fileB.txt, fileC.txt, fileD.txt) in Linux,
use the following combination cat and tee commands:
cat fileA.txt | tee fileB.txt fileC.txt fileD.txt >/dev/null
applicable to any file extensions
only file names and extensions change, everything else remains same.
Use something like the following. It works on zsh.
cat file > firstCopy > secondCopy > thirdCopy
or
cat file > {1..100} - for filenames with numbers.
It's good for small files.
You should use the cp script mentioned earlier for larger files.
I'd recommend creating a general use script and a function (empty-files), based on the script, to empty any number of target files.
Name the script copy-from-one-to-many and put it in your PATH.
#!/bin/bash -e
# _ _____
# | |___ /_ __
# | | |_ \ \/ / Lex Sheehan (l3x)
# | |___) > < https://github.com/l3x
# |_|____/_/\_\
#
# Copy the contents of one file to many other files.
source=$1
shift
for dest in "$#"; do
cp $source $dest
done
exit
NOTES
The shift above removes the first element (the source file path) from the list of arguments ("$#").
Examples of how to empty many files:
Create file1, file2, file3, file4 and file5 with content:
for f in file{1..5}; do echo $f > "$f"; done
Empty many files:
copy-from-one-to-many /dev/null file1 file2 file3 file4 file5
Empty many files easier:
# Create files with content again
for f in file{1..5}; do echo $f > "$f"; done
copy-from-one-to-many /dev/null file{1..5}
Create empty_files function based on copy-from-one-to-many
function empty-files()
{
copy-from-one-to-many /dev/null "$#"
}
Example usage
# Create files with content again
for f in file{1..5}; do echo $f > "$f"; done
# Show contents of one of the files
echo -e "file3:\n $(cat file3)"
empty_files file{1..5}
# Show that the selected file no longer has contents
echo -e "file3:\n $(cat file3)"
Don't just steal code. Improve it; Document it with examples and share it. - l3x
Here's a version that will preface each cp command with sudo:
#!/bin/bash -e
# Filename: copy-from-one-to-may
# _ _____
# | |___ /_ __
# | | |_ \ \/ / Lex Sheehan (l3x)
# | |___) > < https://github.com/l3x
# |_|____/_/\_\
#
# Copy the contents of one file to many other files.
# Pass --sudo if you want each cp to be perfomed with sudo
# Ex: copy-from-one-to-many $(mktemp) /tmp/a /tmp/b /tmp/c --sudo
if [[ "$*" == *--sudo* ]]; then
maybe_use_sudo=sudo
fi
source=$1
shift
for dest in "$#"; do
if [ $dest != '--sudo' ]; then
$maybe_use_sudo cp $source $dest
fi
done
exit
You can use standard scripting commands for that instead:
Bash:
for i in file2 file3 ; do cp file1 $i ; done
The simplest/quickest solution I can think of is a for loop:
for target in file2 file3 do; cp file1 "$target"; done
A dirty hack would be the following (I strongly advise against it, and only works in bash anyway):
eval 'cp file1 '{file2,file3}';'
Go with the fastest cp operations
seq 1 10 | xargs -P 0 -I xxx cp file file-xxx
it means
seq 1 10 count from 1 to 10
| pipe it xargs
-P 0 do it in parallel - as many as needed
-I xxx name of each input xargs receives
cp file file-xxx means copy file to file-1, file-2, etc
and if name of files are different here is the other solutions.
First have the list of files which are going to be created. e.g.
one
two
three
four
five
Second save this list on disk and read the list with xargs just like before but without using seq.
xargs -P 0 -I xxx cp file xxx < list
which means 5 copy operations in parallel:
cp file one
cp file two
cp file three
cp file four
cp file five
and for xargs here is the behind the scene (5 forks)
3833 pts/0 Ss 0:00 bash
15954 pts/0 0:00 \_ xargs -P 0 -I xxx cp file xxx < list
15955 pts/0 0:00 \_ cp file one
15956 pts/0 0:00 \_ cp file two
15957 pts/0 0:00 \_ cp file three
15958 pts/0 0:00 \_ cp file four
15959 pts/0 0:00 \_ cp file five
I don't know how correct this is but i have used something like this
echo ./file1.txt ./file2.txt ./file3.txt | xargs -n 1 cp file.txt
Where echo ./file1.txt ... is destination of a file and use it to feed xargs with one "destination" by one. Therefore command xargs -n 1. And lastly cp file.txt, which is self explanatory i think :)