I have a problem working with GNU Nano program code. This is my task:
Generate 100 files and in each one has to be one number(shuf -i1-1000 - n1). Then scan files and write numbers ascending order to a file named "output.txt".
My code:
#!/bin/bash
mkdir files
find /etc/ -name "*.txt"|xargs du -h >output.txt
for x in {1..100}
do
shuf -i 1-1000 -n 1 > files/$x.txt
done
for x in {1..100}
do
input=$(cat files/$x.txt)
done
I wanted to ask how to sort out numbers which are in files and write them all to output.txt file?
Thanks
Use sort to sort the numbers.
#! /bin/bash
mkdir files
shuf -i1-1000 -n100 | for i in {1..100} ; do
read n
echo $n > files/$i.txt
done
sort -n files/*.txt > files/output.txt
Related
So guys,
I need your help trying to identify the fastest and the most "fault" tolerant solution to my problem.
I have a shell script which executes some functions, based on a txt file, in which I have a list of files.
The list can contain from 1 file to X files.
What I would like to do is iterate over the content of the file and execute my scripts for only 4 items out of the file.
Once the functions have been executed for these 4 files, go over to the next 4 .... and keep on doing so until all the files from the list have been "processed".
My code so far is as follows.
#!/bin/bash
number_of_files_in_folder=$(cat list.txt | wc -l)
max_number_of_files_to_process=4
Translated_files=/home/german_translated_files/
while IFS= read -r files
do
while [[ $number_of_files_in_folder -gt 0 ]]; do
i=1
while [[ $i -le $max_number_of_files_to_process ]]; do
my_first_function "$files" & # I execute my translation function for each file, as it can only perform 1 file per execution
find /home/german_translator/ -name '*.logs' -exec mv {} $Translated_files \; # As there will be several files generated, I have them copied to another folder
sed -i "/$files/d" list.txt # We remove the processed file from within our list.txt file.
my_second_function # Without parameters as it will process all the files copied at step 2.
done
# here, I want to have all the files processed and don't stop after the first iteration
done
done < list.txt
Unfortunately, as I am not quite good at shell scripting, I do not know how to structure it so that it won't waste any resources and mostly, to make sure that it "processes" everything from that file.
Do you have any advice on how to achieve what I am trying to achieve?
only 4 items out of the file. Once the functions have been executed for these 4 files, go over to the next 4
Seems to be quite easy with xargs.
your_function() {
echo "Do something with $1 $2 $3 $4"
}
export -f your_function
xargs -d '\n' -n 4 bash -c 'your_function "$#"' _ < list.txt
xargs -d '\n' for each line
-n 4 take for arguments
bash .... - run this command with 4 arguments
_ - the syntax is bash -c <script> $0 $1 $2 etc..., see man bash.
"$#" - forward arguments
export -f your_function - export your function to environment so child bash can pick it up.
I execute my translation function for each file
So you execute your translation function for each file, not for each 4 files. If the "translation function" is really for each file with no inter-file state, consider rather executing 4 processes in parallel with same code and just xargs -P 4.
If you have GNU Parallel it looks something like this:
doit() {
my_first_function "$1"
my_first_function "$2"
my_first_function "$3"
my_first_function "$4"
my_second_function "$1" "$2" "$3" "$4"
}
export -f doit
cat list.txt | parallel -n4 doit
I have a text file which contains multiple paths like below
$ cat directory.txt
/aaaa/bbbbb/ccccc/
/aaaa/bbbbb/eeeee/
/aaaa/bbbbb/ddddd/
I need to change directory to each path in text file and need to get count of files under that paths.Below is the code i used, But it is not working.
i=cat /aaaa/bbbbb/directory.txt
while read $i ;do
cd $i
ls |wc -l
done < /aaaa/bbbbb/count.txt
Actually you're almost there. The line i=... is not needed, read $i should be read i, and you simply need to call ls with the path instead of cd it first.
#!/bin/bash
while read i; do
ls "$i" | wc -l
done < "/xxx/yyy/count.txt"
Thanks every one i tried this code it is working fine
!/bin/bash
for i in cat /nrt/home/directory.txt;
do
cd $i
ls | wc -l
done > /nrt/home/count.txt
I have more than 500 Mp4 files in my server 1
so i want half of them to send to server 2 and half of them to server 3
but i dont know how to make this
Is there a way to select files by alphabet or maybe date or something else
example videos that start with
a,c,e*.mp4
will send to server 2 and videos that start with
b,d,f*.mp4
will send to server 3
or is there any other way you think is better
rsync -avzP /home/user/public_html/domain.com/ ip:/home/user2/public_html/domain.com/
1) use find to make a list of all the files
find /opt/mymp3folder -print > /tmp/foo
2) find the count of lines and split the list in two
wc -l /tmp/foo
387
split -l 200 /tmp/foo
mv xaa xaa.txt
and then rsync like this
rsync -avzP -e ssh `cat xaa.txt` root#0.0.0.0:/var/www/
I think that is better to split files by size than for numbers (I assume that you have several file sizes in your mp4).
#!/bin/bash
FOLDER=$1
TMP_FILE=$(mktemp)
find $FOLDER -type f -exec stat -c "%s;%n" {} \; | sort -t ';' -k 2 | awk 'BEGIN{ sum=0; FS=";"} { sum += $1; print sum";"$1";"$2 }' > $TMP_FILE
TOTAL_SIZE=$(tail -n 1 $TMP_FILE | cut -f 1 -d ';')
HALF_SIZE=$(echo $TOTAL_SIZE / 2 | bc)
echo $TOTAL_SIZE $HALF_SIZE
# split part
IFS=';'
while read A B C ; do
[ $A -lt $HALF_SIZE ] && echo "$C" >> lst_files_1.txt || echo "$C" >> lst_files_2.txt
done < $TMP_FILE
rsync -avzP
rm $TMP_FILE
After execution you have list_files_1.txt and list_files_2.txt that contains half of files depending of size.
You can send this files to each server using rsync:
rsync -avzP $(cat list_files_1.txt) ip:/home/user2/public_html/domain.com/
1) use find to make a list of all the files
find /opt/mymp3folder -print > /tmp/foo
2) find the count of lines and split the list in two
cd /tmp
wc -l /tmp/foo
387
split -l 200 /tmp/foo
3) split by default makes a set of files called xaa xab xac etc. So use xaa to copy to one server and xab to copy to the other
rsync -av --files-from=/tmp/xaa . server1:/opt/newmp3folder/
rsync -av --files-from=/tmp/xab . server2:/opt/newmp3folder/
'.' in the above is the "source" path and allows the use of relative paths in the "files-from" You either need to be in the same path that the find command is run from and use . or set it to an absolute value
Obviously if you wanted to do this on a regular basis probably want to script it properly
There is an input like:
folder1
folder2
folder3
...
foldern
I would like to iterate over taking multiple lines at once and processes each line, remove the first / (and more but for now this is enough) and echo the. Iterating over in bash with a single thread can be slow sometimes. The alternative way of doing this would be splitting up the input file to N pieces and run the same script with different input and output N times, at the end you can merge the results.
I was wondering if this is possible with xargs.
Update 1:
Input:
/a/b/c
/d/f/e
/h/i/j
Output:
mkdir a/b/c
mkdir d/f/e
mkdir h/i/j
Script:
for i in $(<test); do
echo mkdir $(echo $i | sed 's/\///') ;
done
Doing it with xargs does not work as I would expect:
xargs -a test -I line --max-procs=2 echo mkdir $(echo $line | sed 's/\///')
Obviously I need a way to execute the sed on the input for each line, but using $() does not work.
You probably want:
--max-procs=max-procs, -P max-procs
Run up to max-procs processes at a time; the default is 1. If
max-procs is 0, xargs will run as many processes as possible at
a time. Use the -n option with -P; otherwise chances are that
only one exec will be done.
http://unixhelp.ed.ac.uk/CGI/man-cgi?xargs
With GNU Parallel you can do:
cat file | perl -pe s:/:: | parallel mkdir -p
or:
cat file | parallel mkdir -p {= s:/:: =}
Is there a one-line command/script to copy one file to many files on Linux?
cp file1 file2 file3
copies the first two files into the third. Is there a way to copy the first file into the rest?
Does
cp file1 file2 ; cp file1 file3
count as a "one-line command/script"? How about
for file in file2 file3 ; do cp file1 "$file" ; done
?
Or, for a slightly looser sense of "copy":
tee <file1 file2 file3 >/dev/null
just for fun, if you need a big list of files:
tee <sourcefile.jpg targetfiles{01-50}.jpg >/dev/null- Kelvin Feb 12 at 19:52
But there's a little typo. Should be:
tee <sourcefile.jpg targetfiles{01..50}.jpg >/dev/null
And as mentioned above, that doesn't copy permissions.
You can improve/simplify the for approach (answered by #ruakh) of copying by using ranges from bash brace expansion:
for f in file{1..10}; do cp file $f; done
This copies file into file1, file2, ..., file10.
Resource to check:
http://wiki.bash-hackers.org/syntax/expansion/brace#ranges
for FILE in "file2" "file3"; do cp file1 $FILE; done
You can use shift:
file=$1
shift
for dest in "$#" ; do
cp -r $file $dest
done
cat file1 | tee file2 | tee file3 | tee file4 | tee file5 >/dev/null
(no loops used)
To copy the content of one file (fileA.txt) to many files (fileB.txt, fileC.txt, fileD.txt) in Linux,
use the following combination cat and tee commands:
cat fileA.txt | tee fileB.txt fileC.txt fileD.txt >/dev/null
applicable to any file extensions
only file names and extensions change, everything else remains same.
Use something like the following. It works on zsh.
cat file > firstCopy > secondCopy > thirdCopy
or
cat file > {1..100} - for filenames with numbers.
It's good for small files.
You should use the cp script mentioned earlier for larger files.
I'd recommend creating a general use script and a function (empty-files), based on the script, to empty any number of target files.
Name the script copy-from-one-to-many and put it in your PATH.
#!/bin/bash -e
# _ _____
# | |___ /_ __
# | | |_ \ \/ / Lex Sheehan (l3x)
# | |___) > < https://github.com/l3x
# |_|____/_/\_\
#
# Copy the contents of one file to many other files.
source=$1
shift
for dest in "$#"; do
cp $source $dest
done
exit
NOTES
The shift above removes the first element (the source file path) from the list of arguments ("$#").
Examples of how to empty many files:
Create file1, file2, file3, file4 and file5 with content:
for f in file{1..5}; do echo $f > "$f"; done
Empty many files:
copy-from-one-to-many /dev/null file1 file2 file3 file4 file5
Empty many files easier:
# Create files with content again
for f in file{1..5}; do echo $f > "$f"; done
copy-from-one-to-many /dev/null file{1..5}
Create empty_files function based on copy-from-one-to-many
function empty-files()
{
copy-from-one-to-many /dev/null "$#"
}
Example usage
# Create files with content again
for f in file{1..5}; do echo $f > "$f"; done
# Show contents of one of the files
echo -e "file3:\n $(cat file3)"
empty_files file{1..5}
# Show that the selected file no longer has contents
echo -e "file3:\n $(cat file3)"
Don't just steal code. Improve it; Document it with examples and share it. - l3x
Here's a version that will preface each cp command with sudo:
#!/bin/bash -e
# Filename: copy-from-one-to-may
# _ _____
# | |___ /_ __
# | | |_ \ \/ / Lex Sheehan (l3x)
# | |___) > < https://github.com/l3x
# |_|____/_/\_\
#
# Copy the contents of one file to many other files.
# Pass --sudo if you want each cp to be perfomed with sudo
# Ex: copy-from-one-to-many $(mktemp) /tmp/a /tmp/b /tmp/c --sudo
if [[ "$*" == *--sudo* ]]; then
maybe_use_sudo=sudo
fi
source=$1
shift
for dest in "$#"; do
if [ $dest != '--sudo' ]; then
$maybe_use_sudo cp $source $dest
fi
done
exit
You can use standard scripting commands for that instead:
Bash:
for i in file2 file3 ; do cp file1 $i ; done
The simplest/quickest solution I can think of is a for loop:
for target in file2 file3 do; cp file1 "$target"; done
A dirty hack would be the following (I strongly advise against it, and only works in bash anyway):
eval 'cp file1 '{file2,file3}';'
Go with the fastest cp operations
seq 1 10 | xargs -P 0 -I xxx cp file file-xxx
it means
seq 1 10 count from 1 to 10
| pipe it xargs
-P 0 do it in parallel - as many as needed
-I xxx name of each input xargs receives
cp file file-xxx means copy file to file-1, file-2, etc
and if name of files are different here is the other solutions.
First have the list of files which are going to be created. e.g.
one
two
three
four
five
Second save this list on disk and read the list with xargs just like before but without using seq.
xargs -P 0 -I xxx cp file xxx < list
which means 5 copy operations in parallel:
cp file one
cp file two
cp file three
cp file four
cp file five
and for xargs here is the behind the scene (5 forks)
3833 pts/0 Ss 0:00 bash
15954 pts/0 0:00 \_ xargs -P 0 -I xxx cp file xxx < list
15955 pts/0 0:00 \_ cp file one
15956 pts/0 0:00 \_ cp file two
15957 pts/0 0:00 \_ cp file three
15958 pts/0 0:00 \_ cp file four
15959 pts/0 0:00 \_ cp file five
I don't know how correct this is but i have used something like this
echo ./file1.txt ./file2.txt ./file3.txt | xargs -n 1 cp file.txt
Where echo ./file1.txt ... is destination of a file and use it to feed xargs with one "destination" by one. Therefore command xargs -n 1. And lastly cp file.txt, which is self explanatory i think :)