I have these files. Imagine that each "test" represent the name of one server:
test10.txt
test11.txt
test12.txt
test13.txt
test14.txt
test15.txt
test16.txt
test17.txt
test18.txt
test19.txt
test1.txt
test20.txt
test21.txt
test22.txt
test23.txt
test24.txt
test25.txt
test26.txt
test27.txt
test28.txt
test29.txt
test2.txt
test30.txt
test31.txt
test32.txt
test33.txt
test34.txt
test35.txt
test36.txt
test37.txt
test38.txt
test39.txt
test3.txt
test40.txt
test4.txt
test5.txt
test6.txt
test7.txt
test8.txt
test9.txt
In each txt file, I have this type of data:
2019-10-14-00-00;/dev/hd1;1024.00;136.37;/
2019-10-14-00-00;/dev/hd2;5248.00;4230.53;/usr
2019-10-14-00-00;/dev/hd3;2560.00;481.66;/var
2019-10-14-00-00;/dev/hd4;3584.00;67.65;/tmp
2019-10-14-00-00;/dev/hd5;256.00;26.13;/home
2019-10-14-00-00;/dev/hd1;1024.00;476.04;/opt
2019-10-14-00-00;/dev/hd5;384.00;0.38;/usr/xxx
2019-10-14-00-00;/dev/hd4;256.00;21.39;/xxx
2019-10-14-00-00;/dev/hd2;512.00;216.84;/opt
2019-10-14-00-00;/dev/hd3;128.00;21.46;/var/
2019-10-14-00-00;/dev/hd8;256.00;75.21;/usr/
2019-10-14-00-00;/dev/hd7;384.00;186.87;/var/
2019-10-14-00-00;/dev/hd6;256.00;0.63;/var/
2019-10-14-00-00;/dev/hd1;128.00;0.37;/admin
2019-10-14-00-00;/dev/hd4;256.00;179.14;/opt/
2019-10-14-00-00;/dev/hd3;2176.00;492.93;/opt/
2019-10-14-00-00;/dev/hd1;256.00;114.83;/opt/
2019-10-14-00-00;/dev/hd9;256.00;41.73;/var/
2019-10-14-00-00;/dev/hd1;3200.00;954.28;/var/
2019-10-14-00-00;/dev/hd10;256.00;0.93;/var/
2019-10-14-00-00;/dev/hd10;64.00;1.33;/
2019-10-14-00-00;/dev/hd2;1664.00;501.64;/opt/
2019-10-14-00-00;/dev/hd4;256.00;112.32;/opt/
2019-10-14-00-00;/dev/hd9;2176.00;1223.1;/opt/
2019-10-14-00-00;/dev/hd11;22784.00;12325.8;/opt/
2019-10-14-00-00;/dev/hd12;256.00;2.36;/
2019-10-14-06-00;/dev/hd12;1024.00;137.18;/
2019-10-14-06-00;/dev/hd1;256.00;2.36;/
2019-10-14-00-00;/dev/hd1;1024.00;136.37;/
2019-10-14-00-00;/dev/hd2;5248.00;4230.53;/usr
2019-10-14-00-00;/dev/hd3;2560.00;481.66;/var
2019-10-14-00-00;/dev/hd4;3584.00;67.65;/tmp
2019-10-14-00-00;/dev/hd5;256.00;26.13;/home
2019-10-14-00-00;/dev/hd1;1024.00;476.04;/opt
2019-10-14-00-00;/dev/hd5;384.00;0.38;/usr/xxx
2019-10-14-00-00;/dev/hd4;256.00;21.39;/xxx
2019-10-14-00-00;/dev/hd2;512.00;216.84;/opt
2019-10-14-00-00;/dev/hd3;128.00;21.46;/var/
2019-10-14-00-00;/dev/hd8;256.00;75.21;/usr/
2019-10-14-00-00;/dev/hd7;384.00;186.87;/var/
2019-10-14-00-00;/dev/hd6;256.00;0.63;/var/
2019-10-14-00-00;/dev/hd1;128.00;0.37;/admin
2019-10-14-00-00;/dev/hd4;256.00;179.14;/opt/
2019-10-14-00-00;/dev/hd3;2176.00;492.93;/opt/
2019-10-14-00-00;/dev/hd1;256.00;114.83;/opt/
2019-10-14-00-00;/dev/hd9;256.00;41.73;/var/
2019-10-14-00-00;/dev/hd1;3200.00;954.28;/var/
2019-10-14-00-00;/dev/hd10;256.00;0.93;/var/
2019-10-14-00-00;/dev/hd10;64.00;1.33;/
2019-10-14-00-00;/dev/hd2;1664.00;501.64;/opt/
2019-10-14-00-00;/dev/hd4;256.00;112.32;/opt/
I would like to create a directory for each server, create in each directory a txt file for each FS and put in these txt files each lines which correspond to the FS.
For that, I've tried loop :
#!/bin/bash
directory=(ls *.txt | cut -d'.' -f1)
for d in $directory
do
if [ ! -d $d ]
then
mkdir $d
fi
done
for i in $(cat *.txt)
do
file=$(echo $i | awk -F';' '{print $2}' | sort | uniq | cut -d'/' -f3 )
data=$(echo $i | awk -F';' '{print $2}' )
echo $i | grep -w $data >> /xx/xx/xx/xx/xx/${directory/${file}.txt
done
But this loop doesn't work properly. The directories are created but not the file inside each directory.
I would like something like :
test1/hd1.txt ( with each line which for the hd1 fs in the hd1.txt)
And same thing for each server.
Can you show me how to do that?
#!/bin/bash
for src in *.txt; do
# start a subshell so we don't need to cd back afterwards
# make "$src" be stdin before cd, so we don't need full path
# be careful that in subshell only awk reads from stdin
(
# extract server name to use as directory
dir=/xx/xx/xx/xx/xx/"${src%.txt}"
# chain with "&&" so failures don't cause bad files
mkdir -p "$dir" &&
cd "$dir" &&
awk -F \; '{ split($2, dev, "/"); print > dev[3]".txt" }'
) < "$src"
done
The awk script reads lines delimited by semi-colons.
It splits the second field on slashes to extract the device name (assumption is that the devices always have form: /dev/name
Finally, the > sends output to the relevant file.
For reference, you can make your script work by doing directory=$(...); adding the prefix to mkdir (assuming the prefix directories already exist); closing the reference ${directory}; and quoting all variable references for safety:
#!/bin/bash
directory=$(ls *.txt | cut -d'.' -f1)
for d in "$directory"
do
if [ ! -d "$d" ]
then
mkdir /xx/xx/xx/xx/xx/"$d"
fi
done
for i in $(cat *.txt)
do
file=$(echo "$i" | awk -F';' '{print $2}' | sort | uniq | cut -d'/' -f3 )
data=$(echo $i | awk -F';' '{print $2}' )
echo "$i" | grep -w "$data" >> /xx/xx/xx/xx/xx/"${directory}"/"${file}".txt
done
for file in `ls *.txt`
do
echo ${file}
directory=`echo ${file} | cut -d'.' -f1`
#echo ${directory}
if [ ! -d ${directory} ]
then
mkdir ${directory}
fi
FS=`cat ${file} | awk -F';' '{print $2}' | sort | uniq | cut -d'/' -f3`
#echo $FS
for f in $FS
do
cat ${file} |grep -w -e $f > ${directory}/${f}.txt
done
done
Explanation:
For each file in the current directory, the outer for loop will run.
In the loop for the selected file, a respective directory will be created first.
Next using the FS variable we take all the possible file systems from that selected file.
Finally, an inner loop will be run using the FS types to grep and create separate file system files in the directory.
Related
This bash command will move all files that are not already in a directory into a new self-named folder.
ll | grep ^- | awk -F"." '{print $2 "." $3}' | awk -F":" '{print $2}' | awk '{$1=""; print $0}' | cut -c2- | awk -F"." '{print "mkdir ""$1"";mv ""$1"."$2"" ""$1"""}' > tmp;source tmp
To move all files in the current directory into directories named after the base part of the filename:
for f in *
do
[ -f "$f" -o -d "${f%.*}" ] || continue
mkdir "${f%.*}" && mv "$f" "${f%.*}"
done
This skips any entries in the current directory that are not -f files; it also skips any entries that already have directories named for them. It then creates the directory (named by stripping "period followed by anything" off of the filename) and if that's successful, moves the file into that directory.
I have a few .csv files like below.
xyz0900#1#-1637746436.csv
xxx0900#1#-1637746436.csv
zzz0900#2#-1637746439.csv
yyy0900#1#-1637746436.csv
sss0900#2#-1637746439.csv
I have written a script to perform below tasks:
Get the large file based on the pattern which we have passed as a argument to the script.
Merge all other files which are having same pattern and create a new file
Remove duplicate header from new file.
Move new file to the destination based on the parameter passed as a argument.
Example: I am passing "1637746436#home/dest1,1637746436#home/dest2" as
a second argument to the script. Below script will fetch the
pattern(1637746436). Get the bigger file and merge all other
files(having same pattern) with it. New file will be get created and same will be moved to the destination(home/dest1).
The below script will perform the pattern matching and execution sequentially.
How to make 'for loop iteration' should be executed parallelly? I mean pattern matching of "1637746436#home/dest1,1637746436#home/dest2" should be performed simultaneously(not one after another).
Please help on this.
$merge.sh /home/dummy/17 "1637746436#home/dest1,1637746439#home/dest2"
#!/bin/bash
current=`pwd`
source=$1
destination=$2
echo "$destination" | tr "," "\n" > $current/out.txt
cat out.txt | cut -d "#" -f1 > $current/pattern.txt
for var in `cat pattern.txt`
do
getBiggerfile=$(ls -Sl $source/*$var.csv | head -1)
cd $source
getFileName=$(echo $getBiggerfile | cut -d " " -f9-)
newFileName=$(echo $getFileName | cut -d "#" -f1)
cat *$var.csv >> $getFileName
header=$(head -n 1 $getFileName)
(printf "%s\n" "$header";
grep -vFxe "$header" $getFileName
) > $newFileName.csv
rm -rf *$var.csv
cd $current
for var1 in `cat out.txt`
do
target=`echo $var1 | cut -d "#" -f2`
id=$(echo $var1 | cut -c-10)
if [ $id = $var ]
then
mv $newFileName.csv $target
fi
done
done
The cleanest would be to make the internals of the loop a function, and call the function inside the loop, putting it in the background (child processes), then wait for the background (child) processes to finish:
function do_the_thing(){
source="$1"
current="$2"
var="$3"
getBiggerfile=$(ls -Sl $source/*$var.csv | head -1)
cd $source
getFileName=$(echo $getBiggerfile | cut -d " " -f9-)
newFileName=$(echo $getFileName | cut -d "#" -f1)
cat *$var.csv >> $getFileName
header=$(head -n 1 $getFileName)
(printf "%s\n" "$header";
grep -vFxe "$header" $getFileName
) > $newFileName.csv
rm -rf *$var.csv
cd $current
for var1 in `cat out.txt`
do
target=`echo $var1 | cut -d "#" -f2`
id=$(echo $var1 | cut -c-10)
if [ $id = $var ]
then
mv $newFileName.csv $target
fi
done
}
for var in `cat pattern.txt`
do
do_the_thing "$source" "$current" "$var" &
done
wait
So I have a that works as it going to mulitple directories then within these directories takes multiple fields from files and store it in a .txt files.
There are two loop, the first one that loops through all the folder
the second one that loops through all the files.
The problem I encounter is in the second loops that it read only the first file in the folder and then it moves on to the next folder and ignore all other files in the folder.
archive=/imdata/archive
inventory_archive=/imdata/a/shares/b/inventory/c
ls $archive | while read p; do
echo "Project: $p"
mkdir -v $inventory_archive/$p
dir=$inventory_archive/$p
ls -1 $archive/$p/d001 | while read s; do
echo "Searching Session: $s ..."
find $archive/$p/d001/$s -type f -iname "*.txt" | while read f; do
echo "FILE: $f"
study=`/home/me/program/bin/script $f | grep -m1 "field1" | cut -d "[" -f2 | cut -d "]" -f1`
echo "SID: $study"
if [ ! -d "$dir/$study" ]; then
mkdir -v $dir/$study
fi
studydir=$dir/$study
series=`/home/me/program/bin/script $f | grep -m1 "field2" | cut -d "[" -f2 | cut -d "]" -f1`
echo "SID_2: $series"
if [ ! -a "$studydir/$series.txt" ]; then
touch $studydir/$series.txt
fi
sop=`/home/me/program/bin/script $f | grep -m1 "field3" | cut -d "[" -f2 | cut -d "]" -f1`
echo "SID_3: $sop"
grep -qsF $sop $studydir/$series.txt || echo $sop >> $studydir/$series.txt
exit 1;
done;
done;
done;
There are some directories in the working directory with this template
cas-2-32
sat-4-64
...
I want to loop over the directory names and grab the second and third part of folder names. I have wrote this script. The body shows what I want to do. But the awk command seems to be wrong
#!/bin/bash
for file in `ls`; do
if [ -d $file ]; then
arg2=`awk -F "-" '{print $2}' $file`
echo $arg2
arg3=`awk -F "-" '{print $3}' $file`
echo $arg3
fi
done
but it says
awk: cmd. line:1: fatal: cannot open file `cas-2-32' for reading (Invalid argument)
awk expects a filename as input. Since you have said the cas-2-32 etc are directories, awk fails for the same reason.
Feed the directory names to awk using echo:
#!/bin/bash
for file in `ls`; do
if [ -d $file ]; then
arg2=$(echo $file | awk -F "-" '{print $2}')
echo $arg2
arg3=$(echo $file | awk -F "-" '{print $3}')
echo $arg3
fi
done
Simple comand: ls | awk '{ FS="-"; print $2" "$3 }'
If you want the values in each line just add "\n" instead of a space in awk's print.
When executed like this
awk -F "-" '{print $2}' $file
awk treats $file's value as the file to be parsed, instead of parsing $file's value itself.
The minimal fix is to use a here-string which can feed the value of a variable into stdin of a command:
awk -F "-" '{print $2}' <<< $file
By the way, you don't need ls if you merely want a list of files in current directory, use * instead, i.e.
for file in *; do
One way:
#!/bin/bash
for file in *; do
if [ -d $file ]; then
tmp="${file#*-}"
arg2="${tmp%-*}"
arg3="${tmp#*-}"
echo "$arg2"
echo "$arg3"
fi
done
The other:
#!/bin/bash
IFS="-"
for file in *; do
if [ -d $file ]; then
set -- $file
arg2="$2"
arg3="$3"
echo "$arg2"
echo "$arg3"
fi
done
I need a bash script that goes trough a text file, finds lines starting in "Common subdirectories: ", and rmdir -rf the two subdirectories. Example of line:
Common subdirectories: /dir1/dirA and /dir1/dirB
I'm quite new to bash scripting so any help would be great.
grep 'Common subdirectories: ' < in.txt |\
cut -d: -f2 | cut -d" " -f2,4 |\
while read a b
do
rm -rf "$a" "$b"
done
Edit; added quoting, use the same rm command for both
A more succinct version:
awk '/^Common subdirectories:/{ system("rm -rf "$3" "$5) }' input.txt
Here's a more complete example:
for F in `grep 'Common subdirectories' input.txt | cut -d: -f2 | awk 'BEGIN{RS=" "}{ print }' | tr -d ' '`
do
[ -d "$F" ] && rm -rf $F
done
A bit shorter command:
awk '/Common subdirectories:/ { print $3 " " $5 }' in.txt | xargs -n1 rm -rf