splitting folder names with awk in a directory - linux

There are some directories in the working directory with this template
cas-2-32
sat-4-64
...
I want to loop over the directory names and grab the second and third part of folder names. I have wrote this script. The body shows what I want to do. But the awk command seems to be wrong
#!/bin/bash
for file in `ls`; do
if [ -d $file ]; then
arg2=`awk -F "-" '{print $2}' $file`
echo $arg2
arg3=`awk -F "-" '{print $3}' $file`
echo $arg3
fi
done
but it says
awk: cmd. line:1: fatal: cannot open file `cas-2-32' for reading (Invalid argument)

awk expects a filename as input. Since you have said the cas-2-32 etc are directories, awk fails for the same reason.
Feed the directory names to awk using echo:
#!/bin/bash
for file in `ls`; do
if [ -d $file ]; then
arg2=$(echo $file | awk -F "-" '{print $2}')
echo $arg2
arg3=$(echo $file | awk -F "-" '{print $3}')
echo $arg3
fi
done

Simple comand: ls | awk '{ FS="-"; print $2" "$3 }'
If you want the values in each line just add "\n" instead of a space in awk's print.

When executed like this
awk -F "-" '{print $2}' $file
awk treats $file's value as the file to be parsed, instead of parsing $file's value itself.
The minimal fix is to use a here-string which can feed the value of a variable into stdin of a command:
awk -F "-" '{print $2}' <<< $file
By the way, you don't need ls if you merely want a list of files in current directory, use * instead, i.e.
for file in *; do

One way:
#!/bin/bash
for file in *; do
if [ -d $file ]; then
tmp="${file#*-}"
arg2="${tmp%-*}"
arg3="${tmp#*-}"
echo "$arg2"
echo "$arg3"
fi
done
The other:
#!/bin/bash
IFS="-"
for file in *; do
if [ -d $file ]; then
set -- $file
arg2="$2"
arg3="$3"
echo "$arg2"
echo "$arg3"
fi
done

Related

Created directory with for loop in bash

I have these files. Imagine that each "test" represent the name of one server:
test10.txt
test11.txt
test12.txt
test13.txt
test14.txt
test15.txt
test16.txt
test17.txt
test18.txt
test19.txt
test1.txt
test20.txt
test21.txt
test22.txt
test23.txt
test24.txt
test25.txt
test26.txt
test27.txt
test28.txt
test29.txt
test2.txt
test30.txt
test31.txt
test32.txt
test33.txt
test34.txt
test35.txt
test36.txt
test37.txt
test38.txt
test39.txt
test3.txt
test40.txt
test4.txt
test5.txt
test6.txt
test7.txt
test8.txt
test9.txt
In each txt file, I have this type of data:
2019-10-14-00-00;/dev/hd1;1024.00;136.37;/
2019-10-14-00-00;/dev/hd2;5248.00;4230.53;/usr
2019-10-14-00-00;/dev/hd3;2560.00;481.66;/var
2019-10-14-00-00;/dev/hd4;3584.00;67.65;/tmp
2019-10-14-00-00;/dev/hd5;256.00;26.13;/home
2019-10-14-00-00;/dev/hd1;1024.00;476.04;/opt
2019-10-14-00-00;/dev/hd5;384.00;0.38;/usr/xxx
2019-10-14-00-00;/dev/hd4;256.00;21.39;/xxx
2019-10-14-00-00;/dev/hd2;512.00;216.84;/opt
2019-10-14-00-00;/dev/hd3;128.00;21.46;/var/
2019-10-14-00-00;/dev/hd8;256.00;75.21;/usr/
2019-10-14-00-00;/dev/hd7;384.00;186.87;/var/
2019-10-14-00-00;/dev/hd6;256.00;0.63;/var/
2019-10-14-00-00;/dev/hd1;128.00;0.37;/admin
2019-10-14-00-00;/dev/hd4;256.00;179.14;/opt/
2019-10-14-00-00;/dev/hd3;2176.00;492.93;/opt/
2019-10-14-00-00;/dev/hd1;256.00;114.83;/opt/
2019-10-14-00-00;/dev/hd9;256.00;41.73;/var/
2019-10-14-00-00;/dev/hd1;3200.00;954.28;/var/
2019-10-14-00-00;/dev/hd10;256.00;0.93;/var/
2019-10-14-00-00;/dev/hd10;64.00;1.33;/
2019-10-14-00-00;/dev/hd2;1664.00;501.64;/opt/
2019-10-14-00-00;/dev/hd4;256.00;112.32;/opt/
2019-10-14-00-00;/dev/hd9;2176.00;1223.1;/opt/
2019-10-14-00-00;/dev/hd11;22784.00;12325.8;/opt/
2019-10-14-00-00;/dev/hd12;256.00;2.36;/
2019-10-14-06-00;/dev/hd12;1024.00;137.18;/
2019-10-14-06-00;/dev/hd1;256.00;2.36;/
2019-10-14-00-00;/dev/hd1;1024.00;136.37;/
2019-10-14-00-00;/dev/hd2;5248.00;4230.53;/usr
2019-10-14-00-00;/dev/hd3;2560.00;481.66;/var
2019-10-14-00-00;/dev/hd4;3584.00;67.65;/tmp
2019-10-14-00-00;/dev/hd5;256.00;26.13;/home
2019-10-14-00-00;/dev/hd1;1024.00;476.04;/opt
2019-10-14-00-00;/dev/hd5;384.00;0.38;/usr/xxx
2019-10-14-00-00;/dev/hd4;256.00;21.39;/xxx
2019-10-14-00-00;/dev/hd2;512.00;216.84;/opt
2019-10-14-00-00;/dev/hd3;128.00;21.46;/var/
2019-10-14-00-00;/dev/hd8;256.00;75.21;/usr/
2019-10-14-00-00;/dev/hd7;384.00;186.87;/var/
2019-10-14-00-00;/dev/hd6;256.00;0.63;/var/
2019-10-14-00-00;/dev/hd1;128.00;0.37;/admin
2019-10-14-00-00;/dev/hd4;256.00;179.14;/opt/
2019-10-14-00-00;/dev/hd3;2176.00;492.93;/opt/
2019-10-14-00-00;/dev/hd1;256.00;114.83;/opt/
2019-10-14-00-00;/dev/hd9;256.00;41.73;/var/
2019-10-14-00-00;/dev/hd1;3200.00;954.28;/var/
2019-10-14-00-00;/dev/hd10;256.00;0.93;/var/
2019-10-14-00-00;/dev/hd10;64.00;1.33;/
2019-10-14-00-00;/dev/hd2;1664.00;501.64;/opt/
2019-10-14-00-00;/dev/hd4;256.00;112.32;/opt/
I would like to create a directory for each server, create in each directory a txt file for each FS and put in these txt files each lines which correspond to the FS.
For that, I've tried loop :
#!/bin/bash
directory=(ls *.txt | cut -d'.' -f1)
for d in $directory
do
if [ ! -d $d ]
then
mkdir $d
fi
done
for i in $(cat *.txt)
do
file=$(echo $i | awk -F';' '{print $2}' | sort | uniq | cut -d'/' -f3 )
data=$(echo $i | awk -F';' '{print $2}' )
echo $i | grep -w $data >> /xx/xx/xx/xx/xx/${directory/${file}.txt
done
But this loop doesn't work properly. The directories are created but not the file inside each directory.
I would like something like :
test1/hd1.txt ( with each line which for the hd1 fs in the hd1.txt)
And same thing for each server.
Can you show me how to do that?
#!/bin/bash
for src in *.txt; do
# start a subshell so we don't need to cd back afterwards
# make "$src" be stdin before cd, so we don't need full path
# be careful that in subshell only awk reads from stdin
(
# extract server name to use as directory
dir=/xx/xx/xx/xx/xx/"${src%.txt}"
# chain with "&&" so failures don't cause bad files
mkdir -p "$dir" &&
cd "$dir" &&
awk -F \; '{ split($2, dev, "/"); print > dev[3]".txt" }'
) < "$src"
done
The awk script reads lines delimited by semi-colons.
It splits the second field on slashes to extract the device name (assumption is that the devices always have form: /dev/name
Finally, the > sends output to the relevant file.
For reference, you can make your script work by doing directory=$(...); adding the prefix to mkdir (assuming the prefix directories already exist); closing the reference ${directory}; and quoting all variable references for safety:
#!/bin/bash
directory=$(ls *.txt | cut -d'.' -f1)
for d in "$directory"
do
if [ ! -d "$d" ]
then
mkdir /xx/xx/xx/xx/xx/"$d"
fi
done
for i in $(cat *.txt)
do
file=$(echo "$i" | awk -F';' '{print $2}' | sort | uniq | cut -d'/' -f3 )
data=$(echo $i | awk -F';' '{print $2}' )
echo "$i" | grep -w "$data" >> /xx/xx/xx/xx/xx/"${directory}"/"${file}".txt
done
for file in `ls *.txt`
do
echo ${file}
directory=`echo ${file} | cut -d'.' -f1`
#echo ${directory}
if [ ! -d ${directory} ]
then
mkdir ${directory}
fi
FS=`cat ${file} | awk -F';' '{print $2}' | sort | uniq | cut -d'/' -f3`
#echo $FS
for f in $FS
do
cat ${file} |grep -w -e $f > ${directory}/${f}.txt
done
done
Explanation:
For each file in the current directory, the outer for loop will run.
In the loop for the selected file, a respective directory will be created first.
Next using the FS variable we take all the possible file systems from that selected file.
Finally, an inner loop will be run using the FS types to grep and create separate file system files in the directory.

Printing awk output in same line after grep

I have a very crude script getinfo.sh that gets me information from all files with name FILENAME1 and FILENAME2 in all subfolders and the path of the subfolder. The awk result should only pick the nth line from FILENAME2 if the script is called with "getinfo.sh n". I want all the info printed in one line!
The problem is that if i use print instead of printf the info is written to a new line but my script works. If i use printf i can see the last bit of the awk command in the command propt after the script ist done, but it is not paset after the grep command in the same line. All in all the complete line would be pretty long, but that is intentionally. Would you be willing to tell me what i am doing wrong?
#!/bin/bash
IFS=$'\n'
while read -r fname ;
do
pushd $(dirname "${fname}") > /dev/null
printf '%q' "${PWD##*/}"
grep 'Search_term ' FILENAME1 | tail -1
awk '{ if(NR==n) printf "%s",$0 }' n=$1 $2 FILENAME2
popd > /dev/null
done < <(find . -type f -name 'FILENAME1')
I would also be happy to grep the nth line if this is easier?
SOLUTION:
#!/bin/bash
IFS=$'\n'
while read -r fname ;
do
pushd $(dirname "${fname}") > /dev/null
{
printf '%q' "${PWD##*/}"
grep 'Search_term' FILENAME1 | tail -1
} | tr -d '\n'
if [ "$1" -eq "$1" ] 2>/dev/null
then
awk '{ if(NR==n) printf "%s",$0 }' n="$1" FILENAME2
fi
printf "\n"
popd > /dev/null
done < <(find . -type f -name 'FILENAME1')
You made it clearer in the comments.
I want the output of printf '%q' "${PWD##*/}" and grep 'Search_term ' FILENAME1 | tail -1 and awk '{ if(NR==n) printf "%s",$0 }' n=$1 $2 FILENAME2 to be printed in one line
So first, we have three commands, that each print a single line of output. As the commands do not matter, let's wrap them in functions to simplify the answer:
cmd1() { printf '%q\n' "${PWD##*/}"; }
cmd2() { grep .... ; }
cmd3() { awk ....; }
To print them without newlines between them, we can:
Use a command substitution, which removes trailing empty newlines. With some printf:
printf "%s%s%s\n" "$(cmd1)" "$(cmd2)" "$(cmd3)"
or some echo:
echo "$(cmd1) $(cmd2) $(cmd3)"
or append to a variable:
str="$(cmd1)"
str+=" $(cmd2)"
str+=" $(cmd3)"
printf" %s\n" "$str"
and so on.
We can remove newlines from the stream, using tr -d '\n':
{
cmd1
cmd2
cmd3
} | tr -d '\n'
echo # newlines were removed, so add one to the end.
or we can also remove the newlines only from the first n-1 commands, but I think this is less readable:
{
cmd1
cmd2
} | tr -d'\n'
cmd3 # the trailing newline will be added by cmd3
If i do not pass a number the awk command should be omited.
I see that your awk command expands both $1 and $2, and i see only $1 to be passed as the n=$1 environment variable to awk. I don't know what is $2. You can write if-s on the value of $# the number of arguments:
if (($# == 2)); then
awk '{ if(NR==n) printf "%s",$0 }' n="$1" "$2" FILENAME2
fi
and similar for each case you want to handle. Remember about proper quoting.
Your command shows the unused parameter $2, I deleted that one.
You can add a newline at the end of the awk using the END block, but you also want an extra newline when you call your script without a line number. echo will do.
#!/bin/bash
IFS=$'\n'
while read -r fname ;
do
pushd $(dirname "${fname}") > /dev/null
# Add result of grep in same printf statement
printf '%s %s' "${PWD##*/}" "$(grep 'Search_term ' FILENAME1 | tail -1)"
if (( $# -eq 1 )); then
# use $1 as an awk variable, number n
# use $2 as a different file to read from
awk -v n=$1 '{ if(NR==n) printf "%s ",$0 }' FILENAME2
fi
# Add line-ending
echo
popd > /dev/null
done < <(find . -type f -name 'FILENAME1')

How to return string count in multiple files in Linux

I have multiple xml files and I want to count some string in it.
How to return string count with files names in Linux?
The string I want to count InvoıceNo:
Result will be;
test.xml InvoiceCount:2
test1.xml InvoiceCount:5
test2.xml InvoiceCount:10
You can probably use the following code
PATTERN=InvoiceNo
for file in *.xml
do
count=$(grep -o $PATTERN "$file" | wc -l)
echo "$file" InvoiceCount:$count
done
Output
test.xml InvoiceCount:1
test1.xml InvoiceCount:2
test2.xml InvoiceCount:3
Refered from: https://unix.stackexchange.com/questions/6979/count-total-number-of-occurrences-using-grep
Following awk may help you in same, since you haven't shown any sample Inputs so not tested it.
awk 'FNR==1{if(count){print value,"InvoiceCount:",count;count=""};value=FILENAME;close(value)} /InvoiceCount/{count++}' *.xml
Use grep -c to get the count of matching lines
for file in *.xml ; do
count=$(grep -c $PATTERN $file)
if [ $count -gt 0 ]; then
echo "$file $PATTERN: $count"
fi
done
First the test files:
$ cat foo.xml
InvoiceCount InvoiceCount
InvoiceCount
$ cat bar.xml
InvoiceCount
The GNU awk using gsub for counting:
$ awk '{
c+=gsub(/InvoiceCount/,"InvoiceCount")
}
ENDFILE {
print FILENAME, "InvoiceCount: " c
c=0
}' foo.xml bar.xml
foo.xml InvoiceCount: 3
bar.xml InvoiceCount: 1
A little shell skript will do want you want
#!/bin/bash
for file in *
do
awk '{count+=gsub(" InvoıceNo","")}
END {print FILENAME, "InvoiceCount:" count}' $file
done
Put the code in a file (e.g. counter.sh) and run it like this:
counter.sh text.xml text1.xml text2.xml

Renaming long file names in bulk

I have file names like:
5_END_1033_ACAGTG_L002_R1_001.fastq.gz
5_END_1033_ACAGTG_L002_R2_001.fastq.gz
40_END_251_GTGAAA_L002_R1_001.fastq.gz
40_END_251_GTGAAA_L002_R2_001.fastq.gz
I want something like:
END_1033_R1.fastq.gz
END_1033_R2.fastq.gz
END_251_R1.fastq.gz
END_251_R2.fastq.gz
Are there good ways to rename these files in linux?
You could try using a loop to extract the important part of the filename:
for file in ./*.gz; do newname=$(echo $file | sed -re 's/^([^ACAGTG]+).*(R[1-3]).*/\1\2\.fastq\.gz/g'); echo $newname; done
This will simply give you a new list of filenames. You can then move them:
for file in ./*.gz; do newname=$(echo $file | sed -re 's/^([^ACAGTG]+).*(R[1-3]).*/\1\2\.fastq\.gz/g'); mv $file $newname; done
To break this down a little:
loop over the *.gz files
create a variable which strips out the unnecessary content from the name
move the file name to that new name
I expect there are better ways to do this, but it's what I came up with off the top of my head.
Test:
$ ls
40_END_251_GTGAAA_L002_R1_001.fastq.gz 40_END_251_GTGAAA_L002_R2_001.fastq.gz 5_END_1033_ACAGTG_L002_R1_001.fastq.gz 5_END_1033_ACAGTG_L002_R2_001.fastq.gz
$ for file in ./*.gz; do newname=$(echo $file | sed -re 's/^([^ACAGTG]+).*(R[1-3]).*/\1\2\.fastq\.gz/g'); echo $newname; done
./40_END_251_R1.fastq.gz
./40_END_251_R2.fastq.gz
./5_END_1033_R1.fastq.gz
./5_END_1033_R2.fastq.gz
$ for file in ./*.gz; do newname=$(echo $file | sed -re 's/^([^ACAGTG]+).*(R[1-3]).*/\1\2\.fastq\.gz/g'); mv $file $newname; done
$ ls
40_END_251_R1.fastq.gz 40_END_251_R2.fastq.gz 5_END_1033_R1.fastq.gz 5_END_1033_R2.fastq.gz
Note I'm doing this in bash 4.4.5
EDIT
Given I'm not entirely sure which columns in the name are the most important, awk might work better:
for file in ./*.gz; do newname=$(echo $file | awk -F'_' '{print $2 "_" $3 "_" $6}' -); echo $newname; done
This will split the filename by _ and allow you to reference the columns you want using $X:
for file in ./*.gz; do newname=$(echo $file | awk -F'_' '{print $2 "_" $3 "_" $6}' -); mv $file "${newname}.fastq.gz"; done

How to use sed on Linux to get values from file name?

how to retrive every portion separately from following file name? DSA4020_frontcover_20346501_2011-05.doc
I want to retrieve informations as below;
name = DSA4020
type = frontcover
id = 20346501
date = 2011-05
is it possible to do with sed??
Yes, you can:
pax$ echo 'DSA4020_frontcover_20346501_2011-05.doc' | sed
-e 's/^/name=/'
-e 's/_/\ntype=/'
-e 's/_/\nid=/'
-e 's/_/\ndate=/'
-e 's/\..*//'
name=DSA4020
type=frontcover
id=20346501
date=2011-05
That's all on one line, I've just split it for readability.
You could also do it with awk if you wish:
pax$ echo 'DSA4020_frontcover_20346501_2011-05.doc'
| awk -F_ '{print "name="$1"\ntype="$2"\nid="$3"\ndate="substr($4,1,7)}'
name=DSA4020
type=frontcover
id=20346501
date=2011-05
awk may be a better choice
# f=DSA4020_frontcover_20346501_2011-05.doc
# name=$(echo $f | awk -F_ '{print $1}')
# echo $name
DSA4020
# type=$(echo $f | awk -F_ '{print $2}')
# echo $type
frontcover
In pure bash
FILE="DSA4020_frontcover_20346501_2011-05.doc"
eval $(echo $FILE |(IFS="_";read a b c d; echo "name=$a;type=$b;id=$c;date=${d%.doc}"))
echo Name:$name Type:$type ID:$id DATE:$date

Resources