I'm working on bash script.
var=$(ls -t1 | head -n1);
cat $var | sed 's/"//g' > latest.csv
cat latest.csv | sed -e 's/^\|$/"/g' -e 's/,/","/g' > from_epos.csv
echo "LATEST: $var";
Here's the whole script, it's meant to delete all quotation mark from current file and add new one, between each field.
INPUT:
"sku","item","price","qty"
5135,"ITEM1",1.79,5
5338,"ITEM2",1.39,5
5318,"ITEM3",1.09,5
5235,"ITEM4",1.09,5
9706,"ITEM5",1.99,5
OUTPUT:
"sku","item","price","qty"
"5135","ITEM1","1.79","5
"
"5338","ITEM2","1.39","5
"
"5318","ITEM3","1.09","5
"
"5235","ITEM4","1.09","5
"
"9706","ITEM5","1.09","5
"
My ideal output is:
"sku","item","price","qty"
"5135","ITEM1","1.79","5"
"5338","ITEM2","1.39","5"
"5318","ITEM3","1.09","5"
"5235","ITEM4","1.09","5"
"9706","ITEM5","1.99","5"
It seems like it's entering random character between line in current output like
" and quotation mark is between CR and LF.
What's the problem and how to get it to my ideal vision?
Thanks,
Adam
awk 'BEGIN{FS=OFS=","}{gsub(/\"/,"");gsub(/[^,]+/,"\"&\"")}1' input
Solution using sed:
sed -e 's/"//g; s/,/","/g; s/^/"/; s/$/"/'
Long-piped-commented version:
sed -e 's/"//g' | # removes all quotations
sed -e 's/,/","/g' | # changes all colons to ","
sed -e 's/^/"/; s/$/"/' # puts quotations in the start and end of each line
awk can do all this in one command:
awk -F"," 'NR>1{for(i=1; i<=NF; i++) {if (!($i ~ /^"/)) printf("\"%s\"",$i);
else printf("%s",$i); if (i<NF) printf(","); else print "";}}' latest.csv
EDIT:
Try this awk: (modified from JS's suggested command)
awk 'BEGIN{FS=OFS=","}{gsub(/\"/,"");gsub(/[^,\r]+/,"\"&\"")}1'
OR
awk -F"[,\r]" 'NR==1{print} NR>1{for(i=1; i<NF; i++) {if (!($i ~ /^"/))
printf("\"%s\"",$i); else printf("%s",$i); if (i<NF-1) printf(",");
else print "";}}'
Related
I want to check if the 8th field of /etc/shadow of a username has no entry.
This is my code:
#!/bin/bash
for i in $(cat < "users.txt")
do
sudo grep -w $i /etc/shadow | awk -F: "$8 == ' '" | cut -d: -f1 ;
done
But this is the error that i get when i execute the script
awk: line 1: syntax error at or near ==
awk syntax for this purpose would be:
awk -F 'delim' '$n1 == "text" {print $n2}'
sudo grep -w $i /etc/shadow | awk -F':' '$8 == " " {print $0}'
FYI: /etc/shadow does not contain spaces between colons. so if cat shows
bin:*:1:1:::::::
you should run $8 == ""
note that there is no space.
If n2 is 0, you'd return the entire row. Hope this helps!
Your approach can be greatly simplified, typically using grep and awk on one line indicates that you're overthinking things. Invoking one command instead of three ...
#!/bin/bash
for i in $(cat < "users.txt")
do
sudo awk -F: -v user=$i '$1==user && $8==""{print $1}' /etc/shadow
done
I have file that contains below information
$ cat test.txt
Studentename:Ram
rollno:12
subjects:6
Highest:95
Lowest:65
Studentename:Krish
rollno:13
subjects:6
Highest:90
Lowest:45
Studentename:Sam
rollno:14
subjects:6
Highest:75
Lowest:65
I am trying place info of single student in single.
i.e My output should be
Studentename:Ram rollno:12 subjects:6 Highest:95 Lowest:65
Studentename:Krish rollno:13 subjects:6 Highest:90 Lowest:45
Studentename:Sam rollno:14 subjects:6 Highest:75 Lowest:65.
Below is the command I wrote
cat test.txt | tr "\n" " " | sed 's/Lowest:[0-9]\+/Lowest:[0:9]\n/g'
Above command is breaking line at regex Lowest:[0-9] but it doesn't print the pattern. Instead it is printing Lowest:[0-9].
Please help
Try:
$ sed '/^Studente/{:a; N; /Lowest/!ba; s/\n/ /g}' test.txt
Studentename:Ram rollno:12 subjects:6 Highest:95 Lowest:65
Studentename:Krish rollno:13 subjects:6 Highest:90 Lowest:45
Studentename:Sam rollno:14 subjects:6 Highest:75 Lowest:65
How it works
/^Studente/{...} tells sed to perform the commands inside the curly braces only on lines that start with Studente. Those commands are:
:a
This defines a label a.
N
This reads in the next line and appends it to the pattern space.
/Lowest/!ba
If the current pattern space does not contain Lowest, this tells sed to branch back to label a.
In more detail, /Lowest/ is true if the line contains Lowest. In sed, ! is negation so /Lowest/! is true if the line does not containLowest. Inba, thebstands for the branch command anda` is the label to branch to.
s/\n/ /g
This tells sed to replace all newlines with spaces.
Try this using awk :
awk '{if ($1 !~ /^Lowest/) {printf "%s ", $0} else {print}}' file.txt
Or shorter but more obfuscated :
awk '$1!~/^Lowest/{printf"%s ",$0;next}1' file.txt
Or correcting your command :
tr "\n" " " < file.txt | sed 's/Lowest:[0-9]\+/&\n/g'
Explanation: & is whats matched in the left part of substitution
Another possible GNU sed that doesn't assume Lowest is the last item:
sed ':a; N; /\nStudent/{P; D}; s/\n/ /; ba' test.txt
This might work for you (GNU sed):
sed '/^Studentename:/{:a;x;s/\n/ /gp;d};H;$ba;d' file
Use the hold space to gather up the fields and then remove the newlines to produce a record.
Replace comma with space using a shell script
Given the following input:
Test,10.10.10.10,"80,22,3306",connect
I need to get below output using a bash script
Test 10.10.10.10 "80,22,3306" connect
If you have gawk, you can use FPAT (field pattern), setting it to a regular expression.
awk -v FPAT='([^,]+)|(\"[^"]+\")' '{ for(i=1;i<=NF;i++) { printf "%s ",$i } }' <<< "Test,10.10.10.10,\"80,22,3306\",connect"
We set FPAT to separate the text based on anything that is not a comma and also data enclosed in quotation marks as as well as anything that is not a quotation mark. We then print all the fields with a spaces in between.
Considering if your Input_file is same as shown sample then following sed may help you in same too.
sed 's/\(.[^,]*\),\([^,]*\),\(".*"\),\(.*\)/\1 \2 \3 \4/g' Input_file
Assuming you can read your input from the file, this works
#!/usr/bin/bash
while read -r line;do
declare -a begin=$(echo $line | awk -F'"' '{print $1}' | tr "," " " )
declare -a end=$(echo $line |awk -F'"' '{print $3}' | tr "," " " )
declare -a middle=$(echo $line | awk -F'"' '{print $2}' )
echo "${begin[#]} \"${middle[#]}\" ${end[#]}"
done < connect_file
Edit: I see,that you want to keep the commas between port numbers. I have edited the script.
echo Test,10.10.10.10,\"80,22,3306\",connect|awk '{sub(/,/," ")gsub(/,"80,22,3306",/," \4280,22,3306\42 ")}1'
Test 10.10.10.10 "80,22,3306" connect
In my recent project I want to make a script to convert a WireShark Output to show the bytes per time frame. With this I got the working script:
./tshark -r dumps/combinedOutput2.pcapng -T fields -e frame.time -e frame.len \
| sed -e 's/\..*\t/\t/' \
| awk -F"\t" '$1==last {sum += $2; next} {
printf("%s %8d bytes/s (%6.2f Mbit/s)\n",last,sum,sum*8/1024/1024);
last=$1;sum=$2}'
In Windows (with the awk and sed commands from GNU Windows) the escapes for quotes and backslashes are different. However I am not able to convert it the right way and get the following error:
awk: {sed -e ^s/\.."\\t/\t/}
awk: ^ backslash not last character on line
tshark: An error occurred while printing packets: Invalid argument.
Can anyone help me with formatting this correctly? Thank you in advance!
Can you try replacing the awk single-quotes with double-quotes as below
./tshark -r dumps/combinedOutput2.pcapng -T fields -e frame.time -e frame.len \
| sed -e 's/\..*\t/\t/' \
| awk -F"\t" '$1==last {sum += $2; next} {
printf("%s %8d bytes/s (%6.2f Mbit/s)\n",last,sum,sum*8/1024/1024);
last=$1;sum=$2}'
and escape the double-quotes within " using a \
./tshark -r dumps/combinedOutput2.pcapng -T fields -e frame.time -e frame.len \
| sed -e 's/\..*\t/\t/' \
| awk -F"\t" "$1==last {sum += $2; next} {
printf(\"%s %8d bytes/s (%6.2f Mbit/s)\n\",last,sum,sum*8/1024/1024);
last=$1;sum=$2}"
Never execute an awk script from the command line in Windows, always execute it from a file to avoid having to deal with Windows nightmarish quoting rules. You also don't need sed when you're using awk.
Put this in script.awk:
BEGIN { FS="\t" }
{ sub(/\..*\t/,"\t") }
$1==last {sum += $2; next}
{
printf "%s %8d bytes/s (%6.2f Mbit/s)\n", last, sum, sum*8/1024/1024
last = $1
sum = $2
}
and execute it as:
./tshark -r dumps/combinedOutput2.pcapng -T fields -e frame.time -e frame.len \
| awk -f script.awk
I have a text files with a line like this in them:
MC exp. sig-250-0 events & $0.98 \pm 0.15$ & $3.57 \pm 0.23$ \\
sig-250-0 is something that can change from file to file (but I always know what it is for each file). There are lines before and above this, but the string "MC exp. sig-250-0 events" is unique in the file.
For a particular file, is there a good way to extract the second number 3.57 in the above example using bash?
use awk for this:
awk '/MC exp. sig-250-0/ {print $10}' your.txt
Note that this will print: $3.57 - with the leading $, if you don't like this, pipe the output to tr:
awk '/MC exp. sig-250-0/ {print $10}' your.txt | tr -d '$'
In comments you wrote that you need to call it in a script like this:
while read p ; do
echo $p,awk '/MC exp. sig-$p/ {print $10}' filename | tr -d '$'
done < grid.txt
Note that you need a sub shell $() for the awk pipe. Like this:
echo "$p",$(awk '/MC exp. sig-$p/ {print $10}' filename | tr -d '$')
If you want to pass a shell variable to the awk pattern use the following syntax:
awk -v p="MC exp. sig-$p" '/p/ {print $10}' a.txt | tr -d '$'
More lines would've been nice but I guess you would like to have a simple use awk.
awk '{print $N}' $file
If you don't tell awk what kind of field-separator it has to use it will use just a space ' '. Now you just have to count how many fields you have got to get your field you want to get. In your case it would be 10.
awk '{print $10}' file.txt
$3.57
Don't want the $?
Pipe your awk result to cut:
awk '{print $10}' foo | cut -d $ -f2
-d will use the $ als field-separator and -f will select the second field.
If you know you always have the same number of fields, then
#!/bin/bash
file=$1
key=$2
while read -ra f; do
if [[ "${f[0]} ${f[1]} ${f[2]} ${f[3]}" == "MC exp. $key events" ]]; then
echo ${f[9]}
fi
done < "$file"