I have file which contains,
{"value":true,"message":"fail","timestamp":1445956265,"data":"off
"value":true,"message":"error","timestamp":1445956265,"data":"off"}
I want to extract above 2 lines between curly braces by excluding { and }
Expected output should be like,
"value":true,"message":"fail","timestamp":1445956265,"data":"off
"value":true,"message":"error","timestamp":1445956265,"url":"off"
I am using shell script for this. how it can be done...?
awk
awk 'sub("[{}]","")' filename
try:
#!/bin/bash
FILE="temp.txt"
if [[ ! -f $FILE ]]; then
echo "File not found: $FILE"
exit 1
fi
out=$(awk 'sub("[{}]","")' $FILE | grep '"value":true')
if [[ ! -z $out ]]; then
for i in $out; do
echo "i= $i"
done
fi
Related
This is part of my code that I working to find a word in a remote server connecting via ssh to that server
filename=test.repo
word=fail
exists=$(grep -c $word $filename)
file_server=$1
for i in $(cat $file_server)
do echo ''; echo $1 ;
ssh $i "printf '\e[33m'; hostname; printf '\e[0m\n]' ;
cd /etc/yum.repos.d/;
grep -c $word $filename;
echo $exists;
if [[ $exists -gt 0 ]];
then printf 'Keyword found, cleanup starts \n';
else printf ''$word', was not found, nothing to do here with '$filename'. \n';
fi"
done
This works with just the Do command, but if I add the if [[$exist -gt 0]]; , this got an error
bash: -c: line 4: conditional binary operator expected
bash: -c: line 4: syntax error near ;' bash: -c: line 4: if [[ -gt 0]]; '
any suggestion
This line should be cut from the beginning of the script:
exists=$(grep -c $word $filename)
And replace the call to grep inside the loop.
What I do, with some help of a colleague to fix this avoid using the variable on the ssh session.
file_server=$1
filename=testfile.repo
keyword=testword
if [[ -z $1 ]]
then
printf "\t\e[0;31m==================== ERROR ======================\n"
printf "\tServer list not found\e[0;0m\n"
exit 1
fi
for i in $(cat $file_server)
do echo ''; echo $1 ;
ssh $i "printf '\e[33m'; hostname; printf '\e[0m\n';
cd /etc/yum.repos.d/;
if grep -q $keyword $filename;
then
printf '\e[33m$keyword found, cleanup starts\e[0m\n';
sed -i.BAK '/^failovermethod=/d' $filename
grep failovermethod $filename;
printf '\e[33mBackup file created with the $keyword with this name $filename.BAK\e[0m\n';
grep 'failovermethod' pgdg-redhat-all-test.repo.BAK;
else
printf '\e[33m${keyword^^}\e[32m, was not found, nothing to do here. \e[0m\n';
fi;";
done
I'm writing a bash script to read a set of files line by line and perform some edits. To begin with, I'm simply trying to move the files to backup locations and write them out as-is, to test the script is working. However, it is failing to copy the last line of each file. Here is the snippet:
while IFS= read -r line
do
echo "Line is ***$line***"
echo "$line" >> $POM
done < $POM.backup
I obviously want to preserve whitespace when I copy the files, which is why I have set the IFS to null. I can see from the output that the last line of each file is being read, but it never appears in the output.
I've also tried an alternative variation, which does print the last line, but adds a newline to it:
while IFS= read -r line || [ -n "$line" ]
do
echo "Line is ***$line***"
echo "$line" >> $POM
done < $POM.backup
What is the best way to do this do this read-write operation, to write the files exactly as they are, with the correct whitespace and no newlines added?
The command that is adding the line feed (LF) is not the read command, but the echo command. read does not return the line with the delimiter still attached to it; rather, it strips the delimiter off (that is, it strips it off if it was present in the line, IOW, if it just read a complete line).
So, to solve the problem, you have to use echo -n to avoid adding back the delimiter, but only when you have an incomplete line.
Secondly, I've found that when providing read with a NAME (in your case line), it trims leading and trailing whitespace, which I don't think you want. But this can be solved by not providing a NAME at all, and using the default return variable REPLY, which will preserve all whitespace.
So, this should work:
#!/bin/bash
inFile=in;
outFile=out;
rm -f "$outFile";
rc=0;
while [[ $rc -eq 0 ]]; do
read -r;
rc=$?;
if [[ $rc -eq 0 ]]; then ## complete line
echo "complete=\"$REPLY\"";
echo "$REPLY" >>"$outFile";
elif [[ -n "$REPLY" ]]; then ## incomplete line
echo "incomplete=\"$REPLY\"";
echo -n "$REPLY" >>"$outFile";
fi;
done <"$inFile";
exit 0;
Edit: Wow! Three excellent suggestions from Charles Duffy, here's an updated script:
#!/bin/bash
inFile=in;
outFile=out;
while { read -r; rc=$?; [[ $rc -eq 0 || -n "$REPLY" ]]; }; do
if [[ $rc -eq 0 ]]; then ## complete line
echo "complete=\"$REPLY\"";
printf '%s\n' "$REPLY" >&3;
else ## incomplete line
echo "incomplete=\"$REPLY\"";
printf '%s' "$REPLY" >&3;
fi;
done <"$inFile" 3>"$outFile";
exit 0;
After review i wonder if :
{
line=
while IFS= read -r line
do
echo "$line"
line=
done
echo -n "$line"
} <$INFILE >$OUTFILE
is juts not enough...
Here my initial proposal :
#!/bin/bash
INFILE=$1
if [[ -z $INFILE ]]
then
echo "[ERROR] missing input file" >&2
exit 2
fi
OUTFILE=$INFILE.processed
# a way to know if last line is complete or not :
lastline=$(tail -n 1 "$INFILE" | wc -l)
if [[ $lastline == 0 ]]
then
echo "[WARNING] last line is incomplete -" >&2
fi
# we add a newline ANYWAY if it was complete, end of file will be seen as ... empty.
echo | cat $INFILE - | {
first=1
while IFS= read -r line
do
if [[ $first == 1 ]]
then
echo "First Line is ***$line***" >&2
first=0
else
echo "Next Line is ***$line***" >&2
echo
fi
echo -n "$line"
done
} > $OUTFILE
if diff $OUTFILE $INFILE
then
echo "[OK]"
exit 0
else
echo "[KO] processed file differs from input"
exit 1
fi
Idea is to always add a newline at the end of file and to print newlines only BETWEEN lines that are read.
This should work for quite all text files given they are not containing 0 byte ie \0 character, in which case 0 char byte will be lost.
Initial test can be used to decided whether an incomplete text file is acceptable or not.
Add a new line if line is not a line. Like this:
while IFS= read -r line
do
echo "Line is ***$line***";
printf '%s' "$line" >&3;
if [[ ${line: -1} != '\n' ]]
then
printf '\n' >&3;
fi
done < $POM.backup 3>$POM
I am trying to use whether or not a line contains a date as a condition for an if statement:
if [grep -n -v '[0-9][0-9][0-9][0-9]' $line |wc -l==0]
then
...
The above returns an error. I don't necessarily need to use grep. The line processed by grep would look like:
1984 Dan Marino QB Miami Dolphins
Any help is appreciated.
if [[ $(echo $line | grep -q '[0-9][0-9][0-9][0-9]') ]]; then
# do something
fi
You can check this using bash built-ins:
re='\b[[:digit:]]{4}\b'
if [[ $line =~ $re ]] ; then
echo ok;
fi
[grep -n -v '[0-9][0-9][0-9][0-9]' $line |wc -l==0]
problem 1: [(space).....(space)] you need those spaces
problem 2: there is no [ foo==bar ] you can do something like [ $(echo "0") = "0" ] or [[ $(echo "0") == 0 ]] here the $(echo "0") is an example, you should fill with your commands.
You can just call grep with -q option and check the return value:
if [ $(grep -qv '[0-9][0-9][0-9][0-9]' $line) -eq 0 ]; then
# ...
fi
Use command substitution and proper bash syntax.
[[ "`grep -n -v '[0-9][0-9][0-9][0-9]' $line | wc -l`" -eq 0 ]]
Hi I have used this code to find the .java file from the given path and generate the list of the .java files from the path to output file,and count total number of tabs and spaces in each java file.now i need to write the two while loops in function and call that function in case statement, how to do this?
Code:
#!/bin/sh
#
output=/home/user/Desktop/file-list.txt
path=/home/user/Desktop
find $path -type f -name \*.java > $output
echo "Generating files list..."
echo "Done"
while IFS= read file
do
if [ -f "$file" ]; then
spaces=$(tr -cd '\s' < "$file" | wc -c);
echo "$spaces spaces in file $file" >> "/home/user/Desktop/space-count.txt"
fi
done < "$output"
echo "Space Count Done!"
while IFS= read file
do
if [ -f "$file" ]; then
tabs=$(tr -cd '\t' < "$file" | wc -c);
echo "$tabs tabs in file $file" >> "/home/user/Desktop/tab-count.txt"
fi
done < "$output"
echo "Tab Count Done!"
Here is an example of case, but the name of the character could be passed to the function
thereby eliminating the need to case the char.
function count_char
{
flist=$1
ch=$2
case $ch in
" ") chName=space;;
"\t") chName=tab;;
esac
while IFS= read file
do
if [ -f "$file" ]; then
tot=$(tr -cd $ch < "$file" | wc -c);
echo "$tot $chName in file $file" >> "/home/user/Desktop/tab-count.txt"
fi
done < "$flist"
}
# setup code, find command here
# ....
count_char $output " "
count_char $output "\t"
I have a rather large csv file where each line should end with a pipe (|) and if it doesn't combine the next line into it until find a pipe again. This need to done using a shell script.
I got an answer as
awk '!/|$/{l=l""$0|next|}{print l""$0|l=""}' file
But it gives me error as size of each line is quite large for me. I found out that I should be using perl to do that and have tried something as below but it does produce the desired result.
perl -pe 's/^\n(|\n)/ /gs' input.csv > output.csv
My data looks like
A|1|abc|<xml/>|
|2|def|<xml
>hello world</xml>|
|3|ghi|<xml/>|
And the desired output should be
A|1|abc|<xml/>|
|2|def|<xml>hello world</xml>|
|3|ghi|<xml/>|
Obviously the line size is quite large than the sample input here.
Any help would be highly appreciated.
awk '{printf "%s",$0} /[|][[:space:]]*$/ {print ""}'
Print every line without a newline. If the last non-whitespace character is a pipe, you have a complete line so print a newline.
This should work:
perl -lne 'unless(/\|$/){$line=$line.$_}else{print $line." $_";undef $line}' your_file
if you want to do an inplace replacement do this:
perl -i -lne 'unless(/\|$/){$line=$line.$_}else{print $line." $_";undef $line}' your_file
check here regarding your comment
This should happily handle all cases for you, and not break on any line length:
#!/bin/bash
newLine=0
IFS=
while read -r -n 1 char; do
if [[ $char =~ ^$ ]]; then
if [[ $newLine -eq 1 ]]; then
newLine=0
echo '|' # add a newline
fi
elif [[ $char =~ . && ( $newLine -eq 1 ) ]]; then
newLine=0
echo -n "|$char"
elif [[ $char =~ [|] ]]; then
if [[ $newLine -eq 1 ]]; then
echo -n '|'
fi
newLine=1
else
echo -n $char
fi
done < file.txt
Please note that building a lexer by hand in bash is usually a bad idea.