I have a file with the following format:
123 2 3 48 85.64 85.95
Park ParkName Location
12 2.2 3.2 48 5.4 8.9
Now I could like to write a shell script to extract lines from this file.
The first item from each line is a kind of flag. For different flags, I will make different process.
See my code below:
head= ` echo "$line" | grep -P "^\d+" `
if [ "$head" != "" ]; then
(do something...)
fi
head=` echo "$line" | grep -P "^[A-Z]+" `
if [ "$head" != "" ]; then
(do something...)
fi
The code works. But I dislike the complicated way of writing 2 "if".
I would like to have something simple like:
if [ "$head" != "" ]; then
(do something...)
elif [ "$head" != "" ]; then
(do something...)
fi
Any thoughts?
How about pure bash solution? Bash has a built-in regexp functionality, that you trigger with ~ character.
Be aware though that processing huge files with bash read line will not yield in optimal performance..
#!/bin/bash
file=$1
while read line
do
echo "read [$line]"
if [[ $line =~ ^[0-9] ]]; then
echo ' Processing numeric line'
elif [[ $line =~ ^[A-Za-z] ]]; then
echo ' Processing a text line'
fi
done < $file
How about this. I guess it would fulfill your requirement
file
123 2 3 48 85.64 85.95
Park ParkName Location
12 2.2 3.2 48 5.4 8.9
script.sh
while read line
do
echo $line | grep -qP "^\d+" && echo "Line starts with Numbers"
echo $line | grep -qP "^[a-zA-Z]+" && echo "Line Starts with String"
done < file
Output:-
bash script.sh
Line starts with Numbers
Line Starts with String
Line starts with Numbers
Related
I have a shell script which looks like this (myscript.sh)
echo $squad $buildnumber $date > file2.txt
I have 4 squads. I am parsing them to myscript.sh from Jenkins pipeline
those squads are:
DOD
ABCD
UAB_MS
DCF
when script write file2.txt, then it has to replace 2 squads. other 2 should return without replacing.
I have to replace:
UAB_MS, DCF to UAB
let's assume if it parse DCF to script. then expected output would be (file2.txt)
UAB 115 2022-09-23
this is what i tried (myscript.sh)
if [ $squad == 'DCF' ]
then
newsquad= sed -i 's/DCF/UAB/g'
elif [ $squad == 'UAB_MS' ]
then
newsquad= sed -i 's/UAB_MS/UAB/g'
else
newsquad=$squad
fi
echo $newsquad $buildnumber $date > file2.txt
Can someone help me to figure out this? Thanks in advance!
Note: I am not allowed to use general purpose scripting language (JavaScript, Python etc).
You can join the commands to one line. I added double quotes (no difference here), for future work it is good to get used to that.
I also added ^ in the replacement, so it will only match the start of the line (in $squad). That way a string "DCF" in $buildnumberwill not be replaced.
echo "$squad $buildnumber $date" | sed 's/^DCF/UAB/; s/^UAB_MS/UAB/' > file2.txt
Based on your example, this should works too.
file='file2.txt'
IFS='
'
for what in $squad; do
if [ "$what" = 'UAB_MS' ] || [ "$what" = 'DCF' ]; then
echo "UAB $buildnumber $date"
else
echo "$what $buildnumber $date"
fi
done > "$file"
I think you can update your script making use of sed in a sub shell and remove the spaces when assining values to variables.
Also remove -i from the sed commands as that means editing a file inplace
#!/bin/bash
squad="DCF"
buildnumber="115"
date="2022-09-23"
if [ $squad == 'DCF' ]
then
newsquad=$(sed 's/DCF/UAB/g' <<< "$squad")
elif [ $squad == 'UAB_MS' ]
then
newsquad=$(sed 's/UAB_MS/UAB/g' <<< "$squad")
else
newsquad=$squad
fi
echo "$newsquad $buildnumber $date" > file2.txt
But what you might also do is use an alternation | with sed to replace either DCF or UAB_MS with UAB
echo "$squad $buildnumber $date" | sed 's/^\(DCF\|UAB_MS\)/UAB/' > file2.txt
Output
UAB 115 2022-09-23
when using sed -e to update some parameters of a config file and pipe it to | tee (to write the updated content into the file), this randomly breaks and causes the file to be invalid (size 0).
In Summary, this code is used for updating parameters:
# based on the provided linenumber, add some comments, add the new value, delete old line
sed -e "$lineNr a # comments" -e "$lineNr a $newValue" -e "$lineNr d" $myFile | sudo tee $myFile
I set up an script which calls this update command 100 times.
In a Ubuntu VM (Parallels Desktop) on a shared Directory with OSX this
behaviour occurs up to 50 times
In a Ubuntu VM (Parallels Desktop) on the
Ubuntu partition this behaviour occurs up to 40 times
On a native System (IntelNUC with Ubuntu) this behaviour occurs up to 15 times
Can someone explain why this is happening?
Here is a fully functional script where you can run the experiment as well. (All necessary files are generated by the script, so you can simply copy/paste it into a bashscriptfile and run it)
#!/bin/bash
# main function at bottom
#====================
#===HELPER METHOD====
#====================
# This method updates parameters with a new value. The replacement is performed linewise.
doUpdateParameterInFile()
{
local valueOfInterest="$1"
local newValue="$2"
local filePath="$3"
# stores all matching linenumbers
local listOfLines=""
# stores the linenumber which is going to be replaced
local lineToReplace=""
# find value of interest in all non-commented lines and store related lineNumber
lineToReplace=$( grep -nr "^[^#]*$valueOfInterest" $filePath | sed -n 's/^\([0-9]*\)[:].*/\1/p' )
# Update parameters
# replace the matching line with the desired value
oldValue=$( sed -n "$lineToReplace p" $filePath )
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" $filePath | sudo tee $filePath >/dev/null
# Sanity check to make sure file did not get corrupted by updating parameters
if [[ ! -s $filePath ]] ; then
echo "[ERROR]: While updating file it turned invalid."
return 31
fi
}
#===============================
#=== Actual Update Function ====
#===============================
main_script()
{
echo -n "Update Parameter1 ..."
doUpdateParameterInFile "Parameter1" "Parameter1 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 33 ; fi
echo -n "Update Parameter2 ..."
doUpdateParameterInFile "Parameter2" "Parameter2=90" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 34 ; fi
echo -n "Update Parameter3 ..."
doUpdateParameterInFile "Parameter3" "Parameter3 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 35 ; fi
}
#=================
#=== Main Loop ===
#=================
#generate file config.txt
printf "# Configfile with 3 Parameters\n#[Parameter1]\n#only takes YES or NO\nParameter1 NO \n\n#[Parameter2]\n#Parameter2 takes numbers\nParameter2 = 100 \n\n#[Parameter3]\n#Parameter3 takes YES or NO \nParameter3 YES\n" > config.txt
cp config.txt config.txt.bkup
# Start the experiment and let it run 100 times
cnt=0
failSum=0
while [[ $cnt != "100" ]] ; do
echo "==========run: $cnt; fails: $failSum======="
main_script
if [[ $? != "0" ]] ; then cp config.txt.bkup config.txt ; failSum=$(($failSum+1)) ; fi
cnt=$((cnt+1))
sleep 0.5
done
regards
DonPromillo
The problem is that you're using tee to overwrite $filepath at the same time as sed is trying to read from it. If tee truncates it first then sed gets an empty file and you end up with a 0 length file at the other end.
If you have GNU sed you can use the -i flag to have sed modify the file in place (other versions support -i but require an argument to it). If your sed doesn't support it you can have it write to a temp file and move it back to the original name like
tmpname=$(mktemp)
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" "$filePath" > "$tmpname"
sudo mv "$tmpname" "$filePath"
or if you want to preserve the original permissions you could do
sudo sh -c "cat '$tmpname' > '$filePath'"
rm "$tmpname"
or use your tee approach like
sudo tee "$filePath" >/dev/null <"$tmpname"
rm "$tmpname"
I'm writing a bash script to read a set of files line by line and perform some edits. To begin with, I'm simply trying to move the files to backup locations and write them out as-is, to test the script is working. However, it is failing to copy the last line of each file. Here is the snippet:
while IFS= read -r line
do
echo "Line is ***$line***"
echo "$line" >> $POM
done < $POM.backup
I obviously want to preserve whitespace when I copy the files, which is why I have set the IFS to null. I can see from the output that the last line of each file is being read, but it never appears in the output.
I've also tried an alternative variation, which does print the last line, but adds a newline to it:
while IFS= read -r line || [ -n "$line" ]
do
echo "Line is ***$line***"
echo "$line" >> $POM
done < $POM.backup
What is the best way to do this do this read-write operation, to write the files exactly as they are, with the correct whitespace and no newlines added?
The command that is adding the line feed (LF) is not the read command, but the echo command. read does not return the line with the delimiter still attached to it; rather, it strips the delimiter off (that is, it strips it off if it was present in the line, IOW, if it just read a complete line).
So, to solve the problem, you have to use echo -n to avoid adding back the delimiter, but only when you have an incomplete line.
Secondly, I've found that when providing read with a NAME (in your case line), it trims leading and trailing whitespace, which I don't think you want. But this can be solved by not providing a NAME at all, and using the default return variable REPLY, which will preserve all whitespace.
So, this should work:
#!/bin/bash
inFile=in;
outFile=out;
rm -f "$outFile";
rc=0;
while [[ $rc -eq 0 ]]; do
read -r;
rc=$?;
if [[ $rc -eq 0 ]]; then ## complete line
echo "complete=\"$REPLY\"";
echo "$REPLY" >>"$outFile";
elif [[ -n "$REPLY" ]]; then ## incomplete line
echo "incomplete=\"$REPLY\"";
echo -n "$REPLY" >>"$outFile";
fi;
done <"$inFile";
exit 0;
Edit: Wow! Three excellent suggestions from Charles Duffy, here's an updated script:
#!/bin/bash
inFile=in;
outFile=out;
while { read -r; rc=$?; [[ $rc -eq 0 || -n "$REPLY" ]]; }; do
if [[ $rc -eq 0 ]]; then ## complete line
echo "complete=\"$REPLY\"";
printf '%s\n' "$REPLY" >&3;
else ## incomplete line
echo "incomplete=\"$REPLY\"";
printf '%s' "$REPLY" >&3;
fi;
done <"$inFile" 3>"$outFile";
exit 0;
After review i wonder if :
{
line=
while IFS= read -r line
do
echo "$line"
line=
done
echo -n "$line"
} <$INFILE >$OUTFILE
is juts not enough...
Here my initial proposal :
#!/bin/bash
INFILE=$1
if [[ -z $INFILE ]]
then
echo "[ERROR] missing input file" >&2
exit 2
fi
OUTFILE=$INFILE.processed
# a way to know if last line is complete or not :
lastline=$(tail -n 1 "$INFILE" | wc -l)
if [[ $lastline == 0 ]]
then
echo "[WARNING] last line is incomplete -" >&2
fi
# we add a newline ANYWAY if it was complete, end of file will be seen as ... empty.
echo | cat $INFILE - | {
first=1
while IFS= read -r line
do
if [[ $first == 1 ]]
then
echo "First Line is ***$line***" >&2
first=0
else
echo "Next Line is ***$line***" >&2
echo
fi
echo -n "$line"
done
} > $OUTFILE
if diff $OUTFILE $INFILE
then
echo "[OK]"
exit 0
else
echo "[KO] processed file differs from input"
exit 1
fi
Idea is to always add a newline at the end of file and to print newlines only BETWEEN lines that are read.
This should work for quite all text files given they are not containing 0 byte ie \0 character, in which case 0 char byte will be lost.
Initial test can be used to decided whether an incomplete text file is acceptable or not.
Add a new line if line is not a line. Like this:
while IFS= read -r line
do
echo "Line is ***$line***";
printf '%s' "$line" >&3;
if [[ ${line: -1} != '\n' ]]
then
printf '\n' >&3;
fi
done < $POM.backup 3>$POM
enter image description hereI was trying to see how a shell scripts work and how to run them, so I toke some sample code from a book I picked up from the library called "Wicked Cool Shell Scripts"
I re wrote the code verbatim, but I'm getting an error from Linux, which I compiled the code on saying:
'd.sh: line 3: syntax error near unexpected token `{
'd.sh: line 3:`gmk() {
Before this I had the curly bracket on the newline but I was still getting :
'd.sh: line 3: syntax error near unexpected token
'd.sh: line 3:`gmk()
#!/bin/sh
#format directory- outputs a formatted directory listing
gmk()
{
#Give input in Kb, output converted to Kb, Mb, or Gb for best output format
if [$1 -ge 1000000]; then
echo "$(scriptbc -p 2 $1/1000000)Gb"
elif [$1 - ge 1000]; then
echo "$$(scriptbc -p 2 $1/1000)Mb"
else
echo "${1}Kb"
fi
}
if [$# -gt 1] ; then
echo "Usage: $0 [dirname]" >&2; exit 1
elif [$# -eq 1] ; then
cd "$#"
fi
for file in *
do
if [-d "$file"] ; then
size = $(ls "$file"|wc -l|sed 's/[^[:digit:]]//g')
elif [$size -eq 1] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size ="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(gmk $size))|"
fi
done | \
sed 's/ /^^^/g' |\
xargs -n 2 |\
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%39s %-39s\n", $1, $2}'
exit 0
if [$#-gt 1]; then
echo "Usage :$0 [dirname]" >&2; exit 1
elif [$# -eq 1]; then
cd "$#"
fi
for file in *
do
if [ -d "$file" ] ; then
size =$(ls "$file" | wc -l | sed 's/[^[:digit:]]//g')
if [ $size -eq 1 ] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size ="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(convert $size))|"
fi
done | \
sed 's/ /^^^/g' | \
xargs -n 2 | \
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%-39s %-39s\n", $1, $2 }'
exit 0
sh is very sensitive to spaces. In particular assignment (no spaces around =) and testing (must have spaces inside the [ ]).
This version runs, although fails on my machine due to the lack of scriptbc.
You put an elsif in a spot where it was supposed to be if.
Be careful of column alignment between starts and ends. If you mismatch them it will easily lead you astray in thinking about how this works.
Also, adding a set -x near the top of a script is a very good way of debugging what it is doing - it will cause the interpreter to output each line it is about to run before it does.
#!/bin/sh
#format directory- outputs a formatted directory listing
gmk()
{
#Give input in Kb, output converted to Kb, Mb, or Gb for best output format
if [ $1 -ge 1000000 ]; then
echo "$(scriptbc -p 2 $1/1000000)Gb"
elif [ $1 -ge 1000 ]; then
echo "$(scriptbc -p 2 $1/1000)Mb"
else
echo "${1}Kb"
fi
}
if [ $# -gt 1 ] ; then
echo "Usage: $0 [dirname]" >&2; exit 1
elif [ $# -eq 1 ] ; then
cd "$#"
fi
for file in *
do
if [ -d "$file" ] ; then
size=$(ls "$file"|wc -l|sed 's/[^[:digit:]]//g')
if [ $size -eq 1 ] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(gmk $size))|"
fi
done | \
sed 's/ /^^^/g' |\
xargs -n 2 |\
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%39s %-39s\n", $1, $2}'
exit 0
By the way, with respect to the book telling you to modify your PATH variable, that's really a bad idea, depending on what exactly it advised you to do. Just to be clear, never add your current directory to the PATH variable unless you intend on making that directory a permanent location for all of your scripts etc. If you are making this a permanent location for your scripts, make sure you add the location to the END of your PATH variable, not the beginning, otherwise you are creating a major security problem.
Linux and Unix do not add your current location, commonly called your PWD, or present working directory, to the path because someone could create a script called 'ls', for example, which could run something malicious instead of the actual 'ls' command. The proper way to execute something in your PWD, is to prepend it with './' (e.g. ./my_new_script.sh). This basically indicates that you really do want to run something from your PWD. Think of it as telling the shell "right here". The '.' actually represents your current directory, in other words "here".
I'm working on a script that takes commands from MySQL and executes them one by one. My problem is I can't execute the commands from script:
./bash.sh: line 26: /myscript.sh -c "": no such file or directory
Line 26 is the command I wish to execute ("$com"). if i try to run the command manually, by echoing the content of "$com" and than run it from terminal, it's working.
What am I doing wrong?
if [ ! "${#array[*]}" -eq "0" ]; then
for (( i=0 ; i<cnt ; i++ )); do
id=$(echo "${array[$i]}" | sed 's#\t#^#g' | cut -f'1' -d'^')
com=$(echo "${array[$i]}" | sed 's#\t#^#g' | cut -f'2' -d'^')
imp=$(echo "${array[$i]}" | sed 's#\t#^#g' | cut -f'3' -d'^')
if [[ "$id" = [0-9]* ]]; then
"$com"
echo "DELETE FROM list WHERE id='$id'" | mysql "$DB_USER" -u "$DB_USER" -p"$DB_PASS"
fi
done
else
echo "The list is empty"
fi
Added as Community Wiki
This question was solved in the comments by Vaughn Cato and the OP hasn't been seen since March.
The accept answer there was to use eval so the if statement in the script would be:
if [[ "$id" = [0-9]* ]]; then
eval "$com"
echo "DELETE FROM list WHERE id='$id'" | mysql "$DB_USER" -u "$DB_USER" -p"$DB_PASS"
fi
Change
"$com"
rather to
$com
(remember: too many quotes can be as harmful as too few)
than to
eval "$com"
it is unnecessarily complicated to use eval just to undo the unwanted effect of quoting.