sed command issue with string replacement - string

I'm having a weird problem with the sed command.
I have a script that take a c file, copy it X times and then replace the name of the functions inside it by adding number to the name.
For example:
originalFile.c contains these functions check0, check1 check2
The script will generate those file:
originalFile1.c: check0 check1 check2
originalFile2.c: check3 check4 check5
originalFile3.c: check6 check7 check8
... and so on.
Now the problem... If I generate enough files so the number goes up to 10,20 or more I noticed something in the name of the function. The first function of the file is renamed incorrectly but the other are corrects. For example:
originalFileX.c: __check165__ check16 check17
...
originalFileZ.c: __check297__ __check298__ check29 -> in this file 2 names are incorrects.
Also, If I print the name with echo everything is correct. Do you have any idea what could be wrong?
Here is my script (I run it under OSX):
#!/bin/bash
NUMCHECK=3
# $1: filename
# $2: number of function in the file
# $3: number of function I want to generate
# $4: function basename
function replace_name() {
FILE_NUM=$((($3+($2-1))/$2))
TMP=0
for (( i=1; i<$FILE_NUM+1; i++ ))
do
cp $1.mm test/$1$i.mm
for (( j=0; j<$2; j++ ))
do
OLDNAME="$4$j"
NEWNAME="$4$TMP"
echo $OLDNAME:$NEWNAME
sed -i "" "s/$OLDNAME/$NEWNAME/g" test/$1$i.mm
TMP=$(($TMP+1))
done
done
}
replace_name check $NUMCHECK 60 check

Youre doing 3 runs of the sed in each file. Just imagine the following
sed -i s/check0/check150/g test/check51.mm
sed -i s/check1/check151/g test/check51.mm
sed -i s/check2/check152/g test/check51.mm
The
s/check0/check150/g changes the check0 to check150 - ok
s/check1/check151/g will change the previous check150 to check15150 (because it finds the check1 string in the check150 too, from the previous step).
etc...
You need more precisely define your regex. because here isn't any example input, can't help more.

Related

Nested bash loop runs but logic is not correct

I have this nested bash for loop where the first loop represents the range of servers, and the second loop reads each lines of the .txt file. Then will do a zgrep -i and find for a string on the matching line.
keys.txt is as follow:
1121313232323-1
1121313244234-1
1121867575757-1
Bash Loops:
for i in {1..80}; do for j in `cat keys.txt`; do zgrep -i 'InitialRequest' /var/logs/server$i/* >> /var/log/result.log; done; done &
This runs but its logging every instance where 'InitialRequest' is defined. What im looking for is to only log lines where the key line is matched to 'InitialRequest' with their key value.
The result i'm expecting is like this (with Key value from J inner loop 1121313232323-1 and with string 'InitialRequest' plus the remaining of the line:
1121313232323-1 InitialRequest http://someramdomurl.com
What I'm getting is like (just the matching string without the Key value from the J inner loop:
InitialRequest http://someramdomurl.com
InitialRequest http://someotherramdomurl.com
InitialRequest http://someotherotherramdomurl.com
Can anyone advise where is my error?
Your code is not zgreping $j in the files. For better clarity I will replace $i with $server and $j with $key in my answer. Check it out:
for server in server{1..80}; do
for key in $(cat keys.txt); do
zgrep -i "$key InitialRequest" /var/logs/$server/* >>/var/log/result.log
done
done
But since you are always using the same keys, you just need to read them once. Here is a better version:
keys=($(cat keys.txt))
for server in server{1..80}; do
for key in ${keys[*]}; do
zgrep -i "$key InitialRequest" /var/logs/$server/* >>/var/log/result.log
done
done

Using $ notation in middle of C-Shell statement

I have a bunch of directories to process, so I start a for loop like this:
foreach n (1 2 3 4 5 6 7 8)
Then I have a bunch of commands where I am copying over a few files from different places
cp file1 dir$n
cp file2 dir$n
but I have a couple commands where the $n is in the middle of the command like this:
cp -r dir$nstep1 dir$n
When I run this command, the shell complains that it cannot find the variable $nstep1. What i want to do is evaluate the $n first and then concatenate the text around it. I tried using `` and (), but neither of those work. How to do this in csh?
In this respect behavior is similar to POSIX shells:
cp -r "dir${n}step1" "dir${n}"
The quotes prevent string-splitting and glob expansion. To observe what this means, compare the following:
# prints "hello * cruel * world" on one line
set n=" * cruel * "
printf '%s\n' "hello${n}world"
...to this:
# prints "hello" on one line
# ...then a list of files in the current directory each on their own lines
# ...then "cruel" on another line
# ...then a list of files again
# ... and then "world"
set n=" * cruel * "
printf '%s\n' hello${n}world
In real-world cases, correct quoting can thus be the difference between deleting the oddly-named file you're trying to operate on, and deleting everything else in the directory as well.

How to extract key value pairs from a file when values span multiple lines?

I'm a few weeks into bash scripting and I haven't advanced enough yet to get my head wrapped around this problem. Any help would be appreciated!
I have a "script.conf" file that contains the following:
key1=value1
key2=${HOME}/Folder
key3=( "k3v1" "k3 v2" "k3v3")
key4=( "k4v1"
"k4 v2"
"k4v3"
)
key5=value5
#key6="Do Not Include Me"
In a bash script, I want to read the contents of this script.conf file into an array. I've learned how to handle the scenarios for keys 1, 2, 3, and 5, but the key4 scenario throws a wrench into it with it spanning across multiple lines.
I've been exploring the use of sed -n '/=\s*[(]/,/[)]/{/' which does capture key4 and its value, but I can't figure out how to mix this so that the other keys are also captured in the matches. The range syntax is also new to me, so I haven't figured out how to separate the key/value. I feel like there is an easy regex that would accomplish what I want... in plain-text: "find and group the pattern ^(.*)= (for the key), then group everything after the '=' char until another ^(.*)= match is found, rinse and repeat". I guess if I do this, I need to change the while read line to not handle the key/value separation for me (I'll be looking into this while I'm waiting for a response). BTW, I think a solution where the value of key4 is flattened (new lines removed) would be acceptable; I know for key3 I have to store the value as a string and then convert it to an array later when I want to iterate over it since an array element apparently can't contain a list.
Am I on the right path with sed or is this a job for awk or some other tool? (I haven't ventured into awk yet). Is there an easier approach that I'm missing because I'm too deep into the forest (like changing the while read line in the LoadConfigFile function)?
Here is the code that I have so far in script.sh for processing and capturing the other pairs into the $config array:
__AppDir=$(dirname $0)
__AppName=${__ScriptName%.*}
typeset -A config #init config array
config=( #Setting Default Config values
[key1]="defaultValue1"
[key2]="${HOME}/defaultFolder"
[QuietMode]=0
[Verbose]=0 #Ex. Usage: [[ "${config[Verbose]}" -gt 0 ]] && echo ">>>Debug print"
)
function LoadConfigFile() {
local cfgFile="${1}"
shopt -s extglob #Needed to remove trailing spaces
if [ -f ${cfgFile} ]; then
while IFS='=' read -r key value; do
if [[ "${key:0:1}" == "#" ]]; then
#echo "Skipping Comment line: ${key}"
elif [ "${key:-EMPTY}" != "EMPTY" ]; then
value="${value%%\#*}" # Delete in-line, right comments
value="${value%%*( )}" # Delete trailing spaces
value="${value%%( )*}" # Delete leading spaces
#value="${value%\"*}" # Delete opening string quotes
#value="${value#\"*}" # Delete closing string quotes
#Manipulate any variables included in the value so that they can be expanded correctly
# - value must be stored in the format: "${var1}". `backticks`, "$var2", and "doubleQuotes" are left as is
value="${value//\"/\\\"}" # Escape double quotes for eval
value="${value//\`/\\\`}" # Escape backticks for eval
value="${value//\$/\\\$}" # Escape ALL '$' for eval
value="${value//\\\${/\${}" # Undo the protection of '$' if it was followed by a '{'
value=$(eval "printf '%s\n' \"${value}\"")
config[${key}]=${value} #Store the value into the config array at the specified key
echo " >>>DBG: Key = ${key}, Value = ${value}"
#else
# echo "Skipped Empty Key"
fi
done < "${cfgFile}"
fi
}
CONFIG_FILE=${__AppDir}/${__AppName}.conf
echo "Config File # ${CONFIG_FILE}"
LoadConfigFile ${CONFIG_FILE}
#Print elements of $config
echo "Script Config Values:"
echo "----------------------------"
for key in "${!config[#]}"; do #The '!' char gets an array of the keys, without it, we would get an array of the values
printf " %-20s = %s\n" "${key}" "${config[${key}]}"
done
echo "------ End Script Config ------"
#To convert to an array...
declare -a valAsArray=${config[RequiredAppPackages]} #Convert the value from a string to an array
echo "Count = ${#valAsArray[#]}"
for itemCfg in "${valAsArray[#]}"; do
echo " item = ${itemCfg}"
done
As I mentioned before, I'm just starting to learn bash and Linux scripting in general, so if you see that I'm doing some taboo things in other areas of my code too, please feel free to provide feedback in the comments... I don't want to start bad habits early on :-).
*If it matters, the OS is Ubuntu 14.04.
EDIT:
As requested, after reading the script.conf file, I would like for the elements in $config[#] to be equivalent to the following:
typeset -A config #init config array
config=(
[key1]="value1"
[key2]="${HOME}/Folder"
[key3]="( \"k3v1\" \"k3 v2\" \"k3v3\" )"
[key4]="( \"k4v1\" \"k4 v2\" \"k4v3\" )"
[key5]="value5"
)
I want to be able to convert the values of elements 'key4' and 'key3' into an array and iterated over them the same way in the following code:
declare -a keyValAsArray=${config[keyN]} #Convert the value from a string to an array
echo "Count = ${#keyValAsArray[#]}"
for item in "${keyValAsArray[#]}"; do
echo " item = ${item}"
done
I don't think it matters if \n is preserved for key4's value or not... that depends on if declare has a problem with it.
A shell is an environment from which to call tools with a language to sequence those calls. It is NOT a tool to manipulate text. The standard UNIX tool to manipulate text is awk. Trying to manipulate text in shell IS a bad habit, see why-is-using-a-shell-loop-to-process-text-considered-bad-pr‌​actice for SOME of the reasons why
You still didn't post the expected result of populating the config array so I'm not sure but I think this is what you wanted:
$ cat tst.sh
declare -A config="( $(awk '
{ gsub(/^[[:space:]]+|([[:space:]]+|#.*)$/,"") }
!NF { next }
/^[^="]+=/ {
name = gensub(/=.*/,"",1)
value = gensub(/^[^=]+=/,"",1)
n2v[name] = value
next
}
{ n2v[name] = n2v[name] OFS $0 }
END {
for (name in n2v) {
value = gensub(/"/,"\\\\&","g",n2v[name])
printf "[%s]=\"%s\"\n", name, value
}
}
' script.conf
) )"
declare -p config
$ ./tst.sh
declare -A config='([key5]="value5" [key4]="( \"k4v1\" \"k4 v2\" \"k4v3\" )" [key3]="( \"k3v1\" \"k3 v2\" \"k3v3\")" [key2]="/home/Ed/Folder" [key1]="value1" )'
The above uses GNU awk for gensub(), with other awks you'd use [g]sub() instead.

Printing IFS values from file

Script:
#!/bin/ksh
FILENAME=$1
while read RECORD VALUE
do
echo ${RECORD} ${VALUE} "X"
done <"$FILENAME"
input file:
A 1
B 2
The output of script:
X1
X2
If I remove from echo "x", e.g.
echo ${RECORD} ${VALUE}
I am getting
A 1
B 2
what is wrong?
Update:
If I do
echo "X" ${RECORD} ${VALUE}
it prints correctly:
X A 1
X B 2
and :
echo ${RECORD} "X"
also prints correctly, so i am guessing the issues is with VALUE that maybe contains return carriage symbol (as input file was created on windows)
adding this inside the loop:
VALUE=`echo $VALUE| tr -d '\r'`
solved the issue, if you have a better solution you are more than welcome.
There is a parameter expansion operator you can use to remove a character from the end of a value, if it is present.
VALUE=${VALUE%$'\r'}
This is handled in-shell, without needing to start a new process.

CSV Bash loop Issue with Variables

I have a csv file which im trying to loop through with the purpose to find out if an User Input is found inside the csv data. I wrote the following code which sometimes works and others doesn't. It always stops working when I try to compare to a 2+ digit number. It works OK for numbers 1 through 9, but once u enter lets say 56 , or 99 or 100, it stops working.
the csv data is comma delimited, i have about 300 lines they are just like this.
1,John Doe,Calculus I,5.0
1,John Doe,Calculus II,4.3
1,John Doe,Physics II,3.5
2,Mary Poppins,Calculus I,3.7
2,Mary Poppins,Calculus II,4.7
2,Mary Poppins,Physics I,3.7
Data is just like that, all the way down until ID #100 for a total of 300 lines. Both the sh file and csv file are in the same folder, I'm using a fresh installation of Ubuntu 12.04.3, using gedit as the text editor.
I tried Echoing the variables ID and inside the IF conditionals but it doesn't behave the way it should when testing for the same value. Could someone point me out in the right direction. Thanks
Here's the code:
#s!/bin/bash
echo "enter your user ID";
read user;
INPUT_FILE=notas.csv
while IFS="," read r- ID name asignature final;
do
if [$ID = $user]; then
userType=1;
else
userType=2;
fi
done < notas.csv
Well, your code as written has a few issues.
You have r- instead of -r on the read line - I assume that's a typo not present in your actual code or you wouldn't get very far.
Similarly, you need space around the [...] brackets: [$ID is a syntax error.
You need to quote the parameter expansions in your if clause, and/or switch bracket types. You probably make it a numeric comparison as #imp25 suggested, which I would do by using ((...)).
You probably don't want to set userType to 2 in an else clause, because that will set it to 2 for everyone except whoever is listed last in the file (ID 100, presumably). You want to set it to 2 first, outside the loop. Then, inside the loop when you find a match, set it to 1 and break out of the loop:
userType=2
while IFS=, read -r ID name asignature final; do
if (( $ID == $user )); then
userType=1;
break
fi
done < notas.csv
You could also just use shell tools like awk:
userType=$(awk -F, -vtype=2 '($1=="'"$user"'") {type=1}; END {print type}' notas.csv)
or grep:
grep -q "^$user," notas.csv
userType=$(( $? + 1 ))
etc.
You should quote your variables in the if test statement. You should also perform a numeric test -eq rather than a string comparison =. So your if statement should look like:
if [[ "$ID" -eq "$user" ]]

Resources