We work with Red Hat Linux (version 3.10.0-1160.59.1.el7.x86_6) and we are running into a problem with a process we have that reads data from a text file and then uses that data to pass parameters into a script file.
We have a control shell file that looks as follows.
year=$(date +"%Y%m%d")
xargs -L1 --arg-file=/u01/data.txt /u01/process.sh $year $1 $2 $3 $4
exit
As you can see, in the control file we simply get the current system date and then we read the data coming from a text file using xargs. The text file looks as follows.
John McEnroe Tennist US
Willie Nelson Singer US
"Jose Maria" Napoleon Singer Spain
Luciano Pavarotti Singer Italy
Finally, the process file that looks as follows (I have added the echos just to see the data coming into the process file).
. /u01/app/env/bt.env
echo "TK Total parms: $#"
echo "Parm1 $1"
echo "Parm2 $2"
echo "Parm3 $3"
echo "Parm4 $4"
echo "Parm5 $5"
/u01/app/task /task:localhost:CREATION /user:$LINUX_USER:$LINUX_PASSWORD /params:DATE="$1",NAME="$2",LAST_NAME="$3",JOB="$4",COUNTRY="$5" /debug
sleep 2
exit
The problem we have is with the row where the Name is composed by two words ("Jose" and "Maria"). I thought that if we added quotes around the entire value, that would work well, but it does not: the process file is getting the two words and treating them as if "Jose" was parameter #3, and "Maria" as if it was parameter #4 (and this causes the process to crash).
I have tried to modify the data file to use apostrophes instead of quotes (i.e., 'Jose Maria'); I also tried to wrap the quote around apostrophes and viceversa (i.e. "'Jose Maria'", '"Jose Maria"'), but none of these methods fixes the problem.
Could anyone suggest me how to write the data file to make it work as we need?
Thanks in advance!
Related
We have a tool for cutting adaptors https://github.com/vsbuffalo/scythe/blob/master/README.md and we wanted it to be used on all the files in the raw folder and make an output of each file separately as OUT+File Name.
Something is wrong with this script I wrote, because it doesn't take each file separately, and the whole thing doesn't work properly. It's gonna generateing empty file named OUT+files
Expected operation will looks:
take file1, use scythe on it, write output as OUTfile1
take file2 etc.
#!/bin/bash
FILES=/home/dave/raw/*
for f in $FILES
do
echo "Processing the $f file..."
/home/deve/scythe/scythe -a /home/dev/scythe/illumina_adapters.fa -o "OUT"+$f $f
done
Additionally, I noticed (testing for a single file) that the script uses only one core out of 130 available. Is there any way to improve it?
There is no string concatenation operator in shell. Use juxtaposition instead; it's "OUT$f", not "OUT"+$f.
I am new to Shellscripting.I am working on a poc in which a script should read a log file and then append to a existing file for the purpose of alert.It should work as per below
There will be some predefined format according to which it will decide whether to append in file or not.For example:
WWXXX9999XS message
**XXX** - is a 3 letter acronym (application code) like for **tom** for tomcat application
9999 - is a 4 numeric digit in the range 1001-1999
**E or X** - For notification X ,If open/active alerts already existing for same error code and same message,new alerts will not be raised for existing one.Once you have closed existing alerts,it will raise alarm for new error.There is any change in message for same error code from existing one, it will raise a alarm even though open/active alerts present.
X option is only for drop duplicates on code and message otherwise all alert mechanisms are same.
**S** - is the severity level, I.e 2,3
**message** - is any text that will be displayed
The script will examine the log file, and look for error like cloud server is down,then it would append 'wwclo1002X2 cloud server is down'if its a new alert.
2.If the same alert is coming again,then it should append 'wwclo1002E2 cloud server is down
There are some very handy commands you can use to do this type of File manipulation. I've updated this in response to your comment to allow functionality that will check if the error has already been appended to the new file.
My suggestion would be that there is enough functionality here to warrant saving it in a bash script.
My approach would be to use a combination of less, grep and > to read and parse the file and then append to the new file. First save the following into a bash script (e.g. a file named script.sh)
#!/bin/bash
result=$(less $1 | grep $2)
exists=$(less $3 | grep $2)
if [[ "$exists" == "$result" ]]; then
echo "error, already present in file"
exit 1
else
echo $result >> $3
exit 0
fi
Then use this file in the command passing in the log file as the first argument, the string to search for as the second argument and the target results file as the third argument like this:
./script.sh <logFileName> "errorToSearchFor" <resultsTargetFileName>
Don't forget to run the file you will need to change the permissions - you can do this using:
chmod u+x script.sh
Just to clarify as you have mentioned you are new to scripting - the less command will output the entire file, the | command (an unnamed pipe) will pass this output to the grep command which will then search the file for the expression in quotes and return all lines from the file containing that expression. The output of the grep command is then appended to the new file with >>.
You may need to tailor the expression in quotes after grep to get exactly the output you want from the log file.
The filenames are just placeholders, be sure to update these with the correct file names. Hope this helps!
Note updated > to >> (single angle bracket overwrites, double angle bracket appends
This question already has answers here:
Read user input inside a loop
(6 answers)
Closed 5 years ago.
First post here! I really need help on this one, I looked the issue on google, but can't manage to find an useful answer for me. So here's the problem.
I'm having fun coding some like of a framework in bash. Everyone can create their own module and add it to the framework. BUT. To know what arguments the script require, I created an "args.conf" file that must be in every module, that kinda looks like this:
LHOST;true;The IP the remote payload will connect to.
LPORT;true;The port the remote payload will connect to.
The first column is the argument name, the second defines if it's required or not, the third is the description. Anyway, long story short, the framework is supposed to read the args.conf file line by line to ask the user a value for every argument. Here's the piece of code:
info "Reading module $name argument list..."
while read line; do
echo $line > line.tmp
arg=`cut -d ";" -f 1 line.tmp`
requ=`cut -d ";" -f 2 line.tmp`
if [ $requ = "true" ]; then
echo "[This argument is required]"
else
echo "[This argument isn't required, leave a blank space if you don't wan't to use it]"
fi
read -p " $arg=" answer
echo $answer >> arglist.tmp
done < modules/$name/args.conf
tr '\n' ' ' < arglist.tmp > argline.tmp
argline=`cat argline.tmp`
info "Launching module $name..."
cd modules/$name
$interpreter $file $argline
cd ../..
rm arglist.tmp
rm argline.tmp
rm line.tmp
succes "Module $name execution completed."
As you can see, it's supposed to ask the user a value for every argument... But:
1) The read command seems to not be executing. It just skips it, and the argument has no value
2) Despite the fact that the args.conf file contains 3 lines, the loops seems to be executing just a single time. All I see on the screen is "[This argument is required]" just one time, and the module justs launch (and crashes because it has not the required arguments...).
Really don't know what to do, here... I hope someone here have an answer ^^'.
Thanks in advance!
(and sorry for eventual mistakes, I'm french)
Alpha.
As #that other guy pointed out in a comment, the problem is that all of the read commands in the loop are reading from the args.conf file, not the user. The way I'd handle this is by redirecting the conf file over a different file descriptor than stdin (fd #0); I like to use fd #3 for this:
while read -u3 line; do
...
done 3< modules/$name/args.conf
(Note: if your shell's read command doesn't understand the -u option, use read line <&3 instead.)
There are a number of other things in this script I'd recommend against:
Variable references without double-quotes around them, e.g. echo $line instead of echo "$line", and < modules/$name/args.conf instead of < "modules/$name/args.conf". Unquoted variable references get split into words (if they contain whitespace) and any wildcards that happen to match filenames will get replaced by a list of matching files. This can cause really weird and intermittent bugs. Unfortunately, your use of $argline depends on word splitting to separate multiple arguments; if you're using bash (not a generic POSIX shell) you can use arrays instead; I'll get to that.
You're using relative file paths everywhere, and cding in the script. This tends to be fragile and confusing, since file paths are different at different places in the script, and any relative paths passed in by the user will become invalid the first time the script cds somewhere else. Worse, you aren't checking for errors when you cd, so if any cd fails for any reason, then entire rest of the script will run in the wrong place and fail bizarrely. You'd be far better off figuring out where your system's root directory is (as an absolute path), then referencing everything from it (e.g. < "$module_root/modules/$name/args.conf").
Actually, you're not checking for errors anywhere. It's generally a good idea, when writing any sort of program, to try to think of what can go wrong and how your program should respond (and also to expect that things you didn't think of will also go wrong). Some people like to use set -e to make their scripts exit if any simple command fails, but this doesn't always do what you'd expect. I prefer to explicitly test the exit status of the commands in my script, with something like:
command1 || {
echo 'command1 failed!' >&2
exit 1
}
if command2; then
echo 'command2 succeeded!' >&2
else
echo 'command2 failed!' >&2
exit 1
fi
You're creating temp files in the current directory, which risks random conflicts (with other runs of the script at the same time, any files that happen to have names you're using, etc). It's better to create a temp directory at the beginning, then store everything in it (again, by absolute path):
module_tmp="$(mktemp -dt module-system)" || {
echo "Error creating temp directory" >&2
exit 1
}
...
echo "$answer" >> "$module_tmp/arglist.tmp"
(BTW, note that I'm using $() instead of backticks. They're easier to read, and don't have some subtle syntactic oddities that backticks have. I recommend switching.)
Speaking of which, you're overusing temp files; a lot of what you're doing with can be done just fine with shell variables and built-in shell features. For example, rather than reading line from the config file, then storing them in a temp file and using cut to split them into fields, you can simply echo to cut:
arg="$(echo "$line" | cut -d ";" -f 1)"
...or better yet, use read's built-in ability to split fields based on whatever IFS is set to:
while IFS=";" read -u3 arg requ description; do
(Note that since the assignment to IFS is a prefix to the read command, it only affects that one command; changing IFS globally can have weird effects, and should be avoided whenever possible.)
Similarly, storing the argument list in a file, converting newlines to spaces into another file, then reading that file... you can skip any or all of these steps. If you're using bash, store the arg list in an array:
arglist=()
while ...
arglist+=("$answer") # or ("#arg=$answer")? Not sure of your syntax.
done ...
"$module_root/modules/$name/$interpreter" "$file" "${arglist[#]}"
(That messy syntax, with the double-quotes, curly braces, square brackets, and at-sign, is the generally correct way to expand an array in bash).
If you can't count on bash extensions like arrays, you can at least do it the old messy way with a plain variable:
arglist=""
while ...
arglist="$arglist $answer" # or "$arglist $arg=$answer"? Not sure of your syntax.
done ...
"$module_root/modules/$name/$interpreter" "$file" $arglist
... but this runs the risk of arguments being word-split and/or expanded to lists of files.
I need to "Write a script to add users to your system in which the names of the users are given to script as an argument n (in?, spell error by professor) a text file of the format
Last_Name First_Name Middle_Initial Group
(rest of instructions FYI) Your script should create unique user names of up to 8 characters, and generate
random passwords for the users. The users should be assigned home directories
depending on the group they are in. You can assume that the users will belong to
Management (“mgmt”), Employee (“empl”) or Temporary (“temp”) and that their
respective directories are under these group names in /home. For e.g., if John Doe is
in “mgmt.”, and his user name is jdoe1234, then his home directory should be in
/home/mgmt/jdoe1234. Lock the home directories such that no one else has
permissions to the home directories than the users themselves.
Your script should generate a file called users.txt that contains the following columns:
Last Name First Name UID Password
which can be used to distribute the user names and passwords to the users.
The first part (not in italics) is what confuses me. If I understand his wording correctly, he wants me to take text from a separate .txt file and use it as the arguments for my script? With the idea of
./file.sh arg1 arg2 arg3 arg4
except those args are going to be the first four words in the .txt file? Say, if the .txt file contains "Doe John A empl", then it would be like typing
./file.sh arg1 arg2 arg3 arg4
Here's my attempt so far (I've actually tried other things but this is what I have on screen right now, sort of what I started out with):
#!/bin/bash
echo "$(cat file.txt)"
lname=$(echo $1|head -c 3)//to make username use first 3 letters of last name
fname=$(echo $2|head -c 1)//to make username use first letter of first name
mname=$3
group=$4
echo "$lname$fname$mname$group"
As of right now, anything below the first line doesn't do anything. I got the first line from command line arguments from a file content and I used it because I thought it would let me use the content from a .txt file as arguments but it's clearly not doing that. It's just outputting the content of it, not using letting me using each word as an argument. Not sure what to do. Any help? I thought this wasn't going to be very difficult as I started writing the script assuming the user would provide the arguments but I reread the first part of the instructions and assume he wants them to be taken from a separate .txt file.
$(command) returns the output of the command. If you do $(cat some_file) it will return the text of the file. You can use it to give the content of a file as an argument doing:
cmd1 $(cat args_file)
So when you use echo $(cat file.txt), you get the same output as if you were using cat file.txt because cat sends the content of the file to echo which displays it to the standard output.
$n means argument n passed to the script ($0 being the name of the script). Here you simply have to provide one argument, the name of the file. So $2, $3 and $4 will not contain anything.
So, from the file you can only get a string with the names with $names=$(cat $1). In order to get each field separately, you can use cut:
lname=$(cut -d \ -f 1 $1)
fname=$(cut -d \ -f 2 $1)
mname=$(cut -d \ -f 3 $1)
group=$(cut -d \ -f 4 $1)
NOTES:
The symbol for comments in shell is # NOT //.
head displays the first lines of a file, head -c the first bytes. It does not cut the file.
What I understood as the problem is that:
Firstly: You want to pass the contents of a text file as an input/argument to a shell script.
There could be other ways, but I suggest the following:
./YourScriptFile.sh $(cat YourInputFile.txt)
Secondly: You want to use its contents.
I would suggest to use the following inside your script file:
$1, $3, $4, ..., $n
to access the:
1st, 2nd, 3rd, ..., nth
tokens from the input file (irrespective of new lines).
Finally: You want to make username use first 3 letters of last name or in other words you want to extract characters from a string.
Once you have tokens it's just simple. I would suggest the following:
FirstTwoChars_of_FirstToken=${1:0:2}
FirstTwoChars_of_SeventhToken=${7:0:2}
Or
LastTwoChars_of_FirstToken=${1:7:9}
# if the first token is "FirstName" it would return you "me"
Hope this would help you to improve your code.
As a footnote: Your code will become: (# is used for comments here)
#!/bin/bash
lname=$(1:0:3) #to make username use first 3 letters of last name
fname=$(2:0:1) #to make username use first letter of first name
mname=$3
group=$4
echo "$lname$fname$mname$group"
Now you will have to execute your shell script as mentioned above.
I'm trying to extract a substring from an Oracle error message so I can email it off to an administrator using awk, this part of the code is trying to find where the important bit I want to extract.
starts here's what I have....
(The table name is incorrect to generate the error)
validate_iwpcount(){
DB_RETURN_VALUE=`sqlplus -s $DB_CRED <<END
SELECT count(COLUMN)
FROM INCORRECT_TABLE NAME;
exit
END`
a="$DB_RETURN_VALUE"
b="ERROR at line"
awk -v a="$a" -v b="$b" 'BEGIN{print index(a,b)}'
echo $DB_RETURN_VALUE
}
Strange thing is no matter how big that $DB_RETURN_VALUE is the return value from awk is always 28. Im assuming that somewhere in this error message there's something linux either thinks is a implcit delimiter of somesort and its messing with the count or something stranger. This works fine with regular strings as opposed to what oracle gives me.
Could anybody shine a light on this?
Many thanks
28 seems to be the right answer for the query you have (slightly amended to avoid an ORA-00936, and with tabs in the script). The message you're echoing includes a file expansion; the raw message is:
FROM IW_PRODUCTzS
*
ERROR at line 2:
ORA-00942: table or view does not exist
The * is expanded when you echo $DB_RETURN_VALUE, so the directory you're executing this from seem to have logs mail_files scripts in it, and they are being shown through expansion of the *. If you run it from different directories the echoed message length will vary, but the length of the actual message from Oracle stays the same - the length is changing (through the expansion) after the SQL*Plus call and after awk has done its thing. You can avoid that expansion with echo "$DB_RETURN_VALUE" instead, though I don't suppose you actually want to see that full message anyway in the end.
The substring from character 28 gives you what you want though:
validate_iwpcount(){
DB_RETURN_VALUE=`sqlplus -s $CENSYS_ORACLE_UID <<END
SELECT count(COLUMN_NAME)
FROM IW_PRODUCTzS;
exit
END`
# To see the original message; note the double-quotes
# echo "$DB_RETURN_VALUE"
a="$DB_RETURN_VALUE"
b="ERROR at line"
p=`awk -v a="$a" -v b="$b" 'BEGIN{print index(a,b)}'`
if [ ${p} -gt 0 ]; then
awk -v a="$a" -v p="$p" 'BEGIN{print substr(a,p)}'
fi
}
validate_iwpcount
... displays just:
ERROR at line 2:
ORA-00942: table or view does not exist
I'm sure that can be simplified, maybe into a single awk call, but I'm not that familiar with it.