References
Fullcode of what will be discussed here:
https://github.com/djon2003/com.cyberinternauts.linux.backup
activateLogs question that solved how to log to file and screen: https://stackoverflow.com/a/70792273/214898
Limitation
Just a small reminder from the last question: this script is executed on limited environment, on a QNAP (NAS).
Background
I have a function that activate logging which now has three modes: SCREEN, DISK, BOTH. With some help (from the question of the link above), I achieve to make work the BOTH option. DISK & BOTH use a file descriptor numbered 3. The first is pointing to a file and the second to stdout.
On exit of my script (using trap), it detects if there were logged errors and send them via email.
Code
function sendErrorMailOnExit()
{
## If errors happened, then send email
local isFileDescriptor3Exist=$(command 2>/dev/null >&3 && echo "Y")
if [ "$isFileDescriptor3Exist" = "Y" ]; then
local logFile=$(readlink /proc/self/fd/3 | sed s/.log$/.err/)
local logFileSize=$(stat -c %s "$logFile")
if [ $logFileSize -gt 0 ]; then
addLog "N" "Sending error email"
local logFileName=$(basename "$logFile")
local logFileContent=$(cat "$logFile")
sendMail "Y" "QNAP - Backup error" "Error happened on backup. See log file $logFileName\n\nLog error file content:\n$logFileContent"
fi
fi
}
trap sendErrorMailOnExit EXIT
Problem
As you can see, this works well because the file descriptor #3 is using a file. But now, using the BOTH option, the file descriptor #3 is pointing to stdout and the file is written via tee. Hence my question, how could I get the location of the file of tee.
Why not only using a variable coming from my function activateLogs would you say? Because, this function relaunches the script to be able to get all the logs not caught before the function is called. Thus why using this method to retrieve the error file location.
Possible solutions, but not the best (I hope)
One way would be to pass the file location through a script
parameter, but I would prefer not do that if that can be avoid.
Another, would be to create a "fake" file descriptor #4 (probably my best solution up to now) that would always point to the file.
Does anyone have an idea?
I finally opted for the creation of a "fake" file descriptor #4 that does not nothing except pointing to the current log file.
Related
Background
I've a script. It's purpose is to generate config files for various system services from templates whenever my gateway acquires a new IP from my ISP. This process includes making successive edits with sed to replace $[template] strings in my custom templates with the correct information.
And to do that I've created a small function designed to take input from stdin, redirect it to a temporary file passed as an argument, and then move that file to replace the destination (and also, often, source) config file. The "edit-in-place dance", if you will.
I created a simple test script with the problematic function:
#!/bin/bash
inplace_dance() {
read -r -d '' data
printf '%s' "${data}" > "${1}~"
mv "${1}~" "${1}"
}
# ATTN: ls is only being used to generate input for testing. It is not being parsed.
ls -l ~/ | inplace_dance ~/test.out
Unfortunately, this works. So it's not the function itself. I also tried it with my custom logging utility (see "complications" below):
#!/bin/bash
. /usr/local/lib/logging.bash
log_identifier='test'
log_console='on'
inplace_dance() {
read -r -d '' data
printf '%s' "${data}" > "${1}~"
mv "${1}~" "${1}"
}
# ATTN: ls is only being used to generate input for testing. It is not being parsed.
bashlog 'notice' $(ls -l ~/ | inplace_dance '/home/wolferz/test.out')
This also works.
The Problem
In its original context, however, it does not work. I've confirmed that ${data} gets set just fine. And that ${1} contains the correct filename. What fails is the second line of the function. I've confirmed printf is being run (see, "Additional Info - Without The Redirect" below)... but the file its output is being redirected to is never created.
And I've been over the code a dozen-dozen times (not an exaggeration) and have yet to identify the cause. So, in desperation, I'm going to post it here and hope some kind soul will wade through my code and maybe spot the problem that I'm missing. I'd also happily take advice on improvements/replacements to my logging utility (in the hopes of tracking down the problem) or further troubleshooting steps.
Here is the original context. The important lines are 106-110, 136-140, 144-147, and 151-155
Additional Info
☛ PATH/Environment
The PATH is PATH=/usr/local/sbin:/usr/local/bin:/usr/bin. I believe this is being inherited from systemd (systemd=>dhcpcd.service=>dhcpcd=>dhcpcd-run-hooks=>dhcpcd.exit-hook).
dhcpcd-run-hooks (see "Complications" below) does clear the environment (keeping the above PATH) when it runs. Thus, I've added an example of the environment the script runs in to the "original context" gist. In this case, the environment when $reason == 'BOUND'. This is output by printenv | sort at the end of execution (and thus should show the final state of the environment).
NOTE: Be aware this is Arch Linux and the absence of /bin, /sbin, and /usr/sbin in the PATH is normal (they are just symlinks to /usr/bin anyway).
☛ Return Code
Inserting echo $? after the second line of the function gives me a return code of "0". This is true both with the redirect in line 2 and without (just the printf).
☛ Without The Redirect
Without the redirect, in the original context, the second line of the function prints the contents of ${data} to stdout (which is then captured by bashlog()) exactly as expected.
⚠️ Execute Instead of Source.
Turns out that $0 was /usr/lib/dhcpcd/dhcpcd-run-hooks rather than my script. Apparently dhcpcd-run-hooks doesn't execute the script... it sources it. I made some changes to line 196 to fix this.
♔ Aaaaaand that seems to have fixed all problems. ♔
I'm trying to confirm that was the silver bullet now... I didn't notice it was working till I had made several other changes as well. If I can confirm it I'll submit an answer.
Complications
What complicates matters quite a bit is that it's original context is a /etc/dhcpcd.exit-hook script. dhcpcd-run-hooks appears to eat all stderr and stdout which makes troubleshooting... unpleasant. I've implemented my own logging utility to capture the output of commands in the script and pass it to journald but it's not helping in this case. Either no error is being generated or, somehow, the error is not getting captured by my logging utility. The script is running as root and there is no mandatory access control installed so it shouldn't be a permissions issue.
I am new to Shellscripting.I am working on a poc in which a script should read a log file and then append to a existing file for the purpose of alert.It should work as per below
There will be some predefined format according to which it will decide whether to append in file or not.For example:
WWXXX9999XS message
**XXX** - is a 3 letter acronym (application code) like for **tom** for tomcat application
9999 - is a 4 numeric digit in the range 1001-1999
**E or X** - For notification X ,If open/active alerts already existing for same error code and same message,new alerts will not be raised for existing one.Once you have closed existing alerts,it will raise alarm for new error.There is any change in message for same error code from existing one, it will raise a alarm even though open/active alerts present.
X option is only for drop duplicates on code and message otherwise all alert mechanisms are same.
**S** - is the severity level, I.e 2,3
**message** - is any text that will be displayed
The script will examine the log file, and look for error like cloud server is down,then it would append 'wwclo1002X2 cloud server is down'if its a new alert.
2.If the same alert is coming again,then it should append 'wwclo1002E2 cloud server is down
There are some very handy commands you can use to do this type of File manipulation. I've updated this in response to your comment to allow functionality that will check if the error has already been appended to the new file.
My suggestion would be that there is enough functionality here to warrant saving it in a bash script.
My approach would be to use a combination of less, grep and > to read and parse the file and then append to the new file. First save the following into a bash script (e.g. a file named script.sh)
#!/bin/bash
result=$(less $1 | grep $2)
exists=$(less $3 | grep $2)
if [[ "$exists" == "$result" ]]; then
echo "error, already present in file"
exit 1
else
echo $result >> $3
exit 0
fi
Then use this file in the command passing in the log file as the first argument, the string to search for as the second argument and the target results file as the third argument like this:
./script.sh <logFileName> "errorToSearchFor" <resultsTargetFileName>
Don't forget to run the file you will need to change the permissions - you can do this using:
chmod u+x script.sh
Just to clarify as you have mentioned you are new to scripting - the less command will output the entire file, the | command (an unnamed pipe) will pass this output to the grep command which will then search the file for the expression in quotes and return all lines from the file containing that expression. The output of the grep command is then appended to the new file with >>.
You may need to tailor the expression in quotes after grep to get exactly the output you want from the log file.
The filenames are just placeholders, be sure to update these with the correct file names. Hope this helps!
Note updated > to >> (single angle bracket overwrites, double angle bracket appends
I'm writing a script to test a program and I'm getting caught at this portion
if (("$x"==22)); then
echo "Checking for whether wrong input file is detected."
if diff ${arr[$x]} <(./compare ); then
echo Output is as expected.
else
echo Output is not as expected. Check for errors.
fi
else
if diff -q ${arr[$x]} <(./compare $i); then
echo Output is as expected.
else
echo Output is not as expected. Check for errors.
fi
fi
So what it's doing is testing my program against known output. However, for the case where I use ./compare without an argument, I want to receive an error message from my program specifying that the argument is missing. The test file it's using, "22" let's call it result22.txt, has the exact same output my program would give from just running ./compare (no arguments). However, when I run it using the script, it says that result22.txt differs from just running ./compare. I'm pretty sure I"m running the script wrong, any ideas?
Additionaly information, i is a known input test file that's from an array, x is an incremental variable to count which loop we're on. So arr[$x] is just accessing the nth file from the known output files.
Compare is my own compare program to run.
I'm trying to redirect(?) my standard error/output to a text file.
I did my research, but for some reason the online answers are not working for me.
What am I doing wrong?
cd /home/user1/lists/
for dir in $(ls)
do
(
echo | $dir > /root/user1/$dir" "log.txt
) > /root/Desktop/Logs/Update.log
done
I also tried
2> /root/Desktop/Logs/Update.log
1> /root/Desktop/Logs/Update.log
&> /root/Desktop/Logs/Update.log
None of these work for me :(
Help please!
Try this for the basics:
echo hello >> log.txt 2>&1
Could be read as: echo the word hello, redirecting and appending STDOUT to the file log.txt. STDERR (file descriptor 2) is redirected to wherever STDOUT is being pointed. Note that STDOUT is the default and thus there is no "1" in front of the ">>". Works on the current line only.
To redirect and append all output and error of all commands in a script, put this line near the top. It will be in effect for the length of the script instead of doing it on each line:
exec >>log.txt 2>&1
If you are trying to obtain a list of the files in /home/user1/lists, you do not need a loop at all:
ls /home/usr1/lists/ >Update.log
If you are attempting to run every file in the directory as an executable with a newline as its input, and collect the output from all these programs in Update.log, try this:
for file in /home/user1/lists/*; do
echo | "$file"
done >Update.log
(Notice how we avoid the useless use of ls and how there is no redirection inside the loop.)
If you want to create an empty file called *.log.txt for each file in the directory, you would do
for file in /home/user1/lists/*; do
touch "$(basename "$file")"log.txt
done
(Using basename to obtain the file name without the directory part avoids the cd but you could do it the other way around. Generally, we tend to avoid changing the directory in scripts, so that the tool can be run from anywhere and generate output in the current directory.)
If you want to create a file containing a single newline, regardless of whether it already exists or not,
for file in /home/user1/lists/*; do
echo >"$(basename "$file")"log.txt
done
In your original program, you redirect the echo inside the loop, which means that the redirection after done will not receive any output at all, so the created file will be empty.
These are somewhat wild guesses at what you might actually be trying to accomplish, but should hopefully help nudge you slightly in the right direction. (This should properly be a comment, I suppose, but it's way too long and complex.)
I dont know if it is weird that read is not taking the input from the terminal.
The configure script, which is used in source code making process, should ask the user to give the input to select the type of Database either MYSQL or ORACLE(below is the code).
MYSQLLIBPATH="/usr/lib/mysql"
echo "Enter DataBase-Type 1-ORACLE, 2-MySQL (default MySQL):"
read in
echo $? >> /tmp/error.log
if test -z "$in" -o "$in" = "2"
then
DATABASE=-DDB_MYSQL
if true; then
MYSQL_TRUE=
MYSQL_FALSE='#'
else
MYSQL_TRUE='#'
MYSQL_FALSE=
fi
echo "Enter Mysql Library Path: (eg: $MYSQLLIBPATH (default))"
read in
echo $? >> /tmp/error.log
if test -n "$in"
then
MYSQLLIBPATH=`echo $in`
fi
echo "Mysql Lib path is $MYSQLLIBPATH"
else
if false; then
MYSQL_TRUE=
MYSQL_FALSE='#'
else
MYSQL_TRUE='#'
MYSQL_FALSE=
fi
DATABASE=-DDB_ORACLE
LD_PATH=
fi
But, the read command is not asking for the user input. Its failing to take the input from the stdin.
When I checked the status of the command in the error.log it was showing
1
1
Could anyone tell why read is failing to take the input from the stdin.
Are there any builtin variable which can block read taking the input?
Most likely read executes with standard input redirected from a file that has reached EOF. If the above is not the whole of your configure code, check that there are no input redirections. Could the code above be a part of a function which was invoked with some input from a pipe or a file? Otherwise check how configure is executed - are there any redirections?
Otherwise, the universal advice applies: try simplifying and stripping down your code until it is obvious what's happening.
BTW, it is not a good idea to make configure interactive, if you want to have your program packaged for a distribution - it's not easy to control execution of interactive programs. Consider adding support for supplying parameters through command line options.