I just have written a simple logger which append a message with time to a file. Now I also want to add error out to that log file for better understanding what went wrong. Here is my current code:
#!/bin/bash
logprint() {
echo "$(date +%T): $*" >> ./logfile
}
logdate() {
DATE=`date "+%d.%m.%Y"`
echo "-------------------- ${DATE} --------------------" >> ./logfile
}
The log print function takes arguments and simply write the date plus the message to the log file. The log date function simply writes the date at the beginning.
Now I would like to give the error output to the log print function. Whats the best way to do that?
You can use process substitution technique of form > >(cmd) for this. This allows you to re-direct the output from the standard error stream to the function. You can do something like
2> >(logprint)
But you can't read from the output of this process-substitution as if you were reading from the positional arguments, you need to read as if you were reading over standard input. You can tweak your function to something like below. Added a script for demonstration purposes
#!/usr/bin/env bash
logprint() {
args=""
if (( "$#" > 0 )); then
args="$*"
else
while IFS= read -r line; do
args+="$line"
done
fi
echo "$(date +%T): $args" >> ./logfile
}
logprint "foobar"
mv foobar nosuchfile 2> >(logprint)
If you need the timestamp to be in the same line, the simplest way I can think of is partially filling the line with timestamp (without ending with a newline) and then redirect the error output.
echo -n "$(date) " >> error.txt
ls no_such_file_here 2>> error.txt # an error message is generated here
echo "" >> error.txt # add a newline (useful in case an error message is not produced above)
Note that the last echo is to add a new line character to the current line as it guarantees anything appended later will not be added to the same line.
(However, that can result in an empty line if an error is generated in the line above.)
Update:
I was assuming that you are referring to the STDERR stream. However if that is not the case, the same idea can be used.
Related
I am currently making a bash script. The purpose of this script is not important. However, I have a piece of code that is generating an error. The error is as follows:
./script.bs: line 175: read: read error: 0: Key has expired
./script.bs: error reading input file: Key has expired
I have the code below for lines 175-189.
This specific piece of code does the following:
-Reads a txt file, that has a list of targeted files.
-For each targeted file, each line is read. And if that line is contained in $NumbersFile, it will do nothing. If that line is NOT contained in $NumbersFile, it will add that line to NumbersFile.
This general piece of code is working, and added 65810 lines of content to $NumbersFile. It then however got the error I stated above.
I'd like to add that the while loop on line 175 (where the error is happening) is supposed to read about 70'000 lines from the given file.
How do I fix this error so that my script may finish running without a key expired error?
NumbersFile="numbers.txt";
while read line; do
while read gramline; do
has="0";
if grep -Fq -- "$gramline" "$NumbersFile"; then
has="1";
fi
if [ "$has" -eq "0" ]; then
echo "$gramline" >> $NumbersFile;
fi
done < "$line";
done < "targetsfile.txt";
If my comment is accurate, perhaps this might be faster:
{ cat targetsfile.txt; xargs cat < targetsfile.txt; } | sort -u > numbers.txt
Or as clarified:
xargs cat < targetsfile.txt | sort -u > numbers.txt
Notes:
the braces are simply to group the cat and xargs commands so that the combined output can be piped into sort. Documented in the manual at 3.2.4.3 Grouping Commands
The first cat outputs the contents of the "targetsfile.txt" file
the xargs cat < targetsfile.txt construct will execute the cat command for every file listed in the targets file. It's a very concise and efficient way to execute
while IFS= read -r line; do cat "$line"; done < targetsfile.txt
I currently have a Perl script (that I can't edit) that asks the user a few questions, and then prints a generated output to stdout. I want to make another script that calls this script, allows the user to interact with it as normal, and then stores the output from stdout to a variable.
Here's a really simple example of what I'm trying to accomplish:
inner.pl
#!/usr/bin/perl
print "Enter a number:";
$reply = <>;
print "$reply";
outer.sh (based on the answer by Op De Cirkel here)
#!/bin/bash
echo "Calling inner.pl"
exec 5>&1
OUTPUT=$(./inner.pl | tee >(cat - >&5))
echo "Retrieved output: $OUTPUT"
Desired output:
$ ./outer.sh
Calling inner.pl
Enter a number: 7
7
Retrieved output: 7
However, when I try this, the script will output Calling inner.pl before "hanging" without printing anything from inner.sh.
I've found a bit of a workaround by using the script command to store the entire inner.sh exchange to a temporary file, and then using sed and the like to modify it to my needs. But making temporary files for something fairly trivial like that doesn't make a ton of sense (not to mention script likes to add time stamps and \rs to everything). Is there any non-script way to accomplish this?
The answer is simpler than that. Simply redirect inner's output to a variable with $():
#!/bin/bash
echo "Calling inner.sh"
OUTPUT=$(./inner.sh)
echo "Retrieved output: $OUTPUT"
EDIT:
Now, if there's user interaction with the output in inner.sh (example inner.sh asks user for a number, prints any operation with it and asks the user to input a new value based on that printed result). Then the better is a temporary file like this:
#!/bin/bash
echo "Calling inner.sh"
TMPFILE=`mktemp`
./inner.sh | tee "$TMPFILE"
OUTPUT=$(cat "$TMPFILE")
rm "$TMPFILE"
echo "Retrieved output: $OUTPUT"
What "... | tee file" does is take stdin (standard input) and divert it to two places: stdout (standard output) and to a path/file named "file". In effect it does this, as far as I can judge:
#!/bin/bash
var=(cat) # same as var=(cat /dev/stdin)
echo -e "$var"
for file in "$#"
do
echo -e "$var" > "${file}"
done
exit 0
So I use the above code to create tee1 to see if I could emulate what tee does. But my real intent is to write a modified version that appends to existing file(s) rather than redo them from scratch. I call this one tee2:
#!/bin/bash
var=(cat) # same as var=(cat /dev/stdin)
echo -e "$var"
for file in "$#"
do
echo -e "$var" >> "${file}"
done
exit 0
It makes sense to me, but not to bash. Now an alternative approach is to do something like this:
echo -e "$var"
for file in "$#"
do
echo -e "$var"| tee tmpfile
cat tmpfile >> "${file}"
done
rm tmpfile
exit 0
It also makes sense to me to do this:
#!/bin/bash
cp -rfp /dev/stdin tmpfile
cat tmpfile
for file in "$#"
do
cat tmpfile >> "${file}"
done
exit 0
Or this:
#!/bin/bash
cat /dev/stdin
for file in "$#"
do
cat /dev/stdin >> "${file}"
done
exit 0
Some online searches suggest that printf be used in place of echo -e for more consistency across platforms. Other suggest that cat be used in place of read, though since stdin is a device, it should be able to be used in place of catm as in:
> tmpfile
IFS=\n
while read line
do
echo $line >> tmpfile
echo $line
done < /dev/stdin
unset IFS
Then the for loop follows. But I can't get that to work. How can I do it with bash?
But my real intent is to write a modified version that appends to existing file(s) rather than redo them from scratch.
The tee utility is specified to support an -a option, meaning "Append the output to the files." [spec]
(And I'm not aware of any implementations of tee that deviate from the spec in this regard.)
Edited to add: If your question is really "what's wrong with all the different things I tried", then, that's probably too broad for a single Stack Overflow question. But here's a short list:
var=(cat) means "Set the array variable var to contain a single element, namely, the string cat."
Note that this does not, in any way, involve the program cat.
You probably meant var=$(cat), which means "Run the command cat, capturing its standard output. Discard any null bytes, and discard any trailing sequence of newlines. Save the result in the regular variable var."
Note that even this version is not useful for faithfully implementing tee, since tee does not discard null bytes and trailing newlines. Also, tee forwards input as it becomes available, whereas var=$(cat) has to wait until input has completed. (This is a problem if standard input is coming from the terminal — in which case the user would expect to see their input echoed back — or from a program that might be trying to communicate with the user — in which case you'd get a deadlock.)
echo -e "$var" makes a point of processing escape sequences like \t. (That's what the -e means.) This is not what you want. In addition, it appends an extra newline, which isn't what you want if you've managed to set $var correctly. (If you haven't managed to set $var correctly, then this might help compensate for that, but it won't really fix the problem.)
To faithfully print the contents of var, you should write printf %s "$var".
I don't understand why you switched to the | tee tmpfile approach. It doesn't improve anything so far as I can tell, and it introduces the bug that now if you're copying to n files, then you will also write n copies to standard output. (You fixed that bug in later versions, though.)
The versions where you write directly to a file, instead of saving to a variable first, are a massive improvement in terms of faithfully copying the contents of standard input. But they still have the problem of waiting until input is complete.
The version where you cat /dev/stdin multiple times (once for each destination) won't work, because there's no "rewinding" of standard input. Once something is consumed, it's gone. (This makes sense when you consider that standard input is frequently passed around from program to program — your cat-s, for example, are inheriting it from your Bash script, and your Bash script may be inheriting it from the terminal. If some sort of automatic rewinding were to happen, how would it decide how far back to go?) (Note: if standard input is coming from a regular file, then it's possible to explicitly seek backward along it, and thereby "unconsume" already-consumed input. But that doesn't happen automatically, and anyway that's not possible when standard input is coming from a terminal, from a pipe, etc.)
I want to run a program (when executed it produces logdata) out of a shell script and write the output into a text file. I failed to do so :/
$prog is the executed prog -> socat /dev/ttyUSB0,b9600 STDOUT
$log/$FILE is just path to a .txt file
I had a Perl script to do this:
open (S,$prog) ||die "Cannot open $prog ($!)\n";
open (R,">>","$log") ||die "Cannot open logfile $log!\n";
while (<S>) {
my $date = localtime->strftime('%d.%m.%Y;%H:%M:%S;');
print "$date$_";
}
I tried to do this in a shell script like this
#!/bin/sh
FILE=/var/log/mylogfile.log
SOCAT=/usr/bin/socat
DEV=/dev/ttyUSB0
BAUD=,b9600
PROG=$SOCAT $DEV$BAUD STDOUT
exec 3<&0
exec 0<$PROG
while read -r line
do
DATE=`date +%d.%m.%Y;%H:%M:%S;`
echo $DATE$line >> $FILE
done
exec 0<&3
Doesn't work at all...
How do I read the output of that prog and pipe it into my text file using a shell script? What did I do wrong (if I didn't do everything wrong)?
Final code:
#!/bin/sh
FILE=/var/log/mylogfile.log
SOCAT=/usr/bin/socat
DEV=/dev/ttyUSB0
BAUD=,b9600
CMD="$SOCAT $DEV$BAUD STDOUT"
$CMD |
while read -r line
do
echo "$(date +'%d.%m.%Y;%H:%M:%S;')$line" >> $FILE
done
To read from a process, use process substitution
exec 0< <( $PROG )
/bin/sh doesn't support it, so use /bin/bash instead.
To assign several words to a variable, quote or backslash whitespace:
PROG="$SOCAT $DEV$BAUD STDOUT"
Semicolon is special in shell, quote it or backslash it:
DATE=$(date '+%d.%m.%Y;%H:%M:%S;')
Moreover, no exec's are needed:
while ...
...
done < <( $PROG )
You might even add > $FILE after done instead of adding each line separately to the file.
Original answer
You haven't shown the error messages — which would have been helpful.
Your problem, though, is probably this line:
DATE=`date +%d.%m.%Y;%H:%M:%S;`
where the semicolons mark the end of a command, and there likely isn't a command %H that does anything useful, etc.
You need quotes around the format argument to date, and I'd use single quotes for this job:
DATE=$(date +'%d.%m.%Y;%H:%M:%S;')
or even replace the two lines in the body of the loop with:
echo "$(date +'%d.%m.%Y;%H:%M:%S;')$line" >> $FILE
The double quotes prevent a variety of problems.
That assumes you fix a bunch of other problems, such as the setting of the variables FILE and prog. Also, I'd probably use:
exec > $FILE
to initially zap the output file and then all subsequent standard output would go to that file, so the echo line becomes:
echo "$(date +'%d.%m.%Y;%H:%M:%S;')$line"
Amended answer
The question was originally missing lots of key information. It eventually got updated to include the complete code.
The problem I identified originally remains an issue, but you weren't running into it because the input redirection was not working. If you want the input to come from a process, use a pipe, or possibly process substitution. However, note that you have #!/bin/sh as your shebang line, and /bin/sh won't recognized process substitution; either change the shebang or use the pipe notation. Note that process substitution has advantages if the loop is setting variables that need to be accessed after the loop is complete.
$SOCAT $DEV$BAUD STDOUT |
while read -r line
do
…
done
or
while read -r line
do
…
done < <($SOCAT $DEV$BAUD STDOUT)
Note that your code contains the line:
PROG=$SOCAT $DEV$BAUD STDOUT
This runs the command identified by $DEV$BAUD with the argument STDOUT and the environment variable PROG set to the value of $SOCAT. That is not what you wanted.
You could use an array:
PROG=($SOCAT $DEV$BAUD STDOUT)
and then run:
"${PROG[#]}"
either in the pipe line:
"${PROG[#]}" |
while read -r line
do
…
done
or with process substitution:
while read -r line
do
…
done < <("${PROG[#]}")
Note that unless there is code after the final exec 0<&3, there was no particular virtue in the redirections involving file descriptor 3. You should also close 3 when you're done with it:
exec 0<&3 3>&-
The 'final' code includes the lines:
CMD="$SOCAT $DEV$BAUD STDOUT"
$CMD |
while read -r line
This works OK because there are no spaces in the arguments to the command. That's a common case, but beware of spaces in arguments and file paths.
I had a bash file which prompted the user for some parameters and used defaults if nothing was given. The script then went on to perform some other commands with the parameters.
This worked great - no problems until most recent addition.
In an attempt to read the NAMES parameter from a txt file, I've added a while-loop to take in the names in the file, but I would still like the remaining parameters prompted for.
But once I added the while loop, the output shows the printed prompt in get_ans() and never pauses for a read, thus all the defaults are selected.
I would like to read the first parameter from a file, then all subsequent files from prompting the user.
What did I break by adding the while-loop?
cat list.txt |
while read line
do
get_ans "Name" "$line"
read NAME < $tmp_file
get_ans "Name" "$line"
read NAME < $tmp_file
done
function get_ans
{
if [ -f $tmp_file ]; then
rm $tmp_file
PROMPT=$1
DEFAULT=$2
echo -n "$PROMPT [$DEFAULT]: "
read ans
if [ -z "$ans" ]; then
ans="$DEFAULT"
fi
echo "$ans" > $tmp_file
}
(NOTE: Code is not copy&paste so please excuse typos. Actual code has function defined before the main())
You pipe data into your the while loops STDIN. So the read in get_ans is also taking data from that STDIN stream.
You can pipe data into while on a different file descriptor to avoid the issue and stop bothering with temp files:
while read -u 9 line; do
NAME=$(get_ans Name "$line")
done 9< list.txt
get_ans() {
local PROMPT=$1 DEFAULT=$2 ans
read -p "$PROMPT [$DEFAULT]: " ans
echo "${ans:-$DEFAULT}"
}
To read directly from the terminal, not from stdin (assuming you're on a *NIX machine, not a Windows machine):
while read foo</some/file; do
read bar</dev/tty
echo "got <$bar>"
done
When you pipe one command into another on the command line, like:
$ foo | bar
The shell is going to set it up so that bar's standard input comes from foo's standard output. Anything that foo sends to stdout will go directly to bar's stdin.
In your case, this means that the only thing that your script can read from is the standard output of the cat command, which will contain the contents of your file.
Instead of using a pipe on the command line, make the filename be the first parameter of your script. Then open and read from the file inside your code and read from the user as normal.