I write a crontab mission to make 3 POST request every 10 minutes by cURL, and here is pseudo:
#!/bin/sh
echo `date` >>/tmp/log
curl $a >>/tmp/log
curl $b >>/tmp/log
curl $c >>/tmp/log
That is all the code, but after the first echo to my /tmp/log, other output was saved in random file name like "A6E0U9~D", it doesn't happen all the time, I got no clues why.:(
PS. I don't use "$a", I use a raw string which copy from CHROME Dev Tool, and one of them is added below. And every single line's output is good, the only problem is some of the output was redirected to a random name file.
the cURL link is deleted because it contained my login cookie
Not really a solution, but you can redirect the output of everything at once, rather than repeatedly appending to the same file.
#!/bin/sh
{
date
curl ...
curl ...
curl ...
} > /tmp/log
The benefit here is that all the output will appear in the same file, whether that file is /tmp/log or an oddly named file. If you still end up with another file aside from /tmp/log, then you know there must be a problem with one of the curl calls.
(Note that capturing and re-printing the output of date is redundant.)
In order to run each curl in parallel, you'll need to save the output from each, and concatenate them once all have finished.
#!/bin/sh
{
date
tmp1=$(mktemp) && curl ... > "$tmp1" &
tmp2=$(mktemp) && curl ... > "$tmp2" &
tmp3=$(mktemp) && curl ... > "$tmp3" &
wait
cat "$tmp1" "$tmp2" "$tmp3"
} > /tmp/log
rm "$tmp1" "$tmp2" "$tmp3"
Related
i am writing a bash script to accomplish the following.
script runs wget every five minutes to download a small log from a static url.
script uses diff to see if there are any new entries made to the log file (new entries are made at the end of log file).
if new log entries are found - extract the new entries to a new file, format them properly, send me an alert, return to #1.
if no new log entries are found, go back to #1.
wget "https://url-to-logs.org" -O new_log
if diff -q new_log old_log; then
echo "no new log entries to send."
else
echo "new log entries found, sending alert."
diff -u new_log old_log > new_entries
#some logic i have to take the output of "new_entries", properly format the text and send the alert.
rm -rf old_log new_entries
cp new_log old_log
rm -rf new_log
fi
there is one additional thing - every night at midnight the server hosting the logs deletes all entries and displays a blank file until new log entries are made for the new day.
i guess i could always run a cron job at midnight to run "rm -rf" and "touch" the old_log file, but curious if an easier way to do this exists.
thanks in advance for any/all input and help.
If your logs are not rotating - i.e. the old log is guaranteed to be the prefix of the new log, you can just use tail to get the new suffix - something like this:
tail -n+$(( $(wc -l old_log) + 1 )) new_log > new_entries
If there are no new lines in new_log, the new_entries file will be empty, which you can check using stat or some other way.
If your logs are rotating, you should first use grep to check if the last line from the old log exists in the new log, and if not - assume the entire new log is new:
if ! egrep -q "^$(tail -n1 old_log)\$" new_log; then cat new_log > new_entries; fi
Hi expertsI want commands out and err is appended to one file, like this command > logOutErr.txt 2>&1
but I also want that err is also appended to another command 2> logErrOnly.txt
# This is a non working redirection
exec 1>> logOutErr.txt 2>> logOutErr.txt 2>> logErrOnly.txt
# This should be in Out log only
echo ten/two: $((10/2))
# This should be in both Out and Out+Err log files
echo ten/zero: $((10/0))
I understand than the last redirect 2>> overrides the preceding ...so what? tee? but how?
I have to do this once at the beginning of the script, without modifying the rest of the script (because it is dynamically generated and any modification is too complicated)
Please don't answer only with links to the theory, I have already spent two days reading everything with no good results, I would like a working example
Thanks
With the understanding that you lose ordering guarantees when doing this:
#!/usr/bin/env bash
exec >>logOutErr.txt 2> >(tee -a logErrOnly.txt)
# This should be in OutErr
echo "ten/two: $((10/2))"
# This should be in Err and OutErr
echo "ten/zero: $((10/0))"
This works because redirections are processed left-to-right: When tee is started, its stdout is already pointed to logOutErr.txt, so it appends to that location after first writing to logErrOnly.txt.
i run the following function in (git-)bash under windows:
function config_get_container_values() {
local project_name=$1
local container_name=$2
#local container_name="gitea"
echo "###"
buildcmd="jq -r \".containers[]."
echo "$buildcmd"
buildcmd="${buildcmd}${container_name}"
echo "$buildcmd"
buildcmd="${buildcmd}foobar"
echo "$buildcmd"
echo "###"
}
The output of this is the following. Whyever, after using the variable to extend the string, he starts to overwrite $buildcmd. I tried this also with everything in one line as well with the append command (=+). Everytime the same result.
###
jq -r ".containers[].
jq -r ".containers[].gitea
foobar".containers[].gitea
###
The really strange thing is: When i enable the line local container_name="gitea" everything works as expected. The output is:
###
jq -r ".containers[].
jq -r ".containers[].gitea
jq -r ".containers[].giteafoobar
###
When i put this all into a news file, its also works as expected. So i think something goes wrong in the thousands of line before calling this function. Any idea, what could be cause of this behavior?
Regards
Dave
This is not how you should build up the command, DOS line endings aside. Use --arg to pass the name into the filter as a variable. For example,
config_get_container_values() {
local project_name=$1
local container_name=$2
jq -r --arg n "$container_name " '.containers[][$n+"foobar"]'
}
config_get_container foo gitea < some.json
If the function is invoked with
config_get_container_values proj gitea
it produces the "expected" output. If it is invoked with
config_get_container_values proj $'gitea\r'
it produces output that looks like the first output example. $'gitea\r' expands to a string that consists of 'gitea' followed by a Carriage return (CR) character.
One possible cause of the problem is that the container name (gitea) was read from a file that had Windows/DOS line endings (CR-LF). Problems like that are common. See the first question ("Check whether your script or data has DOS style end-of-line characters") in the "Before asking about problematic code" section of the Stack Overflow 'bash' Info page.
I want to analysis some data in one linux server,then send the it as Email text to my Email account , But when i execute this shell scripts in shell command, It works well, Weird is that when i put all the procedure into crontab job, The Email text will turns to an attached file, Can someone help?
#* * * * * sh -x /opt/bin/exec.sh >> /opt/bin/mailerror 2>&1
/* exec.sh */
#/bin/sh
cd /opt/bin
./analysis.sh > test
mail -s "Today's Weather" example#example.com < test
But when i execute exec.sh in shell command line directly, The Email will get text, Can someone explain it for me, grate thanks.
Ran into the same problem myself, only I'm piping text output into mailx - Heirloom mailx 12.4 7/29/08
When running the script on the command line the email came out as normal email with a text body.
However, when I ran the exact same script via crontab the body of the email came as an attachment - ATT00001.BIN (Outlook), application/octet-stream (mutt) or "noname" (Gmail).
Took some research to figure this out, but here goes:
Problem
Mailx will, if it encounters unknown / control characters in text input, convert it into an attachment with application/octet-stream mime-type set.
From the man page:
for any file that contains formatting characters other than newlines and horizontal tabulators
So you need to remove those control characters, which can be done with i.e. tr
echo "$Output" | /usr/bin/tr -cd '\11\12\15\40-\176' | mail ...
However since I had Norwegian UTF8 characters: æøå - the list expand, and you don't really want to maintain such a list, and I need the norwegian characters.
And inspecting the attachment I found I had only \r, \n the "regular" ASCII characters in range 32-176 - all printable and 184 and 195 --> UTF8
Sollution
Explicitly set the locale in your script:
LANG="en_US.UTF8" ; export LANG
Run export in your shell - or setenv if you run csh or tcsh to determine what your locale is set to.
Explanation
Mailx - when run in your shell - with LANG set to .UTF8, will correctly identify the UTF8 chars and continue.
When run in crontab LANG is not set, and default to LANG=C, since by default crontab will run only a restricted set of environment variables (system dependant).
mailx (or other programs) will then not recognize UTF8 characters and determine that the input containes unknown control characters.
My issue was UTF8 characters, yours could be other control characters in your input. Run it through hexdump or od -c, but since it works OK in a regular shell I'm suspecting LANG issues.
References:
linux mail < file.log has Content-Type: application/octet-stream (a noname attachment in Gmail)
http://alvinalexander.com/blog/post/linux-unix/how-remove-non-printable-ascii-characters-file-unix
I had this same issue and none of the above fixed the problem. Moving the extra return in the file fixed the issue for me:
cat logfile | tr -d \\r | mailx -s'the logfile' to-me#.....
Thanks to this forum:
https://forums.opensuse.org/showthread.php/445955-mailx-creates-unwanted-attachment
Make sure you change this in your script
#/bin/sh
to be replaced by
#!/bin/sh
Coming to the problem
Your script assumes that it is being run from a particular directory (note that almost every path is a relative path, not an absolute path). cron happens to be running it from another directory.
The Fix for text appearing on email
mydir=$(dirname "$0") && cd "${mydir}" || exit 1
./opt/bin/analysis.sh > test
mail -s "Today's Weather" example#example.com < /opt/bin/test
Explanation
$0 is the (possibly relative) filename of the shell script being executed. Given a filename, the dirname command returns the directory containing the filename.
So, that line changes directories to the directory containing the script or exits with an error code if either dirname or cd fails.
OR try to have full path like
./opt/bin/analysis.sh > test
mail -s "Today's Weather" example#example.com < /opt/bin/test
Note: The same problem is discussed earlier here
FOLLOW UP:
Try to remove
sh -x /opt/bin/exec.sh >> /opt/bin/mailerror 2>&1
and instead use
sh /opt/bin/exec.sh 2>&1 >> /opt/bin/mailerror
FOLLOW UP
You have to restart cron for changes to take effect if you do not use the crontab command to edit the file.
crontab -l > oldcrontab
cp oldcrontab newcrontab
echo "$newline" >> newcrontab
crontab < newcrontab
In my case, the cron was not a shell script but a PHP script (so I couldn't put the export LANG thing):
0 9 * * * apache php /test/myscript.php | mail -s "CRON - myscript" foo#bar.com
Solution:
In order to fix the same issue (content is mailed as attachment instead of body), I add LANG=fr_FR.UTF-8 at the beginning of the cron file:
MAILTO=vme1.etc-crond-backoffice-conf
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
LANG=fr_FR.UTF-8
0 9 * * * apache php /test/myscript.php | mail -s "CRON - myscript" foo#bar.com
NB: puting LANG=fr_FR.UTF-8 in the /etc/environment file and restarting cron service worked too.
Reference:
Set LANG in crontab https://www.logikdev.com/2010/02/02/locale-settings-for-your-cron-job/
I'm creating a script to download and mirror a site, URLs are taken from a .txt file. The script is supposed to run daily for a few hours, so I need to get it to continue mirroring where it left off.
Here is the script:
# Created by Salik Sadruddin Merani
# email: ssm14293#gmail.com
# site: http://www.dragotech-innovations.tk
clear
echo ' Created by: Salik Sadruddin Merani'
echo ' email: ssm14293#gmail.com'
echo ' site: http://www.dragotech-innovations.tk'
echo
echo ' Info:'
echo ' This script will use the URLs provided in the File "urls.txt"'
echo ' Info: Logs will be saved in logfile.txt'
echo ' URLs are taken from the urls.txt file'
#
url=`< ./urls.txt`
useragent='Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0'
echo ' Mozilla Firefox User agent will be used'
cred='log=abc#123.org&pwd=abc123&wp-submit=Log In&redirect_to=http://abc#123.org/wp-admin/&testcookie=1'
echo ' Loaded Credentails'
echo ' Logging In'
wget --save-cookies cookies.txt --post-data ${cred} --keep-session-cookies http://members.ebenpagan.com/wp-login.php --delete-after
OIFS=$IFS
IFS=','
arr2=$url
for x in $arr2
do
echo ' Loading Cookies'
wget --spider --load-cookies cookies.txt --keep-session-cookies --mirror --convert-links --page-requisites ${x} -U ${useragent} -np --adjust-extension --continue -e robots=no --span-hosts --no-parent -o log-file-$x.txt
done
IFS=$OIFS
Problems with the script:
The script is not referencing its links correctly by making it referable to the file in the parent directory, please tell me about that.
The script is not resuming after being aborted even with the --continue option.
A smarter way to solve the problem is, work with two .txt files, let's affectionately call them "to_mirror.txt" and "mirrored.txt". Keep each URL on a single line. Declare in your script a variable of value 0, for example total_mirrored=0, it will be very important in our code. Therefore, every time the wget command is executed and, consequently, the site is mirrored, increment the value of the "total_mirrored" variable by +1.
Upon exiting the loop, "total_mirrored" will have any integer value.
Then you must extract the lines from "to_mirror.txt" in the range: first line up to "total_mirrored"; then attach this to "mirrored.txt".
After that delete the range from the file "to_mirror.txt".
In this case the sed command can help you, see my example:
sed -n "1,$total_mirrored p" to_mirror.txt >> mirrored.txt && sed -i "1,$total_mirrored d" to_mirror.txt
You can learn a lot about the sed command by running man sed in your terminal, so I won't explain here what each option does as it's redundant.
But know that:
>> appends the existing file, or creates a file if the file of the mentioned name is not present in the directory. && A && B — run B only if A succeeded.
The --continue flag in wget will attempt to resume the downloading of a single file in the current directory. Please refer to the man page of wget for more info. It is quite detailed.
What you need is resuming the mirroring/downloading from where the script previously left off.
So, its more of a modification of script than some setting in wget. I can suggest a way to do that, but mind you, you can use a different approach as well.
Modify the urls.txt file to have one URL per line. Then refer this pseudocode:
get the URL from the file
if (URL ends with a token #DONE), continue
else, wget command
append a token #DONE to the end of the URL in the file
This way, you will know which URL to continue from, the next time you run the script. All URLs that have a "#DONE" at the end will be skipped, and the rest will be downloaded.