I am executing a PHP script via the command line which, for a specific user, runs fine when executed on the commandline, but when the exact same command is put into the same user's crontab, a PHP iconv error is returned.
The commandline is utilising the Yii framework and the Zend Lucene library, but I'm not sure if that's pertinent.
I've made all executable and script paths absolute in the crontab line and can verify that it works when executed directly on the commandline.
I wrapped the actual PHP invocation in a one-line shell script, as I read elsewhere here that this solved a similar problem for someone, but no joy.
The command successfully executed on the commandline is:
/bin/sh /var/www/yii-projects/projectname/protected/scripts/buildIndex.sh >> /var/lucene/lucene.log
The content of the buildIndex.sh script is:
/usr/bin/php /var/www/yii-projects/projectname/protected/scripts/cron.php lucene buildIndex
And the crontab line is:
*/10 * * * * /bin/sh /var/www/yii-projects/projectname/protected/scripts/buildIndex.sh >> /var/lucene/lucene.log
The error shown in the log when the crontab executes is:
PHP Error[8]: iconv(): Detected an illegal character in input string
in file /var/www/yii-projects/projectname/protected/vendors/Zend/Search/Lucene/Analysis/Analyzer/Common/Text.php at line 58
0 /var/www/yii-projects/projectname/protected/vendors/Zend/Search/Lucene/Analysis/Analyzer/Common/Text.php(58): iconv()
1 /var/www/yii-projects/projectname/protected/vendors/Zend/Search/Lucene/Analysis/Analyzer.php(125): Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive->reset()
2 /var/www/yii-projects/projectname/protected/vendors/Zend/Search/Lucene/Index/SegmentWriter/DocumentWriter.php(98): Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive->setInput()
3 /var/www/yii-projects/projectname/protected/vendors/Zend/Search/Lucene/Index/Writer.php(244): Zend_Search_Lucene_Index_SegmentWriter_DocumentWriter->addDocument()
4 /var/www/yii-projects/projectname/protected/vendors/Zend/Search/Lucene.php(1410): Zend_Search_Lucene_Index_Writer->addDocument()
5 /var/www/yii-projects/projectname/protected/vendors/Zend/Search/Lucene/Proxy.php(500): Zend_Search_Lucene->addDocument()
6 /var/www/yii-projects/projectname/protected/commands/LuceneCommand.php(97): Zend_Search_Lucene_Proxy->addDocument()
7 unknown(0): LuceneCommand->actionBuildIndex()
8 /var/www/yii-projects/yii-1.1.12.b600af/framework/console/CConsoleCommand.php(173): ReflectionMethod->invokeArgs()
9 /var/www/yii-projects/yii-1.1.12.b600af/framework/console/CConsoleCommandRunner.php(68): LuceneCommand->run()
10 /var/www/yii-projects/yii-1.1.12.b600af/framework/console/CConsoleApplication.php(92): CConsoleCommandRunner->run()
11 /var/www/yii-projects/yii-1.1.12.b600af/framework/base/CApplication.php(162): CConsoleApplication->processRequest()
12 /var/www/yii-projects/projectname/protected/scripts/cron.php(14): CConsoleApplication->run()
I cannot think of any reason why there is any difference, given the measures taken, and the fact that the user is the same in both cases.
Please help!
Thanks
Edit - I should also confirm that the underlying data that is being indexed is not changing - I've executed both scenarios alternately many times and get the above results consistently.
Try with the -f switch and directly from crontab:
/usr/bin/php -f /var/www/yii-projects/projectname/protected/scripts/cron.php lucene buildIndex
Also are you sure that the text of the command you are passing is in UTF8? Could there be some other symbol there? Maybe a BOM? You can check this with a HEX editor - open your shell script, omit all the letters and see what's left. Usually a BOM in UTF8 is EF BB BF but it may not be a bom at all. Just check.
Necessary shell environment variables are not available to crontab job, so added this to irishhp users's crontab:
PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
LANG=en_US.UTF-8
which resolved.
Related
Good day everyone.
I have an issue, and Googling the issue has not helped me, basically I have the following requirement.
cronjob that runs 1st script, output is written to a file
file that is created, to have a date stamp
2nd script executes, mail the generated file as an attachment
The issue is with adding the timestamp, if I set the cron to run and just create a file with a generic filename the cronjob runs fine.
I have tried the following:
0 8-17/1 * * * python /usr/local/bin/script1.py >> /usr/local/bin/file_`date +\%Y-%m-%d`.txt 2>&1 && python /usr/local/bin/email_script.py
0 8-17/1 * * * python /usr/local/bin/acme_transcoding_check.py >> /usr/local/bin/file_$(date +"%Y-%m-%d").txt 2>&1 && python /usr/local/bin/email_script.py
Server is running Ubuntu 16.04
You need to escape the percent-sign (%) with a backslash as explained in this answer (not mine).
I want to analysis some data in one linux server,then send the it as Email text to my Email account , But when i execute this shell scripts in shell command, It works well, Weird is that when i put all the procedure into crontab job, The Email text will turns to an attached file, Can someone help?
#* * * * * sh -x /opt/bin/exec.sh >> /opt/bin/mailerror 2>&1
/* exec.sh */
#/bin/sh
cd /opt/bin
./analysis.sh > test
mail -s "Today's Weather" example#example.com < test
But when i execute exec.sh in shell command line directly, The Email will get text, Can someone explain it for me, grate thanks.
Ran into the same problem myself, only I'm piping text output into mailx - Heirloom mailx 12.4 7/29/08
When running the script on the command line the email came out as normal email with a text body.
However, when I ran the exact same script via crontab the body of the email came as an attachment - ATT00001.BIN (Outlook), application/octet-stream (mutt) or "noname" (Gmail).
Took some research to figure this out, but here goes:
Problem
Mailx will, if it encounters unknown / control characters in text input, convert it into an attachment with application/octet-stream mime-type set.
From the man page:
for any file that contains formatting characters other than newlines and horizontal tabulators
So you need to remove those control characters, which can be done with i.e. tr
echo "$Output" | /usr/bin/tr -cd '\11\12\15\40-\176' | mail ...
However since I had Norwegian UTF8 characters: æøå - the list expand, and you don't really want to maintain such a list, and I need the norwegian characters.
And inspecting the attachment I found I had only \r, \n the "regular" ASCII characters in range 32-176 - all printable and 184 and 195 --> UTF8
Sollution
Explicitly set the locale in your script:
LANG="en_US.UTF8" ; export LANG
Run export in your shell - or setenv if you run csh or tcsh to determine what your locale is set to.
Explanation
Mailx - when run in your shell - with LANG set to .UTF8, will correctly identify the UTF8 chars and continue.
When run in crontab LANG is not set, and default to LANG=C, since by default crontab will run only a restricted set of environment variables (system dependant).
mailx (or other programs) will then not recognize UTF8 characters and determine that the input containes unknown control characters.
My issue was UTF8 characters, yours could be other control characters in your input. Run it through hexdump or od -c, but since it works OK in a regular shell I'm suspecting LANG issues.
References:
linux mail < file.log has Content-Type: application/octet-stream (a noname attachment in Gmail)
http://alvinalexander.com/blog/post/linux-unix/how-remove-non-printable-ascii-characters-file-unix
I had this same issue and none of the above fixed the problem. Moving the extra return in the file fixed the issue for me:
cat logfile | tr -d \\r | mailx -s'the logfile' to-me#.....
Thanks to this forum:
https://forums.opensuse.org/showthread.php/445955-mailx-creates-unwanted-attachment
Make sure you change this in your script
#/bin/sh
to be replaced by
#!/bin/sh
Coming to the problem
Your script assumes that it is being run from a particular directory (note that almost every path is a relative path, not an absolute path). cron happens to be running it from another directory.
The Fix for text appearing on email
mydir=$(dirname "$0") && cd "${mydir}" || exit 1
./opt/bin/analysis.sh > test
mail -s "Today's Weather" example#example.com < /opt/bin/test
Explanation
$0 is the (possibly relative) filename of the shell script being executed. Given a filename, the dirname command returns the directory containing the filename.
So, that line changes directories to the directory containing the script or exits with an error code if either dirname or cd fails.
OR try to have full path like
./opt/bin/analysis.sh > test
mail -s "Today's Weather" example#example.com < /opt/bin/test
Note: The same problem is discussed earlier here
FOLLOW UP:
Try to remove
sh -x /opt/bin/exec.sh >> /opt/bin/mailerror 2>&1
and instead use
sh /opt/bin/exec.sh 2>&1 >> /opt/bin/mailerror
FOLLOW UP
You have to restart cron for changes to take effect if you do not use the crontab command to edit the file.
crontab -l > oldcrontab
cp oldcrontab newcrontab
echo "$newline" >> newcrontab
crontab < newcrontab
In my case, the cron was not a shell script but a PHP script (so I couldn't put the export LANG thing):
0 9 * * * apache php /test/myscript.php | mail -s "CRON - myscript" foo#bar.com
Solution:
In order to fix the same issue (content is mailed as attachment instead of body), I add LANG=fr_FR.UTF-8 at the beginning of the cron file:
MAILTO=vme1.etc-crond-backoffice-conf
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
LANG=fr_FR.UTF-8
0 9 * * * apache php /test/myscript.php | mail -s "CRON - myscript" foo#bar.com
NB: puting LANG=fr_FR.UTF-8 in the /etc/environment file and restarting cron service worked too.
Reference:
Set LANG in crontab https://www.logikdev.com/2010/02/02/locale-settings-for-your-cron-job/
This question already has answers here:
Is there a special restriction on commands executed by cron? [duplicate]
(2 answers)
Closed 4 years ago.
I can use this command
mysqldump -u"root" myDB| gzip > mydb_`date +%d-%m-%Y`.sql.gz
but when run in crontab
* * * * * mysqldump -u"root" myDB| gzip > mydb_`date +%d-%m-%Y`.sql.gz
( this error cause by function date, when i remove it , crontab run good )
on ubuntu, it happen this error in log file.
ubuntu CRON[xxxx] (user) CMD(mysqldump -u"root" myDB| gzip > mydb_`date+)
ubuntu CRON[xxxx] (CRON) error ( grandchild #5353 failed with exit status 2)
ubuntu CRON[xxxx] (CRON) info (no MTA installed, discarding output)
% signs in a crontab command are converted to newlines, and all data after the first % is sent to the command's stdin. Replace each % with \%.
(And you only had 4 time fields: * * * *; you need 5 (you later fixed the question).)
From the man 5 crontab:
The ``sixth'' field (the rest of the line) specifies the command to be
run.
The entire command portion of the line, up to a newline or %
character, will be executed by /bin/sh or by the shell specified
in the SHELL variable of the crontab file.
Percent-signs (%) in the
command, unless escaped with backslash (), will be changed into
newline characters, and all data after the first % will be sent to the
command as standard input. There is no way to split a single command
line onto multiple lines, like the shell's trailing "\".
The following works fine from command line
/usr/bin/mysqldump -uUser -pPass Db_name > /var/www/db_backup/db.`date +%Y%m%d%H%M`.sql
but when I try to do that in cron, I get the error:
bad ` sign
errors in crontab file, can't install
I saw someone else on the net solve the same problem by escaping the percent signs, but that didn't help and I tried it with just date inside backquotes with no format specifiers and still got the errors.
I've also seen date's argument enclosed in single or double quotes, but that doesn't help either.
Granted I could just throw it into a script and execute that I suppose - but what fun is that?
Any ideas? I'm using RHEL 5.
Try it with $() instead of backticks. And you probably do need to escape the percent signs since cron converts them to newlines otherwise.
* 0 * * * /usr/bin/mysqldump -uUser -pPass Db_name > /var/www/db_backup/db.$(date +\%Y\%m\%d\%H\%M).sql
Also, you should store the password in an option file with secure permissions (eg. 600 or 640) instead of passing it on the command line.
Put your one line script (as shown) into a proper script file and invoke that from cron:
$ cat /usr/local/bin/db-backup
#!/bin/sh
/usr/bin/mysqldump -uUser -pPass Db_name > \
/var/www/db_backup/db.`date +%Y%m%d%H%M`.sql
$ # use RHEL commands to add db-backup to your crontab
This question already has an answer here:
How is % (percent sign) special in crontab?
(1 answer)
Closed 4 years ago.
I have a crontab that looks like
0 0 * * * pg_dump DB_NAME > /path/to/dumps/`date +%Y%m%d`.dmp
which works fine when I run it manually, but not when cron runs it. After digging through the logs, I see
Dec 12 00:00:01 localhost crond[17638]: (postgres) CMD (pg_dump DB_NAME > /path/to/dumps/`date +)
It looks like a problem with percent signs, but the man page doesn't even contain the percent character at all, so I thought they were alright.
You have to escape percent signs with a backslash:
0 0 * * * pg_dump DB_NAME > /path/to/dumps/`date +\%Y\%m\%d`.dmp
From man 5 crontab:
The ‘‘sixth’’ field (the rest of the line) specifies the command to
be
run. The entire command portion of the line, up to a
newline or %
character, will be executed by /bin/sh or by the shell specified in
the
SHELL variable of the crontab file. Percent-signs (%) in the
command,
unless escaped with backslash (\), will be changed into newline
characters, and all data after the first % will be sent to the command
as
standard input. There is no way to split a single command line
onto
multiple lines, like the shell’s trailing "\".
There's another characteristic problem that can affect programs run by cron as compared with the command line (other than the interpretation of '%' signs described by Robert Gamble).
That difference is in the environment. If the program run relies on special environment variables, then it will work when you run it from the command line, with the environment you normally use, and it will likely work if you run it with at because that captures the environment when you create the job. But cron does no special environment setting.
I habitually, therefore, configure cron to run scripts by absolute pathname, and that script does the environment setting that I need (adds my $HOME/bin directory to PATH, for example). I even have a standardized infrastructure for this - a shell script that sets the environment and runs other programs.
# #(#)$Id: crontab,v 4.2 2007/09/17 02:41:00 jleffler Exp $
# Crontab file for Home Directory for Jonathan Leffler (JL)
#-----------------------------------------------------------------------------
#Min Hour Day Month Weekday Command
#-----------------------------------------------------------------------------
0 * * * * /usr/bin/ksh /work1/jleffler/bin/Cron/hourly
1 1 * * * /usr/bin/ksh /work1/jleffler/bin/Cron/daily
23 1 * * 1-5 /usr/bin/ksh /work1/jleffler/bin/Cron/weekday
2 3 * * 0 /usr/bin/ksh /work1/jleffler/bin/Cron/weekly
21 3 1 * * /usr/bin/ksh /work1/jleffler/bin/Cron/monthly
The script in /work1/jleffler/bin/Cron sets the environment and then runs the script of the same name in /work1/jleffler/bin to do the real work. The names in the Cron sub-directory are actually all links to the same script.