in linux ,bash program.
I write this:
msg=`date '+%m-%d %H:%M'`" alipay recharge [$sum] in past 15 mins"
echo $msg >> $MonitorLog
Mostly it works ,but sometime.the result will like this:
07-15 09:01 card recharge 0 in past 30 mins
My sentence changes. not 0, if $sum=0 ,it should be:
07-15 09:01 card recharge [0] in past 30 mins
I don't know where my '[]' is? can you help me ,thanks a lot.
You are hitting shell globbing. See the output below.
$ ls -l
total 4
-rw-r--r-- 1 root root 0 Jul 14 21:40 5
$ sum=10
$ msg=`date '+%m-%d %H:%M'`" alipay recharge [$sum] in past 15 mins"
$ echo $msg
07-14 21:41 alipay recharge [10] in past 15 mins
$ sum=5
$ msg=`date '+%m-%d %H:%M'`" alipay recharge [$sum] in past 15 mins"
$ echo $msg
07-14 21:41 alipay recharge 5 in past 15 mins
$ echo "$msg"
07-14 21:41 alipay recharge [5] in past 15 mins
#Etan Reisinger's answer contains the crucial pointer:
Shell expansions are inadvertently applied to $msg, because it is unquoted.
tl;dr:
Double-quote your variable references to protect them from interpretation by the shell:
echo "$msg" >> "$MonitorLog" # due to double-quoting, contents of $msg used as is
Generally, the only reason NOT to double-quote a variable reference is the express intent to have the shell interpret the value (apply expansions to it) - see below.
In the case at hand, here's what happens if you do not double-quote $msg:
After splitting the value of $msg into words by whitespace (word splitting), pathname expansion is applied to each:
I.e., each word that looks like a glob (a filename pattern), is matched against filenames - in the specified directory or, without a path component, in the current one - and if matches are found, that word is replaced by matching filenames.
A word such as [0] happens to be a valid glob ([...] encloses a set of matching characters; in this case, the set is made up of only 1 char., 0), and if a file named 0 happens to be present in the current directory, [0] is replaced by that matching filename, 0 - effectively making the [] disappear - this is what happened in the OP's case.
(See man bash, section Pathname Expansion, for what constitutes valid globs.)
Related
I am a lot confused about this behavior.
Each time I run this script from terminal, it works fine, but it fails once is executed from a crontab.
Inside the script you can find each step description.
The target is to print date and time with the peso variable to a file.
I have changed line 16 countless times. Same thing for line #4.
Edit for clarity: THIS IS JUST A SMALL PART FROM THE WHOLE SCRIPT.
It runs nice every minute. It does everything, except the peso issue.
Please HELP!!!
1 # Here I compute one decimal value (like z=0.123) with two integers (sd and mean)
2 peso=$( echo "scale=4; ${z}*${sd}/100 + ${mean}/100" | bc -l)
3 echo "peso_mtrx="$peso # This is for checking: shows 40.123 (example), so it is OK
4 peso+=";" # Add a semicolon to check its behaviour
5 echo "peso= "$peso # show it: OK
6 peso1=$(date "+%D %T") # Now I capture date and time
7 echo "fecha= "$peso1 # shows it, so it is OK
8 peso1+=";" # add a semicolon to date
9 peso1+=$peso # concatenate the peso variable
10 echo $(printf '%s' "$peso1") # shows it, so it is ok up to here
11 echo $(printf '%s' "$peso1") >> ~/projects/Files/normal.csv # WRITE TO FILE
12 # whenever I run this script from terminal, all variables showed right and even print all data into file.
13 # File stores a new line like: 02/03/21 08:24:40;40.1709;
14 # BUT... when it is executed from a CRON job... everything except peso are stored.
15 # File stores a line like: 02/03/21 08:24:40;; peso variable just vanishes.
16 # is it something related to subshells? how to solve this rigmarole?
As I was suspicious, the whole thing was related to subshell issues.
I just did something inside crontab.
Once I execute crontab -e, I initially had something like:
*/1 * * * * /absolute/path/to/project.sh
So doing some reading I ended up doing this:
SHELL=/bin/bash
*/1 * * * * exec bash -l /absolute/path/to/project.sh
I beg to an expert to enlighten us about this solution. As far as I do understand, it is related to create a login shell inside cron using the information stored in .bash_profile.
It did enable the environment variables to be reachable.
This question already has answers here:
Change file's numbers Bash
(2 answers)
Closed 2 years ago.
I need to implement a script (duplq.sh) that would rename all the text files existing in the current directory using the command line arguments. So if the command duplq.sh pic 0 3 was executed, it would do the following transformation:
pic0.txt will have to be renamed pic3.txt
pic1.txt to pic4.txt
pic2.txt to pic5.txt
pic3.txt to pic6.txt
etc…
So the first argument is always the name of a file the second and the third always a positive digit.
I also need to make sure that when I execute my script, the first renaming (pic0.txt to pic3.txt), does not erase the existing pic3.txt file in the current directory.
Here's what i did so far :
#!/bin/bash
name="$1"
i="$2"
j="$3"
for file in $name*
do
echo $file
find /var/log -name 'name[$i]' | sed -e 's/$i/$j/g'
i=$(($i+1))
j=$(($j+1))
done
But the find command does not seem to work. Do you have other solutions ?
The problem you're trying to solve is actually somewhat tricky, and I don't think you've fully thought it through. For instance, what's the difference between duplq.sh pic 0 3 and duplq.sh pic 2 5 -- it looks like both should just add 3 to the number, or would the second skip "pic0.txt" and "pic1.txt"? What effect would either one have on files named "pic", "pic.txt", "picture.txt", "picture2.txt", "pic2-2.txt", or "pic999.txt".
There are also a bunch of basic mistakes in the script you have so far:
You should (almost) always put variable references in double-qotes, to avoid unexpected word-splitting and wildcard expansion. So, for example, use echo "$file" instead of echo $file. In for file in $name*, you should put double-quotes around the variable but not the *, because you want that to be treated as a wildcard. Hence, the correct version is for file in "$name"*
Don't put variable references in single-quotes, they aren't expanded there. So in the find and sed commands, you aren't passing the variables' values, you're passing literal dollar signs followed by letters. Again, use double-quotes. Also, you don't have a "$" before "name", so it won't be treated as a variable even in double-quotes.
But the find and sed commands don't do what you want anyway. Consider find /var/log -name "name[1]" -- that looks for files named "name1", not "name1" + some extension. And it looks in the current directory and all subdirectories, which I'm pretty sure you don't want. And the "1" ("$i") may not be the number in the current filename. Suppose there are files named "pic0.jpg", "pic0.png", and "pic0.txt" -- on the first iteration, the loop might find all three with a pattern like "pic0*", then on the second and third iterations try to find "pic1*" and "pic2*, which don't exist. On the other hand, suppose there are files named "pic0.txt", "pic5.txt", and "pic8.txt" -- again, it might look for "pic0*" (ok), then "pic1*" (not found), and then "pic2*" (ditto).
Also, if you get to multi-digit numbers, the pattern "name[10]" will match "file0" and "file1", but not "file10". I don't know why you added the brackets there, but they don't do anything you'd want.
You already have the files being listed one at a time in the $file variable, searching again with different criteria just adds confusion.
Also, at no point in the script do you actually rename anything. The find | sed line will (if it works) print the new name for the file, but not actually rename it.
BTW, when you do use the mv command, use either mv -n or mv -i to keep it from silently and irretrievably overwriting files if/when a name conflict occurs.
To prevent overwriting when incrementing file numbers, you need to do the renames in reverse numeric order (i.e. rename "pic3.txt" to "pic6.txt" before renaming "pic0.txt" to "pic3.txt"). This is especially tricky because if you just sort filenames in reverse alphabetic order, you'll get "pic7.txt" before "pic10.txt". But you can't do a numeric sort without removing the "pic" and ".txt" parts first.
IMO this is actually the trickiest problem to be solved in order to get this script to work right. It might be simplest to specify the largest index number as one of the arguments, and have it start there and count down to 0 (looping over numbers rather than files), and then for each number iterate over matching files (e.g. "pic0.jpg", "pic0.png", and "pic0.txt").
So I assume that 0 3 is just a measurement for the difference of old num and new num and equivalent to 1 4 or 100 103.
To avoid overwriting existing files, create a new temp dir, move all affected files there, and move all of them back in the end.
#/bin/bash
#
# duplq.sh pic 0 3
base="$1"
delta=$(( $3 - $2 ))
# echo delta $delta
target=$(mktemp -d)
echo $target
# /tmp/tmp.7uXD2GzqAb
add () {
f="$1"
b="$2"
d=$3
num=${f#./${b}}
# echo -e "file: $f \tnum: $num \tnum + d: $((num + d))" ;
echo -e "$((num + d))" ;
}
for f in $(find -maxdepth 1 -type f -regex ".*/${base}[0-9]+")
do
newnum=$(add "$f" "${base}" $delta)
echo mv "$f" "$target/${base}$newnum"
done
# exit
echo mv $target/${base}* .
First I tried to just use bash syntax, to check, whether removal of the prefix (pic) results in just digits remaining. I also didn't use the extension .txt - this is left as an exercise for the reader. From the question it is unclear - it is never explicitly told, that all files share the same extension, but all files in the example do.
With the -regex ".*/${base}[0-9]+") in find, the values are guaranteed to be just digits.
num=${f#./${b}}
removes from file f the base ("pic"). Delta d is added.
Instead of really moving, I just echoed the mv-command.
#TODO: Implement the file name extension conservation.
And 2 other pitfalls came to my mind: If you have 3 files pic0, pic00 and pic000 they all will be renamed to pic3. And pic08 will be cut into pic and 08, 08 will then be tried to be read as octal number (or 09 or 012129 and so on) and lead to an error.
One way to solve this issue is, that you prepend the extracted number (001 or 018) with a "1", then add 3, and remove the leading 1:
001 1001 1004 004
018 1018 1021 021
but this clever solution leads to new problems:
999 1999 2002 002?
So a leading 1 has to be cut off, a leading 2 has to be reduced by 1. But now, if the delta is bigger, let's say 300:
018 1018 1318 318
918 1918 2218 1218
Well - that seems to be working.
I have some scripts in my RedHat server which contains Microfocus COBOL programs which generates a huge file of aprox 3GB in a sort of time of 3 hours on average. The programs write their output files directly in the directory /my_test/files/.
The problem is that sometimes (randomly) some files generated contains null character sections in the middle of the file. And when I check them up, if I reexecute the script again (with the same input parameters), the output file is perfectly generated (it doesn't contain any nullchars). I've checked it a lot of times and I'm pretty sure is not the fault of the COBOL programs (they use quite simple operations). The space in use of that folder is 40%.
Some programs updates the database, and if they finish with return code 0, then the changes are commited, and I don't have any backup, so this is the point of what I'm doing.
This is an example of a file declaration of one of the problematic COBOL programs:
FILE-CONTROL.
SELECT MYFILE
ASSIGN TO MYFILE
ORGANIZATION IS SEQUENTIAL
ACCESS MODE IS SEQUENTIAL
FILE STATUS IS FILE-STATUS.
DATA DIVISION.
FILE SECTION.
FD MYFILE
LABEL RECORD STANDARD
RECORDING MODE F.
01 REG-OUTPUT PIC X(400).
I've also checked for the nulls in the COBOL programs before the NULL files, but unfortunately there are no nulls spotted.
Then I thought about creating a crontab which executes the following script each 5 seconds:
if [[ -f /tmp/sorry_im_working ]]; then
exit
fi
trap 'rm -rf /tmp/sorry_im_working' EXIT
touch /tmp/sorry_im_working
lsof | awk 'BEGIN{
sfiles="";
} {
if($1=="PROGRAM" && $9~/my_test\/files/){
sfiles=sfiles" "$9
}
}END{
comm="find "sfiles" -newermt \x27-2 seconds\x27 -exec env LC_ALL=C bash -c \x27grep -Pq \x22\x5Cx00{200}\x22 <(tail -c 1000 {}) && echo {}\x27 \x5C\x3B";
while(comm | getline sout){
print sout;
};
close(comm);
}' >> /home/ouhma/nullfiles.txt
Therefore, I would like to ask you the following questions:
Any idea of what's going on here?
Do you have any other way to trigger the lastest modified files?
What other information of interest could I add to my log?
If you construct a file d with only \x00 :
hexdump -C d
00000000 5c 78 30 30 0a |\x00.|
00000005
and you :
grep -Faq '\x00' d;echo $?
0
But they're no null caracter inside d.
Maybe, is better to use grep -Paq '\x00'
Depending on the configuration and record structure that is used for the file MF will pad different characters with hex null.
Please copy the 'ASSIGN' clause and the 'FD' clause of the COBOL program.
BTW: if your COBOL programs run three ours to do some calculations and write three GB of data back you should investigate the storage and / or get a COBOL programmer to check the programs, sounds much to slow.
I suspect you are have non-printable characters in your file, the null inserts can be controlled, take a look # INSERTNULL file configuration.
echo ^`date`
^Wed Jan 21 05:49:37 CST 2015
deeps=`echo ^`date``
echo $deeps
Actual Result:
^date
Expected Result :
^Wed Jan 21 05:49:37 CST 2015
Need help on this
Try this method
deeps=^$(date)
echo $deeps
Output :
^Wed Jan 21 18:44:25 IST 2015
Backticks are horribly outdated and should not be used any more -- using $() instead will save you many headaches
Use a backtick, or use Command substitution. Like
# Shell command substitution.
echo ^$(date)
or
# backticks.
deeps=`date`
echo ^$deeps
Both output (the requested)
^Wed Jan 21 08:16:01 EST 2015
Simply because neither of the other (correct) answers actually explain the problem I'm adding another answer.
tl;dr Compare the backtick version to the $() version
The difference between
echo ^`date`
and
deeps=`echo ^`date``
is how many backticks are on the line and how the shell is parsing the line.
echo ^`date`
has a single pair of backticks and the shell parses it as (where the [] are marking "parts" of the line)
[echo] [^][`date`]
The
`date`
bit is then expanded via Command Substitution and so the line becomes
[echo] [^Wed Jan 21 05:49:37 CST 2015]
and then echo spits out the desired ^Wed Jan 21 05:49:37 CST 2015.
This line however
deeps=`echo ^`date``
is parsed as
[deeps][=][`echo ^`][date][``]
which you can already see is quite different and not correct (this happens because backticks cannot be nested for the record).
There are now two command substitutions on this line echo ^ and the empty string so the line becomes
[deeps][=][^][date][]
or with the "words" combined
[deeps][=][^date]
which then assigns ^date to deeps and echo $deeps then gets you ^date.
The $() form of command substitution, no the other hand, does nest and thus
deeps=$(echo ^$(date))
parses as
[deeps][=][$([echo] [^][$([date])])]
which properly runs both date and echo on the result. Though, as indicated in the other answers, the wrapping echo is not necessary as deeps=^$(date) will work just fine by itself.
I'm trying to write a prepare-commit-msg git hook script to check the contents of the last 10 commit messages and check to see if the message that you are attempting to enter is unique and prevent the user from checking in (without the --no-verify overload) if it detects it. When I run this command line in Git I get the following output.
dacke#MachineName /c/Development/Project (tests)
$ git log --pretty=format:'%h|%an|%s' --max-count=10
2919dc2|Eric|Test Message
4ef580c|Eric|Test Message
1a0051b|Eric|Test Message
3e2df42|Eric|Test Commit
a08d4c1|Bob|DE6717 - What I did to fix this defect
aff8afc|Bob|DE6717 - Here is some more defect info
bbbfb67|Ralph|Merge branch 'clean_up' into develop
72d0968|Ralph|Forgot to remove deleted class from the project.
bfd1505|Ralph|Clean up.
d21c6dc|Bruce|Merge branch 'Icons' into develop
My prepare-commit-msg is written like so.
1 #!/bin/bash
2
3 printf "Prepare-Commit-Msg Hook Running...\n"
4
5 #$1 = "Commit Message File 'COMMIT_EDITMSG'"
6 #$2 = "message"
7 commitMessage=$(cat "$1")
8
9 # Prevent people putting in the same commit message multiple times by looking for an identical message in the last 10 commits
10 declare -a last10CommitMessages
11 rawMessages=$(git log --pretty=format:'%h|%an|%s«' --max-count=10)
12 printf "Raw Messages Length: %d\n" "${#rawMessages[#]}"
13 for line in ${rawMessages//«/ };
14 do
15 #printf "%s\n" $line
16 last10CommitMessages+=($line);
17 done
18 printf "Last 10 Commit Length: %d\n" "${#last10CommitMessages[#]}"
19
20 # Temp exit 1 to prevent commit during testing
21 exit 1
When I try to run the "commit" I get the following output.
Raw Messages Length: 1
Last 10 Commit Length: 63
If I uncomment line 15 I can see that for every space and line break I'm getting an item added to the array. On top of that the character that I actually wanted to split the lines on is added to the end which means that I would need yet another method to take this off the end.
I am new to bash scripting and I'm coming from a C# / Windows background so I still learning. Can someone please provide me a simple solution to the problem? More important to me than a quick answer is an answer that can explain HOW this actually works. I've found a lot of conflicting information that does not work for me on the web. I plan on writing a blog piece about this after I get it all figured out so it's important that I don't get any "It just works" as an answer. Thanks.
Simply change
rawMessages=$(git log --pretty=format:'%h|%an|%s«' --max-count=10)
to this
rawMessages=($(git log --pretty=format:'%h|%an|%s«' --max-count=10))
$( ) evaluates the command inside and saves it as one string, ignoring line breaks. When you wrap something with ( ), it evaluates the contents as an array.
EDIT:
If you do this you will see you have way more elements in the array than you wanted. This is because the array will split the string by new line character and white space. To ignore white space you can do as hlovdal suggested and do this..
OLD_IFS="$IFS"
IFS=$'\n'
rawMessages=($(git log --pretty=format:'%h|%an|%s«' --max-count=10))
IFS="$OLD_IFS"
The words are split due to the IFS variable (Internal Field Separator - an ancient unix relic...) which has default value "<space><tab><newline>". Change your loop to
oldIFS=$IFS
IFS=«
for line in ${rawMessages}
do
printf "%s\n" $line
last10CommitMessages+=($line);
done
IFS=$oldIFS