Here is a snippet from a script which I generally execute from cron:
if [ "$RESCAN_COMMAND" = "wipecache" ]; then
log "Linking cover art."
find $FLAC_DIR -name "*.jpg" | while read f; do c=`echo $f | sed -e 's/flac/mp3/g'`; ln -s "$f" "$c"; done
log "Done linking cover art"
fi
The script works perfectly when run from the command line. But when run by cron (as the same user) it fails somewhere in the find line. The "Done" message is not logged and the script does not continue beyond the if block.
The find line creates links from files like flac/Artist/Album/cover.jpg to mp3/Artist/Album/cover.jpg. There are a few hundred files to link. The command generates a lot of output to stderr, because most, if not all, of the links already exist.
On a hunch, I tried redirecting the stderr of the ln command to /dev/null:
find $FLAC_DIR -name "*.jpg" | while read f; do c=`echo $f | sed -e 's/flac/mp3/g'`; ln -s "$f" "$c" 2>/dev/null; done
With that change, the script executes successfully from cron (as well as from the command line).
I would be interested to understand why.
Could it be this bug report: https://bugs.launchpad.net/ubuntu/+source/cron/+bug/151231
It's probably producing too much output. This really isn't a bug, but a feature as cron typically send emails with it's output. MTA's don't like text messages with many many lines, so cron just quits. Maybe the silent quit is a bug though.
You could also use ln -f to suppress the ln errors in only the case of pre-existing files.
Related
I'm a bash noob, and I am trying to set up a sort of "hot reload" functionality for a project I'm working on using inotifywait. Ubuntu 20.04 if that matters.
Here is what I hoped would have worked:
inotifywait -m -r ../.. -e modify,create,delete |
while read line; do
custom_command
done
I'm having two problems:
Issue #1 is that custom_command takes some time to work, and so if I make more changes to the directory in the meantime, custom command appears to "queue up" custom_command, where really I just want it to keep the most recent one and drop the others.
Issue #2 is that I'm getting some sort of "double output." So for example if I bash auto-exec.sh and auto-exec.sh looks like this:
inotifywait -m -r . -q -e modify,create,delete
Then each time a change registers, I get this as output (not a mistake that it's doubled -- I get two identical lines each time there is a modification):
./ MODIFY auto-exec-testfile.txt
./ MODIFY auto-exec-testfile.txt
I should note I've tried making changes both with Visual Code Studio and gedit, with the same results.
If I modify the bash file like so:
inotifywait -m -r . -q -e modify,create,delete |
while read line; do
echo "$line"
echo "..."
done
I get the following output each time there is a change:
./ MODIFY auto-exec-testfile.txt
...
./ MODIFY auto-exec-testfile.txt
...
If I modify bash_test.sh to the following:
inotifywait -m -r . -q -e modify,create,delete |
while read line; do
echo "help me..."
done
Then I get the following each time a change is made:
help me...
help me...
What happened to the the ./ MODIFY ... line?? Presumably there's something I don't understand about bash, stdout or similar /related concepts here?
And finally, if I change the .sh file to the following:
inotifywait -m -r . -q -q -e modify,create,delete |
while read _; do
echo "help me..."
done
Then I get no output at all. This one I think I understand, because the -q -q means that inotifywait is in "super silent" mode, so there is no log and therefore nothing to trigger the while.
What I'd love to do is just trigger the code once when something changes, and drop all but the most recent execution. I'm not sure doing this using a while is entirely necessary, but I tried inotifywait -m -r . -q -q -e modify,create,delete | echo "help me..", and the script printed "help me..." once at startup, then exited on modification.
Assistance very much appreciated.
EDIT - 20201-Mar-23
I removed -m and create from the inotifywait line, and it appears to work as expected, except that it doesn't stay "up" in monitor mode. So this at least only gives me one entry from inotifywait:
notifywait -r .. -q -e modify,delete |
while read line1; do
echo ${line1}
done
Related:
inotifywait - pause monitoring while executing command
https://unix.stackexchange.com/questions/140679/using-inotify-to-monitor-a-directory-but-not-working-100
inotifywait not performing the while loop in bash script
while inotifywait -e close_write,delete .; do
pkill custom_command
custom_command&
done
please bear with me if my terminology or syntax is less than stellar (still learning). I currently have a simple bash script that checks the arguments of the command and outputs files names with matching text. This part of my script works correctly via a grep command and piped to xargs for proper formatting.
When running the script, I run through a simple loop to check if the value is null and then move to running my variable/search if not.
My question is: Is it possible to have this script output via stdout AND also save a new file each time it is run with the user input and date/time? (but not overwrite) EX: report-bob-0729161500.rpt
I saw same other suggestions to use tee with the command, but I was trying to get it to work within the script. Similarly, another suggestion stated to utilize exec > >(tee -i logfile.txt), but I am unsure how to properly format this to include the date/time and $1 input into new files each time the script is executed.
Any help or suggested resources?
Thank you.
SEARCH=`[search_variable]`
if [ -z "$SEARCH" ]
then
echo "$1 not found."
else
echo -e "REPORT LISTING\n\n"
echo "$SEARCH"
fi
EDIT: I did try simply piping the echo statements to the tee command, which does work. However, I am still curious if anyone has other suggestions to accomplish this same task via alternative methods. Thank you.
With echo statements piped to tee:
SEARCH=`[search_variable]`
DATE=`date +"%m%d%y%k%M"`
if [ -z "$SEARCH" ]
then
echo "$1 not found."
else
echo -e "REPORT LISTING\n\n" | tee tps-list-$1-$DATE.rpt
echo "$SEARCH" | tee tps-list-$1-$DATE.rpt
fi
If you want to do it within the script, why then not just write to
both standard output and the file (using append where appropriate?).
Maybe a bit more writing, but it gives complete control.
Leon
So I am running into a problem with unix scripts that use curl to make rest calls. I have one script, that runs two other scripts inside of it.
cat example.sh
FILE="file1.txt"
RECIP="wilfred#blamagam.com"
rm -f $FILE
./script1.sh > $FILE
mail -s "subject" $RECIP < $FILE
RECIP="bob#blamagam.com"
rm -f $FILE
./script2.sh > $FILE
mail -s "subject" $RECIP < $FILE
exit 0
Each script makes rest calls to the same service. It is my understanding that script1.sh should completely finish before script2.sh is ran, however that is not the case. In the logs for the rest service I see a rest call from the second script in the middle of the first one still executing. The second script then fails because of this (it does not get any data returned).
I am modifying this process so I am not the one who originally wrote it. I am not seeing any forked processes, or background processes at all and I have been banging my head against the wall.
I do know that script2.sh works. Whenever script1.sh takes under a minute script2.sh works just fine, but more often than not script1.sh takes over a min, causing the second script to fail.
This is ran by a cron, and the contents of the files are mailed out, so I cant just default to running them manually. Any suggestions for what to look into would be much appreciated!
EDIT: Here is a high pseudo code example
script1.sh
ITEMS=`/usr/bin/curl -m 10 -k -u userName:passWord -L https://server/rest-service/rest?where=clause=value;clause2=value2&sel=field 2>/dev/null | sed s/<\/\?Attribute[^>]*>/\n/g | grep -v '^<' | grep -v '^$' | sed 's/ //g'`
echo "\n Subject for these metrics"
echo "$ITEMS"
Both scripts have lots of entries like this. There are 2 or 3 for loops but they are simple and I do not see any background processes being called. Its a large script so I could only provide a snippet. Could the rest call into pipes be causing an issue?
Edit:
Just tested this on my system and it seems to work.
cat example.sh
FILE="file1.txt"
RECIP="wilfred#blamagam.com"
rm -f "$FILE"
(./script1.sh > "$FILE") &
procscript1=$!
wait "$procscript1"
mail -s "subject" "$RECIP" < "$FILE"
RECIP="bob#blamagam.com"
rm -f "$FILE"
(./script2.sh > "$FILE") &
procscript2=$!
wait "$procscript2"
mail -s "subject" "$RECIP" < "$FILE"
exit 0
Put the script executions in the background with the &.
Get the process id's for each script execution.
Use the wait command to block until the execution is done.
I currently use tail -f to monitor a log file: this way I get an autorefreshing console monitoring a web server.
Now, said webserver was moved to another host and I have no shell privileges for that.
Nevertheless I have a .txt network path, which in the end is a log file which is constantly updated.
So, I'd like to do something like tail -f, but on that url.
Would it be possible?In the end "in linux everything is a file" so..
You can do auto-refresh with help of watch combined with wget.
It won't show history, like tail -f, rather update screen like top.
Example of command, that shows content on file.txt on the screen, and update output every five seconds:
watch -n 5 wget -qO- http://fake.link/file.txt
Also, you can output n last lines, instead of the whole file:
watch -n 5 "wget -qO- http://fake.link/file.txt | tail"
In case if you still need behaviour like "tail -f" (with keeping history), I think you need to write a script that will download log file each time period, compare it to previous downloaded version, and then print new lines. Should be quite easy.
I wrote a simple bash script to fetch URL content each 2 seconds and compare with local file output.txt then append the diff to the same file
I wanted to stream AWS amplify logs in my Jenkins pipeline
while true; do comm -13 --output-delimiter="" <(cat output.txt) <(curl -s "$URL") >> output.txt; sleep 2; done
don't forget to create empty file output.txt file first
: > output.txt
view the stream :
tail -f output.txt
original comment : https://stackoverflow.com/a/62347827/2073339
UPDATE:
I found better solution using wget here:
while true; do wget -ca -o /dev/null -O output.txt "$URL"; sleep 2; done
https://superuser.com/a/514078/603774
I've made this small function and added it to the .*rc of my shell. This uses wget -c, so it does not re-download the whole page:
# Poll logs continuously over HTTP
logpoll() {
FILE=$(mktemp)
echo "———————— LOGPOLLING TO $FILE ————————"
tail -f $FILE &
tail_pid=$!
bg %1
stop=0
trap "stop=1" SIGINT SIGTERM
while [ $stop -ne 1 ]; do wget -co /dev/null -O $FILE "$1"; sleep 2; done
echo "——————————— LOGPOLL DONE ————————————"
kill $tail_pid
rm $FILE
trap - SIGINT SIGTERM
}
Explanation:
Create a temporary logfile using mktemp and save its path to $FILE
Make tail -f output the logfile continuously in the background
Make ctrl+c set stop to 1 instead of exiting the function
Loop until stop bit is set, i.e. until the user presses ctrl+c
wget given URL in a loop every two seconds:
-c - "continue getting partially downloaded file", so that wget continues instead of truncating the file and downloading again
-o /dev/null - wget's log messages shall be thrown into the void
-O $FILE - output the contents to the temp logfile we've created
Clean up after yourself: kill the tail -f, delete the temporary logfile, unset the signal handlers.
The proposed solutions periodically download the full file.
To avoid that I've created a package and published in NPM that does a HEAD request ( getting the size of the file ) and requesting only the last bytes.
Check it out and let me know if you need any help.
https://www.npmjs.com/package/#imdt-os/url-tail
I'm creating a bash script that will run a process in the background, which creates a socket file. The socket file then needs to be chmod'd. The problem I'm having is that the socket file isn't being created before trying to chmod the file.
Example source:
#!/bin/bash
# first create folder that will hold socket file
mkdir /tmp/myproc
# now run process in background that generates the socket file
node ../main.js &
# finally chmod the thing
chmod /tmp/myproc/*.sock
How do I delay the execution of the chmod until after the socket file has been created?
The easiest way I know to do this is to busywait for the file to appear. Conveniently, ls returns non-zero when the file it is asked to list doesn't exist; so just loop on ls until it returns 0, and when it does you know you have at least one *.sock file to chmod.
#!/bin/sh
echo -n "Waiting for socket to open.."
( while [ ! $(ls /tmp/myproc/*.sock) ]; do
echo -n "."
sleep 2
done ) 2> /dev/null
echo ". Found"
If this is something you need to do more than once wrap it in a function, but otherwise as is should do what you need.
EDIT:
As pointed out in the comments, using ls like this is inferior to -e in the test, so the rewritten script below is to be preferred. (I have also corrected the shell invocation, as -n is not supported on all platforms in sh emulation mode.)
#!/bin/bash
echo -n "Waiting for socket to open.."
while [ ! -e /tmp/myproc/*.sock ]; do
echo -n "."
sleep 2
done
echo ". Found"
Test to see if the file exists before proceeding:
while [[ ! -e filename ]]
do
sleep 1
done
If you set your umask (try umask 0) you may not have to chmod at all. If you still don't get the right permissions check if node has options to change that.