BACKGROUND
I would like to explain the scenario properly here.
I am running jenkins_2.73.3 in my cloud server with ubuntu 16.04.
Currently, there are 3 users in the server:
root
develop-user (which I had created for many reasons such as test,deploy etc)
jenkins (which was created by jenkins ofcourse, I also added this jenkins user to sudoers group)
PROBLEM
I have a bash script that I am calling from a build step in Jenkins. Within this bash script,there is a nohup command for calling a separate deployScript in the background such as:
#!/bin/bash
nohup deployScript.sh > $WORKSPACE/app.log 2>&1 & echo $! > save_pid.txt
After the build step is completed, I see that a id is generated inside save_pid.txt but app.log is surprisingly empty. I can't kill any processes with this generated pid. So, that means there isn't any process created in the first place here. Also, the deployScript.sh does not seem to have any effect at all. It's just not working. This happens everytime I run the build in Jenkins. I can assure that there is nothing wrong with the deployScript.sh.
I have tried running this bash script with the develop-user manually without Jenkins and it works perfectly. Contents are written to the log file and also I can use the generated pid to kill the process. I have also tested this in my local environment and it works.
QUESTION
I have been looking at this for days. What might be the root cause here ?Where can I look into to see some logs or other info ? How is the pid generated whereas the log file is empty ? Is it a permission issue with the jenkins user ? Please help.
You can use below line inside the execute shell in jenkins to run it in background without the process being killed.
BUILD_ID=dontKillMe <command> &
So, it turned out to be a permission issue and also the script wasn't executable I guess as pointed out in the comments above.
So, now the bash script looks like below:
#!/bin/bash
sudo chmod a+x deployScript.sh
sudo nohup deployScript.sh > $WORKSPACE/app.log 2>&1 & echo $! > save_pid.txt
This works.
Related
I have a discord bot running on a raspberry pi that i need to restart every day, I'm trying to do this through crontab but with no luck.
It manages to kill the active screen processes but never starts an instance, not that I can see with "screen -ls" and I can tell that it doesn't create one that I can't see as the bot itself does not come online.
Here is the script:
#!/bin/bash
sudo pkill screen
sleep 2
screen -dmS discordBot
sleep 2
screen -S "discordBot" -X stuff "node discordBot/NEWNEWNEWN\n"
Here is also the crontab:
0 0 * * * /bin/bash /home/admin/discordBot/script.sh
Is it possible to have crontab run a screen session? And if so how?
Previously I tried putting the screen command stright into cron but now I have it in a bash script instead.
If I run the script in the terminal it works perfectly, it’s just cron where it fails. Also replacing "screen" with the full path "/usr/bin/screen" does not change anything.
So the best way of doing it, without investigating the underlying problem would be to create a systemd service and putting a restart command into cron.
/etc/systemd/system/mydiscordbot.service:
[Unit]
Description=A very simple discord bot
[Service]
Type=simple
ExecStart=node /path/to/my/discordBot.js
[Install]
WantedBy=multi-user.target
Now you can run your bot with systemctl start mydiscordbot and can view your logs with journalctl --follow -u mydiscord bot
Now you only need to put
45 2 * * * systemctl restart discordbot
into root's crontab and you should be good to go.
You probably should also write the logs into a logfile, maybe /var/log/mydiscordbot.log, but how and if you do that is up to you.
OLD/WRONG ANSWER:
cron is run with a minimal environment, and depending on your os, /usr/bin/ is probably not in the $PATH var. screen is mostlikely located at /usr/bin/screen, so cron can't run it because it can't find the screen binary. try replacing screen in your script with /usr/bin/screen
But the other question here is: Why does your discord bot need to be restarted every day. I run multiple bots with 100k+ users, and they don't need to be restarted at all. Maybe you should open a new question with the error and some code snippets.
I don't know what os your rpi is running, but I had a similar issue earlier today trying to get vms on a server to launch a terminal and run a python script with crontab. I found a very easily solution to automatically restart it on crashes, and to run something simply in the background. Don't rely on crontab or an init service. If your rpi as an x server running, or anything that can be launched on session start, there is a simple shell solution. Make a bash script like
while :; do
/my/script/to/keep/running/here -foo -bar -baz >> /var/log/myscriptlog 2>&1
done
and then you would start it on .xprofile or some similar startup operation like
/path/to/launcher.sh &
to have it run the background. It will restart automatically everytime it closes if started in something like .xprofile, .xinitrc, or anything ran at startup.
Then maybe make a cronjob to restart the raspberry pi or whatever system is running the script but this script wil restart the service whenever it's closed anyway. Maybe you can put that cronjob on a script but I don't think it would launch the GUI.
Some other things you can do to launch a GUI terminal in my research with cronjob that you can try, though they didn't work for my situation on these custom linux vms, and that I read was a security risk to do this from a cronjob, is you can specify the display.
* * * * * DISPLAY=:0 /your/gui/stuff/here
but you would would need to make sure the user has the right permissions and it's considered an unsafe hack to even do this as far as I have read.
for my issue, I had to launch a terminal that stayed open, and then changed to the directory of a python script and ran the script, so that the local files in directory would be called in the python script. here is a rough example of the "launcher.sh" I called from the startup method this strange linux distro used lol.
#!/bin/sh
while :; do
/usr/bin/urxvt -e /bin/bash -c "cd /path/to/project && python loader.py"
done
Also check this source for process management
https://mywiki.wooledge.org/ProcessManagement
I am using rclone to mount a folder from my cloudstorage on my local computers. however, on one machine I only connect via terminal and I want to mount the volume on startup.
So I setup a small shell-script with following contents:
rclone mount remoterep:/examplefolder ~/Documents/examplefolder
and I call it in bashrc with exec ~/mount_examplefolder
when I ssh into said computer, it is working as I do not get any errors but the shell refuses to take any further commands as the mount command is executed.
If I add another ssh login, I get an error-prompt, because it can't overwrite the mount folder from the other session.
So how do I fix this, that the rclone is being executed in the background giving me access to shell back?
Or am I restricted to mounting it manually and then using another ssh session to perform the desired actions?
There's a couple of things here that are causing some problems.
First, when you use exec to spawn a process in the shell, that means that you want to replace the existing shell process with the program you've mentioned. When you do that in an SSH session, you replace the shell process that the SSH daemon started (and you were intending to use to log in). SSH will then wait for that process to exit (which it won't until the volume is umounted), which is why you see the hang. You'll want to skip the exec in your shell configuration, which will spawn the process without replacing your shell.
Second, the reason you see the error is that the mount process is designed to be run once, as you've noticed. If you want to skip mounting the folder if it's already mounted, you can use something like the following as your shell script:
#!/bin/sh
if ! grep " $HOME/Documents/examplefolder " /proc/mounts
then
rclone mount remoterep:/examplefolder ~/Documents/examplefolder
fi
Note the spaces inside the quotes that ensure that you haven't matched something else by accident. This will ensure that your script doesn't try to mount multiple times.
Third, you'll probably want to run this command in the background and detached from the shell so that the exit of the shell doesn't cause it to receive SIGHUP and exit (or restart, depending on how it's configured). You can do this by writing the invocation in your shell configuration as nohup ~/mount_examplefolder >/dev/null 2>&1 &. nohup prevents the program from receiving SIGHUP and redirecting output prevents it from printing messages or creating nohup.out files all over the place.
Finally, you may (or may not) want to configure this to run only when you're using an interactive shell; that is, when you're logging in to start a shell for interactive use rather than scripting use. If so, you can make the invocation of nohup condition on PS1 being set like so:
if [ -n "$PS1" ]
then
nohup ~/mount_examplefolder >/dev/null 2>&1 &
fi
I am trying to run a shell script which runs at system login. To try, I used an example script, which has two lines
#!/bin/bash
echo “Hello World”
I Followed all the instructions mentioned on this website http://www.cyberciti.biz/tips/linux-how-to-run-a-command-when-boots-up.html
Even I did some steps such as editing /etc/rc.local, but still when I login, I do not see Hello world output running on terminal.
Can anyone please explain what is wrong here or may be I am missing something?
Looks to me like Ubuntu 16.04 is a systemd system.
Which means you should create a systemd service to run whatever you'd like # startup.
Look here https://wiki.ubuntu.com/SystemdForUpstartUsers#Example_Systemd_service
After you made your service, use systemctl to enable it on boot for systemd.
sudo systemctl enable mycustom.service
To start the service.
sudo systemctl start mycustom.service
You can also schedule a cron job to run once after the reboot command. The line will need "#reboot root /path/script.sh" You can specify whatever user you want to run the script. To make the result easier to find, you might make that second line something like
echo "Hello World" > /root/hello.txt
Add the line in /etc/profile under echo command as follows
echo “Hello World”
Then execute the following command
source /etc/profile
I have a linux shell script that when run from command line works perfectly but when scheduled to run via crontab, it does not give desired results.
The script is quite simple, it checks to see if mysql-proxy is running or not by checking if its pid is found using the pidof command. If found to be off, it attempts to start the proxy.
# Check if mysql proxy is off
# if found off, attempt to start it
if pidof mysql-proxy
then
echo "Proxy running."
else
echo "Proxy off ... attempting to restart"
/usr/local/mysql-proxy/bin/mysql-proxy -P 172.20.10.196:3306 --daemon --proxy-backend-addresses=172.20.10.194 --proxy-backend-addresses=172.20.10.195
if pidof mysql-proxy
then
echo "Proxy started"
else
echo "Proxy restar failed"
fi
fi
echo "==============================================="
The script is saved in a file check-sql-proxy.sh and has permissions set to 777. When I run the script from command line (sh check-sql-proxy.sh) it gives the desired output.
4066
Proxy running.
===============================================
The script is also scheduled to run every 5 minutes in crontab as
*/5 * * * * bash /root/auto-restart-mysql-proxy.sh > /dev/sql-proxy-restart-log.log
However, when I see the sql-proxy-restart-log.log file it contains the output:
Proxy off ... attempting to restart
Proxy restar failed
===============================================
It seems that pidof command fails to return the pid of the running application which brings the flow of script in else condition.
I am unable to figure out how to resolve this since when I run the script manually, it works fine.
Can anyone help what I am missing with regards to permissions or settings?
Thanks in advance.
Mudasser
Check that the shell is what you think it is (usually /bin/sh, not bash)
Also check that PATH environment variable. Usually, for cron jobs it is a good practice to fully qualify all paths to binaries, e.g.
#!/bin/bash
# Check if mysql proxy is off
# if found off, attempt to start it
if /bin/pidof mysql-proxy
etc.
Try pidof /usr/local/mysql-proxy/bin/mysql-proxy (full path to executable)
In common, try use the same command name as was used to start the instance of mysql-proxy.
The problem seems that crontab environment don't have the same environment as you.
You have 2 simple & proper solutions :
In the first lines of crontab :
PATH=/foo:/bar:/qux
SHELL=/bin/bash
or
source ~/.bashrc
in your scripts.
I have a node.js script which need to start at boot and run under the www-data user. During development I always started the script with:
su www-data -c 'node /var/www/php-jobs/manager.js
I saw exactly what happened, the manager.js works now great. Searching SO I found I had to place this in my /etc/rc.local. Also, I learned to point the output to a log file and to append the 2>&1 to "redirect stderr to stdout" and it should be a daemon so the last character is a &.
Finally, my /etc/rc.local looks like this:
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
su www-data -c 'node /var/www/php-jobs/manager.js >> /var/log/php-jobs.log 2>&1 &'
exit 0
If I run this myself (sudo /etc/rc.local): yes, it works! However, if I perform a reboot no node process is running, the /var/log/php-jobs.log does not exist and thus, the manager.js does not work. What is happening?
In this example of a rc.local script I use io redirection at the very first line of execution to my own log file:
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
exec 1>/tmp/rc.local.log 2>&1 # send stdout and stderr from rc.local to a log file
set -x # tell sh to display commands before execution
/opt/stuff/somefancy.error.script.sh
exit 0
On some linux's (Centos & RH, e.g.), /etc/rc.local is initially just a symbolic link to /etc/rc.d/rc.local. On those systems, if the symbolic link is broken, and /etc/rc.local is a separate file, then changes to /etc/rc.local won't get seen at bootup -- the boot process will run the version in /etc/rc.d. (They'll work if one runs /etc/rc.local manually, but won't be run at bootup.)
Sounds like on dimadima's system, they are separate files, but /etc/rc.d/rc.local calls /etc/rc.local
The symbolic link from /etc/rc.local to the 'real' one in /etc/rc.d can get lost if one moves rc.local to a backup directory and copies it back or creates it from scratch, not realizing the original one in /etc was just a symbolic link.
I ended up with upstart, which works fine.
In Ubuntu I noticed there are 2 files. The real one is /etc/init.d/rc.local; it seems the other /etc/rc.local is bogus?
Once I modified the correct one (/etc/init.d/rc.local) it did execute just as expected.
You might also have made it work by specifying the full path to node. Furthermore, when you want to run a shell command as a daemon you should close stdin by adding 1<&- before the &.
I had the same problem (on CentOS 7) and I fixed it by giving execute permissions to /etc/local:
chmod +x /etc/rc.local
if you are using linux on cloud, then usually you don't have chance to touch the real hardware using your hands. so you don't see the configuration interface when booting for the first time, and of course cannot configure it. As a result, the firstboot service will always be in the way to rc.local. The solution is to disable firstboot by doing:
sudo chkconfig firstboot off
if you are not sure why your rc.local does not run, you can always check from /etc/rc.d/rc file because this file will always run and call other subsystems (e.g. rc.local).
I got my script to work by editing /etc/rc.local then issuing the following 3 commands.
sudo mv /filename /etc/init.d/
sudo chmod +x /etc/init.d/filename
sudo update-rc.d filename defaults
Now the script works at boot.
I am using CentOS 7.
$ cd /etc/profile.d
$ vim yourstuffs.sh
Type the following into the yourstuffs.sh script.
type whatever you want here to execute
export LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64:$LD_LIBRARY_PATH
Save and reboot the OS.
I have used rc.local in the past. But I have learned from my experience that the most reliable way to run your script at the system boot time is is to use #reboot command in crontab. For example:
#reboot path_to_the_start_up_script.sh
This is most probably caused by a missing or incomplete PATH environment variable.
If you provide full absolute paths to your executables (su and node) it will work.
It is my understanding that if you place your script in a certain RUN Level, you should use ln -s to link the script to the level you want it to work in.
first make the script executable using
sudo chmod 755 /path/of/the/file.sh
now add the script in the rc.local
sh /path/of/the/file.sh
before exit 0
in the rc.local,
next make the rc.local to executable with
sudo chmod 755 /etc/rc.local
next to initialize the rc.local use
sudo /etc/init.d/rc.local start
this will initiate the rc.local
now reboot the system.
Done..
I found that because I was using a network-oriented command in my rc.local, sometimes it would fail. I fixed this by putting sleep 3 at the top of my script. I don't know why but it seems when the script is run the network interfaces aren't properly configured or something, and this just allows some time for the DHCP server or something. I don't fully understand but I suppose you could give it a try.
I had exactly same issue, the script was running fine locally but when I reboot/power-on it was not.
I resolved the issue by changing the file path. Basically need to give the complete path in the script. While running locally, file can be accessed but when running on reboot, local path will not be understood.
1 Do not recommend using root to run the apps such as node app.
Well you can do it but may catch more exceptions.
2 The rc.local normally runs as root user.
So if the your script should runs as another user such as www U should make sure the PATH and other environment is ok.
3 I find a easy way to run a service as a user:
sudo -u www -i /the/path/of/your/script
Please prefer the sudo manual~
-i [command]
The -i (simulate initial login) option runs the shell specified by the password database entry of the target user as a loginshell...
rc.local only runs on startup. If you reboot and want the script to execute, it needs to go into the rc.0 file starting with the K99 prefix.