I am running a big julia simulation in AWS machine (linux). I use putty to run from the command line.
I type: julia myscript.jl
However, whenever I close the command prompt, or putty or my laptop, the run on the AWS server will stop. I wonder if anyone knows is there a way to keep julia running when I close the command line prompt. As I do not want to keep my laptop open and putty stay connected for several days
Thanks
dtach exists for this purpose. It will keep a process alive even if the terminal session that created the process is closed.
Once you log in through ssh, start a new dtach session:
$ dtach -A /tmp/my-dtach-session julia myscript.jl
Then detach from the session with ctrl+\.
Detaching will not kill your process. Here I check that it is still running after detaching:
[david#blue ~] $ ps aux | grep dtach
david 506 0.0 0.0 8460 1484 ? Ss 16:15 0:00 dtach -A /tmp/my-dtach-session ./pkg/julia-1.3.0/bin/julia
david 517 0.0 0.0 6140 2224 pts/2 S+ 16:16 0:00 grep dtach
After you detach, you can close your ssh session like normal. If your process has not finished, you can log back in and reattach:
$ dtach -a /tmp/my-dtach-session
Related
I have a script on my home theater PC running Ubuntu 22. The purpose of this script is to check
When was mouse/keyboard activity last seen?
Is there active audio playing (music or video)
and then it determines if the htpc is idle or not, and relays this information back to my home automation server.
I have a "helper" script that fires this up in a tmux session (mainly so I can attach and debug since I have it dump output to the terminal) at boot.
If I run this script as the main user (htpc), then it works great. When it launches from the crontab, pulse audio states that there is no pulse audio daemon running. Checking ps shows that both run as htpc.
Output of running directly from a terminal (I'm SSH'd in)
htpc#htpc:~/Documents$ ps aux | grep -i idle
htpc 8397 0.0 0.0 11308 3736 ? Ss 00:03 0:00 tmux new-session -d -s idle-script exec /home/htpc/Documents/report_idle_to_hass.sh
htpc 8398 0.0 0.0 10100 4012 pts/2 Ss+ 00:03 0:00 /bin/bash /home/htpc/Documents/report_idle_to_hass.sh
htpc 8455 0.0 0.0 9208 2180 pts/3 S+ 00:03 0:00 grep --color=auto -i idle
Output of running from a crontab job:
htpc#htpc:~/Documents$ ps aux | grep -i idle
htpc 6720 0.0 0.0 11304 3604 ? Ss 23:57 0:00 tmux new-session -d -s idle-script exec /home/htpc/Documents/report_idle_to_hass.sh
htpc 6721 0.0 0.0 10100 3988 pts/2 Ss+ 23:57 0:00 /bin/bash /home/htpc/Documents/report_idle_to_hass.sh
htpc 6748 0.0 0.0 9208 2416 pts/3 S+ 23:57 0:00 grep --color=auto -i idle
Visually, I don't see why one works and another doesn't.
tmux output of a manually started (me via ssh) instance:
Counter overflowed: No
[][]
Counter overflowed: No
[][]
Note that the [][] simply is a response from my home automation server. It's an empty response because the state did not change. It will populate with previous state information if there was a change.
tmux output of a crontab-started instance:
Counter overflowed: Yes
[][]
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
Counter overflowed: Yes
[][]
No PulseAudio daemon running, or not running as session daemon.
No PulseAudio daemon running, or not running as session daemon.
Crontab entry (running crontab -e):
#reboot exec /home/htpc/Documents/start_idle_report.sh
I'll give the script giving the error first, the only part that doesn't work is the pacmd command. It does successfully grab the window name and keyboard/mouse last input time. Only pacmd has issues.
check_idle.sh:
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export DISPLAY=:0.0
# Set the timeout value to consider device idle
time_to_be_idle_ms=60000
audio_check="`exec pacmd list-sink-inputs | grep \"state: RUNNING\"`"
idle_time="`exec xprintidle`"
current_app="`exec xdotool getwindowfocus getwindowname`"
if [ -z "$audio_check" ]
then
if [ "$idle_time" -gt "$time_to_be_idle_ms" ]
then
if [[ "$current_app" == *"Frigate"* ]]
then
# No audio, idle time, but cameras are in focus
echo "No"
else
# No audio, idle time, and cameras aren't in focus
echo "Yes"
fi
else
# No audio playing, but idle time not met
echo "No"
fi
else
# Playing audio
echo "No"
fi
start_idle_report.sh:
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Kick off the report script
if [[ "`ps aux | grep -i report_idle_to_hass | wc -l`" -eq "1" ]];
then
tmux new-session -d -s "idle-script" "exec /home/htpc/Documents/report_idle_to_hass.sh"
fi
report_idle_to_hass.sh (some info redacted):
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
submit_state() {
# This function sends the desired output to my server
}
submit_state "No"
last_state="No"
counter=0
while [ true ]
do
idle_check="`/home/htpc/Documents/check_idle.sh`"
if [[ "$idle_check" != "$last_state" ]]
then
echo "States changed: $idle_check"
submit_state "$idle_check"
last_state="$idle_check"
counter=0
else
counter=$((counter+1))
fi
if [ "$counter" -gt 10 ]
then
echo "Counter overflowed: $last_state"
submit_state "$last_state"
counter=0
fi
sleep 2
done
I looked up how pulse figures out what to connect to: https://www.freedesktop.org/wiki/Software/PulseAudio/FAQ/
It lists several environment variables. When I look with printenv, there are no display related or pulse related variables set.
I do not have a ~/.pulse folder to contain the config, and the default pulse file in /etc/pulse is empty (nothing but comments in there).
How is it finding a pulse server when I run it but not in crontab?
I thought maybe it was a path issue (even though I define the paths in the script)
htpc#htpc:~/Documents$ which pacmd
/usr/bin/pacmd
I have tried it with the 'exec' removed and added. Tried 'exec' when I found another question on stack overflow that suggested using exec because it was firing off the script with a 'sh -c' instead, and I figured this might be the problem, but it did not fix it.
I expected the behavior to be identical to me sshing in and simply typing the tmux command myself, but for some reason when started via a crontab job it does not work.
Feels a lot like the commands are being executed as a different user, or some variable issue.
If your issue is that it is running as cron and not as you, then you need to place your cron entries in a user-specific crontab under /var/spool/cron/crontabs (Ubuntu MATE 20.04), not in the main crontab.
I am accessing a server running CentOS (linux distribution) with an SSH connection.
Since I can't always stay logged in, I use "nohup [command] &" to run my programs.
I couldn't find how to get a list of all the programs I started using nohup.
"jobs" only works out before I log out. After that, if I log back again, the jobs command shows me nothing, but I can see in my log files that my programs are still running.
Is there a way to get a list of all the programs that I started using "nohup" ?
When I started with $ nohup storm dev-zookeper ,
METHOD1 : using jobs,
prayagupd#prayagupd:/home/vmfest# jobs -l
[1]+ 11129 Running nohup ~/bin/storm/bin/storm dev-zookeeper &
NOTE: jobs shows nohup processes only on the same terminal session where nohup was started. If you close the terminal session or try on new session it won't show the nohup processes. Prefer METHOD2
METHOD2 : using ps command.
$ ps xw
PID TTY STAT TIME COMMAND
1031 tty1 Ss+ 0:00 /sbin/getty -8 38400 tty1
10582 ? S 0:01 [kworker/0:0]
10826 ? Sl 0:18 java -server -Dstorm.options= -Dstorm.home=/root/bin/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dsto
10853 ? Ss 0:00 sshd: vmfest [priv]
TTY column with ? => nohup running programs.
Description
TTY column = the terminal associated with the process
STAT column = state of a process
S = interruptible sleep (waiting for an event to complete)
l = is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
Reference
$ man ps # then search /PROCESS STATE CODES
Instead of nohup, you should use screen. It achieves the same result - your commands are running "detached". However, you can resume screen sessions and get back into their "hidden" terminal and see recent progress inside that terminal.
screen has a lot of options. Most often I use these:
To start first screen session or to take over of most recent detached one:
screen -Rd
To detach from current session: Ctrl+ACtrl+D
You can also start multiple screens - read the docs.
If you have standart output redirect to "nohup.out" just see who use this file
lsof | grep nohup.out
You cannot exactly get a list of commands started with nohup but you can see them along with your other processes by using the command ps x. Commands started with nohup will have a question mark in the TTY column.
You can also just use the top command and your user ID will indicate the jobs running and the their times.
$ top
(this will show all running jobs)
$ top -U [user ID]
(This will show jobs that are specific for the user ID)
sudo lsof | grep nohup.out | awk '{print $2}' | sort -u | while read i; do ps -o args= $i; done
returns all processes that use the nohup.out file
I have a php script which is responsible for sending emails based on a queue contained in a database.
The script works when it is executed from my shell as such:
/usr/bin/php -f /folder/email.php
However, when I execute it to run in the background:
/usr/bin/php -f /folder/email.php > /dev/null &
It never completes, and the process just sits in the process queue:
clickonce: ps T
PID TT STAT TIME COMMAND
1246 s000 Ss 0:00.03 login -pf
1247 s000 S 0:00.03 -bash
1587 s000 T 0:00.05 /usr/bin/php -f /folder/email.php
1589 s000 R+ 0:00.00 ps T
So my question is how can I run this as a background process and have it actually execute? Do I need to configure my OS? Do I need to change the way I execute the command?
"T" in the "STAT" column indicates a stopped process. I would guess that your script is attempting to read input from stdin and is getting stopped because it is not the foreground process and thus is not allowed to read.
You should check if the script does indeed read something while executing.
Here's what's happening to me: I start tmux sessions using tmux -L name1, tmux -L name2; then I detatch them using ctrl+B+d. Then I try to get a list of the currently running sessions on my computer. However, when I run tmux ls, I get an error message:
failed to connect to server: Connection refused
Is this a bug? I'm familiar with screen; I regard screen -ls as a very useful function since I might start a session and leave it running for weeks before the next time I attach to it. Because of this, the ability to list current running tmux sessions is quite important for me. Why does tmux ls return a "connection refused" error when I know tmux is running?
TL;DR: Try sending SIGUSR1 signal to the tmux server process.
In my case, after about 8 days of inactivity, I was not able to reattach:
$ tmux attach
no sessions
However, a grep for tmux process got me this output:
$ ps -aef | fgrep -i tmux
hari 7139 1 1 2016 ? 2-20:32:31 tmux
hari 25943 25113 0 22:00 pts/0 00:00:00 fgrep --color=auto -i tmux
As suggested by #7heo.tk, this indicates that tmux server is still running, but tmux ls was giving failed to connect to server: Connection refused error. I verified that the tmp directory that belonged to the tmux session existed and lsof -p 7139 (the pid of tmux server) showed that the socket file is open:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tmux 7139 hari 5u unix 0x0000000000000000 0t0 1712879255 /tmp/tmux-50440/default
I also tried explicitly specifying the -S /tmp/tmux-50440/default to tmux but it didn't help. However, I read in the tmux man page that sending SIGUSR1 would make tmux recreate the socket file, so I tried that and I was able to immediately find the session and reattach:
$ kill -s USR1 7139
$ tmux ls
0: 12 windows (created Mon Apr 18 21:17:55 2016) [198x62]
This happens to me when I do not have any sessions running. I'm just starting to use tmux and didn't realize that if you restart your computer you lose your sessions which surprised me at first.
For those of you who are thinking the same thing: Restore tmux session after reboot. A summary of the post: Use shell scripts to build your tmux sessions or create a fancy shell history tracker.
This happened to me when the Ubuntu desktop crashed and my gnome-terminal windows exited. I could still see the tmux process was running (ps aux | grep tmux) but for some reason tmux commands would not work to list the existing sessions. Apparently it wasn't finding the existing Unix socket of the still-running tmux process. The fix in this scenario is to locate the existing Unix socket and specify that to tmux using the -S flag; here's how:
You can find the PID of your still-running tmux process with this:
ps -p $(pidof tmux)
Now take your PID (in my case, 6876) and run this to list any open Unix sockets:
sudo lsof -Uap 6876
Hopefully you see output like this:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tmux 6876 abe 3u unix 0x0000000000000000 0t0 408477 socket
tmux 6876 abe 4u unix 0x0000000000000000 0t0 408478 socket
tmux 6876 abe 6u unix 0x0000000000000000 0t0 408479 /tmp/tmux-1000/default
Now you can specify that existing Unix socket to your tmux command (using the -S flag), and you should be able to list-sessions and attach properly:
tmux -S /tmp/tmux-1000/default list-sessions
tmux -S /tmp/tmux-1000/default attach -t 0
You get this error indeed if there are no session open. If there are no sessions open there is no tmux server running so it can't connect to it.
With the -L option, you change the socket name the tmux server uses, it's not a way to name your sessions. You better use the following commands:
tmux new -s name1
tmux new -s name2
These will create 2 sessions on a server with the default socket name. Now you can do:
$ tmux ls
name1: 1 windows (created Mon Sep 22 10:34:40 2014) [158x40] (attached)
name2: 1 windows (created Mon Sep 22 10:34:43 2014) [158x40] (attached)
And you see all the sessions running on the server on the default socket. You can reattach one of them using:
tmux attach -d -s name1
-s specifies the name of the session
-d will detach it from it's previous client (if it is attached)
You can also switch between sessions inside tmux with the choose-tree command which by default is assigned to the keystroke C-s (prefix key + s). This is what I usually do.
You may have an error in your .tmux.conf. I had this problem until i took out this line from my .tmux.conf:
set-window-option -g xterm-keys on
You could also try tmux -v and then look at the logs it prints.
One easy fix is to delete the tmp files left by the tmux server, for example, by doing $ rm -rf /tmp/tmux-xxx/.
The way TMUX(1) works is by having a client process (tmux) connect to a server process (tmux too, but not attached to a TTY), as shown in the following ps output:
PID TTY STAT TIME COMMAND
19229 pts/1 S+ 0:00 tmux
19231 ? Ss 0:00 tmux
That shows that the client actually starts before the server (one could assume it forks it).
After detach/re-attach, the same ps command outputs:
PID TTY STAT TIME COMMAND
19231 ? Ss 0:00 tmux
19290 pts/1 S+ 0:00 tmux attach
This shows the tmux client as tmux attach, thus being a bit easier to understand.
Now, if we look at the output of pstree in both of the above cases, we get in both cases (ignoring the pid change for tmux attach):
pstree -p
init(1)─┬─acpid(1824)
├─cron(1859)
⋮
├─sh(14146)───tmux(19229)
└─tmux(19231)───sh(19233)───pstree(19234)
Clearly showing that the commands typed (pstree in this case) in the client process (PID 19229) get executed by the server one (PID 19231), thus allowing them to continue without SIGHUP in the event where the client terminal is lost (over ssh, for example).
Now, to the question OP asked: what happens in the case where tmux returns failed to connect to server: Connection refused is that the server process (pid 19231 in our case) is unreachable, whatever the reason (it can be because the server process died; but also because the user executing the tmux client doesn't have the permissions to access the tmux socket, etc.)
The solution in that case is to grep for the tmux processes (via ps for example), and pray that you didn't get this error because the server died (so you can attach to it by using lsof to get what sockets it listens to). Otherwise, there is no way to attach to the server, as it is as dead as after a reboot.
TL;DR:
This error can be given for multiple reasons, ranging from bug to critical failure (program died). In a nutshell, use the UNIX tools at your disposal to determine what socket does tmux use, if it is still running (there should be at least two processes if you have the tmux client running - that happens after invoking tmux or tmux attach from the shell) and thus if you lost your session or not.
Note: as other answers pointed out, if the reason for this error to be shown is a socket error, you can use the -L flag to tell tmux to use a specific socket.
This may happen if you or any cleaning process delete files in /tmp/*. All your sessions data are lost if you can't recover those files. Killing all tmux instances and restart it is the only choice left, unfortunately.
I was using another program inside of tmux (reattach-to-user-namespace), and I was getting this error when I switched computers because reattach-to-user-namespace was not installed. The fix was to simply run brew install reattach-to-user-namespace.
This happened to me because I ran tmux with another user (was root), and I tried to list sessions with my normal user..
So you might want to check the user with which you ran your tmux first.
To do so:
$ ps -aef | fgrep -i tmux
root 7139 1 1 2016 ? 2-20:32:31 tmux
centos 25943 25113 0 22:00 pts/0 00:00:00 fgrep --color=auto -i tmux
See the username in the first column: here it's root
Try tmux -L name1 list-session.
In my putty terminal, i typed the command as follows:
[username#vm186 bin]$ nohup ./mongod --dbpath ~/mongodb-data/ &
[1] 5967
[username#vm186 bin]$ nohup: appending output to `nohup.out'
then, ps showed the nohup is apparently invalid !!
[username#vm186 bin]$ ps -auxw | grep mongo
username 5967 0.0 0.0 76172 4716 pts/8 Sl 10:03 0:00 ./mongod --dbpath /home/username/mongodb-data/
username 6140 0.0 0.0 61192 780 pts/8 S+ 10:04 0:00 grep mongo
So, when i close the window, mongod will receive the signal and quit.
What's wrong with my command? or something wrong with my putty configuration?
On my system (FreeBSD) nohup won't show with ps, but the program it starts will show, and will survive closing putty. Did your program exit after closing putty?
Nohup is not supposed to continue running. It just redirects standard output and standard error, ignores SIGHUP, and executes the program you requested. The requested process totally replaces nohup but inherits the file descriptors and SIGHUP ignoring. That's what prevents the process from terminating when you log out. For more information, look at the source. You're probably using nohup from coreutils.