I apologize if this is a stupid questions; I am a complete newbie when it comes to cloud computation.
I am using Google Compute instances to run python scripts with GPU support. Unfortunately, it seems that for the script to run, my computer has to be on and the terminal connecting me to my instance must be open.
I am wondering if there is any way to run python scripts on instances in Google Cloud completely remotely, and just SSH in to see when the script is finished.
I have considered using IPython notebooks or something similar, but that code I am running requires a very specific Anaconda environment, and is meant to be run via terminal.
Edit 1:
The reason I think I need to have the console connecting me to the instance is because I tried to test it out by writing a small script to make files every minute. My process was as follows:
1. Create an instance, SSH in through the google cloud Instances page
2. Create a new python script with this code:
import time
i=0
while 1:
tmp_file = open("tst"+str(i)+".txt","w")
tmp_file.write(str(i))
tmp_file.close()
i += 1
time.sleep(60)
I then ran this code, confirmed it worked by SSHing in with a different console.
I closed the console with the program running in it. After that, files stopped being created.
Ideally, I would like a situation where I could run such a script, close out of the terminal window and have the execution of the script be unassociated with things like whether I have the console open or whether my device is on. I would like to just be able to SSH in and see the result of a script once it is finished.
I'm also a total novice when it comes to GCE and Python, so you're in good company! I had a similar problem when learning to use GCE. I opted to use a start-up script but I am not sure how well this will fit in with the environment that you need to set up. Mine uses a bash boot script and looks like something like this:
#! /bin/bash
sudo apt-get update
sudo apt-get -yq install python-pip
sudo pip install --upgrade google-cloud
sudo pip install --upgrade google-cloud-storage
sudo pip install --upgrade google-api-python-client
sudo pip install --upgrade google-auth-httplib2
echo "Making directories..."
mkdir -p /home/<username>/rawdata
mkdir -p /home/<username>/processeddata
mkdir -p /home/<username>/input
mkdir -p /home/<username>/code
mkdir -p /home/<username>/and so on
echo "Setting ownership..."
sudo chown -R <username> /home/<username>
echo "Directory creation complete..."
gsutil cp gs://<bucket with code>/* /home/<username>/code/
echo "Boot complete."
nohup python /home/<username>/code/workermaster.py &
The gist of the code (in case it isn't self-explanatory!) is that the instance installs various packages to support the code, but some of these might be on GCE instances by default. It then sets up required directories and copies all required code from a storage bucket and sets ownership. The key line I guess is the one containing "ho hangup" that starts the python script.
My "workermaster" script gets tasks from a Storage bucket and puts the output in one of the folders on the instance. I can see what the instance is doing from the console by looking at the logs (so without SSH-ing into the instance). My instances also copy the output to an output storage bucket. If I SSH into the instance I cannot see the script running, I can just see files 'mysteriously' appearing in the output folder.
There are plenty of experts on here that might be able to post a solution that more specific to your needs, but something like the above has worked for me so far. Good luck!
Not sure why you are saying you have to keep the terminal connected to your compute instance. Some more details will be helpful. Are you manually SSHing into your instance thru terminal and running the script? Is that how you want to do it in future?
If you are running your script periodically, you can set it up as cron job.
You may also want to look at Cloud Functions to go serverless.
You can use programs like tmux.
# ssh to the system
ssh user#system-blah-blah
# start a new tmux session
tmux new -s my_remote_session
# detach from session (keyboard shortcut, in sequence)
<ctrl-b> d
# attach to it back
tmux a -t my_remote_session
To be able to let a script running and close the terminal window, you can use a screen session, this is going to let the script running (inside the screen session) and if you close the terminal, it is going to continue working, after that, you can open again the terminal and connect to the screen session to see the results.
Other option is to use ansible, it helps to run commands inside the VM without connecting to it, but you must create a SSH-key in order to be able to connect with ansible.
I have a Raspberry Pi running Debian Lite (terminal only, no desktop). I've installed Node.js and am currently running a script by making an SSH connection through puTTY.
The problem is, when I close the puTTY window the script stops running. I need it to run when my desktop is turned off, otherwise having it on the Pi is pointless.
So far I have tried:
-Nohup | Appended the output but still stopped running when I closed the terminal.
Multiple options:
Install screen or tmux on the PI and start the job with those programs.
(or) Run the job with the nohup command:
nohup command
(or) Run the job in the background and use disown:
command &
disown %1
I recommend option 1.
IM running a VPS server on Ubuntu 14.04 minimal x86. I connect to it using putty via SSH.
On the server i have a simple script that is starting a few instances of bots
nohup node /nodebots/bot10/server.js &
I use root as a user, so all the privileges and +chmod X are set properly (in my opinion)
The idea is that my node.js program is not excelent + the service im running the bots for sometimes has reboots, server crashes etc.
Ive installed crontab and at this moment struggling to set the script to be run at boot.
Ive used various solutions including trying to boot it via init.d , adding a line to rc.local and well using crontab -e. None of it helps.
Currently the code in crontab looked the following ways
`#reboot sh /nodebots/botsrun.sh`
#reboot root /nodebots/sh botsrun.sh
#reboot cd /nodebots/ && sh botsrun.sh
None of it helped.
Im new to Ubuntu, coding and even terminal commands. I would really apreaciate any kind of help. I will be more then grateful for a step by step tutorial on what im doing wrong and what should be done.
I am having some difficulty running jobs over SSH. I have a series of networked machines which all have access to the same Home folder (when my executable is installed). While working on one machine I would like to be able run my code through ssh using the following sort of command:
ssh -q ExecutableDir/MyExecutable InputDir/MyInput
If I ssh in to any of the machines I wish to run the job on remotely and simply run:
ExecutableDir/MyExecutable InputDir/MyInput
It runs without fail, however when I run through SSH I get an error saying some shared libraries can't be found. Has anyone come across this sort of thing before?
ok I figured it out myself.
It seems when you run things through ssh in the way shown above you don't inherit the path variables etc. that you would if you ssh-ed in 'properly'. You can see this by running:
ssh RemoteMachine printenv
and comparing the output to what you would normally get if you were connected to the remote machine. The solution I then went for was to run something like the following:
ssh -q ExecutableDir/MyExecutable source ~/.bash_profile && InputDir/MyInput
Which then gets all the paths and stuff you might need from the bash_profile file on the remote machine
I started Sahi using putty but I got the following error during running scripts:
Script didn't start 150 ms.
I ran both Sahi and script through putty. It's a point to note that running through Linux terminal it's working fine. Please help me regarding this issue.
You might want to check if there's any difference between the environments (env) when running in a terminal and when running in Putty. Also, if those sahi scripts are bash, you can run it with bash -x to see what's going on inside.