I am having some difficulty running jobs over SSH. I have a series of networked machines which all have access to the same Home folder (when my executable is installed). While working on one machine I would like to be able run my code through ssh using the following sort of command:
ssh -q ExecutableDir/MyExecutable InputDir/MyInput
If I ssh in to any of the machines I wish to run the job on remotely and simply run:
ExecutableDir/MyExecutable InputDir/MyInput
It runs without fail, however when I run through SSH I get an error saying some shared libraries can't be found. Has anyone come across this sort of thing before?
ok I figured it out myself.
It seems when you run things through ssh in the way shown above you don't inherit the path variables etc. that you would if you ssh-ed in 'properly'. You can see this by running:
ssh RemoteMachine printenv
and comparing the output to what you would normally get if you were connected to the remote machine. The solution I then went for was to run something like the following:
ssh -q ExecutableDir/MyExecutable source ~/.bash_profile && InputDir/MyInput
Which then gets all the paths and stuff you might need from the bash_profile file on the remote machine
Related
I had a project where I need to run a command at EC2 reboot. I found only information about User Data but that works only at first launch which is not exactly what I needed. I need a command to run everytime I connect to the machine.
You can store a shell script in this directory:
/var/lib/cloud/scripts/per-boot/
It will be automatically run after every boot. (This is done by cloud-init, which also runs User Data scripts.)
After a little bit of research, I found a very easy solution. You can simply run.
echo "YOUR_COMMAND" >> .bashrc
This works also if you write it in the User Data section when launching your EC2 instance so that you don't have to SSH into it after launch.
Enjoy!
I have a script in which I build a mlagents_envs.environments.UnityEnvironment that successfully launches and works when I run the script from terminal sessions started on my ubuntu machine (that has a GUI). And if I ssh into the machine, I can run these scripts from tmux sessions that were originally created locally on my machine. If, however, I try to run the script from a terminal session created through the remote ssh connection, the script hangs when trying to create the UnityEnvironment. It just says:
Found path: <path_to_unity_executable>
and eventually times out.
I've tried to run the script with a virtual display and it still doesn't work. Specifically, I've tried:
$ xvfb-run --auto-servernum --server-args='-screen 1 640x480x24:64' python3 python_script.py -batchmode
$ xvfb-run --auto-servernum --server-args='-screen 1 640x480x24:64' python3 python_script.py
And I've tried the instructions found here: https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Training-on-Amazon-Web-Service.md
Has anyone encountered this issue? Do you have any suggestions?
The solution ended up being fairly simple. I just needed to specify the right DEVICE before running the script.
$ DEVICE=:1 python3 python_script.py
If anyone else runs into this, you might also need to enable X11 forwarding in both the ssh settings on the server and the client. I'm not 100% sure.
I apologize if this is a stupid questions; I am a complete newbie when it comes to cloud computation.
I am using Google Compute instances to run python scripts with GPU support. Unfortunately, it seems that for the script to run, my computer has to be on and the terminal connecting me to my instance must be open.
I am wondering if there is any way to run python scripts on instances in Google Cloud completely remotely, and just SSH in to see when the script is finished.
I have considered using IPython notebooks or something similar, but that code I am running requires a very specific Anaconda environment, and is meant to be run via terminal.
Edit 1:
The reason I think I need to have the console connecting me to the instance is because I tried to test it out by writing a small script to make files every minute. My process was as follows:
1. Create an instance, SSH in through the google cloud Instances page
2. Create a new python script with this code:
import time
i=0
while 1:
tmp_file = open("tst"+str(i)+".txt","w")
tmp_file.write(str(i))
tmp_file.close()
i += 1
time.sleep(60)
I then ran this code, confirmed it worked by SSHing in with a different console.
I closed the console with the program running in it. After that, files stopped being created.
Ideally, I would like a situation where I could run such a script, close out of the terminal window and have the execution of the script be unassociated with things like whether I have the console open or whether my device is on. I would like to just be able to SSH in and see the result of a script once it is finished.
I'm also a total novice when it comes to GCE and Python, so you're in good company! I had a similar problem when learning to use GCE. I opted to use a start-up script but I am not sure how well this will fit in with the environment that you need to set up. Mine uses a bash boot script and looks like something like this:
#! /bin/bash
sudo apt-get update
sudo apt-get -yq install python-pip
sudo pip install --upgrade google-cloud
sudo pip install --upgrade google-cloud-storage
sudo pip install --upgrade google-api-python-client
sudo pip install --upgrade google-auth-httplib2
echo "Making directories..."
mkdir -p /home/<username>/rawdata
mkdir -p /home/<username>/processeddata
mkdir -p /home/<username>/input
mkdir -p /home/<username>/code
mkdir -p /home/<username>/and so on
echo "Setting ownership..."
sudo chown -R <username> /home/<username>
echo "Directory creation complete..."
gsutil cp gs://<bucket with code>/* /home/<username>/code/
echo "Boot complete."
nohup python /home/<username>/code/workermaster.py &
The gist of the code (in case it isn't self-explanatory!) is that the instance installs various packages to support the code, but some of these might be on GCE instances by default. It then sets up required directories and copies all required code from a storage bucket and sets ownership. The key line I guess is the one containing "ho hangup" that starts the python script.
My "workermaster" script gets tasks from a Storage bucket and puts the output in one of the folders on the instance. I can see what the instance is doing from the console by looking at the logs (so without SSH-ing into the instance). My instances also copy the output to an output storage bucket. If I SSH into the instance I cannot see the script running, I can just see files 'mysteriously' appearing in the output folder.
There are plenty of experts on here that might be able to post a solution that more specific to your needs, but something like the above has worked for me so far. Good luck!
Not sure why you are saying you have to keep the terminal connected to your compute instance. Some more details will be helpful. Are you manually SSHing into your instance thru terminal and running the script? Is that how you want to do it in future?
If you are running your script periodically, you can set it up as cron job.
You may also want to look at Cloud Functions to go serverless.
You can use programs like tmux.
# ssh to the system
ssh user#system-blah-blah
# start a new tmux session
tmux new -s my_remote_session
# detach from session (keyboard shortcut, in sequence)
<ctrl-b> d
# attach to it back
tmux a -t my_remote_session
To be able to let a script running and close the terminal window, you can use a screen session, this is going to let the script running (inside the screen session) and if you close the terminal, it is going to continue working, after that, you can open again the terminal and connect to the screen session to see the results.
Other option is to use ansible, it helps to run commands inside the VM without connecting to it, but you must create a SSH-key in order to be able to connect with ansible.
I'm trying to add a build step in jenkins to copy files from my build server to my web application server. I've got the following command working in the command prompt
sudo scp -r /var/lib/jenkins/workspace/demoproj/publish root#0.0.0.0:/usr/temp
but when I run this command, it prompts me for a password every time. I found out about sshpass, but when I run this command...
sudo sshpass -p "passwordhere" scp -r /var/lib/jenkins/workspace/demoproj/pub root#0.0.0.0:/usr/temp
the terminal gets stuck. And never makes it through.
My main problem is if I add the first command to a build step in jenkins, it won't be able to pass the password over. How can I either supply the password in jenkins, or modify the command to pass over my credentials?
Helpful information: I'm using Putty on Windows 10 to connect to my
Ubuntu 16.04.3 LTS x64 servers from another Ubuntu 16.04.3 server.
First, sshpass needs to be installed on both the systems that is, the one running your jenkins instance as well as the one you are trying to access that is: root#0.0.0.0. You can verify it by doing 'which sshpass' or 'whereis sshpass'. If its not installed even in one of them then you need to install it first.
Also, Have you ever tried doing a ssh to the said machine: root#0.0.0.0 from the system where you have your jenkins instance? If not then there might not be an entry in the 'known-hosts' of either system. for that you can do ssh with '-o StrictHostKeyChecking=no' option to make an automatic entry in known-hosts.
Alternatively, if you dont want to enter password again and again you should work with 'keys'. Generate a unique key for both the systems and do an scp or ssh with -i option.
You should use jenkins credentials instead of using sensitive passwords directly into the scripts. Put the whole scp or ssh part inside a block which looks like: withCredentials(){}.
What's the point of having CI if you are required to be nearby to enter password every time? Install "publish over ssh" plugin, it has a step to send stuff over ssh.
https://wiki.jenkins.io/display/JENKINS/Publish+Over+SSH+Plugin
Look at "Use SSH during a build" section, you can use "send files or execute commands over SSH" build step. This shall become available after plugin installation.
Installed msysGit, wrote the code to start ssh-agent in .profile, everything worked.
Installed cygwin, without Git, just ssh and cURL.
SSH Agent no longer starting when Git bash opens.
I can start a new ssh-agent process instance, I see it when running ps in the Git bash, but when trying to use ssh-add, I get this error:
Could not open a connection to your authentication agent.
With cygwin, lots of .profile and .bashrc files were created in it's install folder (C:\cygwin). Not sure if this is the issue.
How can I fix this, please?
Take a look at my answer posted here for the ssh-add issue. Hopefully, this solution should work in your scenario too.