I have a GPU cluster which I can access by ssh-ing into an intermediate computer, and then ssh-ing into the cluster. (The intermediate computer resides on the same network as the cluster, but the cluster is not connected to the Internet)
I prefer not to use in-terminal editors like vim and would also like to avoid copying files to the intermediate computer and then onto the cluster, so I am looking for a way to remotely edit files on the cluster with an IDE on my local computer.
If I were directly ssh-ed into the cluster, I could use SFTP to edit the files on the cluster.
However, there is an intermediate computer. Is there someway I can use SFTP or a similar technology to achieve my goal?
Thanks for any help and let me know if I can be more clear.
You are looking for an SFTP connection to your cluster through a SSH connection to your gateway - so for SSH tunneling.
Use on your local machine
ssh gateway -L localPort:cluster:portToCluster -l user -p portToGateway
to open a SSH tunnel to your cluster. For localPort use any free port on your local machine (e.g. 2222), portToCluster is the port you usually use for connecting to your cluster from the gateway (standard port is 22) and portToGateway is the port you use for connecting to the gateway from your local machine.
Now you can access the cluster simply using a SFTP connection to localhost:2222
Your SSH tunnel must be open as long as you wish to use this SFTP connection. So keep your SSH connection running.
Related
I would like to set up an apache spark Standalone cluster using two different computers (windows 10) of the same wifi network as slaves.
I can successfully set up the master using Windows bash(linux subsystem) and connect a slave from the same machine.
The problem is when I'm trying to connect a slave from a different machine (same wifi) using the local ip of the master. I even temporarily opened up all the ports of both the machines and mamaged to view the spark ui from the slave machine using the local ip of the master. But still trying to set up a slave on master-local-ip:7077 didn't work.
I believe that the correct way is to use password-less ssh between them but I'm not sure how to configure it for local ip addresses using the windows bash. I tried to create the key and when I executed ssh-copy-id the connection refused.
Any suggestions?
I know you can set up remote connection in Netbeans or as others have suggested, mount the remote file system locally and let Netbeans read that.
This current set-up I have to SSH into a remote server and then SSH again into another one.
Is it possible in Netbeans to access the filesystem on the 2nd server hop?
I think you can setup a SSH local port forwarding on your first remote server to archive what you want here. Once ssh connection established with the first server, it will be tunnel to second server. Also, will be much more convenient if using SSH public key as authentication method.
Run below command on your local (Need to keep this running)
ssh -NL 2222:server2.example.com:22 server1.example.com
Test connection with ssh locally
ssh localhost 2222
So in your case, Netbeans just need to connect to localhost port 2222.
Ref: https://www.ssh.com/ssh/tunneling/example
That isn't answer you expect, but to big to put it in comment.
I think best solution is configure autostart some service which will create tunnel to first server on your login and bind local port (let's say 2222) to second remote server ssh port. This is basical feature of ssh client. When you can configure Netbeans to connect to localhost:2222 and login right into second server. You might need to use keys authentication for first server.
Where are examples to make such thing. Like is underneath link on article
http://www.pc-freak.net/blog/start-ssh-tunnel-pc-boots-windows-alwaysup/
Since you can get ssh access via a daisy-chain, you can very likely use SSHFS:
https://github.com/libfuse/sshfs
Available in Debian (and therefore, likely many other distributions):
https://packages.debian.org/jessie/sshfs
As long as you have ssh keys set up properly on each server, you can daisy-chain mounting via SSHFS to the machine where your NetBeans app is running.
https://en.wikipedia.org/wiki/SSHFS
NOTE: There are some caveats, including that SSHFS expects that it is the only mechanism used to update the filesystem, and that there is only one writer at a time. Undesirable behavior can occur if you allow other mechanisms to change the files or directories on the target server, including data loss and/or corruption.
We're migrating from a Windows server box to Virtual Machines on Azure. Our SQL stored procedure uses Integration Services packages to generate feed files and upload them to client ftp servers.
We do this by building script files and using the native ftp.exe client on Windows. My script file looks like this:
open ftp.ourserver.net
myusername
mypassword123
lcd G:\
cd /
prompt
put "G:\DataFeeds\SmithCo_02272017.txt" "SmithCo_DailyFeed.txt"
bye
This works fine on our old server. When I use command prompt to run this code I'm getting:
421 Active data channel timed out.
I can upload the file manually with just a basic connection using WinSCP client on that machine.
I thought it was a port issue so I created an outbound rule in Windows Firewall for port 21, allowed all connection.
Then I created an endpoint in Azure for port 21 as well.
Is there another step to open this up or something I may be missing?
The Windows command-line ftp.exe supports the FTP active mode only. In the active mode, the server has to connect back to the client to establish data connection for a file transfer. More over the ftp.exe does not allow you to pick, what port is used for the data connection. It randomly picks any available port number above 1024. So you would have to create end points for all ports above 1024. That's possibly something you do not want to do. Though you can try.
WinSCP works, because it (as most modern FTP clients) uses the passive mode by default. For the passive mode, you do not need to open any ports on the client side.
Read my article about network configuration for FTP active and passive connection modes for details.
As WinSCP works for you, you can use WinSCP scripting interface to replace your ftp.exe script.
It would be like (ftp.txt):
open ftp://myusername:mypassword123#ftp.ourserver.net/
lcd G:\
cd /
put "G:\DataFeeds\SmithCo_02272017.txt" "SmithCo_DailyFeed.txt"
exit
(Though the lcd is redundant, both in your and WinSCP script).
And run the script like:
winscp.com /script=ftp.txt
See the guide to converting Windows FTP script to WinSCP script.
(I'm the author of WinSCP)
I have to occasionally work remotely. In order to do so, I have to connect to a gateway server, through which I can then connect to the development server. At work I can connect directly to the dev server. I use SSHFS to map the remote folder to a local one (in Ubuntu). My colleagues don't seem to mind using vim for all of their work, but I really prefer and IDE. I know that using just ssh I can tunnel through multiple connections (ssh -t server1 ssh -t server2), but I'd like to do the same with SSHFS. Does anyone know how this could be accomplished?
Yes, it can be done. For this, the remote server has to support local port forwarding (which might be disabled for security reasons). What you bascially do, you instruct the remote server to open a tunnel to a server in the other network for you:
ssh -fL 127.0.0.1:someport:host.in.the.remote.net:22 proxy.host
someport should be an unused port on your machine (for example 2222), host.in.the.remote.net should be the hostname or IP of the development machine you eventually want to connect to, from within the network of the proxy.host, which is the intermediate server you have to go through. -f instructs ssh to detach from the terminal after the connection is established.
You run this command, and after it is in the background, the remote machine's ssh port can be referred to as 127.0.0.1:someport and can be used as such by sshfs. I don't know of a way to automate this though, but you should easily be able to script it.
Make sure you clear this action with the remote administrator beforehands if you're unsure about the policies. You may need to change the 22 in the first command if the development machine serves ssh on a different port.
I have the following setup:
Machine 1: hosts a given svn repository and lives under a private network
Server: lives under the same network as Machine 1 (therefore having access to it), but is also open to outside ssh connections.
Machine 2: my personal computer which (when not under the same private network) can only access Server.
I would like to access Machine 1 directly so that I can interact with the repository that it hosts using subversive on Eclipse. I can ssh to Server and from there ssh to Machine 1 via terminal, but so far I haven't been able to figure out a way to do that on subversive.
Any suggestion is much appreciated!
Ps.: All systems run linux
Consider using stunnel for your tunneling in such cases.
It provides a proxy to connect to your host by localhost itself.
It works on linux very well
PS: we have a similar architecture, and we use the same.