I want to transfer files between two servers , files size is aproximately 170GB.
On one server , there is Direct Admin control panel, and on the other one is Cpanel .
I've ftp & ssh access on both servers. I know about scp command on ssh, but as I've tried it and I didn't succeed , I prefer to use ftp commands. Because there were some connection or other errors on ssh , so the transfer progress was stopping and I couldn't resume the progress by skipping already uploaded files. So what should I do?
You can use rsync, it will continue where it stopped.
Go to one of the servers and do:
rsync -avz other.server.com:/path/to/directory /where/to/save
You can omit z option if the data is not compressible.
This is with assumption that the user name on both servers is the same.
If not you will need to add -e 'ssh -l login_name' to the above command.
Related
I'm trying to get a Ubuntu server to periodically (preferably whenever it gets updated, if possible) to copy a file remotely from an FTP server to a directory on the Ubuntu server. I should note I'm not very advanced with this kind of stuff.
I of course am not doing this without a tutorial, however it doesn't cover grabbing the file from an ftp.
What would be simplest for me is to be able to run:
tail -F ftp://ftp.addr.ess/files/file-i-want.txt | grep --line-buffered": <" | while read x ; do echo -ne $x | curl -X POST -d #- http://url/hook ; done
What I'm following has that FTP address as a local address. This is a problem, because that command returns this:
tail: cannot open 'ftp://ftp.addr.ess/files/file-i-want.txt' for reading: No such file or directory
I've tried to run:
rsync username#ftp.addr.ess:XX/files/file-i-want.txt /home/ubuntu/destination
however this returns:
ssh: connect to host ftp.addr.ess port XX: Connection refused.
So really if I can get rsync to run FTP instead of SSH, I figure I'd be golden. I researched it though and I can't figure out how to do this (keep in mind I'm no programmer). I originally thought the error was because I wasn't giving it a password, because I didn't know how. It might be that also, though.
This however brings me to my next issue. If it's possible to make rsync do FTP instead of SSH, how would I make it periodically do that?
What is being updated? The remote file (my guess) or something on your server? If it's the remote file, you're out of luck unless there is mecanism/process on the remote server that can send you a notification (an email for example).
I've not used ftp for ages, but have a look at this as a starting point.
A periodic task can be quite easily configured with a cron task.
One of my tasks at work is to check the health/status of multiple Linux servers everyday. I'm thinking of a way to automate this task (without having to login to each server everyday). I'm a newbie system admin by the way. Initially, my idea was to setup a cron job that would run scripts and email the output. Unfortunately, it's not possible to send mail from the servers as of the moment.
I was thinking of running the command in parallel, but I don't know how. For example, how can I see output of df -h without logging in to servers one by one.
You can run ssh with the -t flag to open a ssh session, run a command and then close the session. But to get this fully automated you should automate the login process to every server so that you don't need to type the password for every server.
So to run df -hon a remote server and then close the session you would run ssh -t root#server.com "df -h". Then you can process that output however you want.
One way of automating this could be to write a bash script that runs this command for every server and process the output to check the health of the server.
For further information about the -t flag or how you can automate the login process for ssh.
https://www.novell.com/coolsolutions/tip/16747.html
https://serverfault.com/questions/241588/how-to-automate-ssh-login-with-password
You can use ssh tunnels or just simply ssh for this purpose. With ssh tunnel you can redirect the outputs to your machine, or as an alternative, you can run the ssh with the remote commands on your machine then get the ouput on your machine too.
Please check the following pages for further reading:
http://blog.trackets.com/2014/05/17/ssh-tunnel-local-and-remote-port-forwarding-explained-with-examples.html
https://www.google.hu/amp/s/www.cyberciti.biz/faq/unix-linux-execute-command-using-ssh/amp/
If you want to avoid manual login, use ssh keys.
Create a file /etc/sxx/hosts
populate like so:
[grp_ips]
1.1.1.1
2.2.2.2
3.3.3.3
share ssh key on all machines.
Install sxx from package:
https://github.com/ericcurtin/sxx/releases
Then run command like so:
sxx username#grp_ips "whatever bash command"
Can anyone please let me know if there are any ways that we can monitor a directory in centOS (linux) which is present in different server and when a new file arrives in that directory I need to copy that file to my server.
One way would be to have rync command running periodically on cron job. rysnc is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending
only the differences between the source files and the existing files in
the destination.
If you want to run transfer from remote server to local server like every 10 minutes you add a line like below in your crontab. It also does few other things:
Zip the files
Transfer over ssh
Logs output
Note: You will have to do ssh key exchange to avoid the password prompt.
*/10 * * * * /usr/bin/rsync -zrvh -e ssh root#<remoteIP>:<remoteDir> >/var/log/rsync_my.log 2>&1
Often I face this situation: I sshed into a remote server and ran some programs, and I want to copy their output files back to my local machine. What I do is remember the file path on remote machine, exit the connection, then scp user#remote:filepath .
Obviously this is not optimal. What I'm looking for is a way to let me scp file back to local machine without exiting the connection. I did some searching, almost all results are telling me how to do scp from my local machine, which I already know.
Is this possible? Better still, is it possible without needing to know the IP address of my local machine?
Given that you have an sshd running on your local machine, it's possible and you don't need to know your outgoing IP address. If SSH port forwarding is enabled, you can open a secure tunnel even when you already have an ssh connection opened, and without terminating it.
Assume you have an ssh connection to some server:
local $ ssh user#example.com
Password:
remote $ echo abc > abc.txt # now we have a file here
OK now we need to copy that file back to our local server, and for some reason we don't want to open a new connection. OK, let's get the ssh command line by pressing Enter ~C (Enter, then tilde, then capital C):
ssh> help
Commands:
-L[bind_address:]port:host:hostport Request local forward
-R[bind_address:]port:host:hostport Request remote forward
-D[bind_address:]port Request dynamic forward
-KR[bind_address:]port Cancel remote forward
That's just like the regular -L/R/D options. We'll need -R, so we hit Enter ~C again and type:
ssh> -R 127.0.0.1:2222:127.0.0.1:22
Forwarding port.
Here we forward remote server's port 2222 to local machine's port 22 (and here is where you need the local SSH server to be started on port 22; if it's listening on some other port, use it instead of 22).
Now just run scp on a remote server and copy our file to remote server's port 2222 which is mapped to our local machine's port 22 (where our local sshd is running).
remote $ scp -P2222 abc.txt user#127.0.0.1:
user#127.0.0.1's password:
abc.txt 100% 4 0.0KB/s 00:00
We are done!
remote $ exit
logout
Connection to example.com closed.
local $ cat abc.txt
abc
Tricky, but if you really cannot just run scp from another terminal, could help.
I found this one-liner solution on SU to be a lot more straightforward than the accepted answer. Since it uses an environmental variable for the local IP address, I think that it also satisfies the OP's request to not know it in advance.
based on that, here's a bash function to "DownLoad" a file (i.e. push from SSH session to a set location on the local machine)
function dl(){
scp "$1" ${SSH_CLIENT%% *}:/home/<USER>/Downloads
}
Now I can just call dl somefile.txt while SSH'd into the remote and somefile.txt appears in my local Downloads folder.
extras:
I use rsa keys (ssh-copy-id) to get around password prompt
I found this trick to prevent the local bashrc from being sourced on the scp call
Note: this requires SSH access to local machine from remote (is this often the case for anyone?)
The other answers are pretty good and most users should be able to work with them. However, I found the accepted answer a tad cumbersome and others not flexible enough. A VPN server in between was also causing trouble for me with figuring out IP addresses.
So, the workaround I use is to generate the required scp command on the remote system using the following function in my .bashrc file:
function getCopyCommand {
echo "scp user#remote:$(pwd)/$1 ."
}
I find rsync to be more useful if the local system is almost a mirror of the remote server (including the username) and I require to copy the directory structure also.
function getCopyCommand {
echo "rsync -rvPR user#remote:$(pwd)/$1 /"
}
The generated scp or rsync command is then simply pasted on my local terminal to retrieve the file.
You would need a local ssh server running in your machine, then you can just:
scp [-r] local_content your_local_user#your_local_machine_ip:
Anyway, you don't need to close your remote connection to make a remote copy, just open another terminal and run scp there.
On your local computer:
scp root#remotemachine_name_or_IP:/complete_path_to_file /local_path
This question has been repeated many times and I know to copy a file from ssh server to mac I should follow this:
Copy the file "foobar.txt" from a remote host to the local host
$ scp your_username#remotehost.edu:foobar.txt /some/local/directory
But I want to know how can I copy that to my local machine after connecting to remote ssh.
I meant after connecting to ssh means in terminal I connect to ssh and then I copy them to my pc. what I want to do is one time connect to ssh and enter password and then do all operation. Why? because I am writing a user friendly program which ask password one time and I don't want the user enter every time or save the password.
You're actually trying to reuse existing ssh connection.
Add this to your ~/.ssh/config to set up automatical connection sharing:
ControlMaster auto
ControlPath ~/.ssh/control:%h:%p:%r
Now, if you do
scp your_username#remotehost.edu:foobar.txt /some/local/directory
And if you already have a connection established in another terminal then it wont ask you for a password and connection will be established very quickly.
Not an SO question, but:
http://kb.iu.edu/data/agye.html
The syntax for the scp command is:
scp [options] username1#source_host:directory1/filename1 username2#destination_host:directory2/filename2`
In other words, just switch source/destination if you want to copy something in the other direction :)
For example:
scp foobar.txt your_username#remotehost.edu:some/directory
You can copy files from remote to local by sftp (secure file transfer protocol)
first init sftp
sftp -P typeYourPortNumber username#hostname
now you are inside of sftp terminal. now you can copy file by typing
get absolutePathToSouce absolutePathLocal
you also can transfer file to the server by
put pathToSource pathToDestination