I run a process that generates some files on a server and want to copy them to another server. The two servers are on the same network.
What are the pros/cons of using scp or a network share?
I'm not talking about a one-time copy (which I'd do manually with scp), but about programmatically copying the files after they are generated.
rsync is a third possibility, and very easily scriptable. like scp, it uses ssh by default, and if you have already set up key-based authentication, it doesn't get any easier: rsync -avuz /local/dir/ my.example.com:/remote/dir/
some advantages over scp are the --dry-run and --delete options; the first is self-explanatory, the second deletes anything in the target that's not in the source.
network shares work great when they work, but when they break it can be a major hassle.
as pst said, scp can also be easily scripted, so if you have to choose between the two options you gave, I'd say go with scp simply because it's more reliable and just as easily scripted as copying from a network share.
Related
I have 2 remote servers/machine say s1 and s2 (linux based machines)
Both the server has 1 directory which is very huge. (i mean initially same data in both machines)
s1 is always stable upto date, changes are added by authorized user.
s2 people will make changes to the data here and there.
now requirement is to make content of s2 to inSync with s1.
Condition:
1. No replacement of s2 content with s1 because data is very huge
2. No other software allowed to install in machines
3. Only scp, sftp supported, no ssh or any other sort of access is given because it is production machine.
If anybody come across this sort of requirement Please suggest me any tool, any way to do this task.
If, you say, you have scp, then you must also have ssh. scp requires ssh to work. So, I'll start by challenging your assumption that you can't use rsync over ssh. If you have scp working, then there's no reason why rsync over ssh should not work.
rsync over ssh is the correct answer here. This is the most efficient mechanism for synchronizing content between two different servers. But, I suppose that it's possible that someone who thinks he knows what he's doing, but he really doesn't, hacked up a server to allow only the scp service, and block ssh sessions. Probably under a mistaken notion that this improves security somehow. It really doesn't, but that's a different topic. So, what now...
Well, you say you do have sftp access available. In that case, the next best answer would be a custom sftp client. Learn perl, and use the Net::SFTP module to write a custom perl script, for your specific requirement, to use SFTP to compare the contents of the two servers, and synchronize their contents.
Net::SFTP exposes the underlying SFTP protocol in a way that allows one to write custom applications that uses it. You'll use the SFTP protocol to examine the contents of each server, figure out what's different, then copy what needs to be copied, in order to update their contents.
Using Net::SFTP won't be as efficient as using rsync+ssh. With Net::SFTP, you'll know which files exist on the server, and the size of each file in bytes. However, if both servers appear to have a file with the same name, and the same byte count, you don't really know whether they are, in fact, identical, without downloading each file, and manually comparing them. You'll have to do that, of course. This is the key advantage of rsync+ssh that you do not have an sftp equivalent of. The rsync server works together with the rsync client, and they're able to verify that the file contents are identical, using checksums, without actually transferring the file from one side to another. No way to avoid doing that with sftp in this case, but this is going to be the best you'll be able to do.
If you decide to go the Perl route, don't use Net::SFTP which is an old an unmaintained module. Instead go for Net::SFTP::Foreign that BTW, implements recursive downloads allowing you to select which files to get on the fly, so you can easily do an update.
Another alternative is to use the development version of my other module Net::SSH::Any that has a built-in scp client that is able to download only the files that are newer on the remote side:
my $ssh = Net::SSH::Any->new(...);
$ssh->scp_get( { recursive => 1,
update => 1 },
$remote_dir, $local_dir );
Other scripting languages like Python or Ruby also have SFTP and SCP libraries.
I'm working on a project that needs to be tested on an embedded Linux system. After every little change, I have to scp all files to the device over a SSH connection. Can you suggest a more convenient way to deploy files on a remote target? For example some trick on make's install command:
make install INSTALL='scp 192.168.1.100:/'
or something.
if you can use scp, you can probably also use rsync, specifically rsync over ssh. Use of rsync has as advantage is that it builds a delta of source and destination files, and transfers only what is necessary. In case of transfer after changing very little this would be of considerable benefit. I'd probably invoke it if building completes without error, like make ... && upload (where upload could be a script covering the details of transfer)
Just for completeness, sshfs is often quite useful. You can mount a remote folder visible over ssh on to a folder on your local hard disk. Performance is not great, but certainly serviceable enough for a deploy step, and it's transparent to all tools.
we have a network of several machines and we want to distribute a big directory (ca. 10 GB) to every box.
It is located on an nfs-server and is mounted on all machines, so first approach is to just use normal cp to copy the files from the mounted to a local directory. This is easy, but unfortunately there is no progress bar, because it is not intended to use it for network copies (or is it?).
Using scp is intended for copying across network, but it may encrypt everything and therefore be slow.
Should one be faster, and if so, which: cp on nfs-mount or scp?
You can always use rsync, it can show you the progress (with --progress option) and is more lightweight than scp.
You can enable compression manually with -z.
These days,I am work with jsch-0.1.41,operate resources on a remote linux server via ChannelSftp.I find that there is no function provide the functionality similar to shell command "cp".Now I want to copy a file from a directory to the other,these two directory both remote directory on linux server.
Any wrong point in my presentation,please point it out.Thanks.
The SFTP protocol doesn't offer such a command, and thus also JSch's ChannelSftp doesn't offer it.
You have basically two choices:
Use a combination of get and put, i.e. download the file and upload it again. You can do this without local storage (simply connect one of the streams to the other), but this still requires moving the data twice through the network (and encrypting/decrypting twice), where it wouldn't be really necessary. Use this only if the other way doesn't work.
Don't use SFTP, but use an exec channel to execute a copy command on the server. On unix servers, this command is usually named cp, on Windows servers likely copy. (This will not work if the server's administrator somehow limited your account to SFTP-only access.)
I'm developing on my local machine (apache2, php, mysql). When I want to upload files to my live server (nginx, mysql, php5-fpm), I first backup my www folder, extract the databases, scp everything to my server (which is tedious, because it's protected with opiekey), log myself in, copy the files from my home directory on the server to my www directory and if I'm lucky and the file permissions and everything else works out, I can view the changes online. If I'm unlucky I'll have to research what went wrong.
Today, I changed only one file, and had to go through the entire process just for this file. You can imagine how annoying that is. Is there a faster way to do this? A way to automate it all? Maybe something like "commit" in SVN and off you fly?
How do you guys handle these types of things?
PS: I'm very very new to all this, so bear with me! For example I'm always copying files into my home directory on the server, because scp cannot seem to copy them directly into the /var/www folder?!
There are many utilities which will do that for you. If you know python, try fabric. If you know ruby, you may prefer capistrano. They allow you to script both local and remote operations.
If you have a farm of servers to take care of, those two might not work at the scale you want. For over 10 servers, have a look at chef or puppet to manage your servers completely.
Whether you deploy from local checkout, packaged source (my preferred solution), remote repository, or something entirely different is up to you. Whatever works for you is ok. Just make sure your deployments are reproducible (that is you can always say "5 minutes ago it wasn't broken, I want to have what now what I had 5 minutes ago"). Whatever way of versioning you use is better than no versioning (tagged releases are probably the most comfortable).
I think the "SVN" approach is very close to what you really want. You make a cron job that will run "svn update" every few minutes (or hg pull -u if using mercurial, similar with git). Another option is to use dropbox (we use it for our web servers sometimes) - this one is very easy to setyp and share with non-developers (like UI designers)...
rsync will send only the changes between your local machine and the remote machine. It would be an alternative to scp. You can look into how to set it up to do what you need.
You can't copy to /var/www because the credentials you're using to log in for the copy session doesn't have access to write on /var/www. Assuming you have root access, change the group (chown) on /var/www (or better yet, a sub directory) to your group and change the permissions to allow your group write access (chmod g+w).
rsync is fairly lightweight, so it should be simple to get going.