I have a local Linux server that I'm using to backup two remote Windows 7 boxes over an IPsec VPN tunnel connection. I have the user's Documents folders shared on the remote PC's and have mounted those shares (CIFS) on my local Linux server.
I'm going to use a cron job to run rsync on my local Linux server to create backups of these folders and am currently considering the -avz args to accomplish this.
My question is this: does the -z arg do anything for me since the mount is to a remote machine? As I understand it, -z compresses the data before sending it which definitely makes sense if the job were being run from the remote PC but, it seems like I'm compressing data that's already been pulled through the network given my setup (which seems like it would increase the backup time by adding an unnecessary step).
What are your thoughts? Should I use -z given my setup?
Thanks!
It won't save you anything. To compress the file, rsync needs to read it's contents (in blocks) and then compress them. Since reading the blocks is going to happen over the wire, pre-compression, you save no bandwidth and gain a bit of overhead from the compression itself.
Related
I'm working on a project that needs to be tested on an embedded Linux system. After every little change, I have to scp all files to the device over a SSH connection. Can you suggest a more convenient way to deploy files on a remote target? For example some trick on make's install command:
make install INSTALL='scp 192.168.1.100:/'
or something.
if you can use scp, you can probably also use rsync, specifically rsync over ssh. Use of rsync has as advantage is that it builds a delta of source and destination files, and transfers only what is necessary. In case of transfer after changing very little this would be of considerable benefit. I'd probably invoke it if building completes without error, like make ... && upload (where upload could be a script covering the details of transfer)
Just for completeness, sshfs is often quite useful. You can mount a remote folder visible over ssh on to a folder on your local hard disk. Performance is not great, but certainly serviceable enough for a deploy step, and it's transparent to all tools.
I develop a software that uses a set of big files.
I cannot download all them.
I need to reproduce timeout error that cannot be reproduce otherwise.
There are stage host. I mounted its remote folder with sshfs but I cannot launch local
server instance because it can change these files. It requires write permissions.
With "sshfs -o ro" it fails to start.
I want to know is it possible to say to save changes locally that could overlay actual bytes in remote files?
You should be able to use UnionFS or AUFS (or any other Union mount filesystem) to use these two folders together. You would have the readonly mount with sshfs and merge it with local folder that would be preferred. Reading would occur from the remote filesystem until write has been done.
Is there a way to boost svn performance when the working copy is running over NFS?
(*) It is required for it to be and the NFS mounted partition (/home).
I guess SVN client reads the whole tree looking for changes when commiting. I don't have an idea of what can make a checkout slow.
According to the Subversion FAQ:
Working copies can be stored on NFS (one common scenario is when your home directory is on a NFS server). On Linux NFS servers, due to the volume of renames used internally in Subversion when checking out files, some users have reported that 'subtree checking' should be disabled (it's enabled by default). Please see NFS Howto Server Guide and exports(5) for more information on how to disable subtree checking.
Checkout performance can be constrained by a few factors, but most likely in your case it's I/O to that NFS mount - unless you're saturating the network connection, or the server is undersized.
Use "nolock" option to mount. It is actually OS local lock, not NFS server side lock.
I got better performance from this option.
Checkout performance over NFS is abysmal, to the point that it becomes the major bottleneck, not the bandwidth to the Subversion server. rsvn by Bryce Denney and Wilson Snyder is a perl script that logs into the NFS server using ssh (assuming that is allowed) and runs the svn command remotely. In my tests it gives orders of magnitude faster performance. Excerpt from the man page:
NAME
rsvn - remote svn - run subversion commands on the file server if possible
SYNOPSIS
rsvn ANY_SVN_COMMAND
rsvn update
rsvn --test
DESCRIPTION
When possible, run the svn command on the file server where it does not have to wait for NFS. Otherwise run svn as usual. Some SVN commands will always run locally, either for "safety" or because there is no benefit of running on the file server (svn log).
The commands that will be sent to the file server by default are these (and their abbreviations):
add checkout cleanup diff merge resolved
revert status switch update
Why is commit not run remotely? Because it will either start an editor, which won't always work through noninteractive SSH, or you might use -m "FOO BAR" and the shell's quoting gets all screwed up. It would be good to figure out how to solve these problems, and add "commit" to the list.
we have a network of several machines and we want to distribute a big directory (ca. 10 GB) to every box.
It is located on an nfs-server and is mounted on all machines, so first approach is to just use normal cp to copy the files from the mounted to a local directory. This is easy, but unfortunately there is no progress bar, because it is not intended to use it for network copies (or is it?).
Using scp is intended for copying across network, but it may encrypt everything and therefore be slow.
Should one be faster, and if so, which: cp on nfs-mount or scp?
You can always use rsync, it can show you the progress (with --progress option) and is more lightweight than scp.
You can enable compression manually with -z.
I have many machines (20+) connected in a network. each machine accesses a central database, queries it, processes the information queried, and then writes the results to files on its local hard drive.
Following the processing, I'd like to be able to 'grab' all these files (from all the remote machines) back to the main machine for storage.
I thought of three possible ways to do so:
(1) rsync to each remote machine from the main machine, and 'ask' for the files
(2) rsync from every remote machine to the main machine, and 'send' the files
(3) create a NFS share on each remote machine, to which the main machine can access and read the files (no 'rsync' is needed in such a case)
Is one of the ways better than others? are there better ways I am not aware of?
All machines use Ubuntu 10.04LTS. Thanks in advance for any suggestions.
You could create one NFS share on the master machine and have each remote machine mount that. Seems like less work.
Performance-wise, it's practically the same. You are still sending files over a (relatively) slow network connection.
Now, I'd say which approach you take depends on where you want to handle errors or irregularities. If you want the responsibility to lie on your processing computers, use rsync back to the main one; or the other way round if you want the main one to work on assembling the data and assuring everything is in order.
As for the shared space approach, I would create a share on the main machine, and have the others write to it. They can start as soon as the processing finishes, ensure the file is transferred correctly, and then verify checksums or whatever.
I would prefer option (2) since you know when the processing is finished on the client machine. You could use the same SSH key on all client machines or collect the different keys in the authorized_keys file on the main machine. It's also more reliable if the main machine is unavailable for some reason, you can still sync the results later while in the NFS setup the clients are blocked.