Is there a way to boost svn performance when the working copy is running over NFS?
(*) It is required for it to be and the NFS mounted partition (/home).
I guess SVN client reads the whole tree looking for changes when commiting. I don't have an idea of what can make a checkout slow.
According to the Subversion FAQ:
Working copies can be stored on NFS (one common scenario is when your home directory is on a NFS server). On Linux NFS servers, due to the volume of renames used internally in Subversion when checking out files, some users have reported that 'subtree checking' should be disabled (it's enabled by default). Please see NFS Howto Server Guide and exports(5) for more information on how to disable subtree checking.
Checkout performance can be constrained by a few factors, but most likely in your case it's I/O to that NFS mount - unless you're saturating the network connection, or the server is undersized.
Use "nolock" option to mount. It is actually OS local lock, not NFS server side lock.
I got better performance from this option.
Checkout performance over NFS is abysmal, to the point that it becomes the major bottleneck, not the bandwidth to the Subversion server. rsvn by Bryce Denney and Wilson Snyder is a perl script that logs into the NFS server using ssh (assuming that is allowed) and runs the svn command remotely. In my tests it gives orders of magnitude faster performance. Excerpt from the man page:
NAME
rsvn - remote svn - run subversion commands on the file server if possible
SYNOPSIS
rsvn ANY_SVN_COMMAND
rsvn update
rsvn --test
DESCRIPTION
When possible, run the svn command on the file server where it does not have to wait for NFS. Otherwise run svn as usual. Some SVN commands will always run locally, either for "safety" or because there is no benefit of running on the file server (svn log).
The commands that will be sent to the file server by default are these (and their abbreviations):
add checkout cleanup diff merge resolved
revert status switch update
Why is commit not run remotely? Because it will either start an editor, which won't always work through noninteractive SSH, or you might use -m "FOO BAR" and the shell's quoting gets all screwed up. It would be good to figure out how to solve these problems, and add "commit" to the list.
Related
I am using a load balancer in aws and want to sync files in real time. I was trying to do it by rsync but it's not a real time we set it by cron. I want to do it by real time, I am using it in Singapore region and there is no EFS option.
There is a daemon called lsyncd, which does exactly what you need.
You can read further about it here
"rsync is an excellent and versatile backup tool, but it does have one drawback: you have to run it manually when you want to back up your data. Sure, you can use cron to create scheduled backups, but even this solution cannot provide seamless live synchronization. If this is what you want, then you need the lsyncd tool, a command-line utility which uses rsync to synchronize (or rather mirror) local directories with a remote machine in real time. To install lsyncd on your machine, download the latest .tar.gz archive from the project's Web site, unpack it, and use the terminal to switch to the resulted directory. Run then the ./configure command followed by make, and make install (the latter command requires root privileges). lsyncd is rather straightforward in use, as it features just one command and a handful of options"
I'm working on a project that needs to be tested on an embedded Linux system. After every little change, I have to scp all files to the device over a SSH connection. Can you suggest a more convenient way to deploy files on a remote target? For example some trick on make's install command:
make install INSTALL='scp 192.168.1.100:/'
or something.
if you can use scp, you can probably also use rsync, specifically rsync over ssh. Use of rsync has as advantage is that it builds a delta of source and destination files, and transfers only what is necessary. In case of transfer after changing very little this would be of considerable benefit. I'd probably invoke it if building completes without error, like make ... && upload (where upload could be a script covering the details of transfer)
Just for completeness, sshfs is often quite useful. You can mount a remote folder visible over ssh on to a folder on your local hard disk. Performance is not great, but certainly serviceable enough for a deploy step, and it's transparent to all tools.
I have a local Linux server that I'm using to backup two remote Windows 7 boxes over an IPsec VPN tunnel connection. I have the user's Documents folders shared on the remote PC's and have mounted those shares (CIFS) on my local Linux server.
I'm going to use a cron job to run rsync on my local Linux server to create backups of these folders and am currently considering the -avz args to accomplish this.
My question is this: does the -z arg do anything for me since the mount is to a remote machine? As I understand it, -z compresses the data before sending it which definitely makes sense if the job were being run from the remote PC but, it seems like I'm compressing data that's already been pulled through the network given my setup (which seems like it would increase the backup time by adding an unnecessary step).
What are your thoughts? Should I use -z given my setup?
Thanks!
It won't save you anything. To compress the file, rsync needs to read it's contents (in blocks) and then compress them. Since reading the blocks is going to happen over the wire, pre-compression, you save no bandwidth and gain a bit of overhead from the compression itself.
Despite a lots of topics about this error, I'm still having trouble with setting up av SVN Server. Server is running on Scientific Linux 6 and repositories are supposed to be stored via NFS3 on a SUNOS Storage server.
I read that mounting with "nolocks" options would solve the problem but I don't want to do so as a lot of users are working at the same time on the server, I guess removing the locks would create new problems.
SVN is installed, working on local files, but when I try to create a repo on distant location, files are created but I get the error "database is locked" and cannot use the repo. I use the fsfs system which is supposed to work fine with NFS.
Would anyone have another option for me ?
OK I eventually set up a new share on the NFS server, accessible by my SVN server only, mounted there with "nolock". Then it works, but not really the point, I still don't know how to set that up without removing the locks.
An NFS client will normally use the NFS Lock Manager (NLM) to synchronize locking of certain files on the NFS server with other NFS client accessing/locking the same files. The nolock mount option tells the NFS client not to use the NFS Lock Manager but instead to manage the locks locally on the NFS client machine itself. This is useful if you only have 1 NFS client or several NFS clients where each client works on a different area of the exported file system so that there is no lock contention.
It looks like you have the following:
(A) SVN_Client ==> (B) SVN_Server/NFS_Client ==> (C) NFS_Server
Where: Server (B) is Scientific linux 6 providing SVN services to clients and mounting from Server (C), the SunOS Storage Server.
Assuming you have no other machine mounting from the NFS server and providing the same SVN services, the nolock option will work correctly as server(B) will do all the lock management locally. There is no need/requirement to lock centrally on the NFS server.
This is true for NFSv3 which you mentioned in your question.
I'm trying to figure out how to do this with Eclipse. We currently run SVN and everything works great, but I'd really like to cut my SSH requests in half and use Eclipse to modify some files directly on the server. I'm using the below build of eclipse... how can I do this?
Eclipse for PHP Developers
Build id: 20100218-1602
Update
I have no intention of eliminating SVN from the equation, but when we need to make a hotfix, or run a specific report or function as a one-time thing, I'd much rather use Eclipse than terminal for modifying that kind of thing.
Have a look at How can I use a remote workspace over SSH? on the Eclipse wiki. I'm just quoting the summary below (read the whole section):
Summing up, I would recommend the
following (in order of preference):
VNC or NX when available remotely, Eclipse can be started remotely and
the network is fast enough (try it
out).
Mounted filesystem (Samba or SSHFS) when possible, the network is fast
enough and the workspace is not too
huge.
rsync when offline editing is desired, sufficient tooling is
available locally, and no merge issues
are expected (single user scenario).
RSE on very slow connections or huge workspaces where minimal data
transfer is desired.
EFS on fast connections when all tooling supports it, and options
like VNC or mounted filesystem or
rsync are not available.
But whatever you'll experiment, don't bypassing the version control system.
You could use something like SSHFS, but really, it's a better idea to use some kind of source control system instead of editing files directly on the server. If Subversion isn't sufficient, perhaps you might try a DVCS like Git or Mercurial.