We're using putty and a ssh connection to our webhost. They backup our files daily onto their servers.
Since the backup files use a large amount of space, we now want to copy the backup files to our own server via a cronjob daily.
How do we have to set up the cronjob?
If you know the the backup filepath and filename (eg: backup_ddmmyyyy.tar.gz), you can simply scp that backup file from one server to another.
Put this scp command inside a shell script, and configure it accordingly with server address of the other server, and location where you want to copy the file.
Since your backup files use a large amount of space, my guess is, they are large sized individually as well, so using rsync over ssh instead of a plain scp might be a better option to compensate for network failures.
Once your script is working, you can put in a cronjob for that script for an appropriate time, after the backups on webhost are guaranteed to be over.
Related
Alternate title: How can I prevent the deletion of a file if rsync (or any other app) currently has the file open for reading?
I have a cron on my live system that dumps a database to a backup file every 30 min.
On my backup system I have a cron that runs rsync to fetch that backup file, also every 30 min.
I thought about creating lock files and using them to tell the remote if it's safe to fetch the file or visa versa. I also thought about having the dump script create file names in a rotation and create a "name" file that the remote fetches to know what file is "safe" to retrieve. I know I can synchronize the cron jobs but I'm wondering if there is a better, safer way.
Is there some OS feature I can use to accomplish this?
I have a local Linux server that I'm using to backup two remote Windows 7 boxes over an IPsec VPN tunnel connection. I have the user's Documents folders shared on the remote PC's and have mounted those shares (CIFS) on my local Linux server.
I'm going to use a cron job to run rsync on my local Linux server to create backups of these folders and am currently considering the -avz args to accomplish this.
My question is this: does the -z arg do anything for me since the mount is to a remote machine? As I understand it, -z compresses the data before sending it which definitely makes sense if the job were being run from the remote PC but, it seems like I'm compressing data that's already been pulled through the network given my setup (which seems like it would increase the backup time by adding an unnecessary step).
What are your thoughts? Should I use -z given my setup?
Thanks!
It won't save you anything. To compress the file, rsync needs to read it's contents (in blocks) and then compress them. Since reading the blocks is going to happen over the wire, pre-compression, you save no bandwidth and gain a bit of overhead from the compression itself.
What's a good approach to mirroring a production environment's data for dev? We have one production server in place that mounts many smb shares which several scripts run against routinely.
We now have a separate server for development that we want to keep separate for testing. How do I get sample data from all those smb shares without copying them all? The dev server couldn't hold all that data so I'm looking for something that could routinely run and just copy the first X files out of each directory.
The goal is to have the dev server be "safe" and not mount those same shares during testing.
For a development environment I like to have:
Known good data
Known (constructed) bad data
Random sample of live data
What I mean by "constructed" is data that I have put together in a certain way so I know exactly how it is bad.
In your case I'd have my good and bad data and then write a little Bash script to copy data from the SMB shares to the local dev machine. Maybe run a ls -t on each of the shares so you can grab the newest files, save that output to a file and use head or some other utility to read the first N lines - and copy those files to your dev machine.
pseudo code
clear data directory
copy known good data from some local directory
copy known bad data from some local directory
begin loop: for every SMB share
run `ls -t` and output the results to a file
run `head` or some other utility to get the first N lines (ie file names)
copy those files from the SMB share to my local data directory
end loop
You could set up cron to execute this little script however often you want.
These days,I am work with jsch-0.1.41,operate resources on a remote linux server via ChannelSftp.I find that there is no function provide the functionality similar to shell command "cp".Now I want to copy a file from a directory to the other,these two directory both remote directory on linux server.
Any wrong point in my presentation,please point it out.Thanks.
The SFTP protocol doesn't offer such a command, and thus also JSch's ChannelSftp doesn't offer it.
You have basically two choices:
Use a combination of get and put, i.e. download the file and upload it again. You can do this without local storage (simply connect one of the streams to the other), but this still requires moving the data twice through the network (and encrypting/decrypting twice), where it wouldn't be really necessary. Use this only if the other way doesn't work.
Don't use SFTP, but use an exec channel to execute a copy command on the server. On unix servers, this command is usually named cp, on Windows servers likely copy. (This will not work if the server's administrator somehow limited your account to SFTP-only access.)
I run a process that generates some files on a server and want to copy them to another server. The two servers are on the same network.
What are the pros/cons of using scp or a network share?
I'm not talking about a one-time copy (which I'd do manually with scp), but about programmatically copying the files after they are generated.
rsync is a third possibility, and very easily scriptable. like scp, it uses ssh by default, and if you have already set up key-based authentication, it doesn't get any easier: rsync -avuz /local/dir/ my.example.com:/remote/dir/
some advantages over scp are the --dry-run and --delete options; the first is self-explanatory, the second deletes anything in the target that's not in the source.
network shares work great when they work, but when they break it can be a major hassle.
as pst said, scp can also be easily scripted, so if you have to choose between the two options you gave, I'd say go with scp simply because it's more reliable and just as easily scripted as copying from a network share.