I am trying to restore a big file (~40GB) from Azure backup. I can see my recovery point and mount it as disk drive so I can copy/paste the file I need. The problem is that the copying takes approx. 8 hours, but the disk drive (recovery point) is automatically unmounted after 6 hours and the process fails consistently. I couldn't find any setting in the backup agent to increase this slot.
Any thoughts how to overcome this?
You can extend the mount time by setting the number of hours as a higher value.
RecoveryJobTimeOut under "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Azure Backup\Config\CloudBackupProvider”
Type is DWORD; value is number of hours.
After some struggling I've found a workaround, so I'll post it here for others...
I've mounted the needed recovery point as a disk drive and started a file copy. It shows the standard Windows copy file progress dialog, which has an option of pausing. So after ~5.5 hours, just before the drive is unmounted, I paused the copy, unmounted the drive manually, mounted it again (getting another 6-hours slot), and than resumed the copy. Well, I don't think that this is how Microsoft wanted me to work, but it gets the job done.
Happy restoring!
Try to compress a copy of the file into a .zip on the server, then download the (hopefully smaller file)
Also, if you don't mind me asking, what the heck made a 40Gig file?
Related
I need to copy between 400 - 5000 images, it changes every run.
how can i calculate how many threads will give me the fastest result?
should i open new SSH connection to each thread?
i use paramiko to open ssh connection.
and use sftp to copy the images.
thx
I guess best solution before copying it's add images to one archive, because each time it's checks that each file copied and creating of new file it's very consumable operation.
If you will copy archive in one thread it's can have much faster speed of copying, because it's will not wait for each image copy.
So, will be much faster
pack to archive
copy
unpack
You can check it even without connection between any computers, just copy about 1 gb little files from one hard drive to another, and than pack these files into archive and copy again, you will notice that 2nd way will be muuuuuch faster
I'm working on a backup service. It tracks changes of the files in a the directory to backup. It does that by setting a watch (using inotify with Linux) and comparing the modification time and size after a file has been changed. When it is, the whole file is copied to backup. I'm thinking, could this be done much more efficient? If the backup service can determine the offset and the number of bytes written, it can just copy that, in stead of copy the whole file. I've been looking to fanotify, which offers some interesting features, like an fd to the file modified (by the other process). Now here it stops I think. There is no way as far as I can see it how the process using fanotify can determine from the fd how the file is changed.
Do I overlook something, or is it not possible to get this information?
There is a very useful "remote copy" feature in s3cmd sync where any duplicate copies of a file existing in the source directory will not be transferred more than once to the S3 bucket, but a remote copy will take place instead, reducing the bandwidth taken for the transfer.
I have been searching for a similar solution to do a similar file transfer between 2 Linux servers. I've used rsync many times in the past, doesn't look like it has an option for this but perhaps I have missed something.
Simple example :
/sourcedir/dir1/filea
/sourcedir/dir1/fileb
/sourcedir/dir1/filec
/sourcedir/dir2/filea
/sourcedir/dir2/filed
/sourcedir/dir2/filee
/sourcedir/dir3/filea
/sourcedir/dir3/filef
/sourcedir/dir3/fileg
With a typical transfer, filea would be transferred across the network 3 times.
I'd like to transfer this file only once and have the remote server copy the file twice to restore it to the correct directories on the other side.
I need to perform a sync on a large directory with many duplicates in the fastest time possible.
I know it would be possible to script a solution to this, but if anyone knows an application with this native functionality then that would be great!
Thanks
I have for some time being thinking I can save some money on external hard drives by making this backup scheme: If I have 3TB data to backup, where less than 1TB changes from one backup to the next and I always want to have 1 copy out of the house, it should be enough to have 3 2TB external hard drives. The idea is that each time a disk is used for backup it is completely filled - a full backup is however never made as 3TB>>2TB.
So the backup starts by taking disk1 filling it with 2TB of data. Then take disk2 filling it with 1TB of data and 1TB of redundant data as it also exist on disk1. Now disk1 and disk2 can be taken out of the house.
When the next backup is made disk2 will already contain 2TB of data, where at least 2TB-1TB=1TB is still valid as only 1TB have changes. So by backing up 2TB of data (where some may also exist on disk2) to disk3 we have a complete backup on disk2+disk3. Now disk3 can be moved out of the house and disk1 can be moved back in, deleted and reused for backup.
This can of course be made better so we can use different sizes of disks, have different number of disks, have higher requirement for number of copies out of the house etc.
In theory it is quite easy to make by having stored checksums of which files is on all disks, so we can check for changes by checking the checksums.
However in practice there is a lot of cases to handle: out of disk-space, hardlinks, softlinks, file permissions, file ownership, etc.
I've tried to find existing backup programs that can do this but I have not found any.
So my question is: How do I most easily do this? Writing it from scratch would probably take too much time. So I was wondering if I could put it on top of something existing. Any ideas?
If I were in your shoes I would buy an external hard drive that is large enough to hold all your data.
Then write a Bash script that would:
Mount the external hard drive
Execute rsync to back up everything that has changed
Unmount the external hard drive
Send me a message (e-mail or whatever) letting me know the backup is complete
So you'd plug in your external drive, execute the Bash script and then return the external hard drive to a safe deposit box at a bank (or other similarly secure location).
http://www.petri.co.il/virtual_creating_differencing_disks_with.htm
I followed these steps to create a "Differencing Disks" of the WSSv3 demo VHD from microsoft. Well some time has passed, forgot that it was a "differencing" disk and upon defragging, the VUD (Virtual Undo Disk) expanded to consume the remainder of the free space on my hard drive.
Other then committing these changes back to the original VHD file, is they any other way for me to shrink a VUD.
Thanks
[Update]
Unfortunaelty the change history seems to keep every change to a file, even file fragmentation (why).
org -> verA -> verB -> verC -> verD -> verE -> current.
A tool would be nice to collapse the history tree to something like org -> current and drop/ignore the change history in between as well as defragement the change log for optimization.
[Update#2]
First, Sorry for the extensions to my questions
Second, Is it possible to shrink a differential disk by merging it with its differential parent disk.
Base.VHD
-> Child.VHD (Differential)
-> Grandchild.VHD (Differential)
In merging the Grandchild with the Child would the size be [Child Size] + [GrandChild Size] or would it be something like [Child Size] + [Size of Actual File Differences in Grandchild]?
Thanks again.
The differencing disk recorded every change made by the defragmentation program, which is why it grew out of control. I doubt you can shrink it since it contains changes that have been made and not yet committed.
I think you are going to have to either commit the changes to the original VHD, or throw away all the changes.
Thanks Grant, you're correct and I was stuck with the bloated VHD to merge, but somehow managed to screw that up and lost the changes.
Here is what else I found.
http://www.invirtus.com/blog/?p=7
This is a great article explaining why differentials are so large. Apparently each byte is written into its own 512 byte sector waisting tons of space.
http://www.invirtus.com/downloads/Differencing_Disk_Discussion.ppt
This presentation explains how to use disk compression when storing differencing disks and that undo disks utilize less space. In short, placing you differencing VHD or VUD into a NTFS compressed folder will save you tons of space.
[Example]
I created a differencing disk for the WSSv3 image from Microsoft (5GB), booted it up and installed software. Just the boot processed added 300mb to the VHD, installing TortoiseSVN (20MB) added 200mb, and installing WSPBuilderExtensions (800KB) added 1GB to the VHD.
The end result was 1.5GB differential from installing 21MB. I merged it with the base and resulted in only adding 29MB back to the parent.
I then created another differential VHD inside a NTFS compressed folder, started it up and created a new WSS Web Application through Central Admin. The file size jumped up to 900MB, but only resulted in 90MB on the file system do to the NTFS compression. I then created a VUD, renamed to VHD and completed the same action. The file size increased to 300MB which resulted in 12MB on the file system.
So yes, the differential VHD is highly ineffective and has no intelligence it in what so ever, but the bloating allows for some nice compression.
For development you should also create a new VHD, attach it as a secondary drive and move your file here since any and every file IO is captured in the differencing or undo disk.