I'm planning to transfer data from an old ftp to a www.backblaze.com/b2 bucket.
I'm considering to use rclone for this.
Running rclone in a aws machine. In rclone I will configure two remotes, the ftp and the b2 bucket. Then will execute something like:
./rclone sync ftp:/myfolder b2:/myfolder
Full data size goes from 100GB to 500GB.
The aws machine is on south america and I guess the ftp also, not sure about b2 bucket.
Question: Do this consumes my amazon aws network transfer? And will it cost a lot?
Data transfer into AWS is free. You will be charged only when you transfer data out from AWS to internet. It depends on the region and the size of the data you are transferring out. Based on 100GB to 500GB transfer, data transfer alone will cost you $9 to $45. See Data Transfer Cost for more information.
Related
We have a large amount, 1PB, of (live) data that we have to transfer periodically between S3 and Azure Blob Storage. What tools do you use for that? And what strategy do you use to minimize cost of transfer and downtime?
We have evaluated a number of solutions, including AzCopy, but none of them satisfy all of our requirements. We are a small startup so we want to avoid homegrown solutions.
Thank you
Azure Data Factory is probably your best bet.
Access the ever-expanding portfolio of more than 80 prebuilt connectors—including Azure data services, on-premises data sources, Amazon S3 and Redshift, and Google BigQuery¬—at no additional cost. Data Factory provides efficient and resilient data transfer by using the full capacity of underlying network bandwidth, delivering up to 1.5 GB/s throughput.
read this mount -t nfs vs cifs already :( ?
Our requirement is that we have an application hosted in AWS using nfs-utils to mount a EFS for use ? My question is how can this be done in Azure. I know they have Azure files which works in quite similar way to EFS but as per azure documentation it is done only through cifs-util. Point is that though it will mount a Azure file share in Azure will it work without any issue or do we need to do something in our commands to make it happen?
I am not good in linux, so please pardon me if I am sounding total stupid.
Our requirement is that we have an application hosted in AWS using
nfs-utils to mount a EFS for use ? My question is how can this be done
in Azure.
Amazon Elastic File System (Amazon EFS) provides simple, scalable file storage for use with Amazon EC2 instances in the AWS Cloud.
If you want do the same thing in Azure, I think you are talking about Azure storage blob(new disk).
In Azure, we can via Azure portal to add a new disk to Azure VM as a data disk, works like add a physical data disk to a host. Then we can use fdisk to create the file system on the new partition.
We can follow this article to attach a new disk to Azure VM via Azure portal.
After that completed, we can follow this article to initialize a new data disk in Linux.
I know they have Azure files which works in quite similar way to EFS
but as per azure documentation it is done only through cifs-util.
You are right, Azure files share works like EFS, but Azure files share use Server Message Block (SMB) protocol(also known as Common Internet File System, or CIFS).
The maximum size of an Azure file share is 5 Tib, there is a quota of 20,000 open handles on a single file, and the max IOPS per share is 1000 IOPS.
We can create data disk from Azure storage blob, the maxium size of data disk is 4 Tib(we can create multiple data disks to that VM), and OS disk is 2 Tib.
AWS EFS suppoer Network file system versions 4.0 and 4.1(NFSv4) protocal.
Here a article about performance about Azure file share and Azure storage blob.
I have worked with EXSi Servers lot of times.They provide snapshot option which can be used to revert back the same server to any point of snapshot that we taken.
I was unable to find the same in AWS and Azure.These cloud enterprises provide the option to backup the server.
AWS backups the whole volume.
Azure provide vault800 backup wizard which is incremental.
We can create a new Server with that backup, but we cannot revert back the same server.The EXSi Server take snapshot 10% of 100% volume of server and revert back as per our requirement.
For Azure, take a look at blob snapshots.
Azure Storage provides the capability to take snapshots of blobs. Snapshots capture the blob state at that point in time.
Pretty much the same story with AWS:
You can back up the data on your Amazon EBS volumes to Amazon S3 by taking point-in-time snapshots. Snapshots are incremental backups, which means that only the blocks on the device that have changed after your most recent snapshot are saved
how about using a 3rd party backup solution like Veeam or cloudberry to take image based backup copies and replicate them onto preferred cloud storage.
Veeam also supports instant VM recovery, you can immediately restore a VM into your production environment by running it directly from the backup file. Instant VM recovery helps improve recovery time objectives (RTO), minimise disruption and downtime of production VMs. It is like having a "temporary spare" for a VM: users remain productive while you can troubleshoot an issue with the failed VM.
My company has a disaster recovery requirement, this requirement considered AWS failure, such as AWS is unusable for some reason, we will need to be able to quickly restore our application in other cloud providers, like Azure.
Currently our solution is
taking snapshot of ec2 instance where our application is running regularly, these snapshots are the basic backup stored in AWS (invisibly on S3).
Take the latest snapshot at 0 am every day, and create a EBS volume
Create a temporary EC2 instance
Attache and Mount the volume created in step 2 to the instance created in step 3.
Tar the files on the instance, and upload to s3 (we have some magic tools to copy files in specific bucket to Azure.).
Delete the volume and terminate the instance.
This solutions works, but it has some problem, is there any other better way to complete this requirement?
If you have automated the steps (AWS CLI would suffice) then your solution is simple and effective, the only simpler thing would be to just tar the files on the source instance but that could impact whatever service it's providing.
Microsoft's solution would be to use Azure Site Recovery:
https://learn.microsoft.com/en-us/azure/site-recovery/site-recovery-migrate-aws-to-azure
If you are expecting an AWS outage, isn't it non effective to assume that you will be able to create another instance and restore your snapshot? I mean, if that's the case, why don't you just keep running on AWS?
I think your snapshot policy is OK to prevent EC2 failures, but not AWS. To cover the second case, backup your data to S3 and copy it periodically to Azure.
You will have extra data transfer costs (and they can get huge) but it's the safer way to ensure that you won't have to wait AWS gets back (how long could it take?) to move your data.
I have a large volume of data (~20 TB) in Azure blob storage that I want to access from Spark cluster setup in Amazon EMR. What is the best way to do this? Is transferring this data to S3 the only option? If yes, what is the cheapest way to transfer this data to S3?
Thanks!
Based on your description, you have about 20TB data want to transfer to Amazon S3. I am not familiar with Amazon. But in Azure we will be charged for the data transfer. Here is the pricing site. For example you need $0.08 per GB. 20*1024*0.08= $1638.4. It is very expensive. I would suggest you to consider other approaches. If you do not care about money at all, please try to search tool in google or write your own code to transfer these data.