I am looking for some inputs on how to do a GCP cloud to AZURE cloud data migration.
Scenario -
I have a snowflake instance configured on GCP cloud (multiple databases holding legacy data) and I have another snowflake instance configured on Azure Cloud (DWH created on this instance).
I want to move/copy the data of all the databases (including all child objects - schema, table, views etc) sitting on GCP snowflake instance to snowflake instance configured on Azure Cloud.
Can you please guide me on what can be the best solution for such data migration and any steps or documentation link would be really helpful.
Many thanks - Minti
Please check the Database replication mechanism which can be used as a migration tool for SF account from 1 cloud platform to another. https://docs.snowflake.com/en/user-guide/database-replication-intro.html
Not something I've done before to be honest but if you didn't want to use external tools one possible method would be to secure share your GCP databases with your Azure Snowflake account.
You then might be able to create a new database that is a clone of this share (not sure if this is possible).
Most objects get cloned apart from stages and pipes but tables, views etc should carry over
This is a pretty easy process with a couple of prerequisites.
Make sure you have Organizations enabled on your GCP account.
This feature allows you to self-provision Snowflake accounts on any cloud provider/region. Open a support case to enable it.
Introduction to Organizations
Create a new account on Azure if you haven't already.
Enable Replication on both accounts
This can be done when logged into the account with the ORGADMIN role
Replicate your databases
Note: this will work for having a replica of the databases in the GCP Snowflake account databases in your Azure Snowflake account. If you want to permanently migrate your databases you need to set up Failover/Failback. This is a Business Critical feature, but Snowflake support will enable it for lower editions until you can complete your migration, at which point they will disable it.
Replicating a Database to Another Account
There are two options
You could make use of the replication feature
High level Steps include the below
a. Target account to be created - Can use the Organizations feature available in Snowflake(Enabled by Snowflake Support upon request)
b. Account level objects should be created manually in the target account
Note: The Failover feature is supported for the accounts whose edition is Business-critical and above. However, for account migration scenarios, this feature will be enabled for a temporary period by the Snowflake Support.
c. Replication - the below links can be referenced for a complete understanding of the process.
https://docs.snowflake.com/en/user-guide/database-replication-intro.html#introduction-to-database-replication-across-multiple-accounts
https://docs.snowflake.com/en/user-guide/database-replication-config.html#replicating-a-database-to-another-account
https://docs.snowflake.com/en/user-guide/database-failover-config.html#failing-over-databases-across-multiple-accounts
Please find the link below to have an overview on the costs associated
https://docs.snowflake.com/en/user-guide/database-replication-billing.html#understanding-billing-for-database-replication
Limitations
https://docs.snowflake.com/en/user-guide/database-replication-intro.html#current-limitations-of-replication
One other option is to create the target account and use the unloading and loading feature
https://docs.snowflake.com/en/user-guide-data-unload.html
https://docs.snowflake.com/en/user-guide-data-load.html
Related
We are trying to ingest some data from DataLake to Azure Cosmos DB and Spark OLTP Connector seems to be the easiest to use.
But due to the company's policy, we are not supposed to use the master keys and we usually use managed identity for the applications. I see the Cosmos DB Java client builder has the 'TokenCredential' option with sample code as:
CosmosAsyncClient client = CosmosClientBuilder
.credential(new DefaultAzureCredentialBuilder().build())
.buildAsyncClient();
Is there anyway to setup the connector to use the same authentication mechanism with managed identity?
I see the Cosmos DB Java client builder has the 'TokenCredential' option with sample code
In CosmosAsyncClient you also have to mention the maker key. there is no such way to use managed identities.
we are not supposed to use the master keys and we usually use managed identity for the applications.
As you want to transfer data from Datalake to CosmosDB with Managed Identities you can use Copy Data Tool in Azur data factory. Create Linked service for cosmos db and in authentication type select Managed identity either system or user.
You can refer this So Thread by #KarthikBhyresh-MT for more understanding on Copy data tool.
Currently, the Spark Connector does not support MSI. I see you correctly created the Issue on the repo that holds the source code: https://github.com/Azure/azure-sdk-for-java/issues/29958
That will surely be used for tracking purposes or at least linking to the workitem that tracks the progress on that area. The feature will be available in the future but there is currently no ETA.
Is it possible for me to insert some data from one database to another in Azure sql?
Let's say I have a trigger in db1 that updates some values in db2.
I read about elastic queries but it seems like they are read-only so they don't solve my problem.
You can't use cross-database in Azure Sql Server because databases can't see eachother physically , you could use elastic pools but they are Read Only.
A solution is to use SQL Managed Instance to upload your instance . This supports cross-database queries but it was expensive.
There was some previous discussion here about doing similar:
C# Azure Function trigger when SQL Database has a new row added without polling
There is also the Azure SQL Bindings for Azure Functions but they are input bindings and not triggers and they're still in preview and limited to C#, JavaScript and Python.
Azure SQL bindings for Azure Functions overview (preview)
There was a new announcement last week after MS Build however for Azure SQL Database External REST Endpoints Integration (hopefully they don't refer to it as ASDEREI) but this is currently in preview under Early Adoption Program (EAP).
Announcing the “Azure SQL Database External REST Endpoints Integration” Early Adoption Program
I am trying to use Azure Migrate to move a VM and database from Openstack based cloud to Azure as a VM and managed database. What will happen after the migration is complete ? Will Azure Migrate automatically remove the VM and the database from the source infrastructure ? Are the source resources retained as they are so that the migration can be cancelled if needed?
You have to clean up the resource on Openstack on your own:
Database - Migrate your data with Database Migration Service
Migrate your data with Database Migration Service
In the Azure portal, follow these steps to create an instance of Azure Database Migration Service, and then to run it to migrate the data in your databases:
Create an instance of Azure Database Migration Service. Choose the pricing tier based on whether you need an online or offline migration.
Create a new migration project. Choose the type of migration you want to perform, either offline or online.
Specify source and target server details, including the authentication information.
Identify the databases. Map the relevant target database on the target server to the source server.
Run and monitor the migration.
Select the Run migration button to start the migration. The migration activity screen will appear.
Track the progress until the process shows as completed.
After all the required databases are migrated, check them to make sure they're working.
When these steps are complete, your schema and data have been migrated to the Azure SQL Database instance. You can then shut down and decommission your on-premises databases and servers.
VM - Post-migration steps:
After the migration has taken place, review the security settings of the virtual machine after the migration. Restrict network access for unused services by using network security groups. Deploy Azure Disk Encryption to secure the disks from data theft and unauthorized access.
Consider improving the resilience of the migrated machines by:
Adding a backup schedule that uses Azure Backup.
Replicating the machines to a secondary region using Azure Site Recovery.
Complete clean-up tasks for the remaining on-premises servers. Such tasks may include removing the servers from local backups and removing their raw disk files from storage-area network (SAN) storage to free up space. Update documentation related to the migrated servers to reflect their new IP addresses and locations in Azure.
I am working on a solution that uses SQL Azure. Part of the project deals with backups and using the DAC Web Services for backups.
The issue is that there is a different endpoint depending on which region the Azure SQL database is in. As I am working with multiple groups, and cannot ensure which region the database will be in, I am looking for a way to programmatically determine the region.
The region is also important, as I want to copy the backups to a different region just to be on the safe side.
I know that I can look in the Admin console, but I would like to use code to solve this problem.
Additional information:
The application is running on Azure using Worker Roles for functionality.
I do not have access to all of the account-id's to use the full REST API.
I do have access to the master database on the Azure Sql Server.
Working on this in C# (I failed to put the language)
You can use Get Servers request (GET https://management.database.windows.net:8443/<subscription-id>/servers) of Azure REST API to enumerate SQL servers which gives the Location or Region more info at msdn -> http://msdn.microsoft.com/en-us/library/windowsazure/gg715269.aspx
I would like to transfer my existing SQL Azure location to other one, but I think there is no functionality right now to do so on the management portal of Azure.
I just googled it and found one link http://social.msdn.microsoft.com/Forums/en-US/ssdsgetstarted/thread/e6c961cc-5eea-4f07-82c9-a8805d367b05 that says I need to use the data sync option in Azure's portal but I don't have that feature enabled in my Azure portal.
Also if I do use that option, is there any charge for it? Finally, are there any other option that is possible for moving the SQL Azure location?
To Move an Existing SQL Server Database to a New Region on Azure Assuming There Are No Blob Containers Associated With the Database. For further reference see:
https://azure.microsoft.com/en-us/blog/migrating-azure-services-to-new-regions/
Upgrade the database, if necessary, to one of the Premium pricing tiers
Add geo-replication to the existing database. You can choose what region to have the backup of the existing database. Create a new Database server in the target region of your choice. I suggest provisioning that new database server with the same admin username and password as the existing sql database. When creating the secondary database, I suggest making the Secondary type “Readable” as it will allow you the ability to check that all data and schemas were replicated correctly.
Allow the two databases time to sync. Rule of thumb according from Microsoft AzureCAT is: 3 * (5 minutes + database size / 150 MB/minute)
Configure the Firewall settings of the secondary database to allow the necessary IP addresses to access the database
Temporarily shut down whatever users or applications are accessing the existing database.
From the Azure portal select the existing database and change its geo-replication role from primary to secondary.
Run any ddl scripts that rely on the masterdb such as ddl scripts to recreate users and user profiles
Change the connection strings of any applications to point to the new database.
Users and applications can now connect to the new Database
At your discretion you can remove the old database as a backup and add any new regions as backup.
In terms of charges there will be charges for upgrading the old database if it isn't already a premium database. There will also be charges for creating the geo-replicated database. However, those charges can be limited to a day to a few days worth of fees (depending on how long geo-replication takes). Once the new database is up and running, delete the old database as soon as possible to limit additional fees. Finally, if you upgraded the service level of the old database to a premium tier to facilitate the geo-replication, you will want to downgrade the new database to the original service level of the old database to also limit fees.
I think you can use new Import/Export bacpac feature. I have used it to move databases between accounts and can't see why it wouldn't also work between regions.
See how here
If you are able to stop writes to the DB for a time then you can use the Copy feature on the Azure Portal.
Create a new SQL Server in the region of your choosing.
Add your service(s) IP addresses to the new SQL Server firewall.
Stop writes to the origin database.
Open the origin database in the Azure Portal and click Copy at the top of the blade.
Choose your new SQL server located in the destination region.
Wait for the copy to complete.
Update your service(s) to point to the destination DB.
Enable DB writes.
Verify everything is working.
Delete origin database (and server if it was the only DB on the server).
I wouldn't use DataSynch because it creates many objects in your database to perform synchronization (it's an invasive solution). You can indeed try the Import/Export feature; that should work fine. You can also download a trial version of the Enzo backup tool, which comes with a 30-day free trial: http://www.bluesyntax.net/backup.aspx. [disclaimer: I am the author of this tool]
Regarding the pricing question, you may be charged for data being extracted out of the database. Moving data "in" SQL Azure is free of charge for now. If you are transferring the data to a different data center, you will be charged for extracting the data. It's 15 cents per GB in the US and Europe, and 20 cents in Asia. Here are the pricing details: http://www.microsoft.com/windowsazure/pricing/
Keep in mind that a database that requires 4GB of storage doesn't mean you have 4GB of data. Sometimes indexes can take a lot of space. To estimate the size of the data you will need to transfer you can either drop your indexes (and wait a little for the database size to shrink; the database size should be roughly equal to your data transfer needs) or you can calculate the size of your tables by running a command. Here is a link to an article that shows how to do something similar (look at the second command with is a SELECT statement; just run it for all the tables): http://www.sqldocumentor.com/table-size-in-sql-server-find-rows-and-disk-space-usage
Azure has released a new tool called Azure Resource Mover.
Resource mover can for now handle these resources:
Azure VMs and associated disks
NICs
Availability sets
Azure virtual networks
Public IP addresses
Network security groups (NSGs)
Internal and public load balancers
Azure SQL databases and elastic pools
https://learn.microsoft.com/en-us/azure/resource-mover/move-region-within-resource-group
Azure SQL Server is not supported yet but Azure has a complete guide for this anyway:
https://learn.microsoft.com/en-us/azure/resource-mover/tutorial-move-region-sql#move-the-sql-server