Let's say I have a mydev.database.windows.net Azure SQL Server and Azure SQL DW database for development. And I have a myprod.database.windows.net for prod. If I want to restore prod to dev (cross server) is that possible? From what I can see in the documentation (see the -TargetServerName switch documentation), it is not possible.
Are there recommended workarounds other than scripting out all the objects then using a Polybase CREATE EXTERNAL TABLE AS SELECT command to export all tables to blobs then import those tables with Polybase?
The recommended approach to cross server restores with Azure SQL Database (not DW) is to export to a bacpac file then restore, but I don't believe that's an option for Azure SQL DW right?
I may start creating prod and dev on the same Azure SQL Server (as long as the customer wants both in the same Azure subscription). I would prefer the servers be separate, but ease of restore is important.
This will depend on the frequency and freshness of the restores today. The simplest approach is to restore one of the snapshots we take in the background to support RPO. This is called geo-restore. Snapshots are taken at least every eight hours. However, in practice you will see these taken more frequently. As RPO improves over time so will the frequency of snapshots.
To perform a geo-restore of production into dev you can go to the portal and begin the provisioning process. In the provisioning blade for SQL DW select your dev server. Under select source choose "backup". This will extend the provisioning blade as you will need to then choose the backup you want to use. The rest should be straight forward.
If you need to do this much more frequently or against an "on demand" (i.e. times of your choosing) snap then you would need to build out custom code as you suggest. However, if you are ok to live with our snapshots then the geo-restore would be a good option.
The team are looking for customer feedback on RPO and backup / restore requirements. If you have a business need for more frequent snapshots to support a business case then the team would love to hear from you. Please post this on our user voice feedback channel or reach out to us directly at sqldwfeedback#microsoft.com if business sensitive.
Related
I am working on modernizing a reporting solution where the data sources are on prem on the customers' sql servers (2014) and the reports are displayed as Power BI reports on the customer's Power BI Service portal. Today I use SSIS to build a data warehouse, as well as an on premise data gateway to ensure the transport of data up to an Azure analysis services which in turn are used by the Power BI reports.
I have been wondering if I could use Azure Synapse to connect to customer data and in a most cost effective way transport data to Azure and link them to the Power BI workspace as a shared dataset. There are many possibilities, but it is important that the customer experiences that the reports are fast and stable, and if possible can cope with near real time.
I experience SSIS being cumbersome and expensive in azure. Are there mechanisms that make it cheap and fast to get data in azure? Do I need a data warehouse (Azure SQL database) or is it better to use data lake as storage for data? Needs to do incremental load too. And what if I need to do some transformations? Should I use Power BI dataflow or do I need to create Azure Data flows to achieve this?
Does anyone have good experience to use synapse (also with DevOps in mind) and get a good DEV, TEST and Prod environment for this? Or is using Synapse a cost driver and a simpler implementation will do? Give me your opinions and if you have links to good articles, please do so. Looking forward to hear from you
regards Geir
The honest answer is it depends on a lot of different things and I don't know that I can give you a solid answer. What I can do is try to focus down which services might be the best option.
It is worth noting that a Power BI dataset is essentially an Analysis Services database behind the scenes, so unless you are using a feature that is specifically only available in AAS and using a live connection, you may be able to eliminate that step. Refresh options are one of the things that are more limited in Power BI though, so the separate AAS DB might be necessary for your scenario.
There is a good chance that Power BI dataflows will work just fine for you if you can eliminate the AAS instance, and they have the added advantage of have incremental refresh as a core feature. In that case, Power BI will store the data in a data lake for you.
Synapse is an option, but probably not the best one for your scenario unless your dataset is large, SQL pools can get quite expensive, especially if you aren't making use of any of the compute options to do transformations.
Data Factory (also available as Synapse pipelines) without the SSIS integration is generally the least expansive option for moving large amounts of data. It allows you to use data flows to do some transformations and has things like incremental load. Outputting to a Data Lake is probably fine and the most cost effective, though in some scenarios something like an Azure SQL instance could be required if you specifically need some of those features.
If they want true real time, it can be done, but none of those tools really are built for it. In most cases the 48 refreshes per day (aka every 30 minutes) available on a Premium capacity are close enough to real time once you dig into the underlying purpose of a given report.
For true real time reporting, you would look at push and/or streaming datasets in Power BI and feed them with something like a Logic App or possibly Stream Analytics. There are a lot of limitations with push datasets though- more than likely you would want to set up a regular Power BI report and dataset and then add the real time dataset as a separate entity in addition to that.
As far as devops goes, pretty much any Azure service can be integrated with a pipeline. In addition to any code, any service or service settings can be deployed via an ARM template or CLI script.
Power BI has improved in the past couple years to have much better support for devops and dev/test/prod environments. Current best practices can be found in the Power BI documentation: https://learn.microsoft.com/en-us/power-bi/create-reports/deployment-pipelines-best-practices
There has been a previous question on this and the accepted answer was Azure Elastic Job agent. The problem I have is that the feature is in preview and it still lacks a lot of functionality like diagnostics and alerting. I also find it to be very unreliable as job get randomly cancelled because of service restarts.
Azure Automation Accounts also works, but it only has a execution/running time of 3 hours. So if your maintenance takes more than 3 hours, this is not an option.
I have previously developed my own application for doing this, but the maintenance and management of this can become a headache.
Another alternative could be to just leverage Azure Data Factory perhaps, but this is a route I have not yet followed.
So what are people actually using to do long running maintenance against Azure SQL Databases that has enough diagnostic information in case something goes wrong and has at least some level of alerting?
PS: The database I need to do maintenance on is not small.
Is there any way to stop a SQL Azure DB so that it doesn't charge anything towards our account? I don't want to delete it, just while in testing and it's not being used than we set it to "stopped" like we can do with websites, cloud services, and VMs.
As of 10th February, 2023, The answer is No.
They won't allow it. So the billing will continue for your Azure Database starting the day you create it. There really is no way to pause / stop billing for your Azure SQL Database.
Official Source: feedback.azure.com Please add ability to temporarily turn off/on SQL Azure server to pause billing
Microsoft's official answer appears to be "Yes, you can export your database. Delete the Azure SQL database and that will pause billing. Then when you need it you can create a new database and import your previously expored DB."
I don't believe this is acceptable as an answer for "Allow me to temporarily turn off SQL Server to save on my billing"
This is not an option today - the only choice you have is to reduce the size of the Azure SQL Database which will reduce the cost from the next hour of service. If you really don't want to pay for the DB you could backup the DB to blob storage, delete the database and then restore when required. You could orchestrate this using PowerShell or similar.
Update May 2019: There is a new Azure SQL Database "Serverless" tier coming that might meet some of the requirements around reducing costs by not billing when not in use. Official documentation is available to read.
The Azure SQL Database team is happy to announce that there is now an option that may address your request. We just announced a "serverless" option for Azure SQL DB that will pause your database when it is not in use. You can read more about the feature here:
SQL Database Serverless
The databases get backed up automatically just before a drop. so, you can just drop it when you dont need it and restore it when needed.
Restores will take some time depending on the database size and how much log you generated, so it wont be fast for large databases.
Also, there is an expiration policy on how long the backups are retained (depends on the service tier) so just watch out for that.
https://msdn.microsoft.com/en-us/library/azure/jj650016.aspx
Agree with #Shiva answer.
But if you are simply trying out SQL Server on an Azure VM, you would not want to incur charges by accidentally leaving it running over the weekend or weeks. One solution is to use Automatic Shutdown Feature.
This is now possible and the preview feature is public.
Azure SQL Database serverless
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-serverless
Whereas, at this low scale (scale down requirement (pause) rather than scale up), SQL running within a VM might be a better answer for you...
As is it is not supported however there are a few work arounds. It really depends upon how long you want to stop it and how immediately you want it and the size of the database. If it is for a couple hours then it may not worth it because the billing is hourly and you may have round off issues. If days then you can drop it and restore when needed. If weeks then exporting the database and importing when needed is another option. Also you may want to check backup strategy for the edition you choose before preferring export / restore.
The other thing to note is the immediate need vs planned. If immediate, and the db is big then make sure the SLAs in place works for you
You could export the database to Azure storage and Import it when you want to re-enable it, as suggested here:
Temporarily turn off on sql
Here's what I did with my Azure database (4/20/19)
I scaled the database DTUs as low as possible while still reserving up to 250GB, which turns out to be 10 DTUs with an estimated cost of 1.50 per DTU (estimated at $15 per month). To me, that's almost as good as turning it off and on. I can scale it up to 100 DTUs when I want to get more processing and scale it down when I don't. Even at 100 DTUs for a whole month, the cost was only $75.93 last month on my test database.
NOTE: I am NOT using a VM to run a database, just the standard SQL server you get when you create a database in Azure.
Yes you can, with Azure Serverless SQL Database. While your compute resources will be suspended when the database is not in use, you'll definitely save the costs for compute resources, however the billing will continue for storage resources. You can set the inactive timeout for the database after which the compute resources will be suspended. This timeout can be as low as 1 hour.
Read this: https://azure.microsoft.com/en-in/updates/update-to-azure-sql-database-serverless-providing-even-greater-price-optimization/
Elastic PoolIf you have more than one database you can use the Elastic Pool option to bring your total cost down.
Others also mention the option to Drop your database, and rely on restore. That will also work, if you do not leave it deleted for too long...
I'm working on a quiet large and critical application. It's been deployed to azure with 3 web roles and sql azure db.
In case of disaster, we need to be able to restore both web roles and sql azure to different data centers. Could someone please help me how we can restore SQL Azure DB and Web Role(s) to different data center.
The simple answer is that you take regular backups of your SQL Azure database, which can be restored to a database in another datacenter. You will have a problem with the data since the last backup being lost, which becomes a more difficult problem to resolve — the simplest may be to have a hot standby and use SQL Database Data Sync, but it may not be practical for all the data. Web roles are easier — you redeploy them somewhere else, and change the connection strings to the database. You would also have to change the CNAME for your domain as they will be restored to a different cloudapp.net name.
You did ask for restore, and not failover, right? Performing a failover (where you have a hot standby) is a more difficult problem, particularly as far as data synchronisation is concerned.
I would go back and question 'disaster' and correlate with known facts. I am not sure of the outage history of Azure in specific data centres, but there have been significant Azure-wide outages (leap year 2012 and the certificate problem this year). The ability to restore to a different Azure datacentre won't help you in these scenarios. (Although AWS seems to mostly have regional outages) I don't think that a datacenter-specific recovery strategy is necessary on Windows Azure, but you may want to check the history and likelihood of datacenter-specific failures before making a final call. Having a multi-region architecture that distributes load and data across datacentres, and handles live traffic across all (say using traffic manager), has many benefits — of side effect being builtin-disaster recovery - but comes at an architectural, development, hosting and bandwidth cost.
Go back and write the business case for your datacenter disaster recovery scenario. You may find that it is not worth it financially, or doesn't solve your real problem.
As far as I know the key points to migrate an existing database to SQL Azure are:
Tables has to contain a clustered
index. This is mandatory.
Schema and data migration should be
done through data sync, bulk copy,
or the SQL Azure migration
wizard, but not with the restore option in SSMS.
The .NET code should handle the
transient conditions related with
SQL Azure.
The creation of logins is in the
master database.
Some TSQL features may be not
supported.
And I think that's all, am I right? Am I missing any other consideration before starting a migration?
Kind regards.
Update 2015-08-06
The Web and Business editions are no longer available, they are replaced by Basic, Standard and Premium Tiers.
.CLR Stored Procedure Support is now available
New: SQL Server support for Linked Server and Distributed Queries against Windows Azure SQL Database, more info.
Additional considerations:
Basic tier allows 2 GB
Standard tier allows 250 GB
Premium tier allow 500 GB
The following features are NOT supported:
Distributed Transactions, see feature request on UserVoice
SQL Service broker, see feature request on UserVoice
I'd add in bandwidth considerations (for initial population and on-going bandwidth). This has cost and performance considerations.
Another potential consideration is any long running processes or large transactions that could be subject to SQL Azure's rather cryptic throttling techniques.
Another key area to point out are SQL Jobs. Since SQL Agent is not running, SQL Jobs are not supported.
One way to migrate these jobs are to refactor so that a worker role can kick off these tasks. The content of the job might be moved into a stored procedure to reduce re-architecture. The worker role could then be designed to wake up and run at the appropriate time and kick off the stored procedure.