Are there any tutorials available on the subject of migrating from an existing BI-stack based on SQL Server 2008 to Azure SQL Data Warehouse? I'm specifically interested in best practices with regards to how to handle cross database joins on non-premium tiers (our existing procedures and UDFs are full of joins on multiple database objects) and how to migrate existing SSAS cubes and its related programmability and ETL.
What BI-stack are you using? This will determine your next steps for the actual BI tools.
Specifically for cross-database queries when moving to the cloud, the guidance is to move the databases into schemas and then update your scripts to use schema based (2 part names) vs. database (3 part names) when referencing objects. For example, if you have staging and production databases you can simply move your staging objects into a [staging] schema within a single database.
Azure SQL Data Warehouse is commonly used as a backing store for SSAS cubes (MOLAP/ROLAP/Tabular mode). In the Azure cloud, customers have created IaaS SQL Server VMs to host ETL process (SSIS) and cubes (SSAS) with direct connections to SQL Data Warehouse.
Related
I have several SQL DBs in Azure. All have the same structure. Each DB represents a different location. What would be the best practice to aggreate the data of all locations? The goal would be to be able to answer queries like „How much material of type X was used in time range x to y accross all locations?“ or „Give me the location that produces the highest outcome?“
You can use the Azure SQL database Elastic pool.
Add all of your databases to the Elastic pool.
With Elastic query can help you aggreate the data of all locations in Azure.
The elastic query feature (in preview) enables you to run a Transact-SQL query that spans multiple databases in Azure SQL Database. It allows you to perform cross-database queries to access remote tables, and to connect Microsoft and third-party tools (Excel, Power BI, Tableau, etc.) to query across data tiers with multiple databases. Using this feature, you can scale out queries to large data tiers in SQL Database and visualize the results in business intelligence (BI) reports.
Hope this helps.
My recommendation in this scenario is to create a new database which we will name as the "hub" database and it will consolidate the information of all location databases which we will name as "member" databases. Use SQL Data Sync to synchronize each member database to the hub database. Use T-SQL and Power BI against the hub database to answer all your questions involving all locations.
I participated on a project of a Mexican retail store with 72 stores across Mexico they created a hub database to consolidate sales at the end of the day, and use Power BI to show consolidated sales to stakeholders.
I am new to Azure. I would like to learn the architecture deployed in my company which i shown below on diagram. Can anyone point me to some video example or something that could reflect that from diagram below. I also have access to Azure portal that i have some money credit so if it is possible i could create some test environment based on that diagram.
P.S Is it possible to use Visual Studio for any kind of work based on that diagram or everything have to be created and develop from Azure portal?
Datasource Oracle DB --> on prem gateway --> ADF--> Azure DB --> AAS --> PowerBI
SQL EDP --------------------------------------^
You've got a fairly straightforward BI architecture there with the following logical components:
raw / source data
integration
data mart / dimensional model
semantic
visualisation
The physical components look a bit like this:
The physical components can be described like this:
Oracle database - former market leader database product. I would guess your employers have rejected OBIEE for some reason
Self-hosted Integration Runtime (SHIR)On-premises data gateway - the SHIR gateway enables the movement of data from on-prem data sources to the cloud. This must be used when moving data from on-prem to Azure SQL DB using Data Factory. Use the SHIR with Data Factory and the Gateway with Power BI and Azure Analysis Services.
Data Factory - Azure ELT tool for moving data from place to place. ETL feature Data Flow currently in preview.
Azure SQL DB - PaaS SQL database, scalable via service tiers. If your data in Oracle is not already in a data mart / dimensional format, then it can be made so here
Azure Analysis Services (AAS) - PaaS OLAP in-memory engine, scalable for fast slice-and-dice, drill down and semantic modelling. Tabular only.
Power BI - increasingly powerful visualisation tool. Run dashboard in DirectQuery / LiveConnection mode to avoid entirely duplicating the tabular model from AAS in Power BI.
In answer to some of your questions: you can have one Azure Data Factory with many pipelines. The Visual Studio Azure Data Factory project type is now defunct.
As to "why" for certain technologies:
why Oracle - Who knows.
why SHIR - SHIR is compulsory when moving data from on-prem to cloud with ADF
why Azure SQL DB - lightweight and powerful PaaS DB requiring no infra and low TCO; scalable. Might be location for restructuring of data from raw / relational structure to dimensional in readiness for semantic layer if your data is not already in that format in Oracle
why AAS - fast, in-memory slice-and-dice; scalable, can pause, can be interrogated by Excel, Power BI Desktop, SSMS, VS, other clients etc. Optionally has row-level security (RLS)
Power BI - online service Power BI.com offers easy sharing within organisation, even externally.
why all the components together - you could (in theory) go straight from Oracle to Power BI with a Power BI gateway (I think) BUT you would then have to do all the modelling in Power BI and your model is then only really accessible from Power BI. In this model, users with SQL skills can query the data mart, users with DAX (or Excel, or Power BI Desktop) skills can query the AAS tabular model, AAS is very scalable component, etc
These opinions are strictly my own personal ones and the value of them may go down, as well as up.
HTH
Azure Data Factory has a 1:M capability with various data sources. One instance of Azure Data Factory will support multiple data movement capabilities: Data movement activities
Information about On-Premise Gateway:
The on-premises data gateway acts as a bridge, providing secure data transfer between on-premises data sources and your Azure Analysis Services servers in the cloud. In addition to working with multiple Azure Analysis Services servers in the same region, the latest version of the gateway also works with Azure Logic Apps, Power BI, Power Apps, and Microsoft Flow. You can associate multiple services in the same subscription and same region with a single gateway.
Connecting to on-premises data sources with Azure On-premises Data Gateway
Can I do cross database queries from a Azure SQL Database to Azure SQL Data warehouse, I read somewhere that its possible around sql databases but can it be possible in azure datawarehouse and sql database.
It is a "thing" with MS SQL databases, achieved via links using special credentials and external tables (which essentially map a foreign DB's tables to the local DB). It didn't used to be possible in Azure, but it is nowadays.
For more info, see here
I created an SQL database using ASP.NET Core 1.1 Migrations.
After I created the database I added some data to the database.
What options do I have to upload this database to Azure?
I need to send the Scheme and the initial data.
Is it possible to run Entity Framework migrations on Azure?
This article describes the possibilities to migrate an existing database to SQL Azure.
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-cloud-migrate
However, in your scenario, this might be overkill to go through the steps of realy doing a migration.
If your number of tables and data is rather small, why not create a SQL script to create the tables & insert the data?
Connect to your SQL Azure using SQL Server Management Studio and execute the script.
As for the Entity Framework, yes, you can run those on SQL Azure as well.
We are investigating moving to a cloud model and the Microsoft Azure PaaS platform.
Our current data layer is written using LINQ to SQL. Will it be compatible with SQL Azure?
Yes LINQ to SQL is compatible. But you have to keep 2 things in mind:
SQL Azure is not a dedicated machine for you alone. Your databases will run on machines with other customers. You may be throttled, you can encounter transient exceptions, ... That's why you should implement a retry policy. There's an article on TechNet explaining how to do this with LINQ to SQL.
Since SQL Azure is not a real SQL Server there are some limitations. Read all about those limitations here: General Guidelines and Limitations (Windows Azure SQL Database). If you're planning to move an existing database (with or without data) to SQL Azure, consider using a tool that supports SQL Azure like the SQL Database Migration Wizard (SQLAzureMW).