Goal:
Integrate power apps with power bi (vica verse) without setting up a SQL database for direct query to allow data input of internal tables to power bi report.
The database we are reading from cannot be used for data entry purposes.
E.g. there is a business process whereby a client must track the location of each asset. This is done by creating a table of a distinct id of each asset and then a data entry drop down is applied so users can update the location of each asset which is updated in the report visuals.
Resources:
https://www.google.com/search?q=integrate+power+bi+and+power+apps+direct+quert&oq=integrate+power+bi...
https://powerusers.microsoft.com/t5/Building-Power-Apps/Filtering-data-in-PowerApps-based-on-Power-B...
https://learn.microsoft.com/en-us/powerapps/maker/canvas-apps/powerapps-custom-visual
Solution attempt 1: creating a sharepoint list to store user entry results. An app was created and embedded within PowerBI, however the power bi report did filter results to the app. Likewise results were not immediately shown in the PowerBi report. SharePoint does not have a direct query function, and is limited to 8 refreshes a day + it could not be refreshed through the web browser, only by scheduled refreshes.
Attempted solution 2: using internal PowerBI datasets. Accessing the data from PowerBI datasets inside of powerapps seemed to be read only. Is this the case?
Question for the forum: how do you create data entry for a powerbi report without setting up another database to store results.
I am using an Azure SQL Database for our team's reporting and the data size right now is too big to handle by a single data (at least I think so, it has 2 fact tables with around 100m rows in each table).
The Azure SQL Database is named "operation-db" and the Synapse is named "operation-synapse".
I want to make the transition for my team become as smooth as possible. So I'm planning to copy all the tables, views, stored procedure and user-defined function over to Synapse.
Once I'm done with that, is there a way to rename "operation-synapse" to "operation-db" so the team doesn't have to go to their code base to change the name of the db?
Thanks!
It is not possible to rename a SQL Pool via SQL Server Management Studio and you will receive the following error:
ALTER DATABASE NAME statement is not supported in a Synapse workspace.
To update the name of a SQL pool, use the Azure Synapse Portal or the
Synapse REST API. (Microsoft SQL Server, Error: 49978)
The REST API however does list a move method to change names:
POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/sqlPools/{sqlPoolName}/move?api-version=2019-06-01-preview
I couldn't get it to work though. YMMV. Not renaming your db shouldn't be a big deal though. Your team should feel comfortable with changing connection strings etc and it will help them understand they are moving to a different product (Synapse) with different characteristics.
Before you move to Synapse however, have you look at Clustered Columnstore indexes in Azure SQL DB? They are default type of index in a SQL Pool database but are also available in SQL DB. They can compress your data 5-10x so it might end up not that big at all. Columnstore is great for aggregate queries but less so for point lookups so have a think about your workload before you migrate.
100 million rows is not big enough for synapse. Cci data in each shard will only have 1 row group (1mil rows).
Consider using partitioning or CCI in your sql db itself.
Also what's your usage pattern? If you are doing point lookups and updates clustered indexes will perform better.
You can rename a Synapse database easily using the SSMS GUI. (I've just tried this on v18.8).
Just click once on the database name in the Object Explorer to select it, then press the F2 key to rename it.
The Synapse service must be running (i.e. not paused) for the rename to work.
You can rename Synapse database using T-SQL. The command is as follows:
ALTER DATABASE [OldSynapseDBName]
MODIFY NAME = [NewSynapseDBName]
Note you need to be connected to/issue the command from the master database otherwise it will not work.
The command takes can 30 seconds on 100GB DB and there are some caveats such as DB must not be used during operation.
We have on prem SQL Server Analysis Services (SSAS) multidimension with lot of custom complex calculation, lot of measure group, complex model with many more features. We process per day few billion rows and have custom Excel add-in to connect custom pivot as well as standard Pivot table functionality used to create reports, run ad-hoc queries etc. and many more.
Below are the possible solutions in Azure
Approach 1: Azure Synapse, SSAS Multidimensional (ROLAP), Excel and Power BI. Note that SSAS Multidimensional will run as IaaS which will be host in VM. Desktop excel/excel 365 will be able to connect and Cloud Power BI.
Approach 2: Azure Synapse, Azure Analysis Services Tabular model direct query, Excel and Power BI. Desktop excel/excel 365 will be able to connect and Cloud Power BI.
Question:
Which approach will be based on the huge data volume, processing, complex logic, maintenance and custom calculation?
Can users access these cloud-based data cubes specially SSAS multidimensional either via their desktop Excel or via Excel 365?
How will be the performance ROLAP vs DAX on direct query mode?
What will be cost of moving and processing fairly large amounts of data?
With 12TB of data you will probably be looking at 500 - 1200GB of compressed Tabular model size unless you can reduce the model size by not keeping all of history, pruning unused rows from dimensions, and skipping unnecessary columns. That’s extremely large even for a Tabular model that’s only processed weekly. So I agree an import model wouldn’t be practical.
My recommendation would be a Tabular model. A ROLAP Multidimensional model still requires MOLAP dimensions to perform decently and your dimension sizes and refresh frequency will make that impractical.
So a Tabular model in Azure Analysis Services in DirectQuery mode should work. If you optimize Synapse well you should hopefully get query response times in the 10-60 second range. If you do an amazing job you could probably get it even faster. But performance will be largely dependent on Synapse. So materialized views, enabling query resultset cache, ensuring proper distributions and ensuring good quality Columnstore compression will be important. If you aren’t an expert in Synapse and Azure Analysis Services, find someone who is to help.
In Azure Analysis Services, ensure you mark relationships to enforce referential integrity which will change SQL queries to inner joins which helps performance. And keep the model and the calculations as simple as possible since your model is so large.
Another alternative if you want very snappy interactive dashboard performance for previously anticipated visualizations would be to use Power BI Premium instead of Azure Analysis Services and to do composite models. That allows you to create some smaller agg tables which are imported and respond fast to queries at an anticipated grain. But then other queries will “miss” aggs and run SQL queries against Synapse. Phil Seamark describes aggregations in Power BI well.
I am new to SQL Azure and I have a task to Implementing Auditing Techniques on SQL Server Azure database.
Can you someone please help me what are the different techniques available for Auditing data changes in SQL Azure. Any reference links will also helps.
I want to maintain Auditing for tables which has around 40-50 columns and I want to track all the column changes. I am also looking for reliability and performance factors.
Thanks
Auditing in SQL Azure is very easy to set up.Below is the data that will be captured..
Access to data
Schema changes (DDL)
Data changes (DML)
Accounts, roles, and permissions (DCL)
Stored Procedure, Login and, Transaction Management.
Once you set up Auditing,the files will be stored in Storage account which you Can download as excel file...
Now Azure gives an option to monitor Audit logs using power BI as well..
We have configured auditing for tables which are inserted heavily(1 million inserts per day minimum)..we didnt saw any performance degradation.
Updated as per comments:
Auditing is at database level as of now,if you want to audit a single table only ,triggers can be your best bet
Few links which may help you..
https://powerbi.microsoft.com/en-us/blog/monitor-your-azure-sql-database-auditing-activity-with-power-bi/
https://azure.microsoft.com/en-us/documentation/articles/sql-database-auditing-get-started/#subheading-1
Thanks tmullaney for the response. After a deep analysis I have started using Temporal Tables for enabling Auditing in SQL Azure. We can enable Auditing on individual tables/entities using this feature.
All the process will be done internally by SQL Server and no need to write even a single trigger to do the audit.
Here are couple of links that are useful to explore details on Temporal Tables in SQL Server,
Channel 9 Video : Temporal in SQL Server 2016 : https://channel9.msdn.com/Shows/Data-Exposed/Temporal-in-SQL-Server-2016
Temporal Tables : https://msdn.microsoft.com/en-IN/library/dn935015.aspx
I am at the planning stage of a web application that will be hosted in Azure with ASP.NET for the web site and Silverlight within the site for a rich user experience. Should I use Azure Tables or SQL Azure for storing my application data?
Azure Table Storage appears to be less expensive than SQL Azure. It is also more highly scalable than SQL Azure.
SQL Azure is easier to work with if you've been doing a lot of relational database work. If you were porting an application that was already using a SQL database, then moving it to SQL Azure would be the obvious choice, but that's the only situation where I would recommend it.
The main limitation on Azure Tables is the lack of secondary indexes. This was announced at PDC '09 and is currently listed as coming soon, but there hasn't been any time-frame announcement. (See http://windowsazure.uservoice.com/forums/34192-windows-azure-feature-voting/suggestions/396314-support-secondary-indexes?ref=title)
I've seen the proposed use of a hybrid system where you use table and blob storage for the bulk of your data, but use SQL Azure for indexes, searching and filtering. However, I haven't had a chance to try that solution yet myself.
Once the secondary indexes are added to table storage, it will essentially be a cloud based NoSQL system and will be much more useful than it is now.
Despite similar names SQL Azure Tables and Table Storage have very little in common.
Here are a two links that might help you:
Table Storage, a 100x cost factor
Fat Entities on Table Storage
Basically, the first question should wonder about is Does my app really need to scale? If not, then go for SQL Azure.
For those trying to decide between the two options, be sure to factor reporting requirements into the equation. SQL Azure Reporting and other reporting products support SQL Azure out of the box. If you need to generate complex or flexible reports, you'll probably want to avoid Table Storage.
Azure tables are cheaper, simpler and scale better than SQL Azure. SQL Azure is a managed SQL environment, multi-tenant in nature, so you should analyze if your performance requirements are fit for SQL Azure. A premium version of SQL Azure has been announced and is in preview as of this writing (see HERE).
I think the decisive factors to decide between SQL Azure and Azure tables are the following:
Do you need to do complex joins and use secondary indexes? If yes, SQL Azure is the best option.
Do you need stored procedures? If yes, SQL Azure.
Do you need auto-scaling capabilities? Azure tables is the best option.
Rows within an Azure table cannot exceed 4MB in size. If you need to store large data within a row, it is better to store it in blob storage and reference the blob's URI in the table row.
Do you need to store massive amounts of semi-structured data? If yes, Azure tables are advantageous.
Although Azure tables are tremendously beneficial in terms of simplicity and cost, there are some limitations that need to be taken into account. Please see HERE for some initial guidance.
One other consideration is latency. There used to be a site that Microsoft ran with microbenchmarks on throughput and latency of various object sizes with table store and SQL Azure. Since that site's no longer available, I'll just give you a rough approximation from what I recall. Table store tends to have much higher throughput than SQL Azure. SQL Azure tends to have lower latency (by as much as 1/5th).
It's already been mentioned that table store is easy to scale. However, SQL Azure can scale as well with Federations. Note that Federations (effectively sharding) adds a lot of complexity to your application. I'm also not sure how much Federations affects performance, but I imagine there's some overhead.
If business continuity is a priority, consider that with Azure Storage you get cheap geo-replication by default. With SQL Azure, you can accomplish something similar but with more effort with SQL Data Sync. Note that SQL Data Sync also incurs performance overhead since it requires triggers on all of your tables to watch for data changes.
I realize this is an old question, but still a very valid one, so I'm adding my reply to it.
CoderDennis and others have pointed out some of the facts - Azure Tables is cheaper, and Azure Tables can be much larger, more efficient etc. If you are 100% sure you will stick with Azure, go with Tables.
However this assumes you have already decided on Azure. By using Azure Tables, you are locking yourself into the Azure platform. It means writing code very specific to Azure Tables that is not just going to port over to Amazon, you will have to rewrite those areas of your code. On the other hand programming for a SQL database with LINQ will port over much more easily to another cloud service.
This may not be an issue if you've already decided on your cloud platform.
I suggest looking at Azure Cache in combination with Azure Table. Table alone has 200-300ms latencies, with occasional spikes higher, which can significantly slow down response times / UI interactivity. Cache + Table seems to be a winning combination, for me.
For your question, I want to talk about how to decide with logic choose SQL Table and which need to use Azure Table.
As we know SQL Table is a relational database engine. but if you have a big data in one table the SQL Table is not applicable, because SQL query get big data is slow.
At this time you can choose Azure Table, the Azure Table query is so fast then SQL Table for big data, for example, in our website, someone subscribed many articles, we make the article as feed to user, every user have a copy of article title and description, so in the article table there are lots of data, if we use SQL Table, each query execution maybe take more than 30 seconds. But in Azure Table get users article feed by PartitionKey and RowKey is so fast.
From this example you may know how to choose between in SQL Table and Azure Table.
I wonder whether we are going to end up with some "vendor independent" cloud api libraries in due course?
I think that you have first to define what your application usage funnels are. Will your data model be subjected to frequent changes or it is a stable one? You have to be able to perform ultra fast inserts and reads are not so complicated? Do you need advance google like search? Storing BLOBS?
Those are the questions (and not only) that you have to ask and answer yourself in order to decide if you are more likely going to use NoSql or SQL approach in storing your data.
Please consider that both approaches can easily coexist and can be extended with BLOB storage as well.
Both Azure Tables and SQL Azure are two different beasts.Both are meant for different scenarios, one con to azure table is that you cannot move from azure to any other platform, unless you write providers in your code that can handle such shifts.