Using Microsoft Purview we have scanned thousands of data assets (inside Azure Data Lake Gen2 storage and Azure SQL Managed Instance). By using Asset details page in the Microsoft Purview Data Catalog, we can manually add description of an asset as explained here. Question: since we have thousands of assets registered in Purview, is there any automated process of bulk importing of the descriptions of these assets from an Excel or csv file, if so, could someone please either point to an online link explaining such process or explain the process here.
Related
Unity Catalog is the Azure Databricks data governance solution for the Lakehouse. Whereas, Microsoft Purview provides a unified data governance solution to help manage and govern your on-premises, multicloud, and software as a service (SaaS) data.
Question: In our same Azure Cloud project, can we use Unity Catalog for the Azure Databricks Lakehouse, and use Microsoft Purview for the rest of our Azure project?
Update: In our current Azure subscription, we have divided workload as follows:
SQL related workload: we are doing all our SQL database work using Databricks only (no Azure SQL databases are involved). That is, we are using Databricks Lakehouse, Delta Lake, Deatricks SQL etc. to perform ETL and all Data Analytics work.
All Non-SQL workload: All other assets (Excel files, csv files, pdf, media files etc.) are stored in various Azure storage accounts.
MS Purview is doing a good job in scanning assets in scenario 2 above, and it easily creates a holistic, up-to-date map of our data landscape with automated data discovery, sensitive data classification, and end-to-end data lineage. It also enables our data consumers to access valuable, trustworthy data management.
However, our almost 50% of the work (SQL, ETL, Data Analytics etc.) is done in Azure Databricks where we have significant challenges with Purview. We were wondering if it's possible to keep Purview and Unity Catalog separate as follows: Purview does its Data Governance work for scenario 1 only and Unity Catalog does its Data Governance work for scenario 2 only.
This recently released update may resolve our issue of making Purview work better with Azure Databricks but we have not tried it yet: Connect to and manage Azure Databricks in Microsoft Purview (Preview)
As of right now there is no official integration between Unity Catalog and Purview yet, but it may come in the future. You may join Azure Databricks roadmap webinar that will be tomorrow to get more information.
Regarding the actual question - imho, nothing prevents you from using UC & Purview in the same Azure project.
P.S. You can get metadata & lineage information into Purview by loading data from information schema tables and using Purview APIs to store it in Purview.
Its possible to create an App or WebForm App using Power Apps to save and retrieve information from Azure DataLake, Synapse or Data Factory?
Could you give any suggestion about this implementations, please?
I appreciate any help you can share!!
Thanks so much!
There are multiple ways to import and export data into Microsoft Dataverse. You can use dataflows, Power Query, Azure Data Factory, Azure Logic Apps, and Power Automate. See, Importing and exporting data and Import by bringing your own source file
You can configure dataflows to store their data in your organizationās Azure Data Lake Storage Gen2 account. This article describes the general steps necessary to do so, and provides guidance and best practices along the way.
Although I a starting in power apps, you can checkout for further Create, edit, or configure forms using the form designer
use Dataverse in ADF..
Add source to your forms...
Azure Application Insights does not allow telemetry data retention for more then few days, however it has option called "Continuous export" which exports data into Azure Storage Blobs, so question is how do I build reports using data stored in blobs? Is there a way to use Azure Application Insight's Reporting system itself to point to blob storage as "Data Source" and see reports ?
How are others later building reports on Azure Application Insights data that is exported using "Continuous export" option ?
Regards
You can import the data from the Continuous Export into Azure Data Explorer (ADX). Here is a full article that explains this. In ADX you can keep the data as long as you want to.
Then, you can use the cross-query feature of Azure Monitor to also query data from ADX and thus you get a unified view of current and historical data.
I am new to Azure Synapse Studio.
I am working with Synapse analytics and Loaded the data from NYTaxi and successfully created Database using a loading user etc.
But once I create a Workspace in Synapse Analytics and then Launched the Azure Synapse Studio.
I could not see any database
I wanted to know how to create a Dataset
I wanted to know how to deal with PowerBI within Studio
Also related to Apache Spart etc I need help
Thanks in Advance
Vijay Perepa
With Azure Synapse Analytics (Workspace preview) deployment no SQL Pool is deployed. You can do this in the Synapse workspace (create new SQL pool), also with sample data.
A dataset can be created in the data area (tab linked). Main purpose is metadata information (e. g. for Parquet files in your attached Azure Data Lake Store or a SQL Pool table) that can be used in a Data Flow.
PowerBI: You can link a PowerBI Workspace to Azure Synapse Analytics (Manage - Linked Service). With this you can create PowerBI datasets accessing data in your SQL Pool.
As a good starting point I would recommend the Documentation. There you also find some usefull Tutorials. Lot' s of samples are available on GitHub. Hope this helps.
I am new to Azure and hence trying to understand what services to use when and how.
At the moment, I have one excel file that has couple of tabs that require some transformation to create one excel file tab (inside the source file itself - say Tab "x"). The final tab "x" created is then being useful for creating one final excel file that is shared to various team.
At present, everything is done manually.
This needs to change and the excel file shared to team has to be automated. The source of the file is the excel file that has various tabs (excluding tab "x") and the reporting tool will be SSRS with excel data being stored in cloud.
Keeping this scenario in mind, what is the best way to store excel data into cloud? The excel data will be stored in cloud on a monthly basis. I am confused as to whether to store data in Azure-SQL, Azure Data Lake Gen 2 or Azure Data Lake Analytics or Azure SQL VM?
Every month data can be fetched from Excel file and populate into Azure using azure data factory. But I am not sure what is the best way to store data in the cloud considering the fact that some ETL process is needed to generate data in format similar to tab "X".
I think you can think about to using Azure SQL database.
Azure SQL database or SQL server support you import data from the excel( or csv) files. For more details and limits, please see: Import data from Excel to SQL Server or Azure SQL Database.
If your data have stored in Azure SQL database, you also can using EXCEL to get the data from Azure SQL database:
Connect Excel to a single database in Azure SQL Database and import data and create tables and charts based on values in the database. In this tutorial you will set up the connection between Excel and a database table, save the file that stores data and the connection information for Excel, and then create a pivot chart from the database values.
Reference: Import data from Excel to SQL Server or Azure SQL Database.
I think you don't need to store these excel files in Azure Data Lake.Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob storage. It's still a storage.
The more Azure resource you use, the more cost you need to pay.
If your excel file stored in you local computer, you can using Azure Data Factory to access these local files or with self host integration runtime.
Please referenceļ¼ Copy data to or from a file system by using Azure Data Factory.
Hope this helps.
Your storage requirements are very minimal, so I would select Data Lake to store your documents. The alternative is Blob Storage, but I always prefer Data Lake because it works with Azure Active Directory.
In your scenario, drop it in the ADL, and use the ADL as the source in Azure Data Factory.
Edit:
Honestly, your original post is a little confusing. You have a RAW Excel document, you do some transformations on the RAW document, to generate an Excel Source document. This source document holds the final dataset that the dev team will use to build out SSRS reports. You need to make this dataset available to the teams so that they can connect to it to build the reports? My suggestion is to keep it simple and drop the final source dataset in Excel format, into blob or data lake storage and then ask the dev guys to pick it up from the location. If you are going the route of designing and maintaining a data pipeline (Blob > Data Factory > SQL, or CSV, TSV - then you are introducing unnecessary complications.