Hopefully someone can help. I have a requirement of creating something like a "bring your own data model" feature/steps so clients can upload their own files, use it in a PowerBI data flow, link it with other sources (databases, services, etc), and create reports from it. The only problem is I'm having problems creating a data flow from an Azure blob file.
The files are CSV format
I have premium feature
I first tested and created an M query in PowerBI Desktop and it works, I can access the data
What I do is I copy my created M query to the Power BI data flow data source in a blank query. The result will is I can preview the data, but if I try to save it, I'm getting this error:
I tried the steps on this link but no luck.
Any ideas?
Related
I have data source from external (KoboTool Box). This data source provides conversion to excel file (From Web connection) by put specific web link. After I got all data, I put these excel file in the Azure blob storage.
By using Logic App, I have got the file in the blob container but I have no idea how to refresh this excel file to retrieve external data. There are two Excel Online (One drive & Business). I don't know what next steps.
My question, is it possible to refresh this excel file to get the latest update data from external data source by using logic app ?
I've setup a connection from our data factory in azure to a sharepoint site so I can pull some of the lists on the site into blob storage so I can then process into our warehouse. This all works fine and I can see the data I want. I don't however want to pull all the columns contained in the list I'm after. Looking at the connection I can specify a query, however anything I put in here has no affect on the data that comes back. Is there a way to specify the columns from a list in sharepoint through the copy activity into blob storage?
You need to user select query like below -
$select=Title,Number,OrderDate in the Query text field of the Azure Data Factory Source.
You could use Preview Data button to validate the results. Please refer to the documentation for using Custom OData query options.
I have tried this and it works fine for me (see screenshot below)-
Thanks
Saurabh
I have a query being executed in a Azure server periodically and I need to add some code to it, so it can save some data from Tables/Views to a Excel file during the execution.
I have implemented some code like this on other databases (non-Azures), but executing the same code in Azure gives me messages like "Azure doesn't support" some of the tools I used.
What should I use to do this? I just got to save some Tables data to specific sheets in Excel.
Thanks in advance!
In case if the requirement is specific to Excel file creation ; you can use a logic app to query database from Azure SQL database and generate Excel file based on the below link:
https://community.dynamics.com/ax/b/d365fortechies/posts/logic-app-for-azure-sql-db-to-azure-file-storage-workflow
Note: You can select Excel file generation for Logic app rather than CSV mentioned in the above example or generate an CSV file and then convert into Excel
Since OPENDATASOURCE is not supported in Azure SQL. You also can use other ETL tools to save some data from Tables/Views to a Excel.
Such as Azure data factory:
Using Copy activity in Azure data factory, you can query from table, execute your sql query and execute stored procudure then convert to a Excel file. There are multiple destinations for you to choose to store this excel, cloud or local server.
I have a working query for my app data to be analyzed.
currently it analyzes the last two weeks data with an ago(14d).
Now i want to use a value containing the release date of the apps current version. Since i havent found a way to add a new database table to the already existing database containing the log data in azure analytics, i created a new database in azure and entered my data there.
Now i just don't know, if i can get access to that database at all from within the web query interface of Azure log analytics, or if i have to use some other tool for that?.
i hope that somebody can help me on this.
As always with azure there is a lot of stuff to read about it, but nothing concrete for my issue (or at least i haven't found it yet).
And yes, i know how to insert the data into the query with a let, but since I want to use the same data in different queries, an external location which can be accessed from all the queries would be the solution I prefer.
Thx in advance.
Maverick
You cannot access a db directly. You are better of using a csv/json file in blob storage. In the following example I uploaded a txt file with csv data like this:
2a6c024f-9093-434c-b3b1-000821a15b1a,"Customer 1"
28a658a8-5466-45ea-862c-003b20507dd4,"Customer 2"
c46fb949-d807-4eea-8de4-005dd4beb39a,"Customer 3"
e05b67ee-ff83-4805-b004-0064449f196c,"Customer 4"
Then I can reference this data from log analytics / application insights in a query like this using the externaldata operator:
let customers = externaldata(id:string, companyName:string) [
h#"https://xxx.blob.core.windows.net/myblob.txt?sv=2019-10-10&st=2020-09-29T11%3A39%3A22Z&se=2050-09-30T11%3A39%3A00Z&sr=b&sp=r&sig=xxx"
] with(format="csv");
requests
| extend CompanyId = tostring(customDimensions.CustomerId)
| join kind=leftouter
(
customers
)
on $left.CompanyId == $right.id
The url https://xxx.blob.core.windows.net/myblob.txt?sv=2019-10-10&st=2020-09-29T11%3A39%3A22Z&se=2050-09-30T11%3A39%3A00Z&sr=b&sp=r&sig=xxx is created by creating a url including a SAS token by using the Microsoft Azure Storage Explorer, selecting a blob and then right click -> Get Shared Access Signature. In the popup create a SAS and then copy the uri.
i know Log Analytics uses Azure Data Explorer in the back-end and Azure Data Explorer has a feature to use External Tables within the queries but I am not sure if Log Analytics support External Tables.
External Tables in Azure Data Explorer
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/schema-entities/externaltables#:~:text=An%20external%20table%20is%20a,and%20managed%20outside%20the%20cluster.
I am new to Azure and hence trying to understand what services to use when and how.
At the moment, I have one excel file that has couple of tabs that require some transformation to create one excel file tab (inside the source file itself - say Tab "x"). The final tab "x" created is then being useful for creating one final excel file that is shared to various team.
At present, everything is done manually.
This needs to change and the excel file shared to team has to be automated. The source of the file is the excel file that has various tabs (excluding tab "x") and the reporting tool will be SSRS with excel data being stored in cloud.
Keeping this scenario in mind, what is the best way to store excel data into cloud? The excel data will be stored in cloud on a monthly basis. I am confused as to whether to store data in Azure-SQL, Azure Data Lake Gen 2 or Azure Data Lake Analytics or Azure SQL VM?
Every month data can be fetched from Excel file and populate into Azure using azure data factory. But I am not sure what is the best way to store data in the cloud considering the fact that some ETL process is needed to generate data in format similar to tab "X".
I think you can think about to using Azure SQL database.
Azure SQL database or SQL server support you import data from the excel( or csv) files. For more details and limits, please see: Import data from Excel to SQL Server or Azure SQL Database.
If your data have stored in Azure SQL database, you also can using EXCEL to get the data from Azure SQL database:
Connect Excel to a single database in Azure SQL Database and import data and create tables and charts based on values in the database. In this tutorial you will set up the connection between Excel and a database table, save the file that stores data and the connection information for Excel, and then create a pivot chart from the database values.
Reference: Import data from Excel to SQL Server or Azure SQL Database.
I think you don't need to store these excel files in Azure Data Lake.Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob storage. It's still a storage.
The more Azure resource you use, the more cost you need to pay.
If your excel file stored in you local computer, you can using Azure Data Factory to access these local files or with self host integration runtime.
Please referenceļ¼ Copy data to or from a file system by using Azure Data Factory.
Hope this helps.
Your storage requirements are very minimal, so I would select Data Lake to store your documents. The alternative is Blob Storage, but I always prefer Data Lake because it works with Azure Active Directory.
In your scenario, drop it in the ADL, and use the ADL as the source in Azure Data Factory.
Edit:
Honestly, your original post is a little confusing. You have a RAW Excel document, you do some transformations on the RAW document, to generate an Excel Source document. This source document holds the final dataset that the dev team will use to build out SSRS reports. You need to make this dataset available to the teams so that they can connect to it to build the reports? My suggestion is to keep it simple and drop the final source dataset in Excel format, into blob or data lake storage and then ask the dev guys to pick it up from the location. If you are going the route of designing and maintaining a data pipeline (Blob > Data Factory > SQL, or CSV, TSV - then you are introducing unnecessary complications.