I have a query being executed in a Azure server periodically and I need to add some code to it, so it can save some data from Tables/Views to a Excel file during the execution.
I have implemented some code like this on other databases (non-Azures), but executing the same code in Azure gives me messages like "Azure doesn't support" some of the tools I used.
What should I use to do this? I just got to save some Tables data to specific sheets in Excel.
Thanks in advance!
In case if the requirement is specific to Excel file creation ; you can use a logic app to query database from Azure SQL database and generate Excel file based on the below link:
https://community.dynamics.com/ax/b/d365fortechies/posts/logic-app-for-azure-sql-db-to-azure-file-storage-workflow
Note: You can select Excel file generation for Logic app rather than CSV mentioned in the above example or generate an CSV file and then convert into Excel
Since OPENDATASOURCE is not supported in Azure SQL. You also can use other ETL tools to save some data from Tables/Views to a Excel.
Such as Azure data factory:
Using Copy activity in Azure data factory, you can query from table, execute your sql query and execute stored procudure then convert to a Excel file. There are multiple destinations for you to choose to store this excel, cloud or local server.
Related
I have data in an Excel file (.xlsx). I have to get this data into Azure SQL Database. Table already exists. I am using Azure Database Studio.
I have tried online query generator but it is mixing up the data.
There are multiple approaches to import Excel file into Azure SQL DB. Below I am showing one approach of using Copy activity in Azure Data Factory.
Sample Excel file with 3 columns.
SQL DB with table having same 3 fields.
Use Excel file as Source.
Use Azure SQL database as Sink.
For mapping click on Import Schema and do the mapping.
Finally run pipeline.
Expected Output:-
Receiving an excel file every day with one sheet(Sheet name will be the different every time) and that will be stored in Azure Blob container and is there any possibility to convert the excel to CSV either by using SSIS script Task or Azure Logic Apps.
Any help would be appreciated. Thank you.
There are many ways we can do that.
With Logic app, you could ref the answer here:
Converting should be pretty easy. On high level, you can do following:
Use Excel connector to read into the content of the excel file
Use Data Operations - Create CSV Table to create a CSV format populated with dynamic data from step #1
Use Azure Blob Connector to create and save the new csv file on the blob storage
Since the excel is stored in Blob Storage, I would suggest you use Data factory, it supports Excel file directly:
Create the Source dataset:
Create Sink dataset: set the new csv file name:
Copy active overview:
It works well and very easy and directly:
I am new to Azure and hence trying to understand what services to use when and how.
At the moment, I have one excel file that has couple of tabs that require some transformation to create one excel file tab (inside the source file itself - say Tab "x"). The final tab "x" created is then being useful for creating one final excel file that is shared to various team.
At present, everything is done manually.
This needs to change and the excel file shared to team has to be automated. The source of the file is the excel file that has various tabs (excluding tab "x") and the reporting tool will be SSRS with excel data being stored in cloud.
Keeping this scenario in mind, what is the best way to store excel data into cloud? The excel data will be stored in cloud on a monthly basis. I am confused as to whether to store data in Azure-SQL, Azure Data Lake Gen 2 or Azure Data Lake Analytics or Azure SQL VM?
Every month data can be fetched from Excel file and populate into Azure using azure data factory. But I am not sure what is the best way to store data in the cloud considering the fact that some ETL process is needed to generate data in format similar to tab "X".
I think you can think about to using Azure SQL database.
Azure SQL database or SQL server support you import data from the excel( or csv) files. For more details and limits, please see: Import data from Excel to SQL Server or Azure SQL Database.
If your data have stored in Azure SQL database, you also can using EXCEL to get the data from Azure SQL database:
Connect Excel to a single database in Azure SQL Database and import data and create tables and charts based on values in the database. In this tutorial you will set up the connection between Excel and a database table, save the file that stores data and the connection information for Excel, and then create a pivot chart from the database values.
Reference: Import data from Excel to SQL Server or Azure SQL Database.
I think you don't need to store these excel files in Azure Data Lake.Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob storage. It's still a storage.
The more Azure resource you use, the more cost you need to pay.
If your excel file stored in you local computer, you can using Azure Data Factory to access these local files or with self host integration runtime.
Please referenceļ¼ Copy data to or from a file system by using Azure Data Factory.
Hope this helps.
Your storage requirements are very minimal, so I would select Data Lake to store your documents. The alternative is Blob Storage, but I always prefer Data Lake because it works with Azure Active Directory.
In your scenario, drop it in the ADL, and use the ADL as the source in Azure Data Factory.
Edit:
Honestly, your original post is a little confusing. You have a RAW Excel document, you do some transformations on the RAW document, to generate an Excel Source document. This source document holds the final dataset that the dev team will use to build out SSRS reports. You need to make this dataset available to the teams so that they can connect to it to build the reports? My suggestion is to keep it simple and drop the final source dataset in Excel format, into blob or data lake storage and then ask the dev guys to pick it up from the location. If you are going the route of designing and maintaining a data pipeline (Blob > Data Factory > SQL, or CSV, TSV - then you are introducing unnecessary complications.
I have an Azure database (using SQL Database), and also a separate device that measures floats (not relevant to the question).
As and when the data is being updated, say once every 5 minutes, I wish to update the database so that a new row is being formed with this data. I then intend to connect to PowerBI using the Azure database to form graphs etc.
As mentioned in the title, what would be the best practice? I have done my due diligence and it seems that the best way would just be to update the Azure database. Or should I consider updating a CSV file, then connect the CSV file to the Azure database and update it from there?
Reason why I'm considering to go the CSV file route is because I see that Excel has in-built refresh function, but I couldn't find anything from the Azure side.
https://support.office.com/en-ie/article/refresh-an-external-data-connection-in-excel-1524175f-777a-48fc-8fc7-c8514b984440
If you want to use Excel, you can see this Azure official document: Connect Excel to a single database in Azure SQL database and create a report.
Connect Excel to a single database in Azure SQL Database and import data and create tables and charts based on values in the database. In this tutorial you will set up the connection between Excel and a database table, save the file that stores data and the connection information for Excel, and then create a pivot chart from the database values.
Then, you can use the "Refresh Data" and try the tutorial you have found.
Hope this helps.
I have read this SO question but mine is quite specific to the "import" of CSV and not how to access the blob to get the CSV out
Which is the best way?
1) CSV Stored in the Blob - use a worker role, read the CSV from the blob, parse data and update database
2) Is SQL BulkCopy/BulkInsert an option. The challenge here is that it should not have any on-premise involvement. All within Azure: blob->SQL DAtabase.
3) Will Azure Automation help? Are there PS scripts/workflows that help in such bulk update of CSV data to Azure SQL DB? I haven't found any though
Are there other options that help import blob CSV data to SQL DB without having to write custom code?
Appreciate any thoughts...
Your first method would work. You could also use azcopy (http://aka.ms/azcopy) to download the file locally, and then use BCP to load it into SQL - this way you wont have to write any code for this.
Azure Automation would help if you want to do this repeatedly. You should be able to set this up as a script even if one doesn't exist.
I know this is outdated question but for anyone looking for quick way to do this feel free to check my article on how to do this quickly using SQL prodecure triggered by Logic App.
In short you run on master
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'UNIQUE_STRING_HERE'
Then you run on DB
CREATE DATABASE SCOPED CREDENTIAL BlobCredential
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET = 'sv=SAS_TOKEN_HERE';
CREATE EXTERNAL DATA SOURCE AzureBlob
WITH (
TYPE = BLOB_STORAGE,
LOCATION = 'https://<account_name>.blob.core.windows.net/<container_name>',
CREDENTIAL = BlobCredential
);
And then
BULK INSERT <my_table>
FROM '<file_name>.csv'
WITH (
DATA_SOURCE = 'AzureBlob',
FORMAT = 'CSV',
FIRSTROW = 2
);
Just wrap this insert in procedure and execute it from logic app.
https://marczak.io/posts/azure-loading-csv-to-sql/
or just use ADF like here
https://azure4everyone.com/posts/2019/07/data-factory-intro/
Late answer to old question, but...
If you can use an Azure SQL Data warehouse you could take advantage of PolyBase to directly query the data in CSV format stored in the blob https://learn.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-load-polybase-guide#export-data-to-azure-blob-storage. This will allow you to directly map the data as an external table and query it dynamically.
This saves you the trouble of writing an external tool/solution for extracting, parsing and uploading the data to the Azure SQL database. Unfortunately PolyBase only works for Azure SQL Data warehouse, not Database, but you could setup something that read the structured data from the warehouse to your solution.
I know this question is two years old, but for those just now searching on the topic, I'd like to mention that the new Azure Feature Pack for SSIS makes this an easy task in SSIS. In VS Data Tools, after installing the Azure Feature pack, you would open an empty SSIS project and 1) Create an Azure Storage Connection Manager, then 2) Add a Data Flow Task, then open the Data Flow task and 3) Add a Blob Source tool to connect to the CSV, and then 4) using Destination Assistant connect to the SQL Table where the data is going. You can then execute this as a one-time load interactively inside the VS Data Tools IDE, or publish it to the SQL Server instance and create a recurring job.