We have a requirement to provide data in the form of a text file from our database to different vendors. The file should be generated on a daily basis. Is there any resource or application in azure that we can leverage in order to accomplish this?
Regards,
Lolek
You can use Azure Functions to read from a SQL Azure database as explained on the following resource:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-scenario-database-table-cleanup
The following resource shows how you can write from an Azure Function to a BLOB storage account.
https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/azure-functions/functions-reference-csharp.md#binding-at-runtime-via-imperative-bindings
The following article shows you how to schedule the Azure Function or automate its execution.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-scheduled-function
Hope this helps.
Use Azure Data factory to export the desired data from Sql Azure, schedule the job to put the file in Blob storage.
Related
I want to take an Archival(Backup) of Azure SQL Table to the Azure Blob Storage, I have done the backup in Azure Blob storage using via the Pipeline in CSV file format. And From the Azure Blob Storage, I have restored the data into the Azure SQL Table successfully using the Bulk Insert process.
But now I want to retrieve the data from this CSV file using some kind of filter criteria. Is there any way that I can apply a filter query on Azure Blob storage to retrieve the data?
Is there any other way to take a backup differently and then retrieve the data from Azure Storage?
My end goal is to take a backup of the Azure SQL table in Azure Storage and retrieve the data directly from Azure Storage with a filter.
Note
I know that I can take a backup using the SSMS, but that is not a requirement, I want this process through some kind of Pipeline or using the SQL command.
AFAIK, there is no such filtering option available when restoring the database. But, as you are asking for another way to backup and restoring, SQL Server Management Studio (SSMS) is one the most conveniently used platform for almost all SQL Server related activities.
You can use SSMS to access Azure SQL database using server name and Login Password.
Find this official tutorial from Microsoft about how to take backup of your Azure SQL Database and store it in Storage account and then restore it.
I want to use Azure Log Analytics with the data export feature to export all log tables to a storage account. There used to be an '--export-all-tables' option, but annoyingly this has been removed.
Is there a way I can export all tables? Not just the ones that exist at the moment, but any future ones that may be created?
Azure Policy?
Azure Functions?
Azure Logic App?
We can archive the data with the help of Logic App, as we run a query from a logic app and uses its output in other actions in the workflow. So here Azure Blob Storage connector is used to send query output to blob storage.
Here we just need Log Analytics Workspace and Storage account access to achieve this.
And to add on all the new data, we can create a trigger in logic app where we can run it once in a day according to our requirement.
After setting up the trigger “Click + New step to add an action that runs after the recurrence action. Under Choose an action, type azure monitor and then select Azure Monitor Logs.”
Later after configuring the whole workflow create blob and attach it to workflow as below:
Later we can run the logic app and check the storage for the logs.
Check for the Microsoft Documentation to understand more about Archive data from Log Analytics workspace to Azure storage using Logic App
I have the following ETL requirements for Snowflake on Azure and would like to implement the simplest possible solution because of timeline and technology constraints.
Requirements :
Load CSV data (only a few MBs) from Azure Blob Storage into Snowflake Warehouse daily into a staging table.
Transform the loaded data above within Snowflake itself where transformation is limited to just a few joins and aggregations to obtain a few measures. And finally, park this data into our final tables in a Datamart within the same Snowflake DB.
Lastly, automate the above pipeline using a schedule OR using an event based trigger (i.e. steps to kick in as soon as file lands in Blob Store).
Constraints :
We cannot use use Azure Data Factory to achieve this simplest design.
We cannot use Azure Functions to deploy Python Transformation scripts and schedule them either.
And, I found that Transformation using Snowflake SQL is a limited feature where it only allows certain things as part of COPY INTO command but does not support JOINS and GROUP BY. Furthermore, although the following THREAD suggests that scheduling SQL is possible, but that doesn't address my Transformation requirement.
Regards,
Roy
Attaching the following Idea diagram for more clarity.
https://community.snowflake.com/s/question/0D50Z00009Z3O7hSAF/how-to-schedule-jobs-from-azure-cloud-for-loading-data-from-blobscheduling-snowflake-scripts-since-dont-have-cost-for-etl-tool-purchase-for-scheduling
https://docs.snowflake.com/en/user-guide/data-load-transform.html#:~:text=Snowflake%20supports%20transforming%20data%20while,columns%20during%20a%20data%20load.
You can create snowpipe on Azure blob storage, Once snowpipe created on top of your azure blob storage, It will monitor bucket and file will be loaded into your stage table as soon as new file comes in. After copied the data into stage table you can schedule transformation SQL using snowflake task.
You can refer snowpipe creation step for azure blob storage in below link:
Snowpipe on microsoft Azure blob storage
I tried using HTTP request I am able to send the data from HTTP request to azure SQL but I am manually Send the data through postman but that is not my requirement
Requirement:I need to use a scheduler and a particular time the data from the text file need to be read and to be stored into Azure-SqlDB
if you have any resource or examples please let me know
About how import Text file(CSV) data to Azure SQL by using logic-App, you could reference this tutorial: Quick, easy and cheap way to automate data loading from CSV file into Azure SQL:
Check out how to leverage Azure Blob Storage and Logic Apps for
simple scenario of data loading from CSV into Azure SQL in less than
30 minutes and with almost no coding.
About any developer out there at some point or another had to
automate ETL process for data loading. This article will present a
fast and convinient way to create data loading workflow for CSVs
using Azure SQL and blob storage.
It also introduced other ways you could reference:
How import Text file(CSV) data to Azure SQL by using Data Factory.
How import Text file(CSV) data from Blob stroage to Azure SQL by
using T-SQL.
You also could reference:
Upload Flat File on Azure SQL Database using Azure Logic App
Using Azure Logic Apps to Import CSV to SQL Server
I'm agree with #Mandar Dharmadhikari, Logic app is not the best way to do it.
If your csv file with large data, I also suggest you to use Data Factory，when the copy active pipeline created, you could trigger the pipeline executing in schedule.
Hope this helps.
I would suggest you yo use Azure Data Factory as it is more suited to the task that you want to perform. Following post gives the idea on how to move csv data fo SQL.
http://normalian.hatenablog.com/entry/2017/09/04/233320
I am new Azure functions, One of my task is to read data from Sql database and upload that data as a csv file in azure Blob storage using Azure functions and then using logicapps to retreive it. I am stuck with Sql to file to Azure Blob
I would start with the Azure Functions documentation. I did a quick internet search and found this article on how to access to SQL database from an Azure Function: https://learn.microsoft.com/en-us/azure/azure-functions/functions-scenario-database-table-cleanup
Here is another article which shows how to upload content to blob storage: https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob#output
Apply your learnings from both and you should be able to accomplish this task.
What about if instead you create a trigger to start the logic apps when something happen in your DB. Interesting article here : https://flow.microsoft.com/en-us/blog/introducing-triggers-in-the-sql-connector/
you can then pass the information to a function to process the data and push the new csv file to the storage : https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet?tabs=windows
Optionally you might need to transform what the trigger from sql returns you, there you can use the logic apps transform the input : https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-enterprise-integration-transform