SQL Azure - Importing CSV file that resides in a blob - azure

Question

I have read this SO question but mine is quite specific to the "import" of CSV and not how to access the blob to get the CSV out
Which is the best way?
1) CSV Stored in the Blob - use a worker role, read the CSV from the blob, parse data and update database
2) Is SQL BulkCopy/BulkInsert an option. The challenge here is that it should not have any on-premise involvement. All within Azure: blob->SQL DAtabase.
3) Will Azure Automation help? Are there PS scripts/workflows that help in such bulk update of CSV data to Azure SQL DB? I haven't found any though
Are there other options that help import blob CSV data to SQL DB without having to write custom code?
Appreciate any thoughts...

Answer 1

Your first method would work. You could also use azcopy (http://aka.ms/azcopy) to download the file locally, and then use BCP to load it into SQL - this way you wont have to write any code for this.
Azure Automation would help if you want to do this repeatedly. You should be able to set this up as a script even if one doesn't exist.

Answer 2

I know this is outdated question but for anyone looking for quick way to do this feel free to check my article on how to do this quickly using SQL prodecure triggered by Logic App.
In short you run on master
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'UNIQUE_STRING_HERE'
Then you run on DB
CREATE DATABASE SCOPED CREDENTIAL BlobCredential
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET = 'sv=SAS_TOKEN_HERE';
CREATE EXTERNAL DATA SOURCE AzureBlob
WITH (
TYPE = BLOB_STORAGE,
LOCATION = 'https://<account_name>.blob.core.windows.net/<container_name>',
CREDENTIAL = BlobCredential
);
And then
BULK INSERT <my_table>
FROM '<file_name>.csv'
WITH (
DATA_SOURCE = 'AzureBlob',
FORMAT = 'CSV',
FIRSTROW = 2
);
Just wrap this insert in procedure and execute it from logic app.
https://marczak.io/posts/azure-loading-csv-to-sql/
or just use ADF like here
https://azure4everyone.com/posts/2019/07/data-factory-intro/

Answer 3

Late answer to old question, but...
If you can use an Azure SQL Data warehouse you could take advantage of PolyBase to directly query the data in CSV format stored in the blob https://learn.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-load-polybase-guide#export-data-to-azure-blob-storage. This will allow you to directly map the data as an external table and query it dynamically.
This saves you the trouble of writing an external tool/solution for extracting, parsing and uploading the data to the Azure SQL database. Unfortunately PolyBase only works for Azure SQL Data warehouse, not Database, but you could setup something that read the structured data from the warehouse to your solution.

Answer 4

I know this question is two years old, but for those just now searching on the topic, I'd like to mention that the new Azure Feature Pack for SSIS makes this an easy task in SSIS. In VS Data Tools, after installing the Azure Feature pack, you would open an empty SSIS project and 1) Create an Azure Storage Connection Manager, then 2) Add a Data Flow Task, then open the Data Flow task and 3) Add a Blob Source tool to connect to the CSV, and then 4) using Destination Assistant connect to the SQL Table where the data is going. You can then execute this as a one-time load interactively inside the VS Data Tools IDE, or publish it to the SQL Server instance and create a recurring job.