Is it possible to expose an Azure Data Lake Store via OData? The main goal is consume this service in Salesforce (via Salesforce Connect).
If so, should it take place through Azure Data Factory?
Update
A little bit more of context:
We have historical data stored in Azure Data Lake Storage (ADLS) that we want to expose via OData (to be visualised in Salesforce via Salesforce Connect / External objects). After digging into the issue and potential solutions, we don't think ADLS is the right service to be used in this particular case. Instead, we'll might need to configure a Data Factory pipeline to copy the data we are interested in to a SQL Database and read the data from there via a simple ASP.Net application using Entity Data Model and WCF Data Services Entity Framework Provider (got some insights from this website).
I don't think OData has a connector for ADLS. However, given OData is basically a REST API, you could probably build an OData API over the existing ADLS REST APIs if they are not providing what you need. I am not sure how ADF would come into this picture?
Maybe it would be useful if you tell us what you want to achieve?
Related
I am trying to see whether it is possible to access some data stored within a table in a dedicated SQL in Azure Synapse using REST API but I have not been able to figure much out. I checked the official docs at Microsoft and at most I have been able to query for a specific column within a table, not much more beyond that. I am wondering whether it is even possible to get data through the Azure Synapse REST API. Would appreciate any help.
Docs for reference: https://learn.microsoft.com/en-us/rest/api/synapse/
It is not possible to access the data in Synapse Dedicated SQL pools using REST APIs, today it is possible only to manage compute using REST API.
I cannot find any connector related to this and wanted to know if this is possible.
By copying data from Azure analysis service , do you mean to pull the aggregated data from the OLAP system into a target store?
If yes, then there is no direct connector but you can pull the data based on a linked server concept via ADF v2.
We need to create a linked server between an IaaS/On Prem/SQL MI data instance and AAS Server :
In ADF, create a linked service connecting to the database instance and query the Azure analysis service through database instance via OpenQuery :
select * from openquery(AZUREAS,
'evaluate SUMMARIZECOLUMNS(
''Geography''[City],
"My Measure", [My Measure]
)')
This is how you would be able to copy the AAS data from ADF into a destination :)
Blog post for the same:
https://datasharkx.wordpress.com/2021/03/16/copy-data-from-ssas-aas-through-azure-data-factory/
You also can use Azure Analysis Services REST API.
We are working on implementing a new project in Azure. The idea is to move out of on-premise systems into the cloud as we have our vendors, partners and clients moving into the cloud. The option we are trying out is to use Azure Data Share and have Azure SQL Database subscribe to the data.
The thing we are now trying to explore is once a new data snapshot is created how do we import this data into Azure SQL Database?
For instance we have Partner information and this information is made available via Azure Data Share and new data snapshot is created daily.
The part that I am not sure of is how to synchronize this data between Azure Data Share and Azure SQL Database.
Also, Is there an api available to expose this data out to external vendors, partners or clients from Azure SQL Database after we have data sync to Azure SQL Database from Azure Data Share?
Azure Data Share -> Azure SQL Database
Yes, Azure SQL Database is a supported.
Azure Data Share -> SQL Server Database (on-prem)? Is this option supported?
No, SQL Server Database (on-prem) is not supported.
Is there an api that could be consumed to read data?
Unfortunately, there is no such API that could be consumed to read data.
Azure Data Share enables organizations to simply and securely share data with multiple customers and partners. In just a few clicks, you can provision a new data share account, add datasets, and invite your customers and partners to your data share. Data providers are always in control of the data that they have shared. Azure Data Share makes it simple to manage and monitor what data was shared, when and by whom.
Azure Data Share helps enhance insights by making it easy to combine data from third parties to enrich analytics and AI scenarios. Easily use the power of Azure analytics tools to prepare, process, and analyze data shared using Azure Data Share.
Which Azure data stores does Data Share support?
Data Share supports data sharing to and from Azure Synapse Analytics, Azure SQL Database, Azure Data Lake Storage, Azure Blob Storage, and Azure Data Explorer. Data Share will support more Azure data stores in the future.
The below table details the supported data sources for Azure Data Share.
How to synchronize this data between Azure Data Share and Azure SQL Database.
You need to choose “Snapshot setting” to refresh data automatically.
A data provider can configure a data share with a snapshot setting. This allows incremental updates to be received on a regular schedule, either daily or hourly. Once configured, the data consumer has the option to enable the schedule.
I want to copy data from an on-premise oracle db to sql server on a real time basis.
Data Factory has many strengths, but frequency is not one of them. Have you considered a different approach?
For real time integrations I would recommend Functional App, Logic App and/or Service Bus. I would call this App whenever there is a relevant change in the oracle DB. Alternatively you could have an API on top of this on-premise oracle DB that you call from a scheduled APP.
If you expect heavy traffic you might want to consider using a Service Bus. The illustration below shows how Azure Service Bus sends data from a Publisher(on-premise) to a Subscriber(Azure sql DB) using Topic Message.
Illustration of large scale service bus dataflow
Ref. Azure Service Bus
Welcome to Stack Overflow!
You can copy data from an Oracle database to any supported sink data store. For a list of data stores that are supported as sources or sinks by the copy activity, see the Supported data stores table.
To copy data from and to an Oracle database that isn't publicly accessible, you need to set up a Self-hosted Integration Runtime. The integration runtime provides a built-in Oracle driver. Therefore, you don't need to manually install a driver when you copy data from and to Oracle.
For more details and step by step procedure, refer Copy data from and to Oracle by using Azure Data Factory.
You can also use custom activity as per your needs. Refer custom activities.
Hope this helps.
You could copy data incrementally. But there is frequency limitation. Please reference this post. https://social.msdn.microsoft.com/Forums/en-US/54380f98-716b-4a95-88af-cad2ab7e47b5/what-type-of-data-ingestion-does-azure-data-factory-use?forum=AzureDataFactory
I am looking for a best programmatic way to extract data from Azure Data Lake to MSSQL database, which is installed on a VM within Azure.
Currently I am considering following options:
Azure Data Factory
SSIS (Using Azure Data Lake Store Connection Manager)
User-Defined Outputter Example1, Example2
Custom C# code that reads Azure Data Lake data and inserts it into SQL Server DB
Any other good ways I am missing?
Data factory v2 (currently in public preview), also supports hosting SSIS to give you a data factory AND ssis option.
And not necessarily a good idea for many scenarios, but Azure Logic Apps has both a data lake store connector and SQL Server connector, which could be useful in scenarios such as writing lots of small files on a schedule or trigger.
You also may not need to go full on c# and instead use PowerShell, there are powershell modules for both data lake store and sql server.