I have a concern about Azure Synapse and Azure Data Factory like this:
We using Azure Data lake gen 2 to export data from D365FO. After that, use Azure Data Factory or Azure Synapse to transform (complex) and create a xml file.
My customer decided to use Azure Synapse to transform, but i checked, look like Azure Synapse only support to XML as Source, not Sink and doesn't support transform (flexiable like XSLT)?
You guys have any opinions about that? please help, thanks!
Currently, XML format is only supported as a source in the Azure Data Factory and not supported as sink.
You can raise a feature request here.
https://feedback.azure.com/d365community/post/
Related
I'm new to Azure Data Explorer. Here I need to migrate the data from Azure Table Storage table data into Azure Data Explorer Cluster's Database's Table without using Azure Data Factory.
May be if able to do programatically using .NET, kindly suggest it.
Thanks in advance.
I have a requirement to export data from Azure SQL managed instance to data lake storage as json documents. I have to use SQL Server Integration Services to accomplish this. I tried using the Flexible File Destination Data Flow task but when I see the supported file formats there's no json being supported. What other options do I have to accomplish this.
Azure Data Factory support data movement between Azure Managed Instance and Data lake account, but unfortunately when the destination is Azure Data Lake storage it also doesn't support JSON format using SSIS.
Azure Data Lake Store Destination
The Azure Data Lake Store Destination component enables an SSIS
package to write data to an Azure Data Lake Store. The supported file
formats are: Text, Avro, and ORC.
Workaround: The possible workaround you can try to use Data Flow activity in Azure Data Factory. Load the data from Managed Instance and transform it using Pivot transformation and store the processed data in Data Lake. This approach doesn't involve SSIS. Check this similar kind of request and approach here.
I'm looking for some advice around using Azure Data Factory to migrate data from Salesforce to Dynamics365.
My research has discovered plenty of articles about moving salesforce data to sinks such as azure data lakes or blob storage and also articles that describe moving data from azure data lakes or blob storage into D365.
I haven't found any examples where the source is salesforce and the sink is D365.
Is it possible to do it this way or do I need to copy the SF data to an intermediate sink such as Azure Data Lake or blob storage and then use that as the source of a copy/dataflow to then send to D365?
I will need to perform transformations on the SF data before storing it in D365.
Thanks
I would recommend to add ADLS Gen 2 as a Stage between SalesForce and D365
I am afraid that a direct sink as D365 can be done
Req: This actually for export last 6 months data from Timeseries insights to CSV file to download to User from web application.
The historical data is available on Blob & Timeseries insights.
Approach:
Create azure data factory pipeline which pulls the data from azure time series insights based on aggregate/filter query and convert it into CSV to store in Blob storage.
Is it possible to do that or any other approach can be considered for this requirement.
Please suggest.
Thanks
Data Factory doesn't support Azure Time Series Insights as the dataset(Source or Sink).
Ref the connector overview.
That means we can't create azure data factory pipeline which pulls the data from azure time series insights based on aggregate/filter query and convert it into CSV to store in Blob storage.
Some other way, you may could using Azure Blob storage client library to create and upload the file to Blob Storage.
Here's the tutorials may be helpful:
Query data from the Azure Time Series Insights Gen2 environment
using C Sharp
Quickstart: Azure Blob storage client library v12 for .NET
have you ever made an azure data convert Azure Data Lake excel conversion to CSV file.
first, I have tried using SSIS with Azure Data Lake Source, but when Mapping is not possible, the choice is to add text.
second, says try using azure apps logic with create CSV table but the csv that comes out is only the structure in that folder
Thank you in advance
There is not a built-in way to extract from excel file in Azure data lake. I would suggest you to try one of the below approaches:
Write Custom .NET library for converting Excel to CSV and deploy that to Azure Data Lake Analytics. Azure Data Lake Analytics Programming Guide
Write a custom .NET activity in Azure Data Factory to do this. Custom Activities in Azure Data Factory
Use Azure Functions and Open XML do this activity as detailed in the stack overflow post
Use SSIS Package to do the conversion. You can have SSIS Runtimes in Azure Data Factory. SSIS packages running in Azure Data Factory
As I know about Azure, These isn't any way can help convert the excel file to csv directly.
You could follow these steps:
Download the excel file to you computer.
Import the excel file to the you SQL database.
Then export the table data as CSV file to you Blob Storage.
You could reference this document:
Import data from Excel to SQL Server or Azure SQL Database
Connect to Azure Blob Storage (SQL Server Import and Export
Wizard)
Hope this helps.