Validation Error
I've got this weird issue where validation fails on 'additional columns' for my data sink to Azure SQL coming from a blob storage source in the Azure Data Factory GUI. No matter how many times we recreate the dataset (or specify another dataset, new) we can't get past this validation issue.
The irony of this is we deploy these pipelines from code and when we run them, we get no errors at all. This issue we have had just made life really difficult developing pipelines further as we have to do everything by code. We cant use the pipepline publish option.
Here are some screen grabs for you of the pipeline so you can see the flow.
Pipeline
Inside copyCustomer.
Source
Mapping
Sink
Any ideas on how to fix this validation would be greatly appreciated.
For what it's worth, we have recreated the dataset multiple times (clone and new) to avoid any issue with the dataset model not being the latest as per what's documented here https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview#add-additional-columns-during-copy
Sometimes by setting the table in sink to autocreate has shown the validation to be 'fixed' but then when we go to publish it errors out again.
When your Azure SQL dataset was created long time before and is still utilizing an outdated dataset model that Additional Columns do not support, this is expected behavior.
As per official Microsoft documentation
To resolve this issue, you can just follow the error message to create a new Azure SQL dataset and use this as copy sink.
I followed error message and created new data set and it is working fine for me.
Source:
Mapping:
Sink:
Output:
I suspect here, your dataset of Sink type is incorrect. I reproduced,
same at my end. Its working fine. Kindly make sure you create a sink dataset type with Azure SQL database type connector only.
Please check below screenshots from my implementation.
If still it helps, feel free to share your sink dataset type connector details along with screenshots.
Related
I have been trying to set up a snowpipe to ingest data from blob storage in Azure into snowflake, following this guide, I think I have done everything correctly although I am new to azure and snowflake so may have missed something obvious. Everything seems to have been set up correctly on both sides, but whenever I check the pipe status using SELECT SYSTEM$PIPE_STATUS('azure_pipe');, I get the following:
{"executionState":"RUNNING","pendingFileCount":0,"notificationChannelName":"https://snowflakedata.queue.core.windows.net/snowflakequeue","numOutstandingMessagesOnChannel":2,"lastReceivedMessageTimestamp":"2022-02-18T13:25:12.107Z","channelErrorMessage":"downloadAttributes error:Queue not found for channel Name=https://snowflakedata2.queue.core.windows.net/snowflakequeue, AccountId=6713, NotificationChannelID=2045, IntegrationID=1784764","lastErrorRecordTimestamp":"2022-02-18T17:32:47.854Z"}
I'm not sure what I have done wrong, the snowflake app has the queue contributor role in azure and I'm fairly sure I set everything else up correctly. If anyone could point me in the right direction as to how to troubleshoot this that would be really helpful!
I had the same issue as you did just this week when trying to create a Snowpipe for Azure. Using SELECT SYSTEM$PIPE_STATUS('azure_pipe'); gave the exact same error message as you have shown above. Thankfully, Snowflake Support has provided me with the answer and an explanation.
Answer:
Drop all of the objects relating to the Snowpipe (integrations, pipe, stage, etc). Then recreate them in the exact order and specification as shown in this documentation.
Explanation:
The issue for me was caused because I kept using create or replace on the objects when I was modifying them (eg changing the comment on a pipe). This re-created the object and broke the links between the objects in the Snowpipe and prevented the Snowpipe from working as intended. Dropping and starting again solved it for me.
I've been working with Powerapps for about 3 to 4 months now, and this particular app I've created connects to 3 tables in an Azure SQL Server Database. It's been working great until today. This morning I logged in to edit a feature on the app and my galleries were empty with an error message saying: "Bad Data source Inner exception, Not a valid data Source".
My first thought was, something is off on the database, some planned work possibly during the night, but that's not the case, and the DB is working perfectly as I also use it in PowerBI.
I googled for the error and found several people with a similar issue, and most of them reported that deleting the Data Source connections from power apps and adding them again would fix the issue, as well as deleting and creating the Galleries again. I've performed both actions, and the same error returns. I tried creating a fresh new canvas app, with a gallery and a brand new connection to the Azure Sql DB and still got the same error. I've also tried to restore one of the oldest versions of the app, and I still got the same error. This to me all indicates there's a problem with the DB, but...when I publish the APP, and run the app after publishing, It works perfectly, and I have full usability of the app, which means the connection to the DB is actually working.
It feels like a bug to me. I'll share the screenshot of the error:
Error Message
I'm worried I will lose all my work, so if someone could help me, that would be great.
Finally, I got the solution for this issue. This are the steps I followed to solve the issue.
Remove all the Database connectors from your app.
Go to File -> Settings -> Advanced Settings -> Disable enhanced Microsoft SQL Server Connector.
Add all your Database connectors back again.
I have selected the option of logging all the incompatible rows into the storage account default container, but there have been no logs written inside the storage account, I am wondering why is that not happening?
Is there anything which can be done to make this work?
It's a regression and we are working on the fix, it's expected to be deployed by end of this week. Please try after that.
Update:
The issue is fixed, can you try again?
I am trying to load data to an excel file using SSIS Package. Please find below the details
Source : SQL Server Table
Destination : Excel File
No.of rows:646K
No.of columns:132
I have deployed the package in the SQL Server Integration Services Catalog and trying to execute it from there.
But the following errors are being thrown:
Not enough storage is available to complete this operation.
The attempt to add a row to the Data Flow task buffer failed with
error code 0xC0047020.
SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on
SRC_MDM_ENTITYDUPLICATE returned error code 0xC02020C4. The
component returned a failure code when the pipeline engine called
PrimeOutput(). The meaning of the failure code is defined by the
component, but the error is fatal and the pipeline stopped executing.
There may be error messages posted before this with more information
about the failure.
My DFT looks like the following:
I am using Data Conversion since I am facing some datatype mismatch between Unicode and Non-Unicode characters.
The package is working fine in my local machine with 95-99% resource utilization.
Since I have deployed the package in production environment, I can't do any modifications in the Server Settings. Also I guess the high resource utilization is creating issue while executing the package in production server.
I tried reducing DefaultBufferMaxRows size and increasing DefaultBufferSize which didn't help me anyhow.
Can somebody help me to optimize my package and fix this issue.
Thanks much in Advance.
I realized that the solution of the error is that the column is not excel in your package, as a solution you will either delete that column from the package or add empty columns
My azure search indexer which reads from an sql table with Change Tracking, is failing with the following error
"Unable to cast object of type 'Newtonsoft.Json.Linq.JObject' to type 'System.String'."
If I reset the indexer, it'll start working fine. What is the root cause of the problem here?
We’ve identified an issue in SQL indexers that use SQL integrated change detection that affects a very small number of customers. We’re working on a fix, which will probably be deployed in production next week. We'll also improve our telemetry so that we'll be able to identify this class of issues proactively.
The workaround you’ve already used (resetting the indexer) is the best workaround for this issue.
Sorry for the trouble!