I have been using Azure Data Factory V2 for a while to execute SSIS packages from the SSISDB catalouges.
Today (16-11-2018) I have encountered "Unexpected Termination" Failure message without any Warning and Error message.
Things than I have done:
Executing the SSIS package manually from the SSISDB catalogue in SQL Server Management Services (SSMS). What i have noticed is that it took an exceptionally long time to assign the task to a machine. When the package is assigned to a machine, within 1 or two minutes it throws back the Failure message.
There are 3 SSIS packages that is excecuted "sequentially" with the Azure Data Factory Pipeline. Often the 1st package is executed successfully, however the 2nd and 3rd package never succeded.
Another error message that I got is "Failed pull task from SSISDB, please check if SSISDB has exceeded its limit".
I hope anyone can help me with this issue. I have been searching the web and could not find anything on this subject.
What tier of Azure sql server have you provisioned for the SSISDB to run on? If its too small, it may be taking too much time starting and throw a timeout.
Personally, I've had no problems provisioning an S3 Azure Sql Server.
Hope this helped!
Martin
Related
I setup a Databricks instance on Azure using terraform. The deployment seems to be good. But, I am getting the following error when creating/starting a new cluster,
Message
Cluster terminated. Reason: Cloud provider launch failure
Help
A cloud provider error was encountered while launching worker nodes.
See Knowledge base: Cloud provider initiated terminations for remediations.
Search for error code NetworkingInternalOperationError
Details
Azure error code: NetworkingInternalOperationError
Azure error message: An unexpected error occured while processing the network profile of the VM. Please retry later.
Any idea why this is happening?
Usually such errors are returned when there are temporary problems with underlying VM infrastructure. Usually they mitigated very fast, so you just need to try later, although it makes sense to check Azure Databricks and Azure status pages - they may show if outage is in progress.
Over the weekend, our ADF solution cannot validate any more.
Error message at validation:
DF_Postcode Could not load resource 'DF_Postcode'. Please ensure no
mistakes in the JSON and that referenced resources exist. Status:
UnknownError, Possible reason: undefined
This includes triggers, pipelines, and dataflows.
We did not do any deployments between Friday and this morning. Any thoughts?
-- Update --
Possibly related, starting a data flow debug is not succesful.
-- Update 2 --
Multiple pop-ups appear when doing a shift+F5 refresh of the page. The error message itself is not very helpful.
It does appear there were few changes pushed to ADF over the weekend. However, as the error says, could you check the resources if they are intact or if any properties or values got reset! just in case to be sure to remove user configuration issue.
Check in the ADF studio for all the resources referenced in the error.
If you are using Powershell modules at any point in there, make sure you use the latest one.
Also, for quick check you can raise an issue here to get an official response.
Looking at the Service Blade in the Azure Portal I found an emerging issue listed.
Starting at 09:00 UTC, customers may experience errors using Azure
Data Factory in West Europe, using the Azure Portal UX. We are aware
of the issue and are investigating. Updates to follow in 60 minutes or
as events warrant. Workaround: Customers can manage Data Factory using
Azure Data Studio, Azure CLI or Powershell.
https://aka.ms/azuredatastudio
-- Update from Microsoft --
Summary of impact: Between approximately 06:30 UTC and 12:30 UTC on 13 Dec 2021, you were identified as a
customer using Data Factory V2 in West Europe who may have experienced intermittent errors when accessing
resources in this region.
Preliminary Root Cause: We determined a backend service, responsible for processing API requests became
unhealthy. This led to intermittent API failing calls for Azure Data Factory resources.
Mitigation: We restarted the backend service which mitigated the issue.
I have had an Azure SQL DB point in time restore running for two days. I want to cancel it as I think there is an issue. I can see the DB restoring in SSMS but can't find the deployment in my Azure Portal. Does anyone know how to cancel it? I have tried using Azure CLI but I can't see the resource.
It's called Azure Hiccups, it happened to me yesterday on Switzerland West region between 10:20 and 10:40.
I re-run it and everything was fixed.
If I check the Activity Log I can see the error:
But if I browse in the Service Health it says everything was good:
What to do in case of Azure Hiccups:
FIX: Re-run the task, hopefully it will fix the issue, like when you hit an old TV with your fist.
PREVENT: You can try to create an Activity Log alert but once again it will be based on Service Health (which says that everything is good) and not on the actual Activity Log. So you will probably miss issues like this and will discover the problem 24h later.
POST-MORTEM: You can take a screenshot of the failed task/service in the Activity Log, show it to Microsoft and ask for a refund if possible. For the future you can check the current status of Azure in the official Status page and subscribe to the RSS feed. You can browse the Azure Status History. But as I said none of the last two reports the Azure Hiccups so the screenshot of the Activity Log is still the only proof that a tree yesterday has fallen in the forest.
As Microsoft SLA says that the High availability for Azure SQL Database and SQL Managed Instance is 99.99% of the year you can start collecting those screenshot and open tickets with their support.
After dropping the Database this morning, the operation status of which was unsuccessful. The Restore has finally been canceled 8 hrs after attempting to drop the database.
Found a solution, just create a new database of the same name. And the restoring one will be replaced with the one created, then you can delete it.
I can’t seem to get the 1st project collection created in Azure DevOps Server 2019.
The creation process keeps timing out (within an hour range) and stating that it can’t connect to the DB, but it’s clear it’s been connected prior to step #67 in the creation process, which is where it times out.
Here is what I have tried:
Using 2 machines:
One machine has Azure DevOps Server 2019 and SQL 2017 (w/ SSMS) installed with following configuration
4G RAM
At least 12 G free on hard drive now (will be increasing this)
Second machine will house build server
Error from Logs:
Caught exception in ExecuteScripts: Microsoft.TeamFoundation.Framework.Server.DatabaseRuntimeException:
TF246017: Azure DevOps Server could not connect to the database.
Verify that the instance is specified correctly, that the server that
is hosting the database is operational, and that network problems are
not blocking communication with the server. --->
System.Data.SqlClient.SqlException: A timeout occurred while waiting
for memory resources to execute the query in resource pool 'internal'
(1). Rerun the query
I had an additional 4G RAM installed to the server and that fixed the issue.
We are using the SQL Database Sync feature in Azure and whilst it generally works without issues it does occasionally fail. The failures can be due to lost database connections or failures in which the error messages in the Azure Sync Log don't help. I see no way in the Azure portal to set up alerts to email us when a sync fails and I don't see any Azure cmdlets that will return the status of a database sync. Does anyone have any ideas how we can add monitoring to the service?
Thanks!
I did crosspost this later to the msdn forum that user6133663 listed below.
The answer from Xu Ye (who I assume works for MS) was:
You are right. Currently SQL Data Sync did not support API & failure alert. We will keep you update if any update on this.
So, to answer my original question, there is currently no way to monitor the success or failure of a SQL database sync.