I am trying to add a vm scale set to an agent pool in Azure Devops. I have set up the VMSS and now am adding DSC as an extension which will then call "SendConfigurationApply" to send to the agent pool.
However when I upgrade the VMs I get this error:
VM has reported a failure when processing extension 'DSC'. Error
message: "DSC Configuration 'AgentsInstalled' completed with error(s).
Following are the first few: The request was aborted: Could not create
SSL/TLS secure channel. The PowerShell DSC resource
'[AgentsResource]Agents' with SourceInfo
'C:\Packages\Plugins\Microsoft.Powershell.DSC\2.83.1.0\DSCWork\agentsConnectedInt.ps1.1\agentsConnectedInt.ps1::26::9::AgentsResource'
threw one or more non-terminating errors while running the Set
functionality. These errors are logged to the ETW channel called
Microsoft-Windows-DSC/Operational. Refer to this channel for more
details. The SendConfigurationApply function did not succeed." More
information on troubleshooting is available at
https://aka.ms/VMExtensionDSCWindowsTroubleshoot
I have set the protocols on the VMs to use TLS 1.1 and 1.2.
The VMs are on an older version of Windows 10 client because we need to run Edge Legacy. Could that be related to this issue?
Related
I'm having a very weird issue:
I'm deploying to an azure web app via MSDeploy from an on-prem CI/CD pipeline with Bamboo.
I get:
27-Jun-2022 20:53:50 Verbose: Pre-authenticating to remote agent URL 'https://app-xxxxx-stg.scm.azurewebsites.net:443/msdeploy.axd?site=app-xxxxx-stg' as '$app-xxxxx-stg'.
27-Jun-2022 20:53:50 Error: Could not complete the request to remote agent URL 'https://app-xxxx-stg.scm.azurewebsites.net/msdeploy.axd?site=app-xxxxx-stg'.
27-Jun-2022 20:53:50 Error: The request was aborted: Could not create SSL/TLS secure channel.
27-Jun-2022 20:53:50 Error count: 1.
What is weird is that the same deployment script works just fine with other webapps that have been created in the past (the dev and tst environments) and also fails with the same error if I try to deploy to prod environment (also just created).
The environments are created via ARM template, so they are exactly the same.
I've read other similar issues, but my webapp is configured to allow only TLS 1.2 min. But as mentioned, all the web apps are configured the same way, and the deployments all start from the same machine.
What could be the issue? how can I solve this connection problem?
Thank you
Just to let everyone know: problem is solved by forcing net framework applications (like MS Deploy) to default to TLS1.2.
As per this article: https://learn.microsoft.com/en-us/mem/configmgr/core/plan-design/security/enable-tls-1-2-client#bkmk_net
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft.NETFramework\v2.0.50727]
"SystemDefaultTlsVersions" = dword:00000001
"SchUseStrongCrypto" = dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions" = dword:00000001
"SchUseStrongCrypto" = dword:00000001
We'd a discussion on this on Microsoft Q&A thread. Just posting here to benefit the community. Self hosted agent connected to site using 1.0 or 1.1, we may encounter this error. Hence, for self hosted agent/custom agent for DevOps pipeline deployment ensure that the agent has >TLS1.2 installed.
I'd like to add an offsite Windows VM to Azure Arc for health monitoring. The VM is hosted by Vultr and runs Windows Server 2016 Standard Build 14393.
However, installing AzureConnectedMachineAgent.msi on the target VM fails with error code 1603. Installation log also contains this error:
Start-Service : Service 'Guest Configuration Extension service
WixQuietExec64: (ExtensionService)' cannot be started due to the following error: Cannot start
WixQuietExec64: service ExtensionService on computer '.'.
WixQuietExec64: At C:\Program Files\AzureConnectedMachineAgent\ExtensionService\GC\Modules\Exte
WixQuietExec64: nsionService\ServiceHelper.psm1:367 char:5
Any suggestions on how to fix this?
You may Check if the user with which you are logged into the VM have
sufficient permissions to start a system service
If you find the following in the
%ProgramData%\AzureConnectedMachineAgent\Log\himds.log or in installation logs :
time="2021-02-11T08:39:38-08:00" level=error msg="Cannot open event source: Azure Hybrid Instance Metadata Service."
You can verify the permissions by collecting the following registry
key from an impacted server.
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Eventlog\Application\CustomS
Mitigation can be to grant the permission to write to the
SECURITY_SERVICE_RID S-1-5-6 which would grant the required
permissions to the himds service account.
https://learn.microsoft.com/en-us/windows/win32/secauthz/well-known-sids.
If the registry key does NOT exist on the impacted VM, then this
resolution will NOT apply as there will be a separate root cause such
as AV interference.
If the root cause is not found here ,then a procmon trace needs to be
taken to analyze the root cause for the msi not being able to start a
service.
( In case a procmon trace has to be analyzed , please open an MS
Support ticket)
To get support for Windows Agent and extensions in Azure, the Windows
Agent on the Windows VM must be later than or equal to version
2.7.41491.911. However the cause for the failure of agent installation is different in this case.
You may also want to check %programdata%\ext_mgr_logs\gc_ext_telemetry.txt log which must have had an entry something like this :
<GCLOG>........ Not starting Extension Service since machine is an Azure VM</GCLOG>
Cause:
This can happen while attempting to install the agent on an Azure VM.This is an unsupported production scenario.One Should not be installing this agent on an Azure VM as it conflicts with the Azure Guest Agent and interferes with Azure VM management.
If one wishes to use an Azure VM simply for testing purposes then
they can follow the below document for guidance
https://learn.microsoft.com/en-us/azure/azure-arc/servers/plan-evaluate-on-azure-virtual-machine
This is what I followed to setup IR.
In the final step of Registering Azure Data factory self hosted intergration runtime, we need to provide the Authentication Key. then the installation is making a call to internet. Isn't this strange as the VM could be in a private network?
If the VM is not connected to internet and it gets this error then what to do? "Failed to connect to: we.frontend.clouddatahub.net"
This is the error I get
Failed to execute the command ' -Key xxx'. Error message: Microsoft.DataTransfer.DIAgentClient.HostServiceException: Failed to get service token from ADF service with key xxxx and time cost is: 3.0786307 seconds, the error code is: UnexpectedFault, activityId is: xxx and detailed error message is An error occurred while sending the request.
The underlying connection was closed: An unexpected error occurred on a send.
Authentication failed because the remote party has closed the transport stream.
The issue seems to be disabled remote access. How can I enable it? Dmgcmd -era 8060 is not working.
I have also a related issue logged as another VM works and this fails
Even if you have some private network where the communication can go without any restrictions between your data sources and your integration runtime, the integration runtime application needs to be able to communicate with the Azure data factory services as well. Try whitelisting the IPs for your region in the networking settings of your Azure VM or in your firewall - according to this:
https://learn.microsoft.com/sv-se/azure/data-factory/azure-integration-runtime-ip-addresses
In Azure pipeline when creating a VM through deployment template, we have the option to 'Configure with WinRM agent' as given below.
This acts as a custom extension behind the scenes. But the downloading of this custom extension can be blocked by an internal vnet in Azure. This is the error we are getting.
<datetime> Adding extension 'WinRMCustomScriptExtension' on virtual machine <vmname>
<datetime> Failed to add the extension to the vm: <vmname>. Error: "VM has reported a failure when processing extension 'WinRMCustomScriptExtension'. Error message: \"Failed to download all specified files. Exiting. Error Message: The underlying connection was closed: An unexpected error occurred on a send.\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot "
Since the files cannot be downloaded, I am thinking of a couple of solutions:
How can I know which powershell files azure is using to setup winrm?
Location to store files would be storage account (same vnet as VM)
Perhaps not use WinRM at all and use custom script extension to resolve
everything (with all files from storage account). I hope error from extension stops the pipeline if it happens.
Is there a better solution to resolve this? To me it looks like a bad design by azure as it is not covering non-public VMs.
EDIT:
Found answer to #1) https://aka.ms/vstsconfigurewinrm. This was shown in Raw logs of the pipeline when diagnostics were enabled
Even if you know - how does it help you? It won't be able to download them anyway and you cant really tell it to use local files
If you enable service endpoins and allow your subnet to talk to the storage account - it should work
there is a way to configure WinRM when you create the VM. Keyvault example
You could use script extension like you wanted to as well, but script extension has to download stuff to the Vm as well. Example
I have a script that facilitates an ARM template to provision an Azure Service Fabric cluster (official windows servers) among other dependencies like storage and such. I do not provision through the portal.
Facts:
Two days ago, I used this script to provision the cluster with complete success.
I tried the same again yesterday, and the provisioning failed (with the error below).
just to reassure you that the provisioning script works, I can successfully provision with this script on other subscription and it constantly and reliably succeeds.
The error:
Resource Microsoft.Insights/autoscaleSettings '1NodeVMSetAutoScale' failed with message 'The metric with namespace '' and name '\Processor(_Total)\% Processor Time' is not supported for this resource id '/subscriptions/----/resourceGroups/-cluster/providers/Microsoft.Compute/virtualMachineScaleSets/1'.' 8:10:01 PM - Resource Microsoft.Insights/autoscaleSettings '2NodeVMSetAutoScale' failed with message 'The metric with namespace '' and name '\Processor(_Total)\% Processor Time' is not supported for this resource id '/subscriptions/----/resourceGroups/cluster/providers/Microsoft.Compute/virtualMachineScaleSets/2'.' 8:10:01 PM - "Template output evaluation skipped: at least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." 'string' does not contain a definition for 'error'
My question is why? What could be the reason for it not to consistently succeed? Can you please help with troubleshooting steps?
Related info: https://azure.microsoft.com/en-us/documentation/articles/insights-autoscale-common-metrics/
2 questions:
1) what region are you deploying in?
2) In the new subscription, can you check what resource providers you have registered, and in what regions? In the CLI, the commands look like:
azure config mode arm
azure provider list
azure provider show Microsoft.Insights
I faced the same issue since a week in my subscriptions. The way out was to make changes to the Diagnostic configurations, by adding the counter "\Processor(_Total)\% Processor Time" under the waddiagnostic performace counters section. You can also take sneak peak here were autoscale is discussed: Service Fabric Autoscale
Please share your template/ part of it to analyse further.