I'm using AzureFileCopy task in Azure DevOps for copying a file to a blob storage. However, during execution, the task fails, and the logged error is about AzureRM module not being present.
`Starting: Copying file to blob storage
Task : Azure file copy
Description : Copy files to Azure Blob Storage or virtual machines
Version : 4.210.3
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
##[error]Could not find the modules: 'AzureRM' with Version: 'Any version'. If the module was recently installed, retry after restarting the Azure Pipelines task agent.
Finishing: Copying file to blob storage`
I tried to create a PowerShell task which was supposed to install this module, but couldn't make it to work.
Any suggestions?
Thanks.
You could use PowerShell script Uninstall-AzureRm to remove all AzureRm modules from the machine, refer to this official doc: https://learn.microsoft.com/en-us/powershell/module/az.accounts/uninstall-azurerm?view=azps-9.1.0#examples
From my test, I am failing with Azure File Copy task version 5.*
The same configuration succeeds with Azure File Copy task version 2.*
Have a simple Azure powershell Function. This is triggered by EventGrid, connected to a storage container. Basic function works without any gripe.
The moment I add module dependency everything runtime barfs. As you can see, I included Storage, Account & compute dependency. I tried both complete version and wildcard version. Both fail. I'd appreciate if someone can tell me what is amiss
Requirements.psd1
# This file enables modules to be automatically managed by the Functions service.
# See https://aka.ms/functionsmanageddependency for additional information.
#
#{
# For latest supported version, go to 'https://www.powershellgallery.com/packages/Az'.
# To use the Az module in your function app, please uncomment the line below.
Az = '8.*'
Accounts = '2.*'
Compute = '4.*'
Storage = '4.*'
}
In order for these modules to be included I added import statements in profile.ps1
# import statements
Import-Module Az.Accounts
Import-Module Az.Compute
Import-Module Az.Storage
# Authenticate with Azure PowerShell using MSI.
# Remove this if you are not planning on using MSI or Azure PowerShell.
if ($env:MSI_SECRET) {
Disable-AzContextAutosave -Scope Process | Out-Null
Connect-AzAccount -Identity
}
run.ps1 file is simple. It parses the input and destructure's all input for processing and attempts to ivoke a remote VM to execute a script. this is where I need compute module
param($eventGridEvent, $TriggerMetadata)
# Make sure to pass hashtables to Out-String so they're logged correctly
# $eventGridEvent | Out-String | Write-Host
# $response = $eventGridEvent | ConvertTo-JSON -Depth 5 | Out-String -Width 200
$topic = $eventGridEvent.topic
$api = $eventGridEvent.data.api
$file = $eventGridEvent.data.url
$fileType = $eventGridEvent.data.contentType
# check for container name and file type and invoke appropriate
# operation in VM
Write-Information "A file change operation happened in $topic"
Write-Host "A file change operation happened in $topic"
Write-Host "$api was invoked on $file of type $fileType"
Write-Host "START execution of script in remote host"
Invoke-AzVMRunCommand -ResourceGroupName 'autosys100-rg' -Name 'autosyslinuxvm' -CommandId 'RunShellScript' -ScriptPath 'install_nginx.sh'
Write-Host "COMPLETED execution of script on remote host"
There's nothign fancy and ludicrously basic :(
Here's the error
Connected!
2022-07-25T21:41:19 Welcome, you are now connected to log-streaming service. The default timeout is 2 hours. Change the timeout with the App Setting SCM_LOGSTREAM_TIMEOUT (in seconds).
2022-07-25T21:41:28.181 [Information] Executing 'Functions.egExample' (Reason='EventGrid trigger fired at 2022-07-25T21:41:28.1674630+00:00', Id=718522d6-3705-4be8-b4dc-1e1f93383c9f)
2022-07-25T21:41:28.236 [Warning] The first managed dependency download is in progress, function execution will continue when it's done. Depending on the content of requirements.psd1, this can take a few minutes. Subsequent function executions will not block and updates will be performed in the background.
2022-07-25T21:41:28.333 [Error] Executed 'Functions.egExample' (Failed, Id=718522d6-3705-4be8-b4dc-1e1f93383c9f, Duration=115ms)Result: FailureException: Failed to install function app dependencies. Error: 'Failed to get latest version for module 'Storage' with major version '4'. 'Stack: at Microsoft.Azure.Functions.PowerShellWorker.DependencyManagement.DependencyManager.WaitOnDependencyInstallationTask() in /mnt/vss/_work/1/s/src/DependencyManagement/DependencyManager.cs:line 246at Microsoft.Azure.Functions.PowerShellWorker.DependencyManagement.DependencyManager.WaitForDependenciesAvailability(Func`1 getLogger) in /mnt/vss/_work/1/s/src/DependencyManagement/DependencyManager.cs:line 164at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.ProcessInvocationRequest(StreamingMessage request) in /mnt/vss/_work/1/s/src/RequestProcessor.cs:line 247
2022-07-25T21:41:38.569 [Information] Executing 'Functions.egExample' (Reason='EventGrid trigger fired at 2022-07-25T21:41:38.5683456+00:00', Id=57490df5-c718-436e-87e3-211406b00f9d)
2022-07-25T21:41:39.236 [Warning] The Function app may be missing the 'Az.Accounts' module. If 'Az.Accounts' is available on the PowerShell Gallery, add a reference to this module to requirements.psd1. Make sure this module is compatible with PowerShell 7. For more details, see https://aka.ms/functions-powershell-managed-dependency.
2022-07-25T21:41:40.002 [Error] ERROR: The specified module 'Az.Accounts' was not loaded because no valid module file was found in any module directory.Exception :Type : System.IO.FileNotFoundExceptionMessage : The specified module 'Az.Accounts' was not loaded because no valid module file was found in any module directory.HResult : -2147024894TargetObject : Az.AccountsCategoryInfo : ResourceUnavailable: (Az.Accounts:String) [Import-Module], FileNotFoundExceptionFullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Commands.ImportModuleCommandInvocationInfo :MyCommand : Import-ModuleScriptLineNumber : 13OffsetInLine : 1HistoryId : 1ScriptName : C:\home\site\wwwroot\profile.ps1Line : Import-Module Az.AccountsPositionMessage : At C:\home\site\wwwroot\profile.ps1:13 char:1+ Import-Module Az.Accounts+ ~~~~~~~~~~~~~~~~~~~~~~~~~PSScriptRoot : C:\home\site\wwwrootPSCommandPath : C:\home\site\wwwroot\profile.ps1InvocationName : Import-ModuleCommandOrigin : InternalScriptStackTrace : at <ScriptBlock>, C:\home\site\wwwroot\profile.ps1: line 13PipelineIterationInfo :Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: ERROR: The specified module 'Az.Accounts' was not loaded because no valid module file was found in any module directory.Exception :Type : System.IO.FileNotFoundExceptionMessage : The specified module 'Az.Accounts' was not loaded because no valid module file was found in any module directory.HResult : -2147024894TargetObject : Az.AccountsCategoryInfo : ResourceUnavailable: (Az.Accounts:String) [Import-Module], FileNotFoundExceptionFullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Commands.ImportModuleCommandInvocationInfo :MyCommand : Import-ModuleScriptLineNumber : 13OffsetInLine : 1HistoryId : 1ScriptName : C:\home\site\wwwroot\profile.ps1Line : Import-Module Az.AccountsPositionMessage : At C:\home\site\wwwroot\profile.ps1:13 char:1+ Import-Module Az.Accounts+ ~~~~~~~~~~~~~~~~~~~~~~~~~PSScriptRoot : C:\home\site\wwwrootPSCommandPath : C:\home\site\wwwroot\profile.ps1InvocationName : Import-ModuleCommandOrigin : InternalScriptStackTrace : at <ScriptBlock>, C:\home\site\wwwroot\profile.ps1: line 13PipelineIterationInfo :Exception: The specified module 'Az.Accounts' was not loaded because no valid module file was found in any module directory.Stack:
Your requirements.psd1 has incorrect module names as you have forgotten the Az. prefix.
#{
# For latest supported version, go to 'https://www.powershellgallery.com/packages/Az'.
# To use the Az module in your function app, please uncomment the line below.
# Az = '8.*'
'Az.Accounts' = '2.*'
'Az.Compute' = '4.*'
'Az.Storage' = '4.*'
}
I would usually recommend not installing the entire Az module as it takes a long time however the compute, accounts and storage modules should be installed with the Az module so you wouldn't need to specify both.
You also don't need to specify the imports in the profile.ps1 of the function.
If the modules are available on the PSModulePath then you can just use the functions in your run.ps1
I currently have a script that is similair to the script that is described inside https://learn.microsoft.com/en-us/sql/relational-databases/security/encryption/configure-always-encrypted-keys-using-powershell?view=sql-server-ver15
However the script uses powershell 5 which isn't available to me on linux agents in our azure-devops environment. Because of this a majority of the azure-sql commands aren't available to us. Is there an alternative to these sqlserver module cmdlets.
The recommended alternative is to use the cloud console, but you may be able to get away with linux powershell like so:
I can't guarantee if this all works, but that specific command only creates a SqlColumnMasterKeySettings object that contains information about the location of your column master key. You can probably just create one manually, but you'll need to know the exact values. I would recommend running it from a windows machine first to see what the values should be for your environment.
# On Windows [Optional]
$cmkSettings = New-SqlAzureKeyVaultColumnMasterKeySettings -KeyURL $akvKey.Key.Kid
$cmkSettings | Format-List KeystoreProviderName,KeyPath
KeystoreProviderName : # Take these strings
KeyPath : # And use them in your script on linux
# Now on Linux, using the values above:
$cmkSettings = [Microsoft.SqlServer.Management.PowerShell.AlwaysEncrypted.SqlColumnMasterKeySettings]::new("YourKeystoreProviderName","YourKeyPath")
New-SqlColumnMasterKey -Name 'CMK1' -InputObject $database -ColumnMasterKeySettings $cmkSettings
# Success!
The key settings properties are just strings that get saved to your SQL Instance, so this should work fine. The harder part is authenticating to Azure to create keys from your master key, but you can try importing the desktop version of the commands like so:
# Start a NEW powershell session without the sqlserver module:
pwsh
# Get the module directory:
$d = (Get-Item (Get-Module SqlServer).path).DirectoryName
# Import the desktop version of these assemblies:
Import-Module "$d/Microsoft.SqlServer.Diagnostics.Strace.dll"
Import-Module "$d/Microsoft.SqlServer.Management.PSSnapins.dll"
Import-Module "$d/Microsoft.SqlServer.Management.AzureAuthenticationManagement.dll"
Import-Module "$d/Microsoft.SqlServer.Management.AlwaysEncrypted.Types.dll"
# Then import the module normally (there will be errors - you can ignore these)
Import-Module SqlServer
# Now you may be able to authenticate to Azure to generate new keys:
# Required to generate new keys
# NOTE: -Interactive fails on linux
Add-SqlAzureAuthenticationContext -ClientID "YourID" -Secret "YourSecret" -Tenant "YourTenant"
# Create a key using your master key:
New-SqlColumnEncryptionKey -Name 'CEK1' -InputObject $database -ColumnMasterKey 'CMK1'
This worked on my installation of centos7/pwsh7.1.3 - make sure you have SqlServer version 21.1.18245 (only 10 days old at the moment) as many new sql commands got ported to pwsh 7.1.
In an Azure Function I am trying to load a PowerShell module but getting the error Assembly with same
name is already loaded.
Code Sample
Import-Module "D:\home\site\wwwroot\HelloWorld\modules\MsrcSecurityUpdates\1.7.2\MsrcSecurityUpdates.psd1"
Error Message
Import-Module : Assembly with same name is already loaded
At C:\home\site\wwwroot\HelloWorld\run.ps1:25 char:5
+ Import-Module "D:\home\site\wwwroot\HelloWorld\modules\MsrcSecuri ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Import-Module], FileLoadException
+ FullyQualifiedErrorId : FormatXmlUpdateException,Microsoft.PowerShell.Commands.ImportModuleCommand
Some additional background..
This code was working yesterday. I have made a lot of edits so cannot clearly state the same code which was working yesterday is now failing.
I am editing the code directly via the browser.
I have restarted the web app, to potentially flush out any assemblies loaded during my code. Did not make a difference.
I checked if the module is available with the following, which returns the MsrcSecurityUpdates is NOT installed.
if (-not (Get-Module -Name "MsrcSecurityUpdates"))
{
Write-Output "MsrcSecurityUpdates NOT installed";
}
else
{
Write-Output "MsrcSecurityUpdates YES installed";
}
I downloaded the module with
Save-Module -Name MsrcSecurityUpdates -Path "C:\TEMP" -Force
and subsequently uploaded to the Azure Function File Share using the Kudo console. As per the steps outlined in this Stackoverflow question
This module seems to conflict with other modules in your app, or with assemblies loaded explicitly from your code. It is also possible that the module content is corrupted.
First of all, I would recommend relying on the Managed Dependencies feature instead of uploading the module via Kudu. Just include a reference to your module into the requirements.psd1 file at the root of your app:
#{
...
'MsrcSecurityUpdates' = '1.*'
}
If you edit this file in the Portal, you may need to restart your app. The next time you invoke any function, the latest version of this module will be automatically installed from the PowerShell Gallery and will be available on PSModulePath, so you can import it without specifying any path:
Import-Module MsrcSecurityUpdates
Try this on a brand new app without any other modules: MsrcSecurityUpdates will be loaded. However, if you are still getting the same error, this means MsrcSecurityUpdates is in conflict with other modules your app is using. You can narrow it down by removing other modules from your app (including cleaning up the modules uploaded via Kudu) and reducing your code.
[UPDATE] Potential workarounds:
Try to import (Import-Module) the modules in a certain fixed order, to make sure the more recent assembly versions are loaded first. This may or may not help, depending on the design of the modules.
Try executing commands from one of the modules in a separate process (using PowerShell jobs or sessions, or even invoking pwsh.exe).
I setup a new Win2012 VM in Azure with the Chef plugin and have it connected to manage.chef.io. Added a cookbook which uses the WebPi cookbook to install ServiceBus and its dependencies. The install fails with the following error:
“Error opening installation log file. Verify that the specified log file location exists and is writable.”
After some searching it looks like this is not new in Azure based on this 2013 blog post - https://nemetht.wordpress.com/2013/02/27/web-platform-installer-in-windows-azure-startup-tasks/
It offers a hack to disabled security on the folder temporarily but I'm looking for a better solution.
Any ideas?
More of the log output -
Started installing: 'Microsoft Windows Fabric V1 RTM'
.
Install completed (Failure): 'Microsoft Windows Fabric V1 RTM'
.
WindowsFabric_1_0_960_0 : Failed.
Error opening installation log file. Verify that the specified log file location exists and is writable.
DependencyFailed: Microsoft Windows Fabric V1 CU1
DependencyFailed: Windows Azure Pack: Service Bus 1.1
.
..
Verifying successful installation...
Microsoft Visual C++ 2012 SP1 Redistributable Package (x64) True
Microsoft Windows Fabric V1 RTM False
Log Location: C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\Web Platform Installer\logs\install\2015-05-11T14.15.51\WindowsFabric.txt
Microsoft Windows Fabric V1 CU1 False
Windows Azure Pack: Service Bus 1.1 False
Install of Products: FAILURE
STDERR:
---- End output of "WebpiCmd.exe" /Install /products:ServiceBus_1_1 /suppressreboot /accepteula /Log:c:/chef/cache/WebPI.log ----
Ran "WebpiCmd.exe" /Install /products:ServiceBus_1_1 /suppressreboot /accepteula /Log:c:/chef/cache/WebPI.log returned -1
A Chef contact (thanks Bryan!) helped me understand this issue better. Some WebPI packages do not respect the explicit log path provided to WebPIcmd.exe. The author should fix the package to use the provided log path when it is set. So the options became:
Have the author fix the package
Run Chef in a new scheduled task as a different user which has access
to the AppData folder
Edit the cookbook to perform/unperform a registry edit to temporarily move the AppData folder to a location that the System
user has access. Either in my custom cookbook or fork the WebPI
cookbook.
Obviously, waiting on the author (Microsoft in this case) to fix the package would not happen quickly.
Changing how the Azure VM runs Chef doesn't make sense considering the whole idea is to provide the configuration at the time of provisioning and it just work. Plus changing the default setup may have unintended consequences and puts us in a non-standard environment.
In the short term, I decided to alter the registry in my custom cookbook.
registry_key 'HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders' do
values [{
:name => "Local AppData",
:type => :expand_string,
:data => "%~dp0appdata"
}]
action :create
end
webpi_product 'ServiceBus_1_1' do
accept_eula true
action :install
end
webpi_product 'ServiceBus_1_1_CU1' do
accept_eula true
action :install
end
registry_key 'HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders' do
values [{
:name => "Local AppData",
:type => :expand_string,
:data => '%%USERPROFILE%%\AppData\Local'
}]
end
This change could also be done in the WebPI cookbook as well to fix this issue for all dependent cookbooks. I decided to not approach this until the WebPI team responds to a feature request for the framework to verify packages respect the log path instead.
http://forums.iis.net/t/1225061.aspx?WebPI+Feature+Request+Validate+product+package+log+path+usage
Please go and reply to this thread to try to get the team to help protect against this common package issue.
Here is the solution with POWERSHELL
I had the same error while installing "Service Fabric SDK" during VMSS VM creation. Also the system user was used.
Issue: when I was connecting with RDP with my "admin" user and run the same, it worked.
Solution: change the registry entry as above, install and reset back
here is my solution using "powershell"
I installed 2 .reg files into %TEMP% folder. The content is the old and new exported key / value for the
plugin-sf-SDK-temp.reg
Windows Registry Editor Version 5.00
[HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders]
"Local AppData"=hex(2):25,00,54,00,45,00,4d,00,50,00,25,00,00,00
plugin-sf-SDK-orig.reg
Windows Registry Editor Version 5.00
[HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders]
"Local AppData"=hex(2):25,00,55,00,53,00,45,00,52,00,50,00,52,00,4f,00,46,00,\
49,00,4c,00,45,00,25,00,5c,00,41,00,70,00,70,00,44,00,61,00,74,00,61,00,5c,\
00,4c,00,6f,00,63,00,61,00,6c,00,00,00
Integrate the following code into your custom-powershelgl script:
Write-Output "Reset LocalApp Folder to TEMP"
Start-Process "$($env:windir)\regedit.exe" `
-ArgumentList "/s", "$($env:TEMP)\plugin-sf-SDK-temp.reg"
## replace the following lines with your installation - here my SF SDK installation via WebWPIcmd
Write-Output "Installing /Products:MicrosoftAzure-ServiceFabric-CoreSDK"
Start-Process "$($env:programfiles)\microsoft\web platform installer\WebPICMD.exe" `
-ArgumentList '/Install', `
'/Products:"MicrosoftAzure-ServiceFabric-CoreSDK"', `
'/AcceptEULA', "/Log:$($env:TEMP)\WebPICMD-install-service-fabric-sdk.log" `
-NoNewWindow -Wait `
-RedirectStandardOutput "$($env:TEMP)\WebPICMD.log" `
-RedirectStandardError "$($env:TEMP)\WebPICMD.error.log"
Write-Output "Reset LocalApp Folder to ORIG"
Start-Process "$($env:windir)\regedit.exe" `
-ArgumentList "/s", "$($env:TEMP)\plugin-sf-SDK-orig.reg"