How to add event subscription to Azure storage using New-AzEventGridSubscription? - azure

I am trying to add subscription to storage account using New-AzEventGridSubscription. The subscription should be triggered by blob modification in a container and put message to a certain queue. I created the following script:
$ResourceGroup = "test"
$includedEventTypes = "Microsoft.Storage.BlobCreated", "Microsoft.Storage.BlobDeleted"
New-AzEventGridSubscription `
-ResourceId "/subscriptions/[id]/resourceGroups/[group]/providers/Microsoft.Storage/storageAccounts/[name]" `
-EventSubscriptionName DummyName `
-Endpoint "/subscriptions/[id]/resourceGroups/[group]/providers/Microsoft.Storage/storageAccounts/[name]/queueServices/default/queues/my-queue" `
-ResourceGroup $ResourceGroup `
-EndpointType "storagequeue" `
-SubjectBeginsWith "prefix" `
-SubjectEndsWith "suffix"
but it throws an error:
New-AzEventGridSubscription : Parameter set cannot be resolved using the specified named parameters.
At line:1 char:1
+ New-AzEventGridSubscription `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [New-AzEventGridSubscription], ParameterBindingException
+ FullyQualifiedErrorId : AmbiguousParameterSet,Microsoft.Azure.Commands.EventGrid.NewAzureEventGridSubscription
I made similar command to add subscription to a custom topic, which worked well:
New-AzEventGridSubscription `
-ResourceGroup $ResourceGroup `
-EventSubscriptionName SubscriptionName `
-TopicName MyCustomTopic `
-EndpointType "storagequeue" `
-Endpoint "/subscriptions/[id]/resourceGroups/[group]/providers/Microsoft.Storage/storageAccounts/[account]/queueServices/default/queues/my-queue" `
-SubjectBeginsWith "prefix" `
-SubjectEndsWith "suffix"
I tried several modifications, but to no avail. What am I doing wrong?

After throwing out "ResourceGroup" parameter, reordering other parameters, and putting some arguments into variables, I finally managed to get my script up and running... If anyone ever needs similar stuff, here it is:
$subscriptionId = "<id>"
$ResourceGroup = "<group>"
$includedEventTypes = "Microsoft.Storage.BlobCreated", "Microsoft.Storage.BlobDeleted"
$storageAccount = "<account name>"
$endpoint = "/subscriptions/"+$subscriptionId+"/resourceGroups/"+$ResourceGroup+"/providers/Microsoft.Storage/storageAccounts/"+$storageAccount+"/queueServices/default/queues/my-queue"
$resourceId = "/subscriptions/"+$subscriptionId+"/resourceGroups/"+$ResourceGroup+"/providers/Microsoft.Storage/storageAccounts/"+$storageAccount
$subject = "prefix"
New-AzEventGridSubscription `
-EventSubscriptionName MySubscriptionName `
-Endpoint $endpoint `
-ResourceId $resourceId `
-EndpointType "storagequeue" `
-SubjectBeginsWith $subject `
-SubjectEndsWith "suffix" `
-IncludedEventType $includedEventTypes

Related

Restoring a managed SQL instance in Azure using PowerShell

I am trying to figure out how to restore a database from one managed SQL instance to another. I'm following the tutorials, but I keep running into inscrutable error messages.
Here's my command:
Restore-AzSqlInstanceDatabase `
-Name "SomeDatabase" `
-InstanceName "our-oltp-dev" `
-ResourceGroupName "dev-managedsqlinstances" `
-PointInTime "4/7/2020 12:00:00" `
-TargetInstanceDatabaseName "SomeDatabase_FROM_DEV" `
-TargetInstanceName "our-oltp-sandbox" `
-TargetResourceGroupName "sandbox-managedsqlinstances"
Here's the output:
PS C:\WINDOWS\system32> Restore-AzSqlInstanceDatabase `
-Name "SomeDatabase" `
-InstanceName "our-oltp-dev" `
-ResourceGroupName "dev-managedsqlinstances" `
-PointInTime "4/7/2020 12:00:00" `
-TargetInstanceDatabaseName "SomeDatabase_FROM_DEV" `
-TargetInstanceName "our-oltp-sandbox" `
-TargetResourceGroupName "sandbox-managedsqlinstances"
Restore-AzSqlInstanceDatabase : Parameter set cannot be resolved using the specified named parameters.
At line:1 char:1
+ Restore-AzSqlInstanceDatabase `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Restore-AzSqlInstanceDatabase], ParameterBindingException
+ FullyQualifiedErrorId : AmbiguousParameterSet,Microsoft.Azure.Commands.Sql.ManagedDatabase.Cmdlet.RestoreAzureRmSqlManagedDatabase
It's a copy-and-paste from the Azure docs; so I'm not sure what I'm doing wrong. Any help would be appreciated.
As mentioned in the comment, you need to pass the PointInTime as a DateTime instead of String, you could specify the 4/7/2020 12:00:00 as below.
Sample:
$PointInTime = Get-Date -Year 2020 -Month 4 -Day 7 -Hour 12 -Minute 0 -Second 0
Restore-AzSqlInstanceDatabase `
-Name "SomeDatabase" `
-InstanceName "our-oltp-dev" `
-ResourceGroupName "dev-managedsqlinstances" `
-PointInTime $PointInTime `
-TargetInstanceDatabaseName "SomeDatabase_FROM_DEV" `
-TargetInstanceName "our-oltp-sandbox" `
-TargetResourceGroupName "sandbox-managedsqlinstances"

Set-AzVMCustomScriptExtension in catch?

Attempting to add an extension when not detected but keep failing to find the secret sauce to get this to work. Mind you I am a BASH guy and this is a first foray into PowerShell..
#requires -version 2
# Required parameter $subscription: name of the subscription to enable Custom Script Extensions in
param (
# NOTE: See below for reason...
# [Parameter(Mandatory = $true)] [String] $subscription
# NOTE: Prompting is great for using the script interactively, but if this will also be executed
# from a build server or ...
# NOTE: Once the parameter is marked as mandatory PowerShell it will prompt for value. That said,
# if you remove the mandatory attribute then you can set a default value as a T_THROW ...
# NOTE: This _does_ contain shortcomings if this will be used as a pipeline param ...
# https://stackoverflow.com/questions/33600279/is-it-possible-to-force-powershell-script-to-throw-if-a-required-pipeline-para
[Parameter()]
[ValidateNotNullOrEmpty()]
[String]$SubscriptionName=$(Throw "`SubscriptionName` is mandatory, please provide a value...")
)
# Connect to the current Azure account
Write-Output "Pulling Azure account credentials..."
Start-Process "https://microsoft.com/devicelogin" # steals focus...
# Login to Azure account
Connect-AzAccount
# Set the active subscription
$null = Get-AzSubscription -SubscriptionName "$SubscriptionName" |Set-AzContext
# TODO: error handling
$vms = Get-AzVM
$cseName = "VulnerabilityManagementTools"
ForEach ($vm in $vms) {
try {
$cseStatus = Get-AzVMCustomScriptExtension `
-ResourceGroupName $vm.ResourceGroupName `
-VMName $vm.Name `
-Name $cseName `
-Status
}
catch {
Write-Output "Enabling Custom Script Extension for $vm."
Set-AzVMCustomScriptExtension `
-ResourceGroupName $vm.ResourceGroup `
-Location $vm.Location `
-VMName $vm.Name `
-Name $cseName `
-TypeHandlerVersion "1.1" `
-StorageAccountName "VulnerabilityManagementTools" `
-FileName "VulnerabilityManagementInstaller.ps1" `
-ContainerName "VulnerabilityManagementTools"
}
}
End up err'ing out with
PS /.../automation-scripts> ./EnableCustomScriptExtension.ps1 SubscriptionName
Pulling Azure account credentials...
WARNING: To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code XXXXXX to authenticate.
Account SubscriptionName TenantId Environment
------- ---------------- -------- -----------
XXXX#analytics.com SubName XXXXXX-XXXX AzureCloud
Get-AzVMCustomScriptExtension : The Resource 'Microsoft.Compute/virtualMachines/XXXX/extensions/VulnerabilityManagementTools' under resource group '{NAME}' was not found.
ErrorCode: ResourceNotFound
ErrorMessage: The Resource 'Microsoft.Compute/virtualMachines/XXXX/extensions/VulnerabilityManagementTools' under resource group '{NAME}' was not found.
ErrorTarget:
StatusCode: 404
ReasonPhrase: Not Found
At /.../automation-scripts/EnableCustomScriptExtension.ps1:59 char:18
+ $cseStatus = Get-AzVMCustomScriptExtension `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Get-AzVMCustomScriptExtension], ComputeCloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.GetAzureVMCustomScriptExtensionCommand
Get-AzVMCustomScriptExtension : The Resource 'Microsoft.Compute/virtualMachines/XXXXX/extensions/VulnerabilityManagementTools' under resource group '{RESOURCE_GROUPNAME}' was not found.
ErrorCode: ResourceNotFound
ErrorMessage: The Resource 'Microsoft.Compute/virtualMachines/XXXX/extensions/VulnerabilityManagementTools' under resource group '{RESOURCE_GROUPNAME}' was not found.
ErrorTarget:
StatusCode: 404
ReasonPhrase: Not Found
At /.../automation-scripts/EnableCustomScriptExtension.ps1:59 char:18
+ $cseStatus = Get-AzVMCustomScriptExtension `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Get-AzVMCustomScriptExtension], ComputeCloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.GetAzureVMCustomScriptExtensionCommand
Get-AzVMCustomScriptExtension : The Resource 'Microsoft.Compute/virtualMachines/{VMName}/extensions/VulnerabilityManagementTools' under resource group '{RESOURCEX_GROUPNAME}' was not found.
ErrorCode: ResourceNotFound
ErrorMessage: The Resource 'Microsoft.Compute/virtualMachines/{VMName}/extensions/VulnerabilityManagementTools' under resource group '{RESOURCEX_GROUPNAME}' was not found.
ErrorTarget:
StatusCode: 404
ReasonPhrase: Not Found
At /.../automation-scripts/EnableCustomScriptExtension.ps1:59 char:18
+ $cseStatus = Get-AzVMCustomScriptExtension `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Get-AzVMCustomScriptExtension], ComputeCloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.GetAzureVMCustomScriptExtensionCommand`
In your case, you just need to use the if(){}else{} statement, try the script as below instead of the ForEach part of yours, it works fine on my side.
ForEach ($vm in $vms) {
$cseStatus = Get-AzVMCustomScriptExtension `
-ResourceGroupName $vm.ResourceGroupName `
-VMName $vm.Name `
-Name $cseName `
-Status `
-ErrorAction SilentlyContinue
if ($cseStatus){
Write-Host "The extension has been set for" $vm.Name
}else{
Write-Host "Enabling Custom Script Extension for" $vm.Name
Set-AzVMCustomScriptExtension `
-ResourceGroupName $vm.ResourceGroup `
-Location $vm.Location `
-VMName $vm.Name `
-Name $cseName `
-TypeHandlerVersion "1.1" `
-StorageAccountName "VulnerabilityManagementTools" `
-FileName "VulnerabilityManagementInstaller.ps1" `
-ContainerName "VulnerabilityManagementTools"
}
}
Test result:
You'll need to create an Azure AD Service Principal using password authentication and use the credentials of this to pass to the Connect-AzAccount cmdlet as follows:
$credentials = Get-Credential
Connect-AzAccount -ServicePrincipal -Credentials $credentials
The service account will need to have the necessary permissions to use the Set-AzVMCustomScriptExtensions cmdlet.
More information on creating the service account here:
https://learn.microsoft.com/en-us/powershell/azure/create-azure-service-principal-azureps?view=azps-2.8.0

Why am i getting "New-AzSqlDatabaseImport : Not found: Entity not found to invoke import"

I have a script to handle some parameters and deploy an arm template with those parameters, and then create a database using a bacpac file in my storage account. When my script is trying to invoke the cmdlet "New-AzSqlDatabaseImport" i'm getting "NotFound: Entity not found to invoke import"
I have trying updating all of my Az modules, and tried to run the cmdlet manually and fill in the parameters manually.
Write-Host "Creating database and importing data"
$bloblink = (Get-AzStorageAccount -ResourceGroupName $ResourceGroup -Name $StorageName).PrimaryEndpoints.Blob
$uri = $bloblink + $containerName + "/" + $bacpacFile
$importRequest = New-AzSqlDatabaseImport `
-ResourceGroupName $ResourceGroup `
-ServerName $SQLServer `
-DatabaseName $database_ql_name `
-DatabaseMaxSizeBytes $databaseMaxSizeBytes `
-StorageKeyType "StorageAccessKey" `
-StorageKey $(Get-AzStorageAccountKey -ResourceGroupName $ResourceGroup -StorageAccountName $StorageName).Value[0] `
-StorageUri $uri `
-Edition $DatabaseEdition `
-ServiceObjectiveName $DatabaseSize `
-AdministratorLogin $dbUsername `
-AdministratorLoginPassword $dbPassword
I expect it to start a job on Azure to create the database and start the import job.
This is the error im getting instead (i replaced the full path).
At *PATH*/Deployment.ps1:262 char:18
+ $importRequest = New-AzSqlDatabaseImport `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [New-AzSqlDatabaseImport], CloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Sql.ImportExport.Cmdlet.NewAzureSqlDatabaseImport
I just encountered the same error. I found that the -ServerName argument is case-sensitive. So check your $SQLServer-variable.

Export SQL Azure db to blob - Start-AzureSqlDatabaseExport : Cannot convert AzureStorageContainer to AzureStorageContainer

I am using this code found on stack , and all connections are correct.
Import-Module Azure
Import-Module Azure.Storage
Get-AzureRmSubscription –SubscriptionName “Production” | Select-AzureRmSubscription
# Username for Azure SQL Database server
$ServerLogin = "username"
# Password for Azure SQL Database server
$serverPassword = ConvertTo-SecureString "abcd" -AsPlainText -Force
# Establish credentials for Azure SQL Database Server
$ServerCredential = new-object System.Management.Automation.PSCredential($ServerLogin, $serverPassword)
# Create connection context for Azure SQL Database server
$SqlContext = New-AzureSqlDatabaseServerContext -FullyQualifiedServerName “myspecialsqlserver.database.windows.net” -Credential $ServerCredential
$StorageContext = New-AzureStorageContext -StorageAccountName 'prodwad' -StorageAccountKey 'xxxxx'
$Container = Get-AzureStorageContainer -Name 'automateddbbackups' -Context $StorageContext
$exportRequest = Start-AzureSqlDatabaseExport -SqlConnectionContext $SqlContext -StorageContainer $Container -DatabaseName 'Users' -BlobName 'autobackupotest.bacpac' -Verbose -Debug
I am getting this error. I have spent hours on this.
Start-AzureSqlDatabaseExport : Cannot bind parameter 'StorageContainer'. Cannot convert the
"Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageContainer" value of type
"Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageContainer" to type
"Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageContainer".
At line:31 char:99
+ ... SqlConnectionContext $SqlContext -StorageContainer $Container -Databa ...
+ ~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Start-AzureSqlDatabaseExport], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.WindowsAzure.Commands.SqlDatabase.Database.Cmdlet.StartAzureSqlDatabaseExport
I am using AzureRM 3.8.0
According to your description and codes, If you want to start a new sqldatabase export.I suggest you could try below codes. It will work well.
$subscriptionId = "YOUR AZURE SUBSCRIPTION ID"
Login-AzureRmAccount
Set-AzureRmContext -SubscriptionId $subscriptionId
# Database to export
$DatabaseName = "DATABASE-NAME"
$ResourceGroupName = "RESOURCE-GROUP-NAME"
$ServerName = "SERVER-NAME"
$serverAdmin = "ADMIN-NAME"
$serverPassword = "ADMIN-PASSWORD"
$securePassword = ConvertTo-SecureString -String $serverPassword -AsPlainText -Force
$creds = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $serverAdmin, $securePassword
# Generate a unique filename for the BACPAC
$bacpacFilename = $DatabaseName + (Get-Date).ToString("yyyyMMddHHmm") + ".bacpac"
# Storage account info for the BACPAC
$BaseStorageUri = "https://STORAGE-NAME.blob.core.windows.net/BLOB-CONTAINER-NAME/"
$BacpacUri = $BaseStorageUri + $bacpacFilename
$StorageKeytype = "StorageAccessKey"
$StorageKey = "YOUR STORAGE KEY"
$exportRequest = New-AzureRmSqlDatabaseExport -ResourceGroupName $ResourceGroupName -ServerName $ServerName `
-DatabaseName $DatabaseName -StorageKeytype $StorageKeytype -StorageKey $StorageKey -StorageUri $BacpacUri `
-AdministratorLogin $creds.UserName -AdministratorLoginPassword $creds.Password
$exportRequest
# Check status of the export
Get-AzureRmSqlDatabaseImportExportStatus -OperationStatusLink $exportRequest.OperationStatusLink
Then you could use Get-AzureRmSqlDatabaseImportExportStatus to see the details information, like below:
I got the same problem after updating some powershell packages which I do not remember exactly. After the update my scripts started to fail.
My solution is :
Install the latest AzureRM from nuget via powershell
Use the other overload of Start-AzureSqlDatabaseExport which utilizes the parameters
-StorageContainerName and -StorageContext rather than -StorageContainer
It looks like if you pass the parameters to the function it will create the container object internally without crashing.

Cannot create Azure HDInsight cluster with Hive metastore using Powershell

I get error when I try to create Azure HDInsight cluster using powershell cmdlet:
New-AzureRmHDInsightClusterConfig `
| Add-AzureRmHDInsightMetastore `
-SqlAzureServerName "$sqlDatabaseServerName.database.windows.net" `
-DatabaseName $hiveMetaStoreDBName `
-Credential $sqlServerCred `
-MetastoreType HiveMetaStore `
| New-AzureRmHDInsightCluster `
-ResourceGroupName $resourceGroupName `
-HttpCredential $clusterCreds `
-ClusterName $clusterName `
-Location $location `
-ClusterType $clusterType `
-OSType $OSType `
-Version "$hdVersion" `
-SshCredential $clusterCreds `
-DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" `
-DefaultStorageAccountKey $storageAccountKey `
-ClusterSizeInNodes $clusterNodes
Looks like parameters are not recognized by powershell because it asks to input them (see below). I input required parameters (Location, ClusterName, ClusterSizeInNodes) and then error occurs.
cmdlet New-AzureRmHDInsightCluster at command pipeline position 3
Supply values for the following parameters:
(Type !? for Help.)
Location: West Europe
ClusterName: xxxxxxxxx
ClusterSizeInNodes: 1
New-AzureRmHDInsightCluster : BadRequest: ParameterNullOrEmpty,Parameter 'ASVAccount.AccountName' cannot be null or
empty.
At line:117 char:11
+ | New-AzureRmHDInsightCluster `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [New-AzureRmHDInsightCluster], CloudException
+ FullyQualifiedErrorId : Hyak.Common.CloudException,Microsoft.Azure.Commands.HDInsight.NewAzureHDInsightClusterCom
mand
Does somebody know why its happens or what is wrong in smdlet?
From the error message, it seems like your $storageAccountName parameter to the New-AzureRmHDInsightCluster cmdlet is Null or Empty, you may want to inspect further on this.
Besides, I would also strongly recommend you to specify the -DefaultStorageContainer to the New-AzureRmHDInsightCluster cmdlet as well. This will ensures that the cmdlet will be able to resolve the FQDN of your storage account Uri.
E.g. asv://YourDefaultContainer#YourDefaultStorageAccountName.blob.core.windows.net/
Hope this helps!
Use below command for Cluster with Hive Metastore.
Here is a working PowerShell script, to be used with Azure ARM PowerShell, 1.0.1 or later – you can install Azure RM PS via web platform installer or follow this blog https://azure.microsoft.com/en-us/blog/azps-1-0/
Add-AzureRmAccount
$MyClusterName = "clustername";
$MyClusterLocation = "East US 2";
$NumClusterNodes = 2;
$MyClusterVersion = "3.2";
$MyHDInsightUserName = ""
$MyHDInsightPwd = ""
$MySqlAzureUserName = ""
$MySqlAzurePwd = ""
$MySqlAzureServerName = "*.database.windows.net"
$MySqlAzureDbName = "Dbtest"
$MyDefaultContainerName = "tastoreps"
$clusterResourceGroupName = "dirg"
# Use the correct Azure Subscription!
$subid = ""
Select-AzureRmSubscription -SubscriptionId $subid
# Storage key
$primaryStorageAcctName = "toragesouth"
$primaryStorageResourceGroupName = "storagerg"
# you need to use an ARM based storage as the primary account , classic storage won’t work as a primary account, known issue
$storageAccountKey = Get-AzureRmStorageAccountKey -ResourceGroupName $primaryStorageResourceGroupName -Name $primaryStorageAcctName | %{ $_.Key1 }
# credentials
$HdInsightPwd = ConvertTo-SecureString $MyHDInsightPwd -AsPlainText -Force
$HdInsightCreds = New-Object System.Management.Automation.PSCredential ($MyHDInsightUserName, $HdInsightPwd)
$SqlAzurePwd = ConvertTo-SecureString $MySqlAzurePwd -AsPlainText -Force
$SqlAzureCreds = New-Object System.Management.Automation.PSCredential ($MySqlAzureUserName, $SqlAzurePwd)
$config = New-AzureRmHDInsightClusterConfig -ClusterType Hadoop |
Add-AzureRmHDInsightMetastore -SqlAzureServerName $MySqlAzureServerName -DatabaseName $MySqlAzureDbName -Credential $SqlAzureCreds -MetastoreType HiveMetastore |
Add-AzureRmHDInsightMetastore -SqlAzureServerName $MySqlAzureServerName -DatabaseName $MySqlAzureDbName -Credential $SqlAzureCreds -MetastoreType OozieMetastore
$config.DefaultStorageAccountName="$StorageAcctName.blob.core.windows.net"
$config.DefaultStorageAccountKey=$storageAccountKey
#create cluster
New-AzureRmHDInsightCluster -config $config -OSType Windows -clustername $MyClusterName -HttpCredential $HdInsightCreds -DefaultStorageContainer $MyDefaultContainerName -Location $MyClusterLocation -ResourceGroupName $clusterResourceGroupName -ClusterSizeInNodes $NumClusterNodes -Version $MyClusterVersion

Resources