I get error when I try to create Azure HDInsight cluster using powershell cmdlet:
New-AzureRmHDInsightClusterConfig `
| Add-AzureRmHDInsightMetastore `
-SqlAzureServerName "$sqlDatabaseServerName.database.windows.net" `
-DatabaseName $hiveMetaStoreDBName `
-Credential $sqlServerCred `
-MetastoreType HiveMetaStore `
| New-AzureRmHDInsightCluster `
-ResourceGroupName $resourceGroupName `
-HttpCredential $clusterCreds `
-ClusterName $clusterName `
-Location $location `
-ClusterType $clusterType `
-OSType $OSType `
-Version "$hdVersion" `
-SshCredential $clusterCreds `
-DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" `
-DefaultStorageAccountKey $storageAccountKey `
-ClusterSizeInNodes $clusterNodes
Looks like parameters are not recognized by powershell because it asks to input them (see below). I input required parameters (Location, ClusterName, ClusterSizeInNodes) and then error occurs.
cmdlet New-AzureRmHDInsightCluster at command pipeline position 3
Supply values for the following parameters:
(Type !? for Help.)
Location: West Europe
ClusterName: xxxxxxxxx
ClusterSizeInNodes: 1
New-AzureRmHDInsightCluster : BadRequest: ParameterNullOrEmpty,Parameter 'ASVAccount.AccountName' cannot be null or
empty.
At line:117 char:11
+ | New-AzureRmHDInsightCluster `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [New-AzureRmHDInsightCluster], CloudException
+ FullyQualifiedErrorId : Hyak.Common.CloudException,Microsoft.Azure.Commands.HDInsight.NewAzureHDInsightClusterCom
mand
Does somebody know why its happens or what is wrong in smdlet?
From the error message, it seems like your $storageAccountName parameter to the New-AzureRmHDInsightCluster cmdlet is Null or Empty, you may want to inspect further on this.
Besides, I would also strongly recommend you to specify the -DefaultStorageContainer to the New-AzureRmHDInsightCluster cmdlet as well. This will ensures that the cmdlet will be able to resolve the FQDN of your storage account Uri.
E.g. asv://YourDefaultContainer#YourDefaultStorageAccountName.blob.core.windows.net/
Hope this helps!
Use below command for Cluster with Hive Metastore.
Here is a working PowerShell script, to be used with Azure ARM PowerShell, 1.0.1 or later – you can install Azure RM PS via web platform installer or follow this blog https://azure.microsoft.com/en-us/blog/azps-1-0/
Add-AzureRmAccount
$MyClusterName = "clustername";
$MyClusterLocation = "East US 2";
$NumClusterNodes = 2;
$MyClusterVersion = "3.2";
$MyHDInsightUserName = ""
$MyHDInsightPwd = ""
$MySqlAzureUserName = ""
$MySqlAzurePwd = ""
$MySqlAzureServerName = "*.database.windows.net"
$MySqlAzureDbName = "Dbtest"
$MyDefaultContainerName = "tastoreps"
$clusterResourceGroupName = "dirg"
# Use the correct Azure Subscription!
$subid = ""
Select-AzureRmSubscription -SubscriptionId $subid
# Storage key
$primaryStorageAcctName = "toragesouth"
$primaryStorageResourceGroupName = "storagerg"
# you need to use an ARM based storage as the primary account , classic storage won’t work as a primary account, known issue
$storageAccountKey = Get-AzureRmStorageAccountKey -ResourceGroupName $primaryStorageResourceGroupName -Name $primaryStorageAcctName | %{ $_.Key1 }
# credentials
$HdInsightPwd = ConvertTo-SecureString $MyHDInsightPwd -AsPlainText -Force
$HdInsightCreds = New-Object System.Management.Automation.PSCredential ($MyHDInsightUserName, $HdInsightPwd)
$SqlAzurePwd = ConvertTo-SecureString $MySqlAzurePwd -AsPlainText -Force
$SqlAzureCreds = New-Object System.Management.Automation.PSCredential ($MySqlAzureUserName, $SqlAzurePwd)
$config = New-AzureRmHDInsightClusterConfig -ClusterType Hadoop |
Add-AzureRmHDInsightMetastore -SqlAzureServerName $MySqlAzureServerName -DatabaseName $MySqlAzureDbName -Credential $SqlAzureCreds -MetastoreType HiveMetastore |
Add-AzureRmHDInsightMetastore -SqlAzureServerName $MySqlAzureServerName -DatabaseName $MySqlAzureDbName -Credential $SqlAzureCreds -MetastoreType OozieMetastore
$config.DefaultStorageAccountName="$StorageAcctName.blob.core.windows.net"
$config.DefaultStorageAccountKey=$storageAccountKey
#create cluster
New-AzureRmHDInsightCluster -config $config -OSType Windows -clustername $MyClusterName -HttpCredential $HdInsightCreds -DefaultStorageContainer $MyDefaultContainerName -Location $MyClusterLocation -ResourceGroupName $clusterResourceGroupName -ClusterSizeInNodes $NumClusterNodes -Version $MyClusterVersion
Related
I have tried this approach to change a password of an Azure VM:
$resgroup = "rsource1"
$vmName = "virtualmachine1"
$VM = Get-AzVM -ResourceGroupName $resgroup -Name $vmName
$Credential = Get-Credential
$VM | Set-AzureVMAccessExtension –UserName $Credential.UserName `
–Password $Credential.GetNetworkCredential().Password
$VM | Update-AzVM
But I keep getting this error:
Object reference not set to an instance of an object.
When I console.log the values of $Credential.UserName and $Credential.GetNetworkCredential().Password I got the values of username and password that I have inputted.
What am I missing here?
I've never used Set-AzureVMAccessExtension, but I've used the Az PowerShell equivalant Set-AzVMAccessExtension. It needs you to pass -Credential $Credential instead of -UserName and -Password.
You can try this script I made a while ago to to reset passwords for Azure VMs:
# Replace these values with your own
$resourceGroupName = "Servers-RG"
$vmName = "server1"
# Get the VM into an object
$vm = Get-AzVM -ResourceGroupName $resourceGroupName -Name $vmName
# Store credentials you want to change
$credential = Get-Credential -Message "Enter your username and password for $vmName"
# Store parameters in a hashtable for splatting
# Have a look at https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_splatting?view=powershell-7
$extensionParams = #{
'VMName' = $vmName
'Credential' = $credential
'ResourceGroupName' = $resourceGroupName
'Name' = 'AdminPasswordReset'
'Location' = $vm.Location
}
# Pass splatted parameters and update password
Set-AzVMAccessExtension #extensionParams
# Restart VM
# Don't need to pass any switches since they are inferred ByPropertyName
# Have a look at https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_pipelines?view=powershell-7
$vm | Restart-AzVM
I found that the password update doesn't happen until you restart the VM, so Restart-VM is required.
If anyone interested in the Linux (KISS) version (no VM restart needed):
$settings = '{}'
$protectedSettings = '{
"username": "<yourusername, prefer using Credentials object>",
"password": "<yourpassword, prefer using Credentials object>"
}'
Set-AzVMExtension `
-VMName $vmName `
-ResourceGroupName $rgName `
-Location $location `
-Name "VMAccessForLinux" `
-Publisher "Microsoft.OSTCExtensions" `
-ExtensionType "VMAccessForLinux" `
-TypeHandlerVersion "1.4" `
-Settingstring $settings `
-ProtectedSettingString $protectedSettings
I am trying to scale the CosmosDB Container using Powershell but couldn't find anything in the docs. I tried the following script which didn't work.
$resourceName = $CosmosDB + "/sql/" + $CosmosDatabase + "/" + $CosmosContainer
$ContainerProperties = #{
"resource"=#{
"id"=$CosmosContainer;
"partitionKey"=#{
"paths"=#("/DefaultKey");
"kind"="Hash"
}
};
"options"=#{ "Throughput"=$CosmosScale }
}
Set-AzResource -ResourceType "Microsoft.DocumentDb/databaseAccounts/apis/databases/containers" -ApiVersion "2015-04-08" -ResourceGroupName $resourceGroup -Name $resourceName -PropertyObject $ContainerProperties -Force
Any insights are appreciated.
Here is a PS script that will update throughput on either a database or container for a SQL (Core) API account.
# Update RU for an Azure Cosmos DB SQL (Core) API database or container
$resourceGroupName = "myResourceGroup"
$accountName = "mycosmosaccount"
$databaseName = "database1"
$containerName = "container1"
$databaseResourceName = $accountName + "/sql/" + $databaseName + "/throughput"
$containerResourceName = $accountName + "/sql/" + $databaseName + "/" + $containerName + "/throughput"
$throughput = 500
$updateResource = "database" # or "container"
$properties = #{
"resource"=#{"throughput"=$throughput}
}
if($updateResource -eq "database"){
Set-AzResource -ResourceType "Microsoft.DocumentDb/databaseAccounts/apis/databases/settings" `
-ApiVersion "2015-04-08" -ResourceGroupName $resourceGroupName `
-Name $databaseResourceName -PropertyObject $properties
}
elseif($updateResource -eq "container"){
Set-AzResource -ResourceType "Microsoft.DocumentDb/databaseAccounts/apis/databases/containers/settings" `
-ApiVersion "2015-04-08" -ResourceGroupName $resourceGroupName `
-Name $containerResourceName -PropertyObject $properties
}
else {
Write-Host("Must select database or container")
}
I have used this to update the Azure CosmosDb (SQL API) Collections on automation account but which gets timed Out with the cmdlet Set-AzResource - sometimes it works
Get-AzResource -ResourceType Microsoft.DocumentDB/databaseAccounts
$containerResourceType = "Microsoft.DocumentDb/databaseAccounts/apis/databases/containers/settings"
Using Set command with API Version in Runbook which got failed with ‘Operation failed because a request timed out.’
Set-AzResource -ResourceType $containerResourceType ***
Reference:
https://serverfault.com/questions/967942/scaling-cosmosdb-container-using-powershell
I'm creating VM on Azure from an Image using powershell.
This is the script I'm using .
$UserName = "username"
$Password = ConvertTo-SecureString "password#123" -AsPlainText -Force
$psCred = New-Object System.Management.Automation.PSCredential($UserName, $Password)
New-AzureRmVm `
-ResourceGroupName "RSG" `
-Name "VMName" `
-ImageName "ImageName" `
-Location "West US" `
-VirtualNetworkName "VNName" `
-SubnetName "default" `
-Credential $psCred
-PublicIpAddressName "None" `
-OpenPorts 3389
But, when I got into the Azure portal and see, some Public Ip is getting assigned by default. I have also tried without giving PublicIpAddressName property assuming , it wont assign any IP, but still it is assigning.
I want the Public IP to be none.Can anyone help me achieve this.Thanks!
Currently this an issue which is still in Open state on official azure-powershell github. You can refer it here . Incase if you still want to bypass this you can try using New-AzureReservedIP or after the deployment command try to remove the public ip by yourself Remove-AzureRmPublicIpAddress.
Note : I have'nt tested it yet. Just an idea.
Refer : Docs
To set no public ip address you have can just define it as "" , in powershell you will need to quote that again so it will be """" .
If you are using PowerShell, then you will need to escape all empty parameters by changing "" to '""' to properly pass an empty string into the command. Without this, PowerShell will not pass the empty string, and you will get an error from the command indicating it's missing a parameter.
$winVmCred = Get-Credential `
-Message "Enter username and password for the Windows management virtual machine."
# Create a NIC for the VM.
$winVmNic = New-AzNetworkInterface -Name "winVMNIC01" `
-ResourceGroupName $resourceGroup.ResourceGroupName `
-Location $location `
-SubnetId $targetVMSubnet.Id `
-PrivateIpAddress "10.10.12.10"
# Configure the Windows management VM.
$winVmConfig = New-AzVMConfig -VMName $winVmName -VMSize $winVmSize | `
Set-AzVMOperatingSystem -Windows -ComputerName $winVmName -Credential $winVmCred | `
Set-AzVMSourceImage -PublisherName $winVmPublisher `
-Offer $winVmOffer `
-Skus $winVmSku `
-Version $winVmVersion | `
Add-AzVMNetworkInterface -Id $winVmNic.Id
# Create the VM.
$winVM = New-AzVM -ResourceGroupName $resourceGroup.ResourceGroupName `
-Location $location `
-VM $winVmConfig `
-ErrorAction Stop
I am using this code found on stack , and all connections are correct.
Import-Module Azure
Import-Module Azure.Storage
Get-AzureRmSubscription –SubscriptionName “Production” | Select-AzureRmSubscription
# Username for Azure SQL Database server
$ServerLogin = "username"
# Password for Azure SQL Database server
$serverPassword = ConvertTo-SecureString "abcd" -AsPlainText -Force
# Establish credentials for Azure SQL Database Server
$ServerCredential = new-object System.Management.Automation.PSCredential($ServerLogin, $serverPassword)
# Create connection context for Azure SQL Database server
$SqlContext = New-AzureSqlDatabaseServerContext -FullyQualifiedServerName “myspecialsqlserver.database.windows.net” -Credential $ServerCredential
$StorageContext = New-AzureStorageContext -StorageAccountName 'prodwad' -StorageAccountKey 'xxxxx'
$Container = Get-AzureStorageContainer -Name 'automateddbbackups' -Context $StorageContext
$exportRequest = Start-AzureSqlDatabaseExport -SqlConnectionContext $SqlContext -StorageContainer $Container -DatabaseName 'Users' -BlobName 'autobackupotest.bacpac' -Verbose -Debug
I am getting this error. I have spent hours on this.
Start-AzureSqlDatabaseExport : Cannot bind parameter 'StorageContainer'. Cannot convert the
"Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageContainer" value of type
"Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageContainer" to type
"Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageContainer".
At line:31 char:99
+ ... SqlConnectionContext $SqlContext -StorageContainer $Container -Databa ...
+ ~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Start-AzureSqlDatabaseExport], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.WindowsAzure.Commands.SqlDatabase.Database.Cmdlet.StartAzureSqlDatabaseExport
I am using AzureRM 3.8.0
According to your description and codes, If you want to start a new sqldatabase export.I suggest you could try below codes. It will work well.
$subscriptionId = "YOUR AZURE SUBSCRIPTION ID"
Login-AzureRmAccount
Set-AzureRmContext -SubscriptionId $subscriptionId
# Database to export
$DatabaseName = "DATABASE-NAME"
$ResourceGroupName = "RESOURCE-GROUP-NAME"
$ServerName = "SERVER-NAME"
$serverAdmin = "ADMIN-NAME"
$serverPassword = "ADMIN-PASSWORD"
$securePassword = ConvertTo-SecureString -String $serverPassword -AsPlainText -Force
$creds = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $serverAdmin, $securePassword
# Generate a unique filename for the BACPAC
$bacpacFilename = $DatabaseName + (Get-Date).ToString("yyyyMMddHHmm") + ".bacpac"
# Storage account info for the BACPAC
$BaseStorageUri = "https://STORAGE-NAME.blob.core.windows.net/BLOB-CONTAINER-NAME/"
$BacpacUri = $BaseStorageUri + $bacpacFilename
$StorageKeytype = "StorageAccessKey"
$StorageKey = "YOUR STORAGE KEY"
$exportRequest = New-AzureRmSqlDatabaseExport -ResourceGroupName $ResourceGroupName -ServerName $ServerName `
-DatabaseName $DatabaseName -StorageKeytype $StorageKeytype -StorageKey $StorageKey -StorageUri $BacpacUri `
-AdministratorLogin $creds.UserName -AdministratorLoginPassword $creds.Password
$exportRequest
# Check status of the export
Get-AzureRmSqlDatabaseImportExportStatus -OperationStatusLink $exportRequest.OperationStatusLink
Then you could use Get-AzureRmSqlDatabaseImportExportStatus to see the details information, like below:
I got the same problem after updating some powershell packages which I do not remember exactly. After the update my scripts started to fail.
My solution is :
Install the latest AzureRM from nuget via powershell
Use the other overload of Start-AzureSqlDatabaseExport which utilizes the parameters
-StorageContainerName and -StorageContext rather than -StorageContainer
It looks like if you pass the parameters to the function it will create the container object internally without crashing.
I am trying to create cluster using powershell. Here is script I am executing:
$containerName = "hdfiles"
$location = "Southeast Asia"
$clusterNodes = 2
$userName = "HDUser"
#Generate random password
$rand = New-Object System.Random
$pass = ""
$pass = $pass + [char]$rand.next(97,121) #lower case
$pass = $pass + [char]$rand.next(48,57) #number
$pass = $pass + [char]$rand.next(65,90) #upper case
$pass = $pass + [char]$rand.next(58,62) #special character
1..6 | ForEach { $pass = $pass + [char]$rand.next(97,121) } #6 lower-case characters
$password = ConvertTo-SecureString $pass -AsPlainText -Force
# generate unique random cluster and storage account names
do
{
$clusterName = "hd"
1..6 | ForEach { $clusterName = $clusterName + [char]$rand.next(48,57) }
$storageAccountName = $clusterName + "store"
}
while ((Test-AzureName -Name $storageAccountName -Storage) -and (Test-AzureName -Name $clusterName -Service))
# Create a storage account
Write-Host "Creating storage account..."
New-AzureStorageAccount -StorageAccountName $storageAccountName -Location $location
# Create a Blob storage container
Write-Host "Creating container..."
$storageAccountKey = Get-AzureStorageKey $storageAccountName | %{ $_.Primary }
$destContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey
New-AzureStorageContainer -Name $containerName -Context $destContext
# Create a cluster
Write-Host "Creating HDInsight cluster..."
$credential = New-Object System.Management.Automation.PSCredential ($userName, $password)
New-AzureHDInsightCluster -Name $clusterName -Location $location -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainerName $containerName -ClusterSizeInNodes $clusterNodes -Credential $credential -Version 3.2
But on last line I am getting exception:
New-AzureHDInsightCluster : Validating connection to 'hd662173store.blob.core.windows.net' failed. Inner exception:Could not load file or assembly 'Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
At D:\ProgrammingWorkspace\Edx\Processing BigData with HDInsight\HDILabs\Lab02A\Provision HDInsight.ps1:38 char:1 + New-AzureHDInsightCluster -Name $clusterName -Location $location -Def ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [New-AzureHDInsightCluster], ConfigurationErrorsException
+ FullyQualifiedErrorId : System.Configuration.ConfigurationErrorsException,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.NewAzureHDInsightClusterCmdlet
I am using Azure Powershell release 0.9.7 and Azure SDK 2.7
Looking back on this, the issue you encountered is likely now fixed with the latest version of Azure PowerShell.
To install the latest version of Azure PowerShell, see here:
Installing Azure PowerShell
For samples of how to use HDInsight PowerShell cmdlets (e.g., New-AzureRmHDInsightCluster), see here: Using HDInsight Azure PowerShell cmdlets
I hope this helps!