Azure's New-AzureRmHDInsightCluster command does not accept documented ComponentVersion option - azure

Question

According to MS's docs for New-AzureRmHDInsightCluster,
it should accept -ComponentVersion as an option:
$httpCredential = New-Object System.Management.Automation.PSCredential ($httpUserName, $clusterpassword)
$sparkConfig = New-Object "System.Collections.Generic.Dictionary``2[System.String,System.String]"
$sparkConfig.Add("spark", "2.1")
New-AzureRmHDInsightCluster `
-ClusterName mycluster `
-ComponentVersion $sparkConfig `
-ClusterSizeInNodes 4 `
-HttpCredential $httpCredential `
-Location "Central US" `
-OSType Linux `
-ResourceGroupName tstcluster
However, this command results in:
##[error]A parameter cannot be found that matches parameter name 'ComponentVersion'.
Is there a way to select the Spark version required? We used to use:
Add-AzureRmHDInsightComponentVersion -Config $config -ComponentName "Spark" -ComponentVersion "2.1"
But that's now rejected:
##[error]The term 'Add-AzureRmHDInsightComponentVersion' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
EDIT:
A couple of factors that were key to solving this: First, this script is run by an Azure PowerShell Script task in an Azure Dev Ops pipeline. Second, the PowerShell version used to run this script was 1.*.

Answer 1

Try the command below, it works fine on my side. Make sure you have installed the AzureRM.HDInsight powershell module, you could check it with Get-Module AzureRM.HDInsight.
Sample command:
$httpUserName = "joyhd"
$clusterpassword = ConvertTo-SecureString "<password>" -AsPlainText -Force
$httpCredential = New-Object System.Management.Automation.PSCredential($httpUserName, $clusterpassword)
$SshCredential = New-Object System.Management.Automation.PSCredential($httpUserName, $clusterpassword)
$storageAccountName = "<storageAccountName>"
$storageAccountKey = "xxxxxxx"
$storageContainer = "testhd"
New-AzureRmHDInsightClusterConfig `
| Add-AzureRmHDInsightComponentVersion `
-ComponentName "Spark" `
-ComponentVersion "2.1" `
| New-AzureRmHDInsightCluster `
-ClusterName joytesthd `
-ClusterType "Spark" `
-ClusterSizeInNodes 4 `
-HttpCredential $httpCredential `
-Location "eastus" `
-OSType Linux `
-ResourceGroupName joywebapp `
-DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" `
-DefaultStorageAccountKey $storageAccountKey `
-DefaultStorageContainer $storageContainer `
-SshCredential $SshCredential
Result:

Answer 2

MS was not able to explain why a script that worked in Azure Dev Ops (then VSTS) stopped working. That is, why Add-AzureRmHDInsightComponentVersion was supported in January 2018 but not in September of that same year.
Their solution was to select the latest version of PowerShell available in a pipeline (3.*) and set the Preferred Azure PowerShell Version to 3.8.0.
Making those changes made the existing script workable again.