Create a Data Pipeline in Azure - azure

Question

I've a class which make some extract, transform an load to a dataset located in a different JSON files.
This process work Ok. But, I've the necessity to process manually every month. I submitt an spark application in intelliJ (and submit an Scalla Singleton Object with the transformation)
So, I'm trying to automate this process. But, I didn't find documentation or a tutorial to known what is the best service to accomplish this objective.
The processs Should:
Create a HDInsight Spark Cluster
Run The process (An Scala Class)
Delete the HDInsight Spark Cluster created before
I've searched but the links I find (looking for "Create on demand HD insight spark cluster") are the following:
Access datalake from Azure datafactory V2 using on demand HD Insight
cluster
How to create Azure on demand HD insight Spark cluster using Data
Factory
Other options I've searched:
Host and run your PowerShell scripts in Azure
Azure Logic Apps
Azure Automation
Thanks!

Answer 1

Here are the process which you want to
Create a HDInsight Spark Cluster
Using power shell it should be easy to create HDInsight cluster, here is a sample code:
### Create a Spark 2.3 cluster in Azure HDInsight
# Default cluster size (# of worker nodes), version, and type
$clusterSizeInNodes = "1"
$clusterVersion = "3.6"
$clusterType = "Spark"
# Create the resource group
$resourceGroupName = Read-Host -Prompt "Enter the resource group name"
$location = Read-Host -Prompt "Enter the Azure region to create resources in, such as 'Central US'"
$defaultStorageAccountName = Read-Host -Prompt "Enter the default storage account name"
New-AzResourceGroup -Name $resourceGroupName -Location $location
# Create an Azure storage account and container
# Note: Storage account kind BlobStorage can only be used as secondary storage for HDInsight clusters.
New-AzStorageAccount `
-ResourceGroupName $resourceGroupName `
-Name $defaultStorageAccountName `
-Location $location `
-SkuName Standard_LRS `
-Kind StorageV2 `
-EnableHttpsTrafficOnly 1
$defaultStorageAccountKey = (Get-AzStorageAccountKey `
-ResourceGroupName $resourceGroupName `
-Name $defaultStorageAccountName)[0].Value
$defaultStorageContext = New-AzStorageContext `
-StorageAccountName $defaultStorageAccountName `
-StorageAccountKey $defaultStorageAccountKey
# Create a Spark 2.3 cluster
$clusterName = Read-Host -Prompt "Enter the name of the HDInsight cluster"
# Cluster login is used to secure HTTPS services hosted on the cluster
$httpCredential = Get-Credential -Message "Enter Cluster login credentials" -UserName "admin"
# SSH user is used to remotely connect to the cluster using SSH clients
$sshCredentials = Get-Credential -Message "Enter SSH user credentials" -UserName "sshuser"
# Set the storage container name to the cluster name
$defaultBlobContainerName = $clusterName
# Create a blob container. This holds the default data store for the cluster.
New-AzStorageContainer `
-Name $clusterName `
-Context $defaultStorageContext
$sparkConfig = New-Object "System.Collections.Generic.Dictionary``2[System.String,System.String]"
$sparkConfig.Add("spark", "2.3")
# Create the HDInsight cluster
New-AzHDInsightCluster `
-ResourceGroupName $resourceGroupName `
-ClusterName $clusterName `
-Location $location `
-ClusterSizeInNodes $clusterSizeInNodes `
-ClusterType $clusterType `
-OSType "Linux" `
-Version $clusterVersion `
-ComponentVersion $sparkConfig `
-HttpCredential $httpCredential `
-DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
-DefaultStorageAccountKey $defaultStorageAccountKey `
-DefaultStorageContainer $clusterName `
-SshCredential $sshCredentials
Get-AzHDInsightCluster `
-ResourceGroupName $resourceGroupName `
-ClusterName $clusterName
Run The process (An Scala Class)
You can refer this link to submit an application job remotely to the Spark cluster:
https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-create-standalone-application#run-the-application-on-the-apache-spark-cluster
Delete the HDInsight Spark Cluster created before
Cleaning up the cluster , you can achieve it using powershell, here is a sample code for the same;
# Removes the specified HDInsight cluster from the current subscription.
Remove-AzHDInsightCluster `
-ResourceGroupName $resourceGroupName `
-ClusterName $clusterName
# Removes the specified storage container.
Remove-AzStorageContainer `
-Name $clusterName `
-Context $defaultStorageContext
# Removes a Storage account from Azure.
Remove-AzStorageAccount `
-ResourceGroupName $resourceGroupName `
-Name $defaultStorageAccountName
# Removes a resource group.
Remove-AzResourceGroup `
-Name $resourceGroupName
Additional reference:
https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-jupyter-spark-sql-use-powershell
https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/data-factory/v1/data-factory-build-your-first-pipeline-using-powershell.md
Hope it helps.