I am using Terraform to create an Automation Account in Azure.
The following resource in Azure provider does the job: azurerm_automation_account.
Ok. So I got my AA created... here is when problems arise.
"Run As" account: there seems to be a way to create it from Terraform... but the process is cumbersome. I have lost hope, and will probably resort to enable it manually from Azure portal (it is just one click)... but it will brake my automation pipeline :(
"Start/Stop VM Solution": I need the powershell runbooks in this solution to start-stop VMs according to a given schedule. There is a resource in Azure provider called "azurerm_automation_runbook". It has 2 useful arguments to reference runbook scripts:
"content": with it I could "load" a local powershell script content. I know this would work (I could manually download the .ps1 script used by "Start/Stop VM Solution" and use "content" to load it), but I would be missing any fixes/updates made by Microsoft in its code)
"publish_content_link": by which I could point to the URI of a given powershell runbook. I have looked in the "Runbook Gallery" for the runbooks contained in the "Start/Stop VM Solution" (not found them). Anyone had any luck with this? A different approach could be to "create" the "Start/Stop VM Solution" from a Terraform script (this will automatically populate the desired runbooks in my Automation Account)... but not sure if this would be possible.
Thanks in advance.
For point 1: I also found it very challenging and while things have improved lately, there still doesn't seem to be an easy, straight forward way of creating the Run As Account. I eventually resorted to creating it manually from the Azure Portal but below are potential areas you can explore:
I'm not sure if you've considered using the external data source from terraform to execute the Powershell script from Microsoft. It's still a pain because of the last step where you have to authenticate manually, but it still brings you closer to having a blueprint of your environment. Although I'm not sure how it would behave if running this Terraform script a second time.
For point 2: Could you confirm that the script you want to use is a Powershell script and not a Powershell Workflow script? Also could you please elaborate on this approach (I have a feeling that might be the best approach):
A different approach could be to "create" the "Start/Stop VM Solution" from a Terraform script (this will automatically populate the desired runbooks in my Automation Account)
If you look at the Runbooks Gallery, you'll see most of these Powershell scripts have not been updated for many years and are still working fine. If this will be used in a production environment, it would be better if you have control over the changes and update then at your convenience. If you want to get the URI, you can just click on 'View Source Project' and it will lead you to the GitHub repo. E.g. for the Runbook Stop-Start-AzureVM (Scheduled VM Shutdown/Startup).
You'll also notice most of the scripts is submitted by external parties. If you link to a URI that's maintained by someone else and that person publishes malicious code in there or even accidentally messes up the code, it's not desirable. But again I'm not sure as to the extent of your automation (e.g. if you expect to execute the terraform script once a month to ensure the Runbook is up to date)
If I get the scripts from somewhere, I'll validate it prior to using them in my environment.
data "local_file" "start_vm_parallel" {
filename = "./scripts/start-vm-parallel.ps1"
}
resource "azurerm_automation_runbook" "start_vm_parallel" {
name = local.NAME
location = local.REGION
resource_group_name = local.RG
automation_account_name = azurerm_automation_account.automation_prod.name
log_verbose = "true"
log_progress = "true"
description = "This runbook starts VMs in parallel based on a matching tag value"
runbook_type = "PowerShellWorkflow"
content = data.local_file.start_vm_parallel.content
publish_content_link {
uri = "https://path.to.script/script.ps1"
}
}
If you're using a Powershell Workflow, you need to make sure that the Runbook name matches the workflow name inside the script.
One last thing to remember before you even start using your Runbooks, is to update the modules by creating a 'modules update' Runbook from the Azure Automation team and running it on schedule, once a month.
Related
We have a requirement to force certain tags to our azure resources. Some tags which we need to enforce from subscription level say(eg: st1=st1, st2=st2..), But some are needed to be resourcegroup level(rt1-rt1, rt2=rt2..) and others are to specific resource type (like, aks, appservice, storage account).
By going through the MS doc I found that this can be achieved using azure policy. So the plan is to create azure policy with allowed tags and tag values which need to be enforced on
i) in subscription level
2) resource group level
3) Resource type wise
4) and for other remaining app specific tags to the resources use "az tag update" command.
We need to use automated solution to achieve all these with Azurepipleine and shell commands or scripts as we have only linux machines.
so for the 4th point i got some working pipeline solution to add the app specific tags to the resource level.
But for the requirements 1 to 3, will there any arm or script already available , so that we can integrate them with our azurepipelines.
Any automated solution or suggestion you already tried on this?
update on 12/12
As this docs for Use Tags to organize your azure resources sharing to us, you could use rest api or sdk to add the tags for your resource.
And for the loop for giving the resources with the tags from the resource groups. You could look into this potential workaround for reference. Bash Script
I have created a CD pipeline in Azure DevOps that will deploy an Azure Automation account and a runbook , shedule , jobshedule through ARM templates.
All working fine except when rerunning the template. My template is a part of a large deployment process that is still under construction so until the total scope is finished the ARM template that creates the runbook, shedule, jobshedule will rerun with every release.
The problem right now is the following: Whenever I rerun the template with a new release pipeline, I receive following error
A job schedule for the specified runbook and schedule already
exists.
At first I tried to be smart so added a GUI before the name of my jobshedule but the shedule itself attaches the runbook with the shedule and the deployment was smart enough to figure it out that the shedule was already connected to the runbook. Is there a way of making this still within the DevOps mindset / process so that I can rerun my templates with no problem.
The workaround solution I have created atm is to delete the shedule at every deployment but that seems like a very bad workaround.
Related feature request from UserVoice / feedback forum is here that's currently in triaged state.
Job Schedule id need to be unique for each deployment as per azure documentation.
I've been trying to find a way to run a simple command against one of my existing Azure VMs using Azure Data Factory V2.
Options so far:
Custom Activity/Azure Batch won't let me add existing VMs to the pool
Azure Functions - I have not played with this but I have not found any documentation on this using AZ Functions.
Azure Cloud Shell - I've tried this using the browser UI and it works, however I cannot find a way of doing this via ADF V2
The use case is the following:
There are a few tasks that are running locally (Azure VM) in task scheduler that I'd like to orchestrate using ADF as everything else is in ADF, these tasks are usually python applications that restore a SQL Backup and or purge some folders.
i.e. sqdb-restore -r myDatabase
where sqldb-restore is a command that is recognized locally after installing my local python library. Unfortunately the python app needs to live locally in the VM.
Any suggestions? Thanks.
Thanks to #martin-esteban-zurita, his answer helped me to get to what I needed and this was a beautiful and fun experiment.
It is important to understand that Azure Automation is used for many things regarding resource orchestration in Azure (VMs, Services, DevOps), this automation can be done with Powershell and/or Python.
In this particular case I did not need to modify/maintain/orchestrate any Azure resource, I needed to actually run a Bash/Powershell command remotely into one of my existing VMs where I have multiple Powershell/Bash commands running recurrently in "Task Scheduler".
"Task Scheduler" was adding unnecessary overhead to my data pipelines because it was unable to talk to ADF.
In addition, Azure Automation natively only runs Powershell/Python commands in Azure Cloud Shell which is very useful to orchestrate resources like turning on/off Azure VMs, adding/removing permissions from other Azure services, running maintenance or purge processes, etc, but I was still unable to run commands locally in an existing VM. This is where the Hybrid Runbook Worker came into to picture. A Hybrid worker group
These are the steps to accomplish this use case.
1. Create an Azure Automation Account
2. Install the Windows Hybrid Worker in my existing VM . In my case it was tricky because my proxy was giving me some errors. I ended up downloading the Nuget Package and manually installing it.
.\New-OnPremiseHybridWorker.ps1 -AutomationAccountName <NameofAutomationAccount> -AAResourceGroupName <NameofResourceGroup>
-OMSResourceGroupName <NameofOResourceGroup> -HybridGroupName <NameofHRWGroup>
-SubscriptionId <AzureSubscriptionId> -WorkspaceName <NameOfLogAnalyticsWorkspace>
Keep in mind that in the above code, you will need to find your own parameter values, the only parameter that does not have to be found and will be created is HybridGroupName this will define the name of the Hybrid Group
3. Create a PowerShell Runbook
[CmdletBinding()]
Param
([object]$WebhookData) #this parameter name needs to be called WebHookData otherwise the webhook does not work as expected.
$VerbosePreference = 'continue'
#region Verify if Runbook is started from Webhook.
# If runbook was called from Webhook, WebhookData will not be null.
if ($WebHookData){
# Collect properties of WebhookData
$WebhookName = $WebHookData.WebhookName
# $WebhookHeaders = $WebHookData.RequestHeader
$WebhookBody = $WebHookData.RequestBody
# Collect individual headers. Input converted from JSON.
$Input = (ConvertFrom-Json -InputObject $WebhookBody)
# Write-Verbose "WebhookBody: $Input"
#Write-Output -InputObject ('Runbook started from webhook {0} by {1}.' -f $WebhookName, $From)
}
else
{
Write-Error -Message 'Runbook was not started from Webhook' -ErrorAction stop
}
#endregion
# This is where I run the commands that were in task scheduler
$callBackUri = $Input.callBackUri
# This is extremely important for ADF
Invoke-WebRequest -Uri $callBackUri -Method POST
4. Create a Runbook Webhook pointing to the Hybrid Worker's VM
4. Create a webhook activity in ADF where the above PowerShell runbook script will be called via a POST Method
Important Note: When I created the webhook activity it was timing out after 10 minutes (default), so I noticed in the Azure Automation Account that I was actually getting INPUT data (WEBHOOKDATA) that contained a JSON structure with the following elements:
WebhookName
RequestBody (This one contains whatever you add in the Body plus a default element called callBackUri)
All I had to do was to invoke the callBackUri from Azure Automation. And this is why in the PowerShell runbook code I added Invoke-WebRequest -Uri $callBackUri -Method POST. With this, ADF was succeeding/failing instead of timing out.
There are many other details that I struggled with when installing the hybrid worker in my VM but those are more specific to your environment/company.
This looks like a use case that is supported with Azure Automation, using a hybrid worker. Try reading here: https://learn.microsoft.com/en-us/azure/automation/automation-hybrid-runbook-worker
You can call runbooks with webhooks in ADFv2, using the web activity.
Hope this helped!
How can you get the Azure automation script for just a single service?
In this case I've selected an App Service Plan, but when I click on it's "Automation script" in the Azure Portal, I get the script for everything (minus the various resource types "that cannot be exported yet" )
Unfortunately, export function only allows to export everything in the resource group, so you get an ARM template that represents the resource group.
If you want to get an ARM template solely for that resource, please deploy it into it's own resource group and then use the Automation script option.
Alternatively, you can use armviz to get a visual representation. When you click on a particular element, it will take you directly to that object in the arm template. Not exactly what you are looking for, but might give you enough.
You can't export just a single resource if there are multiple resources in the resource group, but maybe visual studio can help.
If you export the template and load it in VS as an arm template project, you can easily delete the other resources from the sidebar and leave just the one you need. You might also need to clean up the vars/params though.
I am attempting to create a SharePoint Server Farm using the option via the New menu in Azure. I've tried like 5 times now and each time, it fails after about an hour.
The error is so cryptic that I cannot figure out exactly what is going on.
The operation name is: Microsoft.ClassicCompute/virtualMachines/extensions/write
The status is: Failed
And then another,
The operation name is: Update deployment
The status is: Failed.
Thoughts?
I've had a similar experience using the "new" preview portal when deploying Databases. This portal uses Resource Manager to deploy it's resources which is a new way of doing things. Anyhow, I used the PowerShell command:
Get-AzureResourceGroupLog -Name ResourceGroupName
This provided me the detail to understand what was happening with the deployment. I had to adjust my Datacenter because the one I was trying to use was hot.
I talk a little more about the other options this command has here:
http://devian.co/?p=1181