As far as I understand, Windows Virtual Desktop's host pools can be configured in a pooled (assign a user to a VM with free resources) or personal (dedicated VM per user) mode.
I have some users with special needs (available applications, configuration and VM resources) and unpredictable usage times. Would it be possible to assign specific machines to them and tie their lifecycle to the user login? What I'd like to achieve is to shutdown and deallocate the VM if the user logged out or shutdown the VM, and automatically start it (accepting some initial delay) when logging in, to only pay for the VMs when they are actually needed.
Start/Stop VMs during off-hours
It starts or stops machines on user-defined schedules, provides insights through Azure Monitor logs, and sends optional emails by using action groups. The feature can be enabled on both Azure Resource Manager and classic VMs for most scenarios.
This feature uses Start-AzVm cmdlet to start VMs. It uses Stop-AzVM for stopping VMs.
Prerequisites
The runbooks for the Start/Stop VMs during off hours feature work with an Azure Run As account. The Run As account is the preferred authentication method because it uses certificate authentication instead of a password that might expire or change frequently.
An Azure Monitor Log Analytics workspace that stores the runbook job logs and job stream results in a workspace to query and analyze. The Automation account and Log Analytics workspace need to be in the same subscription and supported region. The workspace needs to already exist, you cannot create a new workspace during deployment of this feature.
Recommended: Use a separate Automation account for working
with VMs enabled for the Start/Stop VMs during off-hours feature.
Azure module versions are frequently upgraded, and their parameters
might change. The feature isn't upgraded on the same cadence and it
might not work with newer versions of the cmdlets that it uses. Before
importing the updated modules into your production Automation
account(s), we recommend you import them into a test Automation
account to verify there aren't any compatibility issues.
Permissions
You must have certain permissions to enable VMs for the Start/Stop VMs during off-hours feature. The permissions are different depending on whether the feature uses a pre-created Automation account and Log Analytics workspace or creates a new account and workspace.
You don't need to configure permissions if you're a Contributor on the subscription and a Global Administrator in your Azure Active Directory (AD) tenant. If you don't have these rights or need to configure a custom role, make sure that you have the permissions described below.
Runbooks
The following link lists the runbooks that the feature deploys to your Automation account. Do NOT make changes to the runbook code. Instead, write your own runbook for new functionality.
Don't directly run any runbook with child appended to its name.
All parent runbooks include the WhatIf parameter. When set to True, the parameter supports detailing the exact behavior the runbook takes when run without the parameter and validates that the correct VMs are targeted. A runbook only performs its defined actions when the WhatIf parameter is set to False.
Main default runbooks:
ScheduledStartStop_Parent
SequencedStartStop_Parent
Variables (used by Runbooks)
The following table lists the variables created in your Automation account. Only modify variables prefixed with External. Modifying variables prefixed with Internal causes undesirable effects.
Main variables to use with your Runbooks:
External_Start_ResourceGroupNames: Comma-separated list of one or more resource groups that are targeted for start actions.
External_Stop_ResourceGroupNames: Comma-separated list of one or more resource groups that are targeted for stop actions.
External_ExcludeVMNames: Comma-separated list of VM names to exclude, limited to 140 VMs. If you add more than 140 VMs to the list, VMs specified for exclusion might be inadvertently started or stopped.
Schedules
Don't enable all schedules, because doing so might create overlapping schedule actions. It's best to determine which optimizations you want to do and modify them accordingly.
Scheduled_StopVM: Runs the ScheduledStopStart_Parent runbook with a parameter of Stop every day at the specified time. Automatically stops all VMs that meet the rules defined by variable assets. Enable the related schedule Scheduled-StartVM.
Scheduled_StartVM: Runs the ScheduledStopStart_Parent runbook with a parameter value of Start every day at the specified time. Automatically starts all VMs that meet the rules defined by variable assets. Enable the related schedule Scheduled-StopVM.
Sequenced-StopVM: Runs the Sequenced_StopStop_Parent runbook with a parameter value of Stop every Friday at the specified time. Sequentially (ascending) stops all VMs with a tag of SequenceStop defined by the appropriate variables. For more information on tag values and asset variables, see Runbooks. Enable the related schedule, Sequenced-StartVM.
Sequenced-StartVM: Runs the SequencedStopStart_Parent runbook with a parameter value of Start every Monday at the specified time. Sequentially (descending) starts all VMs with a tag of SequenceStart defined by the appropriate variables. For more information on tag values and variable assets, see Runbooks. Enable the related schedule, Sequenced-StopVM.
How to enable and configure Start/Stop VMs during Off-hours.
Search for and select Automation Accounts.
On the Automation Accounts page, select your Automation account from the list.
From the Automation account, select Start/Stop VM under Related Resources. From here, you can click Learn more about and enable the solution. If you already have the feature deployed, you can click Manage the solution and find it in the list.
On the Start/Stop VMs during off-hours page for the selected deployment, review the summary information and then click Create.
With the resource created, the Add Solution page appears. You're prompted to configure the feature before you can import it into your Automation account.
On the Add Solution page, select Workspace. Select an existing Log Analytics workspace from the list. If there isn't an Automation account in the same supported region as the workspace, you can create a new Automation account in the next step.
On the Add Solution page if there isn't an Automation account available in the supported region as the workspace, select Automation account. You can create a new Automation account to associate with it by selecting Create an Automation account, and on the Add Automation account page, provide the the name of the Automation account in the Name field.
All other options are automatically populated, based on the Log Analytics workspace selected. You can't modify these options. An Azure Run As account is the default authentication method for the runbooks included with the feature.
After you click OK, the configuration options are validated and the Automation account is created. You can track its progress under Notifications from the menu.
On the Add Solution page, select Configure parameters. The Parameters page appears.
Specify a value for the Target ResourceGroup Names field. The field defines group names that contain VMs for the feature to manage. You can enter more than one name and separate the names using commas (values are not case-sensitive). Using a wildcard is supported if you want to target VMs in all resource groups in the subscription. The values are stored in the External_Start_ResourceGroupNames and External_Stop_ResourceGroupNames variables.
The default value for Target ResourceGroup Names is a *. This setting
targets all VMs in a subscription. If you don't want the feature to
target all the VMs in your subscription, you must provide a list of
resource group names before selecting a schedule.
Specify a value for the VM Exclude List (string) field. This value is the name of one or more virtual machines from the target resource group. You can enter more than one name and separate the names using commas (values are not case-sensitive). Using a wildcard is supported. This value is stored in the External_ExcludeVMNames variable.
Use the Schedule field to select a schedule for VM management by the feature. Select a start date and time for your schedule to create a recurring daily schedule starting at the chosen time. Selecting a different region is not available. To configure the schedule to your specific time zone after configuring the feature, see Modify the startup and shutdown schedules.
To receive email notifications from an action group, accept the default value of Yes in the Email notifications field, and provide a valid email address. If you select No but decide at a later date that you want to receive email notifications, you can update the action group that is created with valid email addresses separated by commas. The following alert rules are created in the subscription:
AutoStop_VM_Child
Scheduled_StartStop_Parent
Sequenced_StartStop_Parent
After you have configured the initial settings required for the feature, click OK to close the Parameters page.
Click Create. After all settings are validated, the feature deploys to your subscription. This process can take several seconds to finish, and you can track its progress under Notifications from the menu.
Scenario 1: Start/Stop VMs on a schedule
This scenario is the default configuration when you first deploy Start/Stop VMs during off-hours. For example, you can configure the feature to stop all VMs across a subscription when you leave work in the evening, and start them in the morning when you are back in the office. When you configure the schedules Scheduled-StartVM and Scheduled-StopVM during deployment, they start and stop targeted VMs.
Configuring the feature to just stop VMs is supported. See Modify the startup and shutdown schedules to learn how to configure a custom schedule.
The time zone used by the feature is your current time zone when you
configure the schedule time parameter. However, Azure Automation
stores it in UTC format in Azure Automation. You don't have to do any
time zone conversion, as this is handled during machine deployment.
To control the VMs that are in scope, configure the variables: External_Start_ResourceGroupNames, External_Stop_ResourceGroupNames, and External_ExcludeVMNames.
You can enable either targeting the action against a subscription and resource group, or targeting a specific list of VMs, but not both.
Target the start and stop action by VM list
Run the ScheduledStartStop_Parent runbook with ACTION set to start.
Add a comma-separated list of VMs (without spaces) in the VMList parameter field. An example list is vm1,vm2,vm3.
Set the WHATIF parameter field to True to preview your changes.
Configure the External_ExcludeVMNames variable with a comma-separated list of VMs (VM1,VM2,VM3), without spaces between comma-separated values.
This scenario does not honor the External_Start_ResourceGroupNames and External_Stop_ResourceGroupnames variables. For this scenario, you need to create your own Automation schedule. For details, see Schedule a runbook in Azure Automation.
Scenario 2: Start/Stop VMs in sequence by using tags
Target the start and stop actions against a subscription and resource group
Add a sequencestart and a sequencestop tag with positive integer values to VMs that are targeted in External_Start_ResourceGroupNames and External_Stop_ResourceGroupNames variables. The start and stop actions are performed in ascending order. To learn how to tag a VM, see Tag a Windows virtual machine in Azure and Tag a Linux virtual machine in Azure.
Modify the schedules Sequenced-StartVM and Sequenced-StopVM to the date and time that meet your requirements and enable the schedule.
Run the SequencedStartStop_Parent runbook with ACTION set to start and WHATIF set to True to preview your changes.
Preview the action and make any necessary changes before implementing against production VMs. When ready, manually execute the runbook with the parameter set to False, or let the Automation schedules Sequenced-StartVM and Sequenced-StopVM run automatically following your prescribed schedule.
Scenario 3: Start or stop automatically based on CPU utilization
Start/Stop VMs during off-hours can help manage the cost of running Azure Resource Manager and classic VMs in your subscription by evaluating machines that aren't used during non-peak periods, such as after hours, and automatically shutting them down if processor utilization is less than a specified percentage.
By default, the feature is pre-configured to evaluate the percentage CPU metric to see if average utilization is 5 percent or less. This scenario is controlled by the following variables and can be modified if the default values don't meet your requirements:
External_AutoStop_MetricName
External_AutoStop_Threshold
External_AutoStop_TimeAggregationOperator
External_AutoStop_TimeWindow
External_AutoStop_Frequency
External_AutoStop_Severity
You can enable and target the action against a subscription and resource group, or target a specific list of VMs.
When you run the AutoStop_CreateAlert_Parent runbook, it verifies that the targeted subscription, resource group(s), and VMs exist. If the VMs exist, the runbook calls the AutoStop_CreateAlert_Child runbook for each VM verified by the parent runbook. This child runbook:
Creates a metric alert rule for each verified VM.
Triggers the AutoStop_VM_Child runbook for a particular VM if the CPU drops below the configured threshold for the specified time interval.
Attempts to stop the VM.
Target the autostop action against all VMs in a subscription
Ensure that the External_Stop_ResourceGroupNames variable is empty or set to * (wildcard).
[Optional] If you want to exclude some VMs from the autostop action, you can add a comma-separated list of VM names to the External_ExcludeVMNames variable.
Enable the Schedule_AutoStop_CreateAlert_Parent schedule to run to create the required Stop VM metric alert rules for all of the VMs in your subscription. Running this type of schedule lets you create new metric alert rules as new VMs are added to the subscription.
Target the autostop action against all VMs in a resource group or multiple resource groups
Add a comma-separated list of resource group names to the External_Stop_ResourceGroupNames variable.
If you want to exclude some of the VMs from the autostop, you can add a comma-separated list of VM names to the External_ExcludeVMNames variable.
Enable the Schedule_AutoStop_CreateAlert_Parent schedule to run to create the required Stop VM metric alert rules for all of the VMs in your resource groups. Running this operation on a schedule allows you to create new metric alert rules as new VMs are added to the resource group(s).
Target the autostop action to a list of VMs
Create a new schedule and link it to the AutoStop_CreateAlert_Parent runbook, adding a comma-separated list of VM names to the VMList parameter.
Optionally, if you want to exclude some VMs from the autostop action, you can add a comma-separated list of VM names (without spaces) to the External_ExcludeVMNames variable.
Configure email notifications
In the Azure portal, click on Alerts under Monitoring, then Manage actions. On the Manage actions page, make sure you're on the Action groups tab. Select the action group called StartStop_VM_Notification.
On the StartStop_VM_Notification page, the Basics section will be filled in for you and can't be edited, except for the Display name field. Edit the name, or accept the suggested name. In the Notifications section, click the pencil icon to edit the action details. This opens the Email/SMS message/Push/Voice pane. Update the email address and click OK to save your changes.
Add a VM
There are two ways to ensure that a VM is included when the feature runs:
Each of the parent runbooks of the feature has a VMList parameter. You can pass a comma-separated list of VM names (without spaces) to this parameter when scheduling the appropriate parent runbook for your situation, and these VMs will be included when the feature runs.
To select multiple VMs, set External_Start_ResourceGroupNames and External_Stop_ResourceGroupNames with the resource group names that contain the VMs you want to start or stop. You can also set the variables to a value of * to have the feature run against all resource groups in the subscription.
Exclude a VM
To exclude a VM from Stop/start VMs during off-hours, you can add its name to the External_ExcludeVMNames variable. This variable is a comma-separated list of specific VMs (without spaces) to exclude from the feature. This list is limited to 140 VMs. If you add more than 140 VMs to this list, VMs that are set to be excluded might be inadvertently started or stopped.
Modify the startup and shutdown schedules
Managing the startup and shutdown schedules in this feature follows the same steps as outlined in Schedule a runbook in Azure Automation. Separate schedules are required to start and stop VMs.
Configuring the feature to just stop VMs at a certain time is supported. In this scenario you just create a stop schedule and no corresponding start schedule.
Ensure that you've added the resource groups for the VMs to shut down in the External_Stop_ResourceGroupNames variable.
Create your own schedule for the time when you want to shut down the VMs.
Navigate to the ScheduledStartStop_Parent runbook and click Schedule. This allows you to select the schedule you created in the preceding step.
Select Parameters and run settings and set the ACTION field to Stop.
Select OK to save your changes.
I believe an Automation Runbook with Start/Stop following a specific schedule doesn't fit the requirements of "unpredictable usage times". Using CPU usage would work when logging off (although even if implemented properly, there could be a small acceptable delay). For log-on I think you have a Catch-22. From what I understand a user is generally tied to a WVD pool and if that pool isn't available, the user can't login. So turning on the machine at log on wouldn't work as for that user to even be able to login would require the machine to be turned on prior.
Assuming your users are not tech savvy and giving them access to an Start/Stop Runbook might not be practical, the only other option I can think of is having an Automation Runbook triggered by a custom callback using webhooks. See Azure Automation Webhooks. With a very simple web interface and just two options provided, (Start WVD, Stop WVD) users can trigger the actions themselves.
If you want explore other options like Azure Logic Apps, you get even more event listeners. Some examples: Users could send an email with specific keyword to an inbox that the Logic App is actively checking, which would then trigger the desired action.
I don't know if you're using Log Analytics for collecting Azure Activity Logs but if yes, a mix with the options above could do the trick.
Use Logic App to listen for an event or Runbooks with webhooks, to give users the flexibility to turn on the machine when they need.
Use Logic App and Log Analytics, to listen for a log out event from the machine or to monitor for CPU utilisation over a period of time and act on the result. See Azure Automation Solution for VM Management
I have to implement something similar myself and these are the options I have in mind. I wish I could provide more concrete examples with actual use cases. Will update this in the future when I get there.
EDIT: Did some digging and stumbled upon this repo that provides more info, with screenshots and also the scripts when compared with the official Azure documentation that seem very vague.
Also, since I mentioned Log Analytics earlier, here's a Log Analytics query you can use to find start/end connections in the past 24hrs.
WVDConnections
| where TimeGenerated > ago(24h)
| where State == "Connected"
| project CorrelationId , UserName , StartTime=TimeGenerated, SessionHostName
| join (WVDConnections
| where State == "Completed"
| project EndTime=TimeGenerated, CorrelationId)
on CorrelationId
| parse SessionHostName with Host "domain" //If applicable use parser to remove part of a value to clean it up.
| project StartTime, EndTime, UserName, Host
| sort by StartTime desc
I need to use the command Get-AzureRMResource and return resources created after a particular date . Is it possible to filter the resources w.r.t creation date. Can someone please help?
The Get-AzureRMResource could not get the creation date of Azure RM Resources. It seems there is no other way to get the creation date except the Activity log.
But still it can return only the items created on past 90 days.
For this issue, you could try to Archive the Azure Activity Log, this option is useful if you would like to retain your Activity Log longer than 90 days (with full control over the retention policy) for audit, static analysis, or backup.
Update:
If you want to get resources created after a particular date, try the command below, it returns the resources created after 11/20/2018 1:57:19 AM.
Get-AzureRmResourceGroupDeployment -ResourceGroupName "<ResourceGroupName>" | Where-Object {$_.Timestamp -gt '11/20/2018 1:57:19 AM'}
This information is available via ARM, but you have to call the API directly rather than the PS Get-AzureRMResource (or Get-AzResource) cmdlets.
See Deleting all resources in an Azure Resource Group with age more than x days.
Essentially, you need to add the $expand=createdTime to your query parameters, ie.:
GET
https://management.azure.com/subscriptions/1237f4d2-3dce-4b96-ad95-677f764e7123/resourcegroups?api-version=2019-08-01&%24expand=createdTime
Like #kwill suggested, this site can also help run the command interactively via your browser and return these results for you:
https://learn.microsoft.com/en-us/rest/api/resources/resources/list#code-try-0
Steps below:
Click on the try it now button
Enter your subscription ID
For a key value name use: $expand
For the key value value use: createdTime
Then run the query and it should produce a JSON file for you
Example
I am trying to create an ARM template to deploy an Azure Log Analytics Workspace via ARM. The template works fine, except it needs to understand which SKUs are valid for the target subscription - PerGB2018 for new subscriptions or one of the older SKUs for non-migrated subscriptions.
Pricing models are detailed here:
https://learn.microsoft.com/en-gb/azure/monitoring-and-diagnostics/monitoring-usage-and-estimated-costs#new-pricing-model-and-operations-management-suite-subscription-entitlements
Available SKUs for workspace creation are listed here:
https://learn.microsoft.com/en-us/rest/api/loganalytics/workspaces/createorupdate
I don't know how to identify which ones are valid for the specific subscription prior to deployment and end up with errors and failing deployments where the default I pick is not valid. I cannot assume the person or system calling the template will understand and have access to the correct set of pricing SKUs. PerGB2018 cannot be used on non-migrated subscriptions so cannot be my default.
Can anyone share a method for determining which SKUs will work BEFORE trying to deploy and thus avoiding an error? I have checked the Monitor and Billing APIs in case it is listed there but cannot see anything, and the network calls from the portal page don't offer much insight :(
My preference is to avoid PowerShell as the rest of the deployment uses BASH to request deployment information and build out the parameter files.
Thank You
Inevitably, after asking the question have had a breakthrough - the BASH script below uses Azure CLI 2 to get an AAD Access Token and store it in token. Next we grab the subscription id and store it in subscriptionId.
Once we have the sub ID and a valid Access Token we use curl to call an API endpoint which lists the date of migration to the new pricing model.
token=$(az account get-access-token | jq ".accessToken" -r)
subscriptionId=$(az account show | jq ".id" -ropt)
optedIn=$(curl -X POST -H "Authorization:Bearer $token" -H "Content-Length:0" https://management.azure.com/subscriptions/$subscriptionId/providers/microsoft.insights/listmigrationdate?api-version=2017-10-01 | jq ".optedInDate" -r)
My understanding is that a value of "null" for optedIn means it is the legacy pricing SKUs.
Shout if you disagree or have a better answer!
I'm trying to deploy an HDInsight cluster using an ARM template via Visual Studio. I've created an Azure Resource Group project in Visual Studio 2015, and added my resource definitions to the template JSON files.
However when I've gone to deploy it (by right-clicking the project, choosing Deploy -> New Deployment, entering my parameters), the output of Visual Studio shows (I've snipped out some boring stuff):
17:19:23 - Build started.
17:19:23 - Project "LaunchHdInsightCluster.deployproj" (StageArtifacts target(s)):
[snip]
17:20:27 - [VERBOSE] 17:20:27 - Resource Microsoft.HDInsight/clusters 'groupbhdinsight' provisioning status is running
17:31:06 - [ERROR] New-AzureRmResourceGroupDeployment : ExpiredAuthenticationToken: The access token expiry UTC time '3/14/2016 5:31:06 PM' is earlier than current UTC time '3/14/2016 5:31:07 PM'.
Note that the deploy only ran for 12 minutes before the access token expired - obviously for deploying an HDInsight cluster this is a problem (takes on average 20 minutes).
I'm just trying to understand what's going on under the hood here, as I can't find documentation for this. i.e:
What creates the access token and how? How long does it last for? I wasn't asked for any Azure creds when deploying - I'm assuming it must be the fact that I'm signed into Visual Studio using the same account I use in Azure, and it 'borrows' the authentication session, but this is just a guess
What determines the expiry time of the access token so I can prevent this happening again?
How do I refresh my authentication token?
What's happening here is that the Azure Resource Group deployment in VS uses the PowerShell Script in the project to do deployment (even though the output is hosted in VS, we use that PS script to do the work). The PowerShell script is authenticated by using the token from your VS sign in. That token is only good for an hour and then VS will refresh it. Once it's handed off to PowerShell though, PowerShell doesn't automatically refresh it. So if you have the token for 59 minutes, it's going to expire soon after you start the deployment. The token could last for an hour, or anything less than that. We're working on a fix for this (i.e. have PowerShell automatically refresh the token) but that's a month or so out yet. See: https://github.com/Azure/azure-powershell/issues/1068
Workarounds: Unfortunately there's no good work around from VS. But...
As observed the deployment will continue just fine in Azure, it's just that VS/PS can no longer poll for status. You can monitor the deployment via the portal or PowerShell.
If you drop to PowerShell and run the script, PowerShell will automatically refresh the token when you login with credentials - you can get the exact command that VS runs by sifting through the output window - this doc also gives an overview of running the script manually: https://azure.microsoft.com/en-us/documentation/articles/vs-azure-tools-resource-groups-how-script-works/
Hope that helps...
I bet it was a transient issue. I retried deployment (needed to modify my ARM template) and now it succeeded.
Please check your Azure Resource Group in the portal. You will likely have your resources up and running.
#Cleverguy25 provided an explanation of how I believe the deployment process work.
I am not sure, but I believe that the New-AzureRmResourceGroupDeployment uploads your template file and sets up a deployment to happen in the cloud. Then it queries the deployment to see if it is done and outputs the resources as they are created. Obviously those queries error when the token expires. But the deployment should continue.
You could ignore this error and query the deployment or resource group yourself, to see when it is done.
I follow this post, and simply execute 'Clear-AzureRmContext' this command, then reconnect to Azure, using 'connect-AzAccount', the issue resolved.
https://github.com/Azure/azure-powershell/issues/6585
Open a new powershell and get the current metadata used to authenticate Azure Resource Manager requests using Clear-AzureRmContext.
This worked the magic for me.
I am attempting to create a SharePoint Server Farm using the option via the New menu in Azure. I've tried like 5 times now and each time, it fails after about an hour.
The error is so cryptic that I cannot figure out exactly what is going on.
The operation name is: Microsoft.ClassicCompute/virtualMachines/extensions/write
The status is: Failed
And then another,
The operation name is: Update deployment
The status is: Failed.
Thoughts?
I've had a similar experience using the "new" preview portal when deploying Databases. This portal uses Resource Manager to deploy it's resources which is a new way of doing things. Anyhow, I used the PowerShell command:
Get-AzureResourceGroupLog -Name ResourceGroupName
This provided me the detail to understand what was happening with the deployment. I had to adjust my Datacenter because the one I was trying to use was hot.
I talk a little more about the other options this command has here:
http://devian.co/?p=1181