Creating an Azure VM image with packer - azure

I am trying to create an Azure VM image using packer. My packer template looks like this
variable "version" {
type = string
default = "1.0.0"
}
variable "created_by" {
type = string
}
source "azure-arm" "development_subscription" {
azure_tags = {
CreatedBy = var.created_by
CreatedDate = formatdate("DD/MM/YYYY hh:mm:ss",timestamp())
}
image_offer = "WindowsServer"
image_publisher = "MicrosoftWindowsServer"
image_sku = "2022-datacenter-g2"
managed_image_name = "MyImage_${var.version}"
managed_image_resource_group_name = "Some-RG"
os_type = "Windows"
location = "ukwest"
# client_id = var.client_id
# client_secret = var.client_secret
subscription_id = "e8204745-e84f-4b2e-9e6f-545656fe0922"
vm_size = "Standard_D2s_v3"
winrm_insecure = true
winrm_timeout = "20m"
winrm_use_ssl = true
winrm_username = "packer"
}
However I keep on getting:
==> azure-arm.development_subscription: Waiting for WinRM to become available...
==> azure-arm.development_subscription: Timeout waiting for WinRM.
Other resources I've found online imply I should try increasing the timeout, but this VM doesn't seem likely to take longer than a few seconds to boot. Do I need to do something to disable the system firewall?

I was missing tenant_id. Once I added that, everything worked fine.

I tried your code it also got stuck while connecting to winRM and timed out waiting for the same .
The Major issue I found in your code is that you have not added a communicator ="WinRM" . So ,For that reason the WinRM port doesn't get open and you are not able to connect through it.
So, I added the same as solution in the below code :
variable "version" {
type = string
default = "1.0.0"
}
variable "created_by" {
type = string
default = "ajay"
}
variable "client_secret" {
default = "XXXXXXXXXXXXXXXXXXXXXXXX"
}
variable "client_id" {
default = "XXXXXXXXXXXXXXXXXXXXXXXXXX"
}
source "azure-arm" "development_subscription" {
azure_tags = {
CreatedBy = var.created_by
CreatedDate = formatdate("DD/MM/YYYY hh:mm:ss", timestamp())
}
image_offer = "WindowsServer"
image_publisher = "MicrosoftWindowsServer"
image_sku = "2022-datacenter-g2"
managed_image_name = "MyImage_${var.version}"
managed_image_resource_group_name = "ansumantest"
os_type = "Windows"
location = "ukwest"
client_id = var.client_id
client_secret = var.client_secret
subscription_id = "XXXXXXXXXXXXXXXXXXXX"
vm_size = "Standard_D2s_v3"
communicator = "winrm"
winrm_insecure = true
winrm_timeout = "20m"
winrm_use_ssl = true
winrm_username = "packer"
}
build {
name = "learn-packer"
sources = [
"source.azure-arm.development_subscription"
]
}
Output:

Related

Custom Script extension not Executing on VMSS

I am creating a VMSS using terraform to use for Azure Devops agent pool. I'm able to create VMSS successfully but when I try to run script to enroll it to agent pool, I'm hitting a wall. Nothing seems to work. Here is my TF code:
data "local_file" "template" {
filename = "./agent_install_script.ps1"
}
data "template_file" "script" {
template = data.local_file.template.content
vars = {
agent_name = var.agent_name
pool_name = var.agent_pool_name
token = var.pat_token
user_name = var.vmss_admin_username
logon_password = random_password.vm_password.result
}
}
module "vmss_windows2022g2" {
source = "../modules/vmss_windows"
environment = var.environment
resource_group_name = var.resource_group
vmss_sku = "Standard_DS2_v2"
vmss_nic_subnet_id = module.vnet_mgt.subnet_windows_vmss_id
vmss_nsg_id = module.nsg.vmss_nsg_id
vmss_computer_name = "win2022g2"
vmss_admin_username = var.vmss_admin_username
vmss_admin_password = random_password.vm_password.result
windows_image_id = data.azurerm_image.windows_server2022_gen2.id
vmss_storage_uri = data.azurerm_storage_account.vm_storage.primary_blob_endpoint
overprovision = false
#this will be stored at %SYSTEMDRIVE%\AzureData\CustomData.bin
customData = data.template_file.script.rendered
tags = local.env_tags_map
}
resource "azurerm_virtual_machine_scale_set_extension" "ext" {
name = "InstallDevOpsAgent"
virtual_machine_scale_set_id = module.vmss_windows2022g2.id
publisher = "Microsoft.Azure.Extensions"
type = "CustomScript"
type_handler_version = "2.0"
settings = jsonencode({
"commandToExecute" = "dir C:\\ > C:\\temp\\test.txt"
#"cd C:\\AzureData; mv .\\CustomData.bin .\\install_agent.ps1; powershell -ExecutionPolicy Unrestricted -File .\\install_agent.ps1; del .\\install_agent.ps1;"
})
#protected_settings = var.protected_settings
failure_suppression_enabled = false
auto_upgrade_minor_version = false
automatic_upgrade_enabled = false
provision_after_extensions = []
timeouts {
create = "1h"
}
}
As you can see, I'm copying the powershell script via custom_data and that is working fine with all the variables substituted properly. I have tried executing simple command dir C:\\ > C:\\temp\\test.txt to see if anything works, but am not getting any output.
TF version 1.12, azurerm provider version 3.32.0
Azure DevOps should install an extension on the scale set (and in turn the VM's) which will automatically enrol the agent without the need for a script.
More details here:
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops#lifecycle-of-a-scale-set-agent

Terraform Azure CDN Custom Domain Certificate not supported for this profile

I am trying to enable https for cdn endpoint custom domain. When trying to submit the code, i get the following error:
CertificateType value provided is not supported for this profile for enabling https.
The custom domain code:
resource "azurerm_cdn_endpoint_custom_domain" "endpointfrontend" {
name = "mykappdev"
cdn_endpoint_id = azurerm_cdn_endpoint.cdnendpoint.id
host_name = "${azurerm_dns_cname_record.cnamefrontend.name}.${data.azurerm_dns_zone.dnszone.name}"
cdn_managed_https {
certificate_type = "Dedicated"
protocol_type = "ServerNameIndication"
}
}
The rest of the cdn code:
resource "azurerm_cdn_profile" "cdnprofile" {
name = "mycdn${var.environment}"
location = data.azurerm_resource_group.rg.location
resource_group_name = data.azurerm_resource_group.rg.name
sku = "Standard_Microsoft"
}
resource "azurerm_cdn_endpoint" "cdnendpoint" {
name = "${var.environment}-example"
profile_name = azurerm_cdn_profile.cdnprofile.name
location = azurerm_cdn_profile.cdnprofile.location
resource_group_name = data.azurerm_resource_group.rg.name
is_https_allowed = true
origin {
name = "${var.environment}-origin"
host_name = azurerm_storage_account.frontend.primary_web_host
}
depends_on = [
azurerm_cdn_profile.cdnprofile
]
}
data "azurerm_dns_zone" "dnszone" {
name = "my.app"
resource_group_name = "rg-my"
}
Everything works fine when doing it via UI so the problem has to be in the code.
Edit the block azurerm_cdn_endpoint
resource "azurerm_cdn_endpoint" "cdnendpoint" {
name = "${var.environment}-example"
profile_name = azurerm_cdn_profile.cdnprofile.name
location = azurerm_cdn_profile.cdnprofile.location
resource_group_name = data.azurerm_resource_group.rg.name
is_https_allowed = true
origin {
name = "${var.environment}-origin"
host_name = azurerm_storage_account.frontend.primary_web_host
}
### Code added
delivery_rule {
name = "EnforceHTTPS"
order = "1"
request_scheme_condition {
operator = "Equal"
match_values = ["HTTP"]
}
url_redirect_action {
redirect_type = "Found"
protocol = "Https"
}
}
### End code added
depends_on = [
azurerm_cdn_profile.cdnprofile
]
}
Also, you can check this blog post https://www.emilygorcenski.com/post/migrating-a-static-site-to-azure-with-terraform/
Hope this helps!
After enabling custom https once per hand in the azure portal and than disabling it in portal, it was possible to change it via terraform.
I hope this helps!

Error: waiting for creation of MsSql Database - Terraform failure

I have had to re-develop my pipeline for building out my infrastructure to use local agent pools and change Ubuntu code to windows (PowerShell code).
I am now in the position where I am building out my infrastructure and it failing on the most basic of tasks.
I have created my SQL Server and that seems to be OK. I have also got my logging system infra done OK too, but I am really struggling on building out a DB on my SQL Server.
At the most basic here is my code. Server build OK.
resource "azurerm_mssql_server" "main" {
name = local.sqlServerName
resource_group_name = local.resourceGroupName
location = var.location
version = "12.0"
minimum_tls_version = "1.2"
administrator_login = var.sql_administrator_login
administrator_login_password = var.sql_administrator_login_password
tags = var.tags
}
resource "azurerm_sql_active_directory_administrator" "main" {
server_name = azurerm_mssql_server.main.name
resource_group_name = local.resourceGroupName
login = local.sql_ad_login
tenant_id = data.azurerm_client_config.current.tenant_id
object_id = local.object_id
}
resource "azurerm_sql_firewall_rule" "main" {
name = var.sql_firewall_rule
resource_group_name = local.resourceGroupName
server_name = azurerm_mssql_server.main.name
start_ip_address = "0.0.0.0"
end_ip_address = "0.0.0.0"
}
resource "azurerm_mssql_database" "main" {
name = "${local.raptSqlDatabaseName}-${var.environment}"
server_id = azurerm_mssql_server.main.id
min_capacity = 0.5
max_size_gb = 100
zone_redundant = false
collation = "SQL_Latin1_General_CP1_CI_AS"
sku_name = "GP_S_Gen5_2"
auto_pause_delay_in_minutes = 60
create_mode = "Default"
}
I get an error saying:
Error: waiting for creation of MsSql Database "xxx-xxx-xxx-xxx-Prod" (MsSql Server Name "xxx-xxx-xxx-prod" / Resource Group "rg-xxx-xxx-xxx"): Code="InternalServerError" Message="An unexpected error occured while processing the request.
Before I had to design it all in PS and not bash and use local pool, I never had a problem and this worked fine.
I found this but it's saying the correct error but nothing else seems correct. It is odd because I can build up my other infra fine in the same main.tf file.
https://github.com/hashicorp/terraform-provider-azurerm/issues/13194
I am also noticing that Terraform output is not working:
Here is my output file:
output "sql_server_name" {
value = azurerm_mssql_server.main.fully_qualified_domain_name
}
output "sql_server_user" {
value = azurerm_mssql_server.main.administrator_login
}
output "sql_server_password" {
value = azurerm_mssql_server.main.administrator_login_password
sensitive = true
}
#output "cl_sql_database_name" {
# value = azurerm_mssql_database.cl.name
#}
#output "rapt_sql_database_name" {
# value = azurerm_mssql_database.rapt.name
#}
output "app_insights_instrumentation_key" {
value = azurerm_application_insights.main.instrumentation_key
}
Is there any chance this is linked?
Please use latest terraform version i.e. 1.1.13 and azurerm-provider i.e. 2.92.0 as there were some bugs in previous Azure API version which resulted in 500 Error Code which has been fixed in the recent versions as mentioned in this GitHub Issue . Only after a successful apply , you get the output otherwise it won't get stored if there is an error.
I tested the same code with latest versions on both PowerShell and Bash as below :
provider "azurerm" {
features {}
}
data "azurerm_client_config" "current" {}
locals {
resourceGroupName="ansumantest"
sqlServerName="ansumantestsql"
sql_ad_login="sqladmin"
object_id= data.azurerm_client_config.current.object_id
raptSqlDatabaseName="ansserverdb"
}
variable "location" {
default="eastus"
}
variable "sql_administrator_login" {
default="4dm1n157r470r"
}
variable "sql_administrator_login_password" {
default="4-v3ry-53cr37-p455w0rd"
}
variable "sql_firewall_rule" {
default="ansumantestfirewall"
}
variable "environment" {
default="test"
}
resource "azurerm_mssql_server" "main" {
name = local.sqlServerName
resource_group_name = local.resourceGroupName
location = var.location
version = "12.0"
minimum_tls_version = "1.2"
administrator_login = var.sql_administrator_login
administrator_login_password = var.sql_administrator_login_password
}
resource "azurerm_sql_active_directory_administrator" "main" {
server_name = azurerm_mssql_server.main.name
resource_group_name = local.resourceGroupName
login = local.sql_ad_login
tenant_id = data.azurerm_client_config.current.tenant_id
object_id = local.object_id
}
resource "azurerm_sql_firewall_rule" "main" {
name = var.sql_firewall_rule
resource_group_name = local.resourceGroupName
server_name = azurerm_mssql_server.main.name
start_ip_address = "0.0.0.0"
end_ip_address = "0.0.0.0"
}
resource "azurerm_mssql_database" "main" {
name = "${local.raptSqlDatabaseName}-${var.environment}"
server_id = azurerm_mssql_server.main.id
min_capacity = 0.5
max_size_gb = 100
zone_redundant = false
collation = "SQL_Latin1_General_CP1_CI_AS"
sku_name = "GP_S_Gen5_2"
auto_pause_delay_in_minutes = 60
create_mode = "Default"
}
Output.tf
output "sql_server_name" {
value = azurerm_mssql_server.main.fully_qualified_domain_name
}
output "sql_server_user" {
value = azurerm_mssql_server.main.administrator_login
}
output "sql_server_password" {
value = azurerm_mssql_server.main.administrator_login_password
sensitive = true
}
output "cl_sql_database_name" {
value = azurerm_mssql_database.main.name
}
Output :
Updated Terraform to latest version didnt really help...
I turned on logging with TF_LOGS (env variable in terraform).
Watching the activity logs in the resource group and i noticed it was building the DB server and then there was an error with AAD and then the DB Server build failed after that. So I removed the AAD work in my main.tf - then i reran the pipeline then it worked fine. Phew...

Terraform plan output shows 2 update in place

I ran terraform import for one SQL server & one SQL database. While running the terraform plan I see message 2 to change. But I am not able to find the change in the below plan. It's not showing any null value.
I am not sure what is the change to be in effect.
Here is the information about the terraform plan:
# azurerm_sql_database.sqldb[0m will be updated in-place[0m[0m
2020-12-24T16:01:39.1426150Z [0m [33m~[0m[0m resource "azurerm_sql_database" "sqldb" {
2020-12-24T16:01:39.1426881Z [1m[0mcollation[0m[0m = "SQL_Latin1_General_CP1_CI_AS"
2020-12-24T16:01:39.1427865Z [32m+[0m [0m[1m[0mcreate_mode[0m[0m = "Default"
2020-12-24T16:01:39.1428801Z [1m[0mcreation_date[0m[0m = "2020-07-06T15:20:16.947Z"
2020-12-24T16:01:39.1429581Z [1m[0mdefault_secondary_location[0m[0m = "East US"
2020-12-24T16:01:39.1430271Z [1m[0medition[0m[0m = "GeneralPurpose"
2020-12-24T16:01:39.1474446Z [1m[0mextended_auditing_policy[0m[0m = [
2020-12-24T16:01:39.1481428Z {
2020-12-24T16:01:39.1482165Z retention_in_days = 0
2020-12-24T16:01:39.1483057Z storage_account_access_key = ""
2020-12-24T16:01:39.1483679Z storage_account_access_key_is_secondary = false
2020-12-24T16:01:39.1484293Z storage_endpoint = ""
2020-12-24T16:01:39.1486841Z },
2020-12-24T16:01:39.1487323Z ]
2020-12-24T16:01:39.1488663Z [1m[0mid[0m[0m = "/subscriptions/78bc4018-84c1-4906-94c9-c1d5b84cc907/resourceGroups/rg-us-wus-dev-1/providers/Microsoft.Sql/servers/sql-us-wus-dev/databases/sqldb-us-wus-dev"
2020-12-24T16:01:39.1491489Z [1m[0mlocation[0m[0m = "westus"
2020-12-24T16:01:39.1492160Z [1m[0mmax_size_bytes[0m[0m = "34359738368"
2020-12-24T16:01:39.1492790Z [1m[0mname[0m[0m = "sqldb-us-wus-dev"
2020-12-24T16:01:39.1493436Z [1m[0mread_scale[0m[0m = false
2020-12-24T16:01:39.1494194Z [1m[0mrequested_service_objective_id[0m[0m = "f21733ad-9b9b-4d4e-a4fa-94a133c41718"
2020-12-24T16:01:39.1495057Z [1m[0mrequested_service_objective_name[0m[0m = "GP_Gen5_2"
2020-12-24T16:01:39.1495733Z [1m[0mresource_group_name[0m[0m = "rg-us-wus-dev-1"
2020-12-24T16:01:39.1496437Z [1m[0mserver_name[0m[0m = "sql-us-wus-dev"
2020-12-24T16:01:39.1497190Z [1m[0mtags[0m[0m = {}
2020-12-24T16:01:39.1497905Z [1m[0mzone_redundant[0m[0m = false
2020-12-24T16:01:39.1498494Z
2020-12-24T16:01:39.1498890Z threat_detection_policy {
2020-12-24T16:01:39.1499416Z [1m[0mdisabled_alerts[0m[0m = []
2020-12-24T16:01:39.1500074Z [1m[0memail_account_admins[0m[0m = "Disabled"
2020-12-24T16:01:39.1500670Z [1m[0memail_addresses[0m[0m = []
2020-12-24T16:01:39.1501143Z [1m[0mretention_days[0m[0m = 0
2020-12-24T16:01:39.1501574Z [1m[0mstate[0m[0m = "Disabled"
2020-12-24T16:01:39.1502069Z [1m[0muse_server_default[0m[0m = "Disabled"
2020-12-24T16:01:39.1502411Z }
2020-12-24T16:01:39.1502594Z
2020-12-24T16:01:39.1502851Z timeouts {}
2020-12-24T16:01:39.1503112Z }
2020-12-24T16:01:39.1503279Z
2020-12-24T16:01:39.1503637Z [1m # azurerm_sql_server.sqlserver[0m will be updated in-place[0m[0m
2020-12-24T16:01:39.1504503Z [0m [33m~[0m[0m resource "azurerm_sql_server" "sqlserver" {
2020-12-24T16:01:39.1504979Z [1m[0madministrator_login[0m[0m = "sqladmin"
2020-12-24T16:01:39.1505483Z [32m+[0m [0m[1m[0madministrator_login_password[0m[0m = (sensitive value)
2020-12-24T16:01:39.1506007Z [1m[0mconnection_policy[0m[0m = "Default"
2020-12-24T16:01:39.1506451Z [1m[0mextended_auditing_policy[0m[0m = [
2020-12-24T16:01:39.1506802Z {
2020-12-24T16:01:39.1507156Z retention_in_days = 0
2020-12-24T16:01:39.1507611Z storage_account_access_key = ""
2020-12-24T16:01:39.1508130Z storage_account_access_key_is_secondary = false
2020-12-24T16:01:39.1508695Z storage_endpoint = "https://stuxxwusdev.blob.core.windows.net/"
2020-12-24T16:01:39.1509179Z },
2020-12-24T16:01:39.1509442Z ]
2020-12-24T16:01:39.1510082Z [1m[0mfully_qualified_domain_name[0m[0m = "sql-us-wus-dev.database.windows.net"
2020-12-24T16:01:39.1511114Z [1m[0mid[0m[0m = "/subscriptions/78bc4018-84c1-4906-94c9-c1d5b84cc907/resourceGroups/rg-us-wus-dev-1/providers/Microsoft.Sql/servers/sql-us-wus-dev"
2020-12-24T16:01:39.1511895Z [1m[0mlocation[0m[0m = "westus"
2020-12-24T16:01:39.1512415Z [1m[0mname[0m[0m = "sql-us-wus-dev"
2020-12-24T16:01:39.1512991Z [1m[0mresource_group_name[0m[0m = "wus-dev"
2020-12-24T16:01:39.1513500Z [1m[0mtags[0m[0m = {}
2020-12-24T16:01:39.1514036Z [1m[0mversion[0m[0m = "12.0"
2020-12-24T16:01:39.1514327Z
2020-12-24T16:01:39.1514602Z timeouts {}
2020-12-24T16:01:39.1514890Z }
There are terraform plan symbol meanings, refer to this.
+ create
- destroy
-/+ replace (destroy and then create, or vice-versa if create-before-destroy is used)
~ update in-place i.e. change without destroying
<= read
You can check the ~ mark line to check that the specific attributes will be updated in place.
For example, it will update the retention_in_days from 6 to 0 in the terraform template code.
Please let me know if you still have any questions.

Terraform Azure App Service - ip_restrictions

I am trying to set the IP restrictions block in my Azure App Service App
When performing the Terraform plan or apply, I receive the following error:
Error: azurerm_app_service.app-service-1: : invalid or unknown key: ip_restriction
I used ip_restriction per Terraform Documentation for App Service (Web Apps) Resources
Here is the AppService deployment code i am using:
resource "azurerm_app_service" "app-service-1" {
name = "${var.app_service_1}"
location = "${data.azurerm_resource_group.core-rg.location}"
resource_group_name = "${data.azurerm_resource_group.core-rg.name}"
app_service_plan_id = "${data.azurerm_app_service_plan.app-service-plan-1.id}"
https_only = "True"
enabled = "True"
client_affinity_enabled = "True"
site_config {
always_on = "True"
#default_documents = ""
dotnet_framework_version = "v4.0"
#http2_enabled = ""
#ip_restriction = ""
#java_version = ""
#java_container = ""
#java_container_version = ""
managed_pipeline_mode = "Integrated"
min_tls_version = "1.2"
#php_version = ""
#python_version = ""
remote_debugging_enabled = "False"
#remote_debugging_version = ""
scm_type = "None"
use_32_bit_worker_process = "False"
websockets_enabled = "True"
#ftps_state = ""
}
app_settings {
"KeyVaultURI" = ""
"WEBSITE_NODE_DEFAULT_VERSION" = "6.9.1"
}
ip_restriction {
"ip_address" = ""
}
Thank you
#jamies answer is unfortunately incorrect IP_restriction is not a list taking one or more but a repeatable block.
#gvazzana is the correct format.
However, there is a trap.. that will cause the error you are seeing.
In Tf we are used to typing IP address's in full CIDR format eg 10.23.97.201/23 or 192.68.50.0/24, the azure portal for this section even displays them like this.
But for this particular block, in terraform, you have to do them old school. eg:
site_config {
# For a single IP address
ip_restriction {
ip_address = "81.145.174.78"
}
ip_restriction {
# For an address range
ip_address = "10.240.101.0"
subnet_mask = "255.255.255.0"
}
}
This is of course a pain if you have a long list of address's and ranges.
Now that terraform version 0.12.0 is upon us we should be able to take advantage of the new dynamic block styles and cidrhost and cidrmask functions in order to simplify things.
eg:
dynamic "ip_restriction" {
for_each = var.ip_address_list
content {
ip_address = cidrhost(ip_restriction.value,0)
subnet_mask = cidrmask(ip_restriction.value)
}
}
tested with
Terraform v0.12.13
For those interested, here is the method to use ipRestrictions in Terraform
ip Restrictions is part of the Site_Config {}
See how to use below:
AppService.tf:
resource "azurerm_app_service" "app-service-1" {
name = "${var.app_service_1}"
location = "${data.azurerm_resource_group.core-rg.location}"
resource_group_name = "${data.azurerm_resource_group.core-rg.name}"
app_service_plan_id = "${data.azurerm_app_service_plan.app-service-plan-1.id}"
https_only = "True"
enabled = "True"
client_affinity_enabled = "True"
site_config {
always_on = "True"
#default_documents = ""
dotnet_framework_version = "v4.0"
#http2_enabled = ""
#ip_restriction = ""
#java_version = ""
#java_container = ""
#java_container_version = ""
managed_pipeline_mode = "Integrated"
min_tls_version = "1.2"
#php_version = ""
#python_version = ""
remote_debugging_enabled = "False"
#remote_debugging_version = ""
scm_type = "None"
use_32_bit_worker_process = "False"
websockets_enabled = "True"
#ftps_state = ""
ip_restriction {
ip_address = "${var.ip_address_1}"
}
ip_restriction {
ip_address = "${var.ip_address_2}"
}
ip_restriction {
ip_address = "${var.ip_address_3}"
}
}
app_settings {
"KeyVaultURI" = ""
"WEBSITE_NODE_DEFAULT_VERSION" = "6.9.1"
}
}
So you are running into a syntax error. The documentation can be confusing to read as I have learned over the last year. If you read the section on ip_restriction it says that it takes one or more. This means that it expects an array.
There is also a section of the documentation that tells you that it inside the array it expects an object that has the properties of ip_address and subnet_mask. That is here
So to fix your issue you need the following for ip_restriction.
ip_restriction = [
{
ip_address = "10.0.0.0"
}
]
Hope this helps.

Resources