i am using terraform and i don't get the right parameters to create my glue jobs.
As i am not a terraform pro (i begin), i wonder how it works.
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/glue_job#glue_version
I have not the good updates on my glue job resource using those parameters:
resource "aws_glue_job" "job_name" {
name = "job_name"
description = "job-desc"
role_arn = "${aws_iam_role.service-name.arn}"
max_capacity = 2
max_retries = 1
timeout = 60
command {
script_location = "s3://my_bucket"
python_version = "3"
}
default_arguments = {
"--job-language" = "python"
"--ENV" = "env"
"--spark-event-logs-path" = "s3://my_bucket"
"--job-bookmark-option" = "job-bookmark-enable"
"--glue_version" = "2.0"
"--worker_type" = "G.1X"
"--enable-spark-ui" = "true"
}
execution_property {
max_concurrent_runs = 1
}
}
Idon't know where and how put those params. Could you please help me ?
"--glue_version" = "2.0"
"--worker_type" = "G.1X"
Regards.
The glue_version and worker_type arguments go on the same level as the default_arguments block, not inside of it.
Once you move them out, your resource block may look like this:
resource "aws_glue_job" "job_name" {
name = "job_name"
description = "job-desc"
role_arn = "${aws_iam_role.service-name.arn}"
max_capacity = 2
max_retries = 1
timeout = 60
glue_version = "2.0"
worker_type = "G.1X"
command {
script_location = "s3://my_bucket"
python_version = "3"
}
default_arguments = {
"--job-language" = "python"
"--ENV" = "env"
"--spark-event-logs-path" = "s3://my_bucket"
"--job-bookmark-option" = "job-bookmark-enable"
"--enable-spark-ui" = "true"
}
execution_property {
max_concurrent_runs = 1
}
}
EDIT
The version you are using, 2.30.0 doesn't support these arguments for the aws_glue_job resource.
The glue_version argument was not added until version 2.34.0 of the AWS Provider.
The worker_type argument was not added until version 2.39.0.
You will need to update the provider to support these arguments.
Related
I am trying to create an Azure VM image using packer. My packer template looks like this
variable "version" {
type = string
default = "1.0.0"
}
variable "created_by" {
type = string
}
source "azure-arm" "development_subscription" {
azure_tags = {
CreatedBy = var.created_by
CreatedDate = formatdate("DD/MM/YYYY hh:mm:ss",timestamp())
}
image_offer = "WindowsServer"
image_publisher = "MicrosoftWindowsServer"
image_sku = "2022-datacenter-g2"
managed_image_name = "MyImage_${var.version}"
managed_image_resource_group_name = "Some-RG"
os_type = "Windows"
location = "ukwest"
# client_id = var.client_id
# client_secret = var.client_secret
subscription_id = "e8204745-e84f-4b2e-9e6f-545656fe0922"
vm_size = "Standard_D2s_v3"
winrm_insecure = true
winrm_timeout = "20m"
winrm_use_ssl = true
winrm_username = "packer"
}
However I keep on getting:
==> azure-arm.development_subscription: Waiting for WinRM to become available...
==> azure-arm.development_subscription: Timeout waiting for WinRM.
Other resources I've found online imply I should try increasing the timeout, but this VM doesn't seem likely to take longer than a few seconds to boot. Do I need to do something to disable the system firewall?
I was missing tenant_id. Once I added that, everything worked fine.
I tried your code it also got stuck while connecting to winRM and timed out waiting for the same .
The Major issue I found in your code is that you have not added a communicator ="WinRM" . So ,For that reason the WinRM port doesn't get open and you are not able to connect through it.
So, I added the same as solution in the below code :
variable "version" {
type = string
default = "1.0.0"
}
variable "created_by" {
type = string
default = "ajay"
}
variable "client_secret" {
default = "XXXXXXXXXXXXXXXXXXXXXXXX"
}
variable "client_id" {
default = "XXXXXXXXXXXXXXXXXXXXXXXXXX"
}
source "azure-arm" "development_subscription" {
azure_tags = {
CreatedBy = var.created_by
CreatedDate = formatdate("DD/MM/YYYY hh:mm:ss", timestamp())
}
image_offer = "WindowsServer"
image_publisher = "MicrosoftWindowsServer"
image_sku = "2022-datacenter-g2"
managed_image_name = "MyImage_${var.version}"
managed_image_resource_group_name = "ansumantest"
os_type = "Windows"
location = "ukwest"
client_id = var.client_id
client_secret = var.client_secret
subscription_id = "XXXXXXXXXXXXXXXXXXXX"
vm_size = "Standard_D2s_v3"
communicator = "winrm"
winrm_insecure = true
winrm_timeout = "20m"
winrm_use_ssl = true
winrm_username = "packer"
}
build {
name = "learn-packer"
sources = [
"source.azure-arm.development_subscription"
]
}
Output:
$ terraform -v
Terraform v0.14.6
I have a Terraform plan that sets up alarms for some of my AWS ECS services, and it looks like this
module "ecs_high_cpu_service_aaa_alarm" {
source = "../modules/cw_alarm"
alarm_name = "ecs-high-cpu-service-aaa-alarm"
service_name = "service-aaa"
// Other parameters
}
module "ecs_high_cpu_service_bbb_alarm" {
source = "../modules/cw_alarm"
alarm_name = "ecs-high-cpu-service-bbb-alarm"
service_name = "service-bbb"
// Other parameters
}
module "ecs_high_cpu_service_123_alarm" {
source = "../modules/cw_alarm"
alarm_name = "ecs-high-cpu-service-123-alarm"
service_name = "service-123"
// Other parameters
}
// More alarms with similar setup as above
As you can see, the modules are all set up similarly, differing only in name, alarm_name and service_name parameters. Is there a way to setup a for-loop that will loop over a map to set the modules up for the plan?
From Mark B's (thanks!) comment, this works.
variables.tf
------------
variable "service_map" {
type = map
default = {
service-aaa = "ecs-high-cpu-service-aaa-alarm"
service-bbb = "ecs-high-cpu-service-bbb-alarm"
service-123 = "ecs-high-cpu-service-123-alarm"
}
description = "Service map; key = service name, value = alarm name"
}
main.tf
-------
module "alarms" {
for_each = var.service_map
source = "../modules/cw_alarm"
service_name = each.key
alarm_name = each.value
// Other parameters
}
I ran terraform import for one SQL server & one SQL database. While running the terraform plan I see message 2 to change. But I am not able to find the change in the below plan. It's not showing any null value.
I am not sure what is the change to be in effect.
Here is the information about the terraform plan:
# azurerm_sql_database.sqldb[0m will be updated in-place[0m[0m
2020-12-24T16:01:39.1426150Z [0m [33m~[0m[0m resource "azurerm_sql_database" "sqldb" {
2020-12-24T16:01:39.1426881Z [1m[0mcollation[0m[0m = "SQL_Latin1_General_CP1_CI_AS"
2020-12-24T16:01:39.1427865Z [32m+[0m [0m[1m[0mcreate_mode[0m[0m = "Default"
2020-12-24T16:01:39.1428801Z [1m[0mcreation_date[0m[0m = "2020-07-06T15:20:16.947Z"
2020-12-24T16:01:39.1429581Z [1m[0mdefault_secondary_location[0m[0m = "East US"
2020-12-24T16:01:39.1430271Z [1m[0medition[0m[0m = "GeneralPurpose"
2020-12-24T16:01:39.1474446Z [1m[0mextended_auditing_policy[0m[0m = [
2020-12-24T16:01:39.1481428Z {
2020-12-24T16:01:39.1482165Z retention_in_days = 0
2020-12-24T16:01:39.1483057Z storage_account_access_key = ""
2020-12-24T16:01:39.1483679Z storage_account_access_key_is_secondary = false
2020-12-24T16:01:39.1484293Z storage_endpoint = ""
2020-12-24T16:01:39.1486841Z },
2020-12-24T16:01:39.1487323Z ]
2020-12-24T16:01:39.1488663Z [1m[0mid[0m[0m = "/subscriptions/78bc4018-84c1-4906-94c9-c1d5b84cc907/resourceGroups/rg-us-wus-dev-1/providers/Microsoft.Sql/servers/sql-us-wus-dev/databases/sqldb-us-wus-dev"
2020-12-24T16:01:39.1491489Z [1m[0mlocation[0m[0m = "westus"
2020-12-24T16:01:39.1492160Z [1m[0mmax_size_bytes[0m[0m = "34359738368"
2020-12-24T16:01:39.1492790Z [1m[0mname[0m[0m = "sqldb-us-wus-dev"
2020-12-24T16:01:39.1493436Z [1m[0mread_scale[0m[0m = false
2020-12-24T16:01:39.1494194Z [1m[0mrequested_service_objective_id[0m[0m = "f21733ad-9b9b-4d4e-a4fa-94a133c41718"
2020-12-24T16:01:39.1495057Z [1m[0mrequested_service_objective_name[0m[0m = "GP_Gen5_2"
2020-12-24T16:01:39.1495733Z [1m[0mresource_group_name[0m[0m = "rg-us-wus-dev-1"
2020-12-24T16:01:39.1496437Z [1m[0mserver_name[0m[0m = "sql-us-wus-dev"
2020-12-24T16:01:39.1497190Z [1m[0mtags[0m[0m = {}
2020-12-24T16:01:39.1497905Z [1m[0mzone_redundant[0m[0m = false
2020-12-24T16:01:39.1498494Z
2020-12-24T16:01:39.1498890Z threat_detection_policy {
2020-12-24T16:01:39.1499416Z [1m[0mdisabled_alerts[0m[0m = []
2020-12-24T16:01:39.1500074Z [1m[0memail_account_admins[0m[0m = "Disabled"
2020-12-24T16:01:39.1500670Z [1m[0memail_addresses[0m[0m = []
2020-12-24T16:01:39.1501143Z [1m[0mretention_days[0m[0m = 0
2020-12-24T16:01:39.1501574Z [1m[0mstate[0m[0m = "Disabled"
2020-12-24T16:01:39.1502069Z [1m[0muse_server_default[0m[0m = "Disabled"
2020-12-24T16:01:39.1502411Z }
2020-12-24T16:01:39.1502594Z
2020-12-24T16:01:39.1502851Z timeouts {}
2020-12-24T16:01:39.1503112Z }
2020-12-24T16:01:39.1503279Z
2020-12-24T16:01:39.1503637Z [1m # azurerm_sql_server.sqlserver[0m will be updated in-place[0m[0m
2020-12-24T16:01:39.1504503Z [0m [33m~[0m[0m resource "azurerm_sql_server" "sqlserver" {
2020-12-24T16:01:39.1504979Z [1m[0madministrator_login[0m[0m = "sqladmin"
2020-12-24T16:01:39.1505483Z [32m+[0m [0m[1m[0madministrator_login_password[0m[0m = (sensitive value)
2020-12-24T16:01:39.1506007Z [1m[0mconnection_policy[0m[0m = "Default"
2020-12-24T16:01:39.1506451Z [1m[0mextended_auditing_policy[0m[0m = [
2020-12-24T16:01:39.1506802Z {
2020-12-24T16:01:39.1507156Z retention_in_days = 0
2020-12-24T16:01:39.1507611Z storage_account_access_key = ""
2020-12-24T16:01:39.1508130Z storage_account_access_key_is_secondary = false
2020-12-24T16:01:39.1508695Z storage_endpoint = "https://stuxxwusdev.blob.core.windows.net/"
2020-12-24T16:01:39.1509179Z },
2020-12-24T16:01:39.1509442Z ]
2020-12-24T16:01:39.1510082Z [1m[0mfully_qualified_domain_name[0m[0m = "sql-us-wus-dev.database.windows.net"
2020-12-24T16:01:39.1511114Z [1m[0mid[0m[0m = "/subscriptions/78bc4018-84c1-4906-94c9-c1d5b84cc907/resourceGroups/rg-us-wus-dev-1/providers/Microsoft.Sql/servers/sql-us-wus-dev"
2020-12-24T16:01:39.1511895Z [1m[0mlocation[0m[0m = "westus"
2020-12-24T16:01:39.1512415Z [1m[0mname[0m[0m = "sql-us-wus-dev"
2020-12-24T16:01:39.1512991Z [1m[0mresource_group_name[0m[0m = "wus-dev"
2020-12-24T16:01:39.1513500Z [1m[0mtags[0m[0m = {}
2020-12-24T16:01:39.1514036Z [1m[0mversion[0m[0m = "12.0"
2020-12-24T16:01:39.1514327Z
2020-12-24T16:01:39.1514602Z timeouts {}
2020-12-24T16:01:39.1514890Z }
There are terraform plan symbol meanings, refer to this.
+ create
- destroy
-/+ replace (destroy and then create, or vice-versa if create-before-destroy is used)
~ update in-place i.e. change without destroying
<= read
You can check the ~ mark line to check that the specific attributes will be updated in place.
For example, it will update the retention_in_days from 6 to 0 in the terraform template code.
Please let me know if you still have any questions.
i was able to create vm using terraform but…
when i use the context block im facing an issue
Error: Unsupported block type
on terraform.tf line 34, in resource “opennebula_template” “mytemplate”:
34: context {
Blocks of type “context” are not expected here. Did you mean to define
argument “context”? If so, use the equals sign to assign it a value.
I am adding it exactly as the guide shows in formal terraform docs in here
https://registry.terraform.io/providers/OpenNebula/opennebula/latest/docs/resources/virtual_machine
variable "one_endpoint" {}
variable "one_username" {}
variable "one_password" {}
variable "one_flow_endpoint" {}
provider "opennebula" {
endpoint = var.one_endpoint
flow_endpoint = var.one_flow_endpoint
username = var.one_username
password = var.one_password
}
#########################################################################
resource "opennebula_image" "CentOS7-clone" {
clone_from_image = 35
name = "CentOS7-clone"
datastore_id = 1
persistent = false
permissions = "660"
group = "oneadmin"
}
#########################################################################
resource "opennebula_virtual_machine" "demo" {
count = 1
name = "centos7"
cpu = 2
vcpu = 2
memory = 4096
group = "oneadmin"
permissions = "660"
context {
NETWORK = "YES"
HOSTNAME = "$NAME"
START_SCRIPT ="yum upgrade"
}
graphics {
type = "VNC"
listen = "0.0.0.0"
keymap = "fr"
}
os {
arch = "x86_64"
boot = "disk0"
}
disk {
image_id = opennebula_image.CentOS7-clone.id
size = 10000
target = "vda"
driver = "qcow2"
}
nic {
model = "virtio"
network_id = 7
security_groups = [0]
}
vmgroup {
vmgroup_id = 2
role = "vm-group"
}
tags = {
environment = "dev"
}
timeout = 5
}
you need to define the context block with an equal sign like below:
context = {
NETWORK = "YES"
HOSTNAME = "$NAME"
START_SCRIPT ="yum upgrade"
}
Omitting the equal sing for defining attributes was only supported in Terraform <0.12 (Terraform 0.12 Compatibility for Providers - Terraform by HashiCorp). We have an issue for updating the documentation in the GitHub repository.
I have the following code that will call a module and make target groups for me based off of the information I pass it in the locals variables. This works just fine, my issue being trying to get the arn of each target group it creates in an output.
locals {
targetgroups_beta = {
targetgroup1 = {
name = "example",
path = "/",
environment = "Beta"
}
targetgroup2 = {
name = "example2",
path = "/",
environment = "Beta"
}
}
}
module "target-groups"{
for_each = local.targetgroups_beta
source = ".//modules/targetgroups"
name-beta = each.value.name
path-beta = each.value.path
environment-beta = each.value.environment
vpc-id = "${module.vpc.vpc_id}"
}
The resource name in the module it is calling is target-group, so based off of what I read I should be able to refer to it by something like this:
output{
value = "${aws_lb_target_group.target-group[0].arn}"
}
When I try this I receive the following when running a terraform plan:
"Because aws_lb_target_group.target-group does not have "count" or "for_each" set, references to it must not include an index key. Remove the bracketed index to refer to the single instance of this resource."
My understanding of this is the module that the for_each is calling isn't running a for_each, so I cannot reference the resources in this way. If I do ""${aws_lb_target_group.target-group.arn}" that reference works technically, but includes the arn for every target group and I plan on adding a lot more. Is there a way to take each of these arns out of this list as its own output?
Code in the module that it is calling for reference:
resource "aws_lb_target_group" "target-group" {
name = "example-${var.name-beta}"
port = 80
protocol = "HTTP"
vpc_id = var.vpc-id
deregistration_delay = 5
tags = {
Environment = "${var.environment-beta}"
}
health_check{
healthy_threshold = 2
unhealthy_threshold = 2
timeout = 10
interval = 15
path = var.path-beta
}
}
If I correctly understand, you are using for_each in your target-groups module. If so to get the outputs, you would have to use something like the following in your main.tf file:
module.target-groups[*].arn
The for_each will create multiple modules, not multiple resources in a single module.
Here is good info on using for_each and count with modules in terraform 0.13.
Update for one module
If you want to use only one module, you can do the following:
module "target-groups"{
target_groups_to_create = local.targetgroups_beta
source = ".//modules/targetgroups"
name-beta = each.value.name
path-beta = each.value.path
environment-beta = each.value.environment
vpc-id = "${module.vpc.vpc_id}"
}
Then in the module:
variable "target_groups_to_create" {}
resource "aws_lb_target_group" "target-group" {
for_each = var.target_groups_to_create
name = "example-${each.value.name}"
port = 80
protocol = "HTTP"
vpc_id = var.vpc-id
deregistration_delay = 5
tags = {
Environment = "${each.value.environment}"
}
health_check{
healthy_threshold = 2
unhealthy_threshold = 2
timeout = 10
interval = 15
path = each.value.path
}
}