Reading through the docs here:
https://www.terraform.io/docs/providers/google/r/compute_backend_service.html
We can define backend service:
resource "google_compute_backend_service" "kubernetes-nginx-prod" {
name = "kubernetes-nginx-prod"
health_checks = [google_compute_health_check.kubernetes-nginx-prod-healthcheck.self_link]
backend {
group = replace(google_container_node_pool.pool-1.instance_group_urls[0], "instanceGroupManagers", "instanceGroups")
# TODO missing port 31443
}
}
It seems we are unable to set backend service port via the Terraform settings:
Recreating the backend service without this settings actually leads to downtime for us and the port must be written manually.
We need to reference the port name that we gave in the instance group for e.g.
resource "google_compute_backend_service" "test" {
name = "test-service"
port_name = "test-port"
protocol = "HTTP"
timeout_sec = 5
health_checks = []
backend {
group = "${google_compute_instance_group.test-ig.self_link}"
}
}
resource "google_compute_instance_group" "test-ig" {
name = "test-ig"
instances = []
named_port {
name = "test-port"
port = "${var.app_port}"
}
zone = "${var.zone}"
}
I resolved this by using a terraform data source to extract the instance group data from my google_container_node_pool, and then appending a google_compute_instance_group_named_port resource to it.
NOTE my google_container_node_pool spans 3 zones (a,b,c) and the below code highlights the solution for only zone c
# extract google_compute_instance_group from google_container_node_pool
data "google_compute_instance_group" "instance_group_zone_c" {
# .2 refers to the last of the 3 instance groups in my node pool (zone c)
name = regex("([^/]+)/?$", "${google_container_node_pool.k8s_node_pool.instance_group_urls.2}").0
zone = regex("${var.region}-?[abc]", "${google_container_node_pool.k8s_node_pool.instance_group_urls.2}")
project = var.project_id
}
# define a named port to attach to the google_compute_instance group in my node pool
resource "google_compute_instance_group_named_port" "named_port_zone_c" {
group = data.google_compute_instance_group.instance_group_zone_c.id
zone = data.google_compute_instance_group.instance_group_zone_c.zone
name = "port31090"
port = 31090
}
Related
Here is the setup
a module to create CloudFront distribution
use CloudFront distribution module to create a distribution for web app
add route53 record for the web app to point to CloudFront distribution
Here is the code
locals {
globals = jsondecode(file("${path.module}/../../../globals.json"))
}
module "cloudfront_distribution" {
source = "../../../modules/cloudfront-distribution/"
env = "prod"
resource_name = "www-example-com"
cloudfront_domain_name = "www.example.com"
cloudfront_domain_aliases = ["www.example.com"]
origin_domain_name = "app.example-origin.com"
subdomain_name = ""
price_class = "PriceClass_100"
origin_https_port = 443
origin_http_port = 80
origin_ssl_protocols = ["TLSv1.2"]
is_route53_record_needed = false
is_prerender_function_attached = true
}
resource "aws_route53_record" "www_example_com" {
zone_id = "123456"
name = "www"
type = "CNAME"
ttl = 5
records = [module.cloudfront_distribution.domain_name]
}
I am getting an error in the last line where I am trying to reference to the domain name of the CloudFront created to create the route53 record. This domain name is something like this d8493jg8r.cloudfront.net
You have to declare the distribution as an output inside the module. Then you can reference the module's outputs at the top level.
I followed the instructions from here: https://neo4j.com/docs/operations-manual/4.4/kubernetes/quickstart-cluster/server-setup/ and deployed a cluster of three core members using Terraform.
Used helm-charts: https://github.com/neo4j/helm-charts/releases/tag/4.4.10
Used neo4j version: Neo4j 4.4.11 enterprise
The code structure is as follows:
module/neo4j:
\-main.tf
\-variables.tf
\--core-1/main.tf
\--core-1/variables.tf
\--core-1/core-1.values.yaml
\--core-2/main.tf
\--core-2/variables.tf
\--core-2/core-2.values.yaml
\--core-3/main.tf
\--core-3/variables.tf
\--core-3/core-3.values.yaml
So the root main.tf creates modules of each core. Nothing special, nothing fancy.
The helm deployment is as follows:
resource "helm_release" "neo4j-core-1" {
name = "neo4j-core-1"
chart = "https://github.com/neo4j/helm-charts/releases/download/${var.chart_version}/neo4j-cluster-core-${var.chart_version}.tgz"
namespace = var.namespace
wait = false
values = [
templatefile("${path.module}/core-1.values.yaml", {
share_secret = var.share_secret_name
share_name = var.share_name
share_dir = var.share_dir
image_name = var.image
image_version = var.image_version
})
]
timeout = 600
force_update = true
reset_values = true
set {
name = "neo4j.name"
value = "neo4j-cluster"
}
set_sensitive {
name = "neo4j.password"
value = var.password
}
set {
name = "dbms.mode"
value = "CORE"
}
# backup configuration
set {
name = "dbms.backup.enabled"
value = true
}
set {
name = "neo4j.resources.memory" # sets both requests and limit
value = var.memory
}
set {
name = "neo4j.resources.cpu" # sets both requests and limit
value = var.cpu
}
set {
name = "dbms.memory.heap.initial_size"
value = var.dbms_memory
}
set {
name = "dbms.memory.heap.max_size"
value = var.dbms_memory
}
set {
name = "dbms.memory.pagecache.size"
value = var.dbms_memory
}
set {
name = "causal_clustering.minimum_core_cluster_size_at_formation"
value = 3
}
set {
name = "causal_clustering.minimum_core_cluster_size_at_runtime"
value = 3
}
set {
name = "causal_clustering.discovery_type"
value = "K8S"
}
dynamic "set" {
for_each = local.nodes
content {
name = "nodeSelector.${set.key}"
value = set.value
}
}
}
The problem I am facing is: The deployment passes like only 1 out of 10 times. Whenever the deployment fails, it is due to a time-out of the Terraform helm_release of one or two core members stating: "Secret "neo4j-cluster-auth" exists.
Looking into the log of the one (or two) members already deployed, the startup failed, because the cluster is missing members. (initialDelaySeconds have been configured for each core member and have been increased testwise too)
kubernetes pods
2022-11-17 08:59:22.738+0000 ERROR Failed to start Neo4j on 0.0.0.0:7474.
java.lang.RuntimeException: Error starting Neo4j database server at /var/lib/neo4j/data/databases
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:227) ~[neo4j-4.4.11.jar:4.4.11]
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:180) ~[neo4j-4.4.11.jar:4.4.11]
at com.neo4j.causalclustering.core.CoreGraphDatabase.createManagementService(CoreGraphDatabase.java:38) ~[neo4j-causal-clustering-4.4.11.jar:4.4.11]
at com.neo4j.causalclustering.core.CoreGraphDatabase.<init>(CoreGraphDatabase.java:30) ~[neo4j-causal-clustering-4.4.11.jar:4.4.11]
at com.neo4j.server.enterprise.EnterpriseManagementServiceFactory.createManagementService(EnterpriseManagementServiceFactory.java:34) ~[neo4j-enterprise-4.4.11.jar:4.4.11]
at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:20) ~[neo4j-enterprise-4.4.11.jar:4.4.11]
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:142) [neo4j-4.4.11.jar:4.4.11]
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:95) [neo4j-4.4.11.jar:4.4.11]
at com.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:24) [neo4j-enterprise-4.4.11.jar:4.4.11]
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'com.neo4j.dbms.ClusteredDbmsReconcilerModule#5c2ae7d7' was successfully initialized, but failed to start. Please see the attached cause exception "Failed to join or bootstrap a raft group with id RaftGroupId{00000000} and members RaftMembersSnapshot{raftGroupId=Not yet published, raftMembersSnapshot={ServerId{c72f54d8}=Published as : RaftMemberId{c72f54d8}}} in time. Please restart the cluster. Clue: not enough cores found".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463) ~[neo4j-common-4.4.11.jar:4.4.11]
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) ~[neo4j-common-4.4.11.jar:4.4.11]
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:218) ~[neo4j-4.4.11.jar:4.4.11]
... 8 more
I tried different settings for the following two config parameters:
causal_clustering.discovery_type
causal_clustering.initial_discovery_members
First the combination of using default discovery_type=K8S which omits any set initial_discovery_members.
Second the combination of discovery_type=LIST and defining initial_discovery_members by name and port 5000.
Both settings let to an successful clustering in like 1/10 times.
As the cluster members are searching for each other while getting deployed, another thing being tried is configuring terraform conditions that two of the cluster members get build with "wait=false" and the third member gets a depends on:
module "neo4j-cluster-core-3" { depends_on = [module.neo4j-cluster-core-1, module.neo4j-cluster-core-2]
I'm using the Terraform provider for IBM Cloud to create a LogDNA instance. I'd like to mark this instance as the destination for Platform Logs.
Here is my Terraform:
resource ibm_resource_instance logdna_us_south {
name = "logging-us-south"
location = "us-south"
service = "logdna"
plan = "7-day"
resource_group_id = ibm_resource_group.dev.id
}
Is it possible?
You need to set the default_receiver parameter when creating the instance as described in https://cloud.ibm.com/docs/Log-Analysis-with-LogDNA?topic=Log-Analysis-with-LogDNA-config_svc_logs#platform_logs_enabling_cli
Your terraform should look like:
resource ibm_resource_instance logdna_us_south {
name = "logging-us-south"
location = "us-south"
service = "logdna"
plan = "7-day"
resource_group_id = ibm_resource_group.dev.id
parameters = {
"default_receiver" = true
}
}
I am trying to build a galera cluster using terraform. To do that I need to render the galera config with the nodes ip, so I use a file template.
When applying, terraform fires an error
Error: Cycle: data.template_file.galera_node_config, hcloud_server.galera_node
It seems there is a circular reference when applying because the servers are not being created before the data template is used.
How may I circumvent this ?
Thanks
galera_node.tf
data "template_file" "galera_node_config" {
template = file("sys/etc/mysql/mariadb.conf/galera.cnf")
vars = {
galera_node0 = hcloud_server.galera_node[0].ipv4_address
galera_node1 = hcloud_server.galera_node[1].ipv4_address
galera_node2 = hcloud_server.galera_node[2].ipv4_address
curnode_ip = hcloud_server.galera_node[count.index].ipv4_address
curnode = hcloud_server.galera_node[count.index].id
}
}
resource "hcloud_server" "galera_node" {
count = var.galera_nodes
name = "galera-${count.index}"
image = var.os_type
server_type = var.server_type
location = var.location
ssh_keys = [hcloud_ssh_key.default.id]
labels = {
type = "cluster"
}
user_data = file("galera_cluster.sh")
provisioner "file" {
content = data.template_file.galera_node_config.rendered
destination = "/tmp/galera_cnf"
connection {
type = "ssh"
user = "root"
host = self.ipv4_address
private_key = file("~/.ssh/id_rsa")
}
}
}
The problem here is that you have multiple nodes that all depend on each other, and so there is no valid order for Terraform to create them: they must all be created before any other one can be created.
To address this will require a different approach. There are a few different options for this, but the one that seems closest to what you were already trying is to use the special resource type null_resource to factor out the provisioning into a separate resource that Terraform can work on only after all of the hcloud_server instances are ready.
Note also that the template_file data source is deprecated in favor of the templatefile function, so this is a good opportunity to simplify the configuration by using the function instead.
Both of those changes together lead to this:
resource "hcloud_server" "galera_node" {
count = var.galera_nodes
name = "galera-${count.index}"
image = var.os_type
server_type = var.server_type
location = var.location
ssh_keys = [hcloud_ssh_key.default.id]
labels = {
type = "cluster"
}
user_data = file("galera_cluster.sh")
}
resource "null_resource" "galera_config" {
count = length(hcloud_server.galera_node)
triggers = {
config_file = templatefile("${path.module}/sys/etc/mysql/mariadb.conf/galera.cnf", {
all_addresses = hcloud_server.galera_node[*].ipv4_address
this_address = hcloud_server.galera_node[count.index].ipv4_address
this_id = hcloud_server.galera_node[count.index].id
})
}
provisioner "file" {
content = self.triggers.config_file
destination = "/tmp/galera_cnf"
connection {
type = "ssh"
user = "root"
host = hcloud_server.galera_node[count.index].ipv4_address
private_key = file("~/.ssh/id_rsa")
}
}
}
The triggers argument above serves to tell Terraform that it must re-run the provisioner each time the configuration file changes in any way, which could for example be because you've added a new node: all of the existing nodes would then be reprovisioned to include that additional node in their configurations.
Provisioners are considered a last resort in the Terraform documentation, but in this particular case the alternatives would likely be considerably more complicated. A typical non-provisioner answer to this would be to use a service discovery system where each node can register itself on startup and then discover the other nodes, for example with HashiCorp Consul's service catalog. But unless you have lots of similar use-cases in your infrastructure which could all share the Consul cluster, having to run another service is likely an unreasonable cost in comparison to just using a provisioner.
You really try to use data.template_file.galera_node_config inside of your resource "hcloud_server" "galera_node" and use hcloud_server.galera_node in your data.template_file.
To avoid this problem:
Remove provisioner "file" from your hcloud_server.galera_node
Move this provisioner "file" to a new null_resource e.g. like that:
resource "null_resource" template_upload {
count = var.galera_nodes
provisioner "file" {
content = data.template_file.galera_node_config.rendered
destination = "/tmp/galera_cnf"
connection {
type = "ssh"
user = "root"
host = hcloud_server.galera_nodes[count.index].ipv4_address
private_key = file("~/.ssh/id_rsa")
}
depends_on = [hcloud_server.galera_node]
}
I have a Containerized Network Function (CNF) that connects to three Docker Networks:
...
ip_address = "172.17.0.3"
ip_prefix_length = 16
ipc_mode = "private"
log_driver = "json-file"
log_opts = {}
logs = false
max_retry_count = 0
memory = 4096
memory_swap = -1
must_run = true
name = "c-router-52"
network_data = [
{
gateway = "172.17.0.1"
ip_address = "172.17.0.3"
ip_prefix_length = 16
network_name = "bridge"
},
{
gateway = "172.31.0.1"
ip_address = "172.31.0.4"
ip_prefix_length = 16
network_name = "inside-net"
},
{
gateway = "172.30.0.1"
ip_address = "172.30.0.3"
ip_prefix_length = 16
network_name = "outside-net"
},
]
network_mode = "default"
...
And I am trying to grab the 'outside-net' IP address for use as an input for another container. I am specifying like so:
${docker_container.c-router-52.network_data[2].ip_address}
When its the third element, it works fine.... But the problem is that Terraform (or Docker, one of the two) doesn't always put the 'outside-net' as the third network :(
Is there a way to specify the [network_name="outside-net"] rather than an index number?
Since your code example isn't complete I'm having to guess a little here, but it seems like what you want is a mapping from network name to IP address. You can derive such a data structure from your resource configuration using a for expression, which you can assign to a local value for use elsewhere in the configuration:
locals {
container_ip_addresses = {
for net in docker_container.c-router-52.network_data :
net.network_name => net.ip_address
}
}
With the above definition in your module, you can refer to local.container_ip_addresses elsewhere in your module to refer to this mapping, such as local.container_ip_addresses["outside-net"] to access the outside-net address in particular.
With the network_data structure you showed in your configuration, local.container_ip_addresses would have the following value:
{
bridge = "172.17.0.3"
inside-net = "172.31.0.4"
outside-net = "172.30.0.3"
}
If you need to access the other attributes of those network_data objects, rather than just the ip_address, you can generalize this by making the values of the mapping be the full network objects:
locals {
container_networks = {
for net in docker_container.c-router-52.network_data :
net.network_name => net
}
}
...which would then allow you to access all of the attributes via the network name keys:
local.container_networks["outside-net"].ip_address
local.container_networks["outside-net"].gateway
local.container_networks["outside-net"].ip_prefix_length