It seems that specifying a JDBC_DRIVER_JAR_URI connection property when defining an aws glue connection in terraform does nothing. When I test the glue connection, the cloudwatch logs show that glue is still using version 9.4 JDBC driver or postgres
resource "aws_glue_connection" "glue_connection_2" {
connection_properties = {
JDBC_DRIVER_JAR_URI = "s3://scripts/postgresql.jar"
JDBC_CONNECTION_URL = var.jdbc_connection_url
JDBC_ENGINE_VERSION = "14"
PASSWORD = var.glue_db_password
USERNAME = var.glue_db_user_name
}
name = "${local.glue_connection_name}-custom"
connection_type = "JDBC"
physical_connection_requirements {
availability_zone = var.database_availability_zone
security_group_id_list = var.security_group_id_list
subnet_id = sort(data.aws_subnets.vpc_subnets.ids)[0]
}
}
Is it possible to specify a custom jar for aws glue connections other than creating a custom connector for it?
Related
I tried to connect to Cosmos DB via Databricks. I use the connector from Maven (com.azure.cosmos.spark:azure-cosmos-spark_3-1_2-12:4.14.0). Here is the setup:
cosmosEndpoint = "https://myendpoint.documents.azure.com:443/"
cosmosMasterKey = dbutils.secrets.get(scope = "mykv", key = "my_key")
cosmosDatabaseName = "mydb"
cfg_oro = {
"spark.cosmos.accountEndpoint" : cosmosEndpoint,
"spark.cosmos.accountKey" : cosmosMasterKey,
"spark.cosmos.database" : cosmosDatabaseName,
"spark.cosmos.container" : "mycontainer",
}
spark.conf.set("spark.sql.catalog.cosmosCatalog", "com.azure.cosmos.spark.CosmosCatalog")
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountEndpoint", cosmosEndpoint)
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountKey", cosmosMasterKey)
When I run the statement:
it keeps running and shows no result. Where is the problem coming from?
I am using below terraform datasource for importing shared state from s3. Terraform is giving me error " No stored state was found for the given workspace in the given backend". I am expecting terraform to pick up the workspace "dev-use1" as I have set the workspace using terraform workspace select "dev-use1".
data "terraform_remote_state" "shared_jobs_state" {
backend = "s3"
config = {
bucket = "cicd-backend"
key = "analyticsjobs.tfstate"
workspace_key_prefix = "pipeline/v2/db"
region = "us-east-1"
}
}
Version = Terraform v1.1.9 on darwin_arm64
After enabling the DEBUG in terraform by setting TF_LOG="DEBUG". I can see that s3 api call is giving 404 error.
from the request xml I can see that the prefix is wrong.
As a workaround I have done below changes to datasource.
Not sure this is the recommended way of doing but it works. There is less clarity in docs regards to this https://www.terraform.io/language/state/remote-state-data
data "terraform_remote_state" "shared_jobs_state" {
backend = "s3"
config = {
bucket = "cicd-backend"
key = "pipeline/v2/db/${terraform.workspace}/analyticsjobs.tfstate"
region = "us-east-1"
}
}
We're trying to deploy a Cloud SQL (MSSQL) instance using the google-beta provider with a private IP and after roughly four to five minutes it fails and throws the error "Error waiting for Create Instance: Per-Product Per-Project Service Account is not found"
I am able to create a Cloud SQL instance using the service account via the Cloud Shell CLI and manually in Console.
Has anyone encountered this before and can they provide any insights as to what may be going wrong?
If you look at the errored out resource in console, it appears to have mostly created but this error is shown.
resource "google_sql_database_instance" "cloud_sql_instance" {
provider = google-beta
name = var.cloud_sql_instance_name
region = var.gcp_region
database_version = var.cloud_sql_version
root_password = "wearenothardcodingplaceholdertest"
deletion_protection = var.delete_protection_enabled
project = var.gcp_project
settings {
tier = var.cloud_sql_compute_tier
availability_type = var.cloud_sql_availibility_type
collation = var.cloud_sql_collation
disk_autoresize = var.cloud_sql_auto_disk_resize
disk_type = var.cloud_sql_disk_type
active_directory_config {
domain = var.active_directory_domain
}
backup_configuration {
enabled = var.cloud_sql_backup_enabled
start_time = var.cloud_sql_backup_starttime
point_in_time_recovery_enabled = var.cloud_sql_pitr_enabled
transaction_log_retention_days = var.cloud_sql_log_retention_days
backup_retention_settings {
retained_backups = var.cloud_sql_backup_retention_number
retention_unit = var.cloud_sql_backup_retention_unit
}
}
ip_configuration {
ipv4_enabled = var.cloud_sql_backup_public_ip
private_network = data.google_compute_network.vpc_connection.self_link
require_ssl = var.cloud_sql_backup_require_ssl
allocated_ip_range = var.cloud_sql_ip_range_name
}
maintenance_window {
day = var.cloud_sql_patch_day
hour = var.cloud_sql_patch_hour
update_track = "stable"
}
}
}
I just ran into this issue. You need to create a Service Identity for sqladmin.googleapis.com.
resource "google_project_service_identity" "cloudsql_sa" {
provider = google-beta
project = "cool-project"
service = "sqladmin.googleapis.com"
}
I am using philips-labs/terraform-aws-github-runner.
module "runner" {
source = "philips-labs/github-runner/aws"
version = "0.39.0"
aws_region = var.region
enable_organization_runners = true
environment = var.environment
ghes_url = "<github enterprise server>"
github_app = {
id = var.github_app.id
key_base64 = var.github_app.key_base64
webhook_secret = var.github_app.webhook_secret
}
lambda_security_group_ids = var.lambda_security_group_ids
lambda_subnet_ids = var.lambda_subnet_ids
runner_binaries_syncer_lambda_zip = "${path.module}/resources/runner-binaries-syncer.zip"
runners_lambda_zip = "${path.module}/resources/runners.zip"
subnet_ids = var.subnet_ids
webhook_lambda_zip = "${path.module}/resources/webhook.zip"
vpc_id = var.vpc_id
}
GitHub Enterprise Server: v3.2.8
Workflow event type: check_run
first, there is no registered runner
trigger github actions workflow
immediately, the workflow fails with the error: No runner matching the specified labels was found: self-hosted
then, scale-up lambda says Job not queued and no runners launch
Currently, I have to set enable_job_queued_check=false and the first workflow when there is no runners will fail.
I expect that the workflows wait for the runner become prepared.
Do you have any idea about that?
I am trying to connect sql database to function app on azure.
I tried using "storage_connection_string" key in terraform.It is still not working.
Could someone please help on the issue
I have a Function App deployed into Azure that's also using Azure SQL as well as a storage container. This is how it works for me. My terraform configuration is module-based so my modules for the database and storage accounts are separate, and they pass the required connection strings to my function app module:
resource "azurerm_function_app" "functions" {
name = "fcn-${var.environment}
resource_group_name = "${var.resource_group}"
location = "${var.resource_location}"
app_service_plan_id = "${var.appservice_id}"
storage_connection_string = "${var.storage_prim_conn_string}"
https_only = true
connection_string {
name = "SqlAzureDbConnectionString"
type = "SQLAzure"
value = "${var.fcn_connection_string}"
}
tags {
environment = "${var.environment}"
}
Just remember to check you have the module outputs as well as the variables in place.
Hope that helps.