I have below terraform script, to create a new service account and make it owner.
The scripts creates the service account, but it will throw an error on assigning role
resource "google_service_account" "pci_api_service_account" {
account_id = "pci-api"
display_name = "Api"
project = var.project_id
}
resource "google_service_account_iam_member" "pci_api_owner_binding" {
# service_account_id = "projects/pcb-poc-pci/serviceAccounts/infra-admin-sa#pcb-poc-pci.iam.gserviceaccount.com"
service_account_id = google_service_account.pci_api_service_account.name
role = "roles/owner"
member = "serviceAccount:${google_service_account.pci_api_service_account.email}"
depends_on = [
google_service_account.pci_api_service_account
]
}
and I already autheticated with infra-admin-sa service account by running
gcloud auth activate-service-account --project=pcb-poc-pci --key-file ~/sa/pcb-poc-pci-test-sa-94aa6c81d650.json
When I run terragrunt apply I get this error for the second script
Error: Error applying IAM policy for service account 'projects/pcb-poc-pci/serviceAccounts/pci-api#pcb-poc-pci.iam.gserviceaccount.com': Error setting IAM policy for service account 'projects/pcb-poc-pci/serviceAccounts/pci-api#pcb-poc-pci.iam.gserviceaccount.com': googleapi: Error 403: Permission iam.serviceAccounts.setIamPolicy is required to perform this operation on service account projects/pcb-poc-pci/serviceAccounts/pci-api#pcb-poc-pci.iam.gserviceaccount.com., forbidden
These are the Roles of that service account
Based on google doc here and the error message, Service Account Admin should be enough, which my service account already have
Not sure what I missed
Solution 1
seems command line was not picking the correct credential/service account although I used gcloud auth activate-service-account command.
so I added this to my script
provider "google" {
credentials = file(var.service_account_file_path)
project = var.project_id
}
and now it's working fine
Solution 2
as per #John Hansley comments below
export GOOGLE_APPLICATION_CREDENTIALS=fullpath.json
then terraform will be picking that service account file and scripts will run successfully.
This method is preferred since less issue in CICD pipeline and other deveopers, to set terraform variables
Related
I'm hoping to contribute some documentation on auth_login_aws because I'm trying to use the feature described in this feature request.
TL;DR, despite specifying aws_role_arn in the snippet below, the provider is still trying to use the credentials in my ~.aws/credentials file.
provider "vault" {
address = "http://127.0.0.1:8200"
skip_child_token = true
auth_login_aws {
role = "dev-role-corey-iam"
header_value = "vault.example.com"
aws_role_arn = "arn:aws:iam::1234567890:role/SomeRole"
aws_role_session_name = "corey-test-session"
aws_region = "us-east-1"
}
}
I'm just trying to get Terraform to pull secrets from a local Vault instance using the SomeRole IAM role, which is bound to the dev-role-corey-iam Auth role. I'm currently logged in with a different AWS IAM role (PrimaryRole) that can assume the SomeRole role. When I run terraform plan, the provider is using PrimaryRole instead when I expect it to use SomeRole because aws_role_arn is specified. Am I misinterpreting what aws_role_arn should do?
Whenever I run terraform plan command I have to put client id, secret id, tenant id and subscription id manually.
Here is my setup
I have set env variables in my .zprofile
export ARM_SUBSCRIPTION_ID=*********
export ARM_CLIENT_ID=*********
export ARM_CLIENT_SECRET=*********
export ARM_TENANT_ID=*********
if I want to see them echo $ARM_SUBSCRIPTION_ID it does print out its value
In my providers.tf file this is the code
# Configuration options
provider "azurerm" {
subscription_id = var.SUBSCRIPTION_ID
client_id = var.CLIENT_ID
client_secret = var.CLIENT_SECRET
tenant_id = var.TENANT_ID
features {
}
}
And in my variables.tf file the code looks like this
variable "SUBSCRIPTION_ID" {
}
variable "CLIENT_ID" {
}
variable "CLIENT_SECRET" {
}
variable "TENANT_ID" {
}
so when I run terraform plan commands the terminal prompts me to put all ids manually.
Authenticating to azure by service principal and client secret using terraform:
I tried to authenticate with AzureAD service principal in my environment after finding a workaround and was able to perform it successfully.
I've setup env variables in azCLI as shown here:
export ARM_SUBSCRIPTION_ID="<subscription_id>"
export ARM_CLIENT_ID="<client_id>"
export ARM_TENANT_ID="<tenant_id>"
export ARM_CLIENT_SECRET="<client_secret>"
To authenticate service principal, we need to give some role permissions and API Permissions as well.
Required Permissions to enable:
Goto Roles and Administrators under Azure Active Directory through portal. There are certain administrator positions that come with default configuration. Actively enable the roles of "Application Administrator, Global Administrator, and User Administrator".
and
Goto API permissions under registered App registration and add application permissions for microsoft graph as its the main source to authenticate to Portal and enable below given permissions by clicking on Add permission.
App registration:
To check, I created a resource group and applying a dev tag for it after successful authentication.
Provider.tf
variable "client_secret" {}
terraform {
required_providers {
azuread = {
source = "hashicorp/azuread"
version = "2.30.0" //Give the latest version
}
}
}
provider "azurerm"{
features{}
}
provider "azuread" {
subscription_id = "<subscription_id>"
client_id = "<client_id>"
client_secret = var.client_secret
tenant_id = "<tenant_id>"
features{}
}
resource "azurerm_resource_group" "t_rg" {
name = "<resourcegroupName>"
location = "<location>"
tags = {
Environment = "Dev"
}
}
Executed terraform init:
After setting up everything, I executed terraform plan, it is asking for client_secret only not client_id:
Executed Terraform apply:
Authenticated & deployed successfully in Portal:
Applied tags successfully for resourcegroup as given:
So finally I have found the solution.
If you have multiple subscriptions of Azure (you are working for multiple clients) then the first thing is that you need to check which is your current subscription.
Run this command to check all subscriptions
az account tenant list
and run this command to change switch your subscription
az login --tenant <myTenantID>
and now you can run any Azure cli command and changes will be reflected on your current subscription
In the terraform documentation for google_pubsub_subscription, it mentions having a oidc_token property under push_configuration allows push-subscriptions to utilise service account, but has no mention on how to use service account for pull subscriptions.
How can explicitly set the service account to be used during the creation of pull pubsub subscriptions?
I tried adding oidc_token block, but it did not worked as it does not expect that block directly.
Scenario:
I have a service account that has access to pubsub topics (and necessary permissions to attach subscriptions to it) in Project_A
I want to create subscriptions to those topics in Project_B in terraform.
I need to explicitly use this service account in terraform, so i can create subscriptions at Project_B to topics of Project_A
google_cloudfunctions_function resource for example, has field called service_account_email for setting service account. But there is no for google_pubsub_subscription resource, for pull subscriptions in the documentation.
Actually the service account (service_account_email)is to be specified inside the oidc_token section (you had to go further in the doc :)
below a working example of a push sub with service account and audience (optional)
variable "project" {
type = string
default = "<YOUR_PROJECT_ID>"
}
resource "google_pubsub_topic" "example" {
project = var.project
name = "example-topic"
}
resource "google_pubsub_subscription" "example" {
project = var.project
name = "example-push-subscription-with-auth"
topic = google_pubsub_topic.example.name
push_config {
push_endpoint = "https://example.com/push"
oidc_token {
service_account_email = "${var.project}#appspot.gserviceaccount.com"
audience = "https://example.com/push"
}
}
}
I'm adding another answer, because the question has changes a lot after the different comments.
Here is the solution with the hypothesis that a topic topic-sof is already created in a different project than the subscription one.
I created a service account (SA) on the subscription project, I called it stackoverflow-playground-sa and gave it only the Pub/Sub Editor role as you can see in the screenshot below.
I gave the SA the the Pub/Sub Subscriber role on the topic topic-sof s show in the screenshot below.
If you don't do this step: you will get this error at terraform apply
Error: Error creating Subscription: googleapi: Error 403: User not authorized to perform this action
Of course I did the 2 first steps of roles assignments with a user with enough permissions on both projects
I created a json key file for my SA and downloaded under /path/to/my/key/stackoverflow-playground-sa.json
I authenticated as my SA using
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/my/key/stackoverflow-playground-sa.json
So it can be used by terraform to create the subscription.
Here is the terraform configuration to create the subscription
variable "subscription_project" {
type = string
default = "<YOUR_SUBSCRIPTION_PROJECT>"
}
variable "topic_project" {
type = string
default = "<YOUR_TOPIC_PROJECT>"
}
resource "google_pubsub_subscription" "pull-subscription-of-topic-in-another-project" {
project = var.subscription_project
name = "pull-subscription-of-topic-in-another-project"
topic = "projects/${var.topic_project}/topics/topic-sof"
}
Run (apply) my terraform and the subscription pull-subscription-of-topic-in-another-project is created and attached to the topic topic-sof in the other project.
I published a message to topic-sof using the web ui.
Still authenticated as stackoverflow-playground-sa thanks to step 4, I pull the message (using gcloud in my treminal) et voilà message received:
To summarise : there is no service account to specify in your terraform configuration for this requirement. The service account and its key is set outside terraform (step 1 to 4) so that terraform process can be authenticated and authorised to create the resources configured (the subscription in our case).
Besides using a service account key as in step 4 is a bad practice, security wise. An alternative is to use
gcloud auth application-default login
That will let you set the default credentials of the user of your choice. Given that user has the roles I set for the SA.
What specific changes need to be made to the syntax below in order for the terraform azurerm provider to be able to authenticate the service principal that will be created using the following code?
The Problem
A Second Terraform module needs to authenticate to Azure through the azurerm provider with a client_id and client_secret that is created programatically during an earlier, separate process.
The provider block in the Second Terraform module looks like:
provider "azurerm" {
subscription_id = var.subscriptionId
client_id = var.clientId
client_secret = var.clientSecret
tenant_id = var.tenantId
}
The problem arises when the correct values whcih we validated from the earlier preceding process are not accepted as the var.clientId and the var.clientSecret in the provider code block above.
How the Service Principal is Created:
The client_id and client_secret to be used to authenticate to the Second Terraform module are currently created by a First Terraform module which includes the following:
resource "azuread_application" "appReg" {
name = var.appName
}
resource "azuread_service_principal" "example-sp" {
application_id = azuread_application.appReg.application_id
}
resource "azuread_service_principal_password" "example-sp_pwd" {
service_principal_id = azuread_service_principal.example-sp.id
value = "long-random-string"
end_date = "2021-06-02T01:02:03Z"
}
data "azurerm_subscription" "thisSubscription" {
subscription_id = var.subscriptionId
}
resource "azurerm_role_assignment" "example-sp_role_assignment" {
scope = data.azurerm_subscription.thisSubscription.id
role_definition_name = "Contributor"
principal_id = azuread_service_principal.example-sp.id
}
resource "azuread_application_app_role" "example-role" {
application_object_id = azuread_application.appReg.id
allowed_member_types = ["User", "Application"]
description = "Admins can manage roles and perform all task actions"
display_name = "Admin"
is_enabled = true
value = "administer"
}
Terraform reports Apply complete after the above First module is run, and we are also able to confirm in the Azure Portal that the correct Active Directory has a new app registration with name var.appName and with ID equal to what we find in the First modules tfstate file.
The Error Message:
When Terraform tries to apply the Second module using the Service Principal ID and Secret created by the First module, the following error is thrown:
Error:
Error building account:
Error getting authenticated object ID:
Error listing Service Principals:
autorest.DetailedError{
Original:adal.tokenRefreshError{
message:"adal: Refresh request failed.
Status Code = '400'.
Response body: {
\"error\":\"unauthorized_client\",
\"error_description\":\"AADSTS700016:
Application with identifier 'correct-app-id' was not found in the directory 'the-right-ad-id'.
This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant.
You may have sent your authentication request to the wrong tenant.\\r\\n
Trace ID: some-trace-id\\r\\n
Correlation ID: correlation-id-redacted\\r\\n
Timestamp: 2020-12-31 19:02:19Z\",
\"error_codes\":[700016],
\"timestamp\":\"2020-12-31 19:02:19Z\",
\"trace_id\":\"some-trace-id\",
\"correlation_id\":\"correlation-id-redacted\",
\"error_uri\":\"https://login.microsoftonline.com/error?code=700016\"
}",
resp:(*http.Response)(0xc000ac2000)},
PackageType:"azure.BearerAuthorizer",
Method:"WithAuthorization",
StatusCode:400,
Message:"Failed to refresh the Token for request to https://graph.windows.net/the-right-ad-id/servicePrincipals?%24filter=appId+eq+%27correct-app-id%27&api-version=1.6",
ServiceError:[]uint8(nil),
Response:(*http.Response)(0xc000ac2000)
}
The error message does not seem helpful because we validated that the app is registered with the AAD instance.
How can we resolve this problem and programmatically create a client_id and client_secret that will be accepted and usable by the Second module?
As I see there is no problem with your Terraform code. It should work fine. But you got the error that the application was not found in the tenant. So what you need to do is to check if the tenant Id is really right in the second module.
I had the same issue on Deployment Agents for Terraform on Kubernetes. Several types of error can appear when the memory or the CPU is not large enough.
Below are the Terraform recommendations: https://www.terraform.io/docs/enterprise/before-installing/index.html
You have to be careful with the deployment infrastructures that mutualises the resources (K8s, Hypervisor Pool etc.)when several Terraform deployments are in parallel it causes somewhat random errors.
Terraform which does not stop, API AZure / AWS error, tfstate lock etc.
I'm writing a cloud function in Nodejs (10), and trying to access a secret like so:
const [secret] = await new SecretManagerServiceClient().accessSecretVersion({
name: `projects/PROJECT_NUMBER/secrets/SECRET_NAME/versions/latest`
})
I created the secret in the web console and the name used in code matches that of the existing secret. On the page for the cloud function details, it states that the service account is PROJECT_ID#appspot.gserviceaccount,com, so I added the secretmanager.secretAccessor role to it. However, I'm still getting the same error every time:
Error: 7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/PROJECT_NUMBER/secrets/SECRET_NAME/versions/latest' (or it may not exist).
It makes no difference if I specify a concrete version or just use latest.
HTTP cloud function code:
const { SecretManagerServiceClient } = require('#google-cloud/secret-manager');
const secretManagerServiceClient = new SecretManagerServiceClient();
const name = 'projects/shadowsocks-218808/secrets/workflow/versions/latest';
exports.testSecretManager = async (req, res) => {
const [version] = await secretManagerServiceClient.accessSecretVersion({ name });
const payload = version.payload.data.toString();
console.debug(`Payload: ${payload}`);
res.sendStatus(200);
};
Deploy:
gcloud functions deploy testSecretManager --runtime nodejs10 --trigger-http --allow-unauthenticated
Deploying function (may take a while - up to 2 minutes)...done.
availableMemoryMb: 256
entryPoint: testSecretManager
httpsTrigger:
url: https://us-central1-shadowsocks-218808.cloudfunctions.net/testSecretManager
ingressSettings: ALLOW_ALL
labels:
deployment-tool: cli-gcloud
name: projects/shadowsocks-218808/locations/us-central1/functions/testSecretManager
runtime: nodejs10
serviceAccountEmail: shadowsocks-218808#appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/gcf-upload-us-central1-43476143-b555-4cb2-8f6f-1b2d1952a2d7/42c4cda4-98a8-4994-a3be-d2203b9e646a.zip?GoogleAccessId=service-16536262744#gcf-admin-robot.iam.gserviceaccount.com&Expires=1596513795&Signature=kbLw5teN8EoYmj4fEweKKiIaakxcrhlUg2GGHV4jWJjvmeEfXePpRNOn9yz2zLn%2Fba0UqM9qdJMXujs5afBk%2BVBmywPEiptAZe2qgmldpr%2BsYejFu0woNgsPHVqtJ0NoWDo6W2dq4CuNNwO%2BaQ89mnhahUUQTInkJ55Y3wCIe9smk%2BqWtcvta3zICiToA7RQvPKY5MS6NViyj5mLxuJtDlTY9IKPL%2BqG6JAaQJSFYKYVgLyb6JfirXk8Q7%2FMvnHPpXPlhvsBLQksbF6jDPeefp2HyW4%2FSIQYprfpwKV3hlEIQyRQllz5J9yF83%2FxDPh%2BQPc5QmswKP5XAvYaszJPEw%3D%3D
status: ACTIVE
timeout: 60s
updateTime: '2020-08-04T03:34:32.665Z'
versionId: '2'
Test:
gcloud functions call testSecretManager --data '{}'
Got error same as you:
error: |-
Error: function terminated. Recommended action: inspect logs for termination reason. Details:
7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/shadowsocks-218808/secrets/workflow/versions/latest' (or it may not exist).
solution:
You can find the serviceAccountEmail: shadowsocks-218808#appspot.gserviceaccount.com from the deployment information details of cloud function.
go to IAM & Admin web UI, click ADD ANOTHER ROLE button, add Secret Manager Secret Accessor role to this service account.
Test again:
> gcloud functions call testSecretManager --data '{}'
executionId: 1tsatxl6fndw
result: OK
Read the logs for testSecretManager cloud function:
gcloud functions logs read testSecretManager
You will see the logs for the secret payload string.
I had the same issue and to solve it, I just had to:
Find the Service Account under General of my Google Cloud Function.
It looked like <project-name>#appspot.gserviceaccount.com
In IAM Admin, Add Secret Manager Secret Accessor Role to this Service Account.
After this, everything worked!
I have had similar issues working with secretmanager and the python google-cloud-secretmanager library (2.4). Specifically, after creating a secret and giving my service account the secretmanager.secretAccessor role on this secret (and nothing else, following the principle of least privilege), I was getting the following error when trying to access it:
details = "Permission 'secretmanager.versions.access' denied for resource 'projects/projectid/secrets/keyname/versions/latest' (or it may not exist)."
I could only make it work by also adding the secretmanager.viewer role at the project level, which as far as I can tell is not described in the documentation.
I had similar problem using terraform under gitlab.
I must add two authorizations to the service account which runs the pipeline:
resource "google_project_iam_policy" "gitlab" {
project = "secret_owner_project_id"
policy_data = data.google_iam_policy.iam.policy_data
}
data "google_iam_policy" "iam" {
binding {
role = "roles/secretmanager.secretAccessor"
members = [
"serviceAccount:project_accessing_secret#XYZ.iam.gserviceaccount.com",
]
}
binding {
role = "roles/viewer"
members = [
"serviceAccount:project_accessing_secret#XYZ.iam.gserviceaccount.com",
]
}
}
A bit late, but maybe this answer could be useful for future users. I encountered the same behavior only with Python. I tried lots of things but only thing that worked was creating new service account with zero roles(if I granted it secretmanager.secretAccessor role immediately, I got the same error). Then when empty service account is created, in IAM tab I press +Add, copy my empty service account adress and ONLY then I add secretmanager.secretAccessor role to it. Then I use this account as the account that will execute particular function. You of course may need to add other roles depending on what your function is intended to accomplish.
OAuth scope plays an important role here and please make sure the scope is defined correctly.
To use Secret Manager with workloads running on Compute Engine or GKE, the underlying instance or node must have the cloud-platform OAuth scope. If you receive an error with the following message, it means the instance or node was not provisioned with the correct OAuth scopes.
Request had insufficient authentication scopes
The required OAuth scope to use Secret Manager is:
https://www.googleapis.com/auth/cloud-platform
Example gcloud command to create dataproc with scope
gcloud dataproc clusters create xyz-pqr --region asia-south1 --subnet projects/xyz-pqr/regions/asia-south1/subnetworks/abc-serverless-vpc --zone asia-south1-b --master-machine-type n1-standard-4 --master-boot-disk-size 100 --num-workers 2 --worker-machine-type n1-standard-4 --worker-boot-disk-size 100 --image-version 2.0-debian10 --project xyz-development -scopes https://www.googleapis.com/auth/cloud-platform