In the terraform documentation we are provided with the markup as attached below. I cant find any mention of what the purpose of the field "b" is and how it should be used in general.
resource "aws_s3_bucket" "b" {
bucket = "my_tf_test_bucket"
}
resource "aws_s3_bucket_policy" "b" {
bucket = "${aws_s3_bucket.b.id}"
policy =<<POLICY
{
"Version": "2012-10-17",
"Id": "MYBUCKETPOLICY",
"Statement": [
{
"Sid": "IPAllow",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::my_tf_test_bucket/*",
"Condition": {
"IpAddress": {"aws:SourceIp": "8.8.8.8/32"}
}
}
]
}
POLICY
}
"b" is simply the NAME of the resource that you are creating.
Terraform resources are defined in "blocks", and each resource block creates a resource of the given TYPE (first parameter) and NAME (second parameter). The combination of the type and name must be unique.
So, in your example, you are creating a resource of TYPE aws_s3_bucket and NAME b.
Each resource defined has an id, that you can use to refer this resource in other resources, using a syntax like TYPE.NAME.id, for example ${aws_s3_bucket.b.id}
You can find further information in the doc here: https://www.terraform.io/docs/configuration/resources.html
Related
AWS MSK supports IAM based authentication, but not through resource based policy like S3. So we are trying to come up with Policy creation dynamically and attach it to an existing IAM roles of corresponding applications. Say there are two applications, Application1 with IAM_Role1 is producer and Application2 with IAM_Role2 is consumer. Now, the producer policy looks like
{
"Version": "2012-10-17",
"Statement": [
{
"sid": "mskproducer_policy",
"Effect": "Allow",
"Action": [
"kafka-cluster:Connect",
"kafka-cluster:DescribeTopic",
"kafka-cluster:WriteData"
],
"Resource": "arn:aws:kafka:${var.region}:${data.aws_caller_identity.current.account_id}:topic/${var.cluster_name}/*/${topic-name}"
}
]
}
Similarly consumer policy looks like
{
"Version": "2012-10-17",
"Statement": [
{
"sid": "mskproducer_policy",
"Effect": "Allow",
"Action": [
"kafka-cluster:Connect",
"kafka-cluster:DescribeTopic",
"kafka-cluster:ReadData"
],
"Resource": "arn:aws:kafka:${var.region}:${data.aws_caller_identity.current.account_id}:topic/${var.cluster_name}/*/${topic-name}"
}
]
}
Now planning for a map as input variable which looks like
variable "topic_permissions" {
type = list(any)
default = [{
"app_iamrole" = "iam-role-1"
"producer" = ["topic3"]
"consumer" = ["topic4"]
},
{
"app_iamrole" = "iam-role-2"
"producer" = ["topic5"]
"consumer" = ["topic6"]
}]
}
Option1: By Using Terraform
data "aws_iam_policy_document"
and dynamic "statement" block, I can construct these policies into corresponding producer or consumer policy.json and create these resource policies and output their arn. So in this case I would get two policy arns.
Since these two policies now belong to different IAM roles, how to attach them back into the corresponding IAM role? Meaning how to map which IAM role gets what policy arn dynamically?
"Resource: aws_iam_role_policy_attachment"
expects policy_arn and corresponding role arn
Option 2: As suggested in comments below, another option is to use inline policy.
resource "aws_iam_role_policy".
But again the same problem arises where I'm not sure how to attach the policies formed dynamically; back into the corresponding IAM roles as inline policies.
Option 3: Write a bash or python scripts to form the dynamic policies and return the output as JSON which can then be looped through in
resource "aws_iam_role_policy".
I'm using Terraform
data "external"
After TF apply, the output JSON results in following error; though when running the Python module individually and verifying the contents through jsonlint, says its perfect JSON.
Error: command "python" produced invalid JSON: json: cannot unmarshal
object into Go value of type string
Terraform version being used is 0.13
I have 2 eks cluster as part of our upgrade. I want to handle assume policy such that it has access to both eks cluster. Both the cluster in same AWS account.
i want my policy to look like the below policy. such that the we are not updating any roles, but only the assume policy to handle both clusters.
locals.tf
eks_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::11111111111:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/yyyyyyyyyyyyyyyyyy",
"Federated": "arn:aws:iam::11111111111:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/yyyyyyyyyyyyyyyyyy"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/xxxxxxxxxxxxxx:sub": "system:serviceaccount:%s:%s",
"oidc.eks.us-east-1.amazonaws.com/id/xxxxxxxxxxxxx:sub": "system:serviceaccount:%s:%s"
}
}
}
]
}
EOF
Launcher = "job-Launcher"
Role.tf
resource "aws_iam_role" "launcher" {
name = local.Launcher
assume_role_policy = format(local.eks_policy, "my-namepsace", local.Launcher)
tags = {
terraform = "true"
owner = "stg"
}
}
So i tried like this in locals.tf
count = length(var.federated)
eks_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::11111111111:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/${join(",",${element(var.federated, count.index)})}",
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/${join(",", ${element(var.federated, count.index)})}:sub": "system:serviceaccount:%s:%s"
}
}
}
]
}
But i'm getting an error as count cannot be used within locals.tf,
Can someone pls help me.
Update2:
How do we get something like this
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/xxxxxxxxxxxxx:sub": "system:serviceaccount:ihr-system:ihr-system-external-dns18",
"oidc.eks.us-east-1.amazonaws.com/id/yyyyyyyyyyyyy:sub": "system:serviceaccount:ihr-system:ihr-system-external-dns"
}
}
I tried this ,
federated = [
"xxxxxxxxxxxxxxxxxxxxxx",
"yyyyyyyyyyyyyyyyyyyyyy"
]
Condition : {
"StringEquals" : {
join("",[for oidc in local.federated:"oidc.eks.us-east-1.amazonaws.com/id/${oidc}:sub:","system:serviceaccount:%s:%s"])
}
getting syntax error near in , local expected and another error got
',' or '}' expected got '"system:serviceaccount.."'
for oidc in local.federated
Terraform format function expects an argument per each placeholder. From the documentation:
The specification is a string that includes formatting verbs that are introduced with the % character. The function call must then have one additional argument for each verb sequence in the specification. The verbs are matched with consecutive arguments and formatted as directed, as long as each given argument is convertible to the type required by the format verb.
With that said, you need to provide four arguments, even though it's the same local variables in your case:
format(local.eks_policy, "my-namepsace", local.Launcher, "my-namepsace", local.Launcher)
Depending on your use case, you might also consider defining a list of objects with configuration and build the policy statement using loop in order to prepare final string.
Update 1
Example with dynamic generation might look like this, where role could be assumed by any account from the variable local.params:
locals {
# key = account ID, value could be whatever
params = {
"1111" = { foo = "bar" },
"2222" = { x = "y" }
}
assume_role_str = jsonencode({
# skipped beginning for brevity
Effect = "Allow",
Principal = {
Federated: [ for account in keys(local.params): "arn:aws:iam::11111111111:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/:${account}" ]
}
})
}
I'm unable to delete a role policy from my AWS account with Boto3. I get an error:
botocore.errorfactory.NoSuchEntityException: An error occurred (NoSuchEntity) when calling the DeleteRolePolicy operation: The role policy with name potatoman9000Policy cannot be found.
The policy and role are created and deleted within the same script. The policy is detached prior to this particular bit of code occurs. I'm not sure why its finding the policy name.
Here is the creation:
# Create IAM policy and Role
def iam_creation(client_name):
iam_client = boto3.client('iam')
# Policy template
client_onboarding_policy = {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowListingOfUserFolder",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Effect": "Allow",
"Resource": [
f"arn:aws:s3:::{client_name}"
]
},
{
"Sid": "HomeDirObjectAccess",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObjectVersion",
"s3:DeleteObject",
"s3:GetObjectVersion"
],
"Resource": f"arn:aws:s3:::{client_name}/*"
}
]
}
# Role template
role_onboarding_policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"transfer.amazonaws.com",
"s3.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
# Create policy from template
iam_client.create_policy(
PolicyName=f'{client_name}Policy',
PolicyDocument=json.dumps(client_onboarding_policy)
)
# Create Role from template and create trust relationships
iam_client.create_role(
RoleName=f'{client_name}',
AssumeRolePolicyDocument=json.dumps(role_onboarding_policy)
)
# Attach created policy to created role
iam_client.attach_role_policy(
PolicyArn=f'arn:aws:iam::111111111111:policy/{client_name}Policy',
RoleName=f'{client_name}'
)
The creation goes off without any issues. Here is the delete
# Delete IAM policy and role
def iam_delete(client_name):
iam_client = boto3.client('iam')
iam_resource = boto3.resource('iam')
role_policy = iam_resource.RolePolicy(f'{client_name}', f'{client_name}Policy')
role = iam_resource.Role(f'{client_name}')
# Detach policy from role
iam_client.detach_role_policy(
PolicyArn=f'arn:aws:iam::111111111111:policy/{client_name}Policy',
RoleName=f'{client_name}'
)
# Delete policy
role_policy.delete()
# Delete role
role.delete()
I imagine it has something to do with the way I've named the role policy or not named it. I have confirmed that the Role potatoman9000 does exist in IAM as well as the Policy potatoman9000Policy. Any help is greatly appreciated
RolePolicy is for inline policies, not managed policies.
When you call delete, it errors out because you are using managed policies.
From docs about delete:
Deletes the specified inline policy that is embedded in the specified IAM role.
To delete managed policy you should be using delete_policy.
Deletes the specified managed policy.
I came across a terraform example code regarding template_file in the below link
https://blog.james-carr.org/using-templates-in-terraform-17bb8f4a0aac
policies/s3_bucket_readonly.json.tpl
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::${bucket_name}",
"arn:aws:s3:::${bucket_name}/${key_prefix}"
],
"Effect": "Allow"
}
]
}
Terraform script
data "template_file" "cloud-trail-logs-s3-readonly" {
template = "${file("policies/s3_bucket_readonly.json.tpl")}"
vars {
bucket_name = "${aws_s3_bucket.cloudtrail-logs.bucket}"
key_prefix = "AWSLogs/*"
}
}
resource "aws_s3_bucket" "cloudtrail-logs" {
bucket = "cloudtrail-logs"
acl = "private"
lifecycle_rule {
enabled = true
noncurrent_version_expiration {
days = 30
}
}
}
resource "aws_iam_policy" "cloudtrail-logs-readonly" {
name = "prod-cloudtrail-logs-s3-readonly"
path = "/production/"
description = "Readonly access to cloudtrail-logs bucket"
policy = "${data.template_file.cloud-trail-logs-s3-readonly.rendered}"
}
Can someone explains what the ${data.template_file.cloud-trail-logs-s3-readonly.rendered} actually does ?? Does it simply apply the template variable values to the policies/s3_bucket_readonly.json.tpl and add the same as policy ?? If yes, then what does ".rendered" mean or stand for ??
Yes you are right, even if your wording is inaccurate. It's the result of the template policies/s3_bucket_readonly.json.tpl after variables are applied.
Have a look at the docs (attributes reference) https://www.terraform.io/docs/providers/template/d/file.html#attributes-reference
rendered - The final rendered template.
The value of the policy will be the content of the template after terraform renders it.
After the HCL is interpreted (the terraform language is called HCL - Hashicorp Configuration Language), "${bucket_name}" will be equivalent to "${aws_s3_bucket.cloudtrail-logs.bucket}" as passed in the template resource in the vars block, and "${key_prefix}" to AWSLogs/*
I think "${aws_s3_bucket.cloudtrail-logs.bucket}" will refer to the string "cloudtrail-logs" (value of the bucket attribute in the aws_s3_bucket resource)
So the policy value should be:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::cloudtrail-logs",
"arn:aws:s3:::cloudtrail-logs/AWSLogs/*"
],
"Effect": "Allow"
}
]
}
I have a terraform script that provides a lambda function on aws to send emails. I pieced this terraform script from tutorials and templates on the web to use AWS SES, Api Gateway, Lambda and Cloudwatch services.
To get permissions to work though, I had to run the script and then, separately, build a policy in the AWS console and apply it to the lambda function so that it could fully access the SES and Cloudwatch services. But it's not at all not clear to me how to take that working policy and adapt it to my terraform script. Could anyone please provide or point to guidance on this matter?
The limited/inadequate but otherwise working role in my terraform script looks like this:
resource "aws_iam_role" "iam_for_lambda" {
name = "${var.role_name}"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Effect": "Allow",
"Sid": ""
}
]
} EOF
}
... and the working policy generated in the console (by combining two roles together for all-Cloudwatch and all-SES access):
{
"permissionsBoundary": {},
"roleName": "las_role_new",
"policies": [
{
"document": {
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"autoscaling:Describe*",
"cloudwatch:*",
"logs:*",
"sns:*",
"iam:GetPolicy",
"iam:GetPolicyVersion",
"iam:GetRole"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/events.amazonaws.com/AWSServiceRoleForCloudWatchEvents*",
"Condition": {
"StringLike": {
"iam:AWSServiceName": "events.amazonaws.com"
}
}
}
]
},
"name": "CloudWatchFullAccess",
"id": "ANPAIKEABORKUXN6DEAZU",
"type": "managed",
"arn": "arn:aws:iam::aws:policy/CloudWatchFullAccess"
},
{
"document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ses:*"
],
"Resource": "*"
}
]
},
"name": "AmazonSESFullAccess",
"id": "ANPAJ2P4NXCHAT7NDPNR4",
"type": "managed",
"arn": "arn:aws:iam::aws:policy/AmazonSESFullAccess"
}
],
"trustedEntities": [
"lambda.amazonaws.com"
]
}
There are fields
So my question in summary, and put most generally, is this:
given a "policy" built in the aws console (by selecting a bunch of roles, etc. as in ), how do you convert that to a "role" as required for the terraform script?
To anyone else who might struggle to understand terraform-aws-policy matters, here's my understanding after some grappling. The game here is to carefully distinguish the various similar-sounding terraform structures (aws_iam_role, aws_iam_role_policy, aws_iam_role, assume_role_policy, etc.) and to work out how these black-box structures fit together.
First, the point of an aws role is to collect together policies (i.e. permissions to do stuff). By assigning such a role to a service (e.g. lambda), you thereby give that service the permissions described by those policies. A role must have at least one policy sort of built-in to it: the 'assume-role' policy that specifies which service(s) can use ('assume') that role. This assume-role policy is relatively simple and so 'might as well' be included in the terraform script explicitly (using the <<EOF ... EOF syntax above).
Secondly, if you want to now let that service with the (basic) role do anything to other services, then you need to somehow associate additional policies with that role. I've learned that there are several ways to do this but, in order to answer my question most succinctly, I'll now describe the most elegant way I have found to incorporate multiple template policies offered in the AWS console into one's terraform script.
The code is:
# Define variable for name of lambda function
variable "role_name" {
description = "Name for the Lambda role."
default = "las-role"
}
# Create role with basic policy enabling lambda service to use it
resource "aws_iam_role" "iam_for_lambda" {
name = "${var.role_name}"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": [ "lambda.amazonaws.com" ]
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
# Define a list of policy arn's given in the AWS console
variable "iam_policy_arn_list" {
type = list(string)
description = "IAM Policies to be attached to role"
default = ["arn:aws:iam::aws:policy/CloudWatchFullAccess", "arn:aws:iam::aws:policy/AmazonSESFullAccess"]
}
# Create attachment of the policies for the above arn's to our named role
# The count syntax has the effect of looping over the items in the list
resource "aws_iam_role_policy_attachment" "role-policy-attachment" {
role = var.role_name
count = length(var.iam_policy_arn_list)
policy_arn = var.iam_policy_arn_list[count.index]
depends_on = [aws_iam_role.iam_for_lambda]
}
As you can see, the template policies are included here using the arns which can be found in the AWS console. For example, here's the view for finding the arn for full access to Amazon SES via the AWS Management Console:
When you succesfully deploy your lambda to AWS using terraform, it will pull down these policies from the arns and generate a permission json for your lambda function (which you can view in the lambda-service section of the aws console) that looks a lot like the json I posted in the question.