I have built my own Docker container that provides inference code to be deployed as endpoint on Amazon Sagemaker. However, this container needs to have access to some files from s3. The used IAM role has access to all s3 buckets that I am trying to reach.
Code to download files using a boto3 client:
import boto3
model_bucket = 'my-bucket'
def download_file_from_s3(s3_path, local_path):
client = boto3.client('s3')
client.download_file(model_bucket, s3_path, local_path)
The IAM role's policies:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-bucket/*"
]
}
]
}
Starting the docker container locally allows me to download files from s3 just like expected.
Deploying as an endpoint on Sagemaker, however, the request times out:
botocore.vendored.requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='my-bucket.s3.eu-central-1.amazonaws.com', port=443): Max retries exceeded with url: /path/to/my-file (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f66244e69b0>, 'Connection to my-bucket.s3.eu-central-1.amazonaws.com timed out. (connect timeout=60)'))
Any help is appreciated!
for security reasons they don't let it access s3 natively, you need to hook it up to a VPC
https://docs.aws.amazon.com/sagemaker/latest/dg/host-vpc.html
For anyone coming across this question, when creating a model, the 'Enable Network Isolation' property defaults to True.
From AWS docs:
If you enable network isolation, the containers are not able to make any outbound network calls, even to other AWS services such as Amazon S3. Additionally, no AWS credentials are made available to the container runtime environment.
So this property needs to be set to False in order to connect to any other AWS service.
Related
I'm trying to make a bucket of images public read even when uploaded from another AWS account. I have the current bucket policy in place:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AddPerm",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::mybucketname/*"
}
]
}
This works great when I upload using credentials from the primary account, but when I upload from the other account added by ACL it doesn't apply. As I read I found that you can add bucket-owner-full-control or public-read but not both. My end goal is to allow the object to be fully accesses by both AWS accounts AND have public read access on upload. Is this possible (ideally without two requests)?
The above accepted answer is incorrect as S3 bucket policies are ignored on objects published from another account.
The correct way to apply multiple ACLs on a cross account publish is like follows:
aws s3 cp --recursive blahdir s3://bucketname/blahdir/ --cache-control public,max-age=31536000 --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers full=id=S3_ACCOUNT_CANONICAL_ID
No, it seems you can only specify one ACL when creating an object in Amazon S3.
For details of what each Canned ACL means, see: Canned ACL
For details of how ownership works, see: Amazon S3 Bucket and Object Ownership
Personally, I would not recommend using ACLs to control access. They are a hold-over from the early days of Amazon S3. These days, I would recommend using a Bucket Policy if you wish to make a large number of objects public, especially if they are in the same bucket/path.
Thus, an object can be uploaded with bucket-owner-full-control and the Bucket Policy can make them publicly accessible.
I created a private s3 bucket and a fargate cluster with a simple task that attempts to read from that bucket using python 3 and boto3. I've tried this on 2 different docker images and on one I get a ClientError from boto saying HeadObject Bad request (400) and the other I get NoCredentialsError: Unable to locate credentials.
The only real different in the images is that the one saying bad request is being run normally and the other is being run manually by me via ssh to the task container. So I'm not sure why one image is saying "bad request" and the other "unable to locate credentials".
I have tried a couple different IAM policies, including (terraform) the following policies:
data "aws_iam_policy_document" "access_s3" {
statement {
effect = "Allow"
actions = ["s3:ListBucket"]
resources = ["arn:aws:s3:::bucket_name"]
}
statement {
effect = "Allow"
actions = [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetObjectTagging",
"s3:GetObjectVersionTagging",
]
resources = ["arn:aws:s3:::bucket_name/*"]
}
}
Second try:
data "aws_iam_policy_document" "access_s3" {
statement {
effect = "Allow"
actions = ["s3:*"]
resources = ["arn:aws:s3:::*"]
}
}
And the final one I tried was a build in policy:
resource "aws_iam_role_policy_attachment" "access_s3" {
role = "${aws_iam_role.ecstasks.name}"
policy_arn = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
the bucket definition is very simple:
resource "aws_s3_bucket" "bucket" {
bucket = "${var.bucket_name}"
acl = "private"
region = "${var.region}"
}
Code used to access s3 bucket:
try:
s3 = boto3.client('s3')
tags = s3.head_object(Bucket='bucket_name', Key='filename')
print(tags['ResponseMetadata']['HTTPHeaders']['etag'])
except ClientError:
traceback.print_exc()
No matter what I do, I'm unable to use boto3 to access AWS resources from within a Fargate container task. I'm able to access the same s3 bucket with boto3 on an EC2 instance without providing any kind of credentials and only using the IAM roles/policies. What am I doing wrong? Is it not possible to access AWS resources in the same way from a Fargate container?
Forgot to mention that I am assigning the IAM roles to the task definition execution policy and task policy.
UPDATE: It turns out that the unable to find credentials error I was having is a red herring. The reason I could not get the credentials was because my direct ssh session did not have the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable set.
AWS Fargate will inject an environment variable named AWS_CONTAINER_CREDENTIALS_RELATIVE_URI on your behalf which contains a url to what boto should use for grabbing API access credentials. So the Bad request error is the one I'm actually getting and need help resolving. I checked my environment variables inside the container and the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI value is being set by Fargate.
I struggled quite a bit with this issue and constantly having AWS_CONTAINER_CREDENTIALS_RELATIVE_URI wrongly set to None, until I added a custom task role in addition to my current task execution role.
1) The task execution role is responsible for having access to the container in ECR and giving access to run the task itself, while 2) the task role is responsible for your docker container making API requests to other authorized AWS services.
1) For my task execution role I'm using AmazonECSTaskExecutionRolePolicy with the following JSON;
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
2) I finally got rid of the NoCredentialsError: Unable to locate credentials when I added a task role in addition to the task execution role, for instance, responsible of reading from a certain bucket;
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::bucket_name/*"
}
]
}
In summary; make sure to both have a role for 1) executionRoleArn for access to run the task and 2) taskRoleArn for access to make API requests to authorized AWS services set in your task definition.
To allow Amazon S3 read-only access for your container instance role
Open the IAM console at https://console.aws.amazon.com/iam/.
In the navigation pane, choose Roles.
Choose the IAM role to use for your container instances (this role is likely titled ecsInstanceRole). For more information, see Amazon ECS Container Instance IAM Role.
Under Managed Policies, choose Attach Policy.
On the Attach Policy page, for Filter, type S3 to narrow the policy results.
Select the box to the left of the AmazonS3ReadOnlyAccess policy and choose Attach Policy.
You should need an IAM Role to access from your ecs-task to your S3 bucket.
resource "aws_iam_role" "AmazonS3ServiceForECSTask" {
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": [
"ecs-tasks.amazonaws.com"
]
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
data "aws_iam_policy_document" "bucket_policy" {
statement {
principals {
type = "AWS"
identifiers = [aws_iam_role.AmazonS3ServiceForECSTask.arn]
}
actions = [
"s3:ListBucket",
]
resources = [
"arn:aws:s3:::${var.your_bucket_name}",
]
}
statement {
principals {
type = "AWS"
identifiers = [aws_iam_role.AmazonS3ServiceForECSTask.arn]
}
actions = [
"s3:GetObject",
]
resources = [
"arn:aws:s3:::${var.your_bucket_name}/*",
]
}
}
You should need add your IAM Role in task_role_arn of your task definition.
resource "aws_ecs_task_definition" "_ecs_task_definition" {
task_role_arn = aws_iam_role.AmazonS3ServiceForECSTask.arn
execution_role_arn = aws_iam_role.ECS-TaskExecution.arn
family = "${var.family}"
network_mode = var.network_mode[var.launch_type]
requires_compatibilities = var.requires_compatibilities
cpu = var.task_cpu[terraform.workspace]
memory = var.task_memory[terraform.workspace]
container_definitions = module.ecs-container-definition.json
}
ECS Fargate task not applying role
After countless hours of digging this parameter finally solved the issue for me:
auto_assign_public_ip = true inside a network_configuration block on ecs service.
Turns out my tasks ran by these service didn't have IP assigned and thus no connection to outside world.
Boto3 has a credential lookup route: https://boto3.readthedocs.io/en/latest/guide/configuration.html. When you use AWS provided images to create your EC2 instance, the instance pre-install the aws command and other AWS credential environmental variables. However, Fargate is only a container. You need to manually inject AWS credentials to the container. One quick solution is to add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to the fargate container.
I have multiple EC2 instances originating form a single VPC and i want to assign a bucket policy to my s3 to make sure that only that VPC traffic will be allowed to access the bucket so i created an end point for that VPC and it added all the policies and routes in routing table. I assigned a following policy to my bucket
{
"Version": "2012-10-17",
"Id": "Policy1415115909153",
"Statement": [
{
"Sid": "Access-to-specific-VPCE-only",
"Action": "s3:*",
"Effect": "Allow",
"Resource": ["arn:aws:s3:::examplebucket",
"arn:aws:s3:::examplebucket/*"],
"Condition": {
"StringtEquals": {
"aws:sourceVpce": "vpce-111bbb22"
}
}
}
]
}
but it does not work when i connect to my Bucket using AWS-SDK for nodejs i get access denied error. The nodejs application is actually running in the Ec2 instance launched in same VPC as end point.
I even tried VPC level bucket policy but still i get access denied error. Can anyone tell me if i need to include any endpoint parameter in SDK S3 connection or any other thing?
I have created all that are needed for a successful deployment.
I tried to make the deployment without configuring the CodeDeploy agent in the Amazon instance and the deployment [obviously] failed.
After setting it up though, succeeded.
So, my question is, should I configure every instance that I use manually?
What if I have 100 instances in the deployment group?
Should I create an AMI with the CodeDeploy agent tool already configured?
EDIT
I have watched this:
https://www.youtube.com/watch?v=qZa5JXmsWZs
with this:
https://github.com/andrewpuch/code_deploy_example
and read this:
http://blogs.aws.amazon.com/application-management/post/Tx33XKAKURCCW83/Automatically-Deploy-from-GitHub-Using-AWS-CodeDeploy
I just cannot understand why I must configure with the IAM creds the instance. Isn't it supposed to take the creds from the role I launched it with?
I am not an expert in aws roles and policies, but from the CD documentation this is what I understood.
Is there a way to give the IAM user access to the instance so I wont have to setup the CD agent?
EDIT 2
I think that this post kind of answers: http://adndevblog.typepad.com/cloud_and_mobile/2015/04/practice-of-devops-with-aws-codedeploy-part-1.html
But as you can see, I launched multiple instances but I only installed CodeDeploy agent on one instance, what about others? Do I have to repeat myself and login to them and install them separately? It is OK since I just have 2 or 3. But what if I have handers or even thousand of instances? Actually there are different solutions for this. One of them is, I setup all environment on one instances and create an AMI from it. When I launch my working instance, I will create instance from the one I’ve already configured instead of the AWS default ones. Some other solutions are available
Each instance only requires the CodeDeploy agent installed on it. It does not require the AWS CLI to be installed. See AWS CodeDeploy Agent Operations for installation and operation details.
You should create an instance profile/role in IAM that will grant any instance the correct permissions to accept a code deployment through CodeDeploy service.
Create a role called ApplicationServer. To this role, add the following policy. This assumes you are using S3 for your revisions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::codedeploy-example-com/*"
]
},
{
"Sid": "Stmt1414002531000",
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData"
],
"Resource": [
"*"
]
},
{
"Sid": "Stmt1414002720000",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents"
],
"Resource": [
"*"
]
}
]
}
To your specific questions:
So, my question is, should I configure every instance that I use
manually?
What if I have 100 instances in the deployment group? Should I create
an AMI with the aws-cli tool already configured?
Configure AMI with your base tools, or use CloudFormation or puppet to manage software installation on a given instance as needed. Again the AWS CLI is not required for CodeDeploy. Only the most current version of the CodeDeploy agent is required.
I cannot authenticate to AWS Simple Queue Service (SQS) from an EC2 instance using its associated IAM Role with Boto 2.38 library (and Python 3).
I couldn't find anything specific on documentation about it, but as far as I could understand from examples and other questions around, it was supposed to work just opening a connection like this.
conn = boto.sqs.connect_to_region('us-east-1')
queue = conn.get_queue('my_queue')
Instead, I get a null object from the connect method, unless I provide credentials on my environment, or explicitly to the method.
I'm pretty sure my role is ok, because it works for other services like S3, describing EC2 tags, sending metrics to CloudWatch, etc, all transparently. My SQS policy is like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SQSFullAccess",
"Effect": "Allow",
"Action": [
"sqs:*"
],
"Resource": [
"arn:aws:sqs:us-east-1:<account_id>:<queue_name1>",
"arn:aws:sqs:us-east-1:<account_id>:<queue_name2>"
]
}
]
}
In order to get rid of any suspicion about my policy, I even associated a FullAdmin policy to my role temporarily, without success.
I also verified that it won't work with AWS CLI as well (which, as far as I know, uses Boto as well). So, the only conclusion I could come up with is that this is a Boto issue with SQS client.
Would anyone have a different experience with it? I know that switching to Boto 3 would probably solve it, but I don't consider doing it right now and if it is really a bug, I think it should be reported on git, anyway.
Thanks.
Answering myself.
Boto's 2.38 SQS client does work with IAM Roles. I had a bug in my application.
As for AWS CLI, a credential file (~/.aws/credentials) was present in my local account, and being used instead of the instance's role, because the role is the last one to be looked up by the CLI.