Python3.7 script to export CloudWatch logs to S3 - python-3.x

Question

I am using below code to copy CloudWatch logs to S3:-
import boto3
import collections
from datetime import datetime, date, time, timedelta
region = 'eu-west-1'
def lambda_handler(event, context):
yesterday = datetime.combine(date.today()-timedelta(1),time())
today = datetime.combine(date.today(),time())
unix_start = datetime(1970,1,1)
client = boto3.client('logs')
response = client.create_export_task(
taskName='Export_CloudwatchLogs',
logGroupName='/aws/lambda/stop-instances',
fromTime=int((yesterday-unix_start).total_seconds() * 1000),
to=int((today -unix_start).total_seconds() * 1000),
destination='bucket',
destinationPrefix='bucket-{}'.format(yesterday.strftime("%Y-%m-%d"))
)
return 'Response from export task at {} :\n{}'.format(datetime.now().isoformat(),response)
I gave below policy to role:-
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams",
"logs:CreateExportTask",
"logs:DescribeExportTasks",
"logs:DescribeLogGroups"
],
"Resource": [
"arn:aws:logs:*:*:*"
]
}
]
}
EOF
2nd policy:-
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetBucketAcl"
],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::${var.source_market}-${var.environment}-${var.bucket}/*"],
"Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }
}
]
}
EOF
I am getting below error if I execute this in AWS console:-
{
"errorMessage": "An error occurred (InvalidParameterException) when calling the CreateExportTask operation: GetBucketAcl call on the given bucket failed. Please check if CloudWatch Logs has been granted permission to perform this operation.",
"errorType": "InvalidParameterException"
I have referred many blocks after appending role with appropriate policies.

Answer 1

Check the encryption settings on your bucket. I had the same problem and it was because I had it set to AWS-KMS. I was getting this error with the same permissions you have and then it started working as soon as I switched the encryption to AES-256

Answer 2

It seems like an issue with s3 bucket permissions. You need to attach this policy to your s3 bucket. Please amend the policy by changing the bucket name and aws region for cloudwatch.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:GetBucketAcl",
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-exported-logs",
"Principal": { "Service": "logs.us-west-2.amazonaws.com" }
},
{
"Action": "s3:PutObject" ,
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
"Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } },
"Principal": { "Service": "logs.us-west-2.amazonaws.com" }
}
]}
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.html

Answer 3

I had the same error, the issue was that I put on "destination" parameter something like bucket/something while on the policy I just had bucket, so removing the something prefix on the parameter fixed the problem, so check that the policy and the parameter match.