serverless s3 AccessDenied - node.js

I spent hours on this issue without understanding why the Access Denied
Here my corresponding part of the serverless.yml
provider:
name: aws
runtime: nodejs12.x
region: eu-central-1
environment:
STAGE: ${opt:stage, self:provider.stage}
iamRoleStatements:
- Effect: Allow
Action:
- s3:GetObject
- s3:PutObject
Resource:
- arn:aws:s3:::<bucket-1>/*
- arn:aws:s3:::<bucket-2>/*
- Effect: Allow
Action:
- s3:ListBucket
Resource:
- arn:aws:s3:::<bucket-1>
- arn:aws:s3:::<bucket-2>
Then running
return s3DataProvider.upload({
Bucket: store.bucket,
ACL: 'public-read',
Body: sm.toString(),
Key: `front/${process.env.STAGE}/sitemap.xml`,
ContentType: 'text/xml'
}).promise()
where store.bucket is by <bucket-1> or <bucket-2>
I always have
ERROR AccessDenied: Access Denied
at Request.extractError (/var/task/node_modules/aws-sdk/lib/services/s3.js:837:35)
at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'AccessDenied',
region: null,
time: 2021-01-25T21:48:47.259Z,
requestId: '546A64CC9D503FA8',
extendedRequestId: 'hoRF0wDih8jRimR7Ew0ajMhgf4qQ88DCXjWM6bdd1CUsP+9OdpNkiXwZz1UFAK+s7L/clFH4U2c=',
cfId: undefined,
statusCode: 403,
retryable: false,
retryDelay: 24.899574651815936
}

I'm not 100% sure but I guess there's s3:PutObjectAcl missing as your setting public-read for the object.
Edit: probably be safe and also grant s3:GetObjectAcl. A lot of discussions and similar issues can be found here: Getting Access Denied when calling the PutObject operation with bucket-level permission

Related

DigitalOcean Spaces / Amazon S3 "InvalidArgument: null at Request.extractError"

InvalidArgument: null
at Request.extractError (P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\services\s3.js:700:35)
at Request.callListeners (P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\sequential_executor.js:106:20)
at Request.emit (P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\sequential_executor.js:78:10)
at Request.emit (P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\request.js:688:14)
at Request.transition (P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\request.js:22:10)
at AcceptorStateMachine.runTo (P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\state_machine.js:14:12)
at P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\request.js:38:9)
at Request.<anonymous> (P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\request.js:690:12)
at Request.callListeners (P:\Upwork\MyProject\backend\node_modules\aws-sdk\lib\sequential_executor.js:116:18) {
code: 'InvalidArgument',
region: null,
time: 2020-12-31T15:39:45.724Z,
requestId: '',
extendedRequestId: undefined,
cfId: undefined,
statusCode: 400,
retryable: false,
retryDelay: 85.1667642693943
}
This error occurs when trying to upload to DigitalOcean Spaces or Amazon S3.
The error message is very vague just supplying InvalidArgument as a reason.
code: 'InvalidArgument' suggests that there is an issue with the supplied uploadParams.
const uploadParams = {
Bucket: process.env.DIGITAL_OCEAN_PUBLIC_SPACE_NAME,
Key: `profile-picture/${userUUID}.jpg`,
Body: body,
ACL: 'public',
};
In my case we can see above that I was trying to use ACL: 'public'.
This is not a valid ACL value, a value of ACL: 'public-read' should be used. For private objects you should use ACL: 'private'. The documentation can be found here.
If you receive this error check that your bucket name, ACL values and all of your uploadParams are valid.

how do i use batchGet() to fetch tow tables

I am trying to fetch two tables using batch Get().
Error { ResourceNotFoundException: Requested resource not found
at Request.extractError (D: \Posts\node_modules\aws-sdk\lib\protocol\json.js:51:27)
at Request.callListeners (D:\Posts\node_modules\aws-sdk\lib\sequential_executor.js:106:20)
at Request.emit (D:\Posts\node_modules\aws-sdk\lib\sequential_executor.js:78:10)
at Request.emit (D: \Posts\node_modules\aws-sdk\lib\request.js:683:14)
at Request.transition (D: \Posts\node_modules\aws-sdk\lib\request.js:22:10)
at AcceptorStateMachine.runTo (D: \Posts\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D: \Posts\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Posts\node_modules\aws-sdk\lib\request.js:38:9)
at Request.<anonymous> (D:\Posts\node_modules\aws-sdk\lib\request.js:685:12)
at Request.callListeners (D:\Posts\node_modules\aws-sdk\lib\sequential_executor.js:116:18)
message: 'Requested resource not found',
code: 'ResourceNotFoundException',
time: 2019-09-23T07:29:13.128Z,
requestId: 'VVSRDL4V2299JSMFKD88R8TT77VV4KQNSO5AEMVJF66Q9ASUAAJG',
statusCode: 400,
retryable: false,
retryDelay: 24.23023597884083 }

AWS SDK and NODE JS --> UnknownEndpoint

I have followed this guide: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/ec2-example-creating-an-instance.html
When I run the code below - I get the UnknowEndPoint error.
I have created an AWS IAM user and added the user to a group with the policy name "IAMFullAccess" - I don't know which other policy would fit (I can see in the AWS dashboard that my credentials has been used programmatically - So I guess this part works well...)?
I have placed my credentials in the .aws folder in a file named credentials.
I am not sure what actually to put inside the 'KeyName' below - currently I have entered the Key Pair Name I find by clicking on one of my existing AWS EC2 instances in the dashboard. Is this correct?
Is there some security group or similar I need to edit - to able to connect NodeJS to my AWS account? Otherwise I have no clue what to do?
In start.js file:
process.env.AWS_SDK_LOAD_CONFIG=1;
var AWS_SDK = require('./aws_sdk');
var aws_sdk = new AWS_SDK();
aws_sdk.CopyInstance();
In aws_sdk.js file:
function AWS_SDK() {
this.CopyInstance = function() {
try {
// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');
//Set the region
AWS.config.update({region: 'us-east-2a'});
var instanceParams = {
ImageId: 'ami-0...',
InstanceType: 't1.micro',
KeyName: '<Key_name>',
MinCount: 1,
MaxCount: 1
};
// Create a promise on an EC2 service object
var instancePromise = new AWS.EC2({apiVersion: '2016-11-15'}).runInstances(instanceParams).promise();
// Handle promise's fulfilled/rejected states
instancePromise.then(
function (data) {
console.log(data);
var instanceId = data.Instances[0].InstanceId;
console.log("Created instance", instanceId);
// Add tags to the instance
var tagParams = {
Resources: [instanceId], Tags: [
{
Key: 'Name',
Value: 'SDK Sample'
}
]
};
// Create a promise on an EC2 service object
var tagPromise = new AWS.EC2({apiVersion: '2016-11-15'}).createTags(tagParams).promise();
// Handle promise's fulfilled/rejected states
tagPromise.then(
function (data) {
console.log("Instance tagged");
}).catch(
function (err) {
console.error(err, err.stack);
});
}).catch(
function (err) {
console.error(err, err.stack);
});
}
catch(e){
wl.info('Error: ' + e);
}
}
function create() {
if(globalAWS === null)
globalAWS = new AWS_SDK();
return globalAWS;
}
module.exports = create;
ERROR:
{ UnknownEndpoint: Inaccessible host: ec2.us-east-2a.amazonaws.com'.
This service may not be available in theus-east-2a' region.
at Request.ENOTFOUND_ERROR (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\event_listeners.js:486:46)
at Request.callListeners (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\sequential_executor.js:106:20)
at Request.emit (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\sequential_executor.js:78:10)
at Request.emit (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\request.js:683:14)
at ClientRequest.error (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\event_listeners.js:325:22)
at ClientRequest. (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\http\node.js:93:19)
at ClientRequest.emit (events.js:182:13)
at ClientRequest.EventEmitter.emit (domain.js:442:20)
at TLSSocket.socketErrorListener (_http_client.js:391:9)
at TLSSocket.emit (events.js:182:13) message: 'Inaccessible host: ec2.us-east-2a.amazonaws.com\'. This service may not be
available in theus-east-2a\' region.', code: 'UnknownEndpoint',
region: 'us-east-2a', hostname: 'ec2.us-east-2a.amazonaws.com',
retryable: true, originalError: { Error: getaddrinfo ENOTFOUND
ec2.us-east-2a.amazonaws.com ec2.us-east-2a.amazonaws.com:443
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:57:26)
message:
'getaddrinfo ENOTFOUND ec2.us-east-2a.amazonaws.com ec2.us-east-2a.amazonaws.com:443',
errno: 'ENOTFOUND',
code: 'NetworkingError',
syscall: 'getaddrinfo',
hostname: 'ec2.us-east-2a.amazonaws.com',
host: 'ec2.us-east-2a.amazonaws.com',
port: 443,
region: 'us-east-2a',
retryable: true,
time: 2019-01-14T20:03:42.177Z }, time: 2019-01-14T20:03:42.177Z } 'UnknownEndpoint: Inaccessible host:
ec2.us-east-2a.amazonaws.com\'. This service may not be available in
theus-east-2a\' region.\n at Request.ENOTFOUND_ERROR
(D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\event_listeners.js:486:46)\n
at Request.callListeners
(D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\sequential_executor.js:106:20)\n
at Request.emit
(D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\sequential_executor.js:78:10)\n
at Request.emit
(D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\request.js:683:14)\n
at ClientRequest.error
(D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\event_listeners.js:325:22)\n
at ClientRequest.
(D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\http\node.js:93:19)\n
at ClientRequest.emit (events.js:182:13)\n at
ClientRequest.EventEmitter.emit (domain.js:442:20)\n at
TLSSocket.socketErrorListener (_http_client.js:391:9)\n at
TLSSocket.emit (events.js:182:13)'
Process finished with exit code 0
If I change the REGION to "us-east-2" (delete the 'a' at the end) the error changes to this:
{ Unsupported: The requested configuration is currently not supported. Please check the documentation for supported configurations.
at Request.extractError (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\services\ec2.js:50:35)
at Request.callListeners (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\sequential_executor.js:106:20)
at Request.emit (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\sequential_executor.js:78:10)
at Request.emit (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\request.js:683:14)
at Request.transition (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\request.js:22:10)
at AcceptorStateMachine.runTo (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\request.js:38:9)
at Request.<anonymous> (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\request.js:685:12)
at Request.callListeners (D:\Workspace\BitBucket\Test\node_modules\aws-sdk\lib\sequential_executor.js:116:18)
message:
'The requested configuration is currently not supported. Please check the documentation for supported configurations.',
code: 'Unsupported',
time: 2019-01-14T20:31:55.954Z,
requestId: '815a44e2-5d0d-453e-a4ff-6faac2695064',
statusCode: 400,
retryable: false,
retryDelay: 51.269952198296934 } 'Unsupported: The requested configuration is currently not supported. Please check the documentation for supported configurations.\n at Request.extractError (D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\services\\ec2.js:50:35)\n at Request.callListeners (D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\sequential_executor.js:106:20)\n at Request.emit (D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\sequential_executor.js:78:10)\n at Request.emit (D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\request.js:683:14)\n at Request.transition (D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\request.js:22:10)\n at AcceptorStateMachine.runTo (D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\state_machine.js:14:12)\n at D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\state_machine.js:26:10\n at Request.<anonymous> (D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\request.js:38:9)\n at Request.<anonymous> (D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\request.js:685:12)\n at Request.callListeners (D:\\Workspace\\BitBucket\\Test\\node_modules\\aws-sdk\\lib\\sequential_executor.js:116:18)'
Update 1:
I changed the REGION from to 'us-est-2' and now also updated my ~/.aws/config file (before the file only contained 1 line/row: 'region=us-west-2'):
[default]
region=us-west-2
output=json
Now I get this error (when I try to decode it in the CMD) - I still don't understand what extra roles I have to give/grant my IAM-AWS user to be able read the error message?
An error occurred (AccessDenied) when calling the DecodeAuthorizationMessage operation: User: arn:aws:iam::0046xxxxxxx:user/user_name is not authorized to perform: sts:DecodeAuthorizationMessage
You have configured your AWS region as us-east-2a. That isn't a region, that is an availability zone. Your region should be configured as us-east-2.
After I added the IAM role: "AdministratorAccess" all issues were gone :) The question is now - which role(s) are actually needed to be granted for it to work..."AdministratorAccess" was just for testing purpose.

Error when refreshing credentials Cognito Identity

I am trying to authenticate a cognito user pool user in node, but I am running into an error (404) when refreshing the credentials. I try authenticating with cognitoUser.authenticateUser. That is successful, and inside the onSuccess, I create credentials with AWS.CognitoIdentityCredentials and get the following
CognitoIdentityCredentials {
expired: true,
expireTime: null,
accessKeyId: undefined,
sessionToken: undefined,
params:
{ IdentityPoolId: '...',
Logins:
{ 'cognito...': '...' } },
data: null,
_identityId: null,
_clientConfig: {} }
So I try refreshing the credentials with AWS.config.credentials.refresh but I get this error
{ UnknownError: Not Found
at Request.extractError (/Users/.../projects/nodekb/node_modules/aws-sdk/lib/protocol/json.js:48:27)
at Request.callListeners (/Users/.../projects/nodekb/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/Users/.../projects/nodekb/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/Users/.../projects/nodekb/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/Users/.../projects/nodekb/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/Users/.../projects/nodekb/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /Users/.../projects/nodekb/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/Users/.../projects/nodekb/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/Users/.../projects/nodekb/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/Users/.../projects/nodekb/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
message: 'Not Found',
code: 'UnknownError',
statusCode: 404,
time: 2018-02-21T19:03:29.670Z,
requestId: '...',
retryable: false,
retryDelay: 81.76059666489807 }
Note: these are the two libraries I have required
const AWS = require('aws-sdk');
const AWSCognito = require('amazon-cognito-identity-js');
and I'm following the instructions here. Specifically use case 4
I believe my issue was that my endpoint was set to dynamo. After I changed the endpoint in the AWS configuration, I got another error
I had the same problem, runnining in node, the endpoint was by default set also to dynamodb endpoint. After reseting it (I changed it to undefined), the credentials were refreshed.
( < AWS.CognitoIdentityCredentials > AWS.config.credentials).refresh((error) => {});

AWS S3 parse URI error with docker containers

On my local environment, I develop a file upload feature with a express/node backend and an AWS S3 server (using scality/S3). Both services are dockerized in their own container, and the communication works fine.
My problem is that the S3 server seems not to understand the upload request. Here is the code I try to use :
const s3 = new S3({
accessKeyId: 'accessKey1',
secretAccessKey: 'verySecretKey1',
endpoint: 's3server:8000',
sslEnabled: false,
s3ForcePathStyle: true,
});
function uploadFile(file) {
const params = {
Body: file,
Bucket: 'testbucket',
Key: 'testkey',
ACL: 'public-read',
};
s3.upload(params, (err, data) => {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
}
The S3 server receives the request but send back the error:
{"name":"S3","clientIP":"::ffff:172.18.0.5","clientPort":45066,"httpMethod":"PUT","httpURL":"/testbucket/testkey","time":1502458550488,"req_id":"7f4fac280644b5cf203c","level":"info","message":"received request","hostname":"faf8cb0b47d4","pid":103}
{"name":"S3","bytesSent":192,"clientIP":"::ffff:172.18.0.5","clientPort":45066,"httpMethod":"PUT","httpURL":"/testbucket/testkey","httpCode":400,"time":1502458550491,"req_id":"7f4fac280644b5cf203c","elapsed_ms":2.607924,"level":"info","message":"responded with error XML","hostname":"faf8cb0b47d4","pid":103}
And the node backend logs the error:
{ InvalidURI: Couldn't parse the specified URI.
at Request.extractError (/usr/src/api/node_modules/aws-sdk/lib/services/s3.js:577:35)
at Request.callListeners (/usr/src/api/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/usr/src/api/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/usr/src/api/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/src/api/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/src/api/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/src/api/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/usr/src/api/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/usr/src/api/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/usr/src/api/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at Request.emit (/usr/src/api/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/usr/src/api/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/src/api/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/src/api/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/src/api/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/usr/src/api/node_modules/aws-sdk/lib/request.js:38:9)
message: 'Couldn\'t parse the specified URI.',
code: 'InvalidURI',
region: null,
time: 2017-08-11T13:35:50.510Z,
requestId: '7f4fac280644b5cf203c',
extendedRequestId: '7f4fac280644b5cf203c',
cfId: undefined,
statusCode: 400,
retryable: false,
retryDelay: 57.08331622136704 }
I saw some answers about utf-8 encoding problems but it didn't work on my case :/
Does anyone has an idea about why it can't parse the URI ?
Thank you for your time !
You can fix this by providing your own config.json using volumes, as suggested here.
Copy config.json.
Add "s3server": "us-east-1", to "restEndpoints".
docker run -v $(pwd)/config.json:/usr/src/app/config.json ...
or if you're using docker compose, add something like the following to your docker-compose.yaml:
s3server:
image: scality/s3server
restart: always
expose: [8000]
stdin_open: true
tty: true
container_name: s3server
volumes:
- "${PWD}/s3config.json:/usr/src/app/config.json"

Resources