Lambda: Amazon s3 direct upload error signature does not match - node.js

I want to upload image files to AWS s3 bucket by using pre-signed URLs, But I'm getting an error shown in the screen shot, I've followed the post from this page s3 direct file upload, I would like to know what mistake I'm making and also I want to know whether this is server side issue or there should I use some different approach for making put request to 'pre-signed' URL, thanks ahead.
My serverless.yml
service: my-service-api
provider:
name: aws
runtime: nodejs4.3
stage: dev
region: us-east-1
iamRoleStatements:
- Effect: "Allow"
Action:
- "dynamodb:*"
Resource: "*"
- Effect: "Allow"
Action:
- "s3:*"
Resource: "arn:aws:s3:::profile-images/*"
custom:
globalResultTtlInSeconds: 1
package:
individually: true
include:
- node_modules/mysql/**
- node_modules/bluebird/**
- node_modules/joi/**
exclude:
- .git/**
- .bin/**
- tmp/**
- api/**
- node_modules/**
- utils/**
- package.json
- npm_link.sh
- templates.yml
functions:
profiles:
handler: api/profiles/handler.profiles
events:
- http:
method: POST
path: api/profiles/uploadURL
cors: true
integration: lambda
request: ${file(./templates.yml):request}
authorizer:
arn: arn:aws:lambda:us-east-1:000000000000:function:customAuthorizer
resultTtlInSeconds: ${self:custom.globalResultTtlInSeconds}
identitySource: method.request.header.Authorization
package:
include:
- api/profiles/**
- node_modules/node-uuid/**
- node_modules/jsonwebtoken/**
- node_modules/rand-token/**
resources:
Resources:
UploadBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: profile-images
AccessControl: PublicRead
CorsConfiguration:
CorsRules:
- AllowedMethods:
- GET
- PUT
- POST
- HEAD
AllowedOrigins:
- "*"
AllowedHeaders:
- "*"
IamPolicyInvokeLambdaFunction:
Type: AWS::IAM::Policy
Properties:
PolicyName: "lambda-invoke-function"
Roles:
- {"Ref" : "IamRoleLambdaExecution"}
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- "lambda:InvokeFunction"
Resource: "*"
My handler file
var s3Params = {
Bucket: 'profile-images',
Key: image.name,
ACL: 'public-read'
};
s3.getSignedUrl('putObject', s3Params, function (err, url){
if(err){
console.log('presignedURL err:',err);
context.succeed({error: err});
}
else{
console.log('presignedURL: ',url);
context.succeed({uploadURL: url});
}
});

After spending more time on this issue, I realized that this was not problem on server side but the problem was in making request. I needed to set headers for my PUT request because when AWS s3 receives any request it checks the signature of that request versus the headers so if you are setting 'ContentType' or 'ACL' while creating preSignedURL then you have to provide the 'Content-Type' and 'x-amz-acl' in your request.
This is my updated 's3Params'
var s3Params = {
Bucket: 'profile-images',
Key: image.name,
ACL: 'public-read',
ContentType: image.type
};
And this is my request
Lastly I got some help from this post set headers for presigned PUT s3 requests

Related

API Gateway refuses to connect to S3,DynamodB using Localstack and serverless

I'm developing Localstack application where i need to connect s3 bucket and dynamodB to the API gateway, when i invoke my lambda using command line using
serverless invoke local -f helloagain
it just works fine, but when i invoke it through postman using api gateway e.g. http://localhost:4566/restapis/82wh1mpf11/local/_user_request_/helloagain it gives internal server error.
Read operations on s3 and ddb are working fine, it is getting a tedious task when it comes to dumping the data into these resources.
Sharing my code snippets below
this is docker-compose file
version: "3.1"
services:
localstack:
image: localstack/localstack:latest
environment:
- AWS_DEFAULT_REGION=us-east-1
- EDGE_PORT=4566
- SERVICES=lambda,s3,cloudformation,sts,apigateway,iam,route53
ports:
- "4566-4597:4566-4597"
volumes:
- "${TEMPDIR:-/tmp/localstack}:/temp/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
this is a serverless.yml
service: localstack-lambda
plugins:
- serverless-localstack
custom:
localstack:
debug: true
stages:
- local
- dev
endpointFile: localstack_endpoints.json
# frameworkVersion: "2"
provider:
name: aws
runtime: nodejs12.x
#functions are pretty important
functions:
hello:
handler: handler.hello
events:
- http:
path: hello
method: any
helloagain:
handler: handler.helloagain
events:
- http:
path: helloagain
method: any
createbucket:
handler: s3handler.createbucket
events:
- http:
path: createbucket
method: any
createDDB:
handler: ddb.createDDB
events:
- http:
path: createDDB
method: any
listTables:
handler: ddb.listTables
events:
- http:
path: listTables
method: any
this is helloagain lambda script
const aws = require('aws-sdk');
const fs = require('fs');
const s3 = new aws.S3({
apiVersion: '2006-03-01',
endpoint: `http://localhost:4566`, // This two lines are
s3ForcePathStyle: true, //only needed for LocalStack.
});
module.exports.helloagain = async (event) => {
const file_buffer = fs.readFileSync("demo2.txt");
const params = {
Bucket: "newbucket",
Key: "demo2.txt",
Body:file_buffer
}
const res = await s3.upload(params).promise();
return {
statusCode: 200,
body: JSON.stringify({
res
})
}
}
i've already checked invoking this lambda through command line that
file demo2.txt already exists
bucket exists
i'm getting this error in the console
localstack.services.awslambda.lambda_executors.InvocationException: Lambda process returned with error. Result: {"errorType":"NetworkingError","errorMessage":"connect ECONNREFUSED 127.0.0.1:4566"}.
and this in the postman

AWS Lambda: 503 Service Unavailable

I am developing a REST API in NodeJs using AWS Lambda and AWS API Gateway. I am using AWS SAM Template as well.
Below is my NodeJS Code. There I am only trying to access a sample API over the internet and make a POST call.
const mysql = require('mysql2');
const errorCodes = require('source/error-codes');
const PropertiesReader = require('properties-reader');
const fetch = require('node-fetch');
const prop = PropertiesReader('properties.properties');
const con = mysql.createConnection({
host: prop.get('server.host'),
user: prop.get("server.username"),
password: prop.get("server.password"),
port: prop.get("server.port"),
database: prop.get("server.dbname")
});
exports.testApi = async (event, context) => {
context.callbackWaitsForEmptyEventLoop = false;
con.config.namedPlaceholders = true;
if (event.body == null && event.body == undefined) {
var response = errorCodes.missing_parameters;
return response;
}
let body = JSON.parse(event.body)
if (body.key == null ) {
console.log("fire 1");
var response = errorCodes.not_null_parameters;
return response;
}
try {
let key = body.key;
console.log("body", body);
var notificationMessage = {
"key": key
};
const notificationResponse = await fetch("https://reqbin.com/sample/post/json", {
method: 'post',
body: JSON.stringify(notificationMessage),
headers: {
'Content-Type': 'application/json'
}
});
const data = await notificationResponse.json();
//Return the response
var response = {
"statusCode": 200,
"headers": {
"Content-Type": "application/json"
},
"body": JSON.stringify({
"message": data
}),
"isBase64Encoded": false
};
return response;
} catch (error) {
console.log(error);
//Return the response
var response = {
"statusCode": 500,
"headers": {
"Content-Type": "application/json"
},
"body": JSON.stringify({
"error": error
}),
"isBase64Encoded": false
};
return response;
}
};
Below is my template.yaml file. It contains nested access to another template.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
xxx-restapi
Sample SAM Template for xxx-restapi
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 5
VpcConfig:
SecurityGroupIds:
- sg-xxxxx
SubnetIds:
- subnet-xxxx
- subnet-aaaa
- subnet-bbbb
- subnet-cccc
- subnet-dddd
- subnet-eeee
Parameters:
FirebaseProjectId:
Type: String
#Dont create this domain in the AWS Console manually, so it will fail here
DomainName:
Type: String
Default: api2.someapp.com
Resources:
# Authentication required HTTP API
AuthGatewayHttpApi:
Type: AWS::Serverless::HttpApi
Properties:
Domain:
DomainName: !Ref DomainName
EndpointConfiguration: REGIONAL
CertificateArn: arn:aws:acm:us-east-1:xxxxxx:certificate/bac44716-xxxx-431b-xxxx-xxxx
Route53:
HostedZoneId: xxxxxxx
IpV6: true
Auth:
Authorizers:
FirebaseAuthorizer:
IdentitySource: $request.header.Authorization
JwtConfiguration:
audience:
- !Ref FirebaseProjectId
issuer: !Sub https://securetoken.google.com/${FirebaseProjectId}
DefaultAuthorizer: FirebaseAuthorizer
# Authentication NOT required HTTP API
NoAuthGatewayHttpApi:
Type: AWS::Serverless::HttpApi
Properties:
Domain:
BasePath: noauth
DomainName: !Ref DomainName
CertificateArn: arn:aws:acm:us-east-1:xxxx:certificate/xxx-420d-xxx-xxx-xxxx
Route53:
HostedZoneId: xxxxxx
# Lambda settings
LambdaRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: root
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- ec2:DescribeNetworkInterfaces
- ec2:CreateNetworkInterface
- ec2:DeleteNetworkInterface
- ec2:DescribeInstances
- ec2:AttachNetworkInterface
Resource: '*'
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
SharedValueOutput:
Value: !Ref FirebaseProjectId
Description: You can refer to any resource from the template.
# HelloWorldApi:
# Description: "API Gateway endpoint URL for Prod stage for functions"
# Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
Below is the nested template file
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
xxx-restapi
Sample SAM Template for xxx-restapi
Globals:
Function:
Timeout: 30
VpcConfig:
SecurityGroupIds:
- sg-xxxx
SubnetIds:
- subnet-xxx
- subnet-aaa
- subnet-ccc
- subnet-sss
- subnet-fff
- subnet-eee
Parameters:
FirebaseProjectId:
Type: String
DomainName:
Type: String
Resources:
NoAuthGatewayHttpApi2:
Type: AWS::Serverless::HttpApi
Properties:
StageName: Prod
MyApiMapping:
DependsOn: NoAuthGatewayHttpApi2
Type: AWS::ApiGatewayV2::ApiMapping
Properties:
ApiMappingKey: no-auth
DomainName: api2.xxx.com
ApiId: !Ref NoAuthGatewayHttpApi2
Stage: !Ref NoAuthGatewayHttpApi2.Stage
TestPostFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: xxx-restapi/
Handler: source/fcm/test-api.testApi
Runtime: nodejs14.x
Events:
GetRtcTokenAPIEvent:
Type: HttpApi
Properties:
Path: /fcm/test-api
Method: post
ApiId: !Ref NoAuthGatewayHttpApi2
When I executed the lambda function in local environment it works fine. But if i executed the same after uploading to AWS, it gives me the following error with 503 status code.
{
"message": "Service Unavailable"
}
Below is my Cloud Watch log.
START RequestId: e84242ea-xxx-4aa0-xxx-xxx Version: $LATEST
2022-06-12T05:55:52.914Z e84242ea-xxx-4aa0-xxx-xxx INFO body { key: 'value' }
END RequestId: e84242ea-xxx-xxx-dd37f2a005c0
REPORT RequestId: e84242ea-xxx-4aa0xxx993e-xxx Duration: 30032.56 ms Billed Duration: 30000 ms Memory Size: 128 MB Max Memory Used: 72 MB Init Duration: 315.65 ms
2022-06-12T05:56:22.936Z xxx-b568-xxx-993e-xxx Task timed out after 30.03 seconds
The timed out error you see above is just a mask to the real problem. I even tried this with time limit set to 2 minutes, same result. In my local environment with Docker this works in seconds.
After googling I figured out I "may" have not enabled internet connection from my Lambda functions to outside. Even in my API, all Lambda functions that require no Lambda to outside internet connection is working fine.
How can I fix this error?
I would first try to debug the networking aspect. Can anything in those subnets connect to the database? Are the security groups correct?
You might find it easier to create an EC2 instance and debug the networking from there. It certainly seems like a connection timing out.
I got the 503 error as well,
turns out to be the Api Gateway timeout problem.

s3.listObjectsV2 AccessDenied

Hi I try to list all files in a "folder" of a S3 bucket. So I try:
const params = {
Bucket: s3bucketname,
Prefix: foldername + "/"
};
const data = await s3.listObjectsV2(params).promise()
console.log(data)
which give me an AccessDenied error.
Files itself are stored like this
const params = {
Bucket: s3bucketname,
Key: foldername + "/" + filename,
ContentType: 'image/jpeg',
ACL: 'public-read',
Body: data
};
await s3.upload(params, tags).promise();
and public available. which works fine. Also using
const params = {
Bucket: s3bucketname,
Key: folderfilename
};
await s3.headObject(params).promise()
works fine.
yml file looks like this
S3BucketampstoryscreenshotsPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket:
Ref: S3Bucketampstoryscreenshots
PolicyDocument:
Statement:
-
Action:
- "s3:Get*"
Effect: "Allow"
I tested multiple small variations but cant find the problem here?
The ListBucket statement was needed. Important here to have Ref once with /* and once without /* Not having it set like that causes
Action does not apply to any resource(s) in statement (Service: Amazon
S3; Status Code: 400; Error Code: MalformedPolicy;
This works in my case:
PolicyDocument:
Statement:
-
Action:
- "s3:GetObject"
Effect: "Allow"
Resource:
Fn::Join:
- ""
-
- "arn:aws:s3:::"
-
Ref: "S3Bucketampstoryscreenshots"
- "/*"
Principal: "*"
-
Action:
- "s3:ListBucket"
Effect: "Allow"
Resource:
Fn::Join:
- ""
-
- "arn:aws:s3:::"
-
Ref: "S3Bucketampstoryscreenshots"
Principal: "*"
Thanks to jarmod for the hint

How to add AWS user permissions using serverless?

I have created a user in the AWS console with access only to the Lambda service.
My question is, using the serverless framework, in my serverless.yaml, is it possible to add S3 Full access to my user and possibly any other service?
Thank you.
handler.js
'use strict';
const aws = require('aws-sdk');
const s3 = new aws.S3({ apiVersion: '2006-03-01' });
module.exports.helloWorld = (event, context, callback) => {
const params = {};
s3.listBuckets(params, function(err, data) {
if (err) console.log(err, err.stack);
else console.log(data);
});
const response = {
statusCode: 200,
message: JSON.stringify({message: 'Success!'})
};
callback(null, response);
};
serverless.yaml
provider:
name: aws
runtime: nodejs8.10
region: eu-blah-1
iamRoleStatements:
- Effect: "Allow"
Action:
- "s3:ListBucket"
- "s3:PutObject"
- "s3:GetObject"
Resource: "arn:aws:s3:::examplebucket/*"
functions:
helloWorld:
handler: handler.helloWorld
events:
- http:
path: hello-world
method: get
cors: true
If you are referring to the permissions that you give to the Lambda Function to have at execution time, after it has been deployed by the Serverless Framework, then you add role permissions in the serverless.yaml file, within the provider section.
Here is an example of permissions for the Lambda to talk to S3, Execute other Lambdas, and Send Emails with SES:
iamRoleStatements:
- Effect: "Allow"
Action:
- "s3:PutObject"
- "s3:DeleteObject"
- "s3:DeleteObjects"
Resource: arn:aws:s3:::${self:custom.s3WwwBucket}/content/pages/*
- Effect: Allow
Action:
- lambda:InvokeFunction
- lambda:InvokeAsync
Resource: arn:aws:lambda:${self:custom.region}:*:function:${self:service}-${opt:stage}-*
- Effect: "Allow"
Action:
- "ses:SendEmail"
- "ses:SendEmailRaw"
Resource: "arn:aws:ses:eu-west-1:01234567891234:identity/noreply#example.com"

How to call an AWS Step Function using the definitions in the serverless-step-functions plugin?

I'm using Serverless Framework to create my Lambda functions and the serverless-step-functions plugin to define my step functions.
Is it possible to call an step function directly from one of the lambda functions using the name defined into the serverless.yml file?
I was trying to solve the same problem and this question and the self answer were very helpful. However, I want to add another answer with more details and a working example to help future readers.
There are two things that you may need:
1- Start a State Machine
2- Invoke one specific function from a State Machine (usually for testing purposes)
The following demo uses both cases.
First, we need to configure the serverless.yml file to declare the State Machine, the Lambda functions and the correct IAM permissions.
service: test-state-machine
provider:
name: aws
runtime: nodejs4.3
region: us-east-1
stage: dev
environment:
AWS_ACCOUNT: 1234567890 # use your own AWS ACCOUNT number here
# define the ARN of the State Machine
STEP_FUNCTION_ARN: "arn:aws:states:${self:provider.region}:${self:provider.environment.AWS_ACCOUNT}:stateMachine:${self:service}-${self:provider.stage}-lambdaStateMachine"
# define the ARN of function step that we want to invoke
FUNCTION_ARN: "arn:aws:lambda:${self:provider.region}:${self:provider.environment.AWS_ACCOUNT}:function:${self:service}-${self:provider.stage}-stateMachineFirstStep"
functions:
# define the Lambda function that will start the State Machine
lambdaStartStateMachine:
handler: handler.lambdaStartStateMachine
role: stateMachine # we'll define later in this file
# define the Lambda function that will execute an arbitrary step
lambdaInvokeSpecificFuncFromStateMachine:
handler: handler.lambdaInvokeSpecificFuncFromStateMachine
role: specificFunction # we'll define later in this file
stateMachineFirstStep:
handler: handler.stateMachineFirstStep
# define the State Machine
stepFunctions:
stateMachines:
lambdaStateMachine:
Comment: "A Hello World example"
StartAt: firstStep
States:
firstStep:
Type: Task
Resource: stateMachineFirstStep
End: true
# define the IAM permissions of our Lambda functions
resources:
Resources:
stateMachine:
Type: AWS::IAM::Role
Properties:
RoleName: stateMachine
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: stateMachine
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: "Allow"
Action:
- "states:StartExecution"
Resource: "${self:provider.environment.STEP_FUNCTION_ARN}"
specificFunction:
Type: AWS::IAM::Role
Properties:
RoleName: specificFunction
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: specificFunction
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: "Allow"
Action:
- "lambda:InvokeFunction"
Resource: "${self:provider.environment.FUNCTION_ARN}"
package:
exclude:
- node_modules/**
- .serverless/**
plugins:
- serverless-step-functions
Define the Lambda functions inside the handler.js file.
const AWS = require('aws-sdk');
module.exports.lambdaStartStateMachine = (event, context, callback) => {
const stepfunctions = new AWS.StepFunctions();
const params = {
stateMachineArn: process.env.STEP_FUNCTION_ARN,
input: JSON.stringify({ "msg": "some input" })
};
// start a state machine
stepfunctions.startExecution(params, (err, data) => {
if (err) {
callback(err, null);
return;
}
const response = {
statusCode: 200,
body: JSON.stringify({
message: 'started state machine',
result: data
})
};
callback(null, response);
});
};
module.exports.lambdaInvokeSpecificFuncFromStateMachine = (event, context, callback) => {
const lambda = new AWS.Lambda();
const params = {
FunctionName: process.env.FUNCTION_ARN,
Payload: JSON.stringify({ message: 'invoked specific function' })
};
// invoke a specific function of a state machine
lambda.invoke(params, (err, data) => {
if (err) {
callback(err, null);
return;
}
const response = {
statusCode: 200,
body: JSON.stringify({
message: 'invoke specific function of a state machine',
result: data
})
};
callback(null, response);
});
};
module.exports.stateMachineFirstStep = (event, context, callback) => {
const response = {
statusCode: 200,
body: JSON.stringify({
message: 'state machine first step',
input: event
}),
};
callback(null, response);
};
Deploy executing:
serverless deploy stepf
serverless deploy
Test using:
serverless invoke -f lambdaStartStateMachine
serverless invoke -f lambdaInvokeSpecificFuncFromStateMachine
Solved using serverless environment variables:
environment:
MYFUNCTION_ARN: "arn:aws:states:${self:provider.region}:${self:provider.environment.AWS_ACCOUNT}:stateMachine:${self:service}-${self:provider.stage}-myFunction"
In the function:
var params = {
stateMachineArn: process.env.MYFUNCTION_ARN
};
Here is how you solve it nowadays.
In your serverless.yml, define your stepFunctions and also Outputs:
# define your step functions
stepFunctions:
stateMachines:
myStateMachine:
name: stateMachineSample
events:
- http:
path: my-trigger
method: GET
# make it match your step functions definition
Outputs:
myStateMachine:
Value:
Ref: StateMachineSample
Then you can set your state machine ARN as an environment using ${self:resources.Outputs.fipeStateMachine.Value}.

Resources