I'm learning aws lambda (creating some rest apis, apply rate limiting). I have read some examples from aws and they said that we need to create/use aws API gateway to route to lambda function (UI based)
But I also found in the internet this serverless.yml. No need to use UI anymore
functions:
simple:
handler: handler.simple
events:
- httpApi: 'PATCH /elo'
extended:
handler: handler.extended
events:
- httpApi:
method: POST
path: /post/just
You guys can see there is no where that mention about api gateway. So my questions are:
If I use configuration like that, how can I know whether it is using API gateway or not? If not, how can I specify it to use API gateway?
Is Lambda-Proxy or Lambda Integration used in this case (read more here)? How can I specify it to use Lambda Integration?
Is aws API gateway suitable for rate limiting? Like allow only 1000 request per user (bearer token) per 120 minutes.
Since I'm still waiting for aws account, I have no environment to test. Any help would be appreciated
First, let's establish that there are two different types of endpoints in API Gateway: REST APIs and HTTPS APIs. These offer different features and customization. For example, REST APIs offer client-level throttling, whereas HTTPS APIs do not. You can see more information about both versions here.
This configuration would create a new HTTPS API gateway endpoint. When you specify that the event triggering the lambda is a post to that specific path, your deployment will create a new endpoint with API gateway to enable that automatically.
The serverless framework allows you to specify whether or not you want to use a REST API or an HTTPS API. The syntax above is for the HTTPS API -- also referred to in serverless' documentation as v2, which by default only supports lambda-proxy. You can opt to use a REST API which can be configured to use either, as you can see reading through the documentation here
You can enable throttling on REST APIs as shown in the (documentation)3:
service: my-service
provider:
name: aws
apiGateway:
apiKeys:
- myFirstKey
- ${opt:stage}-myFirstKey
# you can hide it in a serverless variable
- ${env:MY_API_KEY}
- name: myThirdKey
value: myThirdKeyValue
# let cloudformation name the key (recommended when setting api key value)
- value: myFourthKeyValue
description: Api key description # Optional
customerId: A string that will be set as the customerID for the key # Optional
usagePlan:
quota:
limit: 5000
offset: 2
period: MONTH
throttle:
burstLimit: 200
rateLimit: 100
Then in your function definition:
functions:
hello:
events:
- http:
path: user/create
method: get
private: true
Related
Please help .i am new to cloud endpoint and not able to authenticate my nodejs api using cloud endpoint and api key .
My nodejs api:https://iosapi-dot-ingka-rrm-ugc-dev.appspot.com is working perfectly .However it's not working after authenticate with cloud endpoint and Api key.
For fetching data(get), i am using routing in my api like :
https://iosapi-dot-ingka-rrm-ugc-dev.appspot.com/ugc/iosreviewratings/20200611 : 20200611 : is any date range i have to pass .
https://iosapi-dot-ingka-rrm-ugc-dev.appspot.com/ugc/iosreviewratings/20200611?Limit=2&Offset=1
after endpoint deployment ., whenever i am acessing my api with api key , i am getting error " "message": "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.",
"
My Cloud endpoint has been deployed successfully .The below are my openapi.yaml .(ingka-rrm-ugc-dev : is my project id)
openapi.yaml
swagger: "2.0"
info:
description: "A simple Google Cloud Endpoints API example."
title: "Endpoints Example"
version: "1.0.0"
host: "ingka-rrm-ugc-dev.appspot.com"
consumes:
- "application/json"
produces:
- "application/json"
schemes:
- "https"
paths:
"/ugc/iosreviewratings/*":
get:
produces:
- application/json
operationId: "auth_info_google_jwt"
parameters:
- name: Limit
in: query
required: false
type: string
x-example: '200'
- name: Offset
in: query
required: false
type: string
x-example: '2'
responses:
'200':
description: Definition generated from Swagger Inspector
# This section requires all requests to any path to require an API key.
security:
- api_key: []
securityDefinitions:
# This section configures basic authentication with an API key.
api_key:
type: "apiKey"
name: "key"
in: "query"
app.yaml
========--
runtime: nodejs
env: flex
service: iosapi
# This sample incurs costs to run on the App Engine flexible environment.
# The settings below are to reduce costs during testing and are not appropriate
# for production use. For more information, see:
# https://cloud.google.com/appengine/docs/flexible/nodejs/configuring-your-app-with-app-yaml
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
# [START configuration]
endpoints_api_service:
# The following values are to be replaced by information from the output of
# 'gcloud endpoints services deploy openapi-appengine.yaml' command.
name: ingka-rrm-ugc-dev.appspot.com
rollout_strategy: managed
# [END configuration]
Please help me finding where is issue exactly and why api is not working with end point and api key
already enabled all service for for endpoint
gcloud services enable servicemanagement.googleapis.com
gcloud services enable servicecontrol.googleapis.com
gcloud services enable endpoints.googleapis.com
gcloud services enable ingka-rrm-ugc-dev.appspot.com
.
I am trying to create multiple APIs in APIGateway using a Swagger file.
Right now, when I use the AWS console, I manually import the swagger file and then add the endpoints/authentication in every API created.
I want to complete the same task using AWS Lambda (Node JS), so that i may:
Create APIs using Swagger
Add endpoints (custom server)
Add authorizer (cognito)
I looked into AWS documentation but was not able to understand it in a proper way.
How to complete this task?
If you really have to use swagger, I would say
Write your APIs in swagger json format
Parse it to add your API gateway extensions (endpoints/authorizer/role)
Publish it
Or
Create your API with the given swagger document
Pass the restapi id to a lambda function
Use apigateway SDK (e.g. NodeJS SDK for APIGateway) to get resources and methods for the restapi and add integrations to it.
But have you considered AWS Cloudformation scripts? In my current project, I have considered what you want to do but being no expert in swagger I chose Cloudformation over it. (also considered AWS SAM and serverless). It is a little repetitive but found much easier to write and understand. Here's an example of an endpoint in CFN yaml format -
ResourceUserActions:
Type: AWS::ApiGateway::Resource
DeletionPolicy: Delete
Properties:
RestApiId: !Ref RestAPI
ParentId: !Ref ResourceUser
PathPart: "{action+}"
UserActionsMethod:
Type: AWS::ApiGateway::Method
Properties:
RestApiId: !Ref RestAPI
ResourceId: !Ref ResourceUserActions
HttpMethod: ANY
AuthorizationType: COGNITO_USER_POOLS
AuthorizerId: !Ref ApiAuthorizer
Integration:
Type: AWS_PROXY
IntegrationHttpMethod: POST
Uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HandlerFunction.Arn}/invocations
Credentials: !GetAtt ApiGatewayRole.Arn
Type AWS::ApiGateway::Resource defines a resource, and AWS::ApiGateway::Method adds method, authorizer, integration type details to the resource. For more details, see AWS CFN for APIGateway
I have a custom domain URL (my-custom-domain.com), and the REST API supports query and path parameters.
https://my-custom-domain.com/hello
https://my-custom-domain.com?firstparam=abc&secondparam=def
The invoked lambda has to return the response with some path/query parameters appended to the custom domain URL in json body. Basically the other resources which can be accessed.
Example:
https://my-custom-domain.com/hellofromlambda1123
https://my-custom-domain.com?firstparam=abc&secondparam=yourblogpage&pagenumber=30
An Ideal usecase is pagination, where I have to give the previous and next links. How do I pass the custom domain URL to my lambda.
I am working on node js 8
In conventional JAVA programming we can achieve this by HttpServletRequest.getRequestURL().
What is the way to get the custom Domain URL. I have enabled Headers for DefaultCacheBehavior. The host in the lambda event gives the API gateway URL. Is there a way to get the mapping of the Custom Domain inside lambda?
My Cloud Formation Template for custom domain looks like this
AWSTemplateFormatVersion: '2010-09-09'
Description: Custom domain template
Parameters:
ServiceName:
Description: Name of the Service
Type: String
DeploymentEnv:
Default: dev
Description: The environment this stack is being deployed to.
Type: String
CertificateId:
Description: SSL Certificate Id
Type: String
DomainName:
Description: Name of the custom domain
Type: String
HostedZoneId:
Description: Id of the hosted zone
Type: String
Resources:
APIDistribution:
Type: AWS::CloudFront::Distribution
Properties:
DistributionConfig:
Origins:
- DomainName:
Fn::ImportValue:
!Sub "InvokeURL-${DeploymentEnv}"
Id: !Sub 'Custom-Domain-${DeploymentEnv}'
CustomOriginConfig:
OriginProtocolPolicy: https-only
OriginSSLProtocols: [TLSv1.2]
Enabled: 'true'
DefaultCacheBehavior:
AllowedMethods:
- DELETE
- GET
- HEAD
- OPTIONS
- PATCH
- POST
- PUT
DefaultTTL: 0
TargetOriginId: !Sub 'Custom-Domain-${DeploymentEnv}'
ForwardedValues:
QueryString: 'true'
Cookies:
Forward: none
Headers:
- 'Accept'
- 'api-version'
- 'Authorization'
ViewerProtocolPolicy: https-only
Aliases:
- !Sub '${DomainName}'
ViewerCertificate:
AcmCertificateArn: !Sub '${CertificateId}'
SslSupportMethod: sni-only
MinimumProtocolVersion: TLSv1.2_2018
APIDNSRecord:
Type: AWS::Route53::RecordSet
DependsOn: "APIDistribution"
Properties:
HostedZoneId: !Sub '${HostedZoneId}'
Comment: DNS name for the custom distribution.
Name: !Sub '${DomainName}'
Type: A
AliasTarget:
DNSName: !GetAtt APIDistribution.DomainName
HostedZoneId: Z2FDTNDATAQYW2
EvaluateTargetHealth: false
Outputs:
DomainName:
Value: !GetAtt APIDistribution.DomainName
Thanks to #thomasmichaelwallace for pointing to my post on the AWS Forum that explains a way to inject the original request Host header into an alternate request header, using a Lambda#Edge Origin Request trigger. That is one solution, but requires the Lambda trigger, so there is additional overhead and cost. That solution was really about a CloudFront distribution that handles multiple domain names, but needs to send a single Host header to the back-end application while alerting the application of another request header, which I arbitrarily called X-Forwarded-Host.
There are alternatives.
If the CloudFront distribution is only handling one incoming hostname, you could simply configure a static custom origin header. These are injected unconditionally into requests by CloudFront (and if the original requester sets such a header, it is dropped before the configured header is injected). Set X-Forwarded-Host: api.example.com and it will be injected into all requests and visible at API Gateway.
That is the simplest solution and based on what's in the question, it should work.
But the intuitive solution does not work -- you can't simply whitelist the Host header for forwarding to the origin, because that isn't what API Gateway is expecting.
But there should be a way to make it expect that header.
The following is based on a number of observations that are accurate, independently, but I have not tested them all together. Here's the idea:
use a Regional API Gateway deployment, not Edge-Optimized. You don't want an edge-optimized deployment anyway when you are using your own CloudFront distribution because this increases latency by sending the request through the CloudFront network redundantly. It also won't work in this setup.
configure your API as a custom domain (for your exposed domain)
attaching the appropriate certificate to API Gateway, but
do, not point DNS to the assigned regional domain name API Gateway gives you; instead,
use the assigned regional endpoint hostname as the Origin Domain Name in CloudFront
whitelist the Host header for forwarding
This should work because it will cause API Gateway to expect the original Host header, coupled with the way CloudFront handles TLS on the back-end when the Host header is whitelisted for forwarding.
When using API Gateway + Lambda with the Lambda Proxy integration the event the lambda receives includes the headers.Host and headers.X-Forwarded-Proto keys which can be concatenated to build the full request url.
For example for https://my-custom-domain.com/hellofromlambda1123
{
"headers": {
"Host": "my-custom-domain.com"
"X-Forwarded-Proto": "https"
}
}
I am using a Google Cloud Function (GCF) with a Pubsub trigger which sends a HTTP request to a third party API.
The GCF receives notifications from a Pubsub topic used by a service which should not be aware of the third party API.
The third party API requires an authentication using Basic HTTP Authentication.
In order to not to have to hardcode the password in my source code I am using Google KMS to generate a new encrypted key each time I deploy my function. I am using Google Cloud KMS to decrypt the secret each time the function is instantiated.
For decrypting using KMS I have to provide a private key for a service account to the NodeJS Google API.
My main problem today is that I have to push my private key to the GCloud Bucket if I want my GCF to work properly.
Is it possible by using either the Runtime Configurator or the Deployment Manager to configure secrets for a Google Cloud Function?
Thanks you.
As of December 2019, the preferred way to store and manage secrets on Google Cloud is Secret Manager:
$ echo -n "user:pass" | gcloud beta secrets create "my-basic-auth" \
--data-file=- \
--replication-policy "automatic"
You can also create and manage secrets from API:
// Import the library
const {SecretManagerServiceClient} = require('#google-cloud/secret-manager');
// Create the client
const client = new SecretManagerServiceClient();
// Create the secret
const [secret] = await client.createSecret({
parent: "projects/<YOUR-PROJECT-ID>",
secretId:"my-basic-auth",
secret: {
replication: {
automatic: {},
},
},
});
// Add the version with your data
const [version] = await client.addSecretVersion({
parent: secret.name,
payload: {
data: Buffer.from("user:pass", "utf8"),
},
});
Then, in your Cloud Function:
const [version] = await client.accessSecretVersion({
name:"projects/<YOUR-PROJECT-ID>/secrets/<MY-SECRET>/versions/1",
});
const auth = version.payload.data.toString('utf-8');
// auth is user:pass
The service account with which you deploy your Cloud Function will need roles/secretmanager.secretAccessor permissions.
The other solution to this which came out only in the last few months, is to use Google Cloud Runtime Configuration with Firebase for Functions:
https://firebase.google.com/docs/functions/config-env
Firebase for Functions seems to provide access to several features that are not yet available via other means.
Runtime Configurator does not charge for use, but enforces the following API limits and quotas:
1200 Queries Per Minute (QPM) for delete, create, and update requests
600 QPM for watch requests.
6000 QPM for get and list requests.
4MB of data per user, which consists of all data written to the Runtime Configurator service and accompanying metadata.
https://cloud.google.com/deployment-manager/pricing-and-quotas#runtime_configurator
As an aside, I find this conflict in the Firebase for Functions comical:
The Firebase SDK for Cloud Functions offers built-in environment configuration to make it easy to store and retrieve this type of data for your project without having to redeploy your functions.
Then a moment later:
After running functions:config:set, you must redeploy functions to make the new configuration available.
The KMS solution is a viable alternative, however it seems costly for functions. KMS is billed at $0.06 per month per active key, as well as $0.03 per 10,000 operations.
This would then change the cost of your Cloud Function from $0.40 per million invocations, to $3.40 per million invocations. That is quite the jump.
https://cloud.google.com/kms/
https://cloud.google.com/functions/
Is it possible by using either the Runtime Configurator or the Deployment Manager to configure secrets for a Google Cloud Function?
There is no built-in service that will let you configure secrets to be directly accessed by Google Cloud Functions at this time, so the method you are currently using is the proper way to handle secrets on Cloud functions for the time being. This could change as the product is still in beta.
If you want you can make a feature request to the Cloud Function team by using the appropriate issue tracker.
There's also a Google Cloud Key Management Service: Node.js Client.
cd functions
npm install #google-cloud/kms
For example:
// Imports the Cloud KMS library
const {KeyManagementServiceClient} = require('#google-cloud/kms');
// Instantiates a client
const client = new KeyManagementServiceClient();
// Build the location name
const locationName = client.locationPath(functions.config().firebase.projectId, functions.config().firebase.locationId);
async function listKeyRings() {
const [keyRings] = await client.listKeyRings({
parent: locationName,
});
for (const keyRing of keyRings) {
console.log(keyRing.name);
}
return keyRings;
}
return listKeyRings();
I have created one API endpoint for lambda function, as - https://XXXXXXXXX.execute-api.us-east-1.amazonaws.com/XXXX/XXXXXXXXXXXX/ which is GET method.
While calling that endpoint from postman it is giving me
{
"message": "'XXXXXXXXX3LPDGPBF33Q:XXXXXXXXXXBLh219REWwTsNMyyyfbucW8MuM7' not a valid key=value pair (missing equal-sign) in Authorization header: 'AWS XXXXXXXXX3LPDGPBF33Q:XXXXXXXXXXBLh219REWwTsNMyyyfbucW8MuM7'."
}
This is a screenshot of the Amazon Lambda Upload Site: http://i.stack.imgur.com/mwJ3w.png
I have Access Key Id & Secret Access Key for IAM user. I used it all but no luck. Can anyone suggest tweak about this.
If you're using the latest version of Postman, you can generate the SigV4 signature automatically. The region should correspond to your API region (i.e. "us-east-1") and the service name should be "execute-api"
This is not a solution but it has helped me more than once:
Double-check that you are actually hitting an existing endpoint! Especially if you're working with AWS. AWS will return this error if you don't have the correct handler set up in your Lambda or if your API Gateway is not configured to serve this resource/verb/etc.