I am trying to use encrypted environment variables in an AWS Lambda function running in Node.js 4.3, but the code hangs when trying to decrypt the variables. I don't get any error messages, it just times out. Here is what I have tried:
I created the encryption key in the same region as the Lambda, and ensured that the role the Lambda runs as has access to the key. (I've even tried giving the role full control of the key.)
When creating the Lambda, I enable encryption helpers, select my encryption key, and encrypt the environment variable:
Next I click the "Code" button which gives me javascript code that's supposed to handle the decryption at runtime. Here is the code--the only change I have made is to add console.log statements and I added a try/catch:
"use strict";
const AWS = require('aws-sdk');
const encrypted = process.env['DBPASS'];
let decrypted;
function processEvent(event, context, callback) {
console.log("Decrypted: " + decrypted);
callback();
}
exports.handler = (event, context, callback) => {
if (decrypted) {
console.log('data is already decrypted');
processEvent(event, context, callback);
} else {
console.log('data is NOT already decrypted: ' + encrypted);
// Decrypt code should run once and variables stored outside of the function
// handler so that these are decrypted once per container
const kms = new AWS.KMS();
console.log('got kms object');
try {
var myblob = new Buffer(encrypted, 'base64');
console.log('got blob');
kms.decrypt({ CiphertextBlob: myblob }, (err, data) => {
console.log('inside decrypt callback');
if (err) {
console.log('Decrypt error:', err);
return callback(err);
}
console.log('try to get plaintext');
decrypted = data.Plaintext.toString('ascii');
console.log('decrypted: ' + decrypted);
processEvent(event, context, callback);
});
}
catch(e) {
console.log("exception: " + e);
callback('error!');
}
}
};
Here is what I get when I run the function:
data is NOT already decrypted: AQECAH.....
got kms object
got blob
END RequestId: 9b7af.....
Task timed out after 30.00 seconds
When I run the function, it times out. I see that it prints all log statements up to "got blob" then it just stops. No error message other than timed out. I've tried increasing timeout and memory for the Lambda but it just makes it wait longer before timing out.
How is decryption supposed to work when I never tell the app what decryption key to use? The documentation for decrypt does not mention any way to tell it what decryption key to use. And I am not getting any error messages that would tell me it doesn't know what key to use or anything.
I've tried going through this tutorial but it just tells me to do the same thing I've already done. I've also read all of the environment variables documentation but it says that what I'm doing should just work.
Decrypting the environment variables requires an API call to the KMS service. To do that, your Lambda function must have access to the internet since there are no VPC endpoints for KMS. So, if your Lambda is running in a VPC, make sure you have a NAT configured for the VPC to allow your Lambda function to call KMS.
Related
I am writing a AWS lambda Authorizer in node.js. We are required to call Azure AD API to fetch the public keys/security policies to validate the incoming the Access Token.
However, to optimize the performance, I decided to store the public keys/security policies in node.js as a constant (this will be active until the Lambda is running or TTL of the keys expire).
Question : Is it safe from a security perspective ? I want to avoid "caching" it in DynamoDB as calls to DynamoDB would also incur additional milliseconds. Ours is a very high traffic application and we would like to save any millisecond possible for optimal performance. Also, any best practice is also higly appreciated
Typically, you should not hard-code things like that in your code. Even though it is not a security problem, it is making maintenance harder.
For example: when the key is "rotated" or the policy changed and you had it hard-coded in your Lambda, you would need to update your code and do another deployment. This is often causing issues, because the developer forgot about this etc. causing issues because your authorizer does not work anymore. If the Lambda loads the information from an external service like S3, SSM or directly Azure AD, you don't need another deployment. In theory, it should sort itself out depending on which service you use and how you manage your keys etc.
I think the best way is to load the key from an external service during the initialisation phase of the Lambda. That means when it is "booted" for the first time and then cache that value for the duration of the Lambdas lifetime (a few minutes to a few hours).
You could for example load the public keys and policies either directly from Azure, from S3 or SSM Parameter Store.
The following code uses the AWS SDK NodeJS v3, which is not bundled with the Lambda Runtime. You can use v2 of the SDK as well.
const { SSMClient, GetParameterCommand } = require("#aws-sdk/client-ssm");
// This only happens once, when the Lambda is started for the first time:
const init = async () => {
const config = {}
try {
// use whatever 'paramName' you defined, when you created the SSM parameter
const paramName = "/azure/publickey"
const command = new GetParameterCommand({Name: paramName});
const ssm = new SSMClient();
const data = await ssm.send(command);
config["publickey"] = data.Parameter.Value;
} catch (error) {
return Promise.reject(new Error("unable to read SSM parameter '"+ paramName + "'."));
}
return new Promise((resolve, reject) => {
resolve(config);
reject(new Error("unable to create configuration. Unknown error."));
});
};
const initPromise = init();
exports.handler = async (event) => {
const config = await initPromise;
console.log("My public key '%s'", config.key);
return "Hello World";
};
The most important point of this code is the init "function", which is only run on once, creating a "config" which should contain your AWS SDK clients and all the configuration you need in your code. This way, you don't have to get the policy for every request that the Lambda is processing etc.
I am using sts:assumeRole to connect to a s3 bucket of a different account.
Now, the job that I run takes a few days and along the way the credentials expire and I needed a way to renew them.
I have written the following code to handle expiry of the temporary credentials
This code is inside my downloadFile():
return new Promise((resolve, reject) => {
function responseCallback(error, data) {
if (error) {
const errorMessage = `Fail to download file from s3://${config().s3.bucket}/${path}: ${error};`;
reject(error);
} else {
Logger.info(`Successfully download file from s3://${config().s3.bucket}/${path}`);
resolve(data.Body);
}
}
const fn = this.s3Client.getObject({
Bucket: config().s3.bucket,
Key: path
}, (error, data) => this.handleTokenExpiry(error, data, fn, responseCallback));
});
And this is the handleTokenExpiry()
handleTokenExpiry(error, data, fn, callback) {
if (!error || error.code !== "ExpiredToken") return callback(error, data);
Logger.info("Token expired, creating new token");
this.s3Client = null; // null so that init() doesn't return existing s3Client
return this.init().then(fn);
}
Here init() is the method which sets this.s3Client using sts:assumeRole
and then new AWS.S3()
This works fine but I am not sure if this a clean way to do it. The strange thing is when I test it in local it takes almost two minutes for responseCallback() to be called when token is expired. Though responseCallback() gets executed immediately while the token is active.
For tasks running less than 12h, here is the solution.
When using AssumeRole, you can specify DurationSeconds argument to specify the duration of the temporary credentials returned by STS. This is 15 min minimum, up to 12h.
The role you are assuming needs to be modified to authorize the maximum duration too. See https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session
In your case, with a job running for several days. I would suggest to refactor the app to run in smaller batches, each running for a few hours each.
Another alternative would be to be proactive about token expiration. If your code know the token duration and the time at which it acquired the token, I would suggest to call a method before calling a method that uses the token (such as S3's getObject). That method you check if the token are soon to expire and proactively refresh them. Pseudo code would be like
function refreshToken() {
return new Promise( (resolve, reject) => {
// XX depends how long is your S3 getObject call
if (token_acquisition_time + token_duration <= now() + xx minutes) {
// refresh token
sts.assumeRole(...).promise().then(resolve());
} else {
resolve();
}
});
}
...
refreshToken.then(s3.getObject(...));
The AWS SDK can handle refreshing the credentials for you. For example:
const credentials = new ChainableTemporaryCredentials({
params: {
RoleArn: "optional-role-arn",
RoleSessionName: `required-parameter-${Date.now()}`
}
})
const s3 = new AWS.S3({credentials})
Now AWS SDK will refresh the tokens behind the scenes without any action from caller of s3.
For more information, please see AWSK SDK Documentation. Refresh is limited to validity time of the credentials used.
I have a node.js function for AWS Lambda. It reads a JSON file from an S3 bucket as a stream, parses it and prints the parsed objects to the console. I am using stream-json module for parsing.
It works on my local environment and prints the objects to console. But it does not print the objects to the log streams(CloudWatch) on Lambda. It simply times out after the max duration. It prints other log statements around, but not the object values.
1. Using node.js 6.10 in both environments.
2. callback to the Lambda function is invoked only after the stream ends.
3. Lambda has full access to S3
4. Also tried Promise to wait until streams complete. But no change.
What am I missing? Thank you in advance.
const AWS = require('aws-sdk');
const {parser} = require('stream-json');
const {streamArray} = require('stream-json/streamers/StreamArray');
const {chain} = require('stream-chain');
const S3 = new AWS.S3({ apiVersion: '2006-03-01' });
/** ******************** Lambda Handler *************************** */
exports.handler = (event, context, callback) => {
// Get the object from the event and show its content type
const bucket = event.Records[0].s3.bucket.name;
const key = event.Records[0].s3.object.key;
const params = {
Bucket: bucket,
Key: key
};
console.log("Source: " + bucket +"//" + key);
let s3ReaderStream = S3.getObject(params).createReadStream();
console.log("Setting up pipes");
const pipeline = chain([
s3ReaderStream,
parser(),
streamArray(),
data => {
console.log(data.value);
}
]);
pipeline.on('data', (data) => console.log(data));
pipeline.on('end', () => callback(null, "Stream ended"));
};
I have figured out that it is because my Lambda function is running inside a private VPC.
(I have to run it inside a private VPC because it needs to access my ElastiCache instance. I removed related code when I posted the code, for simplification).
Code can access S3 from my local machine, but not from the private VPC.
There is a process to ensure that S3 is accessible from within your VPC. It is posted here https://aws.amazon.com/premiumsupport/knowledge-center/connect-s3-vpc-endpoint/
Here is another link that explains how you should setup a VPC end point to be able to access AWS resources from within a VPC https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/
I am using aws-sdk for publishing message on topic below is the code:
var AWS = require('aws-sdk');
AWS.config.region = 'us-east-1';
AWS.config.credentials = {
accessKeyId: 'myaccesskeyid',
secretAccessKey: 'mysecretaccesskey'
}
function LEDOnIntent() {
this.iotdata = new AWS.IotData({
endpoint: 'XXXXXXXXX.iot.us-east-1.amazonaws.com'
});
}
LEDOnIntent.prototype.publishMessage = function() {
console.log('>publishMessage');
var params = {
topic: 'test_topic',
/* required */
payload: new Buffer('{action : "LED on"}') || 'STRING_VALUE',
qos: 1
};
this.iotdata.publish(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
console.log("Message published : " + data); // successful response
}
});
}
It works fine in local unit testing but when I deploy this code on AWS lambda then I got very uneven behaviour. For the first few requests it will not publish message then it will work fine when I continuously test it. When I test after some break then again it stop working for some initial requests.
Behind the scene, Lambda runs like a container model. It means to create a container when needs and destroy it if doesn't require the container.
The reason you see a delay in the initial request because It takes time to set up a container and do the necessary bootstrapping, which adds some latency each time the Lambda function is invoked. You typically see this latency when a Lambda function is invoked for the first time or after it has been updated because AWS Lambda tries to reuse the container for subsequent invocations of the Lambda function.
AWS Lambda maintains the container for some time in anticipation of another Lambda function invocation. In effect, the service freezes the container after a Lambda function completes, and thaws the container for reuse if AWS Lambda chooses to reuse the container when the Lambda function is invoked again.
Please read the official documentation here
I recently changed my aws credentials in my .env file
AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
AWS_SECRET_ACCESS_KEY=
However on every s3.getSignedUrl request, the SDK uses the previous (root) credentials.
s3.getSignedUrl('putObject', s3Params, (err, data) => {
if (err) {
return res.end();
}
console.log(data) <---------------
const returnData = {
signedRequest: data,
awsImageUrl: `https://${S3_BUCKET}.s3.amazonaws.com/${imageName}`
};
res.json(returnData);
res.end();
});
This logs
https://my-bucket.s3.amazonaws.com/my-pic.png?AWSAccessKeyId=YYYYYYYYYYYYYYYContent-Type=image%2Fpng&Expires=SOMEDATE&Signature=SOMESIGNATURE&x-amz-acl=public-read
YYYYYYYYYYYYYYY is the previous, root credentials
Is it possible that the SDK caches this data?
If so how do I invalidate it?
Or have I overlooked something in code?
AWS Documentation
Expiring and Refreshing Credentials
Occasionally credentials can expire in the middle of a long-running
application. In this case, the SDK will automatically attempt to
refresh the credentials from the storage location if the Credentials
class implements the refresh() method.
If you are implementing a credential storage location, you will want
to create a subclass of the Credentials class and override the
refresh() method. This method allows credentials to be retrieved from
the backing store, be it a file system, database, or some network
storage. The method should reset the credential attributes on the
object.
When seeking credentials, the JavaScript and Node SDKs use the AWS.CredentialProviderChain.
The default credentials providers are:
AWS.CredentialProviderChain.defaultProviders = [function () {
return new AWS.EnvironmentCredentials('AWS');
}, function () {
return new AWS.EnvironmentCredentials('AMAZON');
}, function () {
return new AWS.SharedIniFileCredentials();
}, function () {
if (AWS.ECSCredentials.prototype.getECSRelativeUri() !== undefined) {
return new AWS.ECSCredentials();
}
return new AWS.EC2MetadataCredentials();
}]
Thus, it looks in the following locations:
Environment credentials
~/.aws/credentials file
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
Instance metadata