I have nodejs/express app from which I want to connect to AWS S3.
I do have a temporary approach to make connection,
environment file
aws_access_key_id=XXX
aws_secret_access_key=XXXX/
aws_session_token=xxxxxxxxxxxxxxxxxxxxxxxxxx
S3-connection-service.js
const AWS = require("aws-sdk");
AWS.config.update({
accessKeyId: `${process.env.aws_access_key_id}`,
secretAccessKey: `${process.env.aws_secret_access_key}`,
sessionToken: `${process.env.aws_session_token}`,
region: `${process.env.LOCAL_AWS_REGION}`
});
const S3 = new AWS.S3();
module.exports = {
listBucketContent: (filePath) =>
new Promise((resolve, reject) => {
const params = { Bucket: bucketName, Prefix: filePath };
S3.listObjects(params, (err, objects) => {
if (err) {
reject(err);
} else {
resolve(objects);
}
});
}),
....
....
}
controller.js
const fetchFile = require("../../S3-connection-service.js");
const AWSFolder = await fetchFile.listBucketContent(filePath);
Fine it's works and I'm able to access S3 bucket files and play with it.
PROBLEM
The problem is connection is not persistent. Since, I use session_token, connection remains alive for sometime and again after sometime new tokens will be generated, I have to copy-paste them in env file and re-run the node app.
I really have no idea how can I make connection persistent ?
Where to store AWS confidential/secrets and how to use them to connect to S3 so connection remains alive ?
Just remove
AWS.config.update({
accessKeyId: `${process.env.aws_access_key_id}`,
secretAccessKey: `${process.env.aws_secret_access_key}`,
sessionToken: `${process.env.aws_session_token}`,
region: `${process.env.LOCAL_AWS_REGION}`
});
code block from lambda source in file S3-connection-service.js
Attach a role to lambda function with proper permissions. You will have same functionally.
For local development.
You can set environment variable before testing your application.
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-west-2
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
If you are using any IDE you can set these environment variables on it.
If you are testing from cli
$ AWS_ACCESS_KEY_ID=EXAMPLE AWS_SECRET_ACCESS_KEY=EXAMPLEKEY AWS_DEFAULT_REGION=us-west-2 npm start
connect to S3 so connection remains alive ?
You can't make one request to S3 and keep it alive forever.
These are your options:
Add a try/catch statement inside your code to handle credentials expired error. Then, generate new credentials and re-initialize the S3 client.
Instead of using a Role, use a User. (IAM Identities). User credentials can be valid forever. You won't need to update the credentials in this case.
Do not provide the credentials to AWS.config.update like you are doing right now. If you don't provide the credentials, the AWS client will try to read them from your ~/.aws/credentials file automatically. If you create a script to update them every hour (ex: a cronjob), then your credentials will be up-to-date at all times.
The problem is when I feed Dynamodb config endpoint some value the AWS Key Management Service stops working altogether.
1. DynamoDB
const awsConfig = {
region: process.env.REGION,
endpoint: process.env.ENDPOINT, //this stops AWS KMS
accessKeyId: process.env.ACCESS_KEY_ID,
secretAccessKey: process.env.ACCESS_KEY,
};
aws.config.update(awsConfig);
2. AWS KMS
constructor() {
this.#KEYAWS = keyAws;
this.#region = process.env.REGION;
this.#secretName = process.env.SECRET_NAME;
this.#secret = process.env.ACCESS_KEY;
this.#_AWS_KEY_ACCESS_KEY_ID = process.env.ACCESS_KEY_ID;
}
async #getPrivateKey() {
this.#KEYAWS.config.update({
accessKeyId: this.#_AWS_KEY_ACCESS_KEY_ID,
secretAccessKey: this.#secret,
});
var client = new this.#KEYAWS.SecretsManager({
region: this.#region,
});
}
When I comment out the endpoint in Dynamo config, the KMS works properly
Note: A VPC endpoint for DynamoDB enables Amazon EC2 instances in your VPC to use their private IP addresses to access DynamoDB with no exposure to the public internet.
This happened because of adding endpoint: process.env.ENDPOINT to global configuration Object. AWS SDK Global Configuration
I removed the endpoint option from global configuration and specifically passed it to dynamodb service aws.DynamoDB.DocumentClient({endpoint: process.env.ENDPOINT})
So, the take here is that whenever you have a conflict, exclude those options from Global Configuration.
Check this How to set multiple aws credentials in nodejs aws-sdk module?
When I am trying to load AWS credentials in my project it gives back an error.
When using credentials in plain text everything works good but when I am trying to use environment variables it's not working.
Error message. :
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
Here is my tried code :
const AWS = require('aws-sdk');
const SESConfig = {
apiVersion: "2010-12-01",
accessKeyId: process.env.AWS_SECRET_KEY,
accessSecretKey: process.env.AWS_SECRET_KEY,
region: "us-east-1"
}
AWS.config.update(SESConfig);
var sns = new AWS.SNS()
var sns = new AWS.SNS();
function sendSMS(to_number, message, cb) {
sns.publish({
Message: message,
Subject: 'Admin',
PhoneNumber:to_number
}, cb);
}
// Example
const PhoneNumberArray = ['any mobile number']
PhoneNumberArray.forEach(number => {
sendSMS(number, "Lorem Ipsum is simply dummy text of the printing and typesetting industry.", (err, result)=>{
console.log("RESULTS: ",err,result)
})
})
By default, the SDK detects AWS credentials set in your environment and uses them to sign requests to AWS. That way you don’t need to manage credentials in your applications.
Unix:
$ export AWS_ACCESS_KEY_ID="your_key_id"
$ export AWS_SECRET_ACCESS_KEY="your_secret_key"
Windows:
SET AWS_ACCESS_KEY_ID="your_key_id"
SET AWS_SECRET_ACCESS_KEY="your_secret_key"
Powershell:
$Env:AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
$Env:AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
you can also add $ export AWS_SESSION_TOKEN='your_token' (optional)
See aws-sdk for more details.
Otherwise you can create a ~/.aws/credentials file and add:
[default]
aws_access_key_id = <YOUR_ACCESS_KEY_ID>
aws_secret_access_key = <YOUR_SECRET_ACCESS_KEY>
See aws for more details.
I noticed that you are setting your accessKeyId and secretAccessKey to the same environment variable.
const SESConfig = {
apiVersion: "2010-12-01",
accessKeyId: process.env.AWS_SECRET_KEY, // should be: process.env.AWS_ACCESS_ID
secretAccessKey: process.env.AWS_SECRET_KEY,
region: "us-east-1"
}
These are supplied as separate values by aws and should be represented by two separate environment variables.
Maybe this is your issue?
You can try create an AWS_PROFILE with the credentials if you have the AWS CLI installed.
$ aws configure --profile testuser
AWS Access Key ID [None]: 1234
AWS Secret Access Key [None]: 1234
Default region name [None]: us-east-1
Default output format [None]: text
After that you can set the AWS_PROFILE as environment variable.
Linux / Mac
export AWS_PROFILE=testuser
Windows
setx AWS_PROFILE testuser
After that you should be able to run your program and AWS will get the credentials from your profile. This way you don't have to save your credentials in .ENV. If you do, remember to add it in .gitignore.
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html
Install dotenv
npm install dotenv --save
Create a .env file and add your Variables
AWS_ACCESS_KEY=1234567890
AWS_SECRET_KEY=XXXXXXXXXXXXXXXXXXX
Load dotenv in your project
require('dotenv').config();
Complete code
require('dotenv').config();
const AWS = require('aws-sdk');
const SESConfig = {
apiVersion: "2010-12-01",
accessKeyId: process.env.AWS_ACCESS_KEY,
accessSecretKey: process.env.AWS_SECRET_KEY,
region: "us-east-1"
}
AWS.config.update(SESConfig);
var sns = new AWS.SNS();
function sendSMS(to_number, message, cb) {
sns.publish({
Message: message,
Subject: 'Admin',
PhoneNumber:to_number
}, cb);
}
const PhoneNumberArray = ['any mobile number']
PhoneNumberArray.forEach(number => {
sendSMS(number, "Lorem Ipsum is simply dummy text of the printing and typesetting industry.", (err, result)=>{
console.log("RESULTS: ",err,result)
})
})
I was able to fix this problem by specifying an apiVersion
AWS.config.update({
region: 'MY_REGION',
apiVersion: 'latest',
credentials: {
accessKeyId: 'MY_ACCESS_KEY',
secretAccessKey: 'MY_SECRET_KEY'
}
})
worked after i followed the exact names from aws guide for the env vars
https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-environment.html
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN (Optional)
Note that the variable names in ~/.aws/credentials are case sensitive. That was what caused my problem
You can simply load the credentials through a dedicated config.json file.
{
"accessKeyId": "<YOUR_ACCESS_KEY_ID>",
"secretAccessKey": "<YOUR_SECRET_ACCESS_KEY>",
"region": "eu-west-3"
}
Then use the AWS load command
AWS.config.loadFromPath('./config.json');
In this case you wouldn't need to update the AWS config AWS.config.update(...); as it is done right from the gecko.
Note that:
Loading credentials from a JSON document is not supported in browser scripts.
Source # https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-json-file.html
I have stored all the credentials in my config file itself. For windows, I got it solved by adding a Environment Variable to my nodejs application in .env.local
AWS_SDK_LOAD_CONFIG=1
I came across a similar problem, so I watched a few videos and read a bunch of documentation, In dotenv file try creating the IAM user that you wish to give permission to access the accountAWS_PROFILE="exampleProfile" this should be the same user that you got your Access key and secret from, then require so it should look something like this.
const SESConfig = {
apiVersion: "2010-12-01",
profile:process.env.AWS_PROFILE,
accessKeyId: process.env.AWS_ACCESS_KEY,
accessSecretKey: process.env.AWS_SECRET_KEY,
region: "us-east-1"
}
I switched to a prod role according to this
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-cli.html#switch-role-cli-scenario-prod-env
where there wasn't a 'prod' entry in my ~/.aws/credentials file
I got my sdk calls in my script working by calling
export AWS_SDK_LOAD_CONFIG=1
before runningg it
I encountered the same issue but in my case, I was forced to authenticate through GSuite. That's because, in my work environment, GSuite (from Google) is the Single Sign-On (SSO) provider.
I noticed that while a CLI command like:
aws s3 ls
worked as expected, the node.js code threw the error discussed in this article.
There are two solutions that work in my case:
Add the relevant lines into the code from the sample below:
const AWS = require('aws-sdk');
const s3 = new AWS.S3();
const credentials = new AWS.SharedIniFileCredentials({ profile: '<your_profile_name>' });
AWS.config.credentials = credentials;
AWS.config.region = '<your_region>';
const s3 = new AWS.S3({ region: '<your_region>' });
(async () => {
await s3.putObject({
Body: 'Hello World',
Bucket: "<your_bucket_name>",
Key: "my-file.txt"
}, function (err, data){
if (err) console.log(err, err.stack);
else console.log(data);
});
})()
The second solution that also worked was using the proper environment variable.
On my macOS, I had set the environment variable incorrectly as:
AWS_DEFAULT_PROFILE=<your_profile>
But when I set the below environment variable, my code worked like a charm:
AWS_PROFILE=<your_profile>
Refer to this article by AWS on environment variables:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
Hope my solution helps.
Different situation I was in, but that was leading to the same error.
I was using a snippet from the example code for a Pinpoint Push Notification Lambda, and it included these lines:
// Specify that you're using a shared credentials file, and specify the
// IAM profile to use.
var credentials = new AWS.SharedIniFileCredentials({ profile: '...' });
AWS.config.credentials = credentials;
I was using this code in my own Amplify CLI generated PushNotification Function. There were no issues when working with the Function on its own.
When I tried to call the PushNotification Function from another resource, I got that same error:
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
The solution for me was to simply remove the SharedIniFileCredentials code from the function entirely.
I presume this works because the Amplify environment is all managed, so that explicit AWS.config.credentials was redundant, as well as broken when running in certain scenarios.
Hope this helps anybody who is having a similar problem with calling Functions from other Function in an Amplify project! – I know that's not best practice, as discussed in this other stack overflow, but it works
I need to use multiple AWS credentials for different services like s3, SNS....
var awsS3 = require('aws-sdk');
var awsSes = require('aws-sdk');
awsS3.config.update({
region: config.awsRegion,
accessKeyId: config.sesAccessKeyId,
secretAccessKey: config.sesSecretAccessKey
});
awsSes.config.update({
region: config.s3Region,
accessKeyId: config.s3AccessKeyId,
secretAccessKey: config.s3SecretAccessKey
});
But above code is not working.
How to configure multiple accessKeyIds, secretAccessKeys for different services?
You can pass config while creating service objects. Following is what you are looking for
const s3 = new aws.S3({ /* s3 config */ });
const ses = new aws.SES({ /* ses config */ });
I would think you want to control it with policies instead of having multiple credentials. Use a single credentials/role, then custom policies on what you want to allows and deny for each service. Then your application can use that role/credentials and will be allows or restricted base on the policies.
I am running into issues when generating a signed URL for a public S3 bucket. I get the issue when doing a PUT request:
<Error><Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
A bit of information - I am using:
node version 5.8
aws-sdkversion 2.7.10
I use the aws-sdk like this:
AWS.config.update({
accessKeyId: ACCESS_KEY,
secretAccessKey: SECRET_ACCESS_KEY,
region: 'eu-west-1'
})
const s3 = new AWS.S3()
I generate the URL this way:
const params = {
Key: FILE_KEY,
Bucket: BUCKET_NAME,
ContentType: image/jpeg,
Expires: 60,
ACL: 'public-read',
Metadata: {
'Cache-Control': 'max-age=31556926'
}
}
const signedUrl = s3.getSignedUrl('putObject', params)
The generated URL looks like this:
https://companyxyz.s3-eu-west-1.amazonaws.com/
image/5843df4a15c6fccf4501cab9.jpg?
AWSAccessKeyId=xxxxxxxxxx&
Content-Type=image%2Fjpeg&
Expires=1480843142&
Signature=YvUEGntDLVUUuyVuDMxF5yXXBnI%3D
&x-amz-acl=public-read&
x-amz-meta-cache-control=max-age%3D31556926
It could be related to sig v2 and sig v4
From the documentation here:
http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
and here
http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
notice that in sig v4,
https://s3.amazonaws.com/examplebucket/test.txt
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=<your-access-key-id>/20130721/us-east-1/s3/aws4_request
your access-key-id is part of X-Amz-Credential
while for sig v2
https://elasticmapreduce.amazonaws.com?
&AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE
AWSAccessKeyId has its own paramater.
Your example shows that you are using sig v2
http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
also mentions that some regions do not support sig v2
EU (Frankfurt) Region
EU (London) Region
EU (Frankfurt) is eu-central-1
which is strange, because it sig v2 should not work in eu-central-1.
I resolved the issue by changing region. I created a new bucket in eu-central-1 and everything worked. No matter what, I was not able to generate a working signed URL for eu-west-1.
Would love to hear any insights.