I need to use multiple AWS credentials for different services like s3, SNS....
var awsS3 = require('aws-sdk');
var awsSes = require('aws-sdk');
awsS3.config.update({
region: config.awsRegion,
accessKeyId: config.sesAccessKeyId,
secretAccessKey: config.sesSecretAccessKey
});
awsSes.config.update({
region: config.s3Region,
accessKeyId: config.s3AccessKeyId,
secretAccessKey: config.s3SecretAccessKey
});
But above code is not working.
How to configure multiple accessKeyIds, secretAccessKeys for different services?
You can pass config while creating service objects. Following is what you are looking for
const s3 = new aws.S3({ /* s3 config */ });
const ses = new aws.SES({ /* ses config */ });
I would think you want to control it with policies instead of having multiple credentials. Use a single credentials/role, then custom policies on what you want to allows and deny for each service. Then your application can use that role/credentials and will be allows or restricted base on the policies.
Related
I have nodejs/express app from which I want to connect to AWS S3.
I do have a temporary approach to make connection,
environment file
aws_access_key_id=XXX
aws_secret_access_key=XXXX/
aws_session_token=xxxxxxxxxxxxxxxxxxxxxxxxxx
S3-connection-service.js
const AWS = require("aws-sdk");
AWS.config.update({
accessKeyId: `${process.env.aws_access_key_id}`,
secretAccessKey: `${process.env.aws_secret_access_key}`,
sessionToken: `${process.env.aws_session_token}`,
region: `${process.env.LOCAL_AWS_REGION}`
});
const S3 = new AWS.S3();
module.exports = {
listBucketContent: (filePath) =>
new Promise((resolve, reject) => {
const params = { Bucket: bucketName, Prefix: filePath };
S3.listObjects(params, (err, objects) => {
if (err) {
reject(err);
} else {
resolve(objects);
}
});
}),
....
....
}
controller.js
const fetchFile = require("../../S3-connection-service.js");
const AWSFolder = await fetchFile.listBucketContent(filePath);
Fine it's works and I'm able to access S3 bucket files and play with it.
PROBLEM
The problem is connection is not persistent. Since, I use session_token, connection remains alive for sometime and again after sometime new tokens will be generated, I have to copy-paste them in env file and re-run the node app.
I really have no idea how can I make connection persistent ?
Where to store AWS confidential/secrets and how to use them to connect to S3 so connection remains alive ?
Just remove
AWS.config.update({
accessKeyId: `${process.env.aws_access_key_id}`,
secretAccessKey: `${process.env.aws_secret_access_key}`,
sessionToken: `${process.env.aws_session_token}`,
region: `${process.env.LOCAL_AWS_REGION}`
});
code block from lambda source in file S3-connection-service.js
Attach a role to lambda function with proper permissions. You will have same functionally.
For local development.
You can set environment variable before testing your application.
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-west-2
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
If you are using any IDE you can set these environment variables on it.
If you are testing from cli
$ AWS_ACCESS_KEY_ID=EXAMPLE AWS_SECRET_ACCESS_KEY=EXAMPLEKEY AWS_DEFAULT_REGION=us-west-2 npm start
connect to S3 so connection remains alive ?
You can't make one request to S3 and keep it alive forever.
These are your options:
Add a try/catch statement inside your code to handle credentials expired error. Then, generate new credentials and re-initialize the S3 client.
Instead of using a Role, use a User. (IAM Identities). User credentials can be valid forever. You won't need to update the credentials in this case.
Do not provide the credentials to AWS.config.update like you are doing right now. If you don't provide the credentials, the AWS client will try to read them from your ~/.aws/credentials file automatically. If you create a script to update them every hour (ex: a cronjob), then your credentials will be up-to-date at all times.
The problem is when I feed Dynamodb config endpoint some value the AWS Key Management Service stops working altogether.
1. DynamoDB
const awsConfig = {
region: process.env.REGION,
endpoint: process.env.ENDPOINT, //this stops AWS KMS
accessKeyId: process.env.ACCESS_KEY_ID,
secretAccessKey: process.env.ACCESS_KEY,
};
aws.config.update(awsConfig);
2. AWS KMS
constructor() {
this.#KEYAWS = keyAws;
this.#region = process.env.REGION;
this.#secretName = process.env.SECRET_NAME;
this.#secret = process.env.ACCESS_KEY;
this.#_AWS_KEY_ACCESS_KEY_ID = process.env.ACCESS_KEY_ID;
}
async #getPrivateKey() {
this.#KEYAWS.config.update({
accessKeyId: this.#_AWS_KEY_ACCESS_KEY_ID,
secretAccessKey: this.#secret,
});
var client = new this.#KEYAWS.SecretsManager({
region: this.#region,
});
}
When I comment out the endpoint in Dynamo config, the KMS works properly
Note: A VPC endpoint for DynamoDB enables Amazon EC2 instances in your VPC to use their private IP addresses to access DynamoDB with no exposure to the public internet.
This happened because of adding endpoint: process.env.ENDPOINT to global configuration Object. AWS SDK Global Configuration
I removed the endpoint option from global configuration and specifically passed it to dynamodb service aws.DynamoDB.DocumentClient({endpoint: process.env.ENDPOINT})
So, the take here is that whenever you have a conflict, exclude those options from Global Configuration.
Check this How to set multiple aws credentials in nodejs aws-sdk module?
I have been using AWS SDK v2.846.0 for create a SSL certificate for diferent domains, but when I do a request the I get this error:
"Inaccessible host: acm.undefined.amazonaws.com'. This service may not be available in the us-east-1' region."
Someone know what I could do?
I am using this documentation: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/ACM.html#requestCertificate-property
It looks like the region is undefined in your ACM credentials object.
To supply options (including access key, secret key, and region), you can either pass them into the service constructor e.g.
options = { secretAccessKey: skid, accessKeyId: akid, region: 'us-east-2' };
acm = new AWS.ACM(options);
Or you can update the SDK's config object but you must do this before you create your service object, for example:
options = { secretAccessKey: skid, accessKeyId: akid, region: 'us-east-2' };
AWS.config.update(options);
acm = new AWS.ACM();
When I am trying to load AWS credentials in my project it gives back an error.
When using credentials in plain text everything works good but when I am trying to use environment variables it's not working.
Error message. :
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
Here is my tried code :
const AWS = require('aws-sdk');
const SESConfig = {
apiVersion: "2010-12-01",
accessKeyId: process.env.AWS_SECRET_KEY,
accessSecretKey: process.env.AWS_SECRET_KEY,
region: "us-east-1"
}
AWS.config.update(SESConfig);
var sns = new AWS.SNS()
var sns = new AWS.SNS();
function sendSMS(to_number, message, cb) {
sns.publish({
Message: message,
Subject: 'Admin',
PhoneNumber:to_number
}, cb);
}
// Example
const PhoneNumberArray = ['any mobile number']
PhoneNumberArray.forEach(number => {
sendSMS(number, "Lorem Ipsum is simply dummy text of the printing and typesetting industry.", (err, result)=>{
console.log("RESULTS: ",err,result)
})
})
By default, the SDK detects AWS credentials set in your environment and uses them to sign requests to AWS. That way you don’t need to manage credentials in your applications.
Unix:
$ export AWS_ACCESS_KEY_ID="your_key_id"
$ export AWS_SECRET_ACCESS_KEY="your_secret_key"
Windows:
SET AWS_ACCESS_KEY_ID="your_key_id"
SET AWS_SECRET_ACCESS_KEY="your_secret_key"
Powershell:
$Env:AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
$Env:AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
you can also add $ export AWS_SESSION_TOKEN='your_token' (optional)
See aws-sdk for more details.
Otherwise you can create a ~/.aws/credentials file and add:
[default]
aws_access_key_id = <YOUR_ACCESS_KEY_ID>
aws_secret_access_key = <YOUR_SECRET_ACCESS_KEY>
See aws for more details.
I noticed that you are setting your accessKeyId and secretAccessKey to the same environment variable.
const SESConfig = {
apiVersion: "2010-12-01",
accessKeyId: process.env.AWS_SECRET_KEY, // should be: process.env.AWS_ACCESS_ID
secretAccessKey: process.env.AWS_SECRET_KEY,
region: "us-east-1"
}
These are supplied as separate values by aws and should be represented by two separate environment variables.
Maybe this is your issue?
You can try create an AWS_PROFILE with the credentials if you have the AWS CLI installed.
$ aws configure --profile testuser
AWS Access Key ID [None]: 1234
AWS Secret Access Key [None]: 1234
Default region name [None]: us-east-1
Default output format [None]: text
After that you can set the AWS_PROFILE as environment variable.
Linux / Mac
export AWS_PROFILE=testuser
Windows
setx AWS_PROFILE testuser
After that you should be able to run your program and AWS will get the credentials from your profile. This way you don't have to save your credentials in .ENV. If you do, remember to add it in .gitignore.
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html
Install dotenv
npm install dotenv --save
Create a .env file and add your Variables
AWS_ACCESS_KEY=1234567890
AWS_SECRET_KEY=XXXXXXXXXXXXXXXXXXX
Load dotenv in your project
require('dotenv').config();
Complete code
require('dotenv').config();
const AWS = require('aws-sdk');
const SESConfig = {
apiVersion: "2010-12-01",
accessKeyId: process.env.AWS_ACCESS_KEY,
accessSecretKey: process.env.AWS_SECRET_KEY,
region: "us-east-1"
}
AWS.config.update(SESConfig);
var sns = new AWS.SNS();
function sendSMS(to_number, message, cb) {
sns.publish({
Message: message,
Subject: 'Admin',
PhoneNumber:to_number
}, cb);
}
const PhoneNumberArray = ['any mobile number']
PhoneNumberArray.forEach(number => {
sendSMS(number, "Lorem Ipsum is simply dummy text of the printing and typesetting industry.", (err, result)=>{
console.log("RESULTS: ",err,result)
})
})
I was able to fix this problem by specifying an apiVersion
AWS.config.update({
region: 'MY_REGION',
apiVersion: 'latest',
credentials: {
accessKeyId: 'MY_ACCESS_KEY',
secretAccessKey: 'MY_SECRET_KEY'
}
})
worked after i followed the exact names from aws guide for the env vars
https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-environment.html
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN (Optional)
Note that the variable names in ~/.aws/credentials are case sensitive. That was what caused my problem
You can simply load the credentials through a dedicated config.json file.
{
"accessKeyId": "<YOUR_ACCESS_KEY_ID>",
"secretAccessKey": "<YOUR_SECRET_ACCESS_KEY>",
"region": "eu-west-3"
}
Then use the AWS load command
AWS.config.loadFromPath('./config.json');
In this case you wouldn't need to update the AWS config AWS.config.update(...); as it is done right from the gecko.
Note that:
Loading credentials from a JSON document is not supported in browser scripts.
Source # https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-json-file.html
I have stored all the credentials in my config file itself. For windows, I got it solved by adding a Environment Variable to my nodejs application in .env.local
AWS_SDK_LOAD_CONFIG=1
I came across a similar problem, so I watched a few videos and read a bunch of documentation, In dotenv file try creating the IAM user that you wish to give permission to access the accountAWS_PROFILE="exampleProfile" this should be the same user that you got your Access key and secret from, then require so it should look something like this.
const SESConfig = {
apiVersion: "2010-12-01",
profile:process.env.AWS_PROFILE,
accessKeyId: process.env.AWS_ACCESS_KEY,
accessSecretKey: process.env.AWS_SECRET_KEY,
region: "us-east-1"
}
I switched to a prod role according to this
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-cli.html#switch-role-cli-scenario-prod-env
where there wasn't a 'prod' entry in my ~/.aws/credentials file
I got my sdk calls in my script working by calling
export AWS_SDK_LOAD_CONFIG=1
before runningg it
I encountered the same issue but in my case, I was forced to authenticate through GSuite. That's because, in my work environment, GSuite (from Google) is the Single Sign-On (SSO) provider.
I noticed that while a CLI command like:
aws s3 ls
worked as expected, the node.js code threw the error discussed in this article.
There are two solutions that work in my case:
Add the relevant lines into the code from the sample below:
const AWS = require('aws-sdk');
const s3 = new AWS.S3();
const credentials = new AWS.SharedIniFileCredentials({ profile: '<your_profile_name>' });
AWS.config.credentials = credentials;
AWS.config.region = '<your_region>';
const s3 = new AWS.S3({ region: '<your_region>' });
(async () => {
await s3.putObject({
Body: 'Hello World',
Bucket: "<your_bucket_name>",
Key: "my-file.txt"
}, function (err, data){
if (err) console.log(err, err.stack);
else console.log(data);
});
})()
The second solution that also worked was using the proper environment variable.
On my macOS, I had set the environment variable incorrectly as:
AWS_DEFAULT_PROFILE=<your_profile>
But when I set the below environment variable, my code worked like a charm:
AWS_PROFILE=<your_profile>
Refer to this article by AWS on environment variables:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
Hope my solution helps.
Different situation I was in, but that was leading to the same error.
I was using a snippet from the example code for a Pinpoint Push Notification Lambda, and it included these lines:
// Specify that you're using a shared credentials file, and specify the
// IAM profile to use.
var credentials = new AWS.SharedIniFileCredentials({ profile: '...' });
AWS.config.credentials = credentials;
I was using this code in my own Amplify CLI generated PushNotification Function. There were no issues when working with the Function on its own.
When I tried to call the PushNotification Function from another resource, I got that same error:
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
The solution for me was to simply remove the SharedIniFileCredentials code from the function entirely.
I presume this works because the Amplify environment is all managed, so that explicit AWS.config.credentials was redundant, as well as broken when running in certain scenarios.
Hope this helps anybody who is having a similar problem with calling Functions from other Function in an Amplify project! – I know that's not best practice, as discussed in this other stack overflow, but it works
< premise>
I'm new cloud computing in general, AWS specifically, and REST API, and am trying to cobble together a "big-picture" understanding.
I am working with LocalStack - which, by my understanding, simulates the real AWS by responding identically to (a subset of) the AWS API if you specify the endpoint address/port that LocalStack listens at.
Lastly, I've been working from this tutorial: https://dev.to/goodidea/how-to-fake-aws-locally-with-localstack-27me
< /premise>
Using the noted tutorial, and per its guidance, I successfully creating a S3 bucket using the AWS CLI.
To demonstrate uploading a local file to the S3 bucket, though, the tutorial switches to node.js, which I think demonstrates the AWS node.js SDK:
# aws.js
# This code segment comes from https://dev.to/goodidea/how-to-fake-aws-locally-with-localstack-27me
#
const AWS = require('aws-sdk')
require('dotenv').config()
const credentials = {
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_KEY,
}
const useLocal = process.env.NODE_ENV !== 'production'
const bucketName = process.env.AWS_BUCKET_NAME
const s3client = new AWS.S3({
credentials,
/**
* When working locally, we'll use the Localstack endpoints. This is the one for S3.
* A full list of endpoints for each service can be found in the Localstack docs.
*/
endpoint: useLocal ? 'http://localhost:4572' : undefined,
/**
* Including this option gets localstack to more closely match the defaults for
* live S3. If you omit this, you will need to add the bucketName to the `Key`
* property in the upload function below.
*
* see: https://github.com/localstack/localstack/issues/1180
*/
s3ForcePathStyle: true,
})
const uploadFile = async (data, fileName) =>
new Promise((resolve) => {
s3client.upload(
{
Bucket: bucketName,
Key: fileName,
Body: data,
},
(err, response) => {
if (err) throw err
resolve(response)
},
)
})
module.exports = uploadFile
.
# test-upload.js
# This code segment comes from https://dev.to/goodidea/how-to-fake-aws-locally-with-localstack-27me
#
const fs = require('fs')
const path = require('path')
const uploadFile = require('./aws')
const testUpload = () => {
const filePath = path.resolve(__dirname, 'test-image.jpg')
const fileStream = fs.createReadStream(filePath)
const now = new Date()
const fileName = `test-image-${now.toISOString()}.jpg`
uploadFile(fileStream, fileName).then((response) => {
console.log(":)")
console.log(response)
}).catch((err) => {
console.log(":|")
console.log(err)
})
}
testUpload()
Invocation :
$ node test-upload.js
:)
{ ETag: '"c6b9e5b1863cd01d3962c9385a9281d"',
Location: 'http://demo-bucket.localhost:4572/demo-bucket/test-image-2019-03-11T21%3A22%3A43.511Z.jpg',
key: 'demo-bucket/test-image-2019-03-11T21:22:43.511Z.jpg',
Key: 'demo-bucket/test-image-2019-03-11T21:22:43.511Z.jpg',
Bucket: 'demo-bucket' }
I do not have prior experience with node.js, but my understanding of the above code is that it uses the AWS.S3.upload() AWS node.js SDK method to copy a local file to a S3 bucket, and prints the HTTP response (is that correct?).
Question: I observe that the HTTP response includes a "Location" key whose value looks like a URL I can copy/paste into a browser to view the image directly from the S3 bucket; is there a way to get this location using the AWS CLI?
Am I correct to assume that AWS CLI commands are analogues of the AWS SDK?
I tried uploading a file to my S3 bucket using the aws s3 cp CLI command, which I thought would be analogous to the AWS.S3.upload() method above, but it didn't generate any output, and I'm not sure what I should have done - or should do - to get a Location the way the HTTP response to the AWS.S3.upload() AWS node SDK method did.
$ aws --endpoint-url=http://localhost:4572 s3 cp ./myFile.json s3://myBucket/myFile.json
upload: ./myFile.json to s3://myBucket/myFile.json
Update: continued study makes me now wonder whether it is implicit that a file uploaded to a S3 bucket by any means - whether by CLI command aws s3 cp or node.js SDK method AWS.S3.upload(), etc. - can be accessed at http://<bucket_name>.<endpoint_without_http_prefix>/<bucket_name>/<key> ? E.g. http://myBucket.localhost:4572/myBucket/myFile.json?
If this is implicit, I suppose you could argue it's unnecessary to ever be given the "Location" as in that example node.js HTTP response.
Grateful for guidance - I hope it's obvious how painfully under-educated I am on all the involved technologies.
Update 2: It looks like the correct url is <endpoint>/<bucket_name>/<key>, e.g. http://localhost:4572/myBucket/myFile.json.
AWS CLI and the different SDKs offer similar functionality but some add extra features and some format the data differently. It's safe to assume that you can do what the CLI does with the SDK and vice-versa. You might just have to work for it a little bit sometimes.
As you said in your update, not every file that is uploaded to S3 is publicly available. Buckets have policies and files have permissions. Files are only publicly available if the policies and permissions allow it.
If the file is public then you can just construct the URL as you described. If you have the bucket setup for website hosting, you can also use the domain you setup.
But if the file is not public or you just want a temporary URL, you can use aws presign s3://myBucket/myFile.json. This will give you a URL that can be used by anyone to download the file with the permissions of whoever executed the command. The URL will be valid for one hour unless you choose a different time with --expires-in. The SDK has similar functionality as well but you have to work a tiny bit harder to use it.
Note: Starting with version 0.11.0, all APIs are exposed via a single edge service, which is accessible on http://localhost:4566 by default.
Considering that you've added some files to your bucket
aws --endpoint-url http://localhost:4566 s3api list-objects-v2 --bucket mybucket
{
"Contents": [
{
"Key": "blog-logo.png",
"LastModified": "2020-12-28T12:47:04.000Z",
"ETag": "\"136f0e6acf81d2d836043930827d1cc0\"",
"Size": 37774,
"StorageClass": "STANDARD"
}
]
}
you should be able to access your file with
http://localhost:4566/mybucket/blog-logo.png