I am generating signed urls on my webapp (nodejs) using the knox nodejs-library.
However the issue arises, that for every request, I need to regenerate an unique GET signed url for the current user, leaving browser's cache-control out of the game.
I've searched the web without success as browsers seem to use the full url as caching key so I am really curious how I can, under the given circumstances (nodejs, knox library) get the issue solved and use caching control while still being able to generated signed urls for each and every request as I need to verify the user's access rights.
I cannot believe there's no solution to that though.
I am working with Java AmazonS3 client, but the process should be the same.
There is a strategy that can be used to handle this situation.
You could use a fixed date time as an expiration date. I set this date to tomorrow at 12 pm.
Now every time you generate a url, it will be the same throughout that day until 00:00. That way browser caching can be used to some extent.
Expanding #semir-deljić Answer.
Every time we call getSignedUrl function, it will generate new URLs. This will result in images not being cached even if Cache Control header is present.
Thus, we are using timekeeper library to freeze time. Now when the function is called, it thinks that the time has not passed, and it returns same URL.
const moment = require('moment');
const tk = require("timekeeper");
function url4download(awsPath, awsKey) {
function getFrozenDate() {
return moment().startOf('week').toDate();
}
// Paramters for getSignedUrl function
const params = {
// Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
// Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
Bucket: awsBucket,
Key: `${awsPath}/${awsKey}`,
// 604800 == 7 days
ResponseCacheControl: `public, max-age=604800, immutable`,
Expires: 604800, // 7 days is max
};
const url = tk.withFreeze(getFrozenDate(), () => {
return S3.getSignedUrl('getObject', params);
});
return url;
}
Note:
Using moment().toDate(), as the timekeeper requires a Native Date Object.
Even tough the question is for using knox library, my answer uses aws official library.
// This is how the AWS & S3 is initiliased.
const AWS = require('aws-sdk');
const S3 = new AWS.S3({
accessKeyId: awsAccessId,
secretAccessKey: awsSecretKey,
region: 'ap-south-1',
apiVersion: '2006-03-01',
signatureVersion: 'v4',
});
Inspiration: https://advancedweb.hu/cacheable-s3-signed-urls/
If you use CloudFront with S3, you can use a Custom Policy, if you restrict each url to the user's IP and a reasonably long timeout, it means that when they request the same content again, they will get the same URL and hence their browser can cache the content but the URL will not work for someone else (on a different IP).
(see: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-creating-signed-url-custom-policy.html)
When calculating signed URL you can set 'signingDate' to a fixed moment in the past e.g. yesterday morning, then calculate expiration from that moment.
Don't forget to use UTC and account for timezones.
import { S3Client, GetObjectCommand, GetObjectCommandInput } from "#aws-sdk/client-s3";
import { getSignedUrl } from "#aws-sdk/s3-request-presigner";
let signingDate = new Date();
signingDate.setUTCHours(0, 0, 0, 0);
signingDate.setUTCDate(signingDate.getUTCDate() - 1);
let params: GetObjectCommandInput = {
Bucket: BUCKET_NAME,
Key: filename
};
const command = new GetObjectCommand(params);
const url = await getSignedUrl(s3Client,
command, {
expiresIn: 3 * 3600 * 24, // 1 day until today + 1 expiration + 1 days for timezones
signableHeaders: new Set < string > (),
signingDate: signingDate
});
Related
< premise>
I'm new cloud computing in general, AWS specifically, and REST API, and am trying to cobble together a "big-picture" understanding.
I am working with LocalStack - which, by my understanding, simulates the real AWS by responding identically to (a subset of) the AWS API if you specify the endpoint address/port that LocalStack listens at.
Lastly, I've been working from this tutorial: https://dev.to/goodidea/how-to-fake-aws-locally-with-localstack-27me
< /premise>
Using the noted tutorial, and per its guidance, I successfully creating a S3 bucket using the AWS CLI.
To demonstrate uploading a local file to the S3 bucket, though, the tutorial switches to node.js, which I think demonstrates the AWS node.js SDK:
# aws.js
# This code segment comes from https://dev.to/goodidea/how-to-fake-aws-locally-with-localstack-27me
#
const AWS = require('aws-sdk')
require('dotenv').config()
const credentials = {
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_KEY,
}
const useLocal = process.env.NODE_ENV !== 'production'
const bucketName = process.env.AWS_BUCKET_NAME
const s3client = new AWS.S3({
credentials,
/**
* When working locally, we'll use the Localstack endpoints. This is the one for S3.
* A full list of endpoints for each service can be found in the Localstack docs.
*/
endpoint: useLocal ? 'http://localhost:4572' : undefined,
/**
* Including this option gets localstack to more closely match the defaults for
* live S3. If you omit this, you will need to add the bucketName to the `Key`
* property in the upload function below.
*
* see: https://github.com/localstack/localstack/issues/1180
*/
s3ForcePathStyle: true,
})
const uploadFile = async (data, fileName) =>
new Promise((resolve) => {
s3client.upload(
{
Bucket: bucketName,
Key: fileName,
Body: data,
},
(err, response) => {
if (err) throw err
resolve(response)
},
)
})
module.exports = uploadFile
.
# test-upload.js
# This code segment comes from https://dev.to/goodidea/how-to-fake-aws-locally-with-localstack-27me
#
const fs = require('fs')
const path = require('path')
const uploadFile = require('./aws')
const testUpload = () => {
const filePath = path.resolve(__dirname, 'test-image.jpg')
const fileStream = fs.createReadStream(filePath)
const now = new Date()
const fileName = `test-image-${now.toISOString()}.jpg`
uploadFile(fileStream, fileName).then((response) => {
console.log(":)")
console.log(response)
}).catch((err) => {
console.log(":|")
console.log(err)
})
}
testUpload()
Invocation :
$ node test-upload.js
:)
{ ETag: '"c6b9e5b1863cd01d3962c9385a9281d"',
Location: 'http://demo-bucket.localhost:4572/demo-bucket/test-image-2019-03-11T21%3A22%3A43.511Z.jpg',
key: 'demo-bucket/test-image-2019-03-11T21:22:43.511Z.jpg',
Key: 'demo-bucket/test-image-2019-03-11T21:22:43.511Z.jpg',
Bucket: 'demo-bucket' }
I do not have prior experience with node.js, but my understanding of the above code is that it uses the AWS.S3.upload() AWS node.js SDK method to copy a local file to a S3 bucket, and prints the HTTP response (is that correct?).
Question: I observe that the HTTP response includes a "Location" key whose value looks like a URL I can copy/paste into a browser to view the image directly from the S3 bucket; is there a way to get this location using the AWS CLI?
Am I correct to assume that AWS CLI commands are analogues of the AWS SDK?
I tried uploading a file to my S3 bucket using the aws s3 cp CLI command, which I thought would be analogous to the AWS.S3.upload() method above, but it didn't generate any output, and I'm not sure what I should have done - or should do - to get a Location the way the HTTP response to the AWS.S3.upload() AWS node SDK method did.
$ aws --endpoint-url=http://localhost:4572 s3 cp ./myFile.json s3://myBucket/myFile.json
upload: ./myFile.json to s3://myBucket/myFile.json
Update: continued study makes me now wonder whether it is implicit that a file uploaded to a S3 bucket by any means - whether by CLI command aws s3 cp or node.js SDK method AWS.S3.upload(), etc. - can be accessed at http://<bucket_name>.<endpoint_without_http_prefix>/<bucket_name>/<key> ? E.g. http://myBucket.localhost:4572/myBucket/myFile.json?
If this is implicit, I suppose you could argue it's unnecessary to ever be given the "Location" as in that example node.js HTTP response.
Grateful for guidance - I hope it's obvious how painfully under-educated I am on all the involved technologies.
Update 2: It looks like the correct url is <endpoint>/<bucket_name>/<key>, e.g. http://localhost:4572/myBucket/myFile.json.
AWS CLI and the different SDKs offer similar functionality but some add extra features and some format the data differently. It's safe to assume that you can do what the CLI does with the SDK and vice-versa. You might just have to work for it a little bit sometimes.
As you said in your update, not every file that is uploaded to S3 is publicly available. Buckets have policies and files have permissions. Files are only publicly available if the policies and permissions allow it.
If the file is public then you can just construct the URL as you described. If you have the bucket setup for website hosting, you can also use the domain you setup.
But if the file is not public or you just want a temporary URL, you can use aws presign s3://myBucket/myFile.json. This will give you a URL that can be used by anyone to download the file with the permissions of whoever executed the command. The URL will be valid for one hour unless you choose a different time with --expires-in. The SDK has similar functionality as well but you have to work a tiny bit harder to use it.
Note: Starting with version 0.11.0, all APIs are exposed via a single edge service, which is accessible on http://localhost:4566 by default.
Considering that you've added some files to your bucket
aws --endpoint-url http://localhost:4566 s3api list-objects-v2 --bucket mybucket
{
"Contents": [
{
"Key": "blog-logo.png",
"LastModified": "2020-12-28T12:47:04.000Z",
"ETag": "\"136f0e6acf81d2d836043930827d1cc0\"",
"Size": 37774,
"StorageClass": "STANDARD"
}
]
}
you should be able to access your file with
http://localhost:4566/mybucket/blog-logo.png
I created a bucket on s3 and added a HTML file, after this I created a Cloud Front key pair using my root user and added a Cloud Front Distribution for that bucket. Tried to access the object using that distribution and it worked, than I restricted access to the bucket using Behaviour and selected "self".
Finally I generated a signed url from node js and tested it using Postman.
The problem is that I always get AccessDenied.
<Error>
<Code>AccessDenied</Code>
<Message>Access denied</Message>
</Error>
Here is my code.
const cfsign = require("aws-cloudfront-sign");
var signingParams = {
keypairId: process.env.PUBLIC_KEY,
privateKeyPath: "./aws/Y3PA.pem",
expireTime: (new Date().getTime() + 999999999)
};
// Generating a signed URL
signedUrl = () => {
console.log("url created " + process.env.PUBLIC_KEY);
return cfsign.getSignedUrl(
"xxxx.cloudfront.net/test.html",
signingParams
);
}
The scheme is part of the URL that is required as input to the signature algorithm, so your error is lilely to be here:
cfsign.getSignedUrl("xxxx.cloudfront.net/...
Instead of that, you need this:
cfsign.getSignedUrl("https://xxxx.cloudfront.net/...
I am using sts:assumeRole to connect to a s3 bucket of a different account.
Now, the job that I run takes a few days and along the way the credentials expire and I needed a way to renew them.
I have written the following code to handle expiry of the temporary credentials
This code is inside my downloadFile():
return new Promise((resolve, reject) => {
function responseCallback(error, data) {
if (error) {
const errorMessage = `Fail to download file from s3://${config().s3.bucket}/${path}: ${error};`;
reject(error);
} else {
Logger.info(`Successfully download file from s3://${config().s3.bucket}/${path}`);
resolve(data.Body);
}
}
const fn = this.s3Client.getObject({
Bucket: config().s3.bucket,
Key: path
}, (error, data) => this.handleTokenExpiry(error, data, fn, responseCallback));
});
And this is the handleTokenExpiry()
handleTokenExpiry(error, data, fn, callback) {
if (!error || error.code !== "ExpiredToken") return callback(error, data);
Logger.info("Token expired, creating new token");
this.s3Client = null; // null so that init() doesn't return existing s3Client
return this.init().then(fn);
}
Here init() is the method which sets this.s3Client using sts:assumeRole
and then new AWS.S3()
This works fine but I am not sure if this a clean way to do it. The strange thing is when I test it in local it takes almost two minutes for responseCallback() to be called when token is expired. Though responseCallback() gets executed immediately while the token is active.
For tasks running less than 12h, here is the solution.
When using AssumeRole, you can specify DurationSeconds argument to specify the duration of the temporary credentials returned by STS. This is 15 min minimum, up to 12h.
The role you are assuming needs to be modified to authorize the maximum duration too. See https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session
In your case, with a job running for several days. I would suggest to refactor the app to run in smaller batches, each running for a few hours each.
Another alternative would be to be proactive about token expiration. If your code know the token duration and the time at which it acquired the token, I would suggest to call a method before calling a method that uses the token (such as S3's getObject). That method you check if the token are soon to expire and proactively refresh them. Pseudo code would be like
function refreshToken() {
return new Promise( (resolve, reject) => {
// XX depends how long is your S3 getObject call
if (token_acquisition_time + token_duration <= now() + xx minutes) {
// refresh token
sts.assumeRole(...).promise().then(resolve());
} else {
resolve();
}
});
}
...
refreshToken.then(s3.getObject(...));
The AWS SDK can handle refreshing the credentials for you. For example:
const credentials = new ChainableTemporaryCredentials({
params: {
RoleArn: "optional-role-arn",
RoleSessionName: `required-parameter-${Date.now()}`
}
})
const s3 = new AWS.S3({credentials})
Now AWS SDK will refresh the tokens behind the scenes without any action from caller of s3.
For more information, please see AWSK SDK Documentation. Refresh is limited to validity time of the credentials used.
I am using a code as the following to create a signed Url for my content:
var storage = require('#google-cloud/storage')();
var myBucket = storage.bucket('my-bucket');
var file = myBucket.file('my-file');
//-
// Generate a URL that allows temporary access to download your file.
//-
var request = require('request');
var config = {
action: 'read',
expires: '03-17-2025'
};
file.getSignedUrl(config, function(err, url) {
if (err) {
console.error(err);
return;
}
// The file is now available to read from the URL.
});
This creates an Url that starts with https://storage.googleapis.com/my-bucket/
If I place that URL in the browser, it is readable.
However, i guess that URL is a direct access to the bucket file and is not passing through my configured CDN.
I see that in the docs (https://cloud.google.com/nodejs/docs/reference/storage/1.6.x/File#getSignedUrl) you can pass a cname option, which transforms the url to replace https://storage.googleapis.com/my-bucket/ to my bucket CDN.
HOWEVER when I copy the resulting URL, the sevice account or resulting url doesn't seem to have access to the resource.
I have added the firebase admin service account to the bucket but still I get no access.
Also, from the docs, the CDN signed url seems a lot different from the one signed through that API. Is it possible to create from the api a CDN signed url, or should i manually create it as explained in: https://cloud.google.com/cdn/docs/using-signed-urls?hl=en_US&_ga=2.131493069.-352689337.1519430995#configuring_google_cloud_storage_permissions?
For anyone interested in the node code for that signing:
var url = 'URL of the endpoint served by Cloud CDN';
var key_name = 'Name of the signing key added to the Google Cloud Storage bucket or service';
var key = 'Signing key as urlsafe base64 encoded string';
var expiration = Math.round(new Date().getTime()/1000) + 600; //ten minutes after, in seconds
var crypto = require("crypto");
var URLSafeBase64 = require('urlsafe-base64');
// Decode the URL safe base64 encode key
var decoded_key = URLSafeBase64.decode(key);
// buILD URL
var urlToSign = url
+ (url.indexOf('?') > -1 ? "&" : "?")
+ "Expires=" + expiration
+ "&KeyName=" + key_name;
//Sign the url using the key and url safe base64 encode the signature
var hmac = crypto.createHmac('sha1', decoded_key);
var signature = hmac.update(urlToSign).digest();
var encoded_signature = URLSafeBase64.encode(signature);
//Concatenate the URL and encoded signature
urlToSign += "&Signature=" + encoded_signature;
The Cloud CDN content delivery network works with HTTP(S) load balancing to deliver content to your users. Are you using HTTPS Load Balancer to deliver content to your users?
You can see this attached document[1] on using Google Cloud CDN and HTTP(S) load balancing and inserting content into the cache.
[1] https://cloud.google.com/cdn/docs/overview
[2] https://cloud.google.com/cdn/docs/concepts
What error code are you getting? Can you use the curl command and send the output with the error code for further analysis.
Could you confirm that configuration you have done meets the requirement of cacheability, as not all the HTTP response are cacheable? Google Cloud CDN caches only those responses that satisfy specific conditions [3], please confirm. Upon confirmation, I will do further investigation and advise you accordingly.
[3] Cacheability: https://cloud.google.com/cdn/docs/caching#cacheability
Could you provide me the output of this two command below, which will help me to verify if there is a permission issue on these objects? These commands will dump all the current permission settings on the object.
gsutil acl get gs://[full_path_to_file_to_be_cached]
gsutil ls -L gs://[full_path_to_file_to_be_cached]
For more details on permission, refer to this GCP documentation [4]
[4] Setting bucket permissions: https://cloud.google.com/storage/docs/cloud-console#_bucketpermission
No, it is not possible to create from the API a CDN signed URL
From what Google documents here. The answer provided by #htafoya seem legit.
However, I spent a couple of hours to struggle why the signed URL not working as CDN endpoint complains access denied. Eventually I found the code using crypto module doesn't produce the same hmac-sha1 hash value as what gcloud compute sign-url computed, I still don't know why.
At the same time, I see this lib (jsSHA) is pretty cool, it generates the HMAC-SHA1 hash value exactly the same as gcloud and it has a neat API, so I think I should comment here so that if others have the same struggle will benefit from this, this is the final code I used to sign gcloud cdn URL:
import jsSHA from 'jssha';
const url = `https://{domain}/{path}`;
const expire = Math.round(new Date().getTime() / 1000) + daySeconds;
const extendedUrl = `${url}${url.indexOf('?') > -1 ? "&" : "?"}Expires=${expire}&KeyName=${keyName}`;
// use jssha
const shaObj = new jsSHA("SHA-1", "TEXT", { hmacKey: { value: signKey, format: "B64" } });
shaObj.update(extendedUrl);
const signature = safeSign(shaObj.getHMAC('B64'));
return `${extendedUrl}&Signature=${signature}`;
working great!
I am trying to generate an url signed by Cloudfront (to set expiration of the link) for S3 object.
My code is in Nodejs:
var cfsign = require('aws-cloudfront-sign');
var moment = require('moment');
var options = {
keypairId: 'XXXXXXXX',
privateKeyPath: './pk-XXXXXX.pem',
expireTime: moment().add(60 * 30, 'seconds') //expired after 30 mins
};
myUrl = cfsign.getSignedUrl('https://xxxxxxx.cloudfront.net/bucket/s3object.mp4', options);
Result: the signed url is generated and worked fine with expiration is 30 mins. ("https://xxxxxxx.cloudfront.net/bucket/s3object.mp4?....")
My question: Is there any way to change the s3 object key (s3object.mp4) in the signed url?
There is no direct way of changing the key name. If you want to have a different convention. You can use AWS Lambda Edge function to change the URL format.