I am trying to generate an url signed by Cloudfront (to set expiration of the link) for S3 object.
My code is in Nodejs:
var cfsign = require('aws-cloudfront-sign');
var moment = require('moment');
var options = {
keypairId: 'XXXXXXXX',
privateKeyPath: './pk-XXXXXX.pem',
expireTime: moment().add(60 * 30, 'seconds') //expired after 30 mins
};
myUrl = cfsign.getSignedUrl('https://xxxxxxx.cloudfront.net/bucket/s3object.mp4', options);
Result: the signed url is generated and worked fine with expiration is 30 mins. ("https://xxxxxxx.cloudfront.net/bucket/s3object.mp4?....")
My question: Is there any way to change the s3 object key (s3object.mp4) in the signed url?
There is no direct way of changing the key name. If you want to have a different convention. You can use AWS Lambda Edge function to change the URL format.
Related
I created a bucket on s3 and added a HTML file, after this I created a Cloud Front key pair using my root user and added a Cloud Front Distribution for that bucket. Tried to access the object using that distribution and it worked, than I restricted access to the bucket using Behaviour and selected "self".
Finally I generated a signed url from node js and tested it using Postman.
The problem is that I always get AccessDenied.
<Error>
<Code>AccessDenied</Code>
<Message>Access denied</Message>
</Error>
Here is my code.
const cfsign = require("aws-cloudfront-sign");
var signingParams = {
keypairId: process.env.PUBLIC_KEY,
privateKeyPath: "./aws/Y3PA.pem",
expireTime: (new Date().getTime() + 999999999)
};
// Generating a signed URL
signedUrl = () => {
console.log("url created " + process.env.PUBLIC_KEY);
return cfsign.getSignedUrl(
"xxxx.cloudfront.net/test.html",
signingParams
);
}
The scheme is part of the URL that is required as input to the signature algorithm, so your error is lilely to be here:
cfsign.getSignedUrl("xxxx.cloudfront.net/...
Instead of that, you need this:
cfsign.getSignedUrl("https://xxxx.cloudfront.net/...
I am using a code as the following to create a signed Url for my content:
var storage = require('#google-cloud/storage')();
var myBucket = storage.bucket('my-bucket');
var file = myBucket.file('my-file');
//-
// Generate a URL that allows temporary access to download your file.
//-
var request = require('request');
var config = {
action: 'read',
expires: '03-17-2025'
};
file.getSignedUrl(config, function(err, url) {
if (err) {
console.error(err);
return;
}
// The file is now available to read from the URL.
});
This creates an Url that starts with https://storage.googleapis.com/my-bucket/
If I place that URL in the browser, it is readable.
However, i guess that URL is a direct access to the bucket file and is not passing through my configured CDN.
I see that in the docs (https://cloud.google.com/nodejs/docs/reference/storage/1.6.x/File#getSignedUrl) you can pass a cname option, which transforms the url to replace https://storage.googleapis.com/my-bucket/ to my bucket CDN.
HOWEVER when I copy the resulting URL, the sevice account or resulting url doesn't seem to have access to the resource.
I have added the firebase admin service account to the bucket but still I get no access.
Also, from the docs, the CDN signed url seems a lot different from the one signed through that API. Is it possible to create from the api a CDN signed url, or should i manually create it as explained in: https://cloud.google.com/cdn/docs/using-signed-urls?hl=en_US&_ga=2.131493069.-352689337.1519430995#configuring_google_cloud_storage_permissions?
For anyone interested in the node code for that signing:
var url = 'URL of the endpoint served by Cloud CDN';
var key_name = 'Name of the signing key added to the Google Cloud Storage bucket or service';
var key = 'Signing key as urlsafe base64 encoded string';
var expiration = Math.round(new Date().getTime()/1000) + 600; //ten minutes after, in seconds
var crypto = require("crypto");
var URLSafeBase64 = require('urlsafe-base64');
// Decode the URL safe base64 encode key
var decoded_key = URLSafeBase64.decode(key);
// buILD URL
var urlToSign = url
+ (url.indexOf('?') > -1 ? "&" : "?")
+ "Expires=" + expiration
+ "&KeyName=" + key_name;
//Sign the url using the key and url safe base64 encode the signature
var hmac = crypto.createHmac('sha1', decoded_key);
var signature = hmac.update(urlToSign).digest();
var encoded_signature = URLSafeBase64.encode(signature);
//Concatenate the URL and encoded signature
urlToSign += "&Signature=" + encoded_signature;
The Cloud CDN content delivery network works with HTTP(S) load balancing to deliver content to your users. Are you using HTTPS Load Balancer to deliver content to your users?
You can see this attached document[1] on using Google Cloud CDN and HTTP(S) load balancing and inserting content into the cache.
[1] https://cloud.google.com/cdn/docs/overview
[2] https://cloud.google.com/cdn/docs/concepts
What error code are you getting? Can you use the curl command and send the output with the error code for further analysis.
Could you confirm that configuration you have done meets the requirement of cacheability, as not all the HTTP response are cacheable? Google Cloud CDN caches only those responses that satisfy specific conditions [3], please confirm. Upon confirmation, I will do further investigation and advise you accordingly.
[3] Cacheability: https://cloud.google.com/cdn/docs/caching#cacheability
Could you provide me the output of this two command below, which will help me to verify if there is a permission issue on these objects? These commands will dump all the current permission settings on the object.
gsutil acl get gs://[full_path_to_file_to_be_cached]
gsutil ls -L gs://[full_path_to_file_to_be_cached]
For more details on permission, refer to this GCP documentation [4]
[4] Setting bucket permissions: https://cloud.google.com/storage/docs/cloud-console#_bucketpermission
No, it is not possible to create from the API a CDN signed URL
From what Google documents here. The answer provided by #htafoya seem legit.
However, I spent a couple of hours to struggle why the signed URL not working as CDN endpoint complains access denied. Eventually I found the code using crypto module doesn't produce the same hmac-sha1 hash value as what gcloud compute sign-url computed, I still don't know why.
At the same time, I see this lib (jsSHA) is pretty cool, it generates the HMAC-SHA1 hash value exactly the same as gcloud and it has a neat API, so I think I should comment here so that if others have the same struggle will benefit from this, this is the final code I used to sign gcloud cdn URL:
import jsSHA from 'jssha';
const url = `https://{domain}/{path}`;
const expire = Math.round(new Date().getTime() / 1000) + daySeconds;
const extendedUrl = `${url}${url.indexOf('?') > -1 ? "&" : "?"}Expires=${expire}&KeyName=${keyName}`;
// use jssha
const shaObj = new jsSHA("SHA-1", "TEXT", { hmacKey: { value: signKey, format: "B64" } });
shaObj.update(extendedUrl);
const signature = safeSign(shaObj.getHMAC('B64'));
return `${extendedUrl}&Signature=${signature}`;
working great!
I'm using IBM Cloud Object Storage and its NodeJS SDK (https://github.com/IBM/ibm-cos-sdk-js).
I want to generate presigned links so that users can access contents in the bucket without the need to authenticate.
When I call cos.getSignedUrl('getObject', ...), I get the error UnsupportedSigner: Presigning only supports S3 or SigV4 signing.
How to solve this issue?
First you need to generate HMAC keys for your service instance as described in How do I create HMAC credentials for IBM Cloud Object Storage using the CLI?
Once you have the HMAC access key and secret access key, change the initialization of the COS SDK as follow:
const config = {
endpoint: 'cos endpoint',
apiKeyId: 'cos api key',
ibmAuthEndpoint: 'https://iam.ng.bluemix.net/oidc/token',
serviceInstanceId: 'cos crn'
// these two are required to generate presigned URLs
credentials: new COS.Credentials('<access key>', '<secret_access_key>', sessionToken = null),
signatureVersion: 'v4'
};
const cos = new COS.S3(config);
then you can generate presigned links with:
const url = await cos.getSignedUrl('getObject', {
Bucket: '<your-bucket-name>',
Key: '<your-key-name>',
Expires: 60 * 5, // 5 minutes
});
I am generating signed urls on my webapp (nodejs) using the knox nodejs-library.
However the issue arises, that for every request, I need to regenerate an unique GET signed url for the current user, leaving browser's cache-control out of the game.
I've searched the web without success as browsers seem to use the full url as caching key so I am really curious how I can, under the given circumstances (nodejs, knox library) get the issue solved and use caching control while still being able to generated signed urls for each and every request as I need to verify the user's access rights.
I cannot believe there's no solution to that though.
I am working with Java AmazonS3 client, but the process should be the same.
There is a strategy that can be used to handle this situation.
You could use a fixed date time as an expiration date. I set this date to tomorrow at 12 pm.
Now every time you generate a url, it will be the same throughout that day until 00:00. That way browser caching can be used to some extent.
Expanding #semir-deljić Answer.
Every time we call getSignedUrl function, it will generate new URLs. This will result in images not being cached even if Cache Control header is present.
Thus, we are using timekeeper library to freeze time. Now when the function is called, it thinks that the time has not passed, and it returns same URL.
const moment = require('moment');
const tk = require("timekeeper");
function url4download(awsPath, awsKey) {
function getFrozenDate() {
return moment().startOf('week').toDate();
}
// Paramters for getSignedUrl function
const params = {
// Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
// Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
Bucket: awsBucket,
Key: `${awsPath}/${awsKey}`,
// 604800 == 7 days
ResponseCacheControl: `public, max-age=604800, immutable`,
Expires: 604800, // 7 days is max
};
const url = tk.withFreeze(getFrozenDate(), () => {
return S3.getSignedUrl('getObject', params);
});
return url;
}
Note:
Using moment().toDate(), as the timekeeper requires a Native Date Object.
Even tough the question is for using knox library, my answer uses aws official library.
// This is how the AWS & S3 is initiliased.
const AWS = require('aws-sdk');
const S3 = new AWS.S3({
accessKeyId: awsAccessId,
secretAccessKey: awsSecretKey,
region: 'ap-south-1',
apiVersion: '2006-03-01',
signatureVersion: 'v4',
});
Inspiration: https://advancedweb.hu/cacheable-s3-signed-urls/
If you use CloudFront with S3, you can use a Custom Policy, if you restrict each url to the user's IP and a reasonably long timeout, it means that when they request the same content again, they will get the same URL and hence their browser can cache the content but the URL will not work for someone else (on a different IP).
(see: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-creating-signed-url-custom-policy.html)
When calculating signed URL you can set 'signingDate' to a fixed moment in the past e.g. yesterday morning, then calculate expiration from that moment.
Don't forget to use UTC and account for timezones.
import { S3Client, GetObjectCommand, GetObjectCommandInput } from "#aws-sdk/client-s3";
import { getSignedUrl } from "#aws-sdk/s3-request-presigner";
let signingDate = new Date();
signingDate.setUTCHours(0, 0, 0, 0);
signingDate.setUTCDate(signingDate.getUTCDate() - 1);
let params: GetObjectCommandInput = {
Bucket: BUCKET_NAME,
Key: filename
};
const command = new GetObjectCommand(params);
const url = await getSignedUrl(s3Client,
command, {
expiresIn: 3 * 3600 * 24, // 1 day until today + 1 expiration + 1 days for timezones
signableHeaders: new Set < string > (),
signingDate: signingDate
});
Filepicker by default allows pretty much everybody to add files to your S3 bucket who was clever enough to copy your API key out of the client code and luckily also offers a security option with expiring policies.
But I have no idea how to implement this in Meteor.js. Tried back and forth, installing meteor-crypto-base package, trying to generate the hashes on the server, tried RGBboy's urlsafe-base64 algorithm on https://github.com/RGBboy/urlsafe-base64. But I just do not get any further. Maybe someone can help! Thank you in advance.
This is an example of how to do filepicker signed URLs in meteor, based on the documentation here:
var crypto = Npm.require('crypto');
var FILEPICKER_KEY = 'Z3IYZSH2UJA7VN3QYFVSVCF7PI';
var BASE_URL = 'https://www.filepicker.io/api/file';
Meteor.methods({
signedUrl: function(handle) {
var expiry = Math.floor(new Date().getTime() / 1000 + 60 * 60);
var policy = new Buffer(JSON.stringify({
handle: handle,
expiry: expiry
})).toString('base64');
var signature = crypto
.createHmac('sha256', FILEPICKER_KEY)
.update(policy)
.digest('hex');
return BASE_URL + "/" + handle +
"?signature=" + signature + "&policy=" + policy;
}
});
Note this will need to exist somewhere inside of your server directory so you don't ship the key to the client. To demonstrate that it works, on the client side you can call it like so:
Meteor.call('signedUrl', 'KW9EJhYtS6y48Whm2S6D', function(err, url){console.log(url)});
If everything worked, you should see a photo when you visit the returned URL.