I have a video stored in amazon s3.
Now I'm serving it to the client with node.js stream
return request(content.url).pipe(res)
But, the following format is not working with safari.
Safari is unable to play streamed data. But, the same code works for chrome and firefox.
I did some research and found out
chrome's request content-range param looks like
[0-]
But, safari does the same with content ranges
[0-10][11-20][21-30]
Now if the content was stored in my server, I could have break the file in chucks with
fs.createReadStream(path).pipe(res)
to serve safari with it's requested content range
As mentioned in this blog https://medium.com/better-programming/video-stream-with-node-js-and-html5-320b3191a6b6
How can I do the same with remote url stored in s3?
FYI, It's not feasible to download the content temporarily on server and delete it after serving. As, the website is supposed to receive good traffic.
How can I do the same with remote url stored in s3?
Don't.
Let S3 serve the data. Sign a URL to temporarily allow access to the client. Then, you don't have to serve or proxy anything and you save a lot of bandwidth. An example from the documentation:
var params = {Bucket: 'bucket', Key: 'key'};
var url = s3.getSignedUrl('getObject', params);
console.log('The URL is', url);
...As, the website is supposed to receive good traffic.
You'll probably also want to use a CDN to further reduce your costs and enhance the performance. If you're already using AWS, CloudFront is a good choice.
Related
My goal is to get the image from Twilio api to store it somewhere else because Twilio deletes media after 4h.
(See here)
I'm using node and I have read https://www.twilio.com/docs/sms/api/media-resource#fetch-a-media-resource
...and it says that to do a request to this URL without the "json" extension should return the media with its original MIME type
https://api.twilio.com/2010-041/Accounts/{AccountSid}/Messages/{MessageSid}/Media/{Sid}.json
However, i need auth, so I need to use
const client = require('twilio')(accountSid, authToken);
How can I fetch the image? Any sample code to achieve it? In the docs seem to do it without auth.
UPDATE ----------------------------------------
After accessing the MediaUrl0 on the browser, twilio redirects me to the following URL:
https://s3-external-1.amazonaws.com/media.twiliocdn.com/{AccountSid}/{?}
I was thinking of building my own URL but i dont know how to get the {?}
You do not need authentication to retrieve media for an incoming sms . They are all hosted (as of now on Aws S3) and accessible publicly through a (hard to guess) url.
you can access them using any http client
So I want to pipe a file straight to the client; how I am currently doing it is create a file to disk, then sending that file straight to the client.
router.get("/download/:name", async (req, res) => {
const s3 = new aws.S3();
const dir = "uploads/" + req.params.name + ".apkg"
let file = fs.createWriteStream(dir);
await s3.getObject({
Bucket: <bucket-name>,
Key: req.params.name + ".apkg"
}).createReadStream().pipe(file);
await res.download(dir);
});
I just looked up that res.download() only serves locally. Is there a way you can do it directly from AWS S3 to Client download? i.e. pipe files straight to user. Thanks in advance
As described in this SO thread:
You can simply pipe the read stream into the response instead of the piping it to the file, just make sure to supply the correct Content-Type and to set it as an attachment, so the browser will know how to handle the response properly.
res.attachment(req.params.name);
await s3.getObject({
Bucket: <bucket-name>,
Key: req.params.name + ".apkg"
}).createReadStream().pipe(res);
On more pattern for this is to create a signed url directly to the S3 object and then let the client download straight from S3, instead of streaming it from your node webserver. This will reduce the workload from your web server.
You will need to use the getSignedUrl method from the AWS S3 SDK for JS.
Then, Once you have the URL, just return it to your client to download the file by themselves.
You should take into account that once you give the client a signed URL that has download permissions for, say, 5 minutes, they will only be able to download that file during those next 5 minutes. And you should also take into account that they will be able to pass that URL to anyone else for download during those 5 minutes, so it is dependant on how secure you need this to be.
S3 can be used to content so I would do the following.
Add CORS headers on your node response. This will enable browser to download from another origin i.e. S3.
Enable S3 web server on your bucket.
Script to download redirect from S3 - this you could achieve in JS.
Use signed URL as suggested in the other post if you need to protect S3 content.
In my app, i'm sending photos directly from the client to s3, using something similar to this suggested heroku recommendation: https://devcenter.heroku.com/articles/s3-upload-node
The main benefit is that it saves server cost (i'm assuming since chunks aren't being sent to the server using something such as multipart-y form data).
However, I wish to be able to share these images to twitter also, which states this requirement:
Ensure the POST is a multipart/form-data request. Either upload the raw binary (media parameter) of the file, or its base64-encoded contents (media_data parameter). Use raw binary when possible, because base64 encoding results in larger file sizes
I've tried sending the base64 needed for the client-side s3 upload back to the server, but depending on the photo size -- I often get an error that it's too big to send back.
TLDR
Do I need to send my photos using mulitparty / multipart form data to my server, so I can have the needed base64 / binary to share a photo to twitter, or can I keep sending photos from my client to s3?
Then, somehow, efficiently obtain the needed base64 / binary on the server (possibly using the request module), so I can then send the image to twitter?
One fairly easy way to do this without changing your client code a whole lot would be to use S3 events. S3 events can trigger a lambda function in AWS that can post the image to twitter. You can use any library inside the lambda function to do efficient posting to twitter. Not sure if you want to use Lambda or stick to Heroku.
If you are directly uploading documents from the client to upload to s3, you are exposing your AWS secret/private keys with the client. A more secure way would be uploading the images to node and node in turn upload it to S3. A recommended way to upload images to node server would be using
multipart/form-data and using Multer middleware.
Regardless of the upload method, you can use the following code to serve images to twitter. This code uses AWS-SDK module.
var s3 = new AWS.S3();
var filename = req.query.filename;
var params = {
Bucket: <bucketname>,
Key: <image path>
};
var extension = filename.split('.')[1];
if (extension == "jpg" || extension == "JPG" || extension == "jpeg" || extension == "JPEG")
{
res.setHeader('Content-Type', 'image');
}
else if (extension == "png" || extension == "PNG")
{
res.setHeader('Content-Type', 'image/png');
}
s3.getObject(params).createReadStream().pipe(res);
This method can scale with easy like any other express app.
I am developing a node express web application and I am trying to figure out how to display private s3 images when performing a query or single request for an image to views for proper html rendering. I've looked around and didn't find enough information to fully wrap my head around this.
To reiterate, I cannot publicly host them. They have to be privately stored and retrieved when a user uses my express app. I've tried knox which is great for piping, but I don't know how to display all images in one go to a query results page for example. Or more importantly, how to show the raw image data recieved from knox. I also read some stuff about Amazon CloudFront and all that stuff, but would like to exhaust some closer options than doing more and more configuration.
So, how can I view these private s3 images from an express web app? More specifically, displaying a collection of images or a single image.
So if your goal is to have the server fetch the images and then send them to the client, it looks like knox should be able to help you:
from their Github page, with tweaks:
app.get('/img/:filename', function(req, res){
client.getFile('/whatever/' + req.query.filename, function(err, s3res){
s3res.pipe(res)
});
}
Note: untested.
Yes. this is a complex question. i will try to nake it brief.
My website fetches resources from s3.
I also have an extension that needs to prefetch that s3 file when someone does a google query, so later when they go on my site ,the resource is cached.
At this point I should probably stress that I'm not doing anything malicious. just a matter of user experience.
My problem is. that making an ajax request to s3 fron the extension (either from content-script or background) doesn't send an origin header.
This means that the resource is downloaded and cached without an allow origin header. s3
doesnt add that allow-origin:* if theres no origin in the request. so later, on my site it fails due to missing allow-origin header in cached file :-(
Any ideas on a better way to prefetch to browser cache?
Is there a way to force the ajax request to send an origin? Any origin?
Since I have an allow-origin:* on my s3 bucket, I think any origin will do accept null.
Thanks
Edit: Ended up using one of Rob W's solutions. You are awesome.
Let me comment on each of the options he suggested:
Not to add the host premissions on my manifest - clever idea but wouldn't work for me since I have a content script which runs on any website, so I must use a catch-all wildcard, and I don't think there is an "exclude" permission option.
I tried it, it issues a null origin, which as expected ends up in S3 sending the allow-origin:* header as required. this means I don't get that "allow-origin header missing" error, however the file is then not served from cache. I guess for it to be actually served from cache in chrome this has to be exactly the same origin. so that was very close but not enough.
third option is a charm. And it is the simplest. I didn't know I was able to manipulate the origin header. So I do that and set the exact origin of my website - And it works. The file is cached and later served from cache. I must stress that i had to add a Url filter to only apply this to requests going out to my s3 bucket, otherwise I expect this would wreak havoc on the user's browser.
Thanks. Well done
You've got three options:
Do not add the host permission for S3 to your manifest file. Then the extension will not have the blanket permission to access the resource, and an Origin request header will be sent with the request.
Use a non-extension frame to perform the AJAX request. For example, the following method will result in a cross-origin GET request with Origin: null.
function prefetchWithOrigin(url) {
var html = '<script>(' + function(url) {
var x = new XMLHttpRequest();
x.open('GET', url);
x.onloadend = function() {
parent.postMessage('done', '*');
};
x.send();
} + ')(' + JSON.stringify(url) + ');</script>';
var f = document.createElement('iframe');
f.src = 'data:text/html,' + encodeURIComponent(html);
(document.body || document.documentElement).appendChild(f);
window.addEventListener('message', function listener(event) {
// Remove frame upon completion
if (event.source === f.contentWindow) {
window.removeEventListener('message', listener);
f.remove();
}
});
}
Use the chrome.webRequest.onBeforeSendHeaders event to manually append the Origin header.