I want to reproduce music files from google drive on a web page. I have the link for each file but the response cache headers for the calls are 'no-cache, no-store, max-age=0, must-revalidate" so it will never be saved on the browser cache. Is there any way to add cache to google drive files requests?
The problem:
When you use the drive link for a music file (mp3) https://drive.google.com/a/pucp.pe/uc?id=1kYYS9FZ9Vxif5WJM9ZQcY4SR35NMgoIE&export=download the GET API call receives a 302 code which generates a redirect to another URL, in this case, to 'https://doc-0o-28-docs.googleusercontent.com/docs/securesc/bgp95l3eabkkpccn0qi0qopvc4e7d4mq/us95e8ush1v4b7vvijq1vj1d7ru4rlpo/1556330400000/01732506421897009934/01732506421897009934/1kYYS9FZ9Vxif5WJM9ZQcY4SR35NMgoIE?h=14771753379018855219&e=download'. Each of these calls has no-cache in headers.
I tried using workbox (cache API) but I don't find a way to cache redirects, probably I need to cache both calls (the first GET and the redirect). However, if I use the redirected URL the caching works, but I don't have access to that URL until the first call is made.
I tried to use a proxy server from a NodeJS server
app.get("/test", (req, res) => {
try {
https.get(
URL,
function(response) {
res.writeHead(response.statusCode, {...response.headers,
"Cache-Control": "public, max-age=120",
"Expires": new Date(Date.now() + 120000).toUTCString() })
response.pipe(res);
}
);
} catch (e) {
console.log(e);
}
});
I tried using the first URL with no luck.
I tried using the redirect URL but I get a "Status Code: 302 Found"
One solution could be to download the file and serve it directly from my server but I will be missing the point of using the drive storage. I really want to use google drive storage and not duplicate all files on my server.
Is there a recommended way to do the caching for this case? maybe there is some google drive configuration that I'm missing. Or do you know another approach I could take in this case?
You should be able to cache redirected responses using Workbox (and the Cache Storage API in general). HTTP 30x redirects are followed automatically by default, and you should only have to route the original, non-redirected URL.
Here's a live example of a Glitch project that uses Workbox to cache that MP3 file: https://glitch.com/edit/#!/horn-sausage?path=sw.js:9:0
The relevant snippet of code, that also accounts for the fact that there is no CORS used when serving the files (so you'll get back an opaque response with a status of 0) is:
workbox.routing.registerRoute(
new RegExp('https://drive.google.com/'),
new workbox.strategies.CacheFirst({
plugins: [
new workbox.cacheableResponse.Plugin({statuses: [0, 200]})
],
})
);
Related
Suppose I enter a (public) website that makes 3 XHR/fetch calls on 3 different endpoints:
https://api.example.com/path1
https://api.example.com/path2
https://api.example.com/path3
What I want to achieve is intercept the call to https://api.example.com/path2 only, redirect it to a local service (localhost:8000) and let path1 and path3 through to the original domain.
What kind of options do I have here? I have studied a lot of approaches to this issue:
DNS rewriting - this solution is not suitable as I still have to intercept path1 and path3, only redirect them to the original IPs and try to mimic the headers as much as possible - which means I would have to do a specific proxy configuration for each intercepted domain - this is unfeasible
Chrome extensions - found none to deal specifically with single endpoint intercepting
Overwriting both fetch and XmlHttpRequest after page load - still doesn't cover all scenarios, maybe some websites cache the values of fetch and XmlHttpRequest before page load (?)
Combining the chrome extension and fetch overwrite will work.
download an webextension that let you load javascript code before a given page loads, e.g. User JavaScript and CSS
Add the following script to run before your page loads, base on: Intercepting JavaScript Fetch API requests and responses
const { fetch: originalFetch } = window;
window.fetch = async (...args) => {
let [resource, config ] = args;
// request interceptor starts
resource = resource === "https://api.example.com/path2" ? "http://localhost:8000/path2" : resource
// request interceptor ends
const response = await originalFetch(resource, config);
// response interceptor here
return response;
};
i have a file on my "website/folder/file" which i would like to redirect to prevent user to access that file, without using htaccess.
My file is a huge DB containing url's, i don't want the users access to that file by typing the direct URL to the file.
That file is called & used by my chrome extension, who block access to the user if he tries to reach one of the url's in that DB.
Problem is by typing the direct url to that file we have access...
i tried everything with the .htaccess file, i know we can block, redirect, etc with the .htaccess file but if i redirect or block the url of the DB with the htaccess, my extension doesn't work anymore because the DB is blocked by the htaccess file.
so i'm trying to find a solution, maybe is an !
my background.js
'use strict';
let db = []; // session Global
// ----- parse & cache the database data
fetch('http://thmywebsite.com/folder/db.dat')
.then(response => response.text())
.then(text => { db = text.trim().split(/[\r\n]+/); })
.catch(error => console.log(error));
chrome.webRequest.onBeforeRequest.addListener(details => {
let url = new URL(details.url);
return { cancel: url && url.hostname && db.includes(url.hostname) };
},
{ urls: ["http://*/*", "https://*/*"] },
["blocking"]
);
chrome.extension.isAllowedIncognitoAccess(function (isAllowedAccess) {
if (isAllowedAccess) return; // Great, we've got access
})
You can't realistically do this. You can't block a resource that needs to be available publicly (by your client-side script).
You can potentially make this a little harder for someone wanting your DB, by perhaps sending a unique HTTP request header as part of your fetch(). You can then check for the presence of this header server-side (in .htaccess) and block the request otherwise. This prevents a user from casually requesting this file directly in their browser. However, this is trivial to bypass for anyone who looks at your script (or monitors the network traffic) as they can construct the request to mimic your script. But let's not forget, your script downloads the file to the browser anyway - so it's already there for anyone to save.
You need to rethink your data model. If you don't want the DB to be publicly accessible then it simply can't be publicly accessible. Instead of your script downloading the DB to the client and processing the request locally, you could send the request to your server. Your server then performs the necessary lookup (on the "hidden" database) and sends back a response. Your script then acts on this response.
I am new to GAE / NODEJS. There is plenty of advice to enable calls hosted by GAE to be called cross domain, but I am having trouble getting my node js app to call another domains API. Is there something I need to do to get GAE to allow cross domain calls? My code is:
app.get('/listProducts', (req, res) => {
request.get('https://[cross domain]/api/2.0/products', function (error, response, body) {
console.log('error:', error); // Print the error if one occurred
console.log('statusCode:', response && response.statusCode); // Print the response status code if a response was received
console.log('body:', body); // Print the HTML for the Google homepage.
}).auth(null, null, true, '[key goes here]');
You can allow configuring your app.yaml file or setting in your http header.
Here the doc (looking for CORS Support )
Follow the example:
You could have a game app mygame.appspot.com that accesses assets
hosted by myassets.appspot.com. However, if mygame attempts to
make a JavaScript XMLHttpRequest to myassets, it will not succeed
unless the handler for myassets returns an
Access-Control-Allow-Origin: response header containing the value
http://mygame.appspot.com.
Here is how you would make your static file handler return that required response header value:
handlers:
- url: /images
static_dir: static/images
http_headers:
Access-Control-Allow-Origin: http://mygame.appspot.com
# ...
Note: if you wanted to allow everyone to access your assets, you could use the wildcard '*', instead of http://mygame.appspot.com.
was doing newbie - i should have returned a response code, eg response.status(200).send("ok").end();
I want to send a request to this Amazon Alexa API.
That page contains the last 50 activities I made with my Amazon Echo. The page returns JSON. Before you can request that page, you need to authorize your account, so the proper cookies are set in your browser.
If I do something simple as:
const rp = require("request-promise");
const options = {
method: "GET",
uri: "https://alexa.amazon.com/api/activities?startTime=&size=50&offset=-1",
json: true
};
rp(options).then(function(data) {
console.log(data);
}).catch(function(err) {
console.log(err);
});
I can send a GET request to that URL. This works fine, except Amazon has no idea it's me who's sending the request, because I haven't authorized my NodeJS application.
I've successfully copied ~10 cookies from my regular browser into an incognito tab and authorized that way, so I know copying the cookies will work. After adding them all using tough-cookie, it didn't work, unfortunately. I still got redirected to the signin page (according to the error response).
How do I authorize for this API, so I can send my requests?
I have been looking for a solution for this too. The best idea I have is to use account linking, but I haven't try it yet. Looks like ASK-CLI has interface for this also, but I can't figure it out how to use it (what is that URL?). For linking account to 3rd party server is not easy, but link it back to Amazon for the json API should not be that complicated.
Yes. this is a complex question. i will try to nake it brief.
My website fetches resources from s3.
I also have an extension that needs to prefetch that s3 file when someone does a google query, so later when they go on my site ,the resource is cached.
At this point I should probably stress that I'm not doing anything malicious. just a matter of user experience.
My problem is. that making an ajax request to s3 fron the extension (either from content-script or background) doesn't send an origin header.
This means that the resource is downloaded and cached without an allow origin header. s3
doesnt add that allow-origin:* if theres no origin in the request. so later, on my site it fails due to missing allow-origin header in cached file :-(
Any ideas on a better way to prefetch to browser cache?
Is there a way to force the ajax request to send an origin? Any origin?
Since I have an allow-origin:* on my s3 bucket, I think any origin will do accept null.
Thanks
Edit: Ended up using one of Rob W's solutions. You are awesome.
Let me comment on each of the options he suggested:
Not to add the host premissions on my manifest - clever idea but wouldn't work for me since I have a content script which runs on any website, so I must use a catch-all wildcard, and I don't think there is an "exclude" permission option.
I tried it, it issues a null origin, which as expected ends up in S3 sending the allow-origin:* header as required. this means I don't get that "allow-origin header missing" error, however the file is then not served from cache. I guess for it to be actually served from cache in chrome this has to be exactly the same origin. so that was very close but not enough.
third option is a charm. And it is the simplest. I didn't know I was able to manipulate the origin header. So I do that and set the exact origin of my website - And it works. The file is cached and later served from cache. I must stress that i had to add a Url filter to only apply this to requests going out to my s3 bucket, otherwise I expect this would wreak havoc on the user's browser.
Thanks. Well done
You've got three options:
Do not add the host permission for S3 to your manifest file. Then the extension will not have the blanket permission to access the resource, and an Origin request header will be sent with the request.
Use a non-extension frame to perform the AJAX request. For example, the following method will result in a cross-origin GET request with Origin: null.
function prefetchWithOrigin(url) {
var html = '<script>(' + function(url) {
var x = new XMLHttpRequest();
x.open('GET', url);
x.onloadend = function() {
parent.postMessage('done', '*');
};
x.send();
} + ')(' + JSON.stringify(url) + ');</script>';
var f = document.createElement('iframe');
f.src = 'data:text/html,' + encodeURIComponent(html);
(document.body || document.documentElement).appendChild(f);
window.addEventListener('message', function listener(event) {
// Remove frame upon completion
if (event.source === f.contentWindow) {
window.removeEventListener('message', listener);
f.remove();
}
});
}
Use the chrome.webRequest.onBeforeSendHeaders event to manually append the Origin header.