How can I prevent my ServiceWorker from intercepting cross-origin requests? - cross-domain

I am trying to clean up some code on my application https://git.sequentialread.com/forest/sequentialread-password-manager
I am using a ServiceWorker to enable the application to run offline -- however, I noticed that the ServiceWorker is intercepting cross-origin requests to backblazeb2.com. The app makes these cross origin requests as a part of its normal operation.
You can see here how I am registering the ServiceWorker:
navigator.serviceWorker.register('/serviceworker.js', {scope: "/"}).then(
(reg) => {
...
And inside the serviceworker.js code, I manually avoid caching any requests to backblazeb2.com:
...
return fetch(event.request).then(response => {
const url = new URL(event.request.url);
const isServerStorage = url.pathname.startsWith('/storage');
const isVersion = url.pathname == "/version";
const isBackblaze = url.host.includes('backblazeb2.com');
const isPut = event.request.method == "PUT";
if(!isServerStorage && !isVersion && !isBackblaze && !isPut) {
... // cache the response
However, this seems silly, I wish there was a way to limit the ServiceWorker to only intercept requests for the current origin.
I already tried inserting the origin into the scope property during registration, but this didn't work:
navigator.serviceWorker.register('/serviceworker.js', {scope: window.location.origin}).then(
(reg) => {
...
It was behaving the same way. I am assuming that perhaps this is because there are CORS headers present on the responses from backblazeb2.com, making those requests "technically" within the "scope" of the current origin ?
One idea I had, I could serve a permanent redirect from / to /static/index.html and then configure the serviceworker with a scope of /static, meaning it would only cache resources in that folder. But that seems like an ugly hack I should not have to do.
Is there a clean and "correct" way to do this??

As far as I can tell, the answer is, you can't do this. the serviceworker api won't let you.
Someone explained it to me as "the scope for the serviceworker limits where FROM the requests can be intercepted, not where TO the requests can be intercepted. So in otherwords, if I register a serviceworker at /app/, then a javascript from / or /foo/ will be able to make requests without them being intercepted.
It turns out that actually what I REALLY needed was to understand how service worker error handling works.
In the old version of my app, when the fetch() promise rejected, my code would return null.
return fetch(event.request).then(response => {
...blahblahblah...
}).catch( e => {
....
return null;
});
This was bad news bears and it was causing me to want to skip the serviceworker. what I didn't understand was; its not the serviceworkers fault per se as much as the fact that my serviceworker did not handle errors correctly. So the solution was to handle errors better.
This is what I did. I introduced a new route on the server called /error that always returns the string "serviceworker request failed".
http.HandleFunc("/error", func(response http.ResponseWriter, request *http.Request) {
response.WriteHeader(200)
fmt.Fprint(response, "serviceworker request failed")
})
And then I made sure to cache that endpoint when the service worker is installed.
self.addEventListener('install', event => {
event.waitUntil(clients.get(event.clientId).then(client => {
return caches.open(cacheVersion).then(cache => {
return cache.addAll([
'/',
'/error',
....
Finally, when the serviceworker fetch() promise rejects, I fall back to returning the cached version.
return fetch(event.request).then(response => {
...blahblahblah...
}).catch( e => {
....
return caches.match('/error');
});
I got the idea from the MDN serviceworker example project which does a similar thing and simply returns a cached image of darth vader if the fetch() promise rejects.
This allowed me to gracefully handle these errors and retry instead of silently failing. I simply had to make sure that my code does the right thing when it encounters an http response that matches the literal string "serviceworker request failed".
const requestFailedBytes = app.sjcl.codec.bytes.fromBits(app.sjcl.codec.utf8String.toBits("serviceworker request failed"));
...
var httpRequest = new XMLHttpRequest();
....
httpRequest.onloadend = () => {
...
if(app.cryptoService.uint8ArrayEquals(new Uint8Array(httpRequest.response), requestFailedBytes)) {
reject(false);
return
}

The fetch event in the service worker has a property on the request called "mode." This property allows you to check if the mode was set to "cors."
Here is an example of how to prevent cors requests.
self.addEventListener('fetch', (e) =>
{
if(e.request.method !== 'GET')
{
return false;
}
if(e.request.mode === 'navigate')
{
e.respondWith(caches.match('index.html'));
return false;
}
else if(e.request.mode === 'cors')
{
return false;
}
const response = this.fetchFile(e);
e.respondWith(response);
});

Related

TypeScript: how to handle async functions inside setTimeout()?

I have the following timed function to periodically refetch credentials from an external API in your usual movie fetching IMDB clone app:
// This variable I pass later to Apollo Server, below all this code.
let tokenToBeUsedLater: string;
// Fetch credentials and schedule a refetch before they expire
const fetchAndRefreshToken = async () => {
try {
// fetchCredentials() sends an http request to the external API:
const { access_token, expires_in} = await fetchCredentials() as Credentials;
tokenToBeUsedLater = access_token;
// The returned token expires, so the timeout below is meant to recursively
// loop this function to refetch fresh credentials shortly before expiry.
// This timeout should not stop the app's execution, so can't await it.
// Have also tried the following with just setTimeout() and
// no `return new Promise()`but it throws a 2nd identical error.
return new Promise(resolve => setTimeout(() => {
resolve(fetchAndRefreshToken());
}, expires_in - 60000));
} catch (err) { throw new Error(err); }
};
fetchAndRefreshToken(); // <-- TypeScript error here
I have tried rewriting it in a thousand ways, but no matter what I do I get a Promises must be handled appropriately or explicitly marked as ignored with the 'void' operator error.
I can get rid the error by by:
Using .then().catch() when calling refreshToken(). Not ideal since I don't want to mix it with async and try/catch.
Putting a void operator ahead of refreshToken(). Feels like cheating.
Putting await ahead of refreshToken(). Essentially breaks my app (pauses execution of the app while waiting for the token to expire, so users in the frontend can't search for movies).
Any idea about how to solve this?
Also, any suggested resources/topics to study about this? Because I had a similar issue yesterday and I still can't figure this one out despite having already solved the other one. Cheers =)

Nodejs proxy request coalescing

I'm running into an issue with my http-proxy-middleware stuff. I'm using it to proxy requests to another service which i.e. might resize images et al.
The problem is that multiple clients might call the method multiple times and thus create a stampede on the original service. I'm now looking into (what some services call request coalescing i.e. varnish) a solution that would call the service once, wait for the response and 'queue' the incoming requests with the same signature until the first is done, and return them all in a single go... This is different from 'caching' results due to the fact that I want to prevent calling the backend multiple times simultaneously and not necessarily cache the results.
I'm trying to find if something like that might be called differently or am i missing something that others have already solved someway... but i can't find anything...
As the use case seems pretty 'basic' for a reverse-proxy type setup, I would have expected alot of hits on my searches but since the problemspace is pretty generic i'm not getting anything...
Thanks!
A colleague of mine has helped my hack my own answer. It's currently used as a (express) middleware for specific GET-endpoints and basically hashes the request into a map, starts a new separate request. Concurrent incoming requests are hashed and checked and walked on the separate request callback and thus reused. This also means that if the first response is particularly slow, all coalesced requests are too
This seemed easier than to hack it into the http-proxy-middleware, but oh well, this got the job done :)
const axios = require('axios');
const responses = {};
module.exports = (req, res) => {
const queryHash = `${req.path}/${JSON.stringify(req.query)}`;
if (responses[queryHash]) {
console.log('re-using request', queryHash);
responses[queryHash].push(res);
return;
}
console.log('new request', queryHash);
const axiosConfig = {
method: req.method,
url: `[the original backend url]${req.path}`,
params: req.query,
headers: {}
};
if (req.headers.cookie) {
axiosConfig.headers.Cookie = req.headers.cookie;
}
responses[queryHash] = [res];
axios.request(axiosConfig).then((axiosRes) => {
responses[queryHash].forEach((coalescingRequest) => {
coalescingRequest.json(axiosRes.data);
});
responses[queryHash] = undefined;
}).catch((err) => {
responses[queryHash].forEach((coalescingRequest) => {
coalescingRequest.status(500).json(false);
});
responses[queryHash] = undefined;
});
};

Is there an alternative for recursive approach I use here in node.js?

I implemented a recursive function in a requestHandler I made to serialize API requests and also to make sure the endpoint isn't currently being requested. To make sure that the endpoint isn't currently being requested, I add it to a Set and verify it with conditionals.
Problem is that this recursive approach consumes quite a lot of memory when a lot of requests are made to the same endpoint. Is there any way I could make it less memory intensive as well as performant at the same time? I would love to hear any alternative approach which I could use instead of recursion. Below you can find my code.
async request(endpoint, domain, method, headers, query, body, attachments) {
const requestURL = `${(domain === "discord") ? this.discordBaseURL :
(domain === "trello") ? this.trelloBaseURL : domain}/${endpoint}`;
if (this.queueCollection.has(endpoint) === false) { // queueCollection is the Set in which I store endpoints that are currently being requested by my requestHandler.
this.queueCollection.add(endpoint);
const response = await this.conditionalsHandler(endpoint, requestURL, method, headers, query, body, attachments);
this.queueCollection.delete(endpoint);
return response;
}
else {
const response = new Promise((resolve) => {
setTimeout(() => { // https://stackoverflow.com/a/20999077
resolve(this.request(endpoint, domain, method, headers, query, body, attachments)); // This is where I make the method recursive to call itself back until the endpoint is no longer in the queueCollection Set.
}, 0);
});
return response;
}
}
Yes, you can remove the recursion by making the queueCollection a Map<string, Promise> instead of a Set<string>, and instead of recursing asynchronously and polling the queue until it's empty, chain the request to the tail of the queue if it exists like this:
async request(endpoint, domain, method, headers, query, body, attachments) {
const requestURL = `${(domain === "discord") ? this.discordBaseURL :
(domain === "trello") ? this.trelloBaseURL : domain}/${endpoint}`;
// get existing queue or create a new one
const queue = this.queueCollection.get(endpoint) || Promise.resolve();
// schedule request on the tail of the queue
const request = queue.then(
() => this.conditionalsHandler(endpoint, requestURL, method, headers, query, body, attachments)
);
// prevent errors from propagating along the queue
const tail = request.catch(() => {});
// enqueue the request
this.queueCollection.set(endpoint, tail);
try {
// propagates error handling to consumer
// waits for request to settle before executing finally block
return await request;
} finally {
// only remove promise from Map if this settled request is at the tail of the queue
if (this.queueCollection.get(endpoint) === tail) this.queueCollection.delete(endpoint);
}
}
This approach allows request to throw without breaking the chain so the consumer can handle the error and all the requests will still happen in sequence without depending on previous requests being successful, and it will always clean up the queueCollection on the last pending request regardless of whether the request throws. The await is not redundant here for that reason.

How to use a Service Worker With BASIC Authentication (NTLM, Negotiate)

I have been trying to use a service worker within a IIS hosted web site that caches some of the static content of the site. The site is an internal application that uses Windows Authentication. I have been able to register and run a service worker without too much hassle, but as soon as I open the caches and start adding files to the cache, the promise fails with an authorisation failure. the returned HTTP result is 401 Unauthorised. This is the usual response for the first few requests until the browser and the server are able to negotiate the authorisation.
I will post some code soon that should help with the explanation.
EDIT
var staticCacheName = 'app-static-v1';
console.log("I AM ALIVE");
this.addEventListener('install', function (event) {
console.log("AND I INSTALLED!!!!");
var urlsToCache = [
//...many js files to cache
'/scripts/numeral.min.js?version=2.2.0',
'/scripts/require.js',
'/scripts/text.js?version=2.2.0',
'/scripts/toastr.min.js?version=2.2.0',
];
event.waitUntil(
caches.open(staticCacheName).then(function (cache) {
cache.addAll(urlsToCache);
}).catch(function (error) {
console.log(error);
})
);
});
This is just a guess, given the lack of code, but if you're doing something like:
caches.open('my-cache').then(cache => {
return cache.add('page1.html'); // Or caches.addAll(['page1.html, page2.html']);
});
you're taking advantage of the implicit Request object creation (see section 6.4.4.4.1) that happens when you pass in a string to cache.add()/cache.addAll(). The Request object that's created uses the default credentials mode, which is 'omit'.
What you can do instead is explicitly construct a Request object containing the credentials mode you'd prefer, which in your case would likely be 'same-origin':
caches.open('my-cache').then(cache => {
return cache.add(new Request('page1.html', {credentials: 'same-origin'}));
});
If you had a bunch of URLs that you were passing an array to cache.addAll(), you can .map() them to a corresponding array of Requests:
var urls = ['page1.html', 'page2.html'];
caches.open('my-cache').then(cache => {
return cache.addAll(urls.map(url => new Request(url, {credentials: 'same-origin'})));
});

Modify HTTP responses from a Chrome extension

Is it possible to create a Chrome extension that modifies HTTP response bodies?
I have looked in the Chrome Extension APIs, but I haven't found anything to do this.
In general, you cannot change the response body of a HTTP request using the standard Chrome extension APIs.
This feature is being requested at 104058: WebRequest API: allow extension to edit response body. Star the issue to get notified of updates.
If you want to edit the response body for a known XMLHttpRequest, inject code via a content script to override the default XMLHttpRequest constructor with a custom (full-featured) one that rewrites the response before triggering the real event. Make sure that your XMLHttpRequest object is fully compliant with Chrome's built-in XMLHttpRequest object, or AJAX-heavy sites will break.
In other cases, you can use the chrome.webRequest or chrome.declarativeWebRequest APIs to redirect the request to a data:-URI. Unlike the XHR-approach, you won't get the original contents of the request. Actually, the request will never hit the server because redirection can only be done before the actual request is sent. And if you redirect a main_frame request, the user will see the data:-URI instead of the requested URL.
I just released a Devtools extension that does just that :)
It's called tamper, it's based on mitmproxy and it allows you to see all requests made by the current tab, modify them and serve the modified version next time you refresh.
It's a pretty early version but it should be compatible with OS X and Windows. Let me know if it doesn't work for you.
You can get it here http://dutzi.github.io/tamper/
How this works
As #Xan commented below, the extension communicates through Native Messaging with a python script that extends mitmproxy.
The extension lists all requests using chrome.devtools.network.onRequestFinished.
When you click on of the requests it downloads its response using the request object's getContent() method, and then sends that response to the python script which saves it locally.
It then opens file in an editor (using call for OSX or subprocess.Popen for windows).
The python script uses mitmproxy to listen to all communication made through that proxy, if it detects a request for a file that was saved it serves the file that was saved instead.
I used Chrome's proxy API (specifically chrome.proxy.settings.set()) to set a PAC as the proxy setting. That PAC file redirect all communication to the python script's proxy.
One of the greatest things about mitmproxy is that it can also modify HTTPs communication. So you have that also :)
Like #Rob w said, I've override XMLHttpRequest and this is a result for modification any XHR requests in any sites (working like transparent modification proxy):
var _open = XMLHttpRequest.prototype.open;
window.XMLHttpRequest.prototype.open = function (method, URL) {
var _onreadystatechange = this.onreadystatechange,
_this = this;
_this.onreadystatechange = function () {
// catch only completed 'api/search/universal' requests
if (_this.readyState === 4 && _this.status === 200 && ~URL.indexOf('api/search/universal')) {
try {
//////////////////////////////////////
// THIS IS ACTIONS FOR YOUR REQUEST //
// EXAMPLE: //
//////////////////////////////////////
var data = JSON.parse(_this.responseText); // {"fields": ["a","b"]}
if (data.fields) {
data.fields.push('c','d');
}
// rewrite responseText
Object.defineProperty(_this, 'responseText', {value: JSON.stringify(data)});
/////////////// END //////////////////
} catch (e) {}
console.log('Caught! :)', method, URL/*, _this.responseText*/);
}
// call original callback
if (_onreadystatechange) _onreadystatechange.apply(this, arguments);
};
// detect any onreadystatechange changing
Object.defineProperty(this, "onreadystatechange", {
get: function () {
return _onreadystatechange;
},
set: function (value) {
_onreadystatechange = value;
}
});
return _open.apply(_this, arguments);
};
for example this code can be used successfully by Tampermonkey for making any modifications on any sites :)
Yes. It is possible with the chrome.debugger API, which grants extension access to the Chrome DevTools Protocol, which supports HTTP interception and modification through its Network API.
This solution was suggested by a comment on Chrome Issue 487422:
For anyone wanting an alternative which is doable at the moment, you can use chrome.debugger in a background/event page to attach to the specific tab you want to listen to (or attach to all tabs if that's possible, haven't tested all tabs personally), then use the network API of the debugging protocol.
The only problem with this is that there will be the usual yellow bar at the top of the tab's viewport, unless the user turns it off in chrome://flags.
First, attach a debugger to the target:
chrome.debugger.getTargets((targets) => {
let target = /* Find the target. */;
let debuggee = { targetId: target.id };
chrome.debugger.attach(debuggee, "1.2", () => {
// TODO
});
});
Next, send the Network.setRequestInterceptionEnabled command, which will enable interception of network requests:
chrome.debugger.getTargets((targets) => {
let target = /* Find the target. */;
let debuggee = { targetId: target.id };
chrome.debugger.attach(debuggee, "1.2", () => {
chrome.debugger.sendCommand(debuggee, "Network.setRequestInterceptionEnabled", { enabled: true });
});
});
Chrome will now begin sending Network.requestIntercepted events. Add a listener for them:
chrome.debugger.getTargets((targets) => {
let target = /* Find the target. */;
let debuggee = { targetId: target.id };
chrome.debugger.attach(debuggee, "1.2", () => {
chrome.debugger.sendCommand(debuggee, "Network.setRequestInterceptionEnabled", { enabled: true });
});
chrome.debugger.onEvent.addListener((source, method, params) => {
if(source.targetId === target.id && method === "Network.requestIntercepted") {
// TODO
}
});
});
In the listener, params.request will be the corresponding Request object.
Send the response with Network.continueInterceptedRequest:
Pass a base64 encoding of your desired HTTP raw response (including HTTP status line, headers, etc!) as rawResponse.
Pass params.interceptionId as interceptionId.
Note that I have not tested any of this, at all.
While Safari has this feature built-in, the best workaround I've found for Chrome so far is to use Cypress's intercept functionality. It cleanly allows me to stub HTTP responses in Chrome. I call cy.intercept then cy.visit(<URL>) and it intercepts and provides a stubbed response for a specific request the visited page makes. Here's an example:
cy.intercept('GET', '/myapiendpoint', {
statusCode: 200,
body: {
myexamplefield: 'Example value',
},
})
cy.visit('http://localhost:8080/mytestpage')
Note: You may also need to configure Cypress to disable some Chrome-specific security settings.
The original question was about Chrome extensions, but I notice that it has branched out into different methods, going by the upvotes on answers that have non-Chrome-extension methods.
Here's a way to kind of achieve this with Puppeteer. Note the caveat mentioned on the originalContent line - the fetched response may be different to the original response in some circumstances.
With Node.js:
npm install puppeteer node-fetch#2.6.7
Create this main.js:
const puppeteer = require("puppeteer");
const fetch = require("node-fetch");
(async function() {
const browser = await puppeteer.launch({headless:false});
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', async (request) => {
let url = request.url().replace(/\/$/g, ""); // remove trailing slash from urls
console.log("REQUEST:", url);
let originalContent = await fetch(url).then(r => r.text()); // TODO: Pass request headers here for more accurate response (still not perfect, but more likely to be the same as the "actual" response)
if(url === "https://example.com") {
request.respond({
status: 200,
contentType: 'text/html; charset=utf-8', // For JS files: 'application/javascript; charset=utf-8'
body: originalContent.replace(/example/gi, "TESTING123"),
});
} else {
request.continue();
}
});
await page.goto("https://example.com");
})();
Run it:
node main.js
With Deno:
Install Deno:
curl -fsSL https://deno.land/install.sh | sh # linux, mac
irm https://deno.land/install.ps1 | iex # windows powershell
Download Chrome for Puppeteer:
PUPPETEER_PRODUCT=chrome deno run -A --unstable https://deno.land/x/puppeteer#16.2.0/install.ts
Create this main.js:
import puppeteer from "https://deno.land/x/puppeteer#16.2.0/mod.ts";
const browser = await puppeteer.launch({headless:false});
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', async (request) => {
let url = request.url().replace(/\/$/g, ""); // remove trailing slash from urls
console.log("REQUEST:", url);
let originalContent = await fetch(url).then(r => r.text()); // TODO: Pass request headers here for more accurate response (still not perfect, but more likely to be the same as the "actual" response)
if(url === "https://example.com") {
request.respond({
status: 200,
contentType: 'text/html; charset=utf-8', // For JS files: 'application/javascript; charset=utf-8'
body: originalContent.replace(/example/gi, "TESTING123"),
});
} else {
request.continue();
}
});
await page.goto("https://example.com");
Run it:
deno run -A --unstable main.js
(I'm currently running into a TimeoutError with this that will hopefully be resolved soon: https://github.com/lucacasonato/deno-puppeteer/issues/65)
Yes, you can modify HTTP response in a Chrome extension. I built ModResponse (https://modheader.com/modresponse) that does that. It can record and replay your HTTP response, modify it, add delay, and even use the HTTP response from a different server (like from your localhost)
The way it works is to use the chrome.debugger API (https://developer.chrome.com/docs/extensions/reference/debugger/), which gives you access to Chrome DevTools Protocol (https://chromedevtools.github.io/devtools-protocol/). You can then intercept the request and response using the Fetch Domain API (https://chromedevtools.github.io/devtools-protocol/tot/Fetch/), then override the response you want. (You can also use the Network Domain, though it is deprecated in favor of the Fetch Domain)
The nice thing about this approach is that it will just work out of box. No desktop app installation required. No extra proxy setup. However, it will show a debugging banner in Chrome (which you can add an argument to Chrome to hide), and it is significantly more complicated to setup than other APIs.
For examples on how to use the debugger API, take a look at the chrome-extensions-samples: https://github.com/GoogleChrome/chrome-extensions-samples/tree/main/mv2-archive/api/debugger/live-headers
I've just found this extension and it does a lot of other things but modifying api responses in the browser works really well: https://requestly.io/
Follow these steps to get it working:
Install the extension
Go to HttpRules
Add a new rule and add a url and a response
Enable the rule with the radio button
Go to Chrome and you should see the response is modified
You can have multiple rules with different responses and enable/disable as required. I've not found out how you can have a different response per request though if the url is the same unfortunately.

Resources