I'm trying to handle proxy authentication though chrome extensions.
On the one hand I have chrome extension (with all permissions established) that sends CONNECT request with onAuthRequired handler (background.js)
chrome.webRequest.onAuthRequired.addListener(
(details, callback) => {
console.log('onAuthRequired', details) // <- this has never been displayed
callback({
authCredentials: {
username: 'someid',
password: 'somepwd'
}
})
},{
urls: ['<all_urls>']
},
['asyncBlocking']
)
const config = {
mode: "pac_script",
pacScript: {
data: "function FindProxyForURL(url, host) {\n if (shExpMatch(host, \"*.pandora.com\"))\n return 'PROXY 127.0.0.1:8124';\n return 'DIRECT';\n }"
}
}
chrome.proxy.settings.set({
value: config,
scope: 'regular',
}, function(){})
And on the other hand I have NodeJS proxy server that always sends the 407 status code as described in the specifications
const http = require('http');
const proxy = http.createServer()
proxy.on('connect', (req, clientSocket, head) => {
clientSocket.write('HTTP/1.1 407 Proxy Authentication Required')
clientSocket.write('Proxy-Authenticate: Basic realm="Access to site"\r\n\n')
});
proxy.listen(8124)
Finally, the browser returns ERR_PROXY_AUTH_UNSUPPORTED which means that the status code is sent correcly...
The fact is onAuthRequired is never triggered, can anyone tell me why ?
Thank you in advance
Chrome aggressively caches the authentication calls to proxy servers. So you will only see your console.log call once per browser session (you need to restart Chrome completely, if you've got more than one profile open you'll need to close ALL instances of Chrome before the authentication cache is cleared).
I'm currently trying to figure out how to clear or empty said cache.
See also (How to delete Proxy-Authorization Cache on Chrome extension?)
Related
A few days ago I've been researching how I can listen for https requests in my browser and overwrite some data (headers, body) from the request before it reaches the final destination (site).
I tried many times. I came to use the modules
mockttp, web-sniffer, hoxy, among others.
As the application I'm developing is with selenium, I tried to create a proxy server with the modules mentioned above and make them go through my application before being "dispatched", however, none of them were successful.
Note: I was able to listen to requests, change their response, and other things. But I didn't get what I really wanted: change the request data before it was sent to the site. Something similar to the breakpoint functionality present in Fiddler, Charles, HTTP Tool Kit and others applications.
What do I really hope?
I'd like to hear my browser's requests -> pause HTTPS requests (breakpoint), change something anyway -> forward/follow/dispatch with/the request.
One of the failed attempts:
The new header is not really inserted in the request! (browser was configured with proxy server and I could see the requests being shown in the console)
import mockttp from "mockttp";
import fs from "fs";
import * as Sniffer from "web-proxy-sniffer";
(async () => {
// Create a proxy server with a self-signed HTTPS CA certificate:
const https = await mockttp.generateCACertificate();
fs.writeFile("key.pem", https.key, console.log);
fs.writeFile("cert.pem", https.cert, console.log);
const proxy = Sniffer.createServer({
certAuthority: {
key: fs.readFileSync(`./key.pem`),
cert: fs.readFileSync(`./cert.pem`),
},
});
proxy.intercept(
{
// Intercept before the request is sent
phase: "request",
},
(request, response) => {
request.headers.test = "a simple test";
return request;
}
);
proxy.listen(8001);
})();
Would this be possible with NodeJS?
(Sorry for this "rude" english)
My application uses an end-to-end testing framework (Cypress) that outputs results in the
terminal. When testing changes to frontend code, I use the Vue dev server's
proxy option to route API requests to a remote backend server.
Due to the high load of requests pummeling this test server, our application
frontend expects certain requests to fall through, and it is able to
intelligently handle retrying a given failed API call. However, the proxy
handler doesn't know this, and as a result the console log becomes cluttered
with ECONNRESET proxy errors, as can be seen below.
Proxy error: Could not proxy request /api/XXXXXX from local.myapp.io:3000 to https://myapp-dev.appspot.com.
See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNRESET).
Proxy error: Could not proxy request /api/XXXXXX from local.myapp.io:3000 to https://myapp-dev.appspot.com.
See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNRESET).
Proxy error: Could not proxy request /api/XXXXXX from local.myapp.io:3000 to https://myapp-dev.appspot.com.
See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNRESET).
Proxy error: Could not proxy request /api/XXXXXX from local.myapp.io:3000 to https://myapp-dev.appspot.com.
See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNRESET).
✓ C22756 should be disabled when no events exist (8018ms)
✓ C22757 should be clickable when events exist (2250ms)
Widget Settings
Proxy error: Could not proxy request /api/XXXXXX from local.myapp.io:3000 to https://myapp-dev.appspot.com.
See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNRESET).
✓ C22758 should have name, date range, and account fields (2435ms)
These errors can get quite verbose and tend to hide/obscure the more laconic
test output, making it difficult to scan and nearly impossible to copy
effectively.
Is there any way to suppress these error messages from the dev server/proxy
handler? Ideally, there would be some form of configuration in an NPM script
or the Vue/Webpack config JS file that would allow me to ignore or hide these
errors just when running our test suite.
For reference, here is the config object being passed to the vue.config.js
file's devServer property:
const config = {
allowedHosts: ['localhost', 'local.myapp.io'],
contentBase: path.join(__dirname, 'root/of/frontend/code'),
disableHostCheck: true,
historyApiFallback: true,
https: {
cert: LocalDevCert.getCert(),
key: LocalDevCert.getKey()
},
port: 3000,
proxy: {
'^(?!/(js|modules|img))': {
target: 'https://myapp-dev.appspot.com',
cookieDomainRewrite: 'local.myapp.io',
changeOrigin: true,
secure: true,
bypass (req, res) {
const xsrfToken = uuid();
if (req.headers.accept && req.headers.accept.includes('html')) {
res.cookie('MyXSRFToken', xsrfToken);
res.cookie('MyXSRFToken_C80', xsrfToken);
return '/index.html';
}
if (req.method === 'GET' && req.url.includes('login') && !req.url.includes('google')) {
res.cookie('MyXSRFToken', xsrfToken);
res.cookie('MyXSRFToken_C80', xsrfToken);
return '/';
}
}
}
},
public: 'local.myapp.io:3000'
};
I have tried:
Using the devServer.client.logging configuration option, but this only changes what is shown in the browser console (not the command-line terminal).
Adding the logLevel configuration option to the corresponding proxy configuration object (i.e. object with key ^(?!/(js|modules|img)) above), but this only seems to increase the amount of logging (info as well as errors).
I suspect something needs to be suppressed at a lower level, and perhaps some bash-script magic is the only plausible solution.
You could add a filter at the top of the test, or globally in /cypress/support/index.js.
Cypress.on('window:before:load', window => {
const originalConsoleLog = window.console.log;
window.console.log = (msg) => {
const isProxyError = msg.includes('ECONNRESET') || msg.startsWith('Proxy error:');
if (!isProxyError) {
originalConsoleLog(msg)
}
}
})
The workaround that I eventually settled on was intercepting process.stdout.write in the Javascript file that configured and ran the dev server/tests. Since our test command is implemented as a Vue CLI plugin, I could just put the override function in that plugin's module.exports as follows:
module.exports = (api, options) => {
api.registerCommand(
'customTestCommand',
{
description: 'A test command for posting on Stack Overflow',
usage: 'vue-cli-service customTestCommand [options]',
options: {
'--headless': 'run in headless mode without GUI'
...
}
},
async (args, rawArgs) => {
// Do other setup work here
process.stdout.write = (function (write) {
let clearBlankLine = false;
return function (...args) {
const string = args[0];
const isProxyError = !!string.match(/Proxy error:|ECONNRESET/);
const hasContent = !!string.trim();
if (isProxyError) {
clearBlankLine = true;
return;
}
if (!clearBlankLine || hasContent) {
write.apply(process.stdout, args);
}
clearBlankLine = false;
};
}(process.stdout.write));
// Wait to run your actual dev server process and test commands
// Until here.
}
);
};
This maintains all coloring, spacing, history clearing, etc. on the command line output, but any line that has the terms Proxy Error or ECONNRESET are removed. You can customize the regex to your specific needs.
For a more detailed example of intercepting both stdout and stderr, as well as redirecting those outputs to different locations (e.g. files), see the following Github gist by Ben Buckman that inspired my solution: https://gist.github.com/benbuckman/2758563
I have been trying to use a service worker within a IIS hosted web site that caches some of the static content of the site. The site is an internal application that uses Windows Authentication. I have been able to register and run a service worker without too much hassle, but as soon as I open the caches and start adding files to the cache, the promise fails with an authorisation failure. the returned HTTP result is 401 Unauthorised. This is the usual response for the first few requests until the browser and the server are able to negotiate the authorisation.
I will post some code soon that should help with the explanation.
EDIT
var staticCacheName = 'app-static-v1';
console.log("I AM ALIVE");
this.addEventListener('install', function (event) {
console.log("AND I INSTALLED!!!!");
var urlsToCache = [
//...many js files to cache
'/scripts/numeral.min.js?version=2.2.0',
'/scripts/require.js',
'/scripts/text.js?version=2.2.0',
'/scripts/toastr.min.js?version=2.2.0',
];
event.waitUntil(
caches.open(staticCacheName).then(function (cache) {
cache.addAll(urlsToCache);
}).catch(function (error) {
console.log(error);
})
);
});
This is just a guess, given the lack of code, but if you're doing something like:
caches.open('my-cache').then(cache => {
return cache.add('page1.html'); // Or caches.addAll(['page1.html, page2.html']);
});
you're taking advantage of the implicit Request object creation (see section 6.4.4.4.1) that happens when you pass in a string to cache.add()/cache.addAll(). The Request object that's created uses the default credentials mode, which is 'omit'.
What you can do instead is explicitly construct a Request object containing the credentials mode you'd prefer, which in your case would likely be 'same-origin':
caches.open('my-cache').then(cache => {
return cache.add(new Request('page1.html', {credentials: 'same-origin'}));
});
If you had a bunch of URLs that you were passing an array to cache.addAll(), you can .map() them to a corresponding array of Requests:
var urls = ['page1.html', 'page2.html'];
caches.open('my-cache').then(cache => {
return cache.addAll(urls.map(url => new Request(url, {credentials: 'same-origin'})));
});
I am debugging a token-based A&A module in node.js + angularJS. In this approach, server authenticates the user and responds with a token. This token is saved in a local memory (client side) and user sends the token on every request to the server using the attached code.
However, I face a weird situation:
I login to www.test.com
Then I can see my dashboard: www.test.com/dashboard
Next, I logout from my dashboard
Afterwards, I open a new tab and type "www.test.com/dashboard" in the address bar of the browser. I expected the user directly go to the login page. However, for 1-2 seconds I see my dashboard (e.g., www.test.com/dashboard) and then it goes to login page !!!! I dont understand why it shows my dashboard for 1-2 seconds !!!
I guess the reason is that developers did not attached the token to GET request! any suggestion to remove the problem?
btw, where is the best location (for both angularJS and EJS) to save the token to be able to send it with all requests to server (ajax, GET, ..., websocket) ?
app.factory('AuthInterceptor', function ($window, $q) {
return {
request: function(config) {
config.headers = config.headers || {};
if ($window.sessionStorage.getItem('token')) {
config.headers.Authorization = 'Bearer ' + $window.sessionStorage.getItem('token');
}
return config || $q.when(config);
},
response: function(response) {
if (response.status === 401) {
// TODO: Redirect user to login page.
}
return response || $q.when(response);
}
};
});
// Register the previously created AuthInterceptor.
app.config(function ($httpProvider) {
$httpProvider.interceptors.push('AuthInterceptor');
Is it possible to create a Chrome extension that modifies HTTP response bodies?
I have looked in the Chrome Extension APIs, but I haven't found anything to do this.
In general, you cannot change the response body of a HTTP request using the standard Chrome extension APIs.
This feature is being requested at 104058: WebRequest API: allow extension to edit response body. Star the issue to get notified of updates.
If you want to edit the response body for a known XMLHttpRequest, inject code via a content script to override the default XMLHttpRequest constructor with a custom (full-featured) one that rewrites the response before triggering the real event. Make sure that your XMLHttpRequest object is fully compliant with Chrome's built-in XMLHttpRequest object, or AJAX-heavy sites will break.
In other cases, you can use the chrome.webRequest or chrome.declarativeWebRequest APIs to redirect the request to a data:-URI. Unlike the XHR-approach, you won't get the original contents of the request. Actually, the request will never hit the server because redirection can only be done before the actual request is sent. And if you redirect a main_frame request, the user will see the data:-URI instead of the requested URL.
I just released a Devtools extension that does just that :)
It's called tamper, it's based on mitmproxy and it allows you to see all requests made by the current tab, modify them and serve the modified version next time you refresh.
It's a pretty early version but it should be compatible with OS X and Windows. Let me know if it doesn't work for you.
You can get it here http://dutzi.github.io/tamper/
How this works
As #Xan commented below, the extension communicates through Native Messaging with a python script that extends mitmproxy.
The extension lists all requests using chrome.devtools.network.onRequestFinished.
When you click on of the requests it downloads its response using the request object's getContent() method, and then sends that response to the python script which saves it locally.
It then opens file in an editor (using call for OSX or subprocess.Popen for windows).
The python script uses mitmproxy to listen to all communication made through that proxy, if it detects a request for a file that was saved it serves the file that was saved instead.
I used Chrome's proxy API (specifically chrome.proxy.settings.set()) to set a PAC as the proxy setting. That PAC file redirect all communication to the python script's proxy.
One of the greatest things about mitmproxy is that it can also modify HTTPs communication. So you have that also :)
Like #Rob w said, I've override XMLHttpRequest and this is a result for modification any XHR requests in any sites (working like transparent modification proxy):
var _open = XMLHttpRequest.prototype.open;
window.XMLHttpRequest.prototype.open = function (method, URL) {
var _onreadystatechange = this.onreadystatechange,
_this = this;
_this.onreadystatechange = function () {
// catch only completed 'api/search/universal' requests
if (_this.readyState === 4 && _this.status === 200 && ~URL.indexOf('api/search/universal')) {
try {
//////////////////////////////////////
// THIS IS ACTIONS FOR YOUR REQUEST //
// EXAMPLE: //
//////////////////////////////////////
var data = JSON.parse(_this.responseText); // {"fields": ["a","b"]}
if (data.fields) {
data.fields.push('c','d');
}
// rewrite responseText
Object.defineProperty(_this, 'responseText', {value: JSON.stringify(data)});
/////////////// END //////////////////
} catch (e) {}
console.log('Caught! :)', method, URL/*, _this.responseText*/);
}
// call original callback
if (_onreadystatechange) _onreadystatechange.apply(this, arguments);
};
// detect any onreadystatechange changing
Object.defineProperty(this, "onreadystatechange", {
get: function () {
return _onreadystatechange;
},
set: function (value) {
_onreadystatechange = value;
}
});
return _open.apply(_this, arguments);
};
for example this code can be used successfully by Tampermonkey for making any modifications on any sites :)
Yes. It is possible with the chrome.debugger API, which grants extension access to the Chrome DevTools Protocol, which supports HTTP interception and modification through its Network API.
This solution was suggested by a comment on Chrome Issue 487422:
For anyone wanting an alternative which is doable at the moment, you can use chrome.debugger in a background/event page to attach to the specific tab you want to listen to (or attach to all tabs if that's possible, haven't tested all tabs personally), then use the network API of the debugging protocol.
The only problem with this is that there will be the usual yellow bar at the top of the tab's viewport, unless the user turns it off in chrome://flags.
First, attach a debugger to the target:
chrome.debugger.getTargets((targets) => {
let target = /* Find the target. */;
let debuggee = { targetId: target.id };
chrome.debugger.attach(debuggee, "1.2", () => {
// TODO
});
});
Next, send the Network.setRequestInterceptionEnabled command, which will enable interception of network requests:
chrome.debugger.getTargets((targets) => {
let target = /* Find the target. */;
let debuggee = { targetId: target.id };
chrome.debugger.attach(debuggee, "1.2", () => {
chrome.debugger.sendCommand(debuggee, "Network.setRequestInterceptionEnabled", { enabled: true });
});
});
Chrome will now begin sending Network.requestIntercepted events. Add a listener for them:
chrome.debugger.getTargets((targets) => {
let target = /* Find the target. */;
let debuggee = { targetId: target.id };
chrome.debugger.attach(debuggee, "1.2", () => {
chrome.debugger.sendCommand(debuggee, "Network.setRequestInterceptionEnabled", { enabled: true });
});
chrome.debugger.onEvent.addListener((source, method, params) => {
if(source.targetId === target.id && method === "Network.requestIntercepted") {
// TODO
}
});
});
In the listener, params.request will be the corresponding Request object.
Send the response with Network.continueInterceptedRequest:
Pass a base64 encoding of your desired HTTP raw response (including HTTP status line, headers, etc!) as rawResponse.
Pass params.interceptionId as interceptionId.
Note that I have not tested any of this, at all.
While Safari has this feature built-in, the best workaround I've found for Chrome so far is to use Cypress's intercept functionality. It cleanly allows me to stub HTTP responses in Chrome. I call cy.intercept then cy.visit(<URL>) and it intercepts and provides a stubbed response for a specific request the visited page makes. Here's an example:
cy.intercept('GET', '/myapiendpoint', {
statusCode: 200,
body: {
myexamplefield: 'Example value',
},
})
cy.visit('http://localhost:8080/mytestpage')
Note: You may also need to configure Cypress to disable some Chrome-specific security settings.
The original question was about Chrome extensions, but I notice that it has branched out into different methods, going by the upvotes on answers that have non-Chrome-extension methods.
Here's a way to kind of achieve this with Puppeteer. Note the caveat mentioned on the originalContent line - the fetched response may be different to the original response in some circumstances.
With Node.js:
npm install puppeteer node-fetch#2.6.7
Create this main.js:
const puppeteer = require("puppeteer");
const fetch = require("node-fetch");
(async function() {
const browser = await puppeteer.launch({headless:false});
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', async (request) => {
let url = request.url().replace(/\/$/g, ""); // remove trailing slash from urls
console.log("REQUEST:", url);
let originalContent = await fetch(url).then(r => r.text()); // TODO: Pass request headers here for more accurate response (still not perfect, but more likely to be the same as the "actual" response)
if(url === "https://example.com") {
request.respond({
status: 200,
contentType: 'text/html; charset=utf-8', // For JS files: 'application/javascript; charset=utf-8'
body: originalContent.replace(/example/gi, "TESTING123"),
});
} else {
request.continue();
}
});
await page.goto("https://example.com");
})();
Run it:
node main.js
With Deno:
Install Deno:
curl -fsSL https://deno.land/install.sh | sh # linux, mac
irm https://deno.land/install.ps1 | iex # windows powershell
Download Chrome for Puppeteer:
PUPPETEER_PRODUCT=chrome deno run -A --unstable https://deno.land/x/puppeteer#16.2.0/install.ts
Create this main.js:
import puppeteer from "https://deno.land/x/puppeteer#16.2.0/mod.ts";
const browser = await puppeteer.launch({headless:false});
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', async (request) => {
let url = request.url().replace(/\/$/g, ""); // remove trailing slash from urls
console.log("REQUEST:", url);
let originalContent = await fetch(url).then(r => r.text()); // TODO: Pass request headers here for more accurate response (still not perfect, but more likely to be the same as the "actual" response)
if(url === "https://example.com") {
request.respond({
status: 200,
contentType: 'text/html; charset=utf-8', // For JS files: 'application/javascript; charset=utf-8'
body: originalContent.replace(/example/gi, "TESTING123"),
});
} else {
request.continue();
}
});
await page.goto("https://example.com");
Run it:
deno run -A --unstable main.js
(I'm currently running into a TimeoutError with this that will hopefully be resolved soon: https://github.com/lucacasonato/deno-puppeteer/issues/65)
Yes, you can modify HTTP response in a Chrome extension. I built ModResponse (https://modheader.com/modresponse) that does that. It can record and replay your HTTP response, modify it, add delay, and even use the HTTP response from a different server (like from your localhost)
The way it works is to use the chrome.debugger API (https://developer.chrome.com/docs/extensions/reference/debugger/), which gives you access to Chrome DevTools Protocol (https://chromedevtools.github.io/devtools-protocol/). You can then intercept the request and response using the Fetch Domain API (https://chromedevtools.github.io/devtools-protocol/tot/Fetch/), then override the response you want. (You can also use the Network Domain, though it is deprecated in favor of the Fetch Domain)
The nice thing about this approach is that it will just work out of box. No desktop app installation required. No extra proxy setup. However, it will show a debugging banner in Chrome (which you can add an argument to Chrome to hide), and it is significantly more complicated to setup than other APIs.
For examples on how to use the debugger API, take a look at the chrome-extensions-samples: https://github.com/GoogleChrome/chrome-extensions-samples/tree/main/mv2-archive/api/debugger/live-headers
I've just found this extension and it does a lot of other things but modifying api responses in the browser works really well: https://requestly.io/
Follow these steps to get it working:
Install the extension
Go to HttpRules
Add a new rule and add a url and a response
Enable the rule with the radio button
Go to Chrome and you should see the response is modified
You can have multiple rules with different responses and enable/disable as required. I've not found out how you can have a different response per request though if the url is the same unfortunately.