I develop small crawler with puppeteer in Node.js.
the target site has Flash contents, so I want to enable Flash in puppeteer.
by default, puppeteer is unable to use Flash and white list of permitted sites is empty.
I know how to enable Flash in puppeteer, but I don't know how to set white list.
How to do that?
Is there flag like this?
const browser = await puppeteer.launch({
headless: false,
args: [
'--ppapi-flash-path = {FLASH_PATH}',
'--white-url = {TARGET_URL}'
]
});
Or, is it only way to simply manipulate DOM in setting page in browser (ex.chrome://settings/content/flash)?
I've solved the problem myself.
I can't find the flag of Flash white list and manipulate DOM of chrome's setting page is too tiring for me, but I got useful some params.
1. 'userDataDir: {PROFILE_FILE}'
First, manually launch chrome or chromium, and add url to the list.
Then, set PROFILE_FILE as its user data path.
2. executablePath: '{PATH_TO_CHROME}'
Basically, run on chrome but not chromium, we can use Flash and HLS and etc. by default.
Set manually white list, and it works.
After much tinkering I found a Preferences file configuration that whitelists all flash content by default.
I've made a small puppeteer wrapper to make using this very easy: puppeteer.setExtra({allowFlash: true})
{
"profile": {
"content_settings": {
"exceptions": {
"flash_data": {
"*,*": {
"setting": {
"flashPreviouslyChanged": true
}
}
},
"permission_autoblocking_data": {
"*,*": {
"setting": {
"Flash": {
"dismiss_count": 1
}
}
}
},
"plugins": {
"*,*": {
"per_resource": {
"adobe-flash-player": 1
}
}
}
},
"pref_version": 1
}
}
}
Related
After the Chrome extension I'm working on is installed, or upgraded, the content scripts (specified in the manifest) are not re-injected so a page refresh is required to make the extension work. Is there a way to force the scripts to be injected again?
I believe I could inject them again programmatically by removing them from the manifest and then handling which pages to inject in the background page, but this is not a good solution.
I don't want to automatically refresh the user's tabs because that could lose some of their data. Safari automatically refreshes all pages when you install or upgrade an extension.
There's a way to allow a content script heavy extension to continue functioning after an upgrade, and to make it work immediately upon installation.
Install/upgrade
The install method is to simply iterate through all tabs in all windows, and inject some scripts programmatically into tabs with matching URLs.
ManifestV3
manifest.json:
"background": {"service_worker": "background.js"},
"permissions": ["scripting"],
"host_permissions": ["<all_urls>"],
These host_permissions should be the same as the content script's matches.
background.js:
chrome.runtime.onInstalled.addListener(async () => {
for (const cs of chrome.runtime.getManifest().content_scripts) {
for (const tab of await chrome.tabs.query({url: cs.matches})) {
chrome.scripting.executeScript({
target: {tabId: tab.id},
files: cs.js,
});
}
}
});
This is a simplified example that doesn't handle frames. You can use getAllFrames API and match the URLs yourself, see the documentation for matching patterns.
ManifestV2
Obviously, you have to do it in a background page or event page script declared in manifest.json:
"background": {
"scripts": ["background.js"]
},
background.js:
// Add a `manifest` property to the `chrome` object.
chrome.manifest = chrome.runtime.getManifest();
var injectIntoTab = function (tab) {
// You could iterate through the content scripts here
var scripts = chrome.manifest.content_scripts[0].js;
var i = 0, s = scripts.length;
for( ; i < s; i++ ) {
chrome.tabs.executeScript(tab.id, {
file: scripts[i]
});
}
}
// Get all windows
chrome.windows.getAll({
populate: true
}, function (windows) {
var i = 0, w = windows.length, currentWindow;
for( ; i < w; i++ ) {
currentWindow = windows[i];
var j = 0, t = currentWindow.tabs.length, currentTab;
for( ; j < t; j++ ) {
currentTab = currentWindow.tabs[j];
// Skip chrome:// and https:// pages
if( ! currentTab.url.match(/(chrome|https):\/\//gi) ) {
injectIntoTab(currentTab);
}
}
}
});
Historical trivia
In ancient Chrome 26 and earlier content scripts could restore connection to the background script. It was fixed http://crbug.com/168263 in 2013. You can see an example of this trick in the earlier revisions of this answer.
The only way to force a content script to be injected without refreshing the page is via programatic injection.
You can get all tabs and inject code into them using the chrome tabs API.
For example you can store a manifest version in local storage and every time check if the manifest version is old one (in background page), if so you can get all active tabs and inject your code programmatically, or any other solution that will make you sure that the extension is updated.
Get all tabs using:
chrome.tabs.query
and inject your code into all pages
chrome.tabs.executeScript(tabId, {file: "content_script.js"});
Try this in your background script. Many of the old methods have been deprecated now, so I have refactored the code. For my use I'm only installing single content_script file. If need you can iterate over
chrome.runtime.getManifest().content_scripts array to get all .js files.
chrome.runtime.onInstalled.addListener(installScript);
function installScript(details){
// console.log('Installing content script in all tabs.');
let params = {
currentWindow: true
};
chrome.tabs.query(params, function gotTabs(tabs){
let contentjsFile = chrome.runtime.getManifest().content_scripts[0].js[0];
for (let index = 0; index < tabs.length; index++) {
chrome.tabs.executeScript(tabs[index].id, {
file: contentjsFile
},
result => {
const lastErr = chrome.runtime.lastError;
if (lastErr) {
console.error('tab: ' + tabs[index].id + ' lastError: ' + JSON.stringify(lastErr));
}
})
}
});
}
Chrome has added a method to listen for the install or upgrade event of the extension. One can re-inject the content script when such an event occur.
https://developers.chrome.com/extensions/runtime#event-onInstalled
Due to https://bugs.chromium.org/p/chromium/issues/detail?id=168263, the connection between your content script and background script is severed. As others have mentioned, one way to get around this issue is by reinjecting a content script. A rough overview is detailed in this StackOverflow answer.
The main tricky part is that it's necessary to "destruct" your current content script before injecting a new content script. Destructing can be really tricky, so one way to reduce the amount of state you must destruct is by making a small reinjectable script, that talks to your main content script over the DOM.
can't you add ?ver=2.10 at the end of css or js you upgraded?
"content_scripts": [ {
"css": [ "css/cs.css?ver=2.10" ],
"js": [ "js/contentScript.js?ver=2.10" ],
"matches": [ "http://*/*", "https://*/*" ],
"run_at": "document_end"
} ],
I'm writing a chrome extension and I want to either disable the popup entirely, or present a message if the url of the current tab is not permitted via the v3 manifest's host_permissions property.
This is for a dev support tool and we don't want it enabled for the production url. So if I set:
"host_permissions": [
"http://localhost:3000/",
"https://*.dev.some-site.com/",
"https://www.uat.some-site.com/"
]
...Then if the user is on www.some-site.com (or anywhere else), I want the popup to be disabled.
I can obtain the relevant url easily enough:
let [currentTab] = await chrome.tabs.query({ active: true, currentWindow: true });
const { url } = currentTab;
const cookieUrl = url.match(/https?:\/\/[^/]+\//)?.[0];
...and I can obtain the array of permitted file patterns with
chrome.runtime.getManifest().host_permissions
But how can I use this or anything else to prevent the popup? Also, converting that wild-card into a genuine regex will be a bit of a pain. Isn't there some out-of-the-box method to do all this??
Use chrome.declarativeContent API to enable the action popup for sites in host_permissions and disable it everywhere else by default.
When disabled, the icon becomes grayed out. Clicking it will show the built-in context menu, chrome.action.onClicked won't be triggered. The name of the extension in the puzzle-piece menu is also grayed out.
manifest.json:
"action": {"default_icon": "icon.png"},
"permissions": ["declarativeContent"],
background script:
chrome.action.disable();
chrome.runtime.onInstalled.addListener(() => {
chrome.declarativeContent.onPageChanged.removeRules(() => {
chrome.declarativeContent.onPageChanged.addRules([{
conditions: chrome.runtime.getManifest().host_permissions.map(h => {
const [, sub, host] = h.match(/:\/\/(\*\.)?([^/]+)/);
return new chrome.declarativeContent.PageStateMatcher({
pageUrl: sub ? {hostSuffix: '.' + host} : {hostEquals: host},
});
}),
actions: [new chrome.declarativeContent.ShowAction()],
}]);
});
});
For simplicity, I don't limit PageStateMatcher's schemes.
Need to be done: I need to simulate user interactions (journeys) across a chain of sites.
Question: Do you have any tips how to programatically controll a tab opened as a result of a simulated click?
My experience:
I'm using the chrome-remote-interface npm package.
I'm able to simulate a click with a custom ChromeController class which initializes the chrome-remote-interface and these methods:
async simulateClick(selector) {
return await this.evaluate(function (selector) {
document.querySelector(selector).click()
}, selector);
}
/**
* Shamelessly stolen from simple-headless-browser
*/
async evaluate (fn, ...args) {
const exp = args && args.length > 0 ? `(${String(fn)}).apply(null, ${JSON.stringify(args)})` : `(${String(fn)}).apply(null)`
const result = await this.client.Runtime.evaluate({
expression: exp,
returnByValue: true
})
return result
}
Now I would like to interact with the recently opened tab. I can get the targetId of the new tab with the experimenetal Target Domain (prototyping in node cli):
var targets;
chromeController.client.Target.getTargets().then(t => targets = t);
Which results in:
{ targetInfos:
[ { targetId: '97556479-cdb6-415c-97a1-6efa4e00b281',
type: 'page',
title: 'xxx/preview/239402/',
url: 'xxx/preview/239402/' },
{ targetId: 'bbfe11d5-8e4a-4879-9081-10bb7234209c',
type: 'page',
title: 'Document',
url: 'xxx/preview/239402/teaser/0/' } ] }
I am able to switch between the tabs with:
chromeController.client.Target.activateTarget({targetId:'xxx'})
However I'm not able to get any interaction with this, I can't find the connection, how to load it into the Page and Runtime objects.
I've searched in the docs and also tried googling: 'site:chromedevtools.github.io targetId' which only lead me to
> chromeController.client.Browser.getWindowForTarget({targetId: '97556479-cdb6-415c-97a1-6efa4e00b281'}).catch(e => console.log(e.message));
Promise { <pending> }
> 'Browser.getWindowForTarget' wasn't found
I've also tried to Target.setDiscoverTargets({discover: true}) and to close the original tab.
Thanks for any help!
Recently faced this same issue and in short I had to create a new dev tools protocol client for each new target I wanted control over.
My experience is with dev tools protocol using direct communication with websocket but the api is the same so it should be similar. So here is a summary of what I had to do.
Initially looking at the docs I would have assumed Target.attachToTarget should give us control of the new tab but I found that it didn't work.
My workaround was to create a listener that listened for the Target.targetCreated event which provides a targetInfos just like you found with Target.getTargets but for every new target created like a new tab, page, or iframe. Note: you need to enable Target.setDiscoverTargets in order to receive these events over the protocol.
[ { targetId: '97556479-cdb6-415c-97a1-6efa4e00b281',
type: 'page',
title: 'xxx/preview/239402/',
url: 'xxx/preview/239402/' },
{ targetId: 'bbfe11d5-8e4a-4879-9081-10bb7234209c',
type: 'page',
title: 'Document',
url: 'xxx/preview/239402/teaser/0/' } ] }
With that listener I looked for targets that were of type page, you could filter on a specific url if you know what the page will be. With the targetId in hand I requested available websocket targets following the HTTPEndpoints section near the bottom of the devtools home page.
GET /json or /json/list
A list of all available websocket targets.
[ {
"description": "",
"devtoolsFrontendUrl": "/devtools/inspector.html?ws=localhost:9222/devtools/page/DAB7FB6187B554E10B0BD18821265734",
"id": "DAB7FB6187B554E10B0BD18821265734",
"title": "Yahoo",
"type": "page",
"url": "https://www.yahoo.com/",
"webSocketDebuggerUrl": "ws://localhost:9222/devtools/page/DAB7FB6187B554E10B0BD18821265734"
} ]
I could then launch a new dev tools protocol client using the webSocketDebuggerUrl and have full control over the tab.
I know this is a pretty round about way but, its the only way I was able to make if work.
Although these days it's probably easier to use something like puppeteer to interface with multiple tabs in chrome if you can. Here is the source code to a puppeteer module that follows new tabs that could be good reference for trying to replicate it pageVideoStreamCollector.ts
This is a very late answer but just putting this here if anyone else has the same issue as help on chrome dev tools is very hard to come by. Hope it helps someone out.
I also am getting "Browser.getWindowForTarget wasn't found" on debian, google-chrome-unstable version 61
var uploader = new qq.azure.FineUploaderBasic({
debug: true,
element: document.getElementById("fine-uploader-gallery"),
signature: {
endpoint: '#Url.Action("GetUploadSASUrl", "Upload")'
},
uploadSuccess: {
endpoint: '#Url.Action("ProcessImage", "Upload")'
},
scaling: {
sendOriginal: false,
sizes: [
{ name: "", maxSize: 800 }
]
},
validation: {
allowedExtensions: ['jpeg', 'jpg', 'png']
}
});
</script>
Trying to create a FineUploader Basic instance for Azure but nothing shows up. What am I doing wrong?
I've added a debug: true directive but nothing shows up in the console. The initial script tag is there and a div with the id of "fine-uploader-gallery" is there. All the scripts and CSS are on the page.
Why would you expect something to "show up"? You are using Fine Uploader's "core" mode, which assumes you are providing the entire UI yourself and simply making use of the API, options, and events to drive it. If you want to render a default but customizable UI, you should use UI mode instead. More information about all of these can be found on the docs site at http://docs.fineuploader.com. Before you go any further, you should spend some time familiarizing yourself with these options.
After the Chrome extension I'm working on is installed, or upgraded, the content scripts (specified in the manifest) are not re-injected so a page refresh is required to make the extension work. Is there a way to force the scripts to be injected again?
I believe I could inject them again programmatically by removing them from the manifest and then handling which pages to inject in the background page, but this is not a good solution.
I don't want to automatically refresh the user's tabs because that could lose some of their data. Safari automatically refreshes all pages when you install or upgrade an extension.
There's a way to allow a content script heavy extension to continue functioning after an upgrade, and to make it work immediately upon installation.
Install/upgrade
The install method is to simply iterate through all tabs in all windows, and inject some scripts programmatically into tabs with matching URLs.
ManifestV3
manifest.json:
"background": {"service_worker": "background.js"},
"permissions": ["scripting"],
"host_permissions": ["<all_urls>"],
These host_permissions should be the same as the content script's matches.
background.js:
chrome.runtime.onInstalled.addListener(async () => {
for (const cs of chrome.runtime.getManifest().content_scripts) {
for (const tab of await chrome.tabs.query({url: cs.matches})) {
chrome.scripting.executeScript({
target: {tabId: tab.id},
files: cs.js,
});
}
}
});
This is a simplified example that doesn't handle frames. You can use getAllFrames API and match the URLs yourself, see the documentation for matching patterns.
ManifestV2
Obviously, you have to do it in a background page or event page script declared in manifest.json:
"background": {
"scripts": ["background.js"]
},
background.js:
// Add a `manifest` property to the `chrome` object.
chrome.manifest = chrome.runtime.getManifest();
var injectIntoTab = function (tab) {
// You could iterate through the content scripts here
var scripts = chrome.manifest.content_scripts[0].js;
var i = 0, s = scripts.length;
for( ; i < s; i++ ) {
chrome.tabs.executeScript(tab.id, {
file: scripts[i]
});
}
}
// Get all windows
chrome.windows.getAll({
populate: true
}, function (windows) {
var i = 0, w = windows.length, currentWindow;
for( ; i < w; i++ ) {
currentWindow = windows[i];
var j = 0, t = currentWindow.tabs.length, currentTab;
for( ; j < t; j++ ) {
currentTab = currentWindow.tabs[j];
// Skip chrome:// and https:// pages
if( ! currentTab.url.match(/(chrome|https):\/\//gi) ) {
injectIntoTab(currentTab);
}
}
}
});
Historical trivia
In ancient Chrome 26 and earlier content scripts could restore connection to the background script. It was fixed http://crbug.com/168263 in 2013. You can see an example of this trick in the earlier revisions of this answer.
The only way to force a content script to be injected without refreshing the page is via programatic injection.
You can get all tabs and inject code into them using the chrome tabs API.
For example you can store a manifest version in local storage and every time check if the manifest version is old one (in background page), if so you can get all active tabs and inject your code programmatically, or any other solution that will make you sure that the extension is updated.
Get all tabs using:
chrome.tabs.query
and inject your code into all pages
chrome.tabs.executeScript(tabId, {file: "content_script.js"});
Try this in your background script. Many of the old methods have been deprecated now, so I have refactored the code. For my use I'm only installing single content_script file. If need you can iterate over
chrome.runtime.getManifest().content_scripts array to get all .js files.
chrome.runtime.onInstalled.addListener(installScript);
function installScript(details){
// console.log('Installing content script in all tabs.');
let params = {
currentWindow: true
};
chrome.tabs.query(params, function gotTabs(tabs){
let contentjsFile = chrome.runtime.getManifest().content_scripts[0].js[0];
for (let index = 0; index < tabs.length; index++) {
chrome.tabs.executeScript(tabs[index].id, {
file: contentjsFile
},
result => {
const lastErr = chrome.runtime.lastError;
if (lastErr) {
console.error('tab: ' + tabs[index].id + ' lastError: ' + JSON.stringify(lastErr));
}
})
}
});
}
Chrome has added a method to listen for the install or upgrade event of the extension. One can re-inject the content script when such an event occur.
https://developers.chrome.com/extensions/runtime#event-onInstalled
Due to https://bugs.chromium.org/p/chromium/issues/detail?id=168263, the connection between your content script and background script is severed. As others have mentioned, one way to get around this issue is by reinjecting a content script. A rough overview is detailed in this StackOverflow answer.
The main tricky part is that it's necessary to "destruct" your current content script before injecting a new content script. Destructing can be really tricky, so one way to reduce the amount of state you must destruct is by making a small reinjectable script, that talks to your main content script over the DOM.
can't you add ?ver=2.10 at the end of css or js you upgraded?
"content_scripts": [ {
"css": [ "css/cs.css?ver=2.10" ],
"js": [ "js/contentScript.js?ver=2.10" ],
"matches": [ "http://*/*", "https://*/*" ],
"run_at": "document_end"
} ],