How do I set proxy in phantomjs - node.js

This https://www.npmjs.com/package/phantom#functionality-details page says:
You can also pass command line switches to the phantomjs process by specifying additional args to phantom.create(), eg:
phantom.create '--load-images=no', '--local-to-remote-url-access=yes', (page) ->
or by specifying them in the options* object:
phantom.create {parameters: {'load-images': 'no', 'local-to-remote-url-access': 'yes'}}, (page) ->
These examples are only in coffee script and also they insinuate that the create function can take
create('string',function)
or
create([object object],function)
but really the first parameter expected is the function!
I really wanted to try http://phantomjs.org/api/command-line.html I might have the wrong idea but to me it looks like they can be used in the create function (right before you do the createPage), am I wrong?
I have tried several things, the most logical one is this:
var phantom = require('phantom');
phantom.create(function(browser){
browser.createPage(function(page){
page.open('http://example.com/req.php', function() {
});},{parameters:{'proxy':'98.239.198.83:21320'}});});
So the page gets opened. I know this because I am making req.php save the $_SERVER object to a txt pad but, the REMOTE_ADDR and REMOTE_PORT headers are not the ones in the proxy I have set. They have no effect. I have also tried:
{options:{'proxy':'98.239.198.83:21320'}}
As the docs call that object the options* object *see above^
and
'--proxy=98.239.198.83:21320'
I have also had a dig through the phantom module to find the create function. It is not written in js I can't see it at least. It must be in C++. It looks like this module has been updated but, the examples deep inside the module look like old code.
How do I do this?
EDIT:
var phantom = require('phantom');
phantom.create(function(browser){
browser.createPage(function(page){
browser.setProxy('98.239.198.83','21320','http', null, null, function(){
page.open(
'http://example.com/req.php', function() {
});});});});
This produces no error and the page gets scraped but the proxy is ignored.

As for as phantom 2.0.10 version the following code is running very well in my windows machine
phantom.create(["--proxy=201.172.242.184:15124", "--proxy-type=socks5"])
.then((instance) => {
phInstance = instance;
return instance.createPage();
})
.then((page) => {
sitepage = page;
return page.open('http://newsdaily.online');
})
.then((status) => {
console.log(status);
return sitepage.property('title');
})
.then((content) => {
console.log(content);
sitepage.close();
phInstance.exit();
})
.catch((error) => {
console.log(error);
phInstance.exit();
});

{ parameters: { 'proxy': 'socks://98.239.198.83:21320' } }
They didn't update their docs.

Time is going on, so PhantomJS now able to set proxy "on the fly" (even on per-page-basis): see this commit: https://github.com/ariya/phantomjs/commit/efd8dedfb574c15ddaac26ae72690fc2031e6749
Here is sample usage of new setProxy function (i did not find web page setting usage, this is general usage of proxy on instance of phantom):
https://github.com/ariya/phantomjs/blob/master/examples/openurlwithproxy.js
If you want per-page proxy, use full URL for proxy (schema, user name,password, host, port - all it the URL)

As a side effect of trying to figure out an issue on Github for phantomjs-nodejs I was able to set a proxy as follows:
phantom = require 'phantom'
parameters = {
loadimages: '--load-images=no',
websecurity: '--web-security=no',
ignoresslerrors: '--ignore-ssl-errors=yes',
proxy: '--proxy=10.0.1.235:8118',
}
urls = {
checktor: "https://check.torproject.org/",
google: "https://google.com",
}
phantom.create parameters.websecurity, parameters.proxy, (ph) ->
ph.createPage (page) ->
page.open urls.checktor, (status) ->
console.log "page opened? ", status
page.evaluate (-> document.title), (result) ->
console.log 'Page title is ' + result
ph.exit()
The result where the proxy uses Tor was:
page opened? success
Page title is Congratulations. This browser is configured to use Tor.

use phantom npm package and co npm package.
co(function*() {
const phantomInstance = yield phantom.create(["--proxy=171.13.36.64:808"]);
crawScheduler.start(phantomInstance);
});

I'm running PhantomJS from windows cmd and syntaxes it use looks bit different from what's I notice if you didn't put http:// PJ wont recognize the value this is complete example
var page = require('webpage').create();
page.settings.loadImages = false; //
page.settings.proxy = 'http://192.168.1.5:8080' ;
page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36';
page.open('http://some.com/page', function() {
page.render('some.png');
phantom.exit();
});

Yet another solution for nodejs:
const phantomInstance = await require('phantom').create();
const page = await phantomInstance.createPage();
// get current settings:
var pageSettings = await page.property('settings');
/*
{
XSSAuditingEnabled: false,
javascriptCanCloseWindows: true,
javascriptCanOpenWindows: true,
javascriptEnabled: true,
loadImages: true,
localToRemoteUrlAccessEnabled: false,
userAgent: 'Mozilla/5.0 (Unknown; Linux x86_64) ... PhantomJS/2.1.1 Safari/538.1',
webSecurityEnabled: true
}
*/
pageSettings.proxy = 'https://78.40.87.18:808';
// update settings (return value is undefined):
await page.property('settings', pageSettings);
const status = await page.open('https://2ip.ru/');
// show IP:
var ip = await page.evaluate(function () {
var el = document.getElementById('d_clip_button');
return !el ? '?' : el.textContent;
});
console.log('IP:', ip);
It's an option to set proxy within specific page.

The CoffeeScript example is a little strange, because it is the browser that is passed into the callback of phantom.create and not page, but otherwise it must be compatible judging by the code.
var phantom = require('phantom');
phantom.create({
parameters: {
proxy: '98.239.198.83:21320'
}
}, function(browser){
browser.createPage(function(page){
page.open('http://example.com/req.php', function() {
...
});
});
});
Proxy settings are set during process creation, not during page opening. Although PhantomJS contains an undocumented phantom.setProxy() function which enables you to change the proxy settings in the middle of the script. The phantom module also seems to support it.

var phantom = require('phantom');
phantom.create(function (browser) {
browser.setProxy(proxyIP, proxyPort);
page.open(url, function (status) {
console.log(status);
});
},{dnodeOpts:{weak: false}});
it works fine on my windows.

Related

Netlify functions not found on dev server

I have followed several blogs and questions related to the same problem I am having. It is exactly like this question. However, I am still having issues.
So, I am running netlify dev and trying to access my netlify functions. I have a function in /netlify/functions/ping. The function works as intended when I access the randomized port for the netlify functions (something like localhost:55832...).
However, using the localhost:8888/.netlify/functions/ping gives me a 404 error.
Here is my /netlify/functions/ping file:
import { Handler } from '#netlify/functions';
const handler: Handler = async (event, context) => {
return {
statusCode: 200,
body: JSON.stringify({ data: "pong" }),
};
};
export { handler };
here is where I am trying to call my function on a page
export default function HomePage() {
useEffect(() => {
async function pingpong() {
const res = await fetch(`/.netlify/functions/ping`);
console.log(res);
}
pingpong();
}, []);
return (
...
I have also tried to alter my netlify.toml with the following
[[redirects]]
from = "/api/*"
to = "/.netlify/functions/:splat"
status = 200
force = true
what's start script you use in package.json?
Keep in mind that, to enable Netlify Functions you have to use netlify-cli, i.e. ntl command to run on local dev server.
No more specific configuration, just follow docs, and use that simple netlify/functions/hello.js example.
Then run using ntl dev, you function will be avaiable on /.netlify/functions/hello. easy.

Jest puppeteer typescript reference error Page not found

Trying to setup typescript jest with puppeteer
i following step by step instructions as mentioned below
Jest-puppeteer with typescript configuration
there is a simple test
describe('Google', () => {
beforeAll(async () => {
await page.goto('https://google.com')
})
it('should display "google" text on page', async () => {
await expect(page).toMatch('google')
})
})
when i run my test i get weird error
ReferenceError: page is not defined
and it is pointing to the 'await page' object inside beforeAll
i also notice chrome try to kick in and but does not launch may be this error is b/c chrome could not launch.
jest-puppeteer library is responsible for launching a browser and providing browser and page objects
and here is the code taken from the page mentioned above in link
//jest-puppeteer.config.js
let jest_puppeteer_conf = {
launch: {
timeout: 30000,
dumpio: true // Whether to pipe the browser process stdout and stderr
}
}
const isDebugMode = typeof v8debug === 'object' || /--debug|--inspect/.test(process.execArgv.join(' '));
if (isDebugMode) {
jest_puppeteer_conf.launch.headless = false; // for debug: to see what the browser is displaying
jest_puppeteer_conf.launch.slowMo = 250; // slow down by 250ms for each step
jest_puppeteer_conf.launch.devtools = true; // This lets you debug code in the application code browser
jest_puppeteer_conf.launch.args = [ '--start-maximized' ]; // maximise the screen
}
module.exports = jest_puppeteer_conf;
there is a small debug section at the bottom which reminds to add following types i already have them and still no luck, any help is appreciated.
"compilerOptions": {
.....
"types": [
.......
"puppeteer",
"jest-environment-puppeteer",
"expect-puppeteer"
]
}
commenting out the following line fix the issue.
// testEnvironment: "node"

How can I force external links from browser-window to open in a default browser from Electron?

I'm using the BrowserWindow to display an app and I would like to force the external links to be opened in the default browser. Is that even possible or I have to approach this differently?
I came up with this, after checking the solution from the previous answer.
mainWindow.webContents.on('new-window', function(e, url) {
e.preventDefault();
require('electron').shell.openExternal(url);
});
According to the electron spec, new-window is fired when external links are clicked.
NOTE: Requires that you use target="_blank" on your anchor tags.
new-window is now deprecated in favor of setWindowOpenHandler in Electron 12 (see https://github.com/electron/electron/pull/24517).
So a more up to date answer would be:
mainWindow.webContents.setWindowOpenHandler(({ url }) => {
shell.openExternal(url);
return { action: 'deny' };
});
Improved from the accepted answer ;
the link must be target="_blank" ;
add in background.js(or anywhere you created your window) :
window.webContents.on('new-window', function(e, url) {
// make sure local urls stay in electron perimeter
if('file://' === url.substr(0, 'file://'.length)) {
return;
}
// and open every other protocols on the browser
e.preventDefault();
shell.openExternal(url);
});
Note : To ensure this behavior across all application windows, this code should be run after each window creation.
If you're not using target="_blank" in your anchor elements, this might work for you:
const shell = require('electron').shell;
$(document).on('click', 'a[href^="http"]', function(event) {
event.preventDefault();
shell.openExternal(this.href);
});
I haven't tested this but I assume this is should work:
1) Get WebContents of the your BrowserWindow
var wc = browserWindow.webContents;
2) Register for will-navigate of WebContent and intercept navigation/link clicks:
wc.on('will-navigate', function(e, url) {
/* If url isn't the actual page */
if(url != wc.getURL()) {
e.preventDefault();
openBrowser(url);
}
}
3) Implement openBrowser using child_process. An example for Linux desktops:
var openBrowser(url) {
require('child_process').exec('xdg-open ' + url);
}
let me know if this works for you!
For anybody coming by.
My use case:
I was using SimpleMDE in my app and it's preview mode was opening links in the same window. I wanted all links to open in the default OS browser. I put this snippet, based on the other answers, inside my main.js file. It calls it after it creates the new BrowserWindow instance. My instance is called mainWindow
let wc = mainWindow.webContents
wc.on('will-navigate', function (e, url) {
if (url != wc.getURL()) {
e.preventDefault()
electron.shell.openExternal(url)
}
})
Check whether the requested url is an external link. If yes then use shell.openExternal.
mainWindow.webContents.on('will-navigate', function(e, reqUrl) {
let getHost = url=>require('url').parse(url).host;
let reqHost = getHost(reqUrl);
let isExternal = reqHost && reqHost != getHost(wc.getURL());
if(isExternal) {
e.preventDefault();
electron.shell.openExternal(reqUrl);
}
}
Put this in renderer side js file. It'll open http, https links in user's default browser.
No JQuery attached! no target="_blank" required!
let shell = require('electron').shell
document.addEventListener('click', function (event) {
if (event.target.tagName === 'A' && event.target.href.startsWith('http')) {
event.preventDefault()
shell.openExternal(event.target.href)
}
})
For Electron 5, this is what worked for me:
In main.js (where you create your browser window), include 'shell' in your main require statement (usually at the top of the file), e.g.:
// Modules to control application life and create native browser window
const {
BrowserWindow,
shell
} = require('electron');
Inside the createWindow() function, after mainWindow = new BrowserWindow({ ... }), add these lines:
mainWindow.webContents.on('new-window', function(e, url) {
e.preventDefault();
shell.openExternal(url);
});
I solved the problem by the following step
Add shell on const {app, BrowserWindow} = require('electron')
const {app, BrowserWindow, shell} = require('electron')
Set nativeWindowOpen is true
function createWindow () {
// Create the browser window.
const mainWindow = new BrowserWindow({
width: 1350,
height: 880,
webPreferences: {
nativeWindowOpen: true,
preload: path.join(__dirname, 'preload.js')
},
icon: path.join(__dirname, './img/icon.icns')
})
Add the following listener code
mainWindow.webContents.on('will-navigate', function(e, reqUrl) {
let getHost = url=>require('url').parse(url).host;
let reqHost = getHost(reqUrl);
let isExternal = reqHost && reqHost !== getHost(wc.getURL());
if(isExternal) {
e.preventDefault();
shell.openExternal(reqUrl, {});
}
})
reference https://stackoverflow.com/a/42570770/7458156 by cuixiping
I tend to use these lines in external .js script:
let ele = document.createElement("a");
let url = "https://google.com";
ele.setAttribute("href", url);
ele.setAttribute("onclick", "require('electron').shell.openExternal('" + url + "')");

How to Use CasperJS in node.js?

I would like to use CasperJS in node.js.
I have referred to the following URL's to use CasperJS in node.js:
https://github.com/sgentle/phantomjs-node
http://casperjs.org/index.html#faq-executable
With the help of the above URLs I have written the following code:
//DISPLAY=:0 node test2.js
var phantom = require('phantom');
console.log('Hello, world!');
phantom.create(function (ph) {
ph.casperPath = '/opt/libs/casperjs'
ph.injectJs('/opt/libs/casperjs/bin/bootstrap.js');
var casper = require('casper').create();
casper.start('http://google.fr/');
casper.thenEvaluate(function (term) {
document.querySelector('input[name="q"]').setAttribute('value', term);
document.querySelector('form[name="f"]').submit();
}, {
term: 'CasperJS'
});
casper.then(function () {
// Click on 1st result link
this.click('h3.r a');
});
casper.then(function () {
console.log('clicked ok, new location is ' + this.getCurrentUrl());
});
casper.run();
});
When I run this code, I got the following error:
ERROR MSG:
tz#tz-ubuntu:/opt/workspaces/TestPhantomjs$ DISPLAY=:0 node test2.js
Hello, world!
Error: Cannot find module 'casper'
at Function._resolveFilename (module.js:332:11)
at Function._load (module.js:279:25)
at Module.require (module.js:354:17)
at require (module.js:370:17)
at /opt/workspaces/TestPhantomjs/test2.js:6:14
at Object.<anonymous> (/opt/workspaces/TestPhantomjs/node_modules/phantom/phantom.js:82:43)
at EventEmitter.<anonymous> (/opt/workspaces/TestPhantomjs/node_modules/phantom/node_modules/dnode/index.js:215:30)
at EventEmitter.emit (events.js:67:17)
at handleMethods (/opt/workspaces/TestPhantomjs/node_modules/phantom/node_modules/dnode-protocol/index.js:138:14)
at EventEmitter.handle (/opt/workspaces/TestPhantomjs/node_modules/phantom/node_modules/dnode-protocol/index.js:98:13)
phantom stdout: Unable to load casper environment: Error: Failed to resolve module fs, tried fs
You can use SpookyJS to drive CasperJS from Node.
https://groups.google.com/group/casperjs/browse_thread/thread/641e9e6dff50fb0a/e67aaef5ab4ec918?hl=zh-CN#e67aaef5ab4ec918
Nicolas Perriault
2012/2/27 天猪 蓝虫. :
I wan to use casperjs in nodejs.
and refs to:
https://github.com/sgentle/phantomjs-node and
http://casperjs.org/index.html#faq-executable
You can't run CasperJS that way; QtWebKit and V8 don't share the same
js environment (and event loop), so your node.js app won't be able to
load and use a CasperJS module. You have to run your CasperJS script
separately using a subprocess call, like this one on github. I
don't plan to make CasperJS compatible with phantomjs-node because it
uses alert()-based dirty hacks I'm not easy with.
Cheers,
-- Nicolas Perriault
CasperJS includes a web server to talk to the outside world. Node (using request, superagent etc) can now talk to casper over HTTP.
In scraper.js:
#!/usr/bin/env casperjs
// I AM NOT NODEJS
// I AM CASPER JS
// I RUN IN QTWEBKIT, NOT V8
var casper = require('casper').create();
var server = require('webserver').create();
var ipAndPort = '127.0.0.1:8585';
server.listen(ipAndPort, function(request, response) {
casper.start('https://connect.data.com/login');
casper.userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36");
casper.then(function(){
// lots of code here, and a few more cassper.then()s
});
casper.run(function(){
console.log('\n\nFinished')
response.statusCode = 200;
var body = JSON.stringify({
phoneNumber: '1800-YOLO-SWAG'
})
response.write(body);
response.close();
});
});
You can now run scraper.js as a web server:
chmod +x scraper.js
./scraper.js
You should run it as a Linux service just like you would for a node app.
One solution (which worked for me) is to start and stop your server on a per-test basis. For example, I have a runtests.coffee which looks like:
http = require 'http'
glob = require 'glob'
spawn = require('child_process').spawn
db = require './db' # Contains all database stuff.
webapp = require './webapp' # Contains all of the Express stuff.
db.connect 'test' # Connects to the db server and creates an empty test db.
server = http.createServer webapp.makeApp()
server.listen 0, ->
port = server.address().port
process.env.URL = "http://localhost:#{ port }"
glob 'tests/*', (err, filenames) ->
child = spawn 'casperjs', ['test'].concat(filenames)
child.stdout.on 'data', (msg) -> process.stdout.write msg
child.stderr.on 'data', (msg) -> process.stderr.write msg
child.on 'exit', (code) ->
db.disconnect() # Drops the test db.
server.close()
process.exit code
And my CasperJS tests in tests/ look like:
URL = require('system').env.URL # Note, Casper code here, not Node.
casper.test.begin 'Test something', 1, (test) ->
casper.start "#{ URL }/welcome"
casper.then ->
test.assertHttpStatus 200
# ....
casper.run ->
test.done()
It basically means that your script can't find Casper; have you checked the path and made sure that
/opt/libs/casperjs
and:
/opt/libs/casperjs/bin/bootstrap.js
Are accessible by a website user ? considering the location it's probably not likely.
/opt is a unix path, but the website will be looking in {websiterootpath}/opt.
I'd create a subfolder 'casperjs' in the root folder of your website and copy the contents of
/opt/libs/casperjs
To there.
Then change your paths from
/opt/libs/casperjs
To
/casperjs
I tried to run casper by node cron job too,
here's my solution
in casper.js echo your response:
casper.then(function() {
var comments = this.evaluate(getComments);
this.echo(JSON.stringify(comments));
})
use node-cmd in node file casper_wrapper.js:
var cmd = require('node-cmd');
module.exports = function(url) {
return new Promise(function(resolve, reject) {
cmd.get(
'casperjs casper.js ' + url, // casper takes args to run the script
function(err, data, stderr){
if (err) {
reject(err);
return;
}
var obj = JSON.parse(data);
resolve(obj);
}
);
});
}

How can I control a browser ( ala Selenium ) with node.js?

I've heard of soda, but it seems like it requires you to signup and there's a limit on the # of minutes ( free acct / 200 minutes ).
Does anyone know if there's some alternative way to control a browser, or more specifically invoke JS on a web page?
https://github.com/LearnBoost/soda/raw/master/examples/google.js
/**
* Module dependencies.
*/
var soda = require('../')
, assert = require('assert');
var browser = soda.createClient({
host: 'localhost'
, port: 4444
, url: 'http://www.google.com'
, browser: 'firefox'
});
browser.on('command', function(cmd, args){
console.log(' \x1b[33m%s\x1b[0m: %s', cmd, args.join(', '));
});
browser
.chain
.session()
.open('/')
.type('q', 'Hello World')
.clickAndWait('btnG')
.getTitle(function(title){
assert.ok(~title.indexOf('Hello World'), 'Title did not include the query');
})
.clickAndWait('link=Advanced search')
.waitForPageToLoad(2000)
.assertText('css=#gen-query', 'Hello World')
.assertAttribute('as_q#value', 'Hello World')
.testComplete()
.end(function(err){
if (err) throw err;
console.log('done');
});
Zombie.js might work for you. It is headless and seems really cool.
There are actually now Selenium bindings for JavaScript that work with Node.js.
Here are some basic steps to get started:
1 Install Node.js, you can find the download here.
Make sure you
have the latest Chrome driver and put it in your path.
Use npm install selenium-webdriver to get the module added to your project.
Write a test, for example:
var webdriver = require('selenium-webdriver');
var driver = new webdriver.Builder().
withCapabilities(webdriver.Capabilities.chrome()).
build();
driver.get('http://www.google.com');
driver.findElement(webdriver.By.name('q')).sendKeys('simple programmer');
driver.findElement(webdriver.By.name('btnG')).click();
driver.quit();</code>
I cover how to do this with some screenshots and how to use Mocha as a test driver in my blog post here.
Here's a pure node.js wrapper around the java API for selenium's webdriver:
https://npmjs.org/package/webdriver-sync
Here's an example:
var webdriverModule = require("webdriver-sync");
var driver = new webdriverModule.ChromeDriver;
var By = webdriverModule.By;
var element = driver.findElement(By.name("q"));
element.sendKeys("Cheese!");
element.submit();
element = driver.findElement(By.name("q"));
assert.equal(element.getAttribute('value'), "Cheese!");
Save that in a .js file and run it with node.
The module is a pure wrapper, so things like sleep or synchronous calls are entirely possible. Here's the current interface of the module:
module.exports={
ChromeDriver:ChromeDriver,
FirefoxDriver:FirefoxDriver,
HtmlUnitDriver:HtmlUnitDriver,
By:new By(),
ExpectedConditions:new ExpectedConditions(),
WebDriverWait:WebDriverWait,
Credentials:UserAndPassword,
Cookie:Cookie,
TimeUnits:TimeUnits,
/**
* #param {number} amount in mills to sleep for.
*/
sleep:function(amount){
java.callStaticMethodSync(
"java.lang.Thread",
"sleep",
new Long(amount)
);
}
};
You can see an integration test that tests the full capabilities here:
https://github.com/jsdevel/webdriver-sync/blob/master/test/integrations/SmokeIT.js
wd is "A node.js javascript client for webdriver/selenium 2"

Resources