Using Selenium with NodeJS and the Brave browser (on MacOS Big Sur) - node.js

I'm trying to run some simple tests on a NodeJS app, using Selenium and the Brave browser.
I'm running Big Sur (11.5.2), using Brave version:
Version 1.28.105 Chromium: 92.0.4515.131 (Official Build) (x86_64)
I've downloaded the ChromeDriver, and I've installed in my path:
> which chromedriver
/Users/dd/bin/scripts/chromedriver
> chromedriver --version
ChromeDriver 92.0.4515.107 (87a818b10553a07434ea9e2b6dccf3cbe7895134-refs/branch-heads/4515#{#1634})
I've created a folder, yarn inited, added the selenium-webdriver as a dev dependency,
{
"name": "seleniumtest",
"version": "1.0.0",
"description": "seleniumtest",
"main": "index.js",
"license": "MIT",
"dependencies": {
"cors": "^2.8.5",
"express": "^4.17.1"
},
"devDependencies": {
"eslint": "^7.32.0",
"jest": "^27.0.6",
"nodemon": "^2.0.12",
"selenium-webdriver": "^4.0.0-beta.4"
},
"scripts": {
"start": "node ./index.js"
}
}
run up the following Javascript file, index.js:
const { Builder, By, Key, until } = require('selenium-webdriver');
(async function example() {
let driver = await new Builder().forBrowser('brave').build();
try {
await driver.get('http://www.google.com/');
await driver.findElement(By.name('q')).sendKeys('webdriver', Key.RETURN);
await driver.wait(until.titleIs('webdriver - Google Search'), 1000);
} finally {
await driver.quit();
}
})();
I've fired up the chromedriver:
> chromedriver
Starting ChromeDriver 92.0.4515.107 (87a818b10553a07434ea9e2b6dccf3cbe7895134-refs/branch-heads/4515#{#1634}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
but when I run yarn start I get :
UnhandledPromiseRejectionWarning: Error: Do not know how to build driver: browser; did you forget to call usingServer(url)?
If I replace Builder().forBrowser('brave').build(); with Builder().forBrowser('chrome').build(); then Chrome fires up, but the window just sits there trying to load data:, in the url bar. I tried chromium - same result as for brave.
What am I doing wrong?

You can use setChromeBinaryPath function and point the brave binary path to it:
const chrome = require('selenium-webdriver/chrome');
(async function helloSelenium(){
let options = await new chrome.Options();
options.setChromeBinaryPath('/opt/brave.com/brave/brave-browser');
in the forBrowser('brave') replace it with 'chrome', as selenium will do a switch case with that and only the values in Browser enum will be valid (chrome, edge, firefox, internet_explorer, safari, opera);

Related

Cannot find module 'node:url' when executing typescript from webstorm

I have written this small typescript hello world example
import axios from 'axios';
import { wrapper } from 'axios-cookiejar-support';
import { CookieJar } from 'tough-cookie';
const jar = new CookieJar();
const client = wrapper(axios.create({ jar }));
client.get('https://example.com');
when I run this from webstorm i get the following error
/usr/bin/node /usr/local/lib/node_modules/ts-node/dist/bin.js /home/nayana/WebstormProjects/hello-world/hello.ts
Error: Cannot find module 'node:url'
anyone have idea on how to resolve this?
I already tried npm install node:url and url
i have isolated the error to this line
const client = wrapper(axios.create({ jar }));
The issue maybe is related to the node version.
The axios-cookiejar-support requires a specific node version ("node": ">=14.18.0 <15.0.0 || >=16.0.0").
Check node --version and package-lock.json.
Sample:
"node_modules/axios-cookiejar-support": {
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/axios-cookiejar-support/-/axios-cookiejar-support-4.0.3.tgz",
"integrity": "sha512-fMQc0mPR1CikWZEwVC6Av+sD4cJuV2eo06HFA+DfhY54uRcO43ILGxaq7YAMTiM0V0SdJCV4NhE1bOsQYlfSkg==",
"dependencies": {
"http-cookie-agent": "^4.0.2"
},
"engines": {
"node": ">=14.18.0 <15.0.0 || >=16.0.0"
},
"peerDependencies": {
"axios": ">=0.20.0",
"tough-cookie": ">=4.0.0"
}
},
You might need to install a later version of node.js.
I was running 14.17.6 and after installing 16.17.0 with nvm then I was able to run the project.
If you have nvm installed you can install a specific version of node e.g.
nvm install 16.17.0
make sure the types array in your tsconfig.json file contains "node"
{
"compilerOptions": {
"types": [
// ... your other types
"node"
],
// ... your other settings
},
}
The only thing you need to do, if you didn't install typescript is to change in the vite.config.js file, the import line like this:
import { fileURLToPath, URL } from 'node:url'
To:
import { fileURLToPath, URL } from 'url'

TypeError: trying to use aws-elasticsearch-connector

I'm attempting to update a legacy elastic-search node app, using the the package aws-elasticsearch-connector
and for some reason I'm unable to get it to work at all, even the simplest provided example...
I installed the packages, exactly as shown...
> npm install --save aws-elasticsearch-connector #elastic/elasticsearch aws-sdk
This is the sample code...
const { Client } = require('#elastic/elasticsearch')
const AWS = require('aws-sdk')
const createAwsElasticsearchConnector = require('aws-elasticsearch-connector')
// (Optional) load profile credentials from file
AWS.config.update({
profile: 'myawsprofile'
})
const client = new Client({
...createAwsElasticsearchConnector(AWS.config),
node: 'https://my-elasticsearch-cluster.us-east-1.es.amazonaws.com'
})
When I attempt to run it with this...
> node .\index.js
I get this error...
class AmazonConnection extends Connection {
TypeError: Class extends value undefined is not a constructor or null
I have no idea how I'm supposed to fix this, since the error seems to be in the module itself, not my sample code.
Most of the examples of this error that I've seen, suggest that it's related to circular references, but that doesn't seem to be of any help to me.
I'm using node v16.14.0
This is my package.json...
{
"name": "test_es",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC",
"dependencies": {
"#elastic/elasticsearch": "^8.0.0",
"aws-elasticsearch-connector": "^9.0.3",
"aws-sdk": "^2.1087.0"
}
}
I'm probably doing something wrong, or the package author may be assuming some additional knowledge that I just don't have.
Any suggestions would be greatly appreciated.
It seems version 8 of #elastic/elasticsearch is not compatible with aws-elasticsearch-connector.
Changing to version 7.17.0 seems to resolve this particular error.

Could not find expected browser chrome locally

This Meteor code uses "puppeteer 8.0.0", "puppeteer-core 10.0.0", puppeteer-extra 3.1.18" and "puppeteer-extra-plugin-stealth 2.7.8", It gives this error:
Error: Could not find expected browser (chrome) locally. Run npm install to download the correct Chromium revision (884014).
Tried "npm install" for no avail. Reading up online, tried removing "puppeteer-core": "^10.0.0" from package.json dependencies for no avail.
Any help is much appriciated. Thanks
const puppeteer = require('puppeteer-extra');
const nameH = require('./NameH');
const puppeteerOptions = {
headless: true,
ignoreHTTPSErrors: true,
args: ['--no-sandbox', '--single-process', '--no-zygote', '--disable-setuid-sandbox']
}
let browser;
let pageNameH;
const init = async () => {
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
console.log('1') //>>>>>>>>>>>> Prints 1
puppeteer.use(StealthPlugin());
console.log('2') //>>>>>>>>>>>> Prints 2
browser = await puppeteer.launch(puppeteerOptions);
console.log('3') //>>>>>>>>> DID NOT PRINT <<<<<<<<<<<<<<<
pageNameH = await browser.newPage();
console.log('4')
await pageNameH.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await pageNameH.setViewport({ width: 1366, height: 768 });
await pageNameH.setRequestInterception(true);
blockResources(pageNameH);
}
const blockResources = page => {
page.on('request', (req) => {
if (req.resourceType() == 'stylesheet' || req.resourceType() == 'font' || req.resourceType() == 'image') {
req.abort();
}
else {
req.continue();
}
});
}
export const abc = async (nm, loc) => {
try {
console.log('name try') //>>>>>>>>>>>> Prints "name try"
if (!(browser && pageNameH))
await init();
//use "required" nameh here
} catch (error) { // print the error <<<<<<<<<<<<<<<<<<<<<<<<<
console.log("Could not launch Puppeteer or open a new page.\n" + error);
if (browser && browser.close === 'function') await browser.close();
}
}
// included in package.json
"dependencies": {
"#babel/runtime": "^7.11.2",
"axios": "^0.21.1",
"check": "^1.0.0",
"cheerio": "^1.0.0-rc.6",
"jquery": "^3.5.1",
"meteor-node-stubs": "^1.0.1",
"nightmare": "^3.0.2",
"pending-xhr-puppeteer": "^2.3.3",
"puppeteer": "^8.0.0",
"puppeteer-core": "^10.0.0",
"puppeteer-extra": "^3.1.18",
"puppeteer-extra-plugin-adblocker": "^2.11.11",
"puppeteer-extra-plugin-block-resources": "^2.2.9",
"puppeteer-extra-plugin-stealth": "^2.7.8"
},
may be you can try this, it works for me on linux(centOS), puppeteer(10.2.0).
cd ./node_modules/puppeteer
npm run install
If that fails, you can also try running:
cd ./node_modules/puppeteer
npm install
This will download the chromium to ./node_modules/puppeteer/.local-chromium
I had the same problem. I checked my env variables, and even though PUPPETEER_SKIP_CHROMIUM_DOWNLOAD was set to false, it was still not working. After I removed the variable (unset PUPPETEER_SKIP_CHROMIUM_DOWNLOAD for mac), it worked.
related dependancies:
"dependencies": {
"chrome-aws-lambda": "^10.0.0",
"puppeteer-core": "^10.0.0",
},
"devDependencies": {
"puppeteer": "^10.0.0",
}
Launching Chromium:
import chromium from "chrome-aws-lambda";
const browser = await chromium.puppeteer.launch({
executablePath: await chromium.executablePath,
});
Fixed it by running this command to install chromium manually.
node node_modules/puppeteer/install.js
if you are testing locally, make sure you have puppeteer installed as dev dependencies. specifically
npm install puppeteer --save-dev
https://github.com/alixaxel/chrome-aws-lambda/wiki/HOWTO:-Local-Development#workaround
this approach allows us to rely on puppeteer for local development and puppeteer-core for production deployments.
I had the same issue. What worked for me was to specify as the executablePath Puppeteer launch option the fullpath to the local chromium used by Puppeteer.
Something like this:
const launchOptions = {
// other options (headless, args, etc)
executablePath: '/home/jack/repos/my-repo/node_modules/puppeteer/.local-chromium/linux-901912/chrome-linux/chrome'
}
As noted in another answer, it seems that also referencing a chromium local binary would work, but I think it's a worse solution, since Puppeteer is guaranteed to work only with the local bundled version of Chromium.
Throwing my answer in, in hopes that it helps someone not waste their entire evening like I did.
I was writing a Typescript server that used Puppeteer, and I'm using ESBuild to transpile from TS to JS. In the build step, esbuild was trying to bundle everything into one file, but I had to instruct it to preserve the Puppeteer import from node_modules.
I did this by marking puppeteer as external. See docs here.
Since npm i downloads a compatible version of chromium to the node_modules folder, once I preserved this import, it was able to find Chromium in the node_modules folder.
My build file looks like:
require("esbuild").buildSync({
entryPoints: ["src/index.ts"],
outdir: "build",
bundle: true,
platform: "node",
target: "node16",
external: ["puppeteer"],
});
And I run it with node prod-build.js.
Now in my code, I can just call launch!
const browser = await puppeteer.launch()
I used the installed version on my pc (maybe not what you're looking for)
const browser = await puppeteer.launch({headless:false, executablePath:
'C:/Program Files/.../chrome.exe' });
You may need to install some dependencies depending on your OS. Check Puppeteer's Troubleshooting page for more details.
I was having this error too and I noticed that I started getting such an error after updating my node from my Dockerfile to version ^16 (my puppeteer is in a container). How did I solve it? I downgraded my node from Dockerfile to version 12.
Hope this resolves it...
OBS: I use Ubuntu 21.04 in machine.
EDIT ------
In newest versions of node, you need to go in ./node_modules/puppeteer and run command npm install, then the correct packages will be installed.
I'm using that solution.
If you are using puppeteer in AWS SAM and you don't have puppeteer in dependencies, you can install the same in puppeteer-core using
node node_modules/puppeteer/install.js
For this setup to work you will have to add chrome-aws-lambda in the devDependencies.
"devDependencies": {
"chrome-aws-lambda": "^10.1.0"
}
Also before you take it to production don't forget to add the layer in your template.yaml file:
Layers:
- !Sub 'arn:aws:lambda:ap-south-1:764866452798:layer:chrome-aws-lambda:25'

Error: Cannot find module 'ipfs' web3.min.js:1:155

I Can't work with node module ipfs.js
console shows error: "Cannot find module 'ipfs'"
Ubuntu 16.04.4 LTS
node --version == v8.10.0
npm --version == 5.6.0
ipfs version == 0.4.13
My package.json:
{
"scripts": {
"dev": "lite-server",
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC",
"devDependencies": {
"lite-server": "^2.3.0"
},
"dependencies": {
"ipfs": "^0.28.2",
"web3": "^0.20.6"
}
}
My app.js:
const IPFS = require('ipfs')
const node = new IPFS()
// // set the provider you want from Web3.providers
web3 = new Web3(new `Web3.providers.HttpProvider("http://127.0.0.1:8545"));`
When I run in command line, its works:
> const IPFS = require('ipfs')
undefined
> const node = new IPFS()
undefined
> Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/ipfs/QmYwqrDJCQEiY2fijnwpPhhsG5w8rVxCTjK7duxtPyt24J
Swarm listening on /ip4/127.0.0.1/tcp/4002/ipfs/QmYwqrDJCQEiY2fijnwpPhhsG5w8rVxCTjK7duxtPyt24J
Swarm listening on /ip4/192.168.2.103/tcp/4002/ipfs/QmYwqrDJCQEiY2fijnwpPhhsG5w8rVxCTjK7duxtPyt24J
I was able to solve my problem.
I had no experience with node, so I completely confused its use, in the issue of server-side and client-side use. My intention was to use ipfs in the browser
I am creating a Dapp using the Truffle framework, and the truffle provides a "web3.min.js" file, this library was conflicting with "var ipfs = require ('ipfs').
The solution was simple, I'm using the js-ipfs library only in the browser:
https://github.com/ipfs/js-ipfs#use-in-the-browser
Now my code it's:
my html:
<script src="js/web3.min.js"></script>
<script src="https://unpkg.com/ipfs/dist/index.min.js"></script>
<script src="js/myjs.js"></script>
my app.js:
var web3 = new Web3(new Web3.providers.HttpProvider("http://127.0.0.1:8545"));
console.log(web3);
console.log(Ipfs);

bundle.js bundle.js:1 Uncaught Error: Cannot find module '/node_modules\mongodb-core\lib\topologies/../../package.json'

I am getting the below error:
bundle.js:1 Uncaught Error: Cannot find module '/node_modules\mongodb-ore\lib\topologies/../../package.json'
Here is my version detial:
OS: Windows10
MongoDB: 2.2.16
MongoDB-core: 2.1.2
Node: 6.9.2
I have used npm install bson-ext and changed \node_modules\mongodb-core\node_modules\bson-ext\ext\index.js to
try {
// Load the precompiled win32 binary
if(process.platform == "win32" && process.arch == "x64") {
bson = require('bson');
} else if(process.platform == "win32" && process.arch == "ia32") {
bson = require('bson');
} else {
bson = require('bson');
}
} catch(err) {
console.log(err)
// Attempt to load the release bson version
try {
bson = require('bindings')('bson.node');
} catch (err) {
throw new Error("js-bson: Failed to load c++ bson extension, using pure JS version");
}
}
while the original is:
bson = require('./win32/x64/bson');
because when I try browserify range.js > bundle.js, it cannot find bson-ext module in mongoDB-core.
I am not sure whether this kind of operation may cause the above error.
Here is my package.json file :
"dependencies": {
"browserify": "^13.1.1",
"bson": "^1.0.1",
"d3": "^4.4.0",
"express": "^4.14.0",
"hbs": "^4.0.1",
"jsdom": "^9.9.1",
"mongodb": "^2.2.16",
"mongodb-core": "^2.1.2"
}
I haven't been able to confirm this yet, but I think the problem is that MongoDB's JavaScript (Node.js) driver is not intended for use in a browser, for security reasons. Not clear if the problem in the OP is due to Browserify incorrectly resolving relative paths or something else, but regardless the preferred technique is to proxy requests to your MongoDB instance via a Node server.
Mongo lists off-the-shelf solutions for this here:
https://docs.mongodb.com/ecosystem/tools/http-interfaces/
Note also the --rest option, which allows an application to read directly from the DB via a URL schema:
https://docs.mongodb.com/ecosystem/tools/http-interfaces/#simple-rest-api
As the docs mention, this is not good practice for security concerns, but may help with prototyping.

Resources