so I wonder what ports does Selenium uses with ChromeDriver to run? like from what port to what port selenium runs. Example 1500-3000
I'm trying to run selenium while using NordVpn but it says that it couldn't find any free port so I'm looking for a port list that selenium chromedriver uses to whitelist them in nordvpn and be able to run Selenium while using NordVPN
You can use Chrome DevTools Protocol. Try with below steps:
Add the path of the Chrome executable to the environment variable PATH.
Launch Chrome with a custom flag, and open a port for remote debugging
Please make sure the path to chrome’s executable is added to the environment variable PATH. You can check it by running the command chrome.exe (on Windows) or Google/ Chrome ( on Mac). It should launch the Chrome browser.
If you get a similar message as below that means Chrome is not added to your system’s path:
'chrome' is not recognized as an internal or external command,
operable program or batch file.
If this is the case, please feel free to Google how to add chrome to PATH?
Launch browser with custom flags
To enable Chrome to open a port for remote debugging, we need to launch it with a custom flag –
chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\selenum\ChromeProfile"
For --remote-debugging-portvalue you can specify any port that is open.
For --user-data-dir flag you need to pass a directory where a new Chrome profile will be created. It is there just to make sure chrome launches in a separate profile and doesn’t pollute your default profile.
You can now play with the browser manually, navigate to as many pages, and perform actions and once you need your automation code to take charge, you may run your automation script. You just need to modify your Selenium script to make Selenium connect to that opened browser.
You can verify if Chrome is launched in the right way:
Launch a new browser window normally (without any flag), and navigate to http://127.0.0.1:9222
Confirm if you see the Google homepage reference in the second browser window
Launch browser with options
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
#Change chrome driver path accordingly
chrome_driver = "C:\chromedriver.exe"
driver = webdriver.Chrome(chrome_driver, chrome_options=chrome_options)
print driver.title
The URL 127.0.0.1 denotes your localhost. We have supplied the same port, i.e 9222, that we used to launch Chrome with --remote-debugging-port flag. While you can use any port, you need to make sure it is open and available to use.
Related
I would like to use chromedriver to scrape some stories from fanfiction.net.
I try the following:
from selenium import webdriver
import time
path = 'D:\chromedriver\chromedriver.exe'
browser = webdriver.Chrome(path)
url1 = 'https://www.fanfiction.net/s/8832472'
url2 = 'https://www.fanfiction.net/s/5218118'
browser.get(url1)
time.sleep(5)
browser.get(url2)
The first link opens (sometimes I have to wait 5 seconds). When I want to load the second url, cloudflare intervens and wants me to solve captchas - which are not solvable, atleast cloudflare does not recognize this.
This happens also, if I enter the links manually in chromedriver (so in the GUI). However, if I do the same things in normal chrome, everything works just as fine (I do not even get the waiting period on the first link) - even in private mode and all cookies deleted. I could reproduce this on several machines.
Now my question: To my intuition, chromedriver was just the normal chrome browser which allowed to be controlled. What is the difference to normal chrome, how does Cloudflare distinguish both, and how can I mask my chromedriver as normal chrome? (I do not intend to load many pages in very short time, so it should not look like a bot).
I hope my question is clear
This error message...
...implies that the Cloudflare have detected your requests to the website as an automated bot and subsequently denying you the access to the application.
Solution
In these cases the a potential solution would be to use the undetected-chromedriver to initialize the Chrome Browsing Context.
undetected-chromedriver is an optimized Selenium Chromedriver patch which does not trigger anti-bot services like Distill Network / Imperva / DataDome / Botprotect.io. It automatically downloads the driver binary and patches it.
Code Block:
import undetected_chromedriver as uc
from selenium import webdriver
import time
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
driver = uc.Chrome(options=options)
url1 = 'https://www.fanfiction.net/s/8832472'
url2 = 'https://www.fanfiction.net/s/5218118'
driver.get(url1)
time.sleep(5)
driver.get(url2)
References
You can find a couple of relevant detailed discussions in:
Selenium app redirect to Cloudflare page when hosted on Heroku
How to bypass being rate limited ..HTML Error 1015 using Python
I'm using Selenium & NodeJS to automate download tests on Chrome.
I noticed that Chrome's download protection behaves differently when clicking a link or redirecting to URL automatically vs actually typing the URL in the address bar and clicking ENTER (Chrome protection doesn't mark some files as blocked when actually typing the URL).
What I tried till now and still got blocks for some files:
driver.get(url)
driver.executeScript to redirect to the url
driver.executeScript to create A element and clicking on it.
opening a new tab and then driver.get(url)
Is there a way to imitate the address bar typing and ENTER clicking with Selenium?
Selenium does not support sending keys to the browser address bar, unfortunately.
Someone suggested a solution with win32com.client library for Node.js here
I haven't tried this but found a similar question on this thread.
I have published my website in IIS. The url is http://localhost/MyDemo. I want to give this url for testing to other people (anyone). How can I do that?
I tried to give sharing permission to "Everyone" in the Edit Permission.
Please help.
There are standard port forwarding tools that allow you to do this.
A helpful link here - https://www.sitepoint.com/accessing-localhost-from-anywhere/
Now ngRok is a tool I have used many a times. Its usage is pretty straightforward.
It can be downloaded from here - https://ngrok.com/download
Once ngRok is downloaded, you have to open the ngrok.exe, and it opens a console window.
Execute the following command
ngrok http 4983 -host-header="localhost:4983"
Here 4983 is a port number, in case you have one.
Once the command is executed the tool opens up a Url, which is accessible from anywhere. The below image is your Url, and it will be valid for a certain duration of time (if you are using the tool for free).
When you're writing the manifest.json file, you have to specify matches for your content scripts. The http and https work fine, but if I try to include chrome://*/* or any variant of it, I get an error that I'm attempting to use an invalid scheme for my matches.
Is it not allowed?
By default you cannot run on a chrome:// url page.
However, there is an option in chrome://flags/#extensions-on-chrome-urls:
Extensions on chrome:// URLs (Mac, Windows, Linux, Chrome OS, Android)
Enables running extensions on chrome:// URLs, where extensions explicitly request this permission.
You still have to specify pages that your extension can run on and wildcards are not accepted - so you have to specify the full URL eg chrome://extensions/
The authorized schemes for matches are http, https, file, ftp.
Therefore, chrome is not a valid scheme.
Yes, it is not allowed. You can't link to them from hrefs on a webpage either.
I want to create a Firefox addon. Based on this page URL
I want to filter a URL.The problem is when I use jpm run command which opens a new instance of Firefox, the instance does not opens any URL at all. I even used an empty index.js file and the problem still remains
jpm run just opens a Firefox instance. You must browse to the url you want to check.
Or you can add this code to your index.js:
var tabs = require("sdk/tabs");
tabs.open("http://www.example.com");
but be aware that this code will be executed every time you addon loads so you should remove it before publishing your add-on because it'd be very annoying.
The opened instance proxy is set to "Use system proxy settings" so the website may not be opened.