I had tried to get the response data from a website https://www.mscdirect.com/product/details/49784960 using python requests.
But am not getting the response and i had tried selenium too. But still the error persists
Using Python requests
import requests
session=requests.session()
response=session.get('https://www.mscdirect.com/product/details/49784960')
print(response.text)
Using Selenium
from selenium import webdriver
options = Options()
#options.add_argument("--headless")
#options.add_argument('--disable-gpu')
driver = webdriver.Chrome(executable_path="chromedriver", chrome_options=options)
driver.delete_all_cookies()
driver.get("https://www.mscdirect.com/product/details/49784960")
print(driver.page_source)
Is there any thing am missing? For python requests, i has passed headers also.
Related
I am trying to close portable browser via selenium
I passed --remote-debugging-port=9222 because if I do not pass it then the program is stuck in object creation of webdriver.Chrome(). It will open the portable browser but not load the URL.
But after the URL is open I want to close the browser but driver.quit() it is not working for me. I have tried some other methods for close the browser but they do not work as well.
I want to close the specific instance of the browser which is opened by this program not other opened instances of the browser.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.binary_location = 'C:/Portable/GoogleChromePortable/GoogleChromePortable.exe'
chrome_options.add_argument("--remote-debugging-port=9222")
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--profile-directory=Person 1")
driver = webdriver.Chrome(options=chrome_options,executable_path='C:/Portabl/chromedriver_win32/chromedriver.exe')
url = "https://www.google.com/"
driver.get(url)
driver.quit()
I am using:
selenium 3.141.0, windows 10, python 3.8.0, portable chrome version 93.0.4577.63 (32-bit)
Your this statement
I passed --remote-debugging-port=9222 because if I do not pass it then
the program is stuck in object creation of webdriver.Chrome()
is not correct. --remote-debugging-port=9222 looks like a port number where your application is deployed and you have used chrome option to send them to browser object.
driver.quit()
this typically should have worked, what is the error when it did not work ?
also, for closing a single instance you could do
driver.close()
see if that helps.
I am new to this, this is my first question here.
Please guide me to improve.
Learning python automation.
Came across scenario to automate "replay server response from .har file".
Could capture, HAR file
from browsermobproxy import Server
server = Server("~/browsermob-proxy")
server.start()
proxy = server.create_proxy()
from selenium import webdriver
co = webdriver.ChromeOptions()
co.add_argument('--proxy-server={host}:{port}'.format(host='localhost', port=proxy.port))
driver = webdriver.Chrome(executable_path = "~/chromedriver", chrome_options=co)
proxy.new_har
driver.get(url)
proxy.har
How can I replay server response from captured HAR file (offline) ?
Is it possible with python or any other ?
And what's the significance of replay server response ?
Could you please help me ?
Check out this https://github.com/louisabraham/har2requests. It should convert the .har file into a python script.
As a part of automation, I am trying to login to a locally hosted website using selenium. The page serves a HTTP Basic authentication popup and I use the below code to send the credentials. However, upon using a debugger and executing the code step-wise, I deciphered that a TimeOut exception occurs repeatedly (at line marked with a comment beside it).
Details
I tried on Chrome browser and its corresponding chrome WebDriver for all versions from 79.0 till the latest 84.0, but this exception seems to occur in all the cases.
OS - Windows Server W2k12 VM. [Tried on Windows 10 as well]
Python version 3.8
Code
import time
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
driver = webdriver.Chrome(chrome_options=options)
driver.maximize_window()
driver.get(url)
wait = WebDriverWait(driver, 10)
alert = wait.until(EC.alert_is_present()) # This line causes the time out exception
alert = driver.switch_to.alert
alert.send_keys('domain\username' + Keys.TAB + 'password')
alert.accept()
time.sleep(5)
Note:
Due to a bug in the internal page, I cannot send the credentials through the URL using the https://username:password#ipaddress:port format, so this has forced me to resort to making the above selenium method as my only choice.
The corresponding code for firefox works well (for the same target internal website)
Probable Hunch
I wonder, if I am missing any packages on the newly created VM which is crucial for chrome WebDriver to work. For example, Firefox Gecko driver required Visual studio redistributables in order to work. Does chrome WebDriver require any such equivalent packages?
I don't believe that the Basic Auth popup is exposed as an "alert" in ChromeDriver, so AFAIK your only option is https://username:password#ipaddress:port. Interesting that you say that you can program to the popup in Firefox.
Until Chrome 78, the auth popup would display and block the test script, and you had to "manually" enter the credentials (or use a more general "desktop window manipulation" API), but I don't think that works anymore.
What worked here finally was to discard using the Selenium way of using send_keys() approach but use other packages to do the job. The following two snippets worked.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import keyboard
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
driver = webdriver.Chrome(r"path_to_chromedriver.exe", options=options)
driver.maximize_window()
driver.get("<replace with url>")
keyboard.write(r"<replace with username>")
keyboard.press_and_release("tab")
keyboard.write("<replace with password>")
keyboard.press_and_release("tab")
keyboard.press_and_release("enter")
Or this one (pip install pywin32 before)
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import win32com.client
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
driver = webdriver.Chrome(r"path_to_chromedriver.exe", options=options)
driver.maximize_window()
driver.get("<replace with url>")
shell = win32com.client.Dispatch("WScript.Shell")
shell.SendKeys(r"<replace with username>")
shell.SendKeys("{TAB}")
shell.SendKeys("<replace with password>")
shell.SendKeys("{TAB}")
When you input http://www.python.or/ (intentionally use the wrong url) in firefox or other browsers, browser show something such as below:
The connection was reset
The connection to the server was reset while the page was loading.
The site could be temporarily unavailable or too busy. Try again in a few moments.
If you are unable to load any pages, check your computer’s network connection.
If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the
Now let's do the same task with selenium.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--headless")
browser = webdriver.Chrome(chrome_options=chrome_options)
browser.get("http://www.python.or")
When you execute the above code in console,no error info,how to catch the exception such as firefox do with selenium?
I suggest you to try HTML requests
First, get the requests from the URL:
import requests
r = requests.get(your_url)
Now you need to get the status_code from your request:
print(r.status_code)
r.status_code contains the http's error handler.
Here is a list of all the HTTP status codes
Try this and you'll know when something wrong happened (ex Error 404)
When I FireFox is loaded through Selenium, my browser is under remote controller and a bot image show in URL section in my browser. For deal with this problem I changed User-Agent by this code:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")
driver = webdriver.Firefox(profile)
User-Agent was changed successfully but, bot image in URL section of my browser was remained.
Would you help me, please?
I used this URL for changing User-Agent:
Change user agent for selenium driver
My whole code is:
MainLink="https://blog.feedspot.com/iot_blogs/"
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities.PHANTOMJS
caps["phantomjs.page.settings.userAgent"] = "whatever you want"
driver = webdriver.Firefox()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")
driver = webdriver.Firefox(profile)
agent = driver.execute_script("return navigator.userAgent")
print(agent)
driver.get(MainLink)
Your code is a little confusing. You don't need to use both phantomjs and firefox as the driver for selenium. Which one are you going to use?
As I understood, you would like to avoid being detected from the page you're interacting with. This is usually a greater concern if you're operating with a headless browser, which is the case when using phantomjs but not when using firefox without explicitly telling it to run in this mode, which apparently is your case.
If you are in fact having issues of this nature, there are many ways to try to mitigate this, starting from changing the user agent, as mentioned by you. Assuming you want to go with firefox, A code example would be:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
profile.set_preference("general.useragent.override", user_agent)
driver = webdriver.Firefox(profile)
MainLink="https://blog.feedspot.com/iot_blogs/"
driver.get(MainLink)
In addition, you could set a different user agent each time you make a request, combining with changing the ip address from where the requests are originated, of course, but this is besides the point...
Hope this helps...
Depending on how "deep" you want to go, you could also add cookies copied from a trusted client:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://www.example.com")
# Adds the cookie into current browser context
driver.add_cookie({"name": "key", "value": "value"})
See also: https://www.selenium.dev/documentation/en/support_packages/working_with_cookies/
I could run my code without loading page or in headless way (headless browser).
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(firefox_options=options)
print("Firefox Headless Browser Invoked")
driver.get('https://blog.feedspot.com/iot_blogs/')
s=driver.find_element_by_xpath("""/html/body/div[1]/div[2]/div/div/div[1]/article/div[1]/h1""")
print(s.text)