I have been testing various automation tasks using Selenium Python library. I faced a very serious issue in that process. That is the delay caused due to poor internet connection.
The starting steps are same for all tasks, namely
- open browser
- go to website
- login
- open a specific link after login
These tasks take a long time, so I was hoping to find a method by which I could reuse an already open session and continue testing on that window instead of opening new browser window every time I run the script.
Here's what I have tested so far:
Example Original File:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome(r'C:\testDir\chromedriver_win32\chromedriver.exe', chrome_options=chrome_options)
url = driver.command_executor._url
session_id = driver.session_id
print("URL:",url,"SESSION ID:",session_id)
driver.get('http://www.facebook.com')
username = "---"
password = "---"
driver.find_element_by_name('email').send_keys(username)
driver.find_element_by_name('pass').send_keys(password)
driver.find_element_by_name('login').click()
# time.sleep(2)
driver.maximize_window()
time.sleep(2)
driver.find_elements_by_name('q').send_keys('Ayush Mandowara')
# element = wait.until(EC.presence_of_element_located((By.xpath, '//input[#placeholder="Search"]'))
driver.find_element_by_xpath('//input[#placeholder="Search"]').send_keys('Ayush Mandowara' + Keys.RETURN)
time.sleep(4)
driver.find_element_by_xpath('//div[contains(text(), "Ayush Mandowara")]').click()
time.sleep(3)
driver.find_element_by_class_name('coverBorder').click()
time.sleep(2)
Connecting File:
from connectingToPrev import url, session_id
driver = webdriver.Remote(command_executor=url,desired_capabilities={})
driver.session_id = session_id
driver.get("http://www.google.com")
This connecting file is following what all has already happened in the previous window, I was expecting it either connect to previous window or to open the last link with correct credentials
Answer in Python is appreciated!
Related
I'm trying to create a code where I can auto sign in and sign out on HR Portal i.e. "https://newage.greythr.com/". However, I'm unable to click the sign in/out button after logging in. Initial part of the code is working fine but the bottom part i.e. # Sign in/out is giving InvalidArgumentException error. I've tried 2 alternatives both mentioned below in the code but none of it is executing and giving the same error. Also, I tried to increase the wait time still it failed.
If anything is required from my end kindly let me know in comments section.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from selenium import webdriver
driver = webdriver.Chrome(executable_path='C:/Users/Selenium Drivers/chromedriver.exe')
# Open the website
driver.get("https://newage.greythr.com/")
time.sleep(2)
# Defining username & password
username = "****"
password = "****"
time.sleep(2)
# Entering username
user = driver.find_element("id", "username")
user.click()
user.send_keys(username)
time.sleep(2)
# Entering password
psw = driver.find_element("id", "password")
psw.click()
psw.send_keys(password)
time.sleep(2)
# Click login button
driver.find_element("id", "password").submit()
# Sign in/out
time.sleep(10)
driver.find_element("XPATH", "//button[contains(.,'Sign In')]").click()
**Upper part is same as above**
# Sign in/out
sign = driver.find_element("XPATH", "/html/body/app/ng-component/div/div/div[2]/div/ghr-home/div[2]/div/gt-home-dashboard/div/div[2]/gt-component-loader/gt-attendance-info/div/div/div[3]/gt-button[1]")
time.sleep(20)
sign.click()
You should be using 'xpath' argument or import the By module and do it like
from selenium.webdriver.common.by import By
driver.find_element(By.XPATH, "//button[contains(.,'Sign In')]").click()
I'm trying to download some zips files from this page to an specific path for an already chrome browser session open using the code down below:
import time
import numpy as np
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
opt = Options() #the variable that will store the selenium options
opt.add_experimental_option("debuggerAddress", "localhost:9222") #this allows bulk-dozer to take control of your Chrome Browser in DevTools mode.
opt.add_experimental_option("prefs", {"download.default_directory": r"C:\Users\ResetStoreX\Downloads\Binance futures data\ADAUSDT-Mark_Prices_Klines_1h_Timeframe"}) #set the path to save the desired zipped data
s = Service(r'C:\Users\ResetStoreX\AppData\Local\Programs\Python\Python39\Scripts\chromedriver.exe') #Use the chrome driver located at the corresponding path
driver = webdriver.Chrome(service=s, options=opt) #execute the chromedriver.exe with the previous conditions
#Why using MarkPrices: https://support.btse.com/en/support/solutions/articles/43000557589-index-price-and-mark-price#:~:text=Index%20Price%20is%20an%20important,of%20cryptocurrencies%20on%20major%20exchanges.&text=Mark%20Price%20is%20the%20price,be%20fair%20and%20manipulation%20resistant.
if driver.current_url == 'https://data.binance.vision/?prefix=data/futures/um/daily/markPriceKlines/ADAUSDT/1h/' :
number = 2 #initialize an int variable to 2 because the desired web elements in this page starts from 2
counter = 0
the_dictionary_links = {}
while number <= np.size(driver.find_elements(By.XPATH, '//*[#id="listing"]/tr')): #iterate over the tbody array
data_file_name = driver.find_element(By.XPATH, f'//*[#id="listing"]/tr[{number}]/td[1]/a').text
if data_file_name.endswith('CHECKSUM') == False:
the_dictionary_links[data_file_name] = driver.find_element(By.XPATH, f'//*[#id="listing"]/tr[{number}]/td[1]/a').get_attribute('href')
print(f'Saving {data_file_name} and its link for later use')
counter += 1
number += 1
print(counter)
i = 0
o = 0
for i,o in the_dictionary_links.items():
driver.get(o)
print(f'Downloading {i}')
time.sleep(1.8)
And unfortunately it's not working, it throws the following error:
InvalidArgumentException: invalid argument: cannot parse capability:
goog:chromeOptions from invalid argument: unrecognized chrome option:
prefs
So, I would like to know what could have gone wrong? I coded the program above based on this solution but it only seems to work for a new chrome session, and I need the download default directory to be capable of being reset when needed for an already open session. Any ideas?
I made my own Twitter complaint bot that tweets at my ISP if the network drops.
Code works perfect, until it has to find the Twitter textbox to type the tweet.
Main error is:
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
I have tried:
Adding time delays
Using Firefox Driver instead of Google
Adding page refreshes before the tweet_at_provider() looks for the textbox
Clicking the "Tweet" button to bring up the textbox to then try type in it
Using find.element_by_id but twitter changes id every pageload
When I comment out the first function call to test, it will find and type 6/10 times.
But when both functions are called the tweet_at_provider() always fails at grabbing the textbox and I get the StaleElement error.
import selenium, time, pyautogui
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import ElementClickInterceptedException, NoSuchElementException, StaleElementReferenceException
PROMISED_DOWN = 200
PROMISED_UP = 10
CHROME_DRIVER_PATH = "C:\Development\chromedriver.exe"
GECKODRIVER_PATH = "C:\\Users\\meeha\\Desktop\\geckodriver\\geckodriver.exe"
TWITTER_USERNAME = "my_username"
TWITTER_PASSWORD = "my_password"
class InternetSpeedTwitterBot():
def __init__(self, driver_path):
self.driver = webdriver.Chrome(executable_path=driver_path)
self.down = 0
self.up = 0
def get_internet_speed(self):
self.driver.get("https://www.speedtest.net/")
self.driver.maximize_window()
time.sleep(2)
go = self.driver.find_element_by_xpath("//*[#id='container']/div/div[3]/div/div/div/div[2]/div[3]/div[1]/a/span[4]")
go.click()
time.sleep(40)
self.down = self.driver.find_element_by_xpath("//*[#id='container']/div/div[3]/div/div/div/div[2]/div[3]/div[3]/div/div[3]/div/div/div[2]/div[1]/div[2]/div/div[2]/span")
self.up = self.driver.find_element_by_xpath("//*[#id='container']/div/div[3]/div/div/div/div[2]/div[3]/div[3]/div/div[3]/div/div/div[2]/div[1]/div[3]/div/div[2]/span")
print(f"Download Speed: {self.down.text} Mbps")
print(f"Upload Speed: {self.up.text} Mbps")
time.sleep(3)
def tweet_at_provider(self):
self.driver.get("https://twitter.com/login")
self.driver.maximize_window()
time.sleep(3)
username = self.driver.find_element_by_name("session[username_or_email]")
password = self.driver.find_element_by_name("session[password]")
username.send_keys(TWITTER_USERNAME)
password.send_keys(TWITTER_PASSWORD)
password.submit()
time.sleep(5)
tweet_compose = self.driver.find_element_by_xpath('//*[#id="react-root"]/div/div/div[2]/header/div/div/div/div[1]/div[3]/a/div/span/div/div/span/span')
tweet_compose.click()
time.sleep(2)
textbox = self.driver.find_element_by_xpath('//*[#id="layers"]/div[2]/div/div/div/div/div/div[2]/div[2]/div/div[3]/div/div/div/div[1]/div/div/div/div/div[2]/div[1]/div/div/div/div/div/div/div/div/div/div[1]/div/div/div/div[2]/div/div/div/div')
textbox.send_keys(f"Hey #Ask_Spectrum, why is my internet speed {self.down.text} down / {self.up.text} up when I pay for {PROMISED_DOWN} down / {PROMISED_UP} up???")
bot = InternetSpeedTwitterBot(CHROME_DRIVER_PATH)
bot.get_internet_speed()
bot.tweet_at_provider()
I had the same error there and figured out that HTML tag was instantly changing as soon as I was typing something on the twitter text-box.
tackle this problem using XPATH of span tag that was showing up after typing space from my side. break tag is the initial tag when there is not any text prompted by you, only after you type anything turns into and that's when you have to copy XPATH and use it for your application
I am using Selenium to automate a few browser actions on a particular website and I am using the below set of tools to achieve it.
Python 3.8
Selenium
Chrome Web Driver 79.0
Chrome 79.0
The tasks that I do are just fill up a form and then click on the submit button in the form. And this works most of the time except that sometimes it just won't no matter what! The filling up of form is very smooth and then, when the click on the submit button happens then Chrome just hangs there forever. There's no error on the console whatsoever and googling this issue I see this is such a common occurrence and almost all of the solution out there are just workarounds and not actual fixes. And I have tried almost all of them but to no avail. There's an issue on the selenium GitHub page as well which the maintainers weren't interested in much and closed it. How do I even go about resolving this issue. At this moment I am out of ideas actually. Any help would be appreciated. Thanks.
Below is the source code that I am trying to execute.
import time
from selenium import webdriver
import os
import csv
from datetime import datetime
url = 'https://www.nseindia.com/products/content/equities/equities/eq_security.htm'
xpath_get_data_button = '//*[#id="get"]'
xpath_download_link = '/html/body/div[2]/div[3]/div[2]/div[1]/div[3]/div/div[3]/div[1]/span[2]/a'
nse_list_file = 'nse_list.csv'
wait_time = 5
wait_time_long = 10
start_year = 2000
stop_year = 2019
curr_year = start_year
browser = webdriver.Chrome("E:/software/chromedriver_win32/chromedriver.exe")
browser.get(url)
time.sleep(wait_time)
with open(nse_list_file, 'r') as file:
reader = csv.reader(file)
for row in reader:
scrip = row[0]
year_registered = datetime.strptime(row[1], '%d-%m-%Y').year
if year_registered > start_year:
curr_year = year_registered
else:
curr_year = start_year
try:
browser.find_element_by_class_name('reporttitle1').clear()
browser.find_element_by_class_name('reporttitle1').send_keys(scrip)
browser.find_element_by_id('rdDateToDate').click()
while curr_year <= stop_year:
from_date = '01-01-' + str(curr_year)
to_date = '31-12-' + str(curr_year)
browser.find_element_by_id('fromDate').clear()
browser.find_element_by_id('fromDate').send_keys(from_date)
browser.find_element_by_id('toDate').clear()
browser.find_element_by_id('toDate').send_keys(to_date)
time.sleep(wait_time)
browser.find_element_by_xpath(xpath_get_data_button).click()
time.sleep(wait_time_long)
download_link_element = browser.find_element_by_xpath(xpath_download_link).click()
curr_year = curr_year + 1
except Exception as ex:
print('Could not find download link')
print(str(ex))
if os.path.isfile("stop_loading"):
break
browser.quit()
print('DONE')
I've written a script in python to parse the name, tweets, following and follower of those available in view all section in my profile page of twitter. It is currently doing it's job. However, I find two problems with this scraper:
Every pages it parses the documents from are jamming on the taskbar.
The scraper has got a clumsy look.
Here is what I've written:
from selenium import webdriver
import time
def twitter_data():
driver = webdriver.Chrome()
driver.get('https://twitter.com/?lang=en')
driver.find_element_by_xpath('//input[#id="signin-email"]').send_keys('username')
driver.find_element_by_xpath('//input[#id="signin-password"]').send_keys('password')
driver.find_element_by_xpath('//button[#type="submit"]').click()
driver.implicitly_wait(15)
#Clicking the viewall link
driver.find_element_by_xpath("//small[#class='view-all']//a[contains(#class,'js-view-all-link')]").click()
time.sleep(10)
for links in driver.find_elements_by_xpath("//div[#class='stream-item-header']//a[contains(#class,'js-user-profile-link')]"):
processing_files(links.get_attribute("href"))
#going on to the each profile falling under viewall section
def processing_files(item_link):
driver = webdriver.Chrome()
driver.get(item_link)
# getting information of each profile holder
for prof in driver.find_elements_by_xpath("//div[#class='route-profile']"):
name = prof.find_elements_by_xpath(".//h1[#class='ProfileHeaderCard-name']//a[contains(#class,'ProfileHeaderCard-nameLink')]")[0]
tweet = prof.find_elements_by_xpath(".//span[#class='ProfileNav-value']")[0]
following = prof.find_elements_by_xpath(".//span[#class='ProfileNav-value']")[1]
follower = prof.find_elements_by_xpath(".//span[#class='ProfileNav-value']")[2]
print(name.text, tweet.text, following.text, follower.text)
twitter_data()
I've used both the implicitly_wait and time.sleep in my scraper cause when i found that it was necessary to keep the bot wait a bit longer I used the latter. Thanks in advance to take a look into it.
You can use driver.quit() to close the pages as given below. This will reduce pages in the task bar.
from selenium import webdriver
import time
def twitter_data():
driver = webdriver.Chrome()
driver.get('https://twitter.com/?lang=en')
driver.find_element_by_xpath('//input[#id="signin-email"]').send_keys('username')
driver.find_element_by_xpath('//input[#id="signin-password"]').send_keys('password')
driver.find_element_by_xpath('//button[#type="submit"]').click()
driver.implicitly_wait(15)
#Clicking the viewall link
driver.find_element_by_xpath("//small[#class='view-all']//a[contains(#class,'js-view-all-link')]").click()
time.sleep(10)
for links in driver.find_elements_by_xpath("//div[#class='stream-item-header']//a[contains(#class,'js-user-profile-link')]"):
processing_files(links.get_attribute("href"))
driver.quit()
#going on to the each profile falling under viewall section
def processing_files(item_link):
driver1 = webdriver.Chrome()
driver1.get(item_link)
# getting information of each profile holder
for prof in driver1.find_elements_by_xpath("//div[#class='route-profile']"):
name = prof.find_elements_by_xpath(".//h1[#class='ProfileHeaderCard-name']//a[contains(#class,'ProfileHeaderCard-nameLink')]")[0]
tweet = prof.find_elements_by_xpath(".//span[#class='ProfileNav-value']")[0]
following = prof.find_elements_by_xpath(".//span[#class='ProfileNav-value']")[1]
follower = prof.find_elements_by_xpath(".//span[#class='ProfileNav-value']")[2]
print(name.text, tweet.text, following.text, follower.text)
driver1.quit ()
twitter_data()