Pulling Excel data across multiple firefox pages in Python, Selenium - excel

Goal: Take a list of First and Last names from Excel, and put them into an online registration form, using multiple firefox pages, with only one first name and one last name per page.
Tasks:
Open firefox page
Fill in "First Name" text box with Excel, cell 'A2'="Bob"
Fill in "Last Name" text box with Excel, cell 'B2'="Apple"
Click 'Submit'. -- End of Registration 1 --
Open a new firefox page
Fill in "First Name" text box with Excel, cell 'A3'="Linda"
Fill in "Last Name" text box with Excel, cell 'B3'= "Orange"
Click 'Submit'.
for x in range(2):
from selenium import webdriver
browser=webdriver.Firefox()
browser.get('The Website')
import openpyxl
wb=openpyxl.load_workbook('Names.xlsx')
sheet=wb.get_sheet_by_name('Full Names')
tuple(sheet['A2':'B3'])
#I'm guessing about this next part:
for rowOfCellObjects in sheet['A2':'B3']:
for cellObj in rowOfCellObjects:
browser.find_element_by_id('first_name').send_keys(????)
browser.find_element_by_id('last_name').send_keys(????)
Using Python 3.6.2. Excel 2016. Windows 10 x64. Selenium.
Please dumb it down in the answers, I'm very new to coding :). Thanks!!

This is my usual format:
import pandas as pd
from selenium import webdriver
driver = webdriver.Firefox()
headers = ['first_name', 'last_name']
data = pd.read_csv('Names.csv', names=headers) #Youll want to change the .xlsx to .csv
depth = len(data['first_names']) #this finds how deep the columns are
url = "www.website.com"
driver.get(url) #opens browser
for i in range (0,depth):
driver.find_element_by_xpath('first_name').send_keys(data['first_name'][i])
driver.find_element_by_xpath('last_name').send_keys(data['last_name'][i])
driver.find_element_by_xpath('submit').click()
Also note that in find_element_by_xpath, the format is:
driver.find_element_by_xpath('//input[#name = "first_name"]')
or similar. Youll need to ctl+i or right click-->Inspect to find the xpath.
'input' is the main tag name, and 'name' will be whatever element of 'input' has the "first_name" string literally embedded.

Related

Selecting options in dropdown Selenium ChromeDriver in VBA

I'm trying to select the options of a dropdown list in Chrome using VBA and Selenium.
I manage to open the dropdown but not navigate the options. Can anyone help?
This is the VBA code:
Sub FormChrome()
'Import Selenium Library
Dim driver As New ChromeDriver
'Navigate to the website
driver.Get "https://fremtindservice.no/privat/bompengekalkulator/"
' Wait for page to load
driver.Wait (10)
driver.Window.Maximize
' Locate the first input element using its class and set its value
driver.FindElementByCss("label[for='origin'] ~ div .pac-target-input").SendKeys "Oslo"
' Locate the second input element using its class and set its value
driver.FindElementByCss("label[for='destination'] ~ div .pac-target-input").SendKeys "Rakkestad"
driver.FindElementByCss("label[for='via'] ~ div .pac-target-input").SendKeys ""
'driver.FindElementByCss("div.dropdown.v-select.toll-road-calculator__vehicle-type.single.searchable").Click
' Import Selenium Library
' Wait for page to load
driver.Wait (10)
' Locate the dropdown element using its class and click to open it
driver.FindElementByCss("div.dropdown.v-select.toll-road-calculator__vehicle-type.single.searchable").Click
' Wait for page to load
driver.Wait (10)
' Locate the desired option using its position in the dropdown and click to select it
driver.FindElementByCss("div.vs__dropdown-menu div.vs__dropdown-item:nth-child(3)").Click
End Sub
Here is the HTML code for the Dropdown:
Biltype
Personbil
× Loading...
As a default it says Personbil. I want that text to be the third option in the dropdownlist which is "Lastebil 3.5 tonn (Euro V og eldre)"
Do you use Visual Basic or .net C# ? As far as I know, there is not VB bindings of Selenium.
Re anser, there is special function for select list(dropbox). For #:
yourSelectList=driver.FindElement(By.Id("select-demo"));
SelectElement dropDown = new SelectElement(yourSelectList);
and then you have 3 alternative to choose options:
dropDown.SelectByText("Four");
dropDown.SelectByValue("two");
dropDown.SelectByIndex(3);
for more details you can click here

how to copy to clipboard text between h2 tags in selenium python

what i try to do here is get email code for verification. so I log in to the email, select and copy the 6 digits code from the email and paste it to the other tab. everything is done except i can not double click to select the 6 digit code and copy it to clipboard. the code is between h2 tag and nothing else, like this: 639094 where 639094 is actually the code which i need to be copied. how can i find the code element or whatever and copy it? here is a screen shot of the email and the chrome inspect element if anything helps.
this is the code that I use to copy the code:
codeID = driver.find_element(By.XPATH,
'//table[#class="main"]//tr//td//p//h2').text
ActionChains = ActionChains(driver)
ActionChains.double_click(codeID).perform()
time.sleep(2)
codeID.send_keys(Keys.CONTROL + 'c')
text = pyperclip.paste()
print(text)
screen shot
element is found however looks like can not be copied. the error is Element is not reachable by keyboard. if i do everything automatically up until the element is selected with double click and copy the element with my actual keyboard the element is copied, however when selenium try to copy i get the error from above. the code i use to double click the element is:
codeID = driver.find_element(By.XPATH, '//*[#id="message-htmlpart1"]/div/table/tbody/tr/td[2]/div/table/tbody/tr/td/table/tbody/tr/td/h2')
ActionChains = ActionChains(driver)
ActionChains.double_click(codeID).perform()
time.sleep(2)
and to do the copy is :
codeID.send_keys(Keys.CONTROL + 'c')
text = pyperclip.paste()
print(text)
this is the part where the error ocur:
codeID.send_keys(Keys.CONTROL + 'c')
text = pyperclip.paste()
print(text)
for some reason it says "Element is not reachable by keyboard" but the element/code numbers are selected.
if I use print(text) they are also printed in the console.
driver.find_element_by_xpath('//table[#class="main"]//tr//td//h2').text this will give you the text/code
Hey i will analyse this problem with you
For the first part :
try to take that XPath you have and past it in the Xpath helper (google chrome extension)
=> If you find that element , than the problem in your code
=> if you don’t than the element is already in a frame or in a table
The solution is to change your drive to the new frame and relocate the element inside the frame
Exemple :
iframe_xpath = driver.find_element_by_xpath('//iframe')
driver.switch_to.frame('iframe_xpath')
Now try to relocate the element starting from the iframe
For the second part :
You say it’s a table so you need to mention the /td[i] and /tr[j] value where the number is located so you can get it
Exemple
d = driver.find_element_by_xpath( "//tr[i]/td[j]").text
I hope that’s help

Openpyxl returns wrong hyperlink address after delete_rows()

Problem: I have a program that scrapes Twitter and returns the results in an excel file. Part of each entry is a column containing a hyperlink to the Tweet and image included in the Tweet if applicable. Entries and hyperlinks work fine except when I run the following code to remove duplicate posts:
#Remove duplicate posts.
values = []
i = 2
while i <= sheet.max_row:
if sheet.cell(row=i,column=3).value in values:
sheet.delete_rows(i,1)
else:
values.append(sheet.cell(row=i,column=3).value)
i+=1
After running the duplicate removal snippet the hyperlinks point to what I assume is the offset of deleted entries. Here is the code for creating a Twitter entry:
sheet.cell(row=row, column=8).hyperlink = "https://twitter.com/"+str(tweet.user.screen_name)+"/status/"+str(tweet.id)
sheet.cell(row=row, column=8).style = "Hyperlink"
Expected Results: Should be able to remove duplicate entries and keep the hyperlink pointed to the correct address.
The hyperlinks point to the correct addresses for whatever reason when I change the code to the this:
sheet.cell(row=row, column=8).value = "https://twitter.com/"+str(tweet.user.screen_name)+"/status/"+str(tweet.id)
sheet.cell(row=row, column=8).style = "Hyperlink"
Requires a rapid double click to work as a hyperlink in the excel sheet versus the one click when inserting using .hyperlink.
So fixed but not fixed.

Google search next pages using selenium

I'm trying to automate the clicking of the next page in google search, after I must have gone into the links in the 1st and 2nd search page.
I've so far been able to do the following:
Spin up the chrome browser
Go to the Google webpage
Type in the search words
Click on the search icon
Go into the links on the 1st and 2nd google page
See my code below:
from time import sleep
from selenium import webdriver
from parsel import Selector
from selenium.webdriver.common.keys import Keys
#path to the chromedriver
driver = webdriver.Chrome('/Users\my_path/chromedriver')
driver.get('https://www.gooogle.com')
#locate search form by name
search_query = driver.find_element_by_name('q')
#Input search words
search_query.send_keys('X-Men')
#Simulate return key
search_query.send_keys(Keys.RETURN)
Xmen_urls = driver.find_elements_by_class_name('iUh30')
for page in range(0,3):
Xmen_urls = [url.text for url in Xmen_urls]
#loop to iterate through all links in the google search query
for Xmen_url in Xmen_urls:
driver.get(Xmen_url)
sel = Selector(text = driver.page_source)
#Go back to google search
driver.get('https://www.gooogle.com')
#locate search form by name
search_query = driver.find_element_by_name('q')
#Input search words
search_query.send_keys('X-Men')
#Simulate return key
search_query.send_keys(Keys.RETURN)
#find next page icon in Google search
Next_Google_page = driver.find_element_by_link_text("Next").click()
page += 1
When I'm done collecting the links on the '2nd' search page, how do I tell the algorithm to start from the '2nd' search page and not the 1st search page (this will enable me go into >2 pages).
I know it's a 'for loop' and syntax re-arranging I'm missing somewhere but my brain is frozen at this point.
I saw this page: How to click the next link in google search results? but it only helps if I'm not navigating away from the google search page
What am I doing wrong?
There are two ways I can see:
Open each X-Men url in a separate window using window_handles, collect page_source, close the window and switch back to the original window.
driver.execute_script("window.open(X-Men_url, 'new_window')")
driver.switch_to.window(driver.window_handles[1])
sel = Selector(text = driver.page_source)
driver.close()
driver.switch_to.window(driver.window_handles[0])
The code above may not work exactly, but something to that effect.
The other way is to simulate a number of clicks on NEXT at the beginning of your FOR loop using a loop:
a = 0;
while a <= page:
driver.find_element_by_xpath("//*[contains(local-name(), 'span') and contains(text(), 'Next')]").click()
a = a+1

Using chromedriver to click on dropdown menu and get table of new webpage

I am creating a code to click on each option of the dropdown menu and then get the content of the new webpage, which has a table. I want to save one file for each option of the dropdown menu.
My code doesn't get these infos right now. I ain't sure if it's possible with chromedriver and Python. Could you give a help?
The website is: http://www2.camara.leg.br/deputados/pesquisa
On the first dropdown menu (below "Legislatura Atual - Deputados em Exercício") you have the name of 513 politicians in Brazil. I should choose one name per time, then select "presença em plenário" and then click on "pesquisar". The table which shows on the new webpage should be saved as a file named with the politician's name.
The same situation happens for the other names.
Below is the code that is workable.
from selenium import webdriver
from selenium.webdriver.support.ui import Select
path_to_chromedriver = 'path_to_chromedriver'
chrome_options = webdriver.ChromeOptions()
browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=path_to_chromedriver)
prefs = {"profile.default_content_setting_values.notifications": 2}
chrome_options.add_experimental_option("prefs", prefs)
chrome_options.add_argument("start-maximized")
browser.get('http://www2.camara.leg.br/deputados/pesquisa')
Contact_data = browser.find_element_by_class_name('form-control')
listing = Contact_data.find_elements_by_tag_name("option")
listing1 =[]
for element in listing:
dropdownvalue = listing1.append(element.text)
for i in range(len(listing1)):
Selection = Select(browser.find_element_by_class_name('form-control'))
Selection.select_by_visible_text(listing1[i+1])
browser.find_element_by_id('rbDeputado7').click()
browser.find_element_by_name('Pesquisa').click()

Resources