I was happily extracting information from a page, which requires clicking on a link to see the email. Everything was working fine until I turned off my laptop for a few days, I'm trying to do this task again but for some reason, it isn't working anymore :( (I never changed my code or anything). Please HELP!
Before anything, I tried to replace that click with Actions, tried to use Java, changed the webdriver, changed the waiting time, tried with CSS_SELECTOR, XPATH, etc (previously worked only with LINK_TEXT) but nothing I do seems to work. Here's my code, I censored my information:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
DRIVER_PATH = '/Users/andre/Desktop/geckodriver'
driver = webdriver.Firefox(executable_path=DRIVER_PATH)
driver.get('https://membership.icsc.com/login')
USERNAME = 'XXXXXXXXXXXX'
PASSWORD = 'XXXXXX'
#login first
login = driver.find_element(By.NAME, '_username').send_keys(USERNAME)
password = driver.find_element(By.NAME, '_password').send_keys(PASSWORD)
submit = driver.find_element(By.CSS_SELECTOR, "body > div > div > div > div > div > div > form > div.text-center > p:nth-child(1) > button").click()
main = driver.get("THEURLISTOOLONGTOPUTITHERE")
#thescraper666
def get_scarping(link):
driver.get(link)
try:
email = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Click here to display email")))
email.click()
driver.implicitly_wait(5)
email2 = driver.find_element(By.XPATH, '/html/body/div[5]/div/div/div/div/div/div[2]/div[2]/div/div/div[1]/div[2]/a').text
print(email2)
except NoSuchElementException:
print("NO EMAIL")
return driver.current_url
links = ['A LIST WITH A BUNCH OF URLS']
scrapings = []
for link in links:
scrapings.append(get_scarping(link))
Related
I have compiled a code that would Create an Account on https://www.walmart.com/ using selenium python. The code opens Walmart website goes to Create an account tab, fills the required details and click on Create Account button. However, the problem is Walmart's Human verification challenge which appears randomly at any stage. Following are the snapshots that shows the Human verification challenge appearing just after opening the URL or after clicking on create account button:
I have found a code on stackoverflow to bypass this challenge (shown below) but it didn;t work fro me.
element = driver.find_element(By.CSS_SELECTOR, '#px-captcha')
action = ActionChains(driver)
action.click_and_hold(element)
action.perform()
time.sleep(100)
action.release(element)
action.perform()
time.sleep(0.2)
action.release(element)
For reference my python code is as follows:
import time
import requests
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
url = "https://www.walmart.com/"
first_name = "chuza"
last_name = "789"
email_id = "chuza789#gmail.com"
password = "Eureka1#"
options = Options()
s=Service('C:/Users/Samiullah/.wdm/drivers/chromedriver/win32/96.0.4664.45/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source":
"const newProto = navigator.__proto__;"
"delete newProto.webdriver;"
"navigator.__proto__ = newProto;"
})
wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)
driver.get(url)
sign_in_btn = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Sign In']")))
actions.move_to_element(sign_in_btn).perform()
time.sleep(0.5)
wait.until(EC.visibility_of_element_located((By.XPATH, '//button[normalize-space()="Create an account"]'))).click()
f_name = driver.find_element(By.ID, 'first-name-su')
l_name = driver.find_element(By.ID, 'last-name-su')
email = driver.find_element(By.ID, 'email-su')
pswd = driver.find_element(By.ID, 'password-su')
f_name.send_keys(first_name)
driver.implicitly_wait(2)
l_name.send_keys(last_name)
driver.implicitly_wait(2.5)
email.send_keys(email_id)
driver.implicitly_wait(3)
pswd.send_keys(password)
driver.implicitly_wait(2.8)
###
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-automation-id='signup-submit-btn']"))).click()
Can anyone please guide me how to solve Walmart's Robot or Human? Challenge and how to integrate it with my existing code? Thanks in advance.
The simplest way to overcome this is to check for this element presence.
In case this element appears - perform the page reloading until this element is no more present.
Something like this:
human_dialog = driver.find_elements(By.XPATH, "//div[#aria-labelledby='ld_modalTitle_0']")
while human_dialog:
driver.refresh()
time.sleep(1)
human_dialog = driver.find_elements(By.XPATH, "//div[#aria-labelledby='ld_modalTitle_0']")
I'm in need of a solution to my code, I tried to web scraping a dynamic web page call easy.cl and just get 4 items and sometimes none (only when I download title, cant download price because don't show anything). Well, anyhow, I need a guide of where is my error, because Selenium don't show me any in my result (Sublime text3). Also, easy.cl is dynamic going with a button to show more info products. Finally, I'm thinking in a scroll solution but can't tell, What would you do in my position? any tip to try to find a solution?
in advanced, thanks.
import random
from time import sleep
from selenium import webdriver
import pandas as pd
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
#baseUrl = 'https://www.easy.cl'
driver.get('https://www.easy.cl/tienda/categoria/ceramicas?cur_page=1&cur_view=grid&bc_links=pisos-y-muros&bc_names=Pisos')
#boton = driver.find_element_by_xpath('//button[#class="primary_plp load-more-products"]')
inicio = []
for i in range(5):
try:
boton = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '//button[#class="primary_plp load-more-products"]')))
boton.click()
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH,'//a[#class="product_image"]')))
except:
break
# Espero que carguen los productos...
links_productos = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.XPATH,'//a[#class="product_image"]')))
# obtengo los links de los detalles de los productos
links_pagina=[]
for tag_a in links_productos:
links_pagina.append(tag_a.get_attribute("href"))
for link in links_pagina:
try:
driver.get(link)
titulo = driver.find_element(By.XPATH, '//h1[#class="product-details__title"]').text
#if driver.find_element(By.XPATH, '//span[#class="priceNumber priceNumberAlignRight"]'):
# precio = find_element_by_xpath('//span[#class="priceNumber priceNumberAlignRight"]').text
#else:
# precio = "Sin precio M2"
print(titulo)
#print(precio)
except:
break
The load more products button appears on the bottom of the page, out of the visible screen, so possibly after the element is presented (loaded) you need to scroll to that element before clicking it
from selenium.webdriver.common.action_chains import ActionChains
boton = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '//button[#class="primary_plp load-more-products"]')))
actions = ActionChains(driver)
actions.move_to_element(boton).perform()
boton.click()
But the main issue here, as I see, is
links_productos = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.XPATH,'//a[#class="product_image"]')))
You assume that this will give you all the elements that can be presented on that page matching that locator, but actually this expected condition is waiting for at least 1 element matching this condition and once it catches it - it returns a list of web elements matching that locator it found.
I see no expected condition in Python like we have in Java like this ExpectedConditions.numberOfElementsToBe or this ExpectedConditions.numberOfElementsToBeMoreThan that is actually waiting for desired amount of elements located by passed locator. So, what I can advise you here, is to add a delay of 1-2 seconds after clicking the boton.click() before reading the links_productos. This should resolve your problem.
I'm trying to automatize the reddit logIn with selenium from python and i'm using the following code in order to do it
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from time import sleep
driver = webdriver.Chrome(executable_path=r'C:\Program Files (x86)\chromedriver.exe')
driver.get("https://www.reddit.com/")
login=driver.find_element_by_link_text("Log In")
login.click()
username = "the-username" # Enter your username
password = "the-password" # Enter your password
def slow_typing(element, text):
for character in text:
element.send_keys(character)
sleep(0.3)
def logIn(): # Log In Function.
try:
sleep(15)
#username_in = driver.find_element_by_class_name("AnimatedForm__textInput")
username_in = driver.find_element_by_xpath("//*[#id='loginUsername']")
slow_typing(username_in, username)
pass_in = driver.find_element_by_xpath("//*[#id='loginPassword']")
slow_typing(pass_in,password)
pass_in.send_keys(Keys.ENTER)
sleep(5)
except NoSuchElementException:
print("Llegue aqui xd xd")
logIn()
There's a little more code, but I'm posting a summary so I can tell my problem to you guys. When it is running, it comes to the moment where the input of the username is selected, but it doesn't send the keys. I don't know what to do or change, so I ask for some help here.
def logIn(): # Log In Function.
try:
driver.switch_to_frame(driver.find_element_by_tag_name('iframe'))
sleep(5)
print("hii")
#username_in = driver.find_element_by_class_name("AnimatedForm__textInput")
username_in = driver.find_element_by_xpath("//*[#id='loginUsername']")
slow_typing(username_in, username)
pass_in = driver.find_element_by_xpath("//*[#id='loginPassword']")
slow_typing(pass_in, password)
pass_in.send_keys(Keys.ENTER)
sleep(5)
driver.switch_to_default_content()
except NoSuchElementException:
print("Llegue aqui xd xd")
driver.switch_to_default_content()
The login is inside an iframe witch to it first
To login within reddit using Selenium and python you need to:
Induce WebDriverWait for the frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using XPATH:
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("prefs", { \
"profile.default_content_setting_values.notifications": 1
})
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.reddit.com/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[starts-with(#href, 'https://www.reddit.com/login')]"))).click()
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[starts-with(#src, 'https://www.reddit.com/login')]")))
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//input[#id='loginUsername']"))).send_keys("debanjanb")
driver.find_element(By.XPATH, "//input[#id='loginPassword']").send_keys("zergcore")
driver.find_element(By.XPATH, "//button[#class='AnimatedForm__submitButton m-full-width']").click()
Note : You have to add the following imports :
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
That happens because you don't click inside the input element - below you'll find a small change in your slow typing method, also if have a close look at their code they also have an animation for those fields, clicking the input should solve your issue.
def slow_typing(element, text):
element.click()
for character in text:
element.send_keys(character)
sleep(0.3)
The second recommendation is to use ids instead of XPath whenever you have the chance. Ids give you the best performance and also helps the framework to find the elements easily and not to parse the entire DOM to match the xpath.
How can I click button "View profile" on this page.
Problem: when I want to copy XPath of "View profile" button, the button disappeares.
https://www.linkedin.com/sales/search/people?savedSearchId=515913166
[]
Another image
This is my code. Reads email and password from config.txt file.
Goes to linkedin site, then goes to Sales Navigator
My code
import os, random, sys, time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
browser = webdriver.Chrome('driver/chromedriver')
browser.get('https://www.linkedin.com/uas/login')
file = open('config.txt')
lines = file.readlines()
username = lines[0]
password = lines[1]
elementID = browser.find_element_by_id('username')
elementID.send_keys(username)
elementID = browser.find_element_by_id('password')
elementID.send_keys(password)
visitingSalesNavID = '/sales/homepage/'
fullLink = 'https://www.linkedin.com' + visitingSalesNavID
browser.get(fullLink)
time.sleep(4)
SavedSearchesID = '/sales/search/saved-searches/people'
fullLink = 'https://www.linkedin.com' + SavedSearchesID
browser.get(fullLink)
time.sleep(4)
SavedSearchID = '/sales/search/people?savedSearchId=515913166'
fullLink = 'https://www.linkedin.com' + SavedSearchID
browser.get(fullLink)
time.sleep(4)
browser.find_element_by_xpath('/html[1]/body[1]/div[5]/main[1]/div[1]/div[1]/section[1]/div[1]/ol[1]/li[1]/div[2]/div[1]/div[1]/div[1]/article[1]/section[1]/div[2]/ul[1]/li[1]/div[1]/div[2]/div[1]/div[1]/button[1]/li-icon[1]/*').click()
You have to freeze the DOM:
1) Open the dev console (F12 key)
2) Select sources tab
3) click F8 to freeze the DOM
Then do whatever you do with that
You don’t need to look element’s detail. You can click with the text. Try below code after open menu.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 50).until(
EC.presence_of_element_located((By.XPATH, "//*[contains(., 'View profile')]")))
element.click()
Working in python, selenium, Win 7, I want to click on sowcomment button after it is able to clickable that is located on this webPage, handled by wait then I want to click on showmore Comments button to see more comments in order to scrape more comments. After first button, i am able to extract comments.
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
import selenium.common.exceptions
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import selenium.webdriver.support.ui as UI
driver = webdriver.Firefox()
driver.get("http://www.aljazeera.com/news/2016/07/erdogan-west-mind-business-160729205425215.html")
wait = UI.WebDriverWait(driver, 10)
next_page_link = wait.until(
EC.element_to_be_clickable((By.ID, 'showcomment')))
next_page_link.click()
wait = UI.WebDriverWait(driver, 20)
next_page_link2 = wait.until(
EC.element_to_be_clickable((By.LINK_TEXT, 'Show more comments')))
next_page_link2.click()
v = driver.find_elements_by_class_name('gig-comment-body')
print(v)
for h in v:
print(h.text)
but second button is unable to click rather giving the exception:
selenium.common.exceptions.TimeoutException:
What is the problem?
I think you should try using execute_script() to perform click as below :
next_page_link2 = wait.until(
EC.element_to_be_clickable((By.XPATH, '//div[contains(text(),"Show more comments")]')))
#now click this using execute_script
driver.execute_script("arguments[0].click()", next_page_link2)
Hope it helps...:)