How to Click Link Being Covered by Another Element? Python 3.6 and Selenium - python-3.x

Question

Having trouble figuring out how to click the Next button at the bottom of the table on this page:
https://www.zacks.com/stocks/industry-rank/reit-and-equity-trust-other-266/stocks-in-industry
This is what I've tried:
from bs4 import BeautifulSoup
import requests
import csv, random, time
from pandas.io.html import read_html
from selenium import webdriver
from selenium.webdriver.support.ui import Select
url = 'https://www.zacks.com/stocks/industry-rank/reit-and-equity-trust-other-266/stocks-in-industry'
# Open Chrome
driver = webdriver.Chrome()
# Send Chrome to the URL
page = driver.get(url)
# Wait for page to load a few seconds
timeDelay = random.randrange(4, 8)
time.sleep(timeDelay)
# Try to click the darn button
element = driver.find_element_by_xpath('//*[#id="industry_rank_table_next"]')
driver.execute_script("arguments[0].click();", element)
...and
element = driver.find_element_by_xpath('//*[#id="industry_rank_table_next"]')
element.send_keys("\n")
...found from other answers but not working for me. Simply using .click() does not work. I've also tried selecting the button using css_selector, partial_link_text, and class_name but still no success. I've ran into this on a few sites. Any ideas?

Answer 1

To click() on the element with text as Next you have to induce WebDriverWait for the element to be clickable and you can use either of the following solutions:
Using LINK_TEXT:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Next"))).click()
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.paginate_button next#industry_rank_table_next"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='paginate_button next' and #id='industry_rank_table_next']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Answer 2

if the element which you have to click have parent element then you can find parent by findElements method and after than you just need to pass the index of the element like 0 or 1 or so on.. and then you can perform click action on that
Want to click on third li from second ul:
<ul id="select-123" style="width: 1180px; display: none;">
<li class="" style="display:none;">
<li class="">
<li class="">
<li class="">
</ul>
<ul id="select-123" style="width: 1180px; display: none;">
<li class="" style="display:none;">
<li class="">
<li class="">
<li class="">
</ul>
Code I am trying is to select third li from second ul which does not work:
driver.findElements(By.css(ul[id*='select-123'])).then(function(elems) {
elems[2].then(function(lis) {
driver.findElement(By.css("ul[id*='select-123'] li:nth-child(3)")).click();
});
});

Answer 3

Another way if you don't wish to add additional wait(s) to your code and just click the button:
using javascript click:
element = driver.find_element_by_xpath("//a[#class='paginate_button next' and #id='industry_rank_table_next']")
driver.execute_script("arguments[0].click();", element)

Answer 4

UPDATE:
I didn't notice this earlier, but
element = driver.find_element_by_xpath('//*[#id="industry_rank_table_next"]')
driver.execute_script("arguments[0].click();", element)
...works as it clicks the link successfully, but gives me a
selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
...error after it clicks the link. So to get around this, I've just added a try/except to handle the error. ie:
try:
element = driver.find_element_by_xpath('//*[#id="industry_rank_table_next"]')
driver.execute_script("arguments[0].click();", element)
except:
pass
...which seems to work. Seems like such a stupid observation on my part, but thank you to everyone for your help. Hopefully something in here will help someone else in the future.