I was trying to remove the ad and other pop-up using selenium after opening the webpage. The pop-up is getting removed but the ad is not getting removed. There is some error in executing javascript code(using it remove ad). For which there is no reason given. Also, for the ad, when I open the webpage myself by typing the link then ad does not appear but when I run the program, in code generated browser ad appears(reason I don't know). I have attached both code and error. Here is the code
from selenium import webdriver
import time
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("http://indianexpress.com/columnists/")
// To remove pop-up
later = driver.find_elements(By.CLASS_NAME, "iz_block_button")
later[0].click()
//to remove ad
all_iframes = driver.find_elements_by_tag_name("iframe")
if len(all_iframes) > 0:
print("Ad Found\n")
driver.execute_script("""
var elems = document.getElementsByTagName('iframe');
for(var i = 0, max = elems.length; i < max; i++)
{
elems[i].visibility=hidden;
}
""")
print('Total Ads: ' + str(len(all_iframes)))
else:
print('No frames found')
driver.close()
Error:-
Traceback (most recent call last):
File "/Users/arjungoyal/Desktop/untitled/a.py", line 23, in <module>
""")
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 627, in execute_script
'args': converted_args})['value']
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: hidden is not defined
(Session info: chrome=66.0.3359.181)
(Driver info: chromedriver=2.38.552518 (183d19265345f54ce39cbb94cf81ba5f15905011),platform=Mac OS X 10.13.4 x86_64)
Please someone tell what the error is and how to remove it.
Related
Please find the attached image . I want to fetch the highlighted part in the image,
i want to fetch this attributes 1 · Trending ,#tuesdaymotivations,7,750 Tweets .
please advise.
URL to fetch=https://twitter.com/explore/tabs/trending
from selenium import webdriver
url = 'https://twitter.com/explore/tabs/trending'
# scrolling and scraping tweets
driver = webdriver.Chrome('/chromedriver')
driver.get(url)
trends = driver.find_element_by_xpath('//div[#data-testid="trends"]')
trend = trends[0]
trend.find_element_by_xpath('.//span').text
output:
Traceback (most recent call last):
File "/home/PycharmProject/Twitter_trending/ss.py", line 13, in <module>
trends = driver.find_element_by_xpath('//div[#data-testid="trends"]')
File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#data-testid="trends"]"}
(Session info: chrome=90.0.4430.72)
expected output:
1
·
Trending
#tuesdaymotivations
7,750 Tweets
2
·
Politics · Trending
#ModiResignOrRepeal
37.6K Tweets
3
·
Trending
#ThankfulTuesday
12.8K Tweets...etc
Try to fetch the text using JavaScript. Use this code.
span = trend.find_element_by_xpath('.//span')
text = driver.execute_script("return arguments[0].innerText", span)
This method can also provide text from nested elements. Still not working then try css selector instead of xpath.
trends = driver.find_elements_by_css_selector('div[data-testid="trends"]')
trend = trends[0]
span = trend.find_element_by_xpath('.//span')
text = driver.execute_script("return arguments[0].innerText", span)
I have made a program to grab a youtube link and type it on a website but its not able to find the input box...plz tell me why
import time
from selenium import webdriver
import pyautogui as auto
url = 'https://www.youtube.com/results?search_query=closer+8d'
web = webdriver.Chrome('chromedriver.exe')
web.get(url)
time.sleep(3)
link = web.find_element_by_id('video-title')
link = link.get_attribute('href')
auto.hotkey('ctrl', 't')
auto.typewrite('https://www.youtubemp3.pro/en2')
auto.press('enter')
time.sleep(3)
search_bar = web.find_element_by_xpath('//*[#id="videoURL"]')
search_bar.click()
search_bar.send_keys(link)
The error I'm getting is:
Traceback (most recent call last):
File "e:\pranil\python\doing shit with python\songs downloader.py", line 17, in <module>
search_bar = web.find_element_by_xpath('//*[#id="videoURL"]')
File "C:\Users\Pranil.DESKTOP-TLQKP4G.000\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\Pranil.DESKTOP-TLQKP4G.000\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Users\Pranil.DESKTOP-TLQKP4G.000\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Pranil.DESKTOP-TLQKP4G.000\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="videoURL"]"}
(Session info: chrome=83.0.4103.116)
The xpath you provided to find an element doesn't seem to work ""//[#id="videoURL"]"" I also opened up a YouTube video to see what you were looking for but I couldn't find videoURL if you are trying to get the URL I recommend you have selenium click on the share button and grab the link from there.
import time
from selenium import webdriver
import pyautogui as auto
url = 'https://www.youtube.com/results?search_query=closer+8d'
web = webdriver.Chrome('./chromedriver')
web.get(url)
time.sleep(3)
link = web.find_element_by_id('video-title')
link = link.get_attribute('href')
web.execute_script("window.open('https://www.youtubemp3.pro/en2', 'new_window')")
web.switch_to_window(web.window_handles[-1])
ylink = web.find_element_by_id("videoURL")
ylink.click()
ylink.send_keys(link)
submit = web.find_element_by_name("submitForm")
submit.click()
you do not have to use pyautogui it will work better if you use seleniu
try to find the element with it's ID.
import time
from selenium import webdriver
url = 'https://www.youtube.com/results?search_query=closer+8d'
web = webdriver.Chrome('../assets/chromedriver')
web.get(url)
time.sleep(3)
link = web.find_element_by_id('video-title')
link = link.get_attribute('href')
web.get('https://www.youtubemp3.pro/en2/')
time.sleep(3)
search_bar = web.find_element_by_id('txt-url')
search_bar.click()
search_bar.send_keys(link)
here is my code so far:
from selenium import webdriver
import time
option = webdriver.ChromeOptions()
option.binary_location = 'C:\Program Files (x86)\AVG\Browser\Application\AVGBrowser.exe'
driver = webdriver.Chrome(executable_path='C:\Program Files (x86)\AVG\Browser\Application\AVGBrowser.exe', options=option)
driver.get("https://www.google.com")
time.sleep(3)
driver.quit() # What im trying to fix
The new browser window opens but does not quit after 3 seconds. It just gives me the error:
C:\Users\Tomas\PycharmProjects\AutoSpotify\venv\Scripts\python.exe C:/Users/Tomas/PycharmProjects/AutoSpotify/venv/Scripts/StartSpotify.py
Traceback (most recent call last):
File "C:/Users/Tomas/PycharmProjects/AutoSpotify/venv/Scripts/StartSpotify.py", line 4, in <module>
driver = webdriver.Chrome(executable_path='C:\Program Files (x86)\AVG\Browser\Application\AVGBrowser.exe')
File "C:\Users\Tomas\PycharmProjects\AutoSpotify\venv\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
self.service.start()
File "C:\Users\Tomas\PycharmProjects\AutoSpotify\venv\lib\site-packages\selenium\webdriver\common\service.py", line 98, in start
self.assert_process_still_running()
File "C:\Users\Tomas\PycharmProjects\AutoSpotify\venv\lib\site-packages\selenium\webdriver\common\service.py", line 111, in assert_process_still_running
% (self.path, return_code)
selenium.common.exceptions.WebDriverException: Message: Service C:\Program Files (x86)\AVG\Browser\Application\AVGBrowser.exe unexpectedly exited. Status code was: 0
Process finished with exit code 1
Hay i just tested the code in my linux it& looks fine :
Try this :
# Optional argument : if not specified WebDriver will search your system PATH environment variable for locating the chromedriver
For windows:
# try to place your driver within (Downloads) for example, no need to place it inside programe file or somewhere far.
driver = webdriver.Chrome("C:\\Users\\username\\Downloads\\chromedriver")
You should call your quit in a finally like so:
...
driver = webdriver.Chrome(executable_path='C:\Program Files (x86)\AVG\Browser\Application\AVGBrowser.exe', options=option)
try:
driver.get("https://www.google.com")
...
finally:
driver.quit()
This way, the quit function is called even if you encounter an exception while working with your driver object.
I'm trying to click a "load more" button with the following code
browser.find_element_by_xpath('//*[#id="mainContent"]/div[1]/div/div[5]/div/div[1]').click()
browser.find_element_by_css_selector('#mainContent > div.left-panel > div > div.result-list > div > div.content').click()
browser.find_element_by_link_text('Load More').click()
I receive the following error:
Traceback (most recent call last):
File "C:\Users\David\eclipse-workspace\Web_Scrap\setup.py", line 38, in <module>
browser.find_element_by_css_selector('#mainContent > div.left-panel > div > div.result-list > div > div.content').click()
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element is not clickable at point (239, 698)
(Session info: chrome=68.0.3440.106)
(Driver info: chromedriver=2.41.578737 (49da6702b16031c40d63e5618de03a32ff6c197e),platform=Windows NT 10.0.17134 x86_64)
I've tried each of those 3 individually but can't seem to get selenium to click the button
When pressing inspect that is the following element I receive
The element code is as follows:
<div class="content" onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROFMIDPANE:LoadMore', linkType:'o' } );">Load More</div>
if anyone has any recommendations on how I can achieve this I would greatly appreciate it!
UPDATE:
I tried the two solutions recommended to me but unfortunately didn't work out, I will post it here if anyone is interested.
iamsankalp89 solution:
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='content' and text()='Load More']")))
element.click()
Error message:
Traceback (most recent call last):
File "C:\Users\David\eclipse-workspace\Web_Scrap\setup.py", line 39, in <module>
element.click()
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element is not clickable at point (239, 698)
(Session info: chrome=68.0.3440.106)
(Driver info: chromedriver=2.41.578737 (49da6702b16031c40d63e5618de03a32ff6c197e),platform=Windows NT 10.0.17134 x86_64)
Julian Moreno solution:
ActionChains(driver).move_to_element("//div[#class='content' and text()='Load More']").click("//div[#class='content' and text()='Load More']").perform()
Error Message:
Traceback (most recent call last):
File "C:\Users\David\eclipse-workspace\Web_Scrap\setup.py", line 42, in <module>
ActionChains(browser).move_to_element("//div[#class='content' and text()='Load More").click("//div[#class='content' and text()='Load More").perform()
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\action_chains.py", line 83, in perform
action()
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\action_chains.py", line 293, in <lambda>
Command.MOVE_TO, {'element': to_element.id}))
AttributeError: 'str' object has no attribute 'id'
Try this xpath
//div[#class='content' and text()='Load More']
The code is like this:
browser.find_element_by_xpath('//div[#class='content' and text()='Load More']').click()
Also use WebDriverWait
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='content' and text()='Load More']")))
element.click()
Try using ActionChains:
class selenium.webdriver.common.action_chains.ActionChains(driver)
ActionChains(driver).move_to_element(your_element).click(your_element).perform()
The move to element will move the mouse (cursor) to the middle of the element (your_element) then the perform function will perform the actions chained.
EDIT
Try this:
load_more = browser.find_element_by_css_selector("#mainContent > div.left-panel > div > div.result-list > div > div.content")
WebDriverWait(browser, timeout).until(EC.visibility_of(load_more))
browser.execute_script("return arguments[0].scrollIntoView(true);", load_more)
ActionChains(browser).move_to_element(load_more).click().perform()
The ActionChains(browser).move_to_element() accepts WebElement object or String which is the id of the element. Since the load more does not have an ID then the move_to_element() could not find the WebElement.
After analysing the web page, I noticed that move_to_element(load_more) moves to the load_more button (only until it is within the screen) but it also triggers scrolling down and that displays the footer of the web page. This footer covers the load_more button. Therefore, you need the browser.execute_script("return arguments[0].scrollIntoView(true);", load_more) which will basically keep scrolling until load_more is in view (regardless of the footer showing up).
My code is:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
baseurl = "https://www.google.ca/?gfe_rd=cr&ei=J5ooWerXOsf_8AebtKKICw&gws_rd=ssl"
search = "panda"
xpaths = { 'searchbox' : ".//*[#id='lst-ib']",
'submit' : ".//*[#id='tsf']/div[2]/div[3]/center/input[1]",
'img' : ".//*[#id='gbw']/div/div/div[1]/div[2]/a"
}
driver = webdriver.Firefox()
driver.get(baseurl)
driver.find_element_by_xpath(xpaths['searchbox']).clear()
driver.find_element_by_xpath(xpaths['searchbox']).send_keys(search)
driver.find_element_by_xpath(xpaths['submit']).click()
#driver.find_element_by_xpath(xpaths['img']).click()
Firefox opens, but nothing at all happens, and written in the terminal is the following:
Traceback (most recent call last):
File "sg1.py", line 21, in <module>
driver.find_element_by_xpath(xpaths['searchbox']).send_keys(search)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webelement.py", line 347, in send_keys
self._execute(Command.SEND_KEYS_TO_ELEMENT, {'value': keys_to_typing(value)})
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webelement.py", line 494, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Expected [object Undefined] undefined to be a string
Things to note:
Firefox is up to date.
When Firefox opens it opens a plain version and not the version that usually opens with add-ons such as adblocker, firebug, etc.
When i ran just the click on 'img' bit that is commented out it did what it was supposed too.
This issue is common with geckodriver v.015; In order to resolve this update your geckodriver version to 0.16 also selenium to 3.4.0.