selenium python ElementNotInteractableException [duplicate] - python-3.x

Here I have the image of my code and the image of my error. Can anyone help me to resolve this issue?

ElementNotInteractableException
ElementNotInteractableException is the W3C exception which is thrown to indicate that although an element is present on the HTML DOM, it is not in a state that can be interacted with.
Reasons & Solutions :
The reason for ElementNotInteractableException to occur can be numerous.
Temporary Overlay of other WebElement over the WebElement of our interest :
In this case, the direct solution would have been to induce ExplicitWait i.e. WebDriverWait in combination with ExpectedCondition as invisibilityOfElementLocated as folllows:
WebDriverWait wait2 = new WebDriverWait(driver, 10);
wait2.until(ExpectedConditions.invisibilityOfElementLocated(By.xpath("xpath_of_element_to_be_invisible")));
driver.findElement(By.xpath("xpath_element_to_be_clicked")).click();
A better solution will be to get a bit more granular and instead of using ExpectedCondition as invisibilityOfElementLocated we can use ExpectedCondition as elementToBeClickable as follows:
WebDriverWait wait1 = new WebDriverWait(driver, 10);
WebElement element1 = wait1.until(ExpectedConditions.elementToBeClickable(By.xpath("xpath_of_element_to_be_clicked")));
element1.click();
Permanent Overlay of other WebElement over the WebElement of our interest :
If the overlay is a permanent one in this case we have to cast the WebDriver instance as JavascriptExecutor and perform the click operation as follows:
WebElement ele = driver.findElement(By.xpath("element_xpath"));
JavascriptExecutor executor = (JavascriptExecutor)driver;
executor.executeScript("arguments[0].click();", ele);

I got this because the element I wanted to interact with was covered by another element. In my case it was an opaque overlay to make everything r/o.
When trying to click an element UNDER another element we usualy get "... other Element would receive the click " but not always :.(

This Exception we get when the element is not in an interactable state. So we can use wait till the element is Located or become clickable.
Try using the Implicit wait:
driver.manage().timeouts().implicitlyWait(Time, TimeUnit.SECONDS);
If this is not working use Explicit wait:
WebDriverWait wait=new WebDriverWait(driver, 20);
WebElement input_userName;
input_userName = wait.until(ExpectedConditions.elementToBeClickable(By.tagName("input")));
input_userName.sendkeys("suryap");
You can use ExpectedCondition.visibilityOfElementLocated() as well.
You can increase the time, for example,
WebDriverWait wait=new WebDriverWait(driver, 90);

A solution to this for Javascript looks like this. You will have to modify the time to suit your need.
driver.manage().setTimeouts({ implicit: 30000 });
Hope this is helpful to someone.
see the docs for reference

Actually the Exception is Element Not Visible
The best practice is to use Implicit wait below driver Instantiation so it get sufficient time to find element throughout the exception
driver.get("http://www.testsite.com");
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
Still facing issue as some element require more time. Use ExplicitWait for individual element to satisfy certain condition
In your case you are facing element not visible exception then use wait condition in following way-
WebDriverWait wait = new WebDriverWait(driver, 120);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.your_Elemetnt));

In my case issue was because there is some animation, and that element is not visible for some duration. And hence exception was occurring.
For some reason I couldn't make ExpectedConditions.visibilityOfElementLocated work, so I created a code to wait for some hardcoded seconds before proceeding.
from selenium.webdriver.support.ui import WebDriverWait
def explicit_wait_predicate(abc):
return False
def explicit_wait(browser, timeout):
try:
Error = WebDriverWait(browser, timeout).until(explicit_wait_predicate)
except Exception as e:
None
And now I am calling this explicit_wait function wherever I want to wait for sometime e.g.
from selenium import webdriver
from selenium.webdriver.common.by import By
browser = webdriver.Safari()
browser.get('http://localhost')
explicit_wait(browser,5) # This will wait for 5 secs
elem_un = browser.find_element(By.ID, 'userName')

Related

Why do I get the error "Element could not be scrolled into view" unless I manually scroll the browser instance?

I wrote a python script that goes to a site, and interacts with some dropdowns. It works perfectly fine if after I run the script, quickly make the browser instance full screen so that the elements are in view. If I don't do that, I get the error "Element could not be scrolled into view".
Here is my script:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://example.com")
driver.implicitly_wait(5)
yearbtn = driver.find_element("id", "dropdown_year")
yearbtn.click()
year = driver.find_element("css selector", '#dropdown_ul_year li:nth-child(5)')
year.click()
makebtn = driver.find_element("id", "dropdown_make")
makebtn.click()
make = driver.find_element("css selector", '#dropdown_ul_make li:nth-child(2)')
make.click()
modelbtn = driver.find_element("id", "dropdown_model")
modelbtn.click()
model = driver.find_element("css selector", '#dropdown_ul_model li:nth-child(2)')
model.click()
trimbtn = driver.find_element("id", "dropdown_trim")
trimbtn.click()
trim = driver.find_element("css selector", '#dropdown_ul_trim li:nth-child(2)')
trim.click()
vehicle = driver.find_element("css selector", '#vehiclecontainer > div > p')
vdata = driver.find_element("css selector", '.top-sect .tow-row:nth-child(2)')
print("--------------")
print("Your Vehicle: " + vehicle.text)
print("Vehicle Data: " + vdata.text)
print("--------------")
print("")
driver.close()
Like I said, it works fine if I make the browser full-screen (or manually scroll) so that the elements in question are in view. It finds the element, so what's the issue here? I've tried both Firefox and Chrome.
Generally firefox opens in maximized mode. Incase for specific version of os it doesn't, the best practice is to open it in maximized mode as follows:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("start-maximized")
driver = webdriver.Firefox(options=options)
However in somecases the desired element may not get rendered within the due coarse, so you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:
vehicle = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#vehiclecontainer > div > p")))
vdata = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".top-sect .tow-row:nth-child(2)")))
print("--------------")
print("Your Vehicle: " + vehicle.text)
print("Vehicle Data: " + vdata.text)
print("--------------")
print("")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python
I'm not sure what all is happening on the page in question. I use a generic method to take care of common actions like click(), find(), etc. that contain a wait, scroll, and whatever else is useful for that action. I've put an example below. It waits for the element to be clickable (always a best practice), scrolls to the element, and then clicks it. You may want to tweak it to your specific use but it will get you pointed in the right direction.
def click(self, locator):
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable(locator))
driver.execute_script("arguments[0].scrollIntoView();", element)
element.click()
I would suggest you try this and then convert all of your clicks to use this method and see if that helps, e.g.
click((By.ID, "dropdown_year"))
or
locator = (By.ID, "dropdown_year")
click(locator)

How to handle Selenium when is_displayed attribute throws ElementNotFoundException error when page is refreshed

I am trying to wait for the loading spinner to finish loading and then perform some operation
My Code looks like
try:
while driver.find_element(By.CSS_SELECTOR, "div.content-container div.content-element-container > div.loading-container").is_displayed() == True:
time.sleep(10)
driver.refresh()
if count > 3:
print("Could not fetch content")
exit(1)
count += 1
except Exception as e:
print(str(e))
the reason for driver refresh is because of signal-r. For some reason the i do not get the data and the load spinner keeps on loading and then prints "Could Not fetch content").
If I refresh the page then I get NoSuchElementException. What would be the best way to handle such a scenario?
This locator strategy:
(By.CSS_SELECTOR, "div.content-container div.content-element-container > div.loading-container")
represents the loading spinner / loader element. So on every refresh() the loader element will resurface again and again.
Solution
An elegant approach to get past the loading spinner will be to induce WebDriverWait for the invisibility_of_element_located() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 30).until(EC.invisibility_of_element_located((By.CSS_SELECTOR, "div.content-container div.content-element-container > div.loading-container")))
Using XPATH:
WebDriverWait(driver, 30).until(EC.invisibility_of_element_located((By.XPATH, "//div[#class='content-container']//div[#class='content-element-container']/div[#class='loading-container']")))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Scrape Product Image with BeautifulSoup (Error)

I need your help. I'm working on a telegram bot which sends me all the sales from amazon.
It works well but this function doesn't work properly. I have always the same error that, however, blocks the script
imgs_str = img_div.img.get('data-a-dynamic-image') # a string in Json format
AttributeError: 'NoneType' object has no attribute 'img'
def take_image(soup):
img_div = soup.find(id="imgTagWrapperId")
imgs_str = img_div.img.get('data-a-dynamic-image') # a string in Json format
# convert to a dictionary
imgs_dict = json.loads(imgs_str)
#each key in the dictionary is a link of an image, and the value shows the size (print all the dictionay to inspect)
num_element = 0
first_link = list(imgs_dict.keys())[num_element]
return first_link
I still don't understand how to solve this issue.
Thanks for All!
From the looks of the error, soup.find didn't work.
Have you tried using images = soup.findAll("img",{"id":"imgTagWrapperId"})
This will return a list
Images are not inserted in HTML Page they are linked to it so you need wait until uploaded. Here i will give you two options;
1-) (not recommend cause there may be a margin of error) simply; you can wait until the image is loaded(for this you can use "time.sleep()"
2-)(recommend) I would rather use Selenium Web Driver. You also have to wait when you use selenium, but the good thing is that selenium has a unique function for this job.
I will show how make it with selenium;
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
browser = webdriver.Chrome()
browser.get("url")
delay = 3 # seconds
try:
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'imgTagWrapperId')))# I used what do you want find
print ("Page is ready!")
except TimeoutException:
print ("Loading took too much time!")
More Documention
Code example for way 1
Q/A for way 2

How to get all comments in youtube with selenium?

The webpage shows that there are 702 Comments.
target youtube sample
I write a function get_total_youtube_comments(url) ,many codes copied from the project on github.
project on github
def get_total_youtube_comments(url):
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--headless")
driver = webdriver.Chrome(options=options,executable_path='/usr/bin/chromedriver')
wait = WebDriverWait(driver,60)
driver.get(url)
SCROLL_PAUSE_TIME = 2
CYCLES = 7
html = driver.find_element_by_tag_name('html')
html.send_keys(Keys.PAGE_DOWN)
html.send_keys(Keys.PAGE_DOWN)
time.sleep(SCROLL_PAUSE_TIME * 3)
for i in range(CYCLES):
html.send_keys(Keys.END)
time.sleep(SCROLL_PAUSE_TIME)
comment_elems = driver.find_elements_by_xpath('//*[#id="content-text"]')
all_comments = [elem.text for elem in comment_elems]
return all_comments
Try to parse all comments on a sample webpage https://www.youtube.com/watch?v=N0lxfilGfak.
url='https://www.youtube.com/watch?v=N0lxfilGfak'
list = get_total_youtube_comments(url)
It can get some comments ,only small party of all comments.
len(list)
60
60 is much less than 702,how to get all comments in youtube with selenium?
#supputuri,i can extract all comments with your code.
comments_list = driver.find_elements_by_xpath("//*[#id='content-text']")
len(comments_list)
709
print(driver.find_element_by_xpath("//h2[#id='count']").text)
717 Comments
comments_list[-1].text
'mistake at 23:11 \nin NOT it should return false if x is true.'
comments_list[0].text
'Got a question on the topic? Please share it in the comment section below and our experts will answer it for you. For Edureka Python Course curriculum, Visit our Website: Use code "YOUTUBE20" to get Flat 20% off on this training.'
Why the comments number is 709 instead of 717 shown in page?
You are getting a limited number of comments as YouTube will load the comments as you keep scrolling down. There are around 394 comments left on that video you have to first make sure all the comments are loaded and then also expand all View Replies so that you will reach the max comments count.
Note: I was able to get 700 comments using the below lines of code.
# get the last comment
lastEle = driver.find_element_by_xpath("(//*[#id='content-text'])[last()]")
# scroll to the last comment currently loaded
lastEle.location_once_scrolled_into_view
# wait until the comments loading is done
WebDriverWait(driver,30).until(EC.invisibility_of_element((By.CSS_SELECTOR,"div.active.style-scope.paper-spinner")))
# load all comments
while lastEle != driver.find_element_by_xpath("(//*[#id='content-text'])[last()]"):
lastEle = driver.find_element_by_xpath("(//*[#id='content-text'])[last()]")
driver.find_element_by_xpath("(//*[#id='content-text'])[last()]").location_once_scrolled_into_view
time.sleep(2)
WebDriverWait(driver,30).until(EC.invisibility_of_element((By.CSS_SELECTOR,"div.active.style-scope.paper-spinner")))
# open all replies
for reply in driver.find_elements_by_xpath("//*[#id='replies']//paper-button[#class='style-scope ytd-button-renderer'][contains(.,'View')]"):
reply.location_once_scrolled_into_view
driver.execute_script("arguments[0].click()",reply)
time.sleep(5)
WebDriverWait(driver, 30).until(
EC.invisibility_of_element((By.CSS_SELECTOR, "div.active.style-scope.paper-spinner")))
# print the total number of comments
print(len(driver.find_elements_by_xpath("//*[#id='content-text']")))
There are a couple of things:
The WebElements within the website https://www.youtube.com/ are dynamic. So are the comments dynamically rendered.
With in the webpage https://www.youtube.com/watch?v=N0lxfilGfak the comments doesn't render unless user scrolls the following element within the Viewport.
The comments are with in:
<!--css-build:shady-->
Which applies, Polymer CSS Builder is used apply Polymer's CSS Mixin shim and ShadyDOM scoping. So some runtime work is still done to convert CSS selectors under the default settings.
Considering the above mentioned factors here's a solution to retrieve all the comments:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, NoSuchElementException, ElementClickInterceptedException, WebDriverException
import time
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.youtube.com/watch?v=N0lxfilGfak')
driver.execute_script("return scrollBy(0, 400);")
subscribe = WebDriverWait(driver, 60).until(EC.visibility_of_element_located((By.XPATH, "//yt-formatted-string[text()='Subscribe']")))
driver.execute_script("arguments[0].scrollIntoView(true);",subscribe)
comments = []
my_length = len(WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//yt-formatted-string[#class='style-scope ytd-comment-renderer' and #id='content-text'][#slot='content']"))))
while True:
try:
driver.execute_script("window.scrollBy(0,800)")
time.sleep(5)
comments.append([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//yt-formatted-string[#class='style-scope ytd-comment-renderer' and #id='content-text'][#slot='content']")))])
except TimeoutException:
driver.quit()
break
print(comment)
If you don't have to use Selenium I would recommend you to look at the google/youtube api.
https://developers.google.com/youtube/v3/getting-started
Example :
https://www.googleapis.com/youtube/v3/commentThreads?key=YourAPIKey&textFormat=plainText&part=snippet&videoId=N0lxfilGfak&maxResults=100
This would give you the first 100 results and gets you a token that you can append on the next request to get the next 100 results.
I'm not familiar with python, but I'll tell you the steps that I would do to get all comments.
First of all, if your code I think the main issue is with the
CYCLES = 7
According to this, you will be scrolling for 2 seconds 7 times. Since you are successfully grabbing 60 comments, fixing the above condition will solve your issue.
I assume you don't have any issue in finding elements on a website using locators.
You need to get the total comments to count to a variable as an int. (in your case, let's say it's COMMENTS = 715)
Define another variable called VISIBLECOUNTS = 0
The use a while loop to scroll if the COMMENTS > VISIBLECOUNTS
The code might look like this ( really sorry if there are syntax issues )
// python - selenium command to get all comments counts.
COMMENTS = 715
(715 is just a sample value, it will change upon the total comments count)
VISIBLECOUNTE = 0
SCROLL_PAUSE_TIME = 2
while VISIBLECOUNTS < COMMENTS :
html.send_keys(Keys.END)
time.sleep(SCROLL_PAUSE_TIME)
VISIBLECOUNTS = len(driver.find_elements_by_xpath('//ytm-comment-thread-renderer'))
With this, you will be scrolling down until the COMMENTS = VISIBLECOUNTS. Then you can grab all the comments as all of them share the same element attributes such as ytm-comment-thread-renderer
Since I'm not familiar with python I'll add the command to get the comments to count from js. you can try this on your browser and convert it into your python command
Run the bellow queries in your console and check.
To get total comments count
var comments = document.querySelector(".comment-section-header-text").innerText.split(" ")
//We can get the text value "Comments • 715" and split by spaces and get the last value
Number(comments[comments.length -1])
//Then convirt string "715" to int, you just need to do these in python - selenium
To get active comments count
$x("//ytm-comment-thread-renderer").length
Note: if it's hard to extract the values you still can use the selenium js executor and do the scrolling with js until all the comments are visible. But I guess it's not hard to do it in python since the logic is the same.
I'm really sorry about not being able to add the solution in python.
But hope this helped.
cheers.
The first thing you need to do is scroll down the video page to load all comments:
$actualHeight = 0;
$nextHeight = 0;
while (true) {
try {
$nextHeight += 10;
$actualHeight = $this->driver->executeScript('return document.documentElement.scrollHeight;');
if ($nextHeight >= ($actualHeight - 50 ) ) break;
$this->driver->executeScript("window.scrollTo(0, $nextHeight);");
$this->driver->manage()->timeouts()->implicitlyWait = 10;
} catch (Exception $e) {
break;
}
}

How do i use “wait until” element before certain page is displayed? (using python 3.7)

First of all, i am a newbie in testing app using appium (python 3.7). Here, i am testing an app where i have to wait right after login process is completed. I have done this using implicit wait. But now, to make the testing process more dynamic i want to wait until the next page is displayed.
Note: I have seen and tried several issues of this forum but could not help myself.
Here's the code:
from appium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
desired_cap = {
"platformName": "Android",
"deviceName": "QDG9X18426W11577",
"newCommandTimeout": "240",
"app": "C:\\Users\\tahmina\\Downloads\\test-v3.10.apk",
"appPackage": "com.cloudapper.android",
"appActivity": "com.cloudapper.android.SplashActivity"
}
#Making connection with Appium server
driver = webdriver.Remote("http://localhost:4723/wd/hub", desired_cap)
#Here i have used implicit wait to load the login page
driver.implicitly_wait(20)
#Login to the app
search_element = driver.find_element_by_id('Username').send_keys('test#yandex.com')
search_element = driver.find_element_by_id('Password').send_keys('1155qQQ')
search_element = driver.find_element_by_xpath(
'/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout[2]/android.webkit.WebView/android.webkit.WebView/android.view.View/android.view.View[2]/android.widget.Button').click()
wait = WebDriverWait(driver, 10)
#Waiting until the next process comes up
if webdriver.wait.until(driver.find_element_by_id('com.cloudapper.android:id/item_bg').is_displayed()):
print('Run the next process')
elif webdriver.wait.until(not driver.find_element_by_id('com.cloudapper.android:id/item_bg')):
print('Something went wrong!')
#Searching employee by using ID
search_element = driver.find_element_by_id('com.cloudapper.android:id/edtSearch').send_keys('1018')
driver.execute_script('mobile:performEditorAction', {'action': 'search'})
Guide me if anyone of you have any solution about it.
With Python, you can use WebDriverWait and ExpectedConditions to solve your problem.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#Login to the app
search_element = driver.find_element_by_id('Username').send_keys('test#yandex.com')
search_element = driver.find_element_by_id('Password').send_keys('1155qQQ')
search_element = driver.find_element_by_xpath(
'/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout[2]/android.webkit.WebView/android.webkit.WebView/android.view.View/android.view.View[2]/android.widget.Button').click()
# Waiting until the next process comes up
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "com.cloudapper.android:id/item_bg")))
If you want to implement the WebDriverWait in a try / except block, you can handle the case where your desired element does not appear on the page:
from selenium.common.exceptions import TimeoutException
# Waiting until the next process comes up
try:
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "com.cloudapper.android:id/item_bg")))
except TimeoutException as ex:
# handle the exception here
print("Exception has been thrown. " + str(ex))
Because you are using the explicit wait.until keyword, you should not set driver Implicit wait. It is bad practice to set both implicit and explicit wait in your automation tests, and can yield unexpected wait times.
On another note -- I noticed you are using explicit XPath notation in some of your selectors. XPath is a great method for selecting elements, but using explicit selectors as such makes your code very brittle. I recommend using relative selectors instead. You can replace this:
search_element = driver.find_element_by_xpath(
'/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout[2]/android.webkit.WebView/android.webkit.WebView/android.view.View/android.view.View[2]/android.widget.Button').click()
with this:
search_element = driver.find_element_by_xpath('//android.widget.Button').click()
You may have to query on additional attributes such as text --
search_element = driver.find_element_by_xpath('//android.widget.Button[#text='Log in']').click()
Overall, this method is much more efficient.
Have you tried something like this:
import time
time.sleep(0.5)
This is for anyone who face the same problem:
Import WebDriverWait and ExpectedConditions
Use this as explicit wait in your code
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "com.cloudapper.android:id/item_bg")))

Resources