Scrolling down with Python Selenium on part of the web page - python-3.x

So I have a web page where a part of the page is a user list. A user can scroll down this list, but scrolling only works if the mouse pointer is located above this list. Keys like page down or end do not work.
I can locate the element using Selenium and Xpath without any issue, but I don't know how to perform the scrolling in selenium as it only works when the focus is on that specific web element.
I've tried sending keys like page down and end or code like
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
Anybody got any ideas?

Because you mentioned you can locate the element, you could try scrolling to the specific element, instead of using document.body.scrollHeight as your scroll target.
element = driver.find_element(someLocatorHere)
driver.execute_script("arguments[0].scrollIntoView(true);", element)
Hope this helps a bit.

Scrolling can be achieved in two different ways.
Javascript
elementId = driver.find_element_by_id("elementId")
driver.execute_script("arguments[0].scrollIntoView();", elementId)
Action class
actions = ActionChains(driver)
actions.move_to_element(elementId).perform()

Related

Selenium/ python performing a click with browser.execute_script VS. normal click line

I wanted to click on something in a webpage so I used
WebDriverWait(browser, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#styleguide-v2 > div.banner-container > a:nth-child(2)")))
except that it doesn't work in the background.I have to switch to the browser manually where it to be seen on my screen so that the code works properly.
Then I added this
x = WebDriverWait(browser, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#styleguide-v2 > div.banner-container > a:nth-child(2)")))
browser.execute_script("arguments[0].click();", x)
now it works like charm, my question is what's the difference? I want to know what's happening behind this
the webpage https://www.imdb.com/title/tt0198781/
presence_of_element_located expected condition finishes and the program continues to the next call while the element already created but still not clickable and still not located on it's final position on the page and still not ready to accept regular click.
JavaScript click can handle this kind of click, however this doesn't really imitates real UI user action.
To mimic real user action you should use element_to_be_clickable expected condition and click the element only when it became clickable.
visibility_of_element_located didn't work because the element is not actually visible itself, so we had to use element_to_be_clickable expected condition.
It is also possible that element is covered by some other element during the page rendering when it is literally become clickable but the page is still rendered. In this case we have to add some hardcoded delay or to wait until the element covering the desired button is disappeared. this can be achieved by invisibility_of_element_located expected condition for the covering element.

How to close popover/popup by <a> element using Selenium

I'm working on a webscraper for https://www.grailed.com/designers/jordan-brand/hi-top-sneakers. When the page has opened a popup for login comes up. Searching through the web design I can locate the X element to close the browser like so: temp = WebDriverWait(driver, 10).until(ec.visibility_of_element_located((By.CLASS_NAME, 'close'))). If I go more into this there is an and element. I have tried using .click() on the element (with class 'close'), as well as the SVG and path elements. None of these close the box, and there is no button or other element of this kind for the X. What can I do to close this popover? I'm not sure if I need to find a button-ish element to click, but I can't find one like that. I've looked at a couple of questions and articles (https://stackoverflow.com/questions/61923909/trying-to-close-popover-python-selenium-glassdoor, https://sqa.stackexchange.com/questions/5310/how-to-close-pop-up-window-in-selenium-webdriver, https://saucelabs.com/resources/articles/the-selenium-click-command) but can't find a solution.
You can do double click with actions for resolving this problem
WebDriverWait(driver, 20).until(ec.visibility_of_element_located((By.XPATH,
'//a[#class = 'close']/*[name()='svg']')))
close = driver.find_element_by_xpath("//a[#class = 'close']/*[name()='svg']")
actionChains = ActionChains(driver)
actionChains.double_click(close).perform()
And the Java code for this:
new WebDriverWait(driver, 20)
.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//a[#class = 'close']/*[name()='svg']")));
WebElement close = driver.findElement(By.xpath("//a[#class = 'close']/*[name()='svg']"));
Actions action = new Actions(driver);
action.doubleClick(close).build().perform();
I actually just ran into the same issue myself. The above solution didn't work for me, so I'll post what I did here:
Essentially, I clicked by the coordinates instead of by trying to find the button element.
Note: some webpages don't display the popup until you try to click something else, so I had to first attempt to click another element. If you need to do that, make sure to wait a second or so for the popup to load.
Then, you can do the rest via ActionChain:
elem = driver.find_element_by_class_name("CLASSNAME")
ac = ActionChains(driver)
ac.move_to_element(elem).click().perform()
You'll want to encase this in a try-except block for extra safety.
Credit to Dirk Bergstrom for providing part of the solution here: Clicking at coordinates without identifying element

Selenium Scroll down on Newly Opened Body

I am working with Selenium python, and would like to scroll down on a page after clicking an element. Clicking the element gives a pop-up menu, and then when I try some scrolling methods they scroll on the original page.
I have selected the body of the new element that has appeared and tried this:
from selenium import webdriver
from selenium import Keys
driver = webdriver.get(url)
button = driver.find_element_by_xpath(pathto)
button.click()
body = driver.find_element_by_css('body')
body.send_keys(Keys.PAGE_DOWN)
This does not actually do anything as is. However, if I first manually click on the new popup element before doing body.send_keys(Keys.PAGE_DOWN), it does scroll.
One way I have resolved this is using ActionChains to right click on the body before scrolling down. Directly using .click() on the body does not work as it clicks a link in the body.
Are there any better ways to do this? In particular, I am somewhat afraid that the right click menu that opens up when I use my current method may mess with the rest of the scraping. Is there at least a method to guarantee that it does not interfere? Another solution could be somehow "selecting" or clicking on the body element within selenium, so are there any suggestions for this?
You could try using END key instead of DOWN key. It's worth a shot:
from selenium.webdriver.common.keys import Keys
driver = webdriver.get(url)
button = driver.find_element_by_xpath(pathto)
button.click()
driver.find_element_by_tag_name('body').send_keys(Keys.END)
You also may need to click the pop-up in order to focus it first:
from selenium.webdriver.common.keys import Keys
driver = webdriver.get(url)
button = driver.find_element_by_xpath(pathto)
button.click()
# click popup menu to get it into focus
popup_menu = driver.find_element_by_xpath(pathToPopupMenu)
popup_menu.click()
driver.find_element_by_tag_name('body').send_keys(Keys.END)

not able to click on a button for scraping using selenium and python

THIS IS THE BUTTON CODE WHICH I WANTED TO CLICK
I used this code.
login_btn1 = driver.find_elements_by_tag_name('button')
login_btn1.submit()
but they are showing the error:
AttributeError: 'list' object has no attribute 'submit'
can someone help me with this please.Where am i going wrong?
By using find_elements_by_tag_name you are looking for all 'button' elements which will return a list.
You need to replace this with find_element_by_tag_name to retrieve a single 'button' element. (this will be the first button element found)
Alternatively, in the likely situation that there is more than one button on the page, you can select by class name using find_element_by_class_name('cdp-view-all-button') or another method from the selenium docs.
When you write this :
login_btn1 = driver.find_elements_by_tag_name('button')
Now login_btn1 is a list of Web Element. you can have method which are applicable for list.
Since you want to click on button and button is a web element. you can't do login_btn1.click() .
If you want to click on a single button, first you have to find the locator then there are multiple ways to click on it such as :
button = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#data-track-app='discovery']")))
button.click()
If the provided xpath is Unique in DOM then the above code would work.

Element not recognized

Working on Coded UI testing and for scripts developed using Record Capture and playback feature (ctrl +I).
The problem is when the page has sub-menus (e.g. I need to hover over menu link then click sub-menu). When I record and capture element using Ctrl+I and executed a script it recognizes, but when I ran the script for the second time the element gets changed and it's not recognized.
I have tried simple x path utility posted here but coudn't able to use this feature. What would be the problem for always element id's getting changed. How to resolve it ?
Are you sure it isn't a nested object?
See http://executeautomation.com/blog/how-to-identify-all-the-child-elements-within-a-page-in-coded-ui-test-cuit/
You could also try EnsureClickable()
There could few reasons behind not recognizing an element:
List item Element is not visible when you are trying to click on it.
If Type of Parent Element is e.g. winclient then it’s difficult in coded UI to identify its child elements.
There could various solutions, you can try:
First Click on Menu Item and then click on Sub Menu Item, if you are directly clicking on sub menu item in your recorded script, this will make sub menu element visible.
Also you can check the visibility from Coded UI Test Builder-> Add Assertion button then going to UI Control Map, then select the element in tree and click on Refresh. It will show if element is visible or not.
If Ids are changing, then you can various other properties like Name, ClassName, InnerText, ControlType, TagInstance, ControlName etc. whichever is supported by Element.

Resources