I'm using the find_element_by_partial_link_text selector to find the "next" button, so that I can click it and continue crawling.
However, the issue I'm having is that sometimes the word "next" is also in other random links on the page, which breaks the script.
I've tried addressing this with the code below, which obviously doesn't work.
pagination_ul = ff.find_element_by_xpath('//*[contains(#id, "pagination-both-")]')
next_button = pagination_ul.find_element_by_partial_link_text('Next')
So my question is...
How can I retrieve and compare the parent element of a find_element_by_partial_link_text element?
In other words, how can I make sure that the parent of the find_element_by_partial_link_text is, say, a li element?
Try using below XPath:
driver.find_element_by_xpath('//div[starts-with(#id, "pagination-") and not(contains(#class, "hidden"))]//li[#class="a-last"]/a[text()="Next"]').click()
It looks more complicated then search by text content, but should work in all cases
Note that you might need to Wait for element to appear in DOM/to be clickable
If you want to find parent element of an element, then use the below approach:
For instance, let us take https://www.google.com webpage. The xpath for the mike icon is //div[#id='gs_st0']. To find the parent of that element, which is the search box, use 'child' axes as below.
//*[child::div[#id='gs_st0']]
The above xpath will fetch you the parent element of the child.
In your case, if your child element's xpath is: //a[text()='Next']
So to find the parent element of the 'Next' link, use below xpath:
parentElement = driver.find_element_by_xpath("//*[child::a[text()='Next']]")
Related
I could scroll down or up a whole web page but I am having trouble in scrolling a particular div element within the webpage
The thing is when I open a webpage like whatsapp web and within a particular chat containing the messages(which is particular div element),I want to scrape all the messages from the beginning of the chat ,but I could only scrape the messages which is in the view (the last few messages),So I want to scroll that particular div element to the top of the chat to scrape all the messages.
Can someone help me with this thing in PYTHON.
Thank you.
Yes, there are multiple ways using which you could scroll to particular element
By using the move_to_element()
element = driver.find_element_by_xpath("_element_")
action = ActionChains(driver)
action.move_to_element(element).perform()
By scrollIntoView()
element = driver.find_element_by_xpath("_element_")
driver.execute_script("arguments[0].scrollIntoView(true);", element)
For reference check here
It is possible.
Let's say the element you need to scroll inside is div_el, it's locator is xpath_locator the code will be like this:
div_locator = "xpath_locator"
div_el = driver.find_element_by_xpath(div_locator)
driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", div_el)
See more here
I am using headless Firefox on Selenium and XPath Helper to identify insanely long paths to elements.
When the page initially loads, I can use XPath Helper to find the xpath of any element of interest, and selenium can find the element when given the xpath.
However, several buttons that I need to interact with on the page open menus when pressed that are either small or take up the whole "screen". No matter their size, these containers are overlaid on the original page, and although I can find their xpaths using XPath Helper, when I try to use those xpaths to find the elements using selenium, they can't be found.
I've checked, and there's no iframe funny business happening. I'm a bit stumped as to what could be happening. My guess is that the page's source code is being dynamically changed after I press the buttons that open the menu containers and when I call find_element_by_xpath on new elements in the containers, the original source is being searched, instead of the new source. Could that be it?
Any other ideas?
As a workaround, I can get around this issue by sending keystrokes to the body of the page, but I feel this solution is rather brittle and likely to fail. Would be a much more robust solution to actually specify all elements.
EDIT:
With selenium I can find the export button, but not the menu it opens.
Here is the code for the export button itself:
The element of interest for me is "Customize Export" which I have not been able to find using selenium. Here is the code for this element:
Notice the very top line of this last image (cdk-overlay-container)
Now, when I refresh the page and do NOT click the export button, the cdk-overlay-container section of the code is empty:
This suggests my that my hypothesis is correct -- that when the page loads initially, the "Customize Export" button is nowhere in the source code, but appears only after "Export" is clicked, and that selenium is using the original source code only --not the dynamically generated code that appears after clicking "Export" -- to find elements
Selenium could find the dynamic content after doing
driver.execute_script("return document.body.innerHTML")
The WebDriverWait is what you need to use to wait for a certain condition of elements. Here is an example of waiting for the elements to be clickable before the click with a timeout in 5 seconds:
wait = WebDriverWait(driver, 5)
button = wait.until(EC.element_to_be_clickable((By.XPATH, 'button xpath')))
button.click()
wait.until(EC.element_to_be_clickable((By.XPATH, 'menu xpath'))).click()
identify insanely long paths
is an anti pattern. You can try to not use XPath Helper and find xpath or selector yourself.
Update:
wait = WebDriverWait(driver, 10)
export_buttons = wait.until(EC. presence_of_all_elements_located((By.XPATH, '//button[contains(#class, "mat-menu-trigger") and contains(.="Export")]')))
print("Export button count: ", len(export_buttons))
export_button = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[contains(#class, "mat-menu-trigger") and contains(.="Export")]')))
export_button.click()
cus_export_buttons = wait.until(EC. presence_of_all_elements_located((By.XPATH, '//button[contains(#class, "mat-menu-item") and contains(.="Customize Export")]')))
print("Customize Export button count: ", len(cus_export_buttons))
I am a little bit new to programming but python really made get into it. I am trying to create a programm that automatically checks for updates in a website. I've successfully implemented the neccessary code to call the page of enrollment but yet there is one element that cannot be located. Since I have to do it for multiple courses and iterate throught them there is no specific id, I've tried to find it by title but also this didn't work.
Is there a way you can locate the button with the title "enroll".
I've tried
driver.find_element_by_xpath("//a\[#title ='enroll']").click()
but this didn't work and I always get
NoSuchElement
error.
The XPATH for the button is simply: //*\[#id="id572"\]
Here is the part of the HTML code:
From the screenshot of HTML code you provided, the element is <button>, not <a>.
Try this xpath expression //button[#title='enroll']
This should do it if it's not in any iframes. Just grab the button whose title is enroll and click. Your css selector was an a tag and it might get id dynamically.
driver.find_element_by_css_selector("button[title ='enroll']").click()
I'm trying to run the code:
for j in range(1,13):
driver.find_element_by_xpath('//*[#id="gateway-page"]/body/table/tbody/tr[3]/td[2]/table/tbody/tr[2]/td[2]/table/tbody/tr/td/table/tbody/tr[2]/td/div/div[2]/ul/li['+str(j)+']').click()
time.sleep(3)
To click every satisfying element on this website. But it ignores some elements every time, while it worked when I tried them not in the for loop but separately. Any idea why this happened?
Seems problem is with /ul/li['+str(j)+'] you are performing the click on <li> tag while actual link reside in it. That's why sometime the actual link won't receive the click without any error as link wrapped inside <li> tag .
Try to locate actual link tag. Use below code. I have tested on my system. Hope this will help you.
driver.get('http://catalog.sps.cuny.edu/preview_program.php?catoid=2&poid=607')
driver.implicitly_wait(10)
links = driver.find_elements_by_xpath("//div//h2[contains(.,'Electives')]/..//ul/li//span/a")
for link in links:
link.click()
time.sleep(3)
After observing xpath, I observed that you are trying to click the Elective option on that website. I think you have stored text of all electives in str array and using the loop, you are trying to click on each elective.
I suggest using another approach. Store all electives in list and then iterate over the elements and click them. e.g.
elements = driver.find_elements_by_xpath('///*[#id="gateway-page"]/body/table/tbody/tr[3]/td[2]/table/tbody/tr[2]/td[2]/table/tbody/tr/td/table/tbody/tr[2]/td/div/div[2]/ul/li')
for element in elements:
element.click()
time.sleep()
Probable problems in your solution
You are storing the name of electives in the array. If there is any typo, xPath will become invalid
You are starting loop from 1 to 13 but str is 0 indexed so start the loop from 0. because in you case you will always miss the first elective
Also after each click, elective expands. So you can also think about scrolling if an element is not found
Suggestion:
Also, use relative xpaths instead of absolute. Relative xpaths are more stable.
Happy Coding~
I'm trying to locate elements within the iframe but got no luck.
I think because of my code cannot switch to the iframe and that's why I couldn't locate the elements within it.
Below is my code.
I have tried
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, '/html/body/iframe')))
driver.find_element_by_name('applicant.name').send_keys('email')
I also tried this
iframe = driver.find_element_by_xpath('/html/body/iframe')
driver.switch_to.frame(iframe)
driver.implicitly_wait(10)
driver.find_element_by_name('applicant.name').send_keys('email')
I tried look for xpath, id of the applicant.name but notthing work so far. Any help or suggestions would be appreciated.
Here the site:
https://www.indeed.com/cmp/Paratus-Partners-LLC/jobs/Full-Stack-Developer-7814e52be25090f3?from=iaBackPress&q=software%20developer&vjs=3
There are at least 2 iframes in the page you're trying to automate:
Therefore you need to
Amend your XPath Expression to uniquely identify first iframe like:
parentIframe = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH,"//iframe[contains(#id,'modal-iframe')]")))
Switch to first iframe
driver.switch_to.frame(parentIframe)
Locate second iframe and switch to it
childIframe = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH,"//iframe[contains(#src,'resumeapply')]")))
driver.switch_to.frame(childIframe)
Once done you should be able to send text into the input fields.
There are some elements with the xpath /html/body/iframe
I was looking for the applicant.name and it was only to create after click on the postulate button.
I think that you have to check the xpath for the iframe, maybe something like this can work
"//iframe[contains(#src,'resumeapply')]"