I am trying to scrape job details in LinkedIn. As per now, I am able to scroll till the end of list.
code:
#scrolling till buttom of page(online job section)
target = driver.find_element_by_xpath('//div[#class="jobs-search-results jobs-search-results--is-two-pane"]')
time.sleep(1)
driver.execute_script('arguments[0].scrollTop = arguments[0].scrollHeight', target)
driver.implicitly_wait(1)
time.sleep(1)
now I want to scroll up to the top of the div.
I want to scroll up in div shown as red rectangle:
Scrolling to top of the page in Python using Selenium
Scroll Element into View with Selenium
There is 2 main way to do that:
Firstly, you can locate the element(at the top of your DOM view) in the DOM and then scrolling up until you find the element
element = driver.find_element_by_xpath("element_xpath")
driver.execute_script("return arguments[0].scrollIntoView(true);", element)
Secondly, you can easily use CTRL+HOME for scrolling to the top of the page with the help of Keys
driver.find_element_by_tag_name('body').send_keys(Keys.CONTROL + Keys.HOME)
But in your situation which dealing with JavaScript DOM, recommended way is the first one.
There are many ways to do this, here is one way:
driver.execute_script("window.scrollTo(0, 0);")
Alternatively you can scroll up with the mouse:
import pyautogui
pyautogui.scroll(10)
Related
I am getting a list of links that I want to click. I get the href of each of them and I am able to locate the element using it, but when I try to click it, it just appears as if I were hovering on top of it.
The list is called links and the href that I am trying to access is links[1]
I have alredy tried normally clicking it:
self.driver.find_element(By.XPATH,'//*[contains(#href,links[1])]').click()
And double clicking it:
actionChains = ActionChains(self.driver)
a=self.driver.find_element(By.XPATH,'//*[contains(#href,links[1])]')
actionChains.double_click(a).perform()
But the behaviour is as if I only put the cursor on top of the link without clicking it.
just fix the xpath
driver.find_element(By.XPATH, "//*[contains(#href,'{}')]".format(links[1])).click()
or if you prefer ActionChains
actionChains = ActionChains(self.driver)
a=self.driver.find_element(By.XPATH,"//*[contains(#href,'{}')]".format(links[1]))
actionChains.double_click(a).perform()
I have a website I'm scraping and need to scroll horizontal (to the far right) and vertical (to the very bottom) of a table that is on a webpage. I am using Python3, Selenium and Chrome Driver.
I can scrape the table data but it only scrapes as far down as what's visible on my webpage. When I inspect the webpage I cannot find any scrolling information.
Researching online I've seen various ways to scroll using Selenium even finding by xpath and then scrolling that way. I have not had luck with any option thus far but could be doing it wrong.
This is an example of something I've tried but it does not work:
html=browser.find_element_by_xpath('/html/body/form/div[3]/div[2]/div/div[2]/div/div[2]')
html.send_keys(Keys.END)
This scrapes the rows without issue from the table (but only the ones visible on the webpage when the page is loaded. If I scroll all the way down, and rerun this code, it scrapes the contents of the entire table since it fully loads)
for tr in browser.find_elements_by_xpath('/html/body/form/div[3]/div[2]/div/div[2]/div/div[2]'):
tds = tr.find_elements_by_tag_name('td')
action_log = ([tr.text for tr in tds])
I just want to be able to scroll vertically all the way to the bottom of the table and horizontally for as far as I can (just once after the page loads). Again, scrolling inside the table on the webpage - not in Chrome itself. As a note my table data size will change.
I kept combing the site to figure out how to scroll it and finally just ended up using this which worked wonderfully (since the site already has all the data pulled in). As a note, this is scrolling a table element that is at the bottom of the webpage not Chrome itself. Though it works best when the window is maximized.
bg = browser.find_element_by_xpath('/html/body/form/div[3]/div[2]/div/div[2]/div/div[2]/div[4]/div[4]')
for _ in range(3):
bg.send_keys(Keys.END)
time.sleep(.5)
when I use selenium to click element by position with python
1: I can through find_element_by_xpath("//div[#class='close']").click()
to colse some tips
2. I use this ActionChains to click element by code
action = webdriver.common.action_chains.ActionChains(driver)
action.move_by_offset(x_pos, y_pos))
action.click()
action.perform()
but not clicked, I think mouse position have changed buy using fisrt click, and How can I reset mouse position to (0,0) or have mouse position reset method?
I have the code below that basically identify the small chrome icon in the windows toolbar, right-click on it using pyautogui and then it should locate the "New Window" option. The problem I face is that, even if I take a screenshot after the right-click, the small menu doesn't show up, making it impossible to locate the "New Window" option.
# this part works
chrome_small_icon = r"C:\Users\chrome_small_icon.png"
elem = pyautogui.locateOnScreen(chrome_small_icon)
elem_center = pyautogui.center(elem)
pyautogui.click(elem_center, duration=0.5, button="right")
time.sleep(0.5)
im_after_right_click = pyautogui.screenshot()
# this part finds zero element, reason being, the right-click menu is like a ghost...
chrome_new_window = r"C:\Users\new_window_text.png"
elements = pyautogui.locateAllOnScreen(chrome_new_window)
does anybody have any suggestion about how to locate elements inside the menu that appears when we right click on an element?
Thanks
EDIT
it seems this issue happens only if I right click on the windows toolbar. It does work if I right click on other locations of the screen.
Instead of trying to locate the new window text use the keyboard to select the new window option. I just ran the following code on MacOS and I was successfully able to open a new Chrome window:
import pyautogui
import time
pyautogui.rightClick(pyautogui.center(pyautogui.locateOnScreen('chrome.png')))
#chrome.png is an image of the chrome icon
pyautogui.typewrite('new window')
pyautogui.press('enter')
I am currently using isotope to organize my portfolio. I found that if I decided to place elements inside an isotope item and make these elements' position as fixed, they will not displace fixed on screen. Instead, these elements will use the isotope item's x and y position as the initial screen position. For instance, if I place an element as position:fixed and left:0; top:0; this element will not display on the top left corner of the screen. Instead, it will display on the top left corner of the isotope item.
Another issue I encounter is that the z-index of the element inside of the isotope item is not independent. If I give the element a greater z-index than other isotope items, say if the element has the z-index of 100, and all isotope items have the z-index of 10, the "portfolio_display" will still display behind all isotope items.
To explain what I am trying to achieve:
I want to make the isotope items become clickable, and when it is being activated, a full screen portfolio display will show up and take over the screen.
I do know plugin such as lightbox 2 will work. However, I like to have my portfolio display have much richer functions than lightbox 2. I want to put case study, images, videos, in the full screen mode, and each screen can be format like web page. So my method is to put multiple formatted divs into an isotope item, and use javescript (written by myself) to control their visibility and scrolling. This method works until I started integrate isotope to my portfolio.
I imagine isotope created a little "web page" inside of the bigger "web page" so it dictates the z-index and the position inside of its own. I like to know if I can disable isotope's dictation to its sub elements without breaking its great functionality.
Thank you.
For your full-window display purposes, you can change the modal overlay technique in this demo to your purposes. No plugin needed, you can put whatever you want to show in there, not just use it as an easy self-made modal.