Hey I'm trying to scroll the amazon offers page using puppeteer but it is not scrolling and there is no mouse event happening.
This is the offers page URL.
https://www.amazon.com/dp/1416545360/ref=olp_aod_early_redir?_encoding=UTF8&aod=1
This is the selector I'm trying to scroll on the above page. #all-offers-display-scroller
I would appreciate your help regarding this.
I need to use the puppeteer own methods to serve this purpose.
You can use pressing space on main scrollable element. Ideally it works in most of the sites just by pressing space you can go down
Here's the code for the page you given,
const element = await page.$('#aod-container')
await element.press('Space')
Related
I'm wondering if anyone can help me with this two part problem.
I want to load all of the "swiper-slide" items inside this swiper container. They will only load if manualy scrolling through. I'm wondering if there's a way to force them to all be rendered using web-inspect console.
Is there a way to select one of the slides / button containers using inspect element console on a browser without resorting to using a headless browser.
Code / swiper slider / slides
I could scroll down or up a whole web page but I am having trouble in scrolling a particular div element within the webpage
The thing is when I open a webpage like whatsapp web and within a particular chat containing the messages(which is particular div element),I want to scrape all the messages from the beginning of the chat ,but I could only scrape the messages which is in the view (the last few messages),So I want to scroll that particular div element to the top of the chat to scrape all the messages.
Can someone help me with this thing in PYTHON.
Thank you.
Yes, there are multiple ways using which you could scroll to particular element
By using the move_to_element()
element = driver.find_element_by_xpath("_element_")
action = ActionChains(driver)
action.move_to_element(element).perform()
By scrollIntoView()
element = driver.find_element_by_xpath("_element_")
driver.execute_script("arguments[0].scrollIntoView(true);", element)
For reference check here
It is possible.
Let's say the element you need to scroll inside is div_el, it's locator is xpath_locator the code will be like this:
div_locator = "xpath_locator"
div_el = driver.find_element_by_xpath(div_locator)
driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", div_el)
See more here
I am learning Python and attempting to build a program that will scrape specific data from a website, store it and then manipulate it.
Currently I run my application, it opens a new chrome browser window and loads the page correctly. The problem is it should begin to start scrolling down and loading the remaining elements on the page.
I know the code works because if I manually click somewhere on the page that doesn't normally illicit a response (white space/empty areas) the browser somehow comes into "focus" and begins to iterate through the loop that scrolls down the page (by sending keys) prints the data I am after. I also noticed if I click another similar "dead space" area that contains the header, it doesn't have the same effect. I am unsure if this is something specific to Chrome, iFrames or something of that nature but I am completely stumped and would greatly appreciate any help.
Any thoughts on why I need to manually click on the new chrome window for it to work would be great.
Update: Still having the same issue, even tried with Safari and the same problem seems to exist.
Fixed this with:
element = driver.find_element_by_css_selector("div[id^='app-container']")
action = ActionChains(driver)
action.click(on_element = element)
action.perform()
I have used python scrapy to extract data from a website. Now i am able to scrape most of the details of a site using scrapy. But my main problem is that iam not able to extract all the reviews of products from the site. I am only able to extract the top 4 reviews which they display on the page and for getting other reviews i have to go to a pop up window which has all the reviews. I looked for 'href' for the popup window but im not able to find it. This is the link that i tried to scrape. The reviews and ratings are at the bottom of the page: https://www.coursera.org/learn/big-data-introduction
Can any one help me by explaining how to extract the reviews from this popup window. Another think to note is that there is infinite scrolling for the pop up.
Thanks in advance.
Scrapy, unlike tools like Selenium and PhantomJS, does not drive a full web browser in the background. You cannot just click a button.
You need to understand what the button does (e.g. does it simply submit a form? Does it do something with JavaScript? Etc.) and reproduce the functionality in your own code.
For example, you might need to read the content of a script element, apply regular expressions to it to pull a URL from a string literal, then make a new HTTP request to that URL, the pell the data you want from the new DOM.
... and then repeat for the next “page” of the infinite scroll.
There are lots of libraries out there to scrape information from web pages. Some of them which I had a look are:
http://phantomjs.org/
http://webdriver.io/
http://casperjs.org/
http://www.nightmarejs.org/
http://codecept.io/
https://data-miner.io
http://chaijs.com/
Surprisingly, none of them provide a way to scrape a popup window. Even if they do, I couldn't figure out how it's done.
The scenario is something like this:
-Visit a url (example.com)
-Fill login form
-Click login button
...and now, webpage opens a popup (an actual browser window) which I need to scrape.
Any suggestions or workarounds for popups?
webdriver.io offers change frame features, so in case your pop up has frame tags or iframe tags you can switch to it and test it. You can read more here
http://webdriver.io/api/protocol/frame.html#Usage