I am learning Python and attempting to build a program that will scrape specific data from a website, store it and then manipulate it.
Currently I run my application, it opens a new chrome browser window and loads the page correctly. The problem is it should begin to start scrolling down and loading the remaining elements on the page.
I know the code works because if I manually click somewhere on the page that doesn't normally illicit a response (white space/empty areas) the browser somehow comes into "focus" and begins to iterate through the loop that scrolls down the page (by sending keys) prints the data I am after. I also noticed if I click another similar "dead space" area that contains the header, it doesn't have the same effect. I am unsure if this is something specific to Chrome, iFrames or something of that nature but I am completely stumped and would greatly appreciate any help.
Any thoughts on why I need to manually click on the new chrome window for it to work would be great.
Update: Still having the same issue, even tried with Safari and the same problem seems to exist.
Fixed this with:
element = driver.find_element_by_css_selector("div[id^='app-container']")
action = ActionChains(driver)
action.click(on_element = element)
action.perform()
Related
I'm wondering if anyone can help me with this two part problem.
I want to load all of the "swiper-slide" items inside this swiper container. They will only load if manualy scrolling through. I'm wondering if there's a way to force them to all be rendered using web-inspect console.
Is there a way to select one of the slides / button containers using inspect element console on a browser without resorting to using a headless browser.
Code / swiper slider / slides
I'm making a small project with selenium. but before I can do anything, I need to wait for the whole page to load. is there anyway, for me to send keys as soon as the search bar loads?
You can use
driver.implicitly_wait(10)
On top of your code so each time you try to find an element it will try for 10 seconds and if it doesn’t find it, it will raise an exception
so basically i am making a program in Python that is going to go to one site, click on one picture (than do something) and than move to another picture (and do the same thing) and so on.. well the problem is, there is like 10 pictures with same img class BUT scr is always changing and i hope that could help me.. so now my code opens one image and it does its job but once it's finished instead of moving to another image(element) with same class it opens the same picture and makes a loop which i don't want..
You can try to use x-path, I'm using a chrome extension: https://chrome.google.com/webstore/detail/xpath-finder/ihnknokegkbpmofmafnkoadfjkhlogph?hl=en
To locate the picture.
And then you can use driver.find_element_by_xpath("X-PATH")
I have a scraping script written in Ruby which uses Selenium, Watir and ChromeDriver, all is working just fine with a Chrome browser window, but trying to run in headless mode just hits;
Selenium::WebDriver::Error::UnknownError: unknown error: Element <input id="desired_element" type="checkbox" name="desired_element" checked="checked"> is not clickable at point (660, 594). Other element would receive the click: <html lang="en">...</html>
I'm using Watir 6.8 and latest Chromedriver 2.33.
Any ideas whats n=behind the different behavior of the page when in headless vs non headless, and how I can deal with this?
The error message is telling you what the problem is when it says "Other element would receive the click:"
This means some other element on the screen is covering the checkbox you are trying to interact with. Likely this is caused by whatever the default browser size in headless mode being different than the default size of your browser when it is run non-headless, resulting in a different arrangement of elements
We can verify if this is the case by asking the size of the window in both headless and normal modes and seeing if the resulting values are the same.
size = browser.window.size
puts "The browser window is #{size.width} wide by #{size.height} high"
There are a few potential ways to solve this:
Specify or alter the browser 'window' size. for example
browser.window.resize_to 1024, 768
I prefer this, and normally have a command such as that to set the
browser size right after it is initialized. Set either to the
minimum supported size for your site, or the minimum recommended
size
Use another means to 'click' on the checkbox, such as sending a space at it
#browser.checkbox(name: "desired_element").send_keys " "
I do not prefer this as it doesn't really solve the source of the problem and you may experience other similar issues interacting with other elements on the site as your script progresses.
The reason for this kind of error is that web page doesn't load well enough to locate the element. As you have mentioned that the tests previously passed when you had headless true such issues might be because of .click() , please try replacing .click with .send_keys
.send_keys(selenium.webdriver.common.keys.Keys.SPACE)
When you use .send_keys() you might hit one more issue if it is failing to find elements, for solving this you will have to find the elements
elements = driver.find_element_by_tag_name("html")
elements.send_keys(Keys.SPACE)
Hope this helps you.
I have a big SVG document here, containing a map of all the quests in a certain online game. Each quest node is inside a SVG <a> element, linking to a distinct named anchor in a big HTML document that loads in another tab, containing further details about that particular quest. This works exactly as desired in desktop Safari, and I'd expect it to work just as well in any browser that supports SVG at all since I'm using only the most basic form of linking, but it fails badly on Mobile Safari (iOS 6) - which is my single most important browser target, considering that the game in question is for the iPad. It only scrolls to the correct anchor on the initial load of the HTML page; clicking a different quest in the SVG tab will cause a switch to the HTML tab, and the hash (fragment ID) in the address bar changes, but the page doesn't auto-scroll.
This appears to be a known limitation in Mobile Safari - hash-only changes in the URL apparently used to force a page reload, and that got over-fixed such that nothing gets triggered at all now. The fixes I've found online all seem to be applicable only in cases where the URL change is being generated programatically, from within the same document, rather than static links from a different document.
Further details:
I've tried doing the named anchors in both the old <a name="..."> form, and the newer <h1 id="..."> form. No difference.
I've tried adding an onhashchange handler, to force the scrolling to take place, but the handler isn't being called at all (verified by putting an alert() in it).
I could presumably fix the problem by having each quest's details in a separate HTML file, but that would severely affect usability - with all the details in a single file, you can use your browser's Find feature to search through them all at once. (Also, deploying 1006 files to my web hosting after each update would be a bit of a pain...)
Anybody have an idea for a work-around?