I am learning Python and attempting to build a program that will scrape specific data from a website, store it and then manipulate it.
Currently I run my application, it opens a new chrome browser window and loads the page correctly. The problem is it should begin to start scrolling down and loading the remaining elements on the page.
I know the code works because if I manually click somewhere on the page that doesn't normally illicit a response (white space/empty areas) the browser somehow comes into "focus" and begins to iterate through the loop that scrolls down the page (by sending keys) prints the data I am after. I also noticed if I click another similar "dead space" area that contains the header, it doesn't have the same effect. I am unsure if this is something specific to Chrome, iFrames or something of that nature but I am completely stumped and would greatly appreciate any help.
Any thoughts on why I need to manually click on the new chrome window for it to work would be great.
Update: Still having the same issue, even tried with Safari and the same problem seems to exist.
Fixed this with:
element = driver.find_element_by_css_selector("div[id^='app-container']")
action = ActionChains(driver)
action.click(on_element = element)
action.perform()
I want to crawl the "question contents" and the "link" of each question asked on Stack Overflow website, but the results I crawled are "None", the type is printed as "NoneType", which is quite weird.
I can crawl down the titles, vote number and tags of the question successfully, but fail with crawling the question contents and the link.
import requests
from bs4 import BeautifulSoup
import re
import json
from selenium import webdriver
browser = webdriver.Chrome('./chromedriver')
browser.get('https://stackoverflow.com/questions/tagged/android?sort=votes&page=1&pagesize=50')
soup = BeautifulSoup(browser.page_source, 'html.parser')
divs = soup.find_all('div', class_ = 'question-summary')
for div in divs:
#title ok
y= div.h3.a.text
#tag ok
tags = div.find('div', class_ = 'summary').find_all('div')[1].find_all('a')
h= [tag.text for tag in tags]
#vote ok
v = div.div.div.div.div.span.strong.text
#link??
l = div.h3.a.href
#print(type(l))
#contents??
b = div.class_excerpt
#print(b)
I expect the "question link" and "contents" within the question can be extracted as well, but don't know how to revise the structure.
Try Following code.It gives me link and content.
browser.get('https://stackoverflow.com/questions/tagged/android?sort=votes&page=1&pagesize=50')
soup = BeautifulSoup(browser.page_source, 'html.parser')
links=soup.find_all('a', class_='question-hyperlink')
contents=soup.find_all('div',class_='excerpt')
for link,content in zip(links,contents):
print('link :' + link['href'] , 'content :' + content.text.replace('\n',' ').strip())
Output:
(link :/questions/2025282/what-is-the-difference-between-px-dip-dp-and-sp', content :What is the difference between Android units of measure? px dip dp sp')
(link :/questions/1109022/close-hide-the-android-soft-keyboard', content :I have an EditText and a Button in my layout. After writing in the edit field and clicking on the Button, I want to hide the virtual keyboard. I assume that this is a simple piece of code, but where ...')
(link :/questions/13375357/proper-use-cases-for-android-usermanager-isuseragoat', content :I was looking at the new APIs introduced in Android 4.2. While looking at the UserManager class I came across the following method: public boolean isUserAGoat() Used to determine whether the ...')
(link :/questions/1554099/why-is-the-android-emulator-so-slow-how-can-we-speed-up-the-android-emulator', content :I have got a 2.67\xa0 GHz Celeron processor, and 1.21\xa0 GB of RAM on a x86 Windows XP Professional machine. My understanding is that the Android Emulator should start fairly quickly on such a ...')
(link :/questions/1555109/stop-edittext-from-gaining-focus-at-activity-startup', u"content :I have an Activity in Android, with two elements: EditText ListView When my Activity starts, the EditText immediately has input focus (flashing cursor). I don't want any control to have input focus ...")
(link :/questions/2785485/is-there-a-unique-android-device-id', content :Do Android devices have a unique ID, and if so, what is a simple way to access it using Java?')
(link :/questions/151777/how-do-save-an-android-activity-state-using-save-instance-state', u"content :I've been working on the Android SDK platform, and it is a little unclear how to save an application's state. So given this minor re-tooling of the 'Hello, Android' example: package com.android.hello;...")
(link :/questions/6343166/how-do-i-fix-android-os-networkonmainthreadexception', content :I got an error while running my Android project for RssReader. Code: URL url = new URL(urlToRssFeed); SAXParserFactory factory = SAXParserFactory.newInstance(); SAXParser parser = factory....')
(link :/questions/101754/is-there-a-way-to-run-python-on-android', content :We are working on an S60 version and this platform has a nice Python API.. However, there is nothing official about Python on Android, but since Jython exists, is there a way to let the snake and the ...')
(link :/questions/541966/lazy-load-of-images-in-listview', content :I am using a ListView to display some images and captions associated with those images. I am getting the images from the Internet. Is there a way to lazy load the images so while the text displays, ...')
(link :/questions/432037/how-do-i-center-text-horizontally-and-vertically-in-a-textview', content :How do I center the text horizontally and vertically in a TextView, so that it appears exactly in the middle of the TextView in Android?')
(link :/questions/2194808/debug-certificate-expired-error-in-eclipse-android-plugins', content :I am using Eclipse Android plugins to build a project, but I am getting this error in the console window: [2010-02-03 10:31:14 - androidVNC]Error generating final archive: Debug certificate expired ...')
(link :/questions/1016896/get-screen-dimensions-in-pixels', content :I created some custom elements, and I want to programmatically place them to the upper right corner (n pixels from the top edge and m pixels from the right edge). Therefore I need to get the screen ...')
(link :/questions/3572463/what-is-context-on-android', content :In Android programming, what exactly is a Context class and what is it used for? I read about it on the developer site, but I am unable to understand it clearly.')
(link :/questions/5761960/what-is-the-difference-between-match-parent-and-fill-parent', u"content :I'm a little confused about two XML properties: match_parent and fill_parent. It seems that both are the same. Is there any difference between them?")
(link :/questions/456211/activity-restart-on-rotation-android', u"content :In my Android application, when I rotate the device (slide out the keyboard) then my Activity is restarted (onCreate is called). Now, this is probably how it's supposed to be, but I do a lot of ...")
(link :/questions/3482742/what-is-the-difference-between-gravity-and-layout-gravity-in-android', content :I know we can set the following values to the android:gravity and android:layout_gravity properties: center center_vertical center_horizontal, etc. But I am confused regarding both of these. What ...')
(link :/questions/2201917/how-can-i-open-a-url-in-androids-web-browser-from-my-application', content :How to open an URL from code in the built-in web browser rather than within my application? I tried this: try { Intent myIntent = new Intent(Intent.ACTION_VIEW, Uri.parse(download_link)); ...')
(link :/questions/2091465/how-do-i-pass-data-between-activities-in-android-application', content :I have a scenario where, after logging in through a login page, there will be a sign-out button on each activity. On clicking sign-out, I will be passing the session id of the signed in user to sign-...')
(link :/questions/477572/strange-out-of-memory-issue-while-loading-an-image-to-a-bitmap-object', content :I have a list view with a couple of image buttons on each row. When you click the list row, it launches a new activity. I have had to build my own tabs because of an issue with the camera layout. The ...')
(link :/questions/1678122/must-override-a-superclass-method-errors-after-importing-a-project-into-eclips', content :Anytime I have to re-import my projects into Eclipse (if I reinstalled Eclipse, or changed the location of the projects), almost all of my overridden methods are not formatted correctly, causing the ...')
(link :/questions/4382178/android-sdk-installation-doesnt-find-jdk', u"content :I'm trying to install the Android SDK on my Windows 7 x64 System. jdk-6u23-windows-x64.exe is installed, but the Android SDK setup refuses to proceed because it doesn't find the JDK installation. Is ...")
(link :/questions/3593420/is-there-a-way-to-get-the-source-code-from-an-apk-file', content :The hard drive on my laptop just crashed and I lost all the source code for an app that I have been working on for the past two months. All I have is the APK file that is stored in my email from when ...')
(link :/questions/15852122/hex-transparency-in-colors', u"content :I'm working on implementing a widget transparency option for my app widget although I'm having some trouble getting the hex color values right. Being completely new to hex color transparency I ...")
(link :/questions/4616095/how-to-get-the-build-version-number-of-your-android-application', content :I need to figure out how to get or make a build number for my Android application. I need the build number to display in the UI. Do I have to do something with AndroidManifest.xml?')
(link :/questions/2033914/is-quitting-an-application-frowned-upon', content :Moving on in my attempt to learn Android, I just read the following: Question: Does the user have a choice to kill the application unless we put a menu option in to kill it? If no such option ...')
(link :/questions/937313/fling-gesture-detection-on-grid-layout', u"content :I want to get fling gesture detection working in my Android application. What I have is a GridLayout that contains 9 ImageViews. The source can be found here: Romain Guys's Grid Layout. That file I ...")
(link :/questions/2850573/activity-has-leaked-window-that-was-originally-added', content :What is this error, and why does it happen? 05-17 18:24:57.069: ERROR/WindowManager(18850): Activity com.mypkg.myP has leaked window com.android.internal.policy.impl.PhoneWindow$DecorView#44c46ff0 ...')
(link :/questions/885009/r-cannot-be-resolved-android-error', content :I just downloaded and installed the new Android SDK. I wanted to create a simple application to test drive it. The wizard created this code: package eu.mauriziopz.gps; import android.app.Activity; ...')
(link :/questions/4535298/how-do-i-rotate-the-android-emulator-display', content :How can I rotate the Android emulator display to see it in landscape mode?')
(link :/questions/3028306/download-a-file-with-android-and-showing-the-progress-in-a-progressdialog', content :I am trying to write a simple application that gets updated. For this I need a simple function that can download a file and show the current progress in a ProgressDialog. I know how to do the ...')
(link :/questions/5369682/get-current-time-and-date-on-android', content :How can I get the current time and date in an Android app?')
(link :/questions/16608135/android-studio-add-jar-as-library', u"content :I'm trying to use the new Android Studio but I can't seem to get it working correctly. I'm using the Gson library to serialize/deserialize JSON-objects. But the library somehow isn't included in the ...")
(link :/questions/2115758/how-do-i-display-an-alert-dialog-on-android', content :I want to display a dialog/popup window with a message to the user that shows "Are you sure you want to delete this entry?" with one button that says \'Delete\'. When Delete is touched, it should delete ...')
(link :/questions/513084/ship-an-application-with-a-database', content :If your application requires a database and it comes with built in data, what is the best way to ship that application? Should I: Precreate the SQLite database and include it in the .apk? Include the ...')
(link :/questions/6495898/findviewbyid-in-fragment', content :I am trying to create an ImageView in a Fragment which will refer to the ImageView element which I have created in the XML for the Fragment. However, the findViewById method only works if I extend an ...')
(link :/questions/2734270/how-do-i-make-links-in-a-textview-clickable', content :I have the following TextView defined: <TextView android:layout_width="wrap_content" android:layout_height="wrap_content" android:text="#string/txtCredits" android:autoLink="web" android:...')
(link :/questions/24885223/why-doesnt-recyclerview-have-onitemclicklistener', content :I was exploring RecyclerView and I was surprised to see that RecyclerView does not have onItemClickListener(). Because RecyclerView extends android.view.ViewGroup and ListView extends ...')
(link :/questions/2680827/conversion-to-dalvik-format-failed-with-error-1-on-external-jar', content :In my Android application in Eclipse I get the following error. UNEXPECTED TOP-LEVEL EXCEPTION: java.lang.IllegalArgumentException: already added: Lorg/xmlpull/v1/XmlPullParser; .... ...')
(link :/questions/11461607/cant-start-eclipse-java-was-started-but-returned-exit-code-13', content :I am trying to get my first taste of Android development using Eclipse. I ran into this problem when trying to run Eclipse, having installed version 4.2 only minutes ago. After first trying to start ...')
(link :/questions/2002288/static-way-to-get-context-in-android', u"content :Is there a way to get the current Context instance inside a static method? I'm looking for that way because I hate saving the 'Context' instance each time it changes.")
(link :/questions/11078487/whats-toolscontext-in-android-layout-files', content :Starting with a recent new version of ADT, I\'ve noticed this new attribute on the layout XML files, for example: <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:...')
(link :/questions/10407159/how-to-manage-startactivityforresult-on-android', u"content :In my activity, I'm calling a second activity from the main activity by startActivityForResult. In my second activity there are some methods that finish this activity (maybe without result), however, ...")
(link :/questions/21814825/you-need-to-use-a-theme-appcompat-theme-or-descendant-with-this-activity', content :Android Studio 0.4.5 Android documentation for creating custom dialog boxes: http://developer.android.com/guide/topics/ui/dialogs.html If you want a custom dialog, you can instead display an ...')
(link :/questions/3035692/how-to-convert-a-drawable-to-a-bitmap', u"content :I would like to set a certain Drawable as the device's wallpaper, but all wallpaper functions accept Bitmaps only. I cannot use WallpaperManager because I'm pre 2.1. Also, my drawables are downloaded ...")
(link :/questions/600207/how-to-check-if-a-service-is-running-on-android', content :How do I check if a background service (on Android) is running? I want an Android activity that toggles the state of the service -- it lets me turn it on if it is off and off if it is on.')
(link :/questions/582185/disable-landscape-mode-in-android', content :How can I disable landscape mode for some of the views in my Android app?')
(link :/questions/3875184/cant-create-handler-inside-thread-that-has-not-called-looper-prepare', content :What does the following exception mean; how can I fix it? This is the code: Toast toast = Toast.makeText(mContext, "Something", Toast.LENGTH_SHORT); This is the exception: java.lang....')
(link :/questions/4893953/run-install-debug-android-applications-over-wi-fi', u"content :I thought there was a way to test your applications in development over Wi-Fi. Is this possible? I'd love to be able to untether my phone and develop wirelessly.")
(link :/questions/2176922/how-do-i-create-a-transparent-activity-on-android', content :I want to create a transparent Activity on top of another activity. How can I achieve this?')
Edited
from selenium import webdriver
from bs4 import BeautifulSoup
browser = webdriver.Chrome('./chromedriver')
browser.get('https://stackoverflow.com/questions/tagged/android?sort=votes&page=1&pagesize=50')
soup = BeautifulSoup(browser.page_source, 'html.parser')
links=soup.find_all('a', class_='question-hyperlink')
for link in links:
print('link :' + link['href'] )
browser.get("https://stackoverflow.com" + link['href'])
soup = BeautifulSoup(browser.page_source, 'html.parser')
print('link :' + link['href'] , 'cotent :' + soup.find('div',class_='post-text').text.replace('\n',' ').strip())
I am trying to automate the process of logging into a website and downloading reports.
When choosing which report to download you have to mouse over a menubar on the website and hovering the mouse over the right tab will open a pulldown menu where you then click the report you want.
Screenshot of the pulldown menu (sorry, but no points embed the image :( )
I have little experience with coding and I am very new to Python 3 so I am struggling on how to do this.
Using Selenium library I am able to find things such as the user and password fields and pass in the information to automatically log into the site. But not sure how to grab or point at the correct report in this pulldown menu in order to simpy click it.
Here is a snippet of the page source... the class name = rmText... and the class name is same for all of them... only the text inside the <span> is unique but I don't know how to grab that with the Python 3 code.
Screenshot of the page source
I know of a way to do this using VB script... but I would prefer to stay with Python 3 (and also learn how to do this via Python in the process). The example below was used for a similar report where multiple lines had same class-name and we needed just the one called "Infill". An example code of the VB script:
Set project = .Document.getElementsByClassName("UPC_Items")
For Each elm In project
If instr(elm.innerhtml,"Infill") then
elm.Click
Exit For
End If
Next
How can I achieve the same using Python 3? And is Selenium the right library to use?
Thanks for reading and hopefully helping out a newbie in Python!
try below code, hope this helps:
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
import time
drive = webdriver.Firefox()
driver.get('your url')
....
steps to reach your place where that menu link is presert
....
element = driver.find_element_by_class("rmText")
putmouse = ActionChains(driver).move_to_element(element)
putmouse.perform()
time.sleep(2)
texttoclick = driver.find_element_by_xpath(".//li[#class='rmText']//span[text()='MCC']")
actions.move_to_element(texttoclick).click().perform()
Image Attached I want to click on the specific button on a webpage, using selenium webdriver and Python. Tried finding the element by CSS, class name and XPath, but it doesn't seem to work. I have attached an image showing the button I want to click (It is not a dropdown as the image suggests) and the HTML details. Any help would be appreciated.
Actually sometimes selenium is not able to interact with some web elements using click try simulating enter key press on that element for e.g. -
element = driver.find_element_by_id("value")
element.send_keys(:return)
Let me know if that works
I am using Selenium 2.40.2 for testing web application. I need the mouse move and click to draw a polygon on the google map.
The code is below:
bottom_label = browser_drive.find_element_by_xpath("//span[text()='Project']")
time.sleep(2)
drawing = ActionChains(browser_drive)\
.move_to_element(bottom_label)\
.move_by_offset(-650, -600)\
.click()\
.move_by_offset(100, -100)\
.click()\
.move_by_offset(300, 0)\
.click()\
.move_by_offset(0, 400)\
.click()\
.move_by_offset(-50, -50)\
.double_click()\
.perform()
drawing.perform()
time.sleep(2)
The bottom_label used to point the mouse at the end of web page then sleep to start again change offset, click and close the polygon.
This code with Selenium version does not work probably. What should I do to draw the polygon through move and click the mouse?
Consider using a FluentWait, ImplicityWait and setScriptsTimeout Instead of Sleep.
See further documentation
http://selenium.googlecode.com/svn/trunk/docs/api/java/org/openqa/selenium/WebDriver.Timeouts.html
The problem was with the new version of Selenium and Firefox.
I changed to the Chrome driver and the click function works fine.