This question already has answers here:
is it possible to display a Google Earth map INSIDE Excel?
(3 answers)
Closed 4 years ago.
I have a code that already searches for the latitude and longitude and pastes to my worksheet, which works perfectly. I'm looking for a way to take that latitude and longitude, load google maps, and either take a screenshot of the google maps page or embed the map into Excel.
In my code below I have a code that already loads google maps for any input address, but I do not know how to either take the screenshot of the map (preferably without the input information on the side of the page) or embed the map into Excel. The extra code at the bottom is for a request/response from a USGS website that pulls official seismic information for a location, but should not effect the top part of the code.
Please note that I want this to just be a static screenshot of the map if possible. I do not want to install Google Earth on multiple desktops to be able to embed an interactive map into the worksheet if at all possible.
Option Explicit
Public Sub Seismicgrab()
Dim browser As New ChromeDriver
Dim URL As String
Dim ws As Object
Dim xmlhttp As New MSXML2.XMLHTTP60
browser.Get "http://www.google.com/maps?q=" & Range("H13").Value
browser.Wait 5000
Cells(19, 13).Value = browser.URL
browser.Close
URL = Range("M24").Value
xmlhttp.Open "GET", URL, False
xmlhttp.Send
Worksheets("Title").Range("M25").Value = xmlhttp.responseText
End Sub
You can use the TakeScreenshot method of the object
browser.TakeScreenshot.SaveAs ".....jpg" '<== put your path and file name here
For more flexibility e.g. cropping consider switching languages and using any of these methods:
How to capture the screenshot of a specific element rather than entire page using Selenium Webdriver?
Additionally, there are ways I believe with standard VBA and API calls to take a screenshot and then crop an image.
Related
I tried to combine code parts i could make work, but it was working with <span>, <meta> but it is not working with <img>
Can anyone help to make it work?
I try to get:
https://www.lego.com/cdn/cs/set/assets/blt34360a0ffaff7811/11015_alt.png?fit=bounds&format=png&width=800&height=800&dpr=1
From this code:
<img src="https://www.lego.com/cdn/cs/set/assets/blt34360a0ffaff7811/11015_alt.png?fit=bounds&format=png&width=800&height=800&dpr=1" alt="" class="Imagestyles__Img-sc-1qqdbhr-0 cajeby">
Code part where i want to get the Src url
Sub picgrab()
Dim Doc As Object
Dim nodeAllPic As Object
Dim nodeOnePic As Object
Dim pic As Object
Set Doc = CreateObject("htmlFile")
With CreateObject("MSXML2.XMLHTTP.6.0")
url = "https://www.lego.com/hu-hu/product/around-the-world-11015"
.Open "GET", url, False
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0"
.send
' It is important that i can't use InternetExplorer.
'This should work i guess, but it skips after 'For Each' line.
Set nodeAllPic = Doc.getElementsByClassName("Imagestyles__Img-sc-1qqdbhr-0 cajeby")
For Each nodeOnePic In nodeAllPic
If nodeOnePic.getAttribute("class") = "Imagestyles__Img-sc-1qqdbhr-0 cajeby" Then
Set pic = nodeOneVip.getElementsByClassName("Imagestyles__Img-sc-1qqdbhr-0 cajeby")(0)
ActiveCell.Value = pic.getAttribute("src")
End If
Next nodeOnePic
End With
End Sub
I tired the code above and modified it many way, but couldn't get the content of Src="" .
Need to write the response
First of all, you never write the HTML response to your htmlfile object. So you won't be able to find anything when you call the method getElementsByClassName on it.
Make sure that you include the following line before trying to use the Doc object:
Doc.Write .responseText
Dynamic Content
Secondly, some of the content on that page is not in the original HTTP request that XMLHTTP receives. The page contains JavaScript code that loads content dynamically.
To test this in Chrome, you can open the Chrome DevTools window on that page, then disable JavaScript and refresh the page.
You'll then see the original HTML and a notification that says that JavaScript is disabled.
And now, if you search inside the Elements tab, you won't find the element you were looking for (at least I couldn't find anything with a class "cajeby").
Browser emulation
So, now what? Well, you'll need to use an object to manipulate the original response to execute the JavaScript code. For that you could use Selenium. It's the modern way of doing web scraping or any browser automation with VBA.
You can easily find tutorials on how to get started with Selenium for VBA, but I would recommend this video by WiseOwlTutorials.
Then your code could look like this:
Dim Browser As New Selenium.WebDriver
Browser.Start "chrome", "https://www.lego.com/hu-hu/product/around-the-world-11015"
Browser.Get "/"
Dim img As WebElement
Set img = Browser.FindElementByCss(".Imagestyles__Img-sc-1qqdbhr-0.cajeby", timeout:=5000)
Debug.Print img.Attribute("src")
Set Browser = Nothing
Some notes on the code
Make sure that you have included a reference to the Selenium Library
Notice the use of FindElementByCss. This is necessary because you are using 2 class names and no other method currently support that, but you'll need to use the CSS selector syntax. (More about this here).
Notice the use of timeout:=5000 that lets Selenium know that you are willing to wait up to 5000 milliseconds for the JavaScript code to load the content you are looking for (More details here).
I am trying to copy the text highlighted in Yellow (from the HTML image) using a Selenium VBA code with MS Excel to a string. However, it is crashing and not getting copied. But the same code is working if I were to use it on other websites where the division does not contain "Data-Role" element within a webpage. I tried to make use of all element type like ByName ById ByClass but none of these technique seems to be working. Please note I am using Selenium to access the Chrome browser. I am trying to store the Yellow highlighted text data into a string "Res". Any help or tip is much appreciated. Below is the sample code which I am trying to use.
Dim Res As String
Dim be As New WebDriver
On Error Resume Next
be.Start "chrome", ""
be.Get "the website from which I am tryig to pull data"
Res = be.FindElementByClass("text--1jzYQ uppercase--tL_HU").Text
[![enter image description here][2]][2]
Without having the website address (if able to test with it), or any error messages, can't tell you why it is crashing. Possibly accessing a dynamic element which is still resolving? Dunno.
However, those classes look dynamic, so in terms of robustness, perhaps try using a combination of more stable looking attribute = value css selectors:
be.FindElementByCss("[data-role=status-bar] [data-role=status-text]").text
To prepare for the eventual 'going away' of IE11, I've been trying to figure out how to replace a couple parts of my code. One involves launching IE and using that browser to scrape some pages. Is there an equivalent way to do the below in Edge? I don't see a way to add a reference to the Edge libraries like I did with 'Microsoft Internet Objects' and IE11.
Dim ie As InternetExplorerMedium: Set ie = New InternetExplorerMedium
Dim html As HTMLDocument
With ie
.Visible = False
.Navigate website 'string that's created above this code
End With
Do While ie.ReadyState <> READYSTATE_COMPLETE
DoEvents
Loop
Application.Wait Now + #12:00:10 AM#
Set html = ie.Document
Thanks everyone for your help.
Ok, a few explanations. I am writing these as a reply so as not to have to split them into several comments.
Does Edge work instead of IE to do web scraping with VBA?
It does not work directly. The reason is that IE has a COM interface (Wikipedia: Component Object Model). No other browser has this interface. Not even Edge.
But for Edge there is also a web driver for Selenium. Even provided directly by MS.
Another alternative - xhr
Since you can't use Selenium because you don't have admin rights, there might be the possibility to use xhr (XML HTTP Request). However, in order to make a statement on this, we would have to know the page that you want to scrape.
Xhr can be used directly from VBA because it does not use a browser. The big limitation is that only static content can be processed. No JavaScript is executed, so nothing is reloaded or generated dynamically in any other way. On the other hand, this option is much faster than browser solutions. Often, a static file provided by the web server is sufficient. This can be an HTML file, a JSON or another data exchange format.
There are many examples of using xhr with VBA here on SO. Take note of the possibility first as another approach. I can't explain the method exhaustively here, also because I don't know everything about it myself. But there are many ways to use it.
By the way
IE will finally be discontinued in June 2022 and will then also no longer be delivered with Windows. That's what I read on the German IT pages a few days ago. But there are already massive restrictions on the use of IE.
The problem that I'm having is quite simple. I'm opening a webpage, looking for the input box where I type some text and then hit a Search button. Once the new webpage is uploaded I gather all the info I need. My problem is in the time spent uploading the webpage. My gathering code doesn't work because the new webpage is still not loaded. I have the following code to wait for that:
Do While ie.ReadyState <> READYSTATE_COMPLETE
DoEvents
Loop
where ie was set like this
Set ie = New InternetExplorer
Is there another code except the application.wait that I can use to fix this?
I've run into similar issues when attempting the same. The issue is that the ready state on the IE object can't always be trusted, or at the very least, it's not signaling what you think. For example it will let you know when each frame is ready, not the whole page. So if you don't actually need to see the web browser control, and you only care about sending and receiving data. My suggestion is to not bother rending the page in a web browser object, instead just send and receive data using a WinHttpRequest.
Tools>References>Microsoft WinHTTP Services
Using this, you can send and receive the HTML data directly. If your page uses URL parameters, you send a "GET" then parse the reply. Otherwise you will have to send a "PUT" and send the edited HTML (Basically take the blank form page you begin with and set all the values). When first using, it can be a bit tricky to get the formatting correct depending on the complexity of the page you are trying to automate. Find a good web dugging tool (such as Fiddler) so that you can see the HTML being sent to your target page.
How to capture the Whole web page when using QTP?
I am aware of 'CaptureBitmap' method for the screenshot. But how to capture the Whole page? Help !!
What do you want to capture? If it's the HTML you can create a checkpoint on the Page test object and check the HTML source checkbox in the HTML verification section.
If you want to capture an image of the page then you can only capture the visible part with CaptureBitmap there is no way to get an image of the scrolled out parts (unless you scroll and use multiple captures).
Use Browser("").Capturebitmap.
This takes the screenshot of the visible browser.
Use the sendkeys method to do a page down, then use Browser("").Capturebitmap again!
A full screen shot can be taken by toggling QTP's run settings rather than using CaptureBitmap. We can tell QTP to always take screen shots, interact with the page (or object) we wish to capture (e.g. call .Exist(0)) and this will feed a screen shot in to the results.
The code to do this:
Dim App 'As Application
Set App = CreateObject("QuickTest.Application")
App.Options.Run.ImageCaptureForTestResults = "Always"
Browser("index:=0").Page("index:=0").sync
App.Options.Run.ImageCaptureForTestResults = "OnError"
Technically this seems to be capturing the html and then presenting this to the user in the run results, rather than an actual image of the browser's presentation of the html. But still, this means we can see what's on the page but not visible.
I have went through lot of surfing but couldn't get right answer or I coudn't implement what I found due to restriction of using third party APIs in my office. By using dot net factory, we can use dot net libraries to take screen shots and merge them. refer the below page for complete code
http://www.testbasket.com/2015/08/capture-whole-web-page-using-uftqtp.html
However here i have pasted the contents from the page and hope it helps.
In order to do take the screenshot of complete page, I have used DotNetFactory and System.Drawing dot net library.
Lets go step by step to the solution,
As part of implementing the solution, we need to get the height and weight of the entire page. In order to get that we using DOM of a page using .object method.
#Get the Full Height of Page
FullHeight = Browser("Wikipedia, the free encycloped").Object.document.body.scrollheight
#Get the Full width of Page
Fullwidth = Browser("Wikipedia, the free encycloped").Object.document.body.scrollwidth
Once we found the complete page size, we need to find the client size (how much browser can show)
#Get the visible height - Viewable part of the page
BrowserHeight = Browser("Wikipedia, the free encycloped").Object.document.body.clientHeight
#Get the visible width - Viewable part of the page
Browserwidth = Browser("Wikipedia, the free encycloped").Object.document.body.clientwidth
Next we need to import required dot net libraries using Dot Net Factory
Set oGraphics=DotNetFactory.CreateInstance("System.Drawing.Graphics")
Set oPoint=DotNetFactory.CreateInstance("System.Drawing.Point")
Set oImgFormat=DotNetFactory.CreateInstance("System.Drawing.Imaging.ImageFormat","System.Drawing", Nothing)
Set oImageLib = DotNetFactory.CreateInstance("System.Drawing.Image")
Set oPens=DotNetFactory.CreateInstance("System.Drawing.Pens","System.Drawing")
As a final step, we need to loop through the page and take screenprints separately. finally using Dotnet library we will merge the images using graphics. draw method. It is easy to implement, complete set of code is available in the above mentioned link for reference
If you would like a single screenshot of the whole page, try using SnagIt.
There's a handy PDF with more info on how to go about it (http://download.techsmith.com/snagit/docs/comserver/enu/snagitcom.pdf)
In QTP it might look like this:
Sub Capture_Scroll_Image ()
Set objShell = CreateObject("WScript.Shell")
Set oSnag = CreateObject("SNAGIT.ImageCapture")
oSnag.IncludeCursor = False
oSnag.OutputImageFile.FileType = 5
oSnag.OutputImageFile.FileNamingMethod = 1
oSnag.OutputImageFile.Directory = "C:\Screens\"
oSnag.OutputImageFile.Filename = "Name"
oSnag.EnablePreviewWindow = False
oSnag.AutoScrollOptions.AutoScrollMethod= 1
oSnag.Capture()
Wait (1)
objShell.SendKeys "{ENTER}"
capDone = oSnag.IsCaptureDone
Do Until oSnag.IsCaptureDone
Loop
Set oSnag=Nothing
Set objShell=NothingEnd Sub
End Sub