How can I scrape worded information from a website? - excel

Question

I am new to VBA and html coding in general. I apologise if I don't understand basic terms or use them incorrectly. I was looking to create and run a macro in excel for work that would make my job a lot easier. Essentially, I need to grab a bunch of information off of a real estate website. This includes address, list price, listing agency, auction date (if any) etc. I have spent the last 4 hours reading all about web scraping and I understand the processes, I just don't know how to code it. From what I have read, I need to write a code to automatically open the website, force-ably wait until it's loaded, then retrieve information by either tag, name or id. Is this correct? How can I go about this. What resources should I look to use.
TL;DR How to web scrape text from a webpage of search results (noob instructions).

Answer 1

I will not tell you all the details, you have to find them on your own. Some web pages are complicated, some are easy. Other are impossible, especially if the text is displayed not in HTML but in some other form - picture, Flash etc.
It is however quite simple to extract data from HTML web pages in Excel. First of all, you want to automate it. So click 'Record macro' on the 'Developer' ribbon. This way, you will have all the reproducible step recorded and then you can have a look at the macro a adjust some steps to suit your needs. I however can't teach you here how to program VBA.
When your macro is being recorded, click on 'From web' on the 'Data' ribbon. This will show up a new web query. Then you enter the address of the web page you want to read and try to select (with the little arrow or check-off mark) as narrow area you are interested in as possible. You can also explore some of the fine-tuning options in this wizard dialog.
When you are done, click 'Import' and you will have in some form the contents of the web page. If you are lucky, the data you are interested in will be always in the same cells. Then you can read the cells and store the values somewhere (possible using another macro). If the data are not in the same cells every time you refresh the query, then you have bad luck and have to use some complicated formulas or macros to find them.
Next stop the macro which you are recording and review the code which was recorded. Try to experiment and play around with it until you discover what you actually need. Then it is up to you, how you want to automate it. The options are many...
Otherwise Excel is maybe not the best tool. If I wanted to load HTML page and extract data from it, I would use some scripting e.g. Python which has much better tools than Excel and VBA. There are also tools for converting HTML to XHTML and then extracting data from it like from well-formed XML.

Answer 2

Below is a very basic example illustrating some of the concepts of web scraping. Other reading you should do, would be how to use the other element selectors such as getElementByID getElementByClassName getElementByName.
Here's some code to get you started.
Public Sub ExampleWebScraper()
Dim Browser As Object: Set Browser = CreateObject("InternetExplorer.Application")
Dim Elements As Object 'Will hold all the elements in a collection
Dim Element As Object 'Our iterator that will show us the properties
'Open a page and wait for it to load
With Browser
.Visible = True
.Navigate "www.google.com"
'Wait for the page to load
While .busy Or .readystate <> 4
Application.Wait (Now() + TimeValue("00:00:01"))
Wend
'Enumerate all Elements on the page
'It will store these elements into a collection which we can
'iterate over. The * is the key for ALL, here you can specify
'any tagName and it will limit your search to just those.
'E.g. the most common is Likely Input
Set Elements = .document.getElementsByTagname("*") ' All elements
'Iterate through all elements, and print out some properties
For Each Element In Elements
On Error Resume Next ' This is needed as not all elements have the properties below
' if you try and return a property that doesn't exist for that element
' you will receive an error
'The following information will be output to the 'Immediate Window'
'If you don't see this window, Press Ctrl+G, and it will pop up. That's where this info will display
Debug.Print "The Inner Text is: " & Element.InnerText
Debug.Print "The Value is: " & Element.Value
Debug.Print "The Name is: " & Element.Name
Debug.Print "The ID is: " & Element.ID
Debug.Print "The ClassName is: " & Element.Class
Next Element
End With
'Clean up, free memory
Set Browser = Nothing
Set Elements = Nothing
Set Element = Nothing
End Sub