I am learning how to retrieve tables from websites into excel.
I usually use the web query function, but sometimes excel doesn't see these tables.
So I am trying to write VBA queries by reading lots of posts, but I have limited knowledge and would appreciate your help.
Website link :
https://www.zacks.com/stocks/industry-rank/sectors/?icid=stocks-industry-nav_tracking-zcom-main_menu_wrapper-zacks_sector_rank
Here's what I wrote so far :
Public Sub GetInfo()
Dim IE As New InternetExplorer, clipboard As Object
With IE
.Visible = True
.navigate "https://www.zacks.com/stocks/industry-rank/sectors/?icid=stocks-industry-nav_tracking-zcom-main_menu_wrapper-zacks_sector_rank"
While .Busy Or .readyState < 4: DoEvents: Wend
Set clipboard = GetObject("New:{1C3B4210-F441-11CE-B9EA-00AA006B1A69}")
With .document.getElementById("industry_sector_data_table").contentDocument
ThisWorkbook.Worksheets("Sheet1").Cells(1, 1) = .getElementsByClassName("industry_sector_data_table")(0).innerText
clipboard.SetText .getElementsByTagName("table")(2).outerHTML
clipboard.PutInClipboard
End With
ThisWorkbook.Worksheets("Sheet1").Cells(2, 1).PasteSpecial
.Quit
End With
End Sub
Here's a picture of the site where the table is located :
First thing, I need to click on the Earnings tab (no idea if it's even possible)
Second item is how do I select the number of entries to 100 ? This is for other tables I need to retrieve in the future. This one actually doesn't need it.
Third, the location of the table. I have "industry_sector_date_table" as table id. Not sure how to actually get the info "td", in this example 80.77%, 83.33% and all the other ones.
The table seem to be number 2.
I also read on one post that there is a way to do it without opening the web browser, not sure if I understood that correctly. Is it possible ?
Hope anybody can help from here.
Thanks
Related
I'm new to VBA, and I just looked up how to autofill data from excel to a certain database (I referred to this youtube video https://www.youtube.com/watch?v=docnLSdj8rI)--I tried other references, but for a newbie, this was the easiest to digest. None of the cases here in SO seemed to fit my scenario, so I thought I'd finally ask.
This is my code so far (I changed the URL of the company website as it is confidential):
Sub autofill()
Dim IE As Object
Dim doc As HTMLDocument
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.navigate "http://tools.filings.irc.company.com/ust/upload/2"
Do While IE.Busy Or IE.readyState 4
Application.Wait DateAdd("s", 1, Now)
Loop
Set doc = IE.document
For intRow = 2 To 5 'or until the last row that still has data, I don't know how to code it that way
IE.document.getElementById("global_CompanyName").Value = ThisWorkbook.Sheets("Sheet1").Range("D" & intRow).Value
doc.getElementById("ddDIVTextContainer").Value = Manila
Application.Wait DateAdd("s", 2, Now)
Next
End Sub
Side note: Code is still incomplete by the way! Haven't written it out for all fields because I keep getting an error 462.
Whenever I press run, it actually opens the company database website on IE, but it doesn't fill any field out. It turns out that I get error 462, which says The remote server machine does not exist or is unavailable. I then press debug, and the thing highlights "Do While IE.Busy Or IE.readyState <> 4" (I get that VBA is case sensitive, and for some reason it automatically makes some upper case letters lowercase (like IE.document instead of IE.Document, IE.readyState instead of IE.ReadyState).)
Any chance I can do something about this? Just to explain, my idea is to have an array where each individual row holds a series of information that the macro will have to input onto the database, much like in the photos below. If there's a way I can also have the macro upload a PDF by linking it from my local drive, that'd also be great!
Here is a picture of the table array I'm planning the macro references the data from to automatically fill out the fields
Here is a photo of the company database website where we have to manually enter the data
I know it's a tough ask, but if you can, please explain things to me in layman's terms. I'm still trying to understand basic concepts, but if I get this done it'd be immensely helpful to my teammates and myself. Cheers!
I'm new at excel VBA programming so for those it may be very easy question but i need to learn it.
I have code part like this :
Sub Calculate()
Dim i As Integer
For i = 1 To 1000
Dim Test As Object
Dim IE As Object
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = False
IE.navigate "https://www.somewebsite.com/" & Cells(i, 2).Value
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
'some code here
.
.
.
.
Next i
End Sub
This code will work as get some data from the same website with different endings. Each has written on some range (for example (A1:A1001)).
My question is each of this window opened visibility as hidden and get some data from web page. It takes too much time
I want each link opened in one window or one window with different tab (I don't know which one takes longer time) Each of solution is okay but preferred one is same window navigate each link.
I would like your help
Thanks in advance
It's not 100% clear what you're after, but from the sound of it you need IE automation (among other things) pulling websites into tabs within the same window.
This may be helpful:
.IE.Navigate2 "www.somewebsite.com", CLng(2048)
I have an intranet based website at work that I'd like to input SKU's and scrape the data that comes up into an excel sheet using VBA and IE8. Currently waiting on Access license approvals and approval to gain access to our IBM as400 server. IE8 does not support getElementsByName, and the search bar has no ID, so most of the examples I've found online aren't relevant.
Sub scraper()
Dim item As Long
item = "10011" 'this will eventually be placed in a loop for multiple searches
Set objIE = CreateObject("InternetExplorer.Application")
objIE.Visible = True
' navigate and download the web page
objIE.Navigate "http://**********.aspx"
Do While objIE.ReadyState <> 4 Or objIE.Busy
DoEvents
Loop
objIE.document.getElementsByName("input").Value = item
End Sub
Here's a photo of the inspected element tree. As you can see, there is only a name (input) and class (st1), no ID. How can i get the search data input into this area? I am stuck with IE8, although I do have access to Chrome as well...
Changing objIE.document.getElementsByName("input").Value = item to objIE.document.getElementsByTagName("input")(0).Value = item ended up resolving this issue.
getElementsByTagName is supported by IE8.
I am new to VBA and html coding in general. I apologise if I don't understand basic terms or use them incorrectly. I was looking to create and run a macro in excel for work that would make my job a lot easier. Essentially, I need to grab a bunch of information off of a real estate website. This includes address, list price, listing agency, auction date (if any) etc. I have spent the last 4 hours reading all about web scraping and I understand the processes, I just don't know how to code it. From what I have read, I need to write a code to automatically open the website, force-ably wait until it's loaded, then retrieve information by either tag, name or id. Is this correct? How can I go about this. What resources should I look to use.
TL;DR How to web scrape text from a webpage of search results (noob instructions).
I will not tell you all the details, you have to find them on your own. Some web pages are complicated, some are easy. Other are impossible, especially if the text is displayed not in HTML but in some other form - picture, Flash etc.
It is however quite simple to extract data from HTML web pages in Excel. First of all, you want to automate it. So click 'Record macro' on the 'Developer' ribbon. This way, you will have all the reproducible step recorded and then you can have a look at the macro a adjust some steps to suit your needs. I however can't teach you here how to program VBA.
When your macro is being recorded, click on 'From web' on the 'Data' ribbon. This will show up a new web query. Then you enter the address of the web page you want to read and try to select (with the little arrow or check-off mark) as narrow area you are interested in as possible. You can also explore some of the fine-tuning options in this wizard dialog.
When you are done, click 'Import' and you will have in some form the contents of the web page. If you are lucky, the data you are interested in will be always in the same cells. Then you can read the cells and store the values somewhere (possible using another macro). If the data are not in the same cells every time you refresh the query, then you have bad luck and have to use some complicated formulas or macros to find them.
Next stop the macro which you are recording and review the code which was recorded. Try to experiment and play around with it until you discover what you actually need. Then it is up to you, how you want to automate it. The options are many...
Otherwise Excel is maybe not the best tool. If I wanted to load HTML page and extract data from it, I would use some scripting e.g. Python which has much better tools than Excel and VBA. There are also tools for converting HTML to XHTML and then extracting data from it like from well-formed XML.
Below is a very basic example illustrating some of the concepts of web scraping. Other reading you should do, would be how to use the other element selectors such as getElementByID getElementByClassName getElementByName.
Here's some code to get you started.
Public Sub ExampleWebScraper()
Dim Browser As Object: Set Browser = CreateObject("InternetExplorer.Application")
Dim Elements As Object 'Will hold all the elements in a collection
Dim Element As Object 'Our iterator that will show us the properties
'Open a page and wait for it to load
With Browser
.Visible = True
.Navigate "www.google.com"
'Wait for the page to load
While .busy Or .readystate <> 4
Application.Wait (Now() + TimeValue("00:00:01"))
Wend
'Enumerate all Elements on the page
'It will store these elements into a collection which we can
'iterate over. The * is the key for ALL, here you can specify
'any tagName and it will limit your search to just those.
'E.g. the most common is Likely Input
Set Elements = .document.getElementsByTagname("*") ' All elements
'Iterate through all elements, and print out some properties
For Each Element In Elements
On Error Resume Next ' This is needed as not all elements have the properties below
' if you try and return a property that doesn't exist for that element
' you will receive an error
'The following information will be output to the 'Immediate Window'
'If you don't see this window, Press Ctrl+G, and it will pop up. That's where this info will display
Debug.Print "The Inner Text is: " & Element.InnerText
Debug.Print "The Value is: " & Element.Value
Debug.Print "The Name is: " & Element.Name
Debug.Print "The ID is: " & Element.ID
Debug.Print "The ClassName is: " & Element.Class
Next Element
End With
'Clean up, free memory
Set Browser = Nothing
Set Elements = Nothing
Set Element = Nothing
End Sub
The question is: is there any way to bind Excel with Salesforce and upload data?
What I am looking for is a code to manipulate Salesforce from excel, that is, click a specified button (New Case), fill in specified fields with information from Excel cells and finally click another button (Save).
Salesforce is opened from Internet Explorer. Does it mean it can be manipulated like IE object? If so, are they methods to navigate through specific fields on the screen?
Any directions very appreciated.
Good example of how to connect to already running instance of IE is this:
Sub connectIE()
Dim sWindows As New ShellWindows
Dim IE As InternetExplorer
For Each IE In sWindows
IE.navigate ("https://en.wikipedia.org/wiki/Main_Page")
'you can put whatever you want to do with IE here
Do
DoEvents
Loop Until IE.readyState = 4
Next
End Sub
please note you have to check the following items in menu>Tools>References:
'Microsoft Scripting Runtime' and 'Microsoft Internet Controls'