I'm trying to get the content of a web site and paste it onto a excel sheet.
Before trying to code something in VBA I was simply doing "Control + A" (to select ALL the content from the page I want), "Control + C" (to copy it to the clipboard) and then "Control + V" (to paste it in my sheet).
This sort of information is all I want, this is all I need.
The way I will manipulate the data is another story and it's all working just fine.
I need no other way to get information from the web site then the one that "Control + A" and "Control + C" provides me.
That being said, this is the code I've found and which I'm trying to make it work:
Sub Get_Text_From_Page()
Dim sURL As String
sURL = "https://www.microsoft.com"
Dim IE As Object
Set IE = New InternetExplorer
With IE
.Visible = True
.navigate (sURL)
While .Busy Or .ReadyState <> 4: DoEvents: Wend
Range("A1").Value = .Document.body.innerText
End With
End Sub
This opens a browser and then it crashes with an Automation Error at "While .Busy Or .ReadyState <> 4: DoEvents: Wend".
I've found a few code examples in many places, but they all crash with that Busy or ReadyState stuff...
Edit: Not sure if this is gonna take me anywhere, but I've just found something about SendKeys "^a" and then SendKeys "^c".
You could try to refer to the code example below.
It will launch the IE browser. Open the website. After the site gets loaded, it will select all the content, copy it, and paste it to sheet1 in Excel.
Sub demo()
Dim IE As Object
Sheets("Sheet1").Select
Range("A1").Select
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate "https://microsoft.com"
Do While IE.ReadyState = 4: DoEvents: Loop
Do Until IE.ReadyState = 4: DoEvents: Loop
IE.ExecWB 17, 0
IE.ExecWB 12, 2
Range("A1").Select
ActiveSheet.PasteSpecial Format:="Text", link:=False, DisplayAsIcon:=False
IE.Quit
End Sub
You could try it and let us know whether it works for you.
If you have some questions about the code, please let me know about it.
Related
I have a site I need to log in and then change the URL. After it loads copy the entire site to excel.
If I'm doing it slowly (F8 button) or do application wait function, it work but its not optimal for different PC, and if I set it too high the data will take forever to load.
I wrote this code:
Set Ie = CreateObject("InternetExplorer.Application")
Ie.Visible = True
Ie.Navigate "Site login"
Do While Ie.Busy = True Or Ie.readyState < READYSTATE_COMPLETE
DoEvents
Loop
Set HTMLDoc = Ie.document
HTMLDoc.getElementById("user_login_name").Value = UserName 'the user name is defined before
HTMLDoc.getElementById("user_password").Value = UserPass 'the password is defined before
LoginButton = Ie.document.getElementsByName("button")
Click = Ie.document.getElementsByTagName
LoginButton.Click
Ie.Navigate "The wanted URL to load fully before the next code activates"
'copy and paste in excel
Ie.ExecWB 17, 0 '// SelectAll
Ie.ExecWB 12, 2 '// Copy selection
Range("A1").Select
ActiveSheet.Paste
Range("A1").Select
Ie.Quit
When I run it now I get the user name from login screen
Anything I'm missing?
Thanks in advance for your time
Nik
Put the code below, after second Ie.Navigate in your VBA code may help to fix your issue.
Do While Ie.Busy = True Or Ie.readyState < READYSTATE_COMPLETE
DoEvents
Loop
Your modified code:
Set Ie = CreateObject("InternetExplorer.Application")
Ie.Visible = True
Ie.Navigate "Site login"
Do While Ie.Busy = True Or Ie.readyState < READYSTATE_COMPLETE
DoEvents
Loop
Set HTMLDoc = Ie.document
HTMLDoc.getElementById("user_login_name").Value = UserName 'the user name is defined before
HTMLDoc.getElementById("user_password").Value = UserPass 'the password is defined before
LoginButton = Ie.document.getElementsByName("button")
Click = Ie.document.getElementsByTagName
LoginButton.Click
Ie.Navigate "The wanted URL to load fully before the next code activates"
Do While Ie.Busy = True Or Ie.readyState < READYSTATE_COMPLETE
DoEvents
Loop
'copy and paste in excel
Ie.ExecWB 17, 0 '// SelectAll
Ie.ExecWB 12, 2 '// Copy selection
Range("A1").Select
ActiveSheet.Paste
Range("A1").Select
Ie.Quit
Output:
Sorry to keep you waiting, I will close this due to still waiting from our users for feedback(they use this files only once a month). which means I can't confirm if my solution was correct or not. However what I did was tweak the code. and added an if to check the current pages link and added the Ie.readyState < READYSTATE_COMPLETE into the code. it looks like the following:
If Ie.LocationURL <> "site link" Then Do While Ie.Busy = True Or Ie.readyState <
READYSTATE_COMPLETE DoEvents Loop
End If
I do this twice. once for the login page, and the second time.
It seems to work for me(did the data scrape at least 10 times and no error)
Thank you for your time and sorry for the wait.
I have searched for a while but cannot find a solution. also find similar link on this page and it is locked due to duplcation but actually it is not duplicate of anything. this is not Java or #c or anyother mentioned languages.
this is vba and selenium codes.
on below link on debugging mode (F8) everything working smoothly however on run mode (F5) it is not filling datefrom and dateto section or filling partily.
do you have any idea? also web site is a corporate web site so unfortunately I cannot share it.
Option Explicit
Sub Download_Report4()
Dim obj As New Selenium.WebDriver
Dim ele As WebElement
obj.Start "Chrome"
obj.Get "https://test.com"
Dim Username, Password As Range
Set Username = ActiveWorkbook.Sheets("LoginData").Range("B1")
Set Password = ActiveWorkbook.Sheets("LoginData").Range("B2")
obj.FindElementById("UserName").SendKeys Username.Text
obj.FindElementById("Password").SendKeys Password.Text
obj.FindElementById(LoginButton").Click
Application.Wait Now + TimeValue("00:00:02")
Dim Rows, x
For x = 2 To Sheet1.UsedRange.Rows.count Step 1
If IsEmpty(Sheet1.Range("A" & x)) Then
Exit For
Else
obj.FindElementByName("date$From").Clear
obj.FindElementByName("date$To").Clear
obj.FindElementByName("date$From").SendKeys ("01,03,2017")
obj.FindElementByName("date$To").Click
obj.FindElementByName("date$To").SendKeys (Sheet1.Range("B" & x))
obj.FindElementById("btnSearch").Click
obj.FindElementById("btndownload").Click
End If
Next
End Sub
by the way, date format is correct. when it is entered as 01/03/2017 chrome add dates odd. month section to date year is correct and nothing add to month section.
I will giving little bit more details.
on below code for internet explorer is working very well unless after download button click, ie 11 prompt "do you want to open or save" question stops my macro and loops ending. this is the reason why I move the macro selenium.
Public Sub Download_Report()
Dim ie As New InternetExplorer
Dim myElem As Variant
ie.Visible = True
ie.navigate "https://test.com"
ie.document.getElementById("UserName").Focus
ie.document.getElementById("LUserName").Value = Sheets("LoginData").Range("B1").Text
ie.document.getElementById("Password").Focus
ie.document.getElementById("Password").Value = Sheets("LoginData").Range("B2").Text
ie.document.getElementById("LoginButton").Click
End If
Dim Rows, x
For x = 2 To Sheet1.UsedRange.Rows.count Step 1
If IsEmpty(Sheet1.Range("A" & x)) Then
Exit For
Else
While .Busy Or .readyState < 4: DoEvents: Wend
ie.document.getElementsByName("date$From")(0).Value = Sheet1.Range("A" & x).Text
While .Busy Or .readyState < 4: DoEvents: Wend
ie.document.getElementsByName("date$To")(0).Value = Sheet1.Range("B" & x).Text
While .Busy Or .readyState < 4: DoEvents: Wend
ie.document.getElementById("btnSearch").Click
While .Busy Or .readyState < 4: DoEvents: Wend
ie.document.getElementById("btndownload").Click
End If
Next
End With
End Sub
date can be use as picker after clicked but also can be filled as text also, date section code on web site as is below. also date to section is as same as .
Website code (website is behind a firewall)
<input name="date$From" type="text" value="18/02/2020" size="6" id="date_tbFrom" class="searchSelect">
<input type="hidden" name="date_From_Client" id="date_From_Client">
yeah there were issue that (I do not know the why but), when .click command is applied to date section cursoir was choosen the half of the section or the end so date was entered by macro is wrong or section cannot be filled that means empty. however at debugging mod everything was working smoothly. than I find another solution. I have find sections one tab before date sections. then added
obj.FindElementByName("date$From").Clear
obj.FindElementByName("date$To").Clear
obj.findelementsbyid("sectionbeforedatefrom").click
obj.sendkeys(keys.Tab)
obj.FindElementByName("date$From").SendKeys ("01,03,2017")
obj.findelementsbyid("sectionbeforedateto").click
obj.sendkeys(keys.Tab)
obj.FindElementByName("date$To").SendKeys (Sheet1.Range("B" & x))
obj.FindElementById("btnSearch").Click
those steps worked for me. but #TimWilliams was right all things are related with time. thanks everyone.
I'm quite new to VBA so please bear with me. I've been trying to create an automation to fill in username and password and login to a site (to start with) but I've been having trouble trying to click on the submit button. Scoured the internet and learnt a whole bunch of things but I didnt find anything that seems to work. The page loads and fills in the details and nothing happens when I run the code below.
Would greatly appreciate some help with this. Thanks in advance as always!
Sub worldcheck()
Dim lastrow As Long
Dim IE As Object
Dim cel As Range
Dim post As Object
Dim ws As Worksheet
Dim element As Object
Set ws = Sheets("sheet1")
Set IE = CreateObject("internetexplorer.application")
lastrow = ws.Range("B" & ws.Rows.Count).End(xlUp).Row
IE.Visible = True
IE.Navigate "https://www.world-check.com/frontend/login/"
Do While IE.busy
DoEvents
Loop
Application.Wait (Now + TimeValue("0:00:2"))
IE.document.getElementbyID("username").Value = ws.Range("D2")
IE.document.getElementbyID("password").Value = ws.Range("D3")
IE.document.getElementbyClass("button").click
Do While IE.busy
DoEvents
Loop
End Sub
Nothing else happens? You should be getting an error message at the very least as you are trying to use a non existent method (VBA Run-time error 438 Object doesn't support this property or method) . The method is getElementsByClassName - note the s indicating it returns a collection and the ending is ClassName. You would then need to index into that collection before attempting to access the Click method
As there is only a single element with classname button you can use a faster css class selector (this is also faster than using a type selector of form; likewise, you can use the faster css equivalent of getElementById for the other two DOM elements). document.querySelector stops at the first match so is also more efficient.
Finally, rather than hard coded waits use proper page load waits as shown below:
Option Explicit
Public Sub WorldCheck()
Dim ie As Object
Set ie = CreateObject("InternetExplorer.Application")
With ie
.Visible = True
.Navigate2 "https://www.world-check.com/frontend/login/"
While .busy Or .readystate <> 4: DoEvents: Wend
With .document
.querySelector("#username").Value = "ABCDEF" ' equivalent to .getElementbyID("username").Value = "ABCDEF"
.querySelector("#password").Value = "GHIJKL" '.getElementbyID("password").Value = "GHIJKL"
.querySelector(".button").Click
End With
While .busy Or .readystate <> 4: DoEvents: Wend
Stop '<== delete me later
.Quit
End With
End Sub
I want to download an Excel file from an url.
Unfortunately, I only have the url that contains the "download button", because once I click the button, the url isn't .../file.xls but one, if I go to, will not activate the download process.
Is there were a way with VBA go to this url and click on the button to download the file.
Dim IE As InternetExplorer
Sub Test()
' Create InternetExplorer Object
Set IE = CreateObject("InternetExplorer.Application")
' You can uncoment Next line To see form results
IE.Visible = True
' Send the form data To URL As POST binary request
IE.navigate "http://webpage.com/"
' Wait while IE loading...
While IE.Busy
DoEvents
Wend
Set objInputs = IE.Document.getElementsByTagName("link")
For Each ele In objInputs
If ele.Title Like "List of control sheets having one or more declared Then alert(s)." Then 'Text on the link
ele.Click
End If
Next
End Sub
However, I think the object I'm aiming at is more like a link than a button, because in the source, there are only several lines of these:
<LINK rel=stylesheet type=text/css href="/frm/wdk/theme/documentum/css/webforms.css">
you can use this to access the page using IE:
Dim ie As Object
Set ie = CreateObject("InternetExplorer.application")
With ie
.Visible = True
.Navigate ("http://yourlink")
While .Busy Or .ReadyState <> 4: DoEvents: Wend
End With
And then use the command SendKeys "{TAB}", True to go to the button and the command SendKeys "{ENTER}", True to click on it
I am trying to download a table of proprietary investments/positions/pricing from Nationwide. The code seems to do what I want, EXCEPT for producing an "object required" error when I attempt to select a particular account (click)
I thought I had the proper code to tell my macro to wait until IE was ready to go on, but clearly I am missing something.
In the code, the relevant line is highlighted. If I enter a STOP above the error line, I can wait until I "see" the link appear, then "continue" the code and it runs as expected.
Because this goes to my financial accounts, I cannot provide the user name and password to allow someone to replicate the exact problem, but here is the code, and the error message and highlight. Suggestions appreciated.
Option Explicit
'set Reference to Microsoft Internet Controls
Sub DownLoadFunds()
Dim IE As InternetExplorer
Dim sHTML
Const sURL As String = "https://www.nationwide.com/access/web/login.htm"
Const sURL2 As String = "https://isc.nwservicecenter.com/iApp/isc/app/ia/balanceDetail.do?basho.menuNodeId=12245"
Dim wsTemp As Worksheet
Set wsTemp = Worksheets("Scratch")
Set IE = New InternetExplorer
With IE
.Navigate sURL
.Visible = True 'for debugging
Do While .ReadyState <> READYSTATE_COMPLETE
DoEvents
Loop
Do While .Busy = True
DoEvents
Loop
'Login: User Name and Password "remembered" by IE
.Document.all("submitButton").Click
Do While .ReadyState <> READYSTATE_COMPLETE
DoEvents
Loop
Do While .Busy = True
DoEvents
Loop
'Select this account to show
.Document.all("RothIRA_#########").Click '<--Error at this line
Do While .ReadyState <> READYSTATE_COMPLETE
DoEvents
Loop
Do While .Busy = True
DoEvents
Loop
.Navigate sURL2
Do While .ReadyState <> READYSTATE_COMPLETE
DoEvents
Loop
Do While .Busy = True
DoEvents
Loop
Set sHTML = .Document.GetElementByID("fundByFundOnly")
With wsTemp
.Cells.Clear
.Range("a2") = sHTML.innertext
End With
.Quit
End With
Set IE = Nothing
End Sub
This is the error message:
This shows the highlighted line:
EDIT:
At Tim Williams suggestion, I added a loop to test for the presence of the desired element. This seems to work:
...
On Error Resume Next
Do
Err.Clear
DoEvents
Application.Wait (Time + TimeSerial(0, 0, 1))
.Document.getelementbyid("RothIRA_#########").Click
Loop Until Err.Number = 0
On Error GoTo 0
....
IE.Document.all("#RothIRA_....") is returning Nothing (null in more refined languages), so calling the Click method is causing the error.
Your code is the same as doing this:
Dim rothElement As Whatever
rothElement = IE.Document.all("#RothIRA_....")
rothElement.Click
...when you should do this:
Dim rothElement As Whatever
rothElement = IE.Document.all("#RothIRA_....")
If rothElement <> Nothing Then
rothElement.Click
End If
I suggest using the modern document.GetElementById method instead of the deprecated (if not obsolete) document.All API.
It's possible/likely that the page is using script to dynamically load some content or generate some layout after your "wait" loop has finished. That loop only waits until all linked content/resources have been loaded - it does not wait for scripts on the loaded page to finish, etc.
One approach is to loop your code waiting for the desired element to be rendered:
Const MAX_WAIT_SEC as Long = 5 'limit on how long to wait...
Dim t
t = Timer
Do While .Document.all("RothIRA_#########") Is Nothing
DoEvents
'or you can Sleep here
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop
'carry on...