I am a self-taught, amateur programmer, and I am new to this forum. Please bear with me.
About two years ago, I wrote a simple Excel vba program to login in to a website and grab a customer statement in the form of a .csv file. My program utilizes GET and POST requests. This program worked perfectly (for my needs) until about three weeks ago, when it unfortunately broke on me. The program could not get through the initial GET request. Specifically, it would break on the getReq.send line.
I came across this post:
Login into website using MSXML2.XMLHTTP instead of InternetExplorer.Application with VBA
Here, I learned that you can use "Msxml2.XMLHTTP.6.0" instead of "Msxml2.ServerXMLHTTP.6.0". I modified my code accordingly, eliminating the need to parse cookies after the Get request, and it worked! But I have no idea. Even though I got it to work, I do not feel like I have learned much in the process.
Some information to note:
My original program broke on my work computer (WindowsXP).
Figuring that it may be an XP issue, and in the market for a new machine anyway, I updated to a new computer running Windows7. The program still did not work, though I received a different error message.
I ran my code on a Windows10 computer and it worked fine.
I use identical code to connect to various other websites and it works fine, regardless of what operating system.
So, my specific questions:
Why might the code work with Msxml2.XMLHTTP.6.0 but not Msxml2.ServerXMLHTTP.6.0?
And why might the code have broken in the first place?
Why would the code work on one particular website, but no another?
Any insight would be greatly appreciated. I have attached my code (with login info X'd out).
Sub RCGInquiry()
Dim postReq, getReq, cookies
Dim p0 As Integer, p1 As Integer, temp As String
Dim result As String, respHead As String
Set getReq = CreateObject("Msxml2.ServerXMLHTTP.6.0")
'Set getReq = CreateObject("Msxml2.XMLHTTP.6.0")
' Visit homepage so we can find the cookies
getReq.Open "GET", "https://www.rcginquiry.com/sfs/Entry", False
getReq.send
respHead = getReq.getAllResponseHeaders
Debug.Print respHead
' Need to parse the cookie from Respone Headers
cookies = ""
p0 = 1
Do While InStr(p0, respHead, "Set-Cookie:") > 0
p0 = InStr(p0, respHead, "Set-Cookie:") + 11
p1 = InStr(p0, respHead, Chr(10))
temp = Trim(Mid(respHead, p0, p1 - p0))
cookies = cookies & temp & "; "
Loop
cookies = Left(cookies, Len(cookies) - 2)
' Debug.Print cookies
' Login
Set postReq = CreateObject("Msxml2.ServerXMLHTTP.6.0")
'Set postReq = CreateObject("Msxml2.XMLHTTP.6.0")
postReq.Open "POST", "https://www.rcginquiry.com/sfs/Entry", False
postReq.setRequestHeader "Cookie", cookies
postReq.setRequestHeader "Content-type", "application/x-www-form-urlencoded" 'send appropriate Headers
postReq.send "Usrid=XXXX&Psswd=XXXX" ' send login info
'-------------------------------------------------------------------------------
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Dim FSO As Object
Dim myFile As Object
Dim path As String
Dim y As Integer
curDate = Format(Date, "mm_dd_yy")
' Download CSV
postReq.Open "POST", "https://www.rcginquiry.com/sfs/Downloads/tmp.csv?filetype=POS&format=MFA20&heading=true&allaccts=true&junk=tmp.csv", False
postReq.setRequestHeader "Cookie", cookies 'must resend cookies so it knows i am logged in
postReq.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
postReq.send "filetype=POS&format=MFA20&heading=true&allaccts=true&junk=temp.csv" 'url query parameters
' Writes responseText to a .csv file
Set FSO = CreateObject("Scripting.FileSystemObject")
Set myFile = FSO.createtextfile("C:\Users\Adam\Desktop\POSITION\" & curDate & ".csv", True)
myFile.write (postReq.responseText)
myFile.Close
Set FSO = Nothing
Set myFile = Nothing
End Sub
This blog post says that ServerXLMHTTP will not work through a proxy on the client, but XMLHTTP will. In Excel VBA I was using ServerXLMHTTP.6.0, and it was failing for some clients inside corporate networks, whereas XMLHTTP worked.
Welcome to VBA and StackOverflow. Your notes are thorough and so the only thing I can suggest is that you check your proxy settings.
https://support.microsoft.com/en-us/help/289481/you-may-need-to-run-the-proxycfg-tool-for-serverxmlhttp-to-work
That link was buried in this link
https://support.microsoft.com/en-us/help/290761/frequently-asked-questions-about-serverxmlhttp
which you were referred to by ComIntern
Related
I have a list of Twitter urls in Column A, for which I am trying to pull some information off, however I am having a lot of trouble. I want to pull off everything in yellow
I am not sure if it is due to having the wrong classes or due to the Twitter Urls NOT opening in excel. If I double click a url in excel and try to open it I get this error message.
The link works fine when I copy and paste them into the browser. I have read some information on the web that states that a HKEY on the PC may need changing LINK. The problem I have the person I am building this for is not pc literate and will struggle, to do any fix.
I have always used the below code for scraping and it has never failed me. When it does pull data off Twitter, I get an error message, see image below columns D + E. I am assuming this is making some contact to Twitter but can not access the page to extract the data. I am NOT using IE as it no longer works with twitter, I am using a MSXML2.ServerXMLHTTP.
This is what i am using to extract the data, it is the same for all the columns, just the class changes and if it is a Span or a child.
''''Element 3 Column D
If doc.getElementsByClassName("css-1dbjc4n")(0) Is Nothing Then
wsSheet.Cells(StartRow + myCounter, 4).Value = "-"
Else
wsSheet.Cells(StartRow + myCounter, 4).Value = doc.getElementsByClassName("css-1dbjc4n")(0).getElementsByTagName("Span")(0).innerText
End If
Public Function NewHTMLDocument(strURL As String) As Object
Dim objHTTP As Object, objHTML As Object, strTemp As String
Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
objHTTP.setOption(2) = 13056
objHTTP.Open "GET", strURL, False
objHTTP.send
If objHTTP.Status = 200 Then
strTemp = objHTTP.responseText
Set objHTML = CreateObject("htmlfile")
objHTML.body.innerHTML = strTemp
Set NewHTMLDocument = objHTML
Else
'There has been an error
End If
End Function
QUESTION
Is the problem due to the urls not opening in excel, or is it because the data is dynamic and it can not be extracted?
Twitter Link 1
Twitter Link 2
As always thanks for having a look and my apologies in advance for NOT adding HTML snippet as it would not let me post, I could not find the error so removed the html, it was stating that a URL had been shortened, but could not find it so removed the whole html snippet in order to post.
UPDATE
I thought this link was in my post, but I must have removed it when I removed the HTML Snippet. I found this on Stackoverflow but could not get it to work form me, nothing would extract Link
I'am new to vba in excel. I managed to write a code which scrapes data from a given website and stores it in an excel worksheet. The code works almost every time i run it but sometimes i get an error:
Object variable or With block variable not set.
So it is very challenging to find out why. Also if you could help me out speeding the code (maybe not using clipboard to pastspecial the table, but I don't know how to use otherwise...). Also for you to know, once the error is promted if I click end and run the sub again, it runs without any problem. The error is promted (sometimes only, most of the time the sub works fine) in the specified line with this comment: 'This is the line which throws the error. I appreciate any kind of help guys, thank you in advance :).
The sub looks like this:
Sub PaData()
Dim c As Object, D As Object, H As Object, PID$, SD As Date, FC$, cf$
Set c = CreateObject("New:{1C3B4210-F441-11CE-B9EA-00AA006B1A69}")
Set D = CreateObject("HTMLFile")
Set H = CreateObject("WinHTTP.WinHTTPRequest.5.1")
FC = "EXA" ' This is used to generate the website url
cf = VBMa ' This is another sub which works fine and i need it to get into the webiste
' Get the page
H.SetAutoLogonPolicy 0
H.SetClientCertificate "CURRENT_USER\MY\" & Environ("USERNAME")
H.Open "GET", "https://confidentialwebsite=" & FC
H.setRequestHeader "Cookie", cf
H.Send
H.waitForResponse
' Put the response into the HTML object
D.body.innerHTML = H.ResponseText
' Copy _only a given Table
c.setText D.getElementByID("giventable").outerHTML 'This is the line which throws the error
c.PutInClipBoard
' Paste into the sheet, remove hyperlinks and unMerge all data
Sheets("Pdata").Cells.Delete
Sheets("Pdata").[A1].PasteSpecial
Sheets("Pdata").Cells.Hyperlinks.Delete
Sheets("Pdata").Cells.UnMerge
'update time
Sheets("SM").Range("B1").Value = Sheets("Pdata").Range("D2").Value + 2 / 24
End Sub
When doing an HTTPRequest to a webserver, you should always verify the return status of this call using .Status (see: this )
An overview of the possible status numbers can be found here: https://httpstatuses.com/ or here: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#1xx_Informational_response
I am trying to create a VBA to download a google sheet into excel so I can compile stock market data daily. I would simply use power query for this but I am doing this on my personal laptop which is a mac and does not support power query. I am relatively new to coding so have been leaning on following online instructions. The instruction includes this:
Set objWebCon = CreateObject("MSXML2.XMLHTTP.3.0")
This line when ran creates an error message saying:
"
Run-time error '429':
ActiveX component can't create object
"
I think the issue lies within the fact that the instruction is based on a windows operating system. Any solution I've searched for is specific to windows operating systems.
Does anybody here know if I can change the "MSXML2.XMLHTTP.3.0" part of my code to fit it better to mac? Not sure if this is what needs to be done but any guidance would be super appreciated.
I attached my full code below but feel free to ignore it if not relavent. Thank you!!
Sub DownloadGoogleSheets()
Dim ShtUrl As String, Location As String, FileName As String
Dim objWebCon, objWrit As Object
'Sheet Url
ShtUrl = "https://docs.google.com/spreadsheets/d/1wpA_epxtlz96sxETqKttJwsy9Aubb15H8xslcSQ20T0/export?format=csv&id=1wpA_epxtlz96sxETqKttJwsy9Aubb15H8xslcSQ20T0" & gid = 1319327791
'Location
Location = ThisWorkbook.Path & "\" '/Users/[myName]/Desktop/Stock Analysis/n"
'FileName
FileName = "GoogleSheet.csv"
'Connection to Website
Set objWebCon = CreateObject("MSXML2.XMLHTTP.3.0")
'Writer
Set objWrit = CreateObject("ADODB.Stream")
'Connecting to the Website
objWebCon.Open "Get", ShtUrl, False
objWebCon.Send (ShtUrl)
'Once page is fully loaded
If objWebCon.Status = 200 Then
'Write the text of the sheet
objWrit.Open
objWrit.Type = 1
objWrit.Write objWebCon.ResponseBody
objWrit.Position = 0
objWrit.SaveToFile Location & FileName
objWrit.Close
End If
Set objWebCon = Nothing
Set objWrit = Nothing
End Sub
I copied code to get stock data from hsbc derivatives. (https://www.youtube.com/watch?v=IOzHacoP-u4)
I changed the URL (to hsbc) and that I want to find the value based on the ID, not the class name.
I changed the ID name.
I get
"Run Time Error-91:
Object variable or With block variable not set".
Sub Get_Web_Data()
Dim request As Object
Dim response As String
Dim html As New HTMLDocument
Dim website As String
Dim price As Variant
' Website to go to.
website = "https://www.hsbc-zertifikate.de/home/details#!/isin:DE000TR8S293"
' Create the object that will make the webpage request.
Set request = CreateObject("MSXML2.XMLHTTP")
' Where to go and how to go there - probably don't need to change this.
request.Open "GET", website, False
' Get fresh data.
'request.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
' Send the request for the webpage.
request.send
' Get the webpage response data into a variable.
response = StrConv(request.responseBody, vbUnicode)
' Put the webpage into an html object to make data references easier.
html.body.innerHTML = response
' Get the price from the specified element on the page.
price = html.getElementById("kursdaten20").innerText
' Output the price into a message box.
MsgBox price
End Sub
You are searching for element id kursdaten20 that does not exist on the page.
html.getElementById("kursdaten20") returns Nothing and you are accessing the innerText property with Nothing/Null reference.
When searching for element, you could add a check if the element exists:
'query the document
Set element = html.getElementById("kursdaten20")
If Not element Is Nothing Then
' Get the price from the specified element on the page.
price = element.innerText
' Output the price into a message box.
MsgBox price
Else
' no price
MsgBox "no price"
End If
I'm afraid it's more complicated than what you expected it to be.
I will assume that the info you're after is this:
Geldkurs (1 Stuck)4,01 EUR
Briefkurs (1 Stuck)4,11 EUR
These fields are not static. They are dynamically updated (I guess whenever a transaction is made) by scripts. That's why you will not find their ID's in the source code of the HTML page.
There is however a way to get the info you need by replicating the HTTP request that is being sent to the server whenever these fields are updated.
To find this request and its parameters you need to inspect the network traffic, when you load the page, using your browser's developer tools.
This request returns a (quite poorly structured IMHO) JSON response containing another JSON (!!) which contains the info you want, in HTML format(!!). Here's how the second JSON looks like:
To make things even worse, the names that you can see under state, change with each request you send.
So, firstly you need to parse the json response. Then you need to parse the json within the initial json response to get your hands on the HTML code. Then, using an HTML document object, you can easily get access to the HTML table, containing the desired information.
Here's the way to do it:
Option Explicit
Sub hsbc()
Dim req As New WinHttpRequest
Dim doc As New HTMLDocument
Dim table As HTMLTable
Dim cell As HTMLTableCell
Dim parsedJSON As Object
Dim key As Variant
Dim htmlCode As String
Dim url As String, reqBody As String, resp As String
url = "https://www.hsbc-zertifikate.de/web-htde-tip-zertifikate-main/?components=YW1wZWw6UnRQdWxsQ29tcG9uZW50KCdhbmltQ3NzLGMtaGlnaGxpZ2h0LXVwLGMtaGlnaGxpZ2h0LWRvd24sYy1oaWdobGlnaHQtY2hhbmdlZCcpO3NlYXJjaGhpbnRfbW9iaWxlOlNlYXJjaEhpbnRNb2JpbGVDb21wb25lbnQoJ3VsU2VhcmNoU21hbGwvc2VhcmNoSW5wdXRNb2JpbGUnKTtzZWFyY2hoaW50OlNlYXJjaEhpbnRDb21wb25lbnQoJ3VsU2VhcmNoRnVsbC9zZWFyY2gtaGVhZGVyJyk7aXNpbjpSZXNwb25zaXZlU25hcHNob3RDb21wb25lbnQoJ2ZhbHNlJyk%3D&pagepath=https%3A%2F%2Fwww.hsbc-zertifikate.de%2Fhome%2Fdetails%23!%2Fisin%3ADE000TR8S293&magnoliaSessionId=B22F70D76986AB6BACDF110E4E7A724C.public7a&v-1566551332455"
reqBody = "v-browserDetails=1&theme=hsbc&v-appId=myApp&v-sh=1080&v-sw=1920&v-cw=1920&v-ch=550&v-curdate=1566551332455&v-tzo=-180&v-dstd=60&v-rtzo=-120&v-dston=true&v-vw=50&v-vh=50&v-loc=https%3A%2F%2Fwww.hsbc-zertifikate.de%2Fhome%2Fdetails%23!%2Fisin%3ADE000TR8S293&v-wn=myApp-0.5436432044490654"
With req
.Open "POST", url, False
.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
.send reqBody
resp = .responseText
End With
Set parsedJSON = JsonConverter.ParseJson(resp)
Set parsedJSON = JsonConverter.ParseJson(parsedJSON("uidl"))
For Each key In parsedJSON("state").Keys
If parsedJSON("state")(key)("contentMode") = "HTML" Then
htmlCode = htmlCode & parsedJSON("state")(key)("text")
End If
Next key
doc.body.innerHTML = htmlCode
Set table = doc.getElementsByTagName("table")(0)
Debug.Print table.Rows(2).innerText
Debug.Print table.Rows(3).innerText
End Sub
For demonstration purposes the result will be printed in your immediate window.
You will need to add the following references to your project (VBE>Tools>References):
Microsoft WinHTTP Services version 5.1
Microsoft HTML Objects Library
Microsoft Scripting Runtime
You will also need to add this JSON parser to your project. Follow the installation instructions in the link and you should be set to go.
i am trying to get past the simple log template in with DocuSign. Can anyone kindly tell me what is missing in the VBA code. Is there quotes or anything I need to make this work. I added some quotes around my email address. I have a sandbox account and my developer key, but i get a 401 error whether i send execute the "GET" or not. i did take this code from another tread in stackoverflow but i dont know what was in the excel cells to make this code work.
Public Sub APICallTest()
Dim httpRequest As MSXML2.XMLHTTP60
Dim httpResult As MSXML2.DOMDocument60
' defined request and result variables
Set httpRequest = New XMLHTTP60
Set httpResult = New DOMDocument60
'open login information url https://demo.docusign.net/restapi/v2
httpRequest.Open "GET", "https://demo.docusign.net/restapi/v2/login_information.XML", False
httpRequest.setRequestHeader "X-DocuSign-Authentication: <DocuSignCredentials><Username>MyUserName</Username><Password>" + Chr(34) + "my#myemail.com" + Chr(34)</Password><IntegratorKey>myintegratorkey</IntegratorKey></DocuSignCredentials>Accept: application/xml Content-Type: application/xml", "text"
' send login information request
httpRequest.send
Debug.Print httpRequest.Status, "A"
Debug.Print httpRequest.statusText, "B"
Debug.Print httpRequest.responseText, "C"
Exit Sub
The error message is telling you what the problem is- your Integrator Key is either not present in the request or is invalid (i.e. incorrect). I see your VB code that shows you apparently including the key in the header, however since you haven't posted the raw request you're sending out my guess is that your code is not working properly and the key or the header is not being set properly.
Try doing this:
Login to your demo account, go into your preference, and enable the Request Logging feature. (more on how to do that below)
Run your code.
Go back into your account preferences and retrieve the log that would have been created.
Inspect the log and make sure your Integrator Key and X-DocuSign-Authentication header are present and correct.
For a more complete guide on how to enable request logging in your account see here:
https://support.docusign.com/guides/ndse-user-guide-api-request-logging
Or here if you're using the Classic DocuSIgn UI:
https://support.docusign.com/articles/API-Request-Logging
I suspect that the issue is caused because you are trying to set more than one header in the httpRequest.setRequestHeader statement.
I recommend splitting them up to seperate statements
httpRequest.setRequestHeader("X-DocuSign-Authentication","<DocuSignCredentials><Username>MyUserName</Username><Password>" + Chr(34) + "my#myemail.com" + Chr(34)+"</Password><IntegratorKey>myintegratorkey</IntegratorKey></DocuSignCredentials>")
httpRequest.setRequestHeader("Accept","application/xml");
httpRequest.setRequestHeader("Content-Type","application/xml");
https://msdn.microsoft.com/en-us/library/ms766589(v=vs.85).aspx
Hope this proves helpful.
It works now!! this is what it ended up looking in VBA. How does someone know to put "application/xml" on the end of the Header definition?
Public Sub APICallTest()
Dim httpRequest As MSXML2.XMLHTTP60
Dim httpResult As MSXML2.DOMDocument60
'defined request and result variables
Set httpRequest = New XMLHTTP60
Set httpResult = New DOMDocument60
'open login information url https://demo.docusign.net/restapi/v2
httpRequest.Open "GET", "https://demo.docusign.net/restapi/v2/login_information"
httpRequest.setRequestHeader "X-DocuSign-Authentication", "<DocuSignCredentials><Username>my#email.com</Username><Password>mypassword</Password><IntegratorKey>mykey</IntegratorKey></DocuSignCredentials>"
httpRequest.setRequestHeader "Accept", "application/xml"
httpRequest.setRequestHeader "Content-Type", "application/xml"
httpRequest.send
Debug.Print httpRequest.Status, "A"
Debug.Print httpRequest.statusText, "B"
Debug.Print httpRequest.responseText, "C"
Set httpRequest = Nothing
Set httpResult = Nothing
Exit Sub