auto increase the date in the middle of the URL in macros - excel

I have made the macros script which retrieves the data from the URL. What I need is that, I need to increase the date one by one and get the data for each. the URL is like this :
https://www.ukdogracing.net/racecards/01-05-2017/monmore
Ia m able to get the data with this script :
Sub GetData()
Dim IE As Object
Dim doc As Object
Dim strURL As String
Dim I As Integer
For I = 1 To 5
strURL = "https://www.ukdogracing.net/racecards/01-05-2017/monmore" + Trim(Str(I))
Set IE = CreateObject("InternetExplorer.Application")
With IE
.navigate strURL
Do Until .ReadyState = 4: DoEvents: Loop
Do While .Busy: DoEvents: Loop
Set doc = IE.Document
GetAllTables doc
.Quit
End With
Next I
End Sub
Sub GetAllTables(doc As Object)
Dim ws As Worksheet
Dim rng As Range
Dim tbl As Object
Dim rw As Object
Dim cl As Object
Dim tabno As Long
Dim nextrow As Long
Dim I As Long
Dim ThisLink As Object 'variable for <a> tags
Set ws = Worksheets.Add
For Each tbl In doc.getElementsByTagName("TABLE")
tabno = tabno + 1
nextrow = nextrow + 1
Set rng = ws.Range("B" & nextrow)
rng.Offset(, -1) = "Table " & tabno
For Each rw In tbl.Rows
For Each cl In rw.Cells
rng.Value = cl.outerText
Set rng = rng.Offset(, 1)
I = I + 1
Next cl
nextrow = nextrow + 1
Set rng = rng.Offset(1, -I)
I = 0
Next rw
Next tbl
I = Range("B" & Rows.Count).End(xlUp).Row 'last row with data
Do While Cells(I, 1).Value = "" 'will loop until first not blank found in column A (starting from last row of data, from end to start)
For Each ThisLink In doc.getElementsByTagName("a") 'we check all <a> tags
If ThisLink.innerText = Cells(I, 2).Value Then Cells(I, 1).Value = ThisLink.href 'If the innertext is the name of the race, in column A we add link
Next ThisLink
I = I - 1 'we decrease row position
Loop
End Sub
But I need the script takes the date part of the URL and add one day each time till today and get the data. for example :
https://www.ukdogracing.net/racecards/01-06-2017/monmore
https://www.ukdogracing.net/racecards/01-07-2017/monmore
etc... How can I make the script to get the data for each day adding one each time.
Thanks in advance.

Replace the first sub with this one and it will run for the specified dates. I couldn't see I having any purpose so i removed it.
Sub GetData()
Dim IE As Object, doc As Object
Dim strURL As String, myDate As Date
Set IE = CreateObject("InternetExplorer.Application")
With IE
For myDate = CDate("01-05-2017") To CDate("01-09-2017")
strURL = "https://www.ukdogracing.net/racecards/" & Format(myDate, "mm-dd-yyyy") & "/monmore" ' Trim(Str(I))
.navigate strURL
Do Until .ReadyState = 4: DoEvents: Loop
Do While .Busy: DoEvents: Loop
Set doc = IE.Document
GetAllTables doc
Next myDate
.Quit
End With
End Sub

Related

How to get the latest emails and append to the existing file instead of looping through all items using VBA?

I have code that loops through all Outlook emails under a subfolder and extracts the body of the email based on the subject. Code takes a lot of time to loop through all emails as there are thousands of them.
How do I modify the code to append data, extracted from the latest emails, to the existing file instead of looping through all the emails and overwriting again & again?
Let's say I want to run the code every day to get the prior day's email data.
Option Explicit
Sub FinalMacro()
Application.DisplayAlerts = False
Dim iCounter As Integer
'iCounter = 1
Dim wkb As Workbook
Set wkb = ThisWorkbook
Sheets("Sheet1").Cells.Clear
' point to the desired email
Const strMail As String = "emailaddress#outlook.com"
Dim oApp As Outlook.Application
Dim oMapi As Outlook.MAPIFolder
'Dim oMail As Outlook.MailItem
Dim x As Long, y As Long
Dim destCell As Range
Dim i As Long
Dim oItem As Object
With ActiveSheet
Set destCell = .Cells(Rows.Count, "A").End(xlUp)
End With
On Error Resume Next
Set oApp = GetObject(, "OUTLOOK.APPLICATION")
If (oApp Is Nothing) Then Set oApp = CreateObject("OUTLOOK.APPLICATION")
On Error GoTo 0
Set oMapi = oApp.GetNamespace("MAPI").Folders(strMail).Folders("inbox").Folders("Other mails")
For Each oItem In oMapi.Items
If oItem.Subject = "Volume data" Then
' get html table from email object
Dim HTMLdoc As MSHTML.HTMLDocument
Dim tables As MSHTML.IHTMLElementCollection
Dim table As MSHTML.HTMLTable
Set HTMLdoc = New MSHTML.HTMLDocument
With HTMLdoc
.Body.innerHTML = oItem.HTMLBody
Set tables = .getElementsByTagName("table")
End With
Dim t As Long, r As Long, c As Long
Dim eRow As Long
For t = 0 To tables.Length - 1
eRow = Sheets(1).Cells(Rows.Count, 1).End(xlUp).Offset(1, 0).Row
For r = 0 To (tables(t).Rows.Length - 1)
For c = 0 To (tables(t).Rows(r).Cells.Length - 1)
Range("A" & eRow).Offset(r, c).Value = tables(t).Rows(r).Cells(c).innerText
Next c
Next r
eRow = Sheets(1).Cells(Rows.Count, 1).End(xlUp).Offset(1, 0).Row
Next t
Cells(eRow, 1) = "Date & Time of Receipt:" & " " & oItem.ReceivedTime
Cells(eRow, 1).Interior.Color = vbRed
Cells(eRow, 1).Font.Color = vbWhite
Cells(eRow, 1).Columns.AutoFit
Set oApp = Nothing
Set oMapi = Nothing
Set HTMLdoc = Nothing
Set tables = Nothing
wkb.Save '"C:\Users\Desktop\Trial_1.xlsm"
End If
Next oItem
Application.DisplayAlerts = True
End Sub
To quickly select (filter) latest emails, you can use Items.Restrict.
To use your workbook for the accumulative storage of information, you just need not to erase the sheet, but to find the last filled line and add the content from the letters after it.
Smth like (not tested):
Set oMapi = oApp.GetNamespace("MAPI").Folders(strMail).Folders("inbox").Folders("Other mails")
Set wkb = ThisWorkbook
Set append_ws = wkb.Sheets("Sheet1") ' this worksheet is for appending
'Sheets("Sheet1").Cells.Clear ' - remove this statement
' set filter to: non-flagged mailitems received < 1 day ago
flt = "[FlagStatus] <> 1 And [MessageClass]='IPM.Note' And [ReceivedTime]>='" & _
Format(Now - 1, "ddddd 0:00") & "'"
Set Restricted = oMapi.Items.Restrict(flt)
For I = Restricted.Count To 1 Step -1
Set oItem = Restricted(I)
If oItem.Subject = "Volume data" Then
content_from_email = "smth from letter" ' get the content from the letter
lastrow = append_ws.Cells(append_ws.Rows.Count, 1).End(xlUp).row + 1
append_ws.Cells(lastrow, 1).Value = content_from_email
oItem.MarkAsTask olMarkComplete ' set flag to the processed items
oItem.Save
End If
Next I

Get URL of the image that is the first result in google images

I have multiple keywords in an excel file, I am looking for a way to get the image URL of the first google image result on another cell in this excel file,
For Example
if my cell A1 contains "tomato"
I want cell A2 to display "https://seeds-gallery.com/4963-large_default/novosadski-jabucar-tomato-450-seeds.jpg" which is the image URL of the first result that shows up on Google Images
Can someone please help me out
You could do that in VBA using something like the following;
Public Sub InsertPicturesFromWeb()
Dim IE As InternetExplorer
Dim HTMLdoc As HTMLDocument
Dim imgElements As IHTMLElementCollection
Dim imgElement As HTMLImg
Dim aElement As HTMLAnchorElement
Dim N As Integer, I As Integer
Dim Url As String, Url2 As String
Dim LastRow As Long
Dim M, sImageSearchString
LastRow = Range("A" & Rows.Count).End(xlUp).Row
For I = 1 To LastRow
Url = "https://www.google.co.in/search?q=" & Cells(I, 1) & "&source=lnms&tbm=isch&sa=X&rnd=1"
Set IE = New InternetExplorer
With IE
.Visible = False
.Navigate Url
Do Until .readyState = 4: DoEvents: Loop
Set HTMLdoc = .document
Set imgElements = HTMLdoc.getElementsByTagName("IMG")
N = 1
For Each imgElement In imgElements
If InStr(imgElement.src, sImageSearchString) Then
If imgElement.ParentNode.nodeName = "A" Then
Set aElement = imgElement.ParentNode
Url2 = imgElement.src
N = N + 1
End If
End If
Next
Call GetShapeFromWeb(Url2, Cells(I, 2))
IE.Quit
Set IE = Nothing
End With
Next I
End Sub
Sub GetShapeFromWeb(strShpUrl As String, rngTarget As Range)
With rngTarget.Parent
.Pictures.Insert strShpUrl
.Shapes(.Shapes.Count).Left = rngTarget.Left
.Shapes(.Shapes.Count).Top = rngTarget.Top
End With
End Sub

Transform InternetExplorer to XML

I just discovered xml or xmlhttp and this is entirely new to me.
I am trying to create a macro wherein it would go through all the list of websites in column J starting at row 2 (header at row 1). Get the information that I want from each website, then display them in column K, which is right next to the websites where the information was taken from.
Column J has a list of websites, starting at J2. Let's say it would go all the way down to J10. From each website, there is a certain information I want to get, so the macro will visit the website at J2, get that information and paste it in K2, then visit the website in J3, paste that information in K3, and so on. I already have an existing list of website at column J, which also happens to be dynamic.
This is the current code that I have using IE that I want to convert into xml/xmlhttp.
Sub CommandButton1_Click()
Dim ie As Object
Dim lastrow As Integer
Dim i As Integer
Dim myURL As String
Dim sdd As String
Dim add As Variant
Dim html As Object
Dim mylinks As Object
Dim mylink As Object
Dim result As String
' Create InternetExplorer Object
Set ie = CreateObject("InternetExplorer.Application")
lastrow = Sheet1.Cells(Rows.Count, "J").End(xlUp).Row
For i = 2 To lastrow
myURL = Sheet1.Cells(i, "J").Value
' Hide InternetExplorer
ie.Visible = False
' URL to get data from
ie.navigate myURL
' Loop until page fully loads
Do While ie.readystate <> READYSTATE_COMPLETE
Loop
' Information i want to get from the URLs
sdd = ie.document.getelementsbyclassname("timeline-text")(0).innerText
' Format the result
add = Split(sdd, "$")
Range("K3") = add(1)
' Close InternetExplorer
ie.Quit
'Return to Normal?
ie.Visible = True
End
Next
' Clean up
Set ie = Nothing
Application.StatusBar = ""
End Sub
I am trying to get the "85100", not the $85,100
<span class="font-size-base font-normal">Est.</span>
<span itemprop="price" content="85100">
$85,100
</span>
I'm hoping you could help me with this problem.
Thank you in advance.
I would structure something like as follows, where the IE object is created outside the loop. You use css selectors throughout. You may need a timed loop to ensure element is present on page. Use a proper page load wait as shown.
Use an explicit worksheet name to put the worksheet in a variable to work with.
You might want to add a test that myURL has http/https in it as you may have blank cells in range and only want to work with likely urls values.
Option Explicit
Public Sub CommandButton1_Click()
Dim ie As Object, lastrow As Long, i As Long
Dim myURL As String, sdd As String, ws As Worksheet
Set ws = ThisWorkbook.Worksheets("Sheet1") ' <change as required
Set ie = CreateObject("InternetExplorer.Application")
lastrow = ws.Cells(Rows.Count, "J").End(xlUp).Row
With ie
.Visible = False
For i = 2 To lastrow
myURL = ws.Cells(i, "J").Value
.navigate2 myURL
While .Busy Or .readyState < 4: DoEvents: Wend
sdd = .document.querySelector(".price").getAttribute("content")
ws.Cells(i, "K") = sdd
Next
.Quit
End With
'Application.StatusBar = ""
End Sub
With a timed loop:
Public Sub CommandButton1_Click()
Dim ie As Object, lastrow As Long, i As Long, t As Date, ele As Object
Const MAX_WAIT_SEC As Long = 10
Dim myURL As String, sdd As String, ws As Worksheet
Set ws = ThisWorkbook.Worksheets("Sheet1") ' <change as required
Set ie = CreateObject("InternetExplorer.Application")
lastrow = ws.Cells(rows.Count, "J").End(xlUp).Row
With ie
.Visible = False
For i = 2 To lastrow
myURL = ws.Cells(i, "J").Value
.Navigate2 myURL
While .Busy Or .readyState < 4: DoEvents: Wend
t = Timer
Do
On Error Resume Next
Set ele = HTMLDoc.querySelector(".price")
On Error GoTo 0
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While ele Is Nothing
If Not ele Is Nothing Then
sdd = ele.getAttribute("content")
ws.Cells(i, "K") = sdd
End If
Next
.Quit
End With
'Application.StatusBar = vbnullstring
End Sub

VBA HTML Listing Info Pull

I am looking to follow a series of URL's that are found in column A (example: https://www.ebay.com/itm/Apple-iPhone-7-GSM-Unlocked-Verizon-AT-T-TMobile-Sprint-32GB-128GB-256GB/352381131997?epid=225303158&hash=item520b8d5cdd:m:mWgYDe4a79NeLuAlV-RmAQA:rk:7:pf:0) and pull the following information from them:
- Title
- Price
- Description
I think there are multiple issues with my code... For one, I can't get the program to follow specific URL's listed in the Excel (only if I specify one within the code). Also, pulling multiple fields has given me issues.
Option Explicit
Public Sub ListingInfo()
Dim ie As New InternetExplorer, ws As Worksheet, t As Date
Dim i As Integer
i = 0
Do While Worksheets("Sheet1").Cells(i, 1).Value <> ""
Const MAX_WAIT_SEC As Long = 10
Set ws = ThisWorkbook.Worksheets("Sheet1")
With ie
.Visible = True
.Navigate2 Worksheets("Sheet1").Cells(i, 1).Value
While .Busy Or .readyState < 4: DoEvents: Wend
Dim Links As Object, i As Long, count As Long
t = Timer
Do
On Error Resume Next
Set Title = .document.querySelectorAll("it-ttl")
Set price = .document.querySelectorAll("notranslate")
Set Description = .document.querySelectorAll("ds_div")
count = Links.Length
On Error GoTo 0
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While count = 0
For i = 0 To Title.Length - 1
ws.Cells(i + 1, 1) = Title.item(i)
ws.Cells(i + 1, 2) = price.item(i)
ws.Cells(i + 1, 3) = Description.item(i)
Next
.Quit
i = i + 1
Loop
End With
End Sub
I would use late binding for MSXML2.XMLHTTP and set a reference to the Microsoft HTML Object Library for the HTMLDocument.
Note: querySelector() references the first item it finds that matches its search string.
Here is the short version:
Public Sub ListingInfo()
Dim cell As Range
With ThisWorkbook.Worksheets("Sheet1")
For Each cell In .Range("A1", .Cells(.Rows.Count, 1).End(xlUp))
Dim Document As MSHTML.HTMLDocument
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", cell.Value, False
.send
Set Document = New MSHTML.HTMLDocument
Document.body.innerHTML = .responseText
End With
cell.Offset(0, 1).Value = Document.getElementByID("itemTitle").innerText
cell.Offset(0, 2).Value = Document.getElementByID("prcIsum").innerText
If Not Document.querySelector(".viSNotesCnt") Is Nothing Then
cell.Offset(0, 3).Value = Document.querySelector(".viSNotesCnt").innerText
Else
'Try Something Else
End If
Next
End With
End Sub
A more elaborate solution would be to break the code up into smaller routines and load the data into an Array. The main advantage of this is that you can test each subroutine separately.
Option Explicit
Public Type tListingInfo
Description As String
Price As Currency
Title As String
End Type
Public Sub ListingInfo()
Dim source As Range
Dim data As Variant
With ThisWorkbook.Worksheets("Sheet1")
Set source = .Range("A1:D1", .Cells(.Rows.count, 1).End(xlUp))
data = source.Value
End With
Dim r As Long
Dim record As tListingInfo
Dim url As String
For r = 1 To UBound(data)
record = getListingInfo()
url = data(r, 1)
record = getListingInfo(url)
With record
data(r, 2) = .Description
data(r, 3) = .Price
data(r, 4) = .Title
End With
Next
source.Value = data
End Sub
Public Function getListingInfo(url As String) As tListingInfo
Dim ListingInfo As tListingInfo
Dim Document As MSHTML.HTMLDocument
Set Document = getHTMLDocument(url)
With ListingInfo
.Description = Document.getElementByID("itemTitle").innerText
.Price = Split(Document.getElementByID("prcIsum").innerText)(1)
.Title = Document.querySelectorAll(".viSNotesCnt")(0).innerText
Debug.Print .Description, .Price, .Title
End With
End Function
Public Function getHTMLDocument(url As String) As MSHTML.HTMLDocument
Const READYSTATE_COMPLETE As Long = 4
Dim Document As MSHTML.HTMLDocument
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", url, False
.send
If .readyState = READYSTATE_COMPLETE And .Status = 200 Then
Set Document = New MSHTML.HTMLDocument
Document.body.innerHTML = .responseText
Set getHTMLDocument = Document
Else
MsgBox "URL: " & vbCrLf & "Ready state: " & .readyState & vbCrLf & "HTTP request status: " & .Status, vbInformation, "URL Not Responding"
End If
End With
End Function
There are a lot of things to fix in your code. It is late here so I will just give pointers (and update fully later) and working code below:
Declare all variables and use appropriate type
Review For Loops and how transpose can be used to create a 1d array of urls pulled from sheet to loop over
Review the difference between querySelector and querySelectorAll methods
Review CSS selectors (you are specifying everything as type selector when in fact you are not selecting by tag for the elements of interest; nor by your stated text)
Think about placement of your IE object creation and of your .Navigate2 to make use of existing object
Make sure to use distinct loop counters
Be sure not to overwrite values in sheet
Code:
Option Explicit
Public Sub ListingInfo()
Dim ie As New InternetExplorer, ws As Worksheet
Dim i As Long, urls(), rowCounter As Long
Dim title As Object, price As Object, description As Object
Set ws = ThisWorkbook.Worksheets("Sheet1")
urls = Application.Transpose(ws.Range("A1:A2").Value) '<= Adjust
With ie
.Visible = True
For i = LBound(urls) To UBound(urls)
If InStr(urls(i), "http") > 0 Then
rowCounter = rowCounter + 1
.Navigate2 urls(i)
While .Busy Or .readyState < 4: DoEvents: Wend
Set title = .document.querySelector(".it-ttl")
Set price = .document.querySelector("#prcIsum")
Set description = .document.querySelector("#viTabs_0_is")
ws.Cells(rowCounter, 3) = title.innerText
ws.Cells(rowCounter, 4) = price.innerText
ws.Cells(rowCounter, 5) = description.innerText
Set title = Nothing: Set price = Nothing: Set description = Nothing
End If
Next
.Quit
End With
End Sub
Here's an approach using Web Requests, using MSXML. It should be significantly faster than using IE, and I'd encourage you to strongly consider using this approach wherever possible.
You'll need references to Microsoft HTML Object Library and Microsoft XML v6.0 to get this working.
Option Explicit
Public Sub SubmitRequest()
Dim URLs As Excel.Range
Dim URL As Excel.Range
Dim LastRow As Long
Dim wb As Excel.Workbook: Set wb = ThisWorkbook
Dim ws As Excel.Worksheet: Set ws = wb.Worksheets(1)
Dim ListingDetail As Variant
Dim i As Long
Dim j As Long
Dim html As HTMLDocument
ReDim ListingDetail(0 To 2, 0 To 10000)
'Get URLs
With ws
LastRow = .Cells(.Rows.Count, 1).End(xlUp).Row
Set URLs = .Range(.Cells(1, 1), .Cells(LastRow, 1))
End With
'Update the ListingDetail
For Each URL In URLs
Set html = getHTML(URL.Value2)
ListingDetail(0, i) = html.getElementByID("itemTitle").innertext 'Title
ListingDetail(1, i) = html.getElementByID("prcIsum").innertext 'Price
ListingDetail(2, i) = html.getElementsByClassName("viSNotesCnt")(0).innertext 'Seller Notes
i = i + 1
Next
'Resize array
ReDim Preserve ListingDetail(0 To 2, 0 To i - 1)
'Dump in Column T,U,V of existing sheet
ws.Range("T1:V" & i).Value = WorksheetFunction.Transpose(ListingDetail)
End Sub
Private Function getHTML(ByVal URL As String) As HTMLDocument
'Add a reference to Microsoft HTML Object Library
Set getHTML = New HTMLDocument
With New MSXML2.XMLHTTP60
.Open "GET", URL
.send
getHTML.body.innerHTML = .responseText
End With
End Function

Use a loop to auto search

I made a code that can use the cell in A2 to search it in a specific website and pull down the criteria that I need. I'm having trouble making the correct loop for this. Can someone point me in the right direction? (cells from A2 to A1829)
Code:
Sub SearchBot()
Dim objIE As InternetExplorer
Dim aEle As HTMLLinkElement
Dim y As Integer
Dim result As String
Dim TR As Object, TD As Object
Dim tbl As Object, obj_tbl As Object
Dim lastRow As Long
Dim start_time As Date
Dim end_time As Date
Set objIE = New InternetExplorer
objIE.Visible = True 'make IE browser visible
objIE.navigate "https://capitaliq.com"
Do While objIE.Busy = True Or objIE.readyState <> 4: DoEvents: Loop
For i = 1 to 1829
objIE.document.getElementById("SearchTopBar").Value = _
Sheets("Sheet2").Range("A" & i).Value
Set oNode = objIE.document.getElementsByClassName("iPadHack tmbsearchright")(0)
oNode.Click
Do While objIE.Busy = True Or objIE.readyState <> 4: DoEvents: Loop
b = 2
Dim tblEle As Object
Set tblEle = objIE.document.getElementsByClassName("cTblListBody")(5)
Sheets("Sheet2").Range("B" & b).Value = tblEle.innerText
Debug.Print tblEle.innerText
b = b + 1
Next i
objIE.Quit
End Sub
you're not using the loop variable i anywhere, guessing it should be this line:
Sheets("Sheet2").Range("A").Value
change to:
Sheets("Sheet2").Range("A" & i).Value

Resources