Number of Search Results from Excel - excel

Question

Given a column of strings I would like to find the number of search results from a website (e.g. sciencedirect.com) for each string. An existing answer Number of Google Results from Excel works well for Google.
Unfortunately this solution creates the search URL from the string i.e. a Google search for example contains the word example. The sites I want to use do not do this.
A search for example returns the URL http://www.sciencedirect.com/science?_ob=ArticleListURL&_method=list&_ArticleListID=1860967815&_sort=r&_st=13&view=c&_acct=C000053194&_version=1&_urlVersion=0&_userid=1495569&md5=0ef30742e917da15236ef1824058a1db&searchtype=a
Any idea how I achived the same result with this type of search engine.

Answer 1

You'll have to examine the form you submit when you click on the Search button. The form sends a GET request which containes the search terms, but then you get redirected to the result list page, and the URL of the results page does not contain the search terms any more.
I have successfully searched for "corpus" with this URL:
http://www.sciencedirect.com/science?_ob=QuickSearchURL&_method=submitForm&_acct=C000228598&_origin=home&_zone=qSearch&md5=61ce8901b141d527683913a240486ac4&qs_all=corpus
Note that what you'll have to do is
download the start page http://www.sciencedirect.com/
extract hidden fields from the search form
assemble the search URL from the hidden fields
add your search term to the search URL in the qs_all field
send GET request with the search URL
follow redirection
Except for qs_all all other fields in this URL comes from the the form as hidden fields.
This is the source of the corresponding form, as I downloaded it (before sending the "corpus" search request.):
<form name="qkSrch" method="get" target="_top" action="/science" style="margin:0px;">
<input type="hidden" name="_ob" value="QuickSearchURL">
<input type="hidden" name="_method" value="submitForm">
<input type="hidden" name="_acct" value="C000228598">
<input type="hidden" name="_origin" value="home">
<input type="hidden" name="_zone" value="qSearch">
<input type="hidden" name="md5" value="61ce8901b141d527683913a240486ac4">
<table border="0" width="100%" cellpadding="0" cellspacing="0" style="margin: 0;">
<tbody><tr valign="middle">
<!-- Code related for toggling labels -->
<td align="right"><label for="qs_all" id="fieldLabel"> All fields</label></td>
<td align="left"><input class="textbox qsinput xpstyle" type="text" name="qs_all" id="qs_all" value="" size="30" maxlength="450" title="For example: heart attack AND behavior?" tabindex="1"></td>
<td align="right"><label for="qs_author"> Author</label></td>
<td align="left" colspan="5"><input class="textbox qsinput xpstyle" type="text" name="qs_author" id="qs_author" value="" size="33" maxlength="450" title="e.g. J S Smith or John Smith or Smith JS" tabindex="2" style="_width:100%"></td>
<td nowrap="nowrap">
</td><td></td><td></td>
<td align="right" nowrap="nowrap" width="90%" valign="middle">
Advanced search
</td>
</tr>
<tr>
<td align="right"><label for="qs_title"> Journal/Book title</label></td>
<td align="left"><input class="textbox qsinput xpstyle" type="text" id="qs_title" name="qs_title" value="" size="30" maxlength="450" title="For example: journal of molecular biology" tabindex="3"></td>
<td align="right" class="toggleQukSrch2"><label for="qs_vol" id="volField"> Volume</label></td>
<td align="left" class="toggleQukSrch"><input class="textbox qsinput xpstyle" type="text" name="qs_vol" id="qs_vol" value="" size="3" maxlength="10" style="width:30px;" tabindex="4"></td>
<td align="right" class="toggleQukSrch2"><label for="qs_issue" id="issueField"> Issue</label></td>
<td align="left" class="toggleQukSrch"><input class="textbox qsinput xpstyle" type="text" name="qs_issue" id="qs_issue" value="" size="3" maxlength="10" style="width:30px" tabindex="5"></td>
<td align="right" class="toggleQukSrch2"><label for="qs_pages" id="pageField"> Page</label></td>
<td align="right" class="toggleQukSrch"><input class="textbox qsinput xpstyle" type="text" name="qs_pages" id="qs_pages" value="" size="3" maxlength="10" title="For example: 14-27" style="width:30px" tabindex="6"></td>
<td align="right" nowrap="nowrap">
<input class="button" id="submit_search" type="Submit" alt="Submit Quick Search" title="Submit Quick Search" value="Search ScienceDirect" tabindex="8" name="sdSearch">
</td>
<td align="right" nowrap="nowrap" colspan="8" valign="bottom">
<a class="icon_qmarkHelpsci_dir" href="/science?_ob=HelpURL&_file=qs_tips.htm&_acct=C000228598&_version=1&_urlVersion=0&_userid=10&md5=2bd779305b31602341744eaa786e2f0a" target="sdhelp" onmouseover="window.status='Help is Available';return true" onmouseout="window.status='';return true" onclick="var helpWin;helpWin=window.open('/science?_ob=HelpURL&_file=qs_tips.htm&_acct=C000228598&_version=1&_urlVersion=0&_userid=10&md5=2bd779305b31602341744eaa786e2f0a','sdhelp','scrollbars=yes,resizable=yes,directories=no,toolbar=no,menubar=no,status=no,width=760,height=570');helpWin.focus();return false" tabindex="9" style="font-size:0.92em;padding-right:0;">Search tips</a>
</td>
</tr>
</tbody></table>
</form>
EDIT
Continued with How to extract the number of results from the results page.
Your assumption is right, you'll have to change other parts of the code, namely that extracts the number of results value.
Let's stick with the previous example.
When searching from "corpus", you will find this line in the source of the result page:
<input type="hidden" name="TOTAL_PAGES" value="2836">
And you'll want to extract 2836. Hence you will search for something like <input type="hidden" name="TOTAL_PAGES" value=" and get the subequent value before the closing quote.
I am not going to tell you how to code in VBA, but it is basic String manipulation so I hope you can handle it.