I would like to create a program that will enter a string into the text box on a site like Google (without using their public API) and then submit the form and grab the results. Is this possible? Grabbing the results will require the use of HTML scraping I would assume, but how would I enter data into the text field and submit the form? Would I be forced to use a public API? Is something like this just not feasible? Would I have to figure out query strings/parameters?
Thanks
Theory
What I would do is create a little program that can automatically submit any form data to any place and come back with the results. This is easy to do in Java with HTTPUnit. The task goes like this:
Connect to the web server.
Parse the page.
Get the first form on the page.
Fill in the form data.
Submit the form.
Read (and parse) the results.
The solution you pick will depend on a variety of factors, including:
Whether you need to emulate JavaScript
What you need to do with the data afterwards
What languages with which you are proficient
Application speed (is this for one query or 100,000?)
How soon the application needs to be working
Is it a one off, or will it have to be maintained?
For example, you could try the following applications to submit the data for you:
Lynx
curl
wget
Then grep (awk, or sed) the resulting web page(s).
Another trick when screen scraping is to download a sample HTML file and parse it manually in vi (or VIM). Save the keystrokes to a file and then whenever you run the query, apply those keystrokes to the resulting web page(s) to extract the data. This solution is not maintainable, nor 100% reliable (but screen scraping from a website seldom is). It works and is fast.
Example
A semi-generic Java class to submit website forms (specifically dealing with logging into a website) is below, in the hopes that it might be useful. Do not use it for evil.
import java.io.FileInputStream;
import java.util.Enumeration;
import java.util.Hashtable;
import java.util.Properties;
import com.meterware.httpunit.GetMethodWebRequest;
import com.meterware.httpunit.SubmitButton;
import com.meterware.httpunit.WebClient;
import com.meterware.httpunit.WebConversation;
import com.meterware.httpunit.WebForm;
import com.meterware.httpunit.WebLink;
import com.meterware.httpunit.WebRequest;
import com.meterware.httpunit.WebResponse;
public class FormElements extends Properties
{
private static final String FORM_URL = "form.url";
private static final String FORM_ACTION = "form.action";
/** These are properly provided property parameters. */
private static final String FORM_PARAM = "form.param.";
/** These are property parameters that are required; must have values. */
private static final String FORM_REQUIRED = "form.required.";
private Hashtable fields = new Hashtable( 10 );
private WebConversation webConversation;
public FormElements()
{
}
/**
* Retrieves the HTML page, populates the form data, then sends the
* information to the server.
*/
public void run()
throws Exception
{
WebResponse response = receive();
WebForm form = getWebForm( response );
populate( form );
form.submit();
}
protected WebResponse receive()
throws Exception
{
WebConversation webConversation = getWebConversation();
GetMethodWebRequest request = getGetMethodWebRequest();
// Fake the User-Agent so the site thinks that encryption is supported.
//
request.setHeaderField( "User-Agent",
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv\\:1.7.3) Gecko/20040913" );
return webConversation.getResponse( request );
}
protected void populate( WebForm form )
throws Exception
{
// First set all the .param variables.
//
setParamVariables( form );
// Next, set the required variables.
//
setRequiredVariables( form );
}
protected void setParamVariables( WebForm form )
throws Exception
{
for( Enumeration e = propertyNames(); e.hasMoreElements(); )
{
String property = (String)(e.nextElement());
if( property.startsWith( FORM_PARAM ) )
{
String fieldName = getProperty( property );
String propertyName = property.substring( FORM_PARAM.length() );
String fieldValue = getField( propertyName );
// Skip blank fields (most likely, this is a blank last name, which
// means the form wants a full name).
//
if( "".equals( fieldName ) )
continue;
// If this is the first name, and the last name parameter is blank,
// then append the last name field to the first name field.
//
if( "first_name".equals( propertyName ) &&
"".equals( getProperty( FORM_PARAM + "last_name" ) ) )
fieldValue += " " + getField( "last_name" );
showSet( fieldName, fieldValue );
form.setParameter( fieldName, fieldValue );
}
}
}
protected void setRequiredVariables( WebForm form )
throws Exception
{
for( Enumeration e = propertyNames(); e.hasMoreElements(); )
{
String property = (String)(e.nextElement());
if( property.startsWith( FORM_REQUIRED ) )
{
String fieldValue = getProperty( property );
String fieldName = property.substring( FORM_REQUIRED.length() );
// If the field starts with a ~, then copy the field.
//
if( fieldValue.startsWith( "~" ) )
{
String copyProp = fieldValue.substring( 1, fieldValue.length() );
copyProp = getProperty( copyProp );
// Since the parameters have been copied into the form, we can
// eke out the duplicate values.
//
fieldValue = form.getParameterValue( copyProp );
}
showSet( fieldName, fieldValue );
form.setParameter( fieldName, fieldValue );
}
}
}
private void showSet( String fieldName, String fieldValue )
{
System.out.print( "<p class='setting'>" );
System.out.print( fieldName );
System.out.print( " = " );
System.out.print( fieldValue );
System.out.println( "</p>" );
}
private WebForm getWebForm( WebResponse response )
throws Exception
{
WebForm[] forms = response.getForms();
String action = getProperty( FORM_ACTION );
// Not supposed to break out of a for-loop, but it makes the code easy ...
//
for( int i = forms.length - 1; i >= 0; i-- )
if( forms[ i ].getAction().equalsIgnoreCase( action ) )
return forms[ i ];
// Sadly, no form was found.
//
throw new Exception();
}
private GetMethodWebRequest getGetMethodWebRequest()
{
return new GetMethodWebRequest( getProperty( FORM_URL ) );
}
private WebConversation getWebConversation()
{
if( this.webConversation == null )
this.webConversation = new WebConversation();
return this.webConversation;
}
public void setField( String field, String value )
{
Hashtable fields = getFields();
fields.put( field, value );
}
private String getField( String field )
{
Hashtable<String, String> fields = getFields();
String result = fields.get( field );
return result == null ? "" : result;
}
private Hashtable getFields()
{
return this.fields;
}
public static void main( String args[] )
throws Exception
{
FormElements formElements = new FormElements();
formElements.setField( "first_name", args[1] );
formElements.setField( "last_name", args[2] );
formElements.setField( "email", args[3] );
formElements.setField( "comments", args[4] );
FileInputStream fis = new FileInputStream( args[0] );
formElements.load( fis );
fis.close();
formElements.run();
}
}
An example properties files would look like:
$ cat com.mellon.properties
form.url=https://www.mellon.com/contact/index.cfm
form.action=index.cfm
form.param.first_name=name
form.param.last_name=
form.param.email=emailhome
form.param.comments=comments
# Submit Button
#form.submit=submit
# Required Fields
#
form.required.to=zzwebmaster
form.required.phone=555-555-1212
form.required.besttime=5 to 7pm
Run it similar to the following (substitute the path to HTTPUnit and the FormElements class for $CLASSPATH):
java -cp $CLASSPATH FormElements com.mellon.properties "John" "Doe" "John.Doe#gmail.com" "To whom it may concern ..."
Legality
Another answer mentioned that it might violate terms of use. Check into that first, before you spend any time looking into a technical solution. Extremely good advice.
Most of the time, you can just send a simple HTTP POST request.
I'd suggest you try playing around with Fiddler to understand how the web works.
Nearly all the programming languages and frameworks out there have methods for sending raw requests.
And you can always program against the Internet Explorer ActiveX control. I believe it many programming languages supports it.
I believe this would put in legal violation of the terms of use (consult a lawyer about that: programmers are not good at giving legal advice!), but, technically, you could search for foobar by just visiting URL http://www.google.com/search?q=foobar and, as you say, scraping the resulting HTML. You'll probably also need to fake out the User-Agent HTTP header and maybe some others.
Maybe there are search engines whose terms of use do not forbid this; you and your lawyer might be well advised to look around to see if this is indeed the case.
Well, here's the html from the Google page:
<form action="/search" name=f><table cellpadding=0 cellspacing=0><tr valign=top>
<td width=25%> </td><td align=center nowrap>
<input name=hl type=hidden value=en>
<input type=hidden name=ie value="ISO-8859-1">
<input autocomplete="off" maxlength=2048 name=q size=55 title="Google Search" value="">
<br>
<input name=btnG type=submit value="Google Search">
<input name=btnI type=submit value="I'm Feeling Lucky">
</td><td nowrap width=25% align=left>
<font size=-2> <a href=/advanced_search?hl=en>
Advanced Search</a><br>
<a href=/preferences?hl=en>Preferences</a><br>
<a href=/language_tools?hl=en>Language Tools</a></font></td></tr></table>
</form>
If you know how to make an HTTP request from your favorite programming language, just give it a try and see what you get back. Try this for instance:
http://www.google.com/search?hl=en&q=Stack+Overflow
If you download Cygwin, and add Cygwin\bin to your path you can use curl to retrieve a page and grep/sed/whatever to parse the results. Why fill out the form when with google you can use the querystring parameters, anyway? With curl, you can post info, too, set header info, etc. I use it to call web services from a command line.
Related
I have a hosted Blazor WebAssembly application.
I need a strategy or a sample on how can I copy values from an excel spreadsheet and paste them into the application with a final goal to add them into my database through the existing API.
So the question here is this: what components should I paste the values into, and how should I handle the whole process:
excel > clipboard > Component > save in db
It was actually more difficult than I initially thought. I've created a repo. The result is this.
You can select any elements in Excel, copy them, focus the content of your Blazor page and paste it. As a simple view, it is displayed in a table.
Let's go through the solution.
Index.razor
#page "/"
<div class="form-group">
<label for="parser">Parser type</label>
<select class="form-control" id="parser" #bind="_parserType">
<option value="text">Text</option>
<option value="html">HTML</option>
</select>
</div>
<PasteAwareComponent OnContentPasted="FillTable">
#if (_excelContent.Any() == false)
{
<p>No Content</p>
}
else
{
<table class="table table-striped">
#foreach (var row in _excelContent)
{
<tr>
#foreach (var cell in row)
{
<td>#cell</td>
}
</tr>
}
</table>
}
</PasteAwareComponent>
<button type="button" class="btn btn-primary" #onclick="#( () => _excelContent = new List<String[]>() )">Clear</button>
#code
{
private IList<String[]> _excelContent = new List<String[]>();
...more content, explained later...
}
If you copy a selection from Excel into the clipboard, not a single text is copied, but multiple representations of the same content. In my experiment, it has been three different types.
I've built two different parser: ExcelHtmlContentParser and ExcelTextContentParser. Regarding the many different possibilities of what a cell content in Excel can be, my implementation is merely completed and should be seen as an inspiration. To see both parsers in action, you can choose between them by changing the value in the select box.
The PasteAwareComponent handles the interaction with Javascript. You can place any content inside this component. If this component (or any child) has focus, the paste event will be handled correctly.
<span #ref="_reference">
#ChildContent
</span>
#code {
private ElementReference _reference;
[Parameter]
public RenderFragment ChildContent { get; set; }
[Parameter]
public EventCallback<IEnumerable<IDictionary<String, String>>> OnContentPasted { get; set; }
[JSInvokable("Pasted")]
public async void raisePasteEvent(IEnumerable<IDictionary<String, String>> items)
{
await OnContentPasted.InvokeAsync(items);
}
}
The component handles the interoperation with javascript. As soon the paste events happen the EventCallback<IEnumerable<IDictionary<String, String>>> OnContentPasted is fired.
Potentially, there could be more than one element inside the clipboard. Hence, we need to handle a collection IEnumerable<>. As seen in the picture before, the same clipboard item can have multiple representations. Each representation has a mime-type like "text/plain" or "text/html" and the value. This is represented by the IDictionary<String, String> where the key is the mime-type, and the value is the content.
Before going into the details about the javascript interop, we go back to the Index component.
<PasteAwareComponent OnContentPasted="FillTable">
...
</PasteAwareComponent>
#code {
private async Task FillTable(IEnumerable<IDictionary<String, String>> content)
{
if (content == null || content.Count() != 1)
{
return;
}
var clipboardContent = content.ElementAt(0);
IExcelContentParser parser = null;
switch (_parserType)
{
case "text":
parser = new ExcelTextContentParser();
break;
case "html":
parser = new ExcelHtmlContentParser();
break;
default:
break;
}
foreach (var item in clipboardContent)
{
if (parser.CanParse(item.Key) == false)
{
continue;
}
_excelContent = await parser.GetRows(item.Value);
}
}
}
The index component uses this event callback in the method FillTable. The method checks if there is one element in the clipboard. Based on the selection, the parser is chosen. Each representation is checked in the next step if the chosen parser can parse it, based on the provided mime-type. If the right parser is found, the parser does its magic, and the content of the field _excelContent is updated. Because it is an EventCallback StateHasChanged is called internally, and the view is updated.
The text parser
In the text representation, Excel uses \r\n as the end of the row and a \t for each cell, even the empty ones. The parser logic is quite simple.
public class ExcelTextContentParser : IExcelContentParser
{
public String ValidMimeType { get; } = "text/plain";
public Task<IList<String[]>> GetRows(String input) =>
Task.FromResult<IList<String[]>>(input.Split("\r\n", StringSplitOptions.RemoveEmptyEntries).Select(x =>
x.Split("\t").Select(y => y ?? String.Empty).ToArray()
).ToList());
}
I haven't tested how this behavior changes if the content is more complex. I guess that the HTML representation is more stable. Hence, the second parser.
The HTML parser
The HTML representation is a table. With <tr> and <td>. I've used the library AngleSharp as HTML parser.
public class ExcelHtmlContentParser : IExcelContentParser
{
public String ValidMimeType { get; } = "text/html";
public async Task<IList<String[]>> GetRows(String input)
{
var context = BrowsingContext.New(Configuration.Default);
var document = await context.OpenAsync(reg => reg.Content(input));
var element = document.QuerySelector<IHtmlTableElement>("table");
var result = element.Rows.Select(x => x.Cells.Select(y => y.TextContent).ToArray()).ToList();
return result;
}
}
We are loading the clipboard content as an HTML document, getting the table and iterating over all rows, and selected each column.
** The js interop ***
#inject IJSRuntime runtime
#implements IDisposable
<span #ref="_reference">
#ChildContent
</span>
#code {
private ElementReference _reference;
private DotNetObjectReference<PasteAwareComponent> _objectReference;
protected override async Task OnAfterRenderAsync(bool firstRender)
{
await base.OnAfterRenderAsync(firstRender);
if (firstRender == true)
{
_objectReference = DotNetObjectReference.Create(this);
await runtime.InvokeVoidAsync("BlazorClipboadInterop.ListeningForPasteEvents", new Object[] { _reference, _objectReference });
}
}
public void Dispose()
{
GC.SuppressFinalize(this);
if (_objectReference != null)
{
_objectReference.Dispose();
}
}
}
The PasteAwareComponent component overrides the OnAfterRenderAsync life cycle, to invoke a js interop method. It has to be the OnAfterRenderAsync because before, the HTML reference wouldn't exist, and we need the reference to add the paste event listener. When the paste event occurred the javascript has to call this object, so we need to create a DotNetObjectReference instance. We implemented the IDisposable interface and disposing the reference correctly to prevent memory leaks.
The last part is the javascript part itself. I've created a file called clipboard-interop.js and placed it inside the wwwroot/js folder.
var BlazorClipboadInterop = BlazorClipboadInterop || {};
BlazorClipboadInterop.ListeningForPasteEvents = function (element, dotNetObject) {
element.addEventListener('paste', function (e) { BlazorClipboadInterop.pasteEvent(e, dotNetObject) });
};
We use the HTML reference to register an event listener for the 'paste' event. In the handling method, we create the object that is passed to the C# method.
BlazorClipboadInterop.pasteEvent =
async function (e, dotNetObject) {
var data = await navigator.clipboard.read();
var items = []; //is passed to C#
for (let i = 0; i < data.length; i++) {
var item = {};
items.push(item);
for (let j = 0; j < data[i].types.length; j++) {
const type = data[i].types[j];
const blob = await data[i].getType(type);
if (blob) {
if (type.startsWith("text") == true) {
const content = await blob.text();
item[type] = content;
}
else {
item[type] = await BlazorClipboadInterop.toBase64(blob);
}
}
}
}
dotNetObject.invokeMethodAsync('Pasted', items);
e.preventDefault();
}
When we are using js interop, we should use objects that are easy to serialize. In the case of a real blob, like an image, it would be based64-encoded string, otherwise just the content.
The solution used the navigator.clipboard capabilities. The user needs to allow it. Hence we see the dialog.
My site works this way:
All content has autoroute for {culture/slug} URLs
Users can select the site culture, so that everything is presented in the language they choose
I'm trying to achieve this functionality:
User selects site in English.
User goes to "site.com/es/content", which is a content in Spanish.
The site has to automatically change the culture to Spanish and return the requested content.
What I think I need is to intercept the request, parse the URL and get the culture to see if it's the same as the current one.
I've tried getting it in the ItemsController in Orchard.Core.Contents using the IHttpContextAccessor but it doesn't have the raw Url.
I've also tried catching the request in Orchard.Autoroute and Orchard.Alias services but they are not the ones processing the request.
Any pointers would be appreciated.
There are some ways to do this.
Implement ICultureSelector
namespace SomeModule
{
using System;
using System.Globalization;
using System.Linq;
using System.Web;
using Orchard.Localization.Services;
public class CultureSelectorByHeader : ICultureSelector
{
private readonly ICultureManager cultureManager;
public CultureSelectorByHeader(ICultureManager cultureManager)
{
this.cultureManager = cultureManager;
}
public CultureSelectorResult GetCulture(HttpContextBase context)
{
var acceptedLanguageHeader = context?.Request?.UserLanguages?.FirstOrDefault();
if ( acceptedLanguageHeader == null )
return null;
var enabledCultures = this.cultureManager.ListCultures();
var siteCulture = this.cultureManager.GetSiteCulture();
// Select the specified culture if it's enabled.
// Otherwise, or if it wasn't found, fall back to the default site culture.
var culture = enabledCultures.Contains(acceptedLanguageHeader, StringComparer.InvariantCultureIgnoreCase)
? CultureInfo.CreateSpecificCulture(acceptedLanguageHeader).Name
: CultureInfo.CreateSpecificCulture(siteCulture).Name;
return new CultureSelectorResult { CultureName = culture, Priority = 0 };
}
}
}
You can go wild in GetCulture, read headers, cookies, a query string or get some settings for the current user from DB. Whatever fits your need.
Set the culture directly
private void SetWorkContextCulture(string cultureTwoLetterIsoCode)
{
if ( !string.IsNullOrWhitespace(cultureTwoLetterIsoCode) )
{
try
{
var culture = CultureInfo.CreateSpecificCulture(cultureTwoLetterIsoCode);
this.Services.WorkContext.CurrentCulture = culture.TwoLetterISOLanguageName;
}
catch ( CultureNotFoundException )
{
Debug.WriteLine("Couldn't change thread culture.");
}
}
}
Just change the current culture of WorkContext before returning your result and you're good to go.
Fun fact: Changing the WorkContext.Culture in a controller will override everything you did in your ICultureSelector implementation.
I'm using ServiceStack with OrmLite, and having great success with it so far. I'm looking for a way to filter out 'soft deleted' records when using AutoQuery. I've seen this suggestion to use a SqlExpression, but I'm not sure where you would place that. In the AppHost when the application starts? I did that, but the deleted records still return. My QueryDb request object in this case is as follows:
public class QueryableStore : QueryDb<StoreDto>
{
}
Other SqlExpressions I've used are in the repository class itself, but being that I'm using QueryDb and only the message itself (not leveraging my repository class) I don't have any other code in place to handle these messages and filter out the 'deleted' ones.
I've also tried using a custom service base as suggested by this approach as well, using the following:
public abstract class MyCustomServiceBase : AutoQueryServiceBase
{
private const string IsDeleted = "F_isdeleted";
public override object Exec<From>(IQueryDb<From> dto)
{
var q = AutoQuery.CreateQuery(dto, Request);
q.And("{0} = {1}", IsDeleted, 0);
return AutoQuery.Execute(dto, q);
}
public override object Exec<From, Into>(IQueryDb<From, Into> dto)
{
var q = AutoQuery.CreateQuery(dto, Request);
q.And("{0} = {1}", IsDeleted, 0);
return AutoQuery.Execute(dto, q);
}
}
This code gets called, but when the Execute call happens I get an error:
System.ArgumentException: 'Conversion failed when converting the varchar value 'F_isdeleted' to data type int.'
The F_isdeleted column is a 'bit' in SQL Server, and represented as a bool in my POCO.
Any ideas on what would work here? I'm kind of at a loss that this seems this difficult to do, yet the docs make it look pretty simple.
The {0} are placeholders for db parameters, so your SQL should only be using placeholders for DB parameters, e.g:
var q = AutoQuery.CreateQuery(dto, Request);
q.And(IsDeleted + " = {0}", false);
Otherwise if you want to use SQL Server-specific syntax you can use:
q.And(IsDeleted + " = 0");
I have somewhat lost touch with custom search engines ever since Google switched from its more legacy search engine api in favor of the google custom search api. I'm hoping someone might be able to tell me whether a (pretty simple) goal can be accomplished with the new framework, and potentially any starting help would be great.
Specifically, I am looking to write a program which will read in text from a text file, then use five words from said document in a google search - the point being to figure out how many results accrue from said search.
An example input/output would be:
Input: "This is my search term" -- quotations included in the search!
Output: there were 7 total results
Thanks so much, all, for your time/help
First you need to create a Google Custom Search project inside you google account.
From this project you must obtain a Custom Search Engine ID , known as cx parameter. You must also obtain a API key parameter. Both of these are available from your Google Custom Search API project inside your google account.
Then, if you prefer Java , here's a working example:
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class GoogleCustonSearchAPI {
public static void main(String[] args) throws Exception {
String key="your_key";
String qry="your_query";
String cx = "your_cx";
//Fetch urls
URL url = new URL(
"https://www.googleapis.com/customsearch/v1?key="+key+"&cx="+cx+"&q="+ qry +"&alt=json&queriefields=queries(request(totalResults))");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream())));
//Remove comments if you need to output in JSON format
/*String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
System.out.println(output);
}*/
//Print the urls and domains from Google Custom Search String searchResult;
while ((searchResult = output.readLine()) != null) {
int startPos=searchResult.indexOf("\"link\": \"")+("\"link\": \"").length();
int endPos=searchResult.indexOf("\",");
if(searchResult.contains("\"link\": \"") && (endPos>startPos)){
String link=searchResult.substring(startPos,endPos);
if(link.contains(",")){
String tempLink = "\"";
tempLink+=link;
tempLink+="\"";
System.out.println(tempLink);
}
else{
System.out.println(link);
}
System.out.println(getDomainName(link));
}
}
conn.disconnect();
}
public static String getDomainName(String url) throws URISyntaxException {
URI uri = new URI(url);
String domain = uri.getHost();
return domain.startsWith("www.") ? domain.substring(4) : domain;
}
The "&queriefields=queries(request(totalResults))" is what makes the difference and gives sou what you need. But keep in mind that you can perform only 100 queries per day for free and that the results of Custom Search API are sometimes quite different from the those returned from Google.com search
If anybody would still need some example of CSE (Google Custom Search Engine) API, this is working method
public static List<Result> search(String keyword){
Customsearch customsearch= null;
try {
customsearch = new Customsearch(new NetHttpTransport(),new JacksonFactory(), new HttpRequestInitializer() {
public void initialize(HttpRequest httpRequest) {
try {
// set connect and read timeouts
httpRequest.setConnectTimeout(HTTP_REQUEST_TIMEOUT);
httpRequest.setReadTimeout(HTTP_REQUEST_TIMEOUT);
} catch (Exception ex) {
ex.printStackTrace();
}
}
});
} catch (Exception e) {
e.printStackTrace();
}
List<Result> resultList=null;
try {
Customsearch.Cse.List list=customsearch.cse().list(keyword);
list.setKey(GOOGLE_API_KEY);
list.setCx(SEARCH_ENGINE_ID);
Search results=list.execute();
resultList=results.getItems();
}
catch ( Exception e) {
e.printStackTrace();
}
return resultList;
}
This method returns List of Result Objects, so you can iterate through it
List<Result> results = new ArrayList<>();
try {
results = search(QUERY);
} catch (Exception e) {
e.printStackTrace();
}
for(Result result : results){
System.out.println(result.getDisplayLink());
System.out.println(result.getTitle());
// all attributes
System.out.println(result.toString());
}
I use gradle dependencies
dependencies {
compile 'com.google.apis:google-api-services-customsearch:v1-rev57-1.23.0'
}
Don't forget to define your own GOOGLE_API_KEY, SEARCH_ENGINE_ID (cx), QUERY and HTTP_REQUEST_TIMEOUT (ie private static final int HTTP_REQUEST_TIMEOUT = 3 * 600000;)
How would I override the tables rendered around the webparts in the "Rich Content" area?
I have successfully removed the tables around webpartzones and their webparts but can't figure how to remove the tables around Rich Content area webparts.
I am not using the Content Editor WebPart.
The "Rich Content" area I am using is created using the PublishingWebControls:RichHtmlField.
This is the control which has content and webparts.
Bounty here.
I have pondered this myself in the past and I've come up with two options, though none are very appealing, so have not implemented them:
Create a custom rich text field. Override render, call base.Render using a TextWriter object and place the resulting html in a variable, which you then "manually" clean up, before writing to output.
Create a custom rich text field. Override render, but instead of calling base.Render, take care of the magic of inserting the webparts yourself. (This is probably trickier.)
Good luck!
Update, some example code I use to minimize the output of the RichHtmlField:
public class SlimRichHtmlField : RichHtmlField
{
protected override void Render(HtmlTextWriter output)
{
if (IsEdit() == false)
{
//This will remove the label which precedes the bodytext which identifies what
//element this is. This is also identified using the aria-labelledby attribute
//used by for example screen readers. In our application, this is not needed.
StringBuilder sb = new StringBuilder();
StringWriter sw = new StringWriter(sb);
HtmlTextWriter htw = new HtmlTextWriter(sw);
base.Render(htw);
htw.Flush();
string replaceHtml = GetReplaceHtml();
string replaceHtmlAttr = GetReplaceHtmlAttr();
sb.Replace(replaceHtml, string.Empty).Replace(replaceHtmlAttr, string.Empty);
output.Write(sb.ToString());
}
else
{
base.Render(output);
}
}
private string GetReplaceHtmlAttr()
{
return " aria-labelledby=\"" + this.ClientID + "_label\"";
}
private string GetReplaceHtml()
{
var sb = new StringBuilder();
sb.Append("<div id=\"" + this.ClientID + "_label\" style='display:none'>");
if (this.Field != null)
{
sb.Append(SPHttpUtility.HtmlEncode(this.Field.Title));
}
else
{
sb.Append(SPHttpUtility.HtmlEncode(SPResource.GetString("RTELabel", new object[0])));
}
sb.Append("</div>");
return sb.ToString();
}
private bool IsEdit()
{
return SPContext.Current.FormContext.FormMode == SPControlMode.Edit || SPContext.Current.FormContext.FormMode == SPControlMode.New;
}
}
This code is then used by your pagelayout like this:
<YourPrefix:SlimRichHtmlField ID="RichHtmlField1" HasInitialFocus="false" MinimumEditHeight="200px" FieldName="PublishingPageContent" runat="server" />
Got it:
https://sharepoint.stackexchange.com/a/32957/7442