Google Data API very slow to read large spreadsheet - google-docs

I am using the Google Data API for .net (https://code.google.com/p/google-gdata/) to read data from some Google Docs spreadsheets.
I have found that reading from a large spreadsheet ( > 14000 records) takes a very long time, several hours, before returning the data.
The code I'm using is below, is there another method to use to read the data, or am I missing something else to speed up this process?
public void ReadSpreadsheet(string applicationName, string title, string sheetName, string cellRange, string userName, string password)
{
var service = new SpreadsheetsService(applicationName);
service.setUserCredentials(userName, password);
SpreadsheetEntry spreadSheetEntry = GetSpreadsheetByTitle(service, title);
WorksheetEntry workSheetEntry = GetWorkSheetByTitle(spreadSheetEntry.Worksheets, sheetName);
AtomEntryCollection list = GetCells(service, workSheetEntry, cellRange);
}
public static SpreadsheetEntry GetSpreadsheetByTitle(SpreadsheetsService service, string title)
{
var query = new SpreadsheetQuery();
query.Title = title;
SpreadsheetFeed feed = service.Query(query);
return (from SpreadsheetEntry entry in feed.Entries
where title.Equals(entry.Title.Text, StringComparison.InvariantCultureIgnoreCase)
select (entry)).FirstOrDefault();
}
public static WorksheetEntry GetWorkSheetByTitle(WorksheetFeed feed, string title = null)
{
return (from WorksheetEntry entry in feed.Entries
where title != null && title.Equals(entry.Title.Text, StringComparison.InvariantCultureIgnoreCase)
select (entry)).FirstOrDefault();
}
public static AtomEntryCollection GetCells(SpreadsheetsService service, WorksheetEntry entry, string range)
{
AtomLink cellFeedLink = entry.Links.FindService(GDataSpreadsheetsNameTable.CellRel, null);
var query = new CellQuery(cellFeedLink.HRef.ToString());
query.Range = range;
query.ReturnEmpty = ReturnEmptyCells.yes;
// this line is what takes the most time
CellFeed feed = service.Query(query);
return (feed.Entries);
}

Related

Excel File Upload with Kendo UI

while I was trying to upload an excel file with kendo ui I found a code on the internet. It is using a keyword named "Constants" but this keyword does not recognize the ".xls" file extension. I am stuck at this and did some research but have no answer to solve this. Here is my code:
public ActionResult Submit(IEnumerable<HttpPostedFileBase> files)
{
if(files!= null)
{
string fileName;
string filePath;
string fileExtension;
foreach(var f in files)
{
//Set file details
SetFileDetails(f, out fileName, out filePath, out fileExtension);
if(fileExtension == Constants.xls || fileExtension == Constants.xlsx)
{
//Save the uploaded file to app folder
string savedExcelFiles = Constants.UploadedFolder + fileName;
f.SaveAs(Server.MapPath(savedExcelFiles));
ReadDataFromExcelFiles(savedExcelFiles);
}
else
{
//file not supported send alert
}
}
}
return RedirectToActionPermanent("Index","Connect");
}
private static void SetFileDetails(HttpPostedFileBase f,out string fileName,out string filePath,out string fileExtension)
{
fileName=Path.GetFileName(f.FileName);
fileExtension=Path.GetExtension(f.FileName);
filePath = Path.GetFullPath(f.FileName);
}
private void ReadDataFromExcelFiles(string savedExcelFiles)
{
var connectionString = string.Format("Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0};Extended Properties=Excel 12.0;",Server.MapPath(savedExcelFiles));
//fill the DataSet by the sheets
var adapter = new OleDbDataAdapter("SELECT * FROM [Sheet1$]",connectionString);
var ds = new DataSet();
List<UploadExcel> uploadExl = new List<UploadExcel>();
adapter.Fill(ds,"Subscriber");
DataTable data=ds.Tables["Subscriber"];
GetSetUploadExcelData(uploadExl,data);
}
private static void GetSetUploadExcelData (List<UploadExcel> uploadExl,DataTable data)
{
for(int i=0;i<data.Rows.Count-1;i++)
{
UploadExcel NewUpload = new UploadExcel();
NewUpload.ID = Convert.ToInt16(data.Rows[i]["ID"]);
NewUpload.CostCenter = Convert.ToString(data.Rows[i]["CostCenter"]);
NewUpload.FirstName = Convert.ToString(data.Rows[i]["FirstName"]);
NewUpload.LastName = Convert.ToString(data.Rows[i]["LastName"]);
NewUpload.MobileNo = Convert.ToString(data.Rows[i]["MobileNo"]);
NewUpload.EmailID = Convert.ToString(data.Rows[i]["EmailID"]);
NewUpload.Services = Convert.ToString(data.Rows[i]["Services"]);
NewUpload.UsageType = Convert.ToString(data.Rows[i]["UsageType"]);
NewUpload.Network = Convert.ToString(data.Rows[i]["Network"]);
NewUpload.UsageIncluded = Convert.ToInt16(data.Rows[i]["UsageIncluded"]);
NewUpload.Unit = Convert.ToString(data.Rows[i]["Unit"]);
uploadExl.Add(NewUpload);
}
}
}
I suspect that the Constants.xls relates to a static class or enum that the original code author is using to hold the .xls/.xlsx extensions.
If you create a constants class something like:
public static class Constants
{
public static string xls = "xls";
public static string xlsx = "xlsx";
}
This would then should help.
If you need any more assistance then please let me know.
edit: Just reviewing the code it seems they are also putting in constant mapping for the uploadfolder location as well so I suspect this is just a static class rather than an enum with application specific details. in a way a bit like using the appSettings within webconfig

converting strings within Entity Framework query

The following query works perfectly fine and populates its dropdown list. The data in the data base is stored in all uppercase, ie PALM BEACH. I want to convert it to Proper case, which obviously i can do after the fact by iterating through the returned list and reformatting BUT I should be able to do it with in the query itself. The following query works fine.
Dim citylist As List(Of String) = (From c In ctx.ziptaxes
Where c.StateID = ddlStates.SelectedIndex
Order By c.City Ascending
Select c.City).ToList()
But if i try to convert it to some thing like this, it fails
Dim citylist As List(Of String) = (From c In ctx.ziptaxes
Where c.StateID = ddlStates.SelectedIndex
Let cityname = StrConv(c.City, VbStrConv.ProperCase)
Order By cityname Ascending
Select cityname).ToList()
I've tried using culture info and String.Format(c.City, vbProperCase) too and nothing other than the original query works. Any help appreciated.
ADDENDUM:
Well some further research is telling me that .Net objects like string conversion and cultureinfo cannot be used prior to the query being run. If that's the case it explains why it isn't working. The following solves my problem BUT I would still like to know if there is way to do it within the LINQ to EF.
Dim citylist As List(Of String) = (From c In ctx.ziptaxes
Where c.StateID = ddlStates.SelectedIndex
Order By c.City Ascending
Select c.City).ToList()
If citylist.Count > 0 Then
For i As Integer = 0 To citylist.Count - 1
citylist(i) = StrConv(citylist(i).ToLower(), vbProperCase)
Next
With ddlCity
.Items.Clear()
.DataSource = citylist.Distinct()
.DataBind()
.Items.Insert(0, "Select a city")
.SelectedIndex = 0
End With
End If
You can do the conversion in your SELECT. Here's an example (with an over-simplified City name converter):
using System.Collections.Generic;
using System.Linq;
using NUnit.Framework;
namespace LinqQuestion
{
[TestFixture]
public class StackOverflowTests
{
private IEnumerable<City> _cities;
[TestFixtureSetUp]
public void Arrange()
{
_cities = new List<City>
{
new City { Id = 1, Name = "FLINT", StateId = 1 },
new City { Id = 2, Name = "SAGINAW", StateId = 1 },
new City { Id = 3, Name = "DETROIT", StateId = 1 },
new City { Id = 4, Name = "FLint", StateId = 1 }
};
}
[Test]
public void TestCountryQuery()
{
var data = _cities
.Where(c => c.StateId == 1)
.OrderBy(c => c.Name)
.Select(c => StrConv(c.Name))
.Distinct().ToList();
Assert.That(data.Count == 3);
}
private static string StrConv(string original)
{
var firstLetter = original.Substring(0, 1).ToUpper();
var theRest = original.Substring(1, original.Length - 1).ToLower();
return firstLetter + theRest;
}
}
public class City
{
public int Id { get; set; }
public int StateId { get; set; }
public string Name { get; set; }
}
}

Issue with SqlScalar<T> and SqlList<T> when calling stored procedure with parameters

The new API for Servicestack.OrmLite dictates that when calling fx a stored procedure you should use either SqlScalar or SqlList like this:
List<Poco> results = db.SqlList<Poco>("EXEC GetAnalyticsForWeek 1");
List<Poco> results = db.SqlList<Poco>("EXEC GetAnalyticsForWeek #weekNo", new { weekNo = 1 });
List<int> results = db.SqlList<int>("EXEC GetTotalsForWeek 1");
List<int> results = db.SqlList<int>("EXEC GetTotalsForWeek #weekNo", new { weekNo = 1 });
However the named parameters doesn't work. You HAVE to respect the order of the parameters in the SP. I think it is because the SP is executed using CommandType=CommandType.Text instead of CommandType.StoredProcedure, and the parameters are added as dbCmd.Parameters.Add(). It seems that because the CommandType is Text it expects the parameters to be added in the SQL querystring, and not as Parameters.Add(), because it ignores the naming.
An example:
CREATE PROCEDURE [dbo].[sproc_WS_SelectScanFeedScanRecords]
#JobNo int = 0
,#SyncStatus int = -1
AS
BEGIN
SET NOCOUNT ON;
SELECT
FSR.ScanId
, FSR.JobNo
, FSR.BatchNo
, FSR.BagNo
, FSR.ScanType
, FSR.ScanDate
, FSR.ScanTime
, FSR.ScanStatus
, FSR.SyncStatus
, FSR.JobId
FROM dbo.SCAN_FeedScanRecords FSR
WHERE ((FSR.JobNo = #JobNo) OR (#JobNo = 0) OR (ISNULL(#JobNo,1) = 1))
AND ((FSR.SyncStatus = #SyncStatus) OR (#SyncStatus = -1) OR (ISNULL(#SyncStatus,-1) = -1))
END
When calling this SP as this:
db.SqlList<ScanRecord>("EXEC sproc_WS_SelectScanFeedScanRecords #SyncStatus",new {SyncStatus = 1});
It returns all records with JobNo = 1 instead of SyncStatus=1 because it ignores the named parameter and add by the order in which they are defined in the SP.
I have to call it like this:
db.SqlList<ScanRecord>("EXEC sproc_WS_SelectScanFeedScanRecords #SyncStatus=1");
Is this expected behavior? I think it defeats the anonymous type parameters if I can't trust it
TIA
Bo
My solution was to roll my own methods for stored procedures. If people finds them handy, I could add them to the project
public static void StoredProcedure(this IDbConnection dbConn, string storedprocedure, object anonType = null)
{
dbConn.Exec(dbCmd =>
{
dbCmd.CommandType = CommandType.StoredProcedure;
dbCmd.CommandText = storedprocedure;
dbCmd.SetParameters(anonType, true);
dbCmd.ExecuteNonQuery();
});
}
public static T StoredProcedureScalar<T>(this IDbConnection dbConn, string storedprocedure, object anonType = null)
{
return dbConn.Exec(dbCmd =>
{
dbCmd.CommandType = CommandType.StoredProcedure;
dbCmd.CommandText = storedprocedure;
dbCmd.SetParameters(anonType, true);
using (IDataReader reader = dbCmd.ExecuteReader())
return GetScalar<T>(reader);
});
}
public static List<T> StoredProcedureList<T>(this IDbConnection dbConn, string storedprocedure, object anonType = null)
{
return dbConn.Exec(dbCmd =>
{
dbCmd.CommandType = CommandType.StoredProcedure;
dbCmd.CommandText = storedprocedure;
dbCmd.SetParameters(anonType, true);
using (var dbReader = dbCmd.ExecuteReader())
return IsScalar<T>()
? dbReader.GetFirstColumn<T>()
: dbReader.ConvertToList<T>();
});
}
They are just modified versions of the SqlScalar and SqlList plus the ExecuteNonQuery

Retrieve Image from Picture Library - REST

I am looking for some suggestion or sample around retrieving images (actual file, not URL), from a picture library using REST API.
Thanks for any input.
Task 1: Getting a List of Image libs on a given site
public static XmlNode GetPicLibListingXML(string imagingServiceURL)
{
Imaging wsImaging = new Imaging();
wsImaging.UseDefaultCredentials = true;
wsImaging.Url = imagingServiceURL;
XmlNode xnPicLibs = wsImaging.ListPictureLibrary();
return xnPicLibs;
}
Sample return XML:
<Library name="{3C1D52F5-5387-490A-9A2D-A9C99A208C00}" title="Tech Images" guid="3c1d52f5-5387-490a-9a2d-a9c99a208c00" url="Tech Images" xmlns="http://schemas.microsoft.com/sharepoint/soap/ois/" />
Task 2: Listing Images in a given library
public static XmlNode GetImageFileListing(string imagingServiceURL, string imageFileLibraryName)
{
Imaging wsImaging = new Imaging();
ImageInfo curImageInfo = new ImageInfo();
wsImaging.UseDefaultCredentials = true;
wsImaging.Url = imagingServiceURL;
XmlNode xnListItems = wsImaging.GetListItems(imageFileLibraryName, "");
return xnListItems;
}
Task 3: Download Image(s)
private const string ATTR_FILENAME = "name";
private const string FILENAMESPACEURI = "http://schemas.microsoft.com/sharepoint/soap/ois/";
public static bool DownloadImageFiles(string imagingServiceURL, string imageFileLibraryName, string[] fileNames, string saveToFolder)
{
Imaging wsImaging = new Imaging();
wsImaging.UseDefaultCredentials = true;
wsImaging.Url = imagingServiceURL;
XmlElement parent = (XmlElement)wsImaging.Download(imageFileLibraryName, string.Empty, fileNames, 0, true);
XmlNodeList files = parent.GetElementsByTagName("File", FILENAMESPACEURI);
foreach (XmlNode file in files)
{
if (Directory.Exists(saveToFolder) == false)
{
Directory.CreateDirectory(saveToFolder);
}
byte[] fileBytes = Convert.FromBase64String(file.InnerText);
using (FileStream fs = File.OpenWrite(saveToFolder + file.Attributes[ATTR_FILENAME].Value))
{
BinaryWriter writer = new BinaryWriter(fs);
writer.Write(fileBytes);
writer.Close();
}
}
return true;
}
Note:
Imaging() class is a web reference to imagining.asmx
The Download call natively returns XML so yo uneed to run it through a conversion to byte
If you need to get a reference on the Imagine web service check this on out on MSDN:
http://msdn.microsoft.com/en-us/library/imaging.imaging.aspx
source:
http://gourangaland.wordpress.com/2008/05/30/using-the-moss-imaging-web-service-to-download-imagesimaging-asmx/

Check if a List Column Exists using SharePoint Client Object Model?

Using the Client Object Model (C#) in SharePoint 2010, how can I determine if a specified column (field) name exists in a given List?
Thanks, MagicAndi.
Just found this while searching for the same thing, but it looks like Sharepoint 2010 has something built in for this, at least for the Server model: list.Fields.ContainsField("fieldName");
Not sure if it exists for Client side though. Figured it would be a good place to store this information however.
Server Object Model
string siteUrl = "http://mysite";
using (SPSite site = new SPSite(siteUrl))
{
using (SPWeb web = site.OpenWeb())
{
SPList list = web.Lists["my forum"];
for (int i = 0; i < list.Fields.Count; i++)
{
if (list.Fields[i].Title == "xyz")
{
-
-
}
}
}
}
Client Object Model
string siteUrl = "http://MyServer/sites/MySiteCollection";
ClientContext clientContext = new ClientContext(siteUrl);
SP.List List = clientContext.Web.Lists.GetByTitle("my forum");
for (int i = 0; i < list.Fields.Count; i++)
{
if (list.Fields[i].Title == "xyz")
{
-
-
}
}
The following method demonstrates how to determine whether a specified column exists in a List using CSOM:
static class FieldCollectionExtensions
{
public static bool ContainsField(this List list,string fieldName)
{
var ctx = list.Context;
var result = ctx.LoadQuery(list.Fields.Where(f => f.InternalName == fieldName));
ctx.ExecuteQuery();
return result.Any();
}
}
Usage
using(var ctx = new ClientContext(webUrl))
{
var list = ctx.Web.Lists.GetByTitle(listTitle);
if(list.ContainsField("Title")){
//...
}
}
Here's an extension code (CSOM) for sharepoint list
public static bool DoesFieldExist(this List list, ClientContext clientContext, string internalFieldname)
{
bool exists = false;
clientContext.Load(list.Fields, fCol => fCol.Include(
f => f.InternalName
).Where(field => field.InternalName == internalFieldname));
clientContext.ExecuteQuery();
if (list.Fields != null && list.Fields.Count > 0)
{
exists = true;
}
return exists;
}
usage
List targetList = this.Context.Web.Lists.GetById(<ListID>);
targetList.DoesFieldExist(<ClientContext>, <Field internal Name>)
enjoy :)
I ended up retrieving the details of the list's fields prior to my operation, and saving them in a generic list of structs (containing details of each field). I then query this (generic) list to see if the current field actually exists in the given (SharePoint) list.
// Retrieve detail sof all fields in specified list
using (ClientContext clientContext = new ClientContext(SharePointSiteUrl))
{
List list = clientContext.Web.Lists.GetByTitle(listName);
_listFieldDetails = new List<SPFieldDetails>();
// get fields name and their types
ClientObjectPrototype allFields = list.Fields.RetrieveItems();
allFields.Retrieve( FieldPropertyNames.Title,
FieldPropertyNames.InternalName,
FieldPropertyNames.FieldTypeKind,
FieldPropertyNames.Id,
FieldPropertyNames.ReadOnlyField);
clientContext.ExecuteQuery();
foreach (Field field in list.Fields)
{
SPFieldDetails fieldDetails = new SPFieldDetails();
fieldDetails.Title = field.Title;
fieldDetails.InternalName = field.InternalName;
fieldDetails.Type = field.FieldTypeKind;
fieldDetails.ID = field.Id;
fieldDetails.ReadOnly = field.ReadOnlyField;
listFieldDetails.Add(fieldDetails);
}
}
// Check if field name exists
_listFieldDetails.Exists(field => field.Title == fieldName);
// Struct to hold details of the field
public struct SPFieldDetails
{
public string Title { get; set; }
public string InternalName { get; set; }
public Guid ID { get; set; }
public FieldType Type { get; set; }
public bool ReadOnly { get; set; }
}
Some good answers above. I personally used this one:
List list = ctx.Web.Lists.GetByTitle("Some list");
FieldCollection fields = list.Fields;
IEnumerable<Field> fieldsColl = ctx.LoadQuery(fields.Include(f => f.InternalName));
ctx.ExecuteQuery();
bool fieldMissing = fieldsColl.Any(f => f.InternalName != "Internal_Name");
You can also use 'Where' after Include method and check if returned collection/field is null. It's about personal preference, because both options are querying on client side.
I prefer the SharePoint Plus Library as it is really clean:
http://aymkdn.github.io/SharepointPlus/symbols/%24SP%28%29.list.html
$SP().list("My List").get({
fields:"Title",
where:"Author = '[Me]'"
},function getData(row) {
console.log(row[0].getAttribute("Title"));
});
You could setup a for loop to loop through the row and check if the column you're looking for exists.
A cut down and simplified version of Mitya's extension method:
public static bool FieldExists(this List list, string internalFieldname)
{
using (ClientContext clientContext = list.Context as ClientContext)
{
clientContext.Load(list.Fields, fCol => fCol.Include(
f => f.InternalName
).Where(field => field.InternalName == internalFieldname));
clientContext.ExecuteQuery();
return (list.Fields != null) && (list.Fields.Count > 0);
}
}
There's no need to pass in a separate client context parameter when you can already use the context that comes in with the list.
to much code use this
load Fields first then
bool exists= clientContext2.Site.RootWeb.Fields.Any(o => o.Id.ToString() == a.Id.ToString());

Resources