SharePoint oData API Only Returns 1000 Records - sharepoint

I am trying to query a SharePoint 2013 list using the Rest API for all items in the list. The problem is it only returns 1000 records max and I need to get all of the records. I am using the oData v4 API and auto generated service references for the site.
I figured it out: I am including the question and answer here in case anyone else needs it.

I created an extension method called SelectAll() that returns all of the records for a given query.
public static List<T> SelectAll<T>(this DataServiceContext dataContext, IQueryable<T> query)
{
var list = new List<T>();
DataServiceQueryContinuation<T> token = null;
var response = ((DataServiceQuery)query).Execute() as QueryOperationResponse<T>;
do
{
if (token != null)
{
response = dataContext.Execute(token);
}
list.AddRange(response);
} while ((token = response.GetContinuation()) != null);
return list;
}
You use it by calling dataContext.SelectAll(query);

I had the same problem, and wanted it to be a generic solution without providing the query. I do use the EntitySetAttribute to determine the listname.
public static List<T> GetAlltems<T>(this DataServiceContext context)
{
return context.GetAlltems<T>(null);
}
public static List<T> GetAlltems<T>(this DataServiceContext context, IQueryable<T> queryable)
{
List<T> allItems = new List<T>();
DataServiceQueryContinuation<T> token = null;
EntitySetAttribute attr = (EntitySetAttribute)typeof(T).GetCustomAttributes(typeof(EntitySetAttribute), false).First();
// Execute the query for all customers and get the response object.
DataServiceQuery<T> query = null;
if (queryable == null)
{
query = context.CreateQuery<T>(attr.EntitySet);
}
else
{
query = (DataServiceQuery<T>) queryable;
}
QueryOperationResponse<T> response = query.Execute() as QueryOperationResponse<T>;
// With a paged response from the service, use a do...while loop
// to enumerate the results before getting the next link.
do
{
// If nextLink is not null, then there is a new page to load.
if (token != null)
{
// Load the new page from the next link URI.
response = context.Execute<T>(token);
}
allItems.AddRange(response);
}
// Get the next link, and continue while there is a next link.
while ((token = response.GetContinuation()) != null);
return allItems;
}

Related

Servicestack automap update endpoints

We are using ServiceStack QueryDb to expose certain business objects for auto-querying, and it is working great.
[Route("/catalog/customers")]
[Authenticate]
public class QueryCustomers : QueryDb<ServiceModel.Catalog.Customer> { }
We'd like to implement some sort of UpdateDb that auto-maps authenticated POST, PUT and DELETE requests to insert, update and delete auto-mapped predefined OrmLite business objects, but don't see any examples on the web. Ideally, we could create an endpoint as simple as:
[Route("/edits/customers")]
[Authenticate]
public class UpdateCustomers : UpdateDb<ServiceModel.Catalog.Customer> { }
I'd prefer to not have to roll our own if this has already been done somewhere in the ServiceStack library or elsewhere... Is this something that ServiceStack already supports, or are we on our own in developing this UpdateDb utility?...
This would require a CRUD version of AutoQuery which doesn't exist yet so you would need to implement your Update services as normal.
Haven't done a lot of testing, but it looks like the UpdateDb will involve one tweak against the QueryDb implementation -
public long CreateUpdate<From>(IUpdateDb<From> dto, Dictionary<string, string> dynamicParams, IRequest req = null)
{
long result;
var db = GetDb<From>(req);
var body = req.GetRawBody();
var jsonObject = JsonObject.Parse(body);
var obj = jsonObject.ConvertTo<From>();
var id = (obj as IHasId<int>)?.Id;
if (id == null)
throw new Exception("Cannot update without identity field defined");
if (req.Verb == "DELETE")
{
result = db.DeleteById<From>(id);
}
else if (req.Verb == "POST")
{
db.InsertAll(new[] { obj });
result = ((IHasId<int>)obj).Id;
}
else
{
result = db.UpdateOnly(obj,
jsonObject.Keys.ToArray(),
u => ((IHasId<int>)u).Id == id);
}
return result;
}

Azure App Service - Update object from table controller

In the Azure app service mobile backend service, REST API requests are handled by TableController implementation. These methods can be invoked by using corresponding methods available in client SDKs. So, i can query for a particular entity and update its status from the client side.
But how to invoke them in the server side or within the same controller? For example, if I want to query for a particular todoItem and update its status from some custom method here like
Use LookUp(id) to get the item
Update the status
Use UpdateAsync(id, item)
Here I don't know how to create a Delta object of TodoItem to call UpdateAsync(id, patch) method.
public class TodoItemController : TableController<TodoItem>
{
protected override void Initialize(HttpControllerContext controllerContext)
{
base.Initialize(controllerContext);
initrackerserviceContext context = new initrackerserviceContext();
DomainManager = new EntityDomainManager<TodoItem>(context, Request);
}
// GET tables/TodoItem
public IQueryable<TodoItem> GetAllTodoItems()
{
return Query();
}
// GET tables/TodoItem/48D68C86-6EA6-4C25-AA33-223FC9A27959
public SingleResult<TodoItem> GetTodoItem(string id)
{
return Lookup(id);
}
// PATCH tables/TodoItem/48D68C86-6EA6-4C25-AA33-223FC9A27959
public Task<TodoItem> PatchTodoItem(string id, Delta<TodoItem> patch)
{
return UpdateAsync(id, patch);
}
// POST tables/TodoItem
public async Task<IHttpActionResult> PostTodoItem(TodoItem item)
{
TodoItem current = await InsertAsync(item);
return CreatedAtRoute("Tables", new { id = current.Id }, current);
}
// DELETE tables/TodoItem/48D68C86-6EA6-4C25-AA33-223FC9A27959
public Task DeleteTodoItem(string id)
{
return DeleteAsync(id);
}
}
Just use the standard Entity Framework mechanisms. For instance, to find and update a record with a status, you can just use the context:
var item = await context.TodoItems.Where(i => i.Id.Equals(myId)).FirstOrDefaultAsync<TodoItem>();
if (item != null) {
item.Complete = true;
context.Entry(item).State = EntityState.Modified;
await context.SaveChangesAsync();
}
My EF coding is not the greatest ad-hoc, but you should get the idea. Just do the Entity Framework thing.
It's better to use TableController.ReplaceAsync() method that is already implemented for us here in the source code of EntityDomainManager.
var item = Lookup(item.Id).Queryable.FirstOrDefault();
if (item != null)
{
item.Complete = true;
item = await ReplaceAsync(item.Id, item);
}
The ReplaceAsync() method correctly handles the exceptions, so I would not recommend working directly with the EF context.

C# Parse how to wait until Query returns a value

I'm trying to retrieve user data from Parse (xamarin.ios using c#). I'm using an async method with await. My challenge is,each time I navigate to the tableView in the app, which should populate the user data in question,the table is always empty.
I would like to wait until the results have been returned before proceeding with the other portion of code.I have tried to use the ContinueWith() function but constantly ran into a build error -
Cannot implicitly convert type 'void' to System.Collections.Generic.IEnumerable<Parse.ParseObject>
My Questions:
Is this the best way to wait for the result?
How do I solve the build error?
Here is my current implementation:
public async void retrieveData(string username)
{
try
{
this.requests.ClearRequests();
refreshed = false;
var query = ParseObject.GetQuery("Requests").WhereEqualTo("username", username);
IEnumerable<ParseObject> results = await query.FindAsync().ContinueWith(t =>{
if(results != null)
{
foreach(ParseObject parseObject in results)
{
UserRequest request = new UserRequest();
request.objectId = parseObject.ObjectId;
request.make = parseObject.Get<string> ("item1");
request.model = parseObject.Get<string> ("item2");
request.year = parseObject.Get<string> ("item3");
request.userName = parseObject.Get<string> ("username");
this.requests.addRequest (request);
}
refreshed = true;
}
});
}
catch(ParseException e) {
Console.WriteLine (e.Message + e.StackTrace);
}
}
You shouldn't need a ContinueWith...that's what the await should handle.
await waits on a Task and then brings back the result with the proper return type. ContinueWith returns a Task, so you would have to grab the Result from the task to make it usable.
For more on this type of thing, you may want to check out Difference between await and ContinueWith
You can try something like this.
public async void retrieveData(string username, )
{
try
{
this.requests.ClearRequests();
refreshed = false;
var query = ParseObject.GetQuery("Requests").WhereEqualTo("username", username);
IEnumerable<ParseObject> results = await query.FindAsync();
if(results != null)
{
foreach(ParseObject parseObject in results)
{
UserRequest request = new UserRequest();
request.objectId = parseObject.ObjectId;
request.make = parseObject.Get<string> ("item1");
request.model = parseObject.Get<string> ("item2");
request.year = parseObject.Get<string> ("item3");
request.userName = parseObject.Get<string> ("username");
this.requests.addRequest (request);
}
refreshed = true;
}
//This is your refresh method for your TableView
this.RefreshTableView();
//or, if in iOS
NSNotificationCenter.DefaultCenter.PostNotificationName("resultsRetrieved", null);
}
catch(ParseException e) {
Console.WriteLine (e.Message + e.StackTrace);
}
}
To show the results in the tableView, I would recommend moving the refreshing of the tableView to a separate method that gets triggered synchronously after the results have been retrieved and parsed. This is shown with the this.RefreshTableView() call above.
If in iOS on Xamarin, another option is to post a notification to the NSNotificationCenter (the Xamarin documentation for which is here). Use the PostNotificationName part seen above instead and then add an observer in the ViewControllers that you want to be dependent on the data. This is done as follows:
Make a notificationToken object:
NSObject notificationToken;
Then in your setup method (you could put this inside of your ViewDidLoad):
void Setup ()
{
notificationToken = NSNotificationCenter.DefaultCenter.AddObserver ("resultsRetrieved", RefreshData);
}
Make your RefeshData method:
void RefreshData (NSString notifString)
{
this.tableView.ReloadData();
}
And then, make sure you dispose of the notification observer when you tear down the class
void Teardown ()
{
NSNotificationCenter.DefaultCenter.RemoveObserver (notificationToken);
}
I had a similar issue so started using callbacks. I'm using them in Xamarin.Android, pretty sure they're available in Xamarin.iOS.
Method that starts the task method - Note I am passing in a method of this class as a parameter
private async void updatedData()
{
await Utils.DataTasks.getNewLiveTips(populateTipsList);
}
Method that calls for data from server
public class DataTasks
{
public static async Task getAllData(Action<IEnumerable<ParseObjects>> callback) {
var query = new ParseQuery<ParseObjects>().OrderByDescending("updatedAt").Limit(5);
IEnumerable<ParseObjects> parseTips = await query.FindAsync();
foreach (var tip in parseTips)
{
// Save data to DB if needed
}
callback(parseTips);
}
Method I passed as parameter in the first instance is now called
private void populateTipsList(IEnumerable<ParseObjects> results)
{
mAdapter = new TipAdapter(this.Activity, results);
mRecyclerView.SetAdapter(mAdapter);
refresher.Refreshing = false;
}

dynamics crm plugin delete entity during update message

Is it possible to delete entity while same is still in plugin update transaction?
It seems following code is not working. I need to delete entity when its get updated and some other circumstances
Something like:
protected void ExecutePosAnnotationtUpdate(LocalPluginContext localContext)
{
if (localContext == null)
{
throw new ArgumentNullException("localContext");
}
if (localContext.PluginExecutionContext.Depth > 1) return;
Entity postEntityImage = null;
if (localContext.PluginExecutionContext.PostEntityImages.Contains("PostImage"))
{
if (localContext.PluginExecutionContext.PostEntityImages["PostImage"] != null)
{
postEntityImage = localContext.PluginExecutionContext.PostEntityImages["PostImage"];
}
}
Entity preEntityImage = null;
if (localContext.PluginExecutionContext.PreEntityImages.Contains("PreImage"))
{
if (localContext.PluginExecutionContext.PreEntityImages["PreImage"] != null)
{
preEntityImage = localContext.PluginExecutionContext.PreEntityImages["PreImage"];
}
}
if ((bool)postEntityImage.Attributes["isdocument"])
{
if ( some condition )
localContext.OrganizationService.Delete(postEntityImage.LogicalName, postEntityImage.Id);
}
}
`
Since you're updating, the record is there in Target.
public void Execute(IServiceProvider serviceProvider)
{
var context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
var serviceFactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
var service = serviceFactory.CreateOrganizationService(context.UserId);
var target = context.InputParameters["Target"] as Entity;
var condition = /* whatever */
if(condition)
{
service.Delete(target.LogicalName, target.Id);
}
}
Works as expected when attached to Update message, Post-Operation, Asynchronous. Works inside the Sandbox, also.
Records will not disappear at once, it takes some time (~20 seconds on my on-premise playground). If you make it Synchronous it will still work but alerts are going to come up because data disappears while being handled by the CRM during the update.

Does the Azure Storage Table query entities really has number limitations?

From MSDN, it seems there's a limitation for the number of entities returned by the Query service:
A query against the Table service may return a maximum of 1,000 entities at one time and may execute for a maximum of five seconds.
But as I wrote a sample to show this issue, I didn't find any limitations for the number of returned entities, here is my key code:
public class DataProvider
{
public static string PartitionKey
{
get { return "PartitionKey"; }
}
public static IEnumerable<CustomerEntity> MoreThanThousandData()
{
var result = new List<CustomerEntity>();
for (int i = 0; i < 1200; i++)
{
result.Add(new CustomerEntity(PartitionKey, Guid.NewGuid().ToString())
{
Name = Guid.NewGuid().ToString(),
Age = new Random().Next(10, 70)
});
}
return result;
}
}
Insert 1200 entities to the table:
public class AfterOptimize
{
public void InsertDataToTable()
{
var cloudData = DataProvider.MoreThanThousandData();
Console.WriteLine("Plan to insert {0} entities to the table.", cloudData.Count());
InsertDataToTableInternal(AzureTableService.Table, cloudData);
}
private void InsertDataToTableInternal(CloudTable table, IEnumerable<ITableEntity> data)
{
var splitedData = data.Chunk(100);
Parallel.ForEach(splitedData, item =>
{
var batchInsertOperation = new TableBatchOperation();
foreach (var tableEntity in item)
{
batchInsertOperation.Add(TableOperation.Insert(tableEntity));
}
table.ExecuteBatch(batchInsertOperation);
});
}
}
Then, read from the table, the partition key are all the same here:
public void ReadCloudData()
{
InsertMoreThanOneThousandDataToTable();
var query =
new TableQuery<CustomerEntity>().Where(TableQuery.GenerateFilterCondition("PartitionKey",
QueryComparisons.Equal, DataProvider.PartitionKey));
var result = AzureTableService.Table.ExecuteQuery(query);
Console.WriteLine("Read {0} entities from table.", result.Count()); // output 1200
}
I only used the latest Azure storage .NET client API.
I'm not able to find a documentation link but ExecuteQuery method handles continuation token internally and will return all entities in a table. Thus the behavior you're seeing is correct.
If you run Fiddler when you are executing this code, you will notice multiple requests are sent to table service. First request would be without continuation token but in subsequent requests you will see NextPartitionKey and NextRowKey querystring parameters.

Resources