From MSDN, it seems there's a limitation for the number of entities returned by the Query service:
A query against the Table service may return a maximum of 1,000 entities at one time and may execute for a maximum of five seconds.
But as I wrote a sample to show this issue, I didn't find any limitations for the number of returned entities, here is my key code:
public class DataProvider
{
public static string PartitionKey
{
get { return "PartitionKey"; }
}
public static IEnumerable<CustomerEntity> MoreThanThousandData()
{
var result = new List<CustomerEntity>();
for (int i = 0; i < 1200; i++)
{
result.Add(new CustomerEntity(PartitionKey, Guid.NewGuid().ToString())
{
Name = Guid.NewGuid().ToString(),
Age = new Random().Next(10, 70)
});
}
return result;
}
}
Insert 1200 entities to the table:
public class AfterOptimize
{
public void InsertDataToTable()
{
var cloudData = DataProvider.MoreThanThousandData();
Console.WriteLine("Plan to insert {0} entities to the table.", cloudData.Count());
InsertDataToTableInternal(AzureTableService.Table, cloudData);
}
private void InsertDataToTableInternal(CloudTable table, IEnumerable<ITableEntity> data)
{
var splitedData = data.Chunk(100);
Parallel.ForEach(splitedData, item =>
{
var batchInsertOperation = new TableBatchOperation();
foreach (var tableEntity in item)
{
batchInsertOperation.Add(TableOperation.Insert(tableEntity));
}
table.ExecuteBatch(batchInsertOperation);
});
}
}
Then, read from the table, the partition key are all the same here:
public void ReadCloudData()
{
InsertMoreThanOneThousandDataToTable();
var query =
new TableQuery<CustomerEntity>().Where(TableQuery.GenerateFilterCondition("PartitionKey",
QueryComparisons.Equal, DataProvider.PartitionKey));
var result = AzureTableService.Table.ExecuteQuery(query);
Console.WriteLine("Read {0} entities from table.", result.Count()); // output 1200
}
I only used the latest Azure storage .NET client API.
I'm not able to find a documentation link but ExecuteQuery method handles continuation token internally and will return all entities in a table. Thus the behavior you're seeing is correct.
If you run Fiddler when you are executing this code, you will notice multiple requests are sent to table service. First request would be without continuation token but in subsequent requests you will see NextPartitionKey and NextRowKey querystring parameters.
Related
Using Azure's DocumentDb and the .NET API, I have the following method which works great for retrieving lists of entire documents:
public async Task<IEnumerable<T>> GetItemsAsync<T>(Expression<Func<T, bool>> predicate)
{
IDocumentQuery<T> query = _Client.CreateDocumentQuery<T>(
UriFactory.CreateDocumentCollectionUri(_DatabaseId, _Collection),
new FeedOptions { MaxItemCount = -1 })
.Where(predicate)
.AsDocumentQuery();
List<T> results = new List<T>();
while (query.HasMoreResults)
{
var item = await query.ExecuteNextAsync<T>();
results.AddRange(item);
}
return results;
}
Now, I don't always want to return the entire document (especially considering the DocumentDb RU pricing model), so I thought I should be able to add a .Select projection like so:
public async Task<List<TResult>> GetItemsAsync<T, TResult>(Expression<Func<T, bool>> predicate, Expression<Func<T, TResult>> select)
{
IDocumentQuery<TResult> query = _Client.CreateDocumentQuery<T>(
UriFactory.CreateDocumentCollectionUri(_DatabaseId, _Collection),
new FeedOptions { MaxItemCount = -1 })
.Where(predicate)
.Select(select)
.AsDocumentQuery();
List<TResult> results = new List<TResult>();
while (query.HasMoreResults)
{
var item = await query.ExecuteNextAsync<TResult>();
results.AddRange(item);
}
return results;
}
Usage:
var rez = await _docs.GetItemsAsync<ApolloAssetDoc, Guid?>(x => x.MyVal == 5, x => x.ID);
But the second method always return 0 results. Obviously I'm barking up the wrong tree.
Any idea what the correct way to return a list of either dynamic objects for queries where more than one property is selected (eg "SELECT d.id, d.MyVal FROM Items d WHERE d.DocType=0")or a simple list where only a single property is selected (eg "SELECT d.id FROM Items d WHERE d.DocType=0")?
I could repro the issue, if there is no [JsonProperty(PropertyName = "id")] for ID property in the entity Class. If it is not included, please have a try to use the following code:
public class ApolloAssetDoc
{
[JsonProperty(PropertyName = "id")]
public Guid ID { get; set; }
public string MyVal { get; set; }
}
Note: The field is case sensitive.
Could some one tell me whether I can use the batch processing to add group memberships?
if yes can you please give me an example
thanks in advance
kind regards,
Snegha
According to the documentation, Graph API support batch processing. The Microsoft Azure Active Directory Client support batch processing.
You can find a lot of samples to use the Azure AD Graph API here :
Azure Samples for active directory
Especially you have a full sample on most of the actions that you can perform with the Graph API here :
Call the Azure AD Graph API from a native client (search for "Batch Operations" in the program.cs file).
Unfortunately batch processing is not working for navigation properties or at least I did not find a way to make it work...
So let's have a look at the documentation.
Graph API support for OData batch requests:
A query is a single query or function invocation.
A change set is a group of one or more insert, update, or delete operations, action invocations, or service invocations.
A batch is a container of operations, including one or more change sets and query operations.
The Graph API supports a subset of the functionality defined by the
OData specification:
A single batch can contain a maximum of five queries and/or change sets combined.
A change set can contain a maximum of one source object modification and up to 20 add-link and delete-link operations
combined. All operations in the change set must be on a single source
entity.
Here is the batch request syntax :
https://graph.windows.net/TenantName/$batch?api-version=1.6
A batch request is sent to the server with a single POST directive.
The payload is a multi-part MIME message containing the batch and its constituent queries and change sets. The payload includes two types of MIME boundaries:
A batch boundary separates each query and/or change set in the batch.
A change set boundary separates individual operations within a change set.
An individual request within a change set is identical to a request made when that operation is called by itself. (here is a sample request)
You can find here the full sample code : azure-active-directory-batchprocessing
So Basically, you need to obtain the authentication token:
var authority = "https://login.microsoftonline.com/mytenantName.onmicrosoft.com";
var resource = "https://graph.windows.net/";
var clientId = "ClientId of the application in the Azure Active Directory";
var clientSecret = "ClientSecret of the application in the Azure Active Directory";
var token = new AuthenticationContext(authority, false).AcquireToken(resource,
new ClientCredential(clientId, clientSecret)).AccessToken;
In your question you'd like to add member to a group (see Graph API Documentation on groups):
// Get the objectId of the group
var groupId = ...
// Get the member ids you'd like to add to the group
var memberIds = ...
Here is the code to add members to a group:
private static async Task AddMemberToGroup(string token, string groupId, IList<string> memberIds)
{
if (memberIds.Count > 100)
{
// A batch can contain up to 5 changesets. Each changeset can contain up to 20 operations.
throw new InvalidOperationException("Cannot send more than 100 operation in an batch");
}
var batch = new BatchRequest("https://graph.windows.net/MyTenantName.onmicrosoft.com");
// A changeset can contain up to 20 operations
var takeCount = 20;
var skipCount = 0;
var take = memberIds.Skip(skipCount).Take(takeCount).ToList();
while (take.Count > 0)
{
AddChangeset(batch, groupId, take);
skipCount += takeCount;
take = memberIds.Skip(skipCount).Take(takeCount).ToList();
}
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", $"Bearer {token}");
var response = await client.SendAsync(batch.Request);
}
}
private static void AddChangeset(BatchRequest batch, string groupId, IEnumerable<string> memberIds)
{
var changeset = batch.AddChangeSet();
foreach (var memberId in memberIds)
{
// Create the HttpRequest to add a member to a group
var request = AddMemberToGroupRequest(groupId, memberId);
// Add the operation to the changeset
changeset.AddOperation(request);
}
}
private static HttpRequestMessage AddMemberToGroupRequest(string groupId, string memberId)
{
// Create a request to add a member to a group
var request = new HttpRequestMessage(HttpMethod.Post,
$"https://graph.windows.net/MyTenantName.onmicrosoft.com/groups/{groupId}/$links/members?api-version=1.6");
// Create the body of the request
var jsonBody =
JsonConvert.SerializeObject(new DirectoryObject($"https://graph.windows.net/MyTenantName.onmicrosoft.com/directoryObjects/{memberId}"));
// Set the content
request.Content = new StringContent(jsonBody, Encoding.UTF8, "application/json");
// Return the request
return request;
}
public class BatchRequest
{
private readonly MultipartContent _batchContent;
public BatchRequest(string tenantUrl)
{
// Create the batch request
Request = new HttpRequestMessage(HttpMethod.Post,
$"{tenantUrl}/$batch?api-version=1.6");
// Initializes the batch content
_batchContent = new MultipartContent("mixed", "batch_" + Guid.NewGuid());
Request.Content = _batchContent;
}
public HttpRequestMessage Request { get; }
public ChangeSet AddChangeSet()
{
// Create a new changeset
var changeSet = new ChangeSet();
// Add the content of the changeset to the batch
_batchContent.Add(changeSet.Content);
// return the changeset
return changeSet;
}
public HttpMessageContent CreateOperation(HttpRequestMessage request)
{
var content = new HttpMessageContent(request);
content.Headers.ContentType = new MediaTypeHeaderValue("application/http");
content.Headers.Add("Content-Transfer-Encoding", "binary");
return content;
}
}
public class ChangeSet
{
public ChangeSet()
{
Content = new MultipartContent("mixed", "changeset_" + Guid.NewGuid());
}
public MultipartContent Content { get; }
public void AddOperation(HttpRequestMessage request)
{
var operationContent = new HttpMessageContent(request);
operationContent.Headers.ContentType = new MediaTypeHeaderValue("application/http");
operationContent.Headers.Add("Content-Transfer-Encoding", "binary");
Content.Add(operationContent);
}
}
public class DirectoryObject
{
public DirectoryObject(string url)
{
this.url = url;
}
public string url { get; }
}
I am trying to query a SharePoint 2013 list using the Rest API for all items in the list. The problem is it only returns 1000 records max and I need to get all of the records. I am using the oData v4 API and auto generated service references for the site.
I figured it out: I am including the question and answer here in case anyone else needs it.
I created an extension method called SelectAll() that returns all of the records for a given query.
public static List<T> SelectAll<T>(this DataServiceContext dataContext, IQueryable<T> query)
{
var list = new List<T>();
DataServiceQueryContinuation<T> token = null;
var response = ((DataServiceQuery)query).Execute() as QueryOperationResponse<T>;
do
{
if (token != null)
{
response = dataContext.Execute(token);
}
list.AddRange(response);
} while ((token = response.GetContinuation()) != null);
return list;
}
You use it by calling dataContext.SelectAll(query);
I had the same problem, and wanted it to be a generic solution without providing the query. I do use the EntitySetAttribute to determine the listname.
public static List<T> GetAlltems<T>(this DataServiceContext context)
{
return context.GetAlltems<T>(null);
}
public static List<T> GetAlltems<T>(this DataServiceContext context, IQueryable<T> queryable)
{
List<T> allItems = new List<T>();
DataServiceQueryContinuation<T> token = null;
EntitySetAttribute attr = (EntitySetAttribute)typeof(T).GetCustomAttributes(typeof(EntitySetAttribute), false).First();
// Execute the query for all customers and get the response object.
DataServiceQuery<T> query = null;
if (queryable == null)
{
query = context.CreateQuery<T>(attr.EntitySet);
}
else
{
query = (DataServiceQuery<T>) queryable;
}
QueryOperationResponse<T> response = query.Execute() as QueryOperationResponse<T>;
// With a paged response from the service, use a do...while loop
// to enumerate the results before getting the next link.
do
{
// If nextLink is not null, then there is a new page to load.
if (token != null)
{
// Load the new page from the next link URI.
response = context.Execute<T>(token);
}
allItems.AddRange(response);
}
// Get the next link, and continue while there is a next link.
while ((token = response.GetContinuation()) != null);
return allItems;
}
I am trying to perform an azure table query.
My table (that saves logs) has thousands of rows of data, and it gets populated with more each second.
Right now i have only 1 partition key, but it doesn't affect the next question.
How can i get back lets say only the 100 latest results.
this is my Entity:
public MainServerLogEntity(string Message)
{
this.PartitionKey = "mainserverlogs";
this.RowKey = (DateTime.MaxValue.Ticks - DateTime.UtcNow.Ticks).ToString();
this.Message = Message;
this.Date = DateTime.UtcNow;
}
public MainServerLogEntity() { }
public string Message { get; set; }
public DateTime Date { get; set; }
Right now this is the query i am performing inside a web api i have:
[Route("MainServerLogs")]
[HttpGet]
public IEnumerable<MainServerLogEntity> GetMainServerLogs()
{
CloudTable table = AzureStorageHelpers.GetWebApiTable(connectionString, "mainserverlogs");
TableQuery<MainServerLogEntity> query = new TableQuery<MainServerLogEntity>().Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "mainserverlogs"));
return table.ExecuteQuery(query);
}
But the problem is that i am getting alot of data, and i am requesting this api every few seconds in order to update the ui.
What should i do? is it possible to define in the query that i only want the 100 first rows?
If it is not possible then what other technique should i use?
Try implementing a .Take(100) on the query like so:
[Route("MainServerLogs")]
[HttpGet]
public IEnumerable<MainServerLogEntity> GetMainServerLogs()
{
CloudTable table = AzureStorageHelpers.GetWebApiTable(connectionString, "mainserverlogs");
TableQuery<MainServerLogEntity> query = new TableQuery<MainServerLogEntity>().Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "mainserverlogs")).Take(100);
return table.ExecuteQuery(query);
}
I have an Azure Mobile service coded in .net Web API. I have a TableController. I want that table controller to be able to insert multiple persons, not just one person with from the client with InsertAsync(myPerson). I have the following code in the TableController:
[RequiresAuthorization(AuthorizationLevel.Admin)]
public async Task<bool> InsertPersons(List<Person> values)
{
try
{
foreach (var item in values)
{
var current = await InsertAsync(item);
}
return true;
}
catch (System.Exception)
{
return false;
}
}
The problem is in the client. Because it is strongly typed it only allows me to insert one item at a time. How must I call the server from the client? Do I have to write a Custom Api Controller and call it with mobileService.InvokeApiAsync? If so, how can I get access to my database from a Custom API Controller that doesn't inherit from TableController?
Thank you so much!
The helper methods in the TableController<T> base class assume that the insert operations apply to a single object - and the InsertAsync method in the client also assumes the same. So even though you can define in a table controller a method that takes an array (or list) of Person, you won't be able to call it via the client SDK (at least not without some heavy-lifting using a handler, for example).
You can, however, create a custom API which takes such a list. And to insert the multiple items from the API, you can access the context directly, without needing to go through the helper methods from the table:
public class PersonController : ApiController
{
test20140807Context context;
protected override void Initialize(HttpControllerContext controllerContext)
{
base.Initialize(controllerContext);
this.context = new test20140807Context();
}
[HttpPost]
public async Task<bool> InsertPersons(List<Person> values)
{
foreach (var value in values)
{
if (string.IsNullOrEmpty(value.Id))
{
value.Id = Guid.NewGuid().ToString();
}
}
try
{
this.context.People.AddRange(values);
await this.context.SaveChangesAsync();
return true;
}
catch (System.Exception ex)
{
Trace.WriteLine("Error: " + ex);
return false;
}
}
}
And on the client:
private async void btnTest_Click(object sender, RoutedEventArgs e)
{
var items = new Person[]
{
new Person { Name = "John Doe", Age = 33 },
new Person { Name = "Jane Roe", Age = 32 }
};
try
{
var response = await App.MobileService.InvokeApiAsync<Person[], bool>("person", items);
Debug.WriteLine("response: " + response);
}
catch (Exception ex)
{
var str = ex.ToString();
Debug.WriteLine(str);
}
}
From Carlos Figueira's post on inserting multiple items at once in azure mobile services, it looks like what you need to do is create another table called AllPersons. In your client, the AllPersons object would have a Persons array member. In your server side script for the AllPersons insert, you iterate through the AllPersons.Persons and insert into the table one by one.