In Service Fabric I am trying to call an ActorService and get a list of all actors. I'm not getting any errors, but no actors are returned. It's always zero.
This is how I add actors :
ActorProxy.Create<IUserActor>(
new ActorId(uniqueName),
"fabric:/ECommerce/UserActorService");
And this is how I try to get a list of all actors:
var proxy = ActorServiceProxy.Create(new Uri("fabric:/ECommerce/UserActorService"), 0);
ContinuationToken continuationToken = null;
CancellationToken cancellationToken = new CancellationTokenSource().Token;
List<ActorInformation> activeActors = new List<ActorInformation>();
do
{
var proxy = GetUserActorServiceProxy();
PagedResult<ActorInformation> page = await proxy.GetActorsAsync(continuationToken, cancellationToken);
activeActors.AddRange(page.Items.Where(x => x.IsActive));
continuationToken = page.ContinuationToken;
}
while (continuationToken != null);
But no matter how many users I've added, the page object will always have zero items. What am I missing?
The second argument int in ActorServiceProxy.Create(Uri, int, string) is the partition key (you can find out more about actor partitioning here).
The issue here is that your code checks only one partition (partitionKey = 0).
So the solutions is quite simple - you have to iterate over all partitions of you service. Here is an answer with code sample to get partitions and iterate over them.
UPDATE 2019.07.01
I didn't spot this from the first time but the reason why you aren't getting any actors returned is because you aren't creating any actors - you are creating proxies!
The reason for such confusion is that Service Fabric actors are virtual i.e. from the user point of view actor always exists but in real life Service Fabric manages actor object lifetime automatically persisting and restoring it's state as needed.
Here is a quote from the documentation:
An actor is automatically activated (causing an actor object to be constructed) the first time a message is sent to its actor ID. After some period of time, the actor object is garbage collected. In the future, using the actor ID again, causes a new actor object to be constructed. An actor's state outlives the object's lifetime when stored in the state manager.
In you example you've never send any messages to actors!
Here is a code example I wrote in Program.cs of newly created Actor project:
// Please don't forget to replace "fabric:/Application16/Actor1ActorService" with your actor service name.
ActorRuntime.RegisterActorAsync<Actor1> (
(context, actorType) =>
new ActorService(context, actorType)).GetAwaiter().GetResult();
var actor = ActorProxy.Create<IActor1>(
ActorId.CreateRandom(),
new Uri("fabric:/Application16/Actor1ActorService"));
_ = actor.GetCountAsync(default).GetAwaiter().GetResult();
ContinuationToken continuationToken = null;
var activeActors = new List<ActorInformation>();
var serviceName = new Uri("fabric:/Application16/Actor1ActorService");
using (var client = new FabricClient())
{
var partitions = client.QueryManager.GetPartitionListAsync(serviceName).GetAwaiter().GetResult();;
foreach (var partition in partitions)
{
var pi = (Int64RangePartitionInformation) partition.PartitionInformation;
var proxy = ActorServiceProxy.Create(new Uri("fabric:/Application16/Actor1ActorService"), pi.LowKey);
var page = proxy.GetActorsAsync(continuationToken, default).GetAwaiter().GetResult();
activeActors.AddRange(page.Items);
continuationToken = page.ContinuationToken;
}
}
Thread.Sleep(Timeout.Infinite);
Pay special attention to the line:
_ = actor.GetCountAsync(default).GetAwaiter().GetResult();
Here is where the first message to actor is sent.
Hope this helps.
Related
I'm struggling to find a code example from MS for the v3 SDK for queries with paging, they provide examples for V2 but that SDK is a completely different code base using the "CreateDocumentQuery" method.
I've tried searching through GitHub here: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos.Samples/Usage/Queries/Program.cs
I believe I'm looking for a method example using continuation tokens, with the assumption that if I cache the previously used continuation tokens in my web app then I can page backwards as well as forwards?
I'm also not quite understanding MS explanation in that MaxItemCount doesn't actually mean it will only try to return X items, but simply limits the No. of items in each search across each partition, confused!
Can anyone point me to the right place for a code example please? I also tried searching through https://learn.microsoft.com/en-us/azure/cosmos-db/sql-query-pagination but appears to lead us to the older SDK (V2 I believe)
UPDATE (following comments from Gaurav below)
public async Task<(List<T>, string)> QueryWithPagingAsync(string query, int pageSize, string continuationToken)
{
try
{
Container container = GetContainer();
List<T> entities = new(); // Create a local list of type <T> objects.
QueryDefinition queryDefinition = new QueryDefinition(query);
using FeedIterator<T> resultSetIterator = container.GetItemQueryIterator<T>(
query, // SQL Query passed to this method.
continuationToken, // Value is always null for the first run.
requestOptions: new QueryRequestOptions()
{
// Optional if we already know the partition key value.
// Not relevant here becuase we're passing <T> which could
// be any model class passed to the generic method.
//PartitionKey = new PartitionKey("MyParitionKeyValue"),
// This does not actually limit how many documents are returned if
// what we're querying resides across multiple partitions.
// If we set the value to 1, then control the number of times
// the loop below performs the ReadNextAsync, then we can control
// the number of items we return from this method. I'm not sure
// whether this is best way to go, it seems we'd be calling
// the API X no. times by the number of items to return?
MaxItemCount = 1
});
// Set var i to zero, we'll use this to control the number of iterations in
// the loop, then once i is equal to the pageSize then we exit the loop.
// This allows us to limit the number of documents to return (hope this is the best way to do it)
var i = 0;
while (resultSetIterator.HasMoreResults & i < pageSize)
{
FeedResponse<T> response = await resultSetIterator.ReadNextAsync();
entities.AddRange(response);
continuationToken = response.ContinuationToken;
i++; // Add 1 to var i in each iteration.
}
return (entities, continuationToken);
}
catch (CosmosException ex)
{
//Log.Error($"Entities was not retrieved successfully - error details: {ex.Message}");
if (ex.StatusCode == HttpStatusCode.NotFound)
{
return (null, null);
}
else { throw; }
}
}
The above method is my latest attempt, and whilst I'm able to use and return continuation tokens, the next challenge is how to control the number of items returned from Cosmos. In my environment, you may notice the above method is used in a repo with where we're passing in model classes from different calling methods, therefore hard coding the partition key is not practical and I'm struggling with configuring the number of items returned. The above method is in fact controlling the number of items I am returning to the calling method further up the chain, but I'm worried that my methodology is resulting in multiple calls to Cosmos i.e. if I set the page size to 1000 items, am I making an HTTP call to Cosmos 1000 times?
I was looking at a thread here https://stackoverflow.com/questions/54140814/maxitemcount-feed-options-property-in-cosmos-db-doesnt-work but not sure the answer in that thread is a solution, and given I'm using the V3 SDK, there does not seem to be the "PageSize" parameter available to use in the request options.
However I also found an official Cosmos code sample here: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos.Samples/Usage/Queries/Program.cs#L154-L186 (see example method "QueryItemsInPartitionAsStreams" line 171) and it looks like they have used a similar pattern i.e. setting the MaxItemCount variable to 1 and then controlling the no. of items returned in the loop before exiting. I guess I'd just like to understand better what, if any impact this might have on the RUs and API calls to Cosmos?
Please try the following code. It fetches all documents from a container with a maximum of 100 documents in a single request.
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Azure.Cosmos;
namespace CosmosDbSQLAPISamples
{
class Program
{
private static string connectionString =
"AccountEndpoint=https://account-name.documents.azure.com:443/;AccountKey=account-key==;";
private static string databaseName = "database-name";
private static string containerName = "container-name";
static async Task Main(string[] args)
{
CosmosClient client = new CosmosClient(connectionString);
Container container = client.GetContainer(databaseName, containerName);
string query = "Select * From Root r";
string continuationToken = null;
int pageSize = 100;
do
{
var (entities, item2) = await GetDataPage(container, query, continuationToken, pageSize);
continuationToken = item2;
Console.WriteLine($"Total entities fetched: {entities.Count}; More entities available: {!string.IsNullOrWhiteSpace(continuationToken)}");
} while (continuationToken != null);
}
private static async Task<(List<dynamic>, string)> GetDataPage(Container container, string query, string continuationToken, int pageSize)
{
List<dynamic> entities = new(); // Create a local list of type <T> objects.
QueryDefinition queryDefinition = new QueryDefinition(query);
QueryRequestOptions requestOptions = new QueryRequestOptions()
{
MaxItemCount = pageSize
};
FeedIterator<dynamic> resultSetIterator = container.GetItemQueryIterator<dynamic>(query, continuationToken, requestOptions);
FeedResponse<dynamic> response = await resultSetIterator.ReadNextAsync();
entities.AddRange(response);
continuationToken = response.ContinuationToken;
return (entities, continuationToken);
}
}
}
UPDATE
I think I understand your concerns now. Essentially there are two things you would need to consider:
MaxItemCount - This is the maximum number of documents that will be returned by Cosmos DB in a single request. Please note that you can get anywhere from 0 to the value specified for this parameter. For example, if you specify 100 as MaxItemCount you can get anywhere from 0 to 100 documents in a single request.
FeedIterator - It keeps track of continuation token internally. Based on the response received, it sets HasMoreResults to true or false if a continuation token is found. Default value for HasMoreResults is true.
Now coming to your code, when you do something like:
while (resultSetIterator.HasMoreResults)
{
//some code here...
}
Because FeedIterator keeps track of the continuation token, this loop will return all the documents that match the query. If you notice, in my code I am not using this logic. I simply send the request once and then return the result.
I think setting MaxItemCount to 1 is a bad idea. If you want to fetch say 100 then you're making a minimum of 100 requests to your Cosmos DB account. If you have a hard need to get exactly 100 (or any fixed number) documents from your API, you can implement your own pagination logic. For example, please see code below. It fetches a total of 1000 documents with a maximum of 100 documents in a single request.
static async Task Main(string[] args)
{
CosmosClient client = new CosmosClient(connectionString);
Container container = client.GetContainer(databaseName, containerName);
string query = "Select * From Root r";
string continuationToken = null;
int pageSize = 100;
int maxDocumentsToFetch = 1000;
List<dynamic> documents = new List<dynamic>();
do
{
var numberOfDocumentsToFetch = Math.Min(pageSize, maxDocumentsToFetch);
var (entities, item2) = await GetDataPage(container, query, continuationToken, numberOfDocumentsToFetch);
continuationToken = item2;
Console.WriteLine($"Total entities fetched: {entities.Count}; More entities available: {!string.IsNullOrWhiteSpace(continuationToken)}");
maxDocumentsToFetch -= entities.Count;
documents.AddRange(entities);
} while (maxDocumentsToFetch > 0 && continuationToken != null);
}
The solution:
Summary:
From the concerns raised in my question and taking note from Gaurav Mantri's comments, if we are fetching the items from Cosmos in a loop then the MaxItemCount does not actually limit the total number of results returned but simply limits the number of results per request. If we continue to fetch more items in the loop then we end up with more results returned than what the user may want to retrieve.
In my case, the reason for paging is to present the items back to the web App using a razor list view, but we want to be able to set the maximum number of results returned per page.
The solution below is based on capturing information on the count of items in each iteration of the loop, therefore if we check the Count of the items returned on each iteration of the loop and if we have achieved less than or equal to the MaxItemCount value then we break from the loop with our set maximum number of items and the continuationToken that we can use on the next method run.
I have tested the method with continuation tokens and am able to affectively page backwards and forwards, but the key difference from the code example in my original question is that we're only calling Cosmos DB once to get the desired number of results back, as opposed to limiting the request to one item per run and having to run multiple requests.
public async Task<(List<T>, string)> QueryWithPagingAsync(string query, int pageSize, string continuationToken)
{
string unescapedContinuationToken = null;
if (!String.IsNullOrEmpty(continuationToken)) // Check if null before unescaping.
{
unescapedContinuationToken = Regex.Unescape(continuationToken); // Needed in my case...
}
try
{
Container container = GetContainer();
List<T> entities = new(); // Create a local list of type <T> objects.
QueryDefinition queryDefinition = new(query); // Create the query definition.
using FeedIterator<T> resultSetIterator = container.GetItemQueryIterator<T>(
query, // SQL Query passed to this method.
unescapedContinuationToken, // Value is always null for the first run.
requestOptions: new QueryRequestOptions()
{
// MaxItemCount does not actually limit how many documents are returned
// from Cosmos, if what we're querying resides across multiple partitions.
// However this parameter will control the max number of items
// returned on 'each request' to Cosmos.
// In the loop below, we check the Count of the items returned
// on each iteration of the loop and if we have achieved less than or
// equal to the MaxItemCount value then we break from the loop with
// our set maximum number of items and the continuationToken
// that we can use on the next method run.
// 'pageSize' is the max no. items we want to return for each page in our list view.
MaxItemCount = pageSize,
});
while (resultSetIterator.HasMoreResults)
{
FeedResponse<T> response = await resultSetIterator.ReadNextAsync();
entities.AddRange(response);
continuationToken = response.ContinuationToken;
// After the first iteration, we get the count of items returned.
// Now we'll either return the exact number of items that was set
// by the MaxItemCount, OR we may find there were less results than
// the MaxItemCount, but either way after the first run, we should
// have the number of items returned that we want, or at least
// the maximum number of items we want to return, so we break from the loop.
if (response.Count <= pageSize) { break; }
}
return (entities, continuationToken);
}
catch (CosmosException ex)
{
//Log.Error($"Entities was not retrieved successfully - error details: {ex.Message}");
if (ex.StatusCode == HttpStatusCode.NotFound)
{
return (null, null);
}
else { throw; }
}
}
In Code:
var sqlQueryText = $"SELECT * FROM c WHERE OFFSET {offset} LIMIT {limit}";
but this is more expensive (more RU/s) then using continuationToken.
When using Offset/Limit continuationToken will be used in background by Azure Cosmos SDK to get all the results.
Let's say I have two types of Hazelcast nodes running on cluster:
"Leader" nodes – these are able to load and populate Hazelcast map M. Leaders will also update values in M from time to time (based on external resource).
"Follower" nodes – these will need to read from M
My intent is for Follower nodes to trigger loading missing elements into M (loading thus needs to be done on Leader side) .
Roughly, the steps made to get an element from map could look like this:
IMap m = hazelcastInstance.getMap("M");
if (!m.containsKey(k)) {
if (iAmLeader()) {
Object fresh = loadByKey(k); // loading from external resource
return m.put(k, fresh);
} else {
makeSomeLeaderPopulateValueForKey(k);
}
}
return m.get(k);
What approach could you suggest?
Notes
I want Followers to act as nodes, not just clients, because there are going to be far more Follower instances than Leaders and I would like them to participate in load distribution.
I could just build another level of service, that would run only on Leader nodes and provide interface to populate map with requested keys. But that would mean adding extra layer of communication and configuration, and I was hoping that the kind of requirements stated above could be solved within single Hazelcast cluster.
I think I may have found an answer in the form of MapLoader (EDIT since originally posting, I have confirmed this is indeed the way to do this).
final Config config = new Config();
config.getMapConfig("MY_MAP_NAME").setMapStoreConfig(
new MapStoreConfig().setImplementation(new MapLoader<KeyType, ValueType>(){
#Override
public ValueType load(final KeyType key) {
//when a client asks for data for corresponding key of type
//KeyType that isn't already loaded
//this function will be invoked and give you a chance
//to load it and return it
ValueType rv = ...;
return rv;
}
#Override
public Map<KeyType, ValueType> loadAll(
final Collection<KeyType> keys) {
//Similar to MapLoader#load(KeyType), except this is
//a batched version of it for performance gains.
//this gets called on first access to the cache,
//where MapLoader#loadAllKeys() is called to get
//the keys parameter for this funcion
Map<KeyType, ValueType> rv = new HashMap<>();
keys.foreach((key)->{
rv.put(key, /*figure out what key means*/);
});
return rv;
}
#Override
public Set<KeyType> loadAllKeys() {
//Prepopulate all the keys. My understanding is that
//this is an initialization step, to give you a chance
//to load data on startup so an initial set of datas
//will be available to anyone using the cache. Any keys
//returned here are sent to MapLoader#loadAll(Collection)
Set<KeyType> rv = new HashSet<>();
//figure out what keys need to be in the return value
//to load a key into cache at first access to this map,
//named "MY_MAP_NAME" in this example
return rv;
}
}));
config.getGroupConfig().setName("MY_INSTANCE_NAME").setPassword("my_password");
final HazelcastInstance hazelcast = Hazelcast
.getOrCreateHazelcastInstance(config);
I have got a question related to CQRS in data centric processes. Let me explain it better.
Consider we have a SOAP/JSON/whatever service, which transfers some data to our system during an integration process. It is said that in CQRS every state change must be achieved by the means of commands (or events if Event Sourcing is used).
When it comes to our integrating process we have got a great deal of structured DATA instead of a set of commands/events and I am wondering how to actually process those data.
// Some Façade service
class SomeService
{
$_someService;
public function __construct(SomeService $someService)
{
$this->_someService = $someService;
}
// Magic function to make it all good and
public function process($dto)
{
// if I get it correctly here I need somehow
// convert incoming dto (xml/json/array/etc)
// to a set of commands, i. e
$this->someService->doSomeStuff($dto->someStuffData);
// SomeStuffChangedEvent raised here
$this->someService->doSomeMoreStuff($dtom->someMoreStuffData);
// SomeMoreStuffChangedEvent raised here
}
}
My question is whether my suggestion is suitable in the given case or there may be some better methods to do what I need. Thank you in advance.
Agreed, a service may have a different interface. If you create a rest-api to update employees, you may want to provide an UpdateEmployeeMessage which contains everything that can change. In a CRUD-kind of service, this message would probably mirror the database.
Inside of the service, you can split the message into commands:
public void Update(UpdateEmployeeMessage message)
{
bus.Send(new UpdateName
{
EmployeeId = message.EmployeeId,
First = message.FirstName,
Last = message.LastName,
});
bus.Send(new UpdateAddress
{
EmployeeId = message.EmployeeId,
Street = message.Street,
ZipCode = message.ZipCode,
City = message.City
});
bus.Send(new UpdateContactInfo
{
EmployeeId = message.EmployeeId,
Phone = message.Phone,
Email = message.Email
});
}
Or you could call the aggregate directly:
public void Update(UpdateEmployeeMessage message)
{
var employee = repository.Get<Employee>(message.EmployeeId);
employee.UpdateName(message.FirstName, message.LastName);
employee.UpdateAddress(message.Street, message.ZipCode, message.City);
employee.UpdatePhone(message.Phone);
employee.UpdateEmail(message.Email);
repository.Save(employee);
}
I have a simple IBackgroundTask implementation that performs a query and then either performs an insert or one or more updates depending on whether a specific item exists or not. However, the updates are not persisted, and I don't understand why. New items are created just as expected.
The content item I'm updating has a CommonPart and I've tried authenticating as a valid user. I've also tried flushing the content manager at the end of the Sweep method. What am I missing?
This is my Sweep, slightly edited for brevity:
public void Sweep()
{
// Authenticate as the site's super user
var superUser = _membershipService.GetUser(_orchardServices.WorkContext.CurrentSite.SuperUser);
_authenticationService.SetAuthenticatedUserForRequest(superUser);
// Create a dummy "Person" content item
var item = _contentManager.New("Person");
var person = item.As<PersonPart>();
if (person == null)
{
return;
}
person.ExternalId = Random.Next(1, 10).ToString();
person.FirstName = GenerateFirstName();
person.LastName = GenerateLastName();
// Check if the person already exists
var matchingPersons = _contentManager
.Query<PersonPart, PersonRecord>(VersionOptions.AllVersions)
.Where(record => record.ExternalId == person.ExternalId)
.List().ToArray();
if (!matchingPersons.Any())
{
// Insert new person and quit
_contentManager.Create(item, VersionOptions.Draft);
return;
}
// There are at least one matching person, update it
foreach (var updatedPerson in matchingPersons)
{
updatedPerson.FirstName = person.FirstName;
updatedPerson.LastName = person.LastName;
}
_contentManager.Flush();
}
Try to add _contentManager.Publish(updatedPerson). If you do not want to publish, but just to save, you don't need to do anything more, as changes in Orchard as saved automatically unless the ambient transaction is aborted. The call to Flush is not necessary at all. This is the case both during a regular request and on a background task.
I'm getting a strange error while trying to select a row from a table under Windows Azure Table Storage. The exception "An item with the same key has already been added." is being thrown even though I'm not inserting anything. The query that is causing the problem is as follows:
var ids = new HashSet<string>() { id };
var fields = new HashSet<string> {"#all"};
using (var db = new AzureDbFetcher())
{
var result = db.GetPeople(ids, fields, null);
}
public Dictionary<string, Person> GetPeople(HashSet<String> ids, HashSet<String> fields, CollectionOptions options)
{
var result = new Dictionary<string, Person>();
foreach (var id in ids)
{
var p = db.persons.Where(x => x.RowKey == id).SingleOrDefault();
if (p == null)
{
continue;
}
// do something with result
}
}
As you can see, there's only 1 id and the error is thrown right at the top of the loop and nothing is being modified.
However, I'm using "" as the Partition Key for this particular row. What gives?
You probably added an object with the same row key (and no partition key) to your DataServiceContext before performing this query. Then you're retrieving the conflicting object from the data store, and it can't be added to the context because of the collision.
The context tracks all object retrieved from the Tables. Since entities are uniquely identified by their partitionKey/rowKey combination, a context, like the tables, cannot contain duplicate partitionkey/rowkey combinations.
Possible causes of such a collison are:
Retrieving an entity, modifying it, and then retrieving it again using the same context.
Adding an entity to the context, and then retrieving one with the same keys.
In both cases, the context the encounters it's already tracking a different object which does however have the same keys. This is not something the context can sort out by itself, hence the exception.
Hope this helps. If you could give a little more information, that would be helpful.