I have three queue in my project.
1.verify email and number.
2.register user.
3.perform investment operation like. deposit, withdrawn, invest etc.
I want the flow of execution is first then second when second is running first run next record. and when second is completed then third.. because we have some data dependency for all.
how I create this kind of sequence of queue
queue 1
Trace.TraceInformation("verification is started");
BrokeredMessage verificationqueuedata = Client.Receive();
try
{
if (creditcheckqueuedata != null)
{
UserModel userModel = verificationqueuedata.GetBody<UserModel>();
if (userModel == null)
{
verificationqueuedata.Abandon();
}
else
{//project code
verificationqueuedata.Complete();
}
}
all the three queue are created in same manner..
Support me for creating of sequence
I have three queue in my project. 1.verify email and number.
2.register user. 3.perform investment operation like. deposit, withdrawn, invest etc.
If you mean, three separate queues for separate tasks: Pick up item from Queue-1 once it is completed put message into Queue-2 & so on. There is no race condition here.
If you are using same queue for three types of messages: You need to maintain correlation-id with each of your message & use some kind of state mechanism (database) to find whether previous operation for this correlation id is completed or no.
Related
I have a simple example of dish washers at a restaurant to illustrate the issue I am having.
Question
How can I ensure that the correct number of dish washers are seized & released when it's depended on the number of agents being used?
Problem
Using a function to assign the resources, the number of dish washers are not always correct due to different times in which sinks are used and not used.
Example
Main:
Generates dishes and randomly assigns them to one of three sinks in the exit block.
Sinks is a population of agents.
dish_washers is a ResourcePool with a capacity of 10.
Sink:
Dishes enter a queue and are entered one at a time using a hold block.
Once the dish is cleaned, the hold is unblocked to grab the next dish.
Details:
I have a shared ResourcePool of dish_washers at a restaurant.
There are 3 sinks at the restaurant.
Dishes are generated and randomly assigned to each sink.
If only 1 sink is being used, then two dish washers are needed.
However, if 2 or more sinks are being used then the number of dish washers becomes:
numberOfDishWashers = 2 + numberOfSinksInUse;
In order to change the numberOfDishWashers as more sinks are being used, I created a function that defines the numberOfDishWashers to be seized from the dish_washer ResourcePool.
int numberOfSinksUsed = 0;
int numberOfWorkersToSeize = 0;
int numberOfWorkersAlreadySeized = 0;
int numberOfWorkersToAssign = 0;
ResourcePool[][] dish_washers;
for(Sink curSink : main.sinks){
if(curSink.queue.size() > 0){
numberOfSinksUsed += 1;
}
}
numberOfWorkersAlreadySeized = main.dish_washers.busy();
numberOfWorkersToSeize = 2 + numberOfSinksUsed;
numberOfWorkersToAssign = numberOfWorkersToSeize - numberOfWorkersAlreadySeized;
dish_washers = new ResourcePool[1][numberOfWorkersToAssign];
for(int i = 0; i < numberOfWorkersToAssign; i++){
dish_washers[0][i] = main.dish_washers;
}
return dish_washers;
Error Description:
However, depending on which sink completes first & releases, the number of dish washer assigned will be incorrect. A traceln at the end of the sink process illustrates this where the numberOfDishWashers seized on the exit block doesn't match "2 + numberOfSinksInUse".
There is an instance where 3 sinks are in used but only 4 workers were seized.
Exit, Sink: C Workers Currently Seized: 4
Sinks in Use: 2
Exit, Sink: C Workers Currently Seized: 4
Sinks in Use: 3
Exit, Sink: C Workers Currently Seized: 5
Sinks in Use: 2
Exit, Sink: C Workers Currently Seized: 4
Sinks in Use: 2
Another way to look at the issue, is this Excel table outlining the current logic.
The number of busy workers doesn't match the number of busy workers there should be based on the number of active sinks.
Methods I have Tried
Custom function to release only the necessary workers to keep the correct total.
Generates an error because the resource gets assigned to the 'agent' or dish.
When the dish gets destroyed it has unreleased resources attached to it.
Passing the "sink" agent through an "enter", "seize", and "exit" block to assign the
resource to the agent "sink" instead of the dish that is generated.
Error regarding the "dish" agent being in the flowchart of the "sink" agent while the
"sink" agent is seizing the workers.
How can I ensure the correct number of dish washers are always grabbed?
So your fundamental problem here is that inside the sink you will seize a dishwasher, then the dish goes into the delay (With the number of dishwashers seized) and once out of the delay it will release what ever dishwashers it seized... But during the time it is in the delay the situation might have changed and you actually wanted to seize a different number of dishwashers for that specific sink...
Your options are to either
Remove dishes from the delay, release the correct amount of dishwashers, return back into the delay and delay for the remainder of the time...
Implement your own logic.
I would go for option 2 as option 1 means that you develop a workaround for the block created by AnyLogic and you will end up not using the blocks the way they were designed, this is unfortunately the issue with blockification
So I would have a collection inside of a sink that shows the number of dishwashers currently assigned to this sink. Then whenever a new dish enters a sink we recalculate the number of dishwashers to assign (perhaps at every sink? ) and then make the correct assignment.
Here is an example with some sample code - I did not test it but you will have something similar
I need to analyse one http event value which should not be greater than 30mins. & 95% event should belong to this bucket. If it fails send the alert.
My first concern is to get the right metrics in /actuator/prometheus
Steps I took:
As in every http request event, I am getting one integer value called eventMinute.
Using micrometer MeterRegistry, I tried below code
// MeterRegistry meterRegistry ...
meterRegistry.summary("MINUTES_ANALYSIS", tags);
where tag = EVENT_MINUTE which receives some integer value in each
http event.
But this way, it floods the metrics due to millions of event.
Guide me a way please, i am beginner to this. Thanks!!
The simplest solution (which I would recommend you start with) would be to just create 2 counters:
int theThing = //getTheThing()
if(theThing > 30) {
meterRegistry.counter("my.request.counter.abovethreshold").inc()
}
meterRegistry.counter("my.request.counter.total").inc()
You would increment the counter that matches your threshold and another that tracks all requests (or reuse another meter that does that for you).
Then it is simple to setup a chart or alarm:
my_request_counter_abovethreshold/my_request_counter_total < .95
(I didn't test the code. It might need a tiny bit of tweaking)
You'll be able to do a similar thing with DistributionSummary by setting various SLOs (I'm not familiar with them to be able to offer one), but start with something simple first and if it is sufficient, you won't need the other complexity.
There are certain ways to solve this problem
1 ; here is a function which receives tags, name of metrics and a value
public void createOrUpdateHistogram(String metricName, Map<String, String> stringTags, double numericValue)
{
DistributionSummary.builder(metricName)
.tags(tags)
//can enforce slo if required
.publishPercentileHistogram()
.minimumExpectedValue(1.0D) // can take this based on how you want your distibution
.maximumExpectedValue(30.0D)
.register(this.meterRegistry)
.record(numericValue);
}
then it produce metrics like
delta_bucket{mode="CURRENT",le="30.0",} 11.0
delta_bucket{mode="CURRENT", le="+Inf",} 11.0
so as infinte also hold the less than value, so subtract the le=30 from le=+Inf
Another ways could be
public void createOrUpdateHistogram(String metricName, Map<String, String> stringTags, double numericValue)
{
Timer.builder(metricName)
.tags(tags)
.publishPercentiles(new double[]{0.5D, 0.95D})
.publishPercentileHistogram()
.serviceLevelObjectives(new Duration[]{Duration.ofMinutes(30L)})
.minimumExpectedValue(Duration.ofMinutes(30L))
.maximumExpectedValue(Duration.ofMinutes(30L))
.register(this.meterRegistry)
.record((long)timeDifference, TimeUnit.MINUTES);
}
it will only have two le, the given time and +inf
it can be change based on our requirements also it gives us quantile.
So i have 3 lambdas, one with an API event that triggers a lambda that pulls down around 50,000 objects and pushes them all to a queue.
The second lambda reads from the queue, 10 at a time, in a loop 30 times - meaning it reads, does stuff, invokes the third lambda, returns promise, then reads again - 30 times for a total of 300 reads in the time the lambda executes
The 3rd lambda takes the information from the queue and hits another endpoint with it.
The issue is in that second lambda...First i call a function that returns the number of messages in the queue and if it's more than zero i read them. However, even if there's 20,000 messages in the queue it often comes back with nothing. I'm not sure why.
I have WaitTimeSeconds set to 20 for long polling. Any help would be greatly appreciated, the docs claim i can read up to 3,000/second with a FIFO queue and i'm having trouble getting anywhere near that performance.
Here's the code:
exports.handler = (event, context, callback) => {
const sqs = new AWS.SQS({ region: process.env.AWS_REGION });
getMessageCount(sqs)
.then((messageCount) => {
if (messageCount > 0) {
mapSeries(range(0, 30), getMessages(sqs))
.then((messageRes) => {
callback(null, messageRes);
})
.catch(e => Promise.reject(e));
}
callback(null, 'No more messages');
})
.catch((e) => {
callback(e);
});
};
getMessageCount makes a call to sqs.getQueueAttributes and returns a promise that receives the number of messages.
mapSeries allows the loop to wait for the previous promise to be resolved/rejected before iterating and on each iteration it calls getMessages which calls sqs.receiveMessage and invokes the 3rd lambda with the data.
Any perspective on this is appreciated, thank you!
As i understand your questions, the problem lies with getting the number of messages in the queue. If you had also given the getMessageCount(sqs) as well, we could have determined the types of attributes you are trying to retrieve from SQS.
There are three types of attributes relevant, to get the message count in SQS. These attributes are given below.
ApproximateNumberOfMessages - Returns the approximate number of visible
messages in a queue
ApproximateNumberOfMessagesNotVisible - Returns the approximate number of messages that have not timed-out and aren't deleted.
If you want to include the messages that are waiting to be added, you can consider the following property as well.
ApproximateNumberOfMessagesDelayed - Returns the approximate number of
messages that are waiting to be added to the queue.
By considering these attributes, you can get a much more accurate count from SQS.
Also if I may suggest, I implemented a similar system, but without looking for the count.I retrieve 10 messages at a time via polling, process them and delete them from the queue. As per your example, you can repeat this for 30 times. But if the getMessages(sqs) function returns an empty set, we could assume that the list is empty. (This depends on whether you are using short polling or long polling). Nevertheless, checking for the number of messages at every step seems to be redundant. This is according to this example, but it might defer according to the use case.
Read through the API documentation: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SQS.html#receiveMessage-property
Parameters:
MaxNumberOfMessages — (Integer)
The maximum number of messages to return. Amazon SQS never returns
more messages than this value (however, fewer messages might be
returned). Valid values are 1 to 10. Default is 1.
Wrap your code in a while loop and anticipate a frequent case of 0 messages since 0 is fewer than 1 to 10.
Something like...
var messages = [];
while(messages.length < NUMBER_OF_MSGS_YOU_REALLY_WANT) {
var new_messages = await getSQSMessages(NUMBER_OF_MSGS_YOU_REALLY_WANT - messages.length);
if(new_messages.Data.Messages.length > 0) {
messages.push(new_messages.Data.Messages);
}
}
I have a Purchase Order content type in my Orchard application. Among other properties it has a PurchaseOrderNumber. The purchase order number is assigned when the user saves the purchase order for the first time. I use a custom controller and views for implementing the purchase order CRUD operations.
I have a purchase order number definition part which is attached to a company content type where the next purchase order number, a prefix and padding is saved. So when the system generates the next purchase order number, the prefix (e.g. PO) is used together with the next number (e.g. 123) and the padding (e.g. 5) to generate a string - e.g. PO00123.
When the purchase order number is generated the next purchase order number stored in the purchase order definition part attached to the company content item is incremented and saved so that when a user creates another purchase order it will be assigned the next number.
My challenge here is to prevent duplicate purchase order numbers from being assigned if two users create a new purchase order at the same time.
I was thinking of creating an ISingletonDependency that uses lock (_lock) {...} to wrap code that will generate the next number. This way multiple request can ask for the next number and always get the next unique number. How do I implement this though? I can't figure out how to get access to an IContentManager that has its own database transaction.
Or is there a different pattern that I should rather use?
I figured it out after looking at the Orchard.Tasks.Locking.Services.DistributedLockService class. You need to take a dependency on ILifetimeScope and then resolve ITransactionManager and IContentManager.
lock (_lock) {
using (var childLifetimeScope = _lifetimeScope.BeginLifetimeScope()) {
var transactionManager = childLifetimeScope.Resolve<ITransactionManager>();
var contentManager = childLifetimeScope.Resolve<IContentManager>();
try {
transactionManager.RequireNew(IsolationLevel.Serializable);
var contentItem = contentManager.GetLatest(contentItemId);
var number = CompileNewNumber(contentItem);
contentManager.Publish(contentItem);
return number;
}
catch (Exception exception) {
Logger.Error(exception, "Error compiling next number.");
transactionManager.Cancel();
return "";
}
}
}
I have records that have an index attribute to maintain their position in relation to each other.
I have a plugin that performs a renumbering operation on these records when the index is changed or new one created. There are specific rules that apply to items that are at the first and last position in the list.
If a new (or existing changed) item is inserted into the middle (not technically the middle...just somewhere between start and end) of the list a renumbering kicks off to make room for the record.
This renumbering process fires in a new execution pipeline...We are updating record D. When I tell record E to change (to make room for D) that of course fires the plugin on update message.
This renumbering is fine until we reach the end of the list where the plugin then gets into a loop with the first business rule that maintains the first and last record differently.
So I am trying to think of ways to pass a flag to the execution context spawned by the renumbering process so the recursion skips the boundary edge business rules if IsRenumbering == true.
My thoughts / ideas:
I have thought of using the Depth check > 1 but that isn't a reliable value as I can't explicitly turn it on or off....it may happen to work but that is not engineering a solid solution that is hoping nothing goes bump. Further a colleague far more knowledgeable than I said that when a workflow calls a plugin the depth value is off and can't be trusted.
All my variables are scoped at the execute level so as to avoid variable pollution at the class level....However if I had a dictionary object, tuple, something at the class level and one value would be the thread id and the other the flag value then perhaps my subsequent execution context could check if the same owning thread id had any values entered.
Any thoughts or other ideas on how to pass context information to a new pipeline would be greatly appreciated.
Per Nicknow sugestion I tried sharedvariables but they seem to be going out of scope...:
First time firing post op:
if (base.Stage == EXrmPluginStepStage.PostOperation)
{
...snip...
foreach (var item in RenumberSet)
{
Context.ParentContext.SharedVariables[recordrenumbering] = "googly";
Entity renumrec = new Entity("abcd") { Id = item.Id };
#region We either add or subtract indexes based upon sortdir
...snip...
renumrec["abc_indexfield"] = TmpIdx + 1;
break;
.....snip.....
#endregion
OrganizationService.Update(renumrec);
}
}
Now we come into Pre-Op of the recursion process kicked off by the above post-op OrganizationService.Update(renumrec); and it seems based upon this check the sharedvariable didn't carry over...???
if (!Context.SharedVariables.Contains(recordrenumbering))
{
//Trace.Trace("Null Set");
//Context.SharedVariables[recordrenumbering] = IsRenumbering;
Context.SharedVariables[recordrenumbering] = "Null Set";
}
throw invalidpluginexception reveals:
Sanity Checks:
Depth : 2
Entity: ...
Message: Update
Stage: PreOperation [20]
User: 065507fe-86df-e311-95fe-00155d050605
Initiating User: 065507fe-86df-e311-95fe-00155d050605
ContextEntityName: ....
ContextParentEntityName: ....
....
IsRenumbering: Null Set
What are you looking for is IExecutionContext.SharedVariables. Whatever you add here is available throughout the entire transaction. Since you'll have child pipelines you'll want to look at the ParentContext for the value. This can all get a little tricky, so be sure to do a lot of testing - I've run into many issues with SharedVariables and looping operations in Dynamics CRM.
Here is some sample (very untested) code to get you started.
public static bool GetIsRenumbering(IPluginExecutionContext pluginContext)
{
var keyName = "IsRenumbering";
var ctx = pluginContext;
while (ctx != null)
{
if (ctx.SharedVariables.Contains(keyName))
{
return (bool)ctx.SharedVariables[keyName];
}
else ctx = ctx.ParentContext;
}
return false;
}
public static void SetIsRenumbering(IPluginExecutionContext pluginContext)
{
var keyName = "IsRenumbering";
var ctx = pluginContext;
ctx.SharedVariables.Add(keyName, true);
}
A very simple solution: add a bit field to the entity called "DisableIndexRecalculation." When your first plugin runs, make sure to set that field to true for all of your updates. In the same plugin, check to see if "DisableIndexRecalculation" is set to true: if so, set it to null (by removing it from the TargetEntity entirely) and stop executing the plugin. If it is null, do your index recalculation.
Because you are immediately removing the field from the TargetEntity if it is true the value will never be persisted to the database so there will be no performance penalty.