Is there a way in Orleans to make sure X reminder grains of same type are hosted on different silos?
Motivation:
I want to use Orleans in an Azure cloud service that runs background processing and should be highly resilient to failures and to silos downtime due to in-place upgrade. In my scenario there is a need for a grain that is always active doing it's job using timers, and if it’s hosting silo goes down then it should be activated again on another silo in few seconds. During an Azure cloud service upgrade, the instances are divided into 5 groups, one group is upgraded at a time, which means during an upgrade I always have 20% of my instances shut down, which makes it challenging to have an always active grain. I consulted with Sergey Bykov and we came up with several ideas, one of them was introducing “keep alive” grains that their purpose is to ping the “always active” grain to make sure it’s active. In order to make that work during upgrades, I need to make sure there is always at least one “keep alive” grain that its hosting silo isn’t in the same upgrade group as the silo that hosts the “always active” grain, so that “keep alive” grain will be able to make the “always active” grain be re-activated on another silo that isn’t shutdown at that moment.
I recently was reading the Orleans Timers and Reminders documentation which contained the following nugget which may be relevant. Highlighting is my own.
Reminders
Description
Reminders are similar to timers with a few important differences:
Reminders are persistent and will continue to trigger in all situations (including partial or full cluster restarts) unless explicitly cancelled.
Reminders are associated with a grain, not any specific activation.
If a grain has no activation associated with it and a reminder ticks, one will be created. e.g.: If an activation becomes idle and is deactivated, a reminder associated with the same grain will reactivate the grain when it ticks next.
Reminders are delivered by message and are subject to the same interleaving semantics as all other grain methods.
Reminders should not be used for high-frequency timers– their period should be measured in minutes, hours, or days.
Further Comments
It is possible that this isn't a solution in your case because of the final note about high-frequency. If you have a single point of failure that can't go down, it might be relevant to consider alternative solutions. :-/
The particular case that I am looking at using this for is precisely for a 5 minute polling interval. I need to call a web service, gather some data and then process it.
My only other thought is to make my polling-based implementation super lightweight so that it can be quickly brought up to speed as the underlying infrastructure is changed.
It appears that there is a decent bit of documentation here regarding activation placement strategies.
Related
I've come across many clients who aren't really able to provide real production data about a website's peak usage. I often do not get peak pageviews per hour, etc.
In these circumstances, besides just guessing or going with what "feels right" (i.e. making it all up), how exactly does one come up with a realistic workload model with an appropriate # of virtual users and a good pacing value?
I use Loadrunner for my performance/load testing.
Ask for the logs for a month.
Find the stats for session duration, then count the number of distinct IP's blocked by session duration.
Once you have the high volume hour, count the number of page instances. Business processes will typically have a termination page which is distinct and allows you to understand how many times a particular action takes place, such as request new password, update profile, business process 1, etc...
With this you will have a measurement of users and actions. You will want your stakeholder to take ownership of this data. As Quality assurance, we should not own both the requirement and the test against it. We should own one, but not both. If your client will not own the requirement, cascading it down to rest of the organization, assume you will be left out in the cold with a result they do not like....i.e., defects that need to be addressed before deployment to production.
Now comes your largest challenge, which is something that needs to be fixed with a process issue by your client.....You are about to test using requirements that no other part of the organization, architecture, development, platform engineering, had when they built the solution. Even if your requirements are a perfect recovery, plus some amount for growth, any defects you find will be challenged aggressively.
Your test will not match any assumptions or requirements used by any other portion of the organization.
And, in a sense, these other orgs will be correct in aggressively challenging your results. It really isn't fair to hold their designed solution to a set of requirements which were not in place when they made decisions which impacted scalability and response times for the system. You would be wise to call this out with your clients before the first execution of any performance test.
You can buy yourself some time. If the client does have a demand for a particular response time, such as an adoption of the Google RAIL model, then you can implement a gate before accepting any code for multi-user performance testing that the code SHALL BE compliant for a single user. It is not going to get any faster for two or more users. Implementing this hard gate will solve about 80% of your performance issues, for the changes required to bring code into compliance for a single user most often will have benefits on the multi-user front.
You can buy yourself some time in a second way as well. Take a look at their current site using tools such as Google Lighthouse and GTMetrix. Most of us are creatures of habit, that includes architect, developers, and ops personnel. We design, build, deploy to patterns we know and are comfortable with....usually the same ones over and over again until we are forced to make a change. It is highly likely that the performance antipatterns pulled from Lighthouse and GTMetrix will be carried forward into a future release unless they are called out for mitigation. Begin citing defects directly off of these tools before you even run a performance test. You will need management support, but you might consider not even accepting a build for multi-user performance testing until GTMetrix scores at least a B across the board and Lighthouse a score of 90 or better.
This should leave edge cases when you do get to multi-user performance testing, such as too early allocation of a resource, holding onto resources too long, too large of a resource allocation, hitting something too often, lock contention on a shared resource. An architectural review might pick up on these, where someone might say, "we are pre-allocating this because.....," or "Marketing says we need to hold the cart for 30 minutes before de-allocation," or "...." Well, you get the idea.
Don't forget to have the database profiler running while functional testing is going on. You are likely to pick up a few missing indexes or high cost queries here which should be addressed before multi-user performance testing as well.
You are probably wondering why am I pointing out all of these things before your performance test takes place. Darnit, you are hired to engage in a performance test! The test you are about to conduct is very high risk politically. Even if it finds something ugly, because the other parts of the organization did not benefit from the requirements, the result is likely to be rejected until the issue shows up in production. By shifting the focus to objective measures even before you need to run two users in anger together there are many avenues to finding and fixing performance issues which are far less politically volatile. Food for thought.
For small app they are no problem.
But for apps with traffic you can hit limits easily.
Http protocol is req-res driven. Just because your backend is stuck with limit, you can't really wait to send respond back until rate limit allows you to resume making your api calls.
What do you do?
I can think of several scenarios:
Wait it out: while it sucks, but sometimes it's easy fix, as you don't need to do anything.
Queue it: this a lot of work oppose to making just api call. This requires that first you store it in database, then have background task go through database and do the task. Also user would be told "it is processing" not "it's done"
Use lot of apis: very hacky... and lot of trouble to manage. Say you are using amazon, now you would have to create, verify, validate like 10 accounts. Not even possible for where you need to verify with say domain name. Since amazon would know account abc already owns it.
To expand on what your queueing options are:
Unless you can design the problem of hitting this rate limit out of existence as #Hammerbot walks through, I would go with some implementation of queue. The solution can scale in complexity and robustness according to what loads you're facing and how many rate limited APIs you're dealing with.
Recommended
You use some library to take care of this for you. Node-rate-limiter looks promising. It still appears you would have to worry about how you handle your user interaction (make them wait, write to a db/cache-service and notify them later).
"Simplest case" - not recommended
You can implement a minimally functioning queue and back it with a database or cache. I've done this before and it was fine, initially. Just remember you'll run into needing to implement your own retry logic, will have to worry about things like queue starvation **. Basically, the caveats of rolling your own < insert thing whose implementation someone already worried about > should be taken into consideration.
**(e.g. your calls keep failing for some reason and all of a sudden your background process is endlessly retrying large numbers of failing queue work elements and your app runs out of memory).
Complex case:
You have a bunch of API calls that all get rate-limited and those calls are all made at volumes that make you start considering decoupling your architecture so that your user-facing app doesn't have to worry about handling this asynchronous background processing.
High-level architecture:
Your user-facing server pushes work units of different type onto different queues. Each of these queues corresponds to a differently rate-limited processing (e.g. 10 queries per hour, 1000 queries per day). You then have a "rate-limit service" that acts as a gate to consuming work units off the different queues. Horizontally distributed workers then only consume items from the queues if and only if the rate limit service says they can. The results of these workers could then be written to a database and you could have some background process to then notify your users of the result of the asynchronous work you had to perform.
Of course, in this case you're wading into a whole world of infrastructure concerns.
For further reading, you could use Lyft's rate-limiting service (which I think implements the token bucket algorithm to handle rate limiting). You could use Amazon's simple queueing service for the queues and Amazon lambda as the queue consumers.
There are two reasons why rate limits may cause you problems.
Chronic: (that is, a sustained situation). You are hitting rate limits because your sustained demand exceeds your allowance.
In this case, consider a local cache, so you don't ask for the same thing twice. Hopefully the API you are using has a reliable "last-modified" date so you can detect when your cache is stale. With this approach, your API calling is to refresh your cache, and you serve requests from your cache.
If that can't help, you need higher rate limits
Acute: your application makes bursts of calls that exceed the rate limit, but on average your demand is under the limit. So you have a short term problem. I have settled on a brute-force solution for this ("shoot first, ask permission later"). I burst until I hit the rate limit, then I use retry logic, which is easy as my preferred tool is python, which supports this easily. The returned error is trapped and retry handling takes over. I think every mature library would have something like this.
https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html
The default retry logic is to backoff in increasingly big steps of time.
This has a starvation risk, I think. That is, if there are multiple clients using the same API, they share the same rate limit as a pool. On your nth retry, your backoff may be so long that newer clients with shorter backoff times are stealing your slots ... by the time your long backoff time expires, the rate limit has already been consumed by a younger competitor, so you now retry even longer, making the problem worse,although at the limit, this just means the same as the chronic situation: the real problem is your total rate limit is insufficient, but you might not be sharing fairly among jobs due to starvation. An improvement is to provide a less naive algorithm, it's the same locking problem that you do in computer science (introducing randomisation is a big improvement). Once again, a mature library is aware of this and should help with built-in retry options.
I think that this depends on which API you want to call and for what data.
For example, Facebook limits their API call to 200 requests per hour and per user. So if your app grows, and you are using their OAuth implementation correctly, you shouldn't be limited here.
Now, what data do you need? Do you really need to make all these calls? Is the information you call somewhat storable on any of your server?
Let's imagine that you need to display an Instagram feed on a website. So at each visitor request, you reach Instagram to get the pictures you need. And when your app grows, you reach the API limit because you have more visitors than what the Instagram API allows. In this case, you should definitely store the data on your server once per hour, and let your users reach your database rather than Instagram's one.
Now let's say that you need specific information for every user at each request. Isn't it possible to let that user handle his connection to the API's? Either by implementing the OAuth 2 flow of the API or by asking the user their API informations (not very secure I think...)?
Finally, if you really can't change the way you are working now, I don't see any other options that the ones you listed here.
EDIT: And Finally, as #Eric Stein stated in his comment, some APIs allow you to rise your API limit by paying (a lot of SaaS do that), so if your app grows, you should afford to pay for those services (they are delivering value to you, it's fair to pay them back)
We're transitioning to a managed service provider for our IT service desk and deskside and we're working out the details of their SLAs. Many of the SLAs are based on ticket status. An example of this is the following:
"Measures the amount of time it takes to assess, schedule, test, and package application packages before they are available for User Acceptance Testing."
My first thought was to try using SLAs to measure this, as they neatly tie together calendars and priorities, but I'm having a really hard time finding any information about how I could do this.
Now I'm looking into using the TKSTATUS.STATUSTRACKING attribute on tickets, but I believe this just tracks straight 24/7 time instead of taking into consideration any calendars.
Has anyone tried this before? Any suggestions?
We are in a similar process, but we are measuring time on site of vendors for our work orders. Opposed to using any SLAs we have diffrent statuses which mark different events. Then when we want to know how long it took for an event to finish, we look to see the time required to change status in the wostatus table.
Basically, my question is to get a feel for what time of day you do things like DNS transfers when moving sites across servers, and provide updates to sites. Does anyone do this during the day, or do most developers do this at night?
I think different approaches can be valid, depending on your situation, the actual change to be performed, how often a change happens and what your users expect.
If you do it during the day, you potentially have a longer time to fix issues if there are any, including if they are issues that need to be dealt with your service provider (which might be more responsive during business hours than at night or during week-ends).
If you enjoy working at night or during week-ends and you don't rely on any service provider to solve any issue related to the move/migration you're asking about, doing it at this time might be less disruptive to your users.
But all that also depends on who are your actual users. More and more users are spread throughout the globe and can interact with your sites at any time of day or night.
Common systems administration practice, when dealing with big changes or migrations, usually recommends not doing it just before off-time (be it night, or week-end, or holidays), unless you're sure that you (or the team) is available to work to fix issues. Basically, you need to plan for issues, because they always happen.
Something else to consider is any SLA you, or your company, might have with the users. In that case, you might need to perform your maintenance action outside of business hours, or with provisional notice, or risking any penalty to be applied.
A proper answer to the question you're asking really depends on your situation and the business you're in.
Starting out with Windows Azure, but how do I know which is better to handle web-traffic and a background processor. Would 2x extra small instances be better or a single small instance.
If I were to use a small instance, I would make the background processor in the web-role, what are the cons of doing it this way?
In future this would also apply where multiple small instances or fewer big instances.
Is there some sort of tool to help decide which way I'll be able to get the best bang for my buck etc.?
I know that for the Microsoft's SLA to be met 2x instances need to be running.
It is better to have 2 extra-small rather that 1 small instance as far service availability is concerned. That being said there are multiple gotchas:
You need to put your 2 VMs into 2 distinct upgrade domains (done in role definition file).
Your app needs to support multi-VM, aka not rely on non-shared session state.
Better availability does not mean better performance, in particular, local cache is basically halved.
Size of cache and overall difficulties at spreading an app over many small VMs typically explain why most dev stick to a single but larger VM until they reach a point that really calls for scaling out (which is likely to never happen for most apps anyway).
For SLA purposes, you need at least two instances, as Joannes alluded to when talking about service availability. Other things to consider:
It's easy to handle background tasks in a web role - you get the same OnStart() and Run() as a worker role.
When scaling, remember that, if you combine functionality into a single role, it all scales together as a single unit. So, if your background processing is being starved out due to excessive web traffic, you'll want to consider splitting them into separate roles
Extra Small instances have shared CPU. More importantly, they're going to have less network throughput. A Small instance has approx. 100Mbps. An Extra Small instance is a fraction of that (I'll need to look up the number). And... memory is 768MB vs. 1.75GB for a Small.
If you have an MSDN subscription, the included Windows Azure subscription comes with 1,500 CPU-hours monthly. But... that excludes Extra Small instances. You'll pay for those. Be sure to use Small when using your MSDN-supplied account. Edit: MSDN allowances are now friendly to extra-small instances.
I guess we can't really know without usage figures and more info and even then I think only time will tell but...
Why not sign up for one of the free trial accounts that gives you an extra small instance. See if your app copes well enough then when it goes live get a second one for load balancing, SLA etc.
If it doesn't cope then get a bigger one - but I'd still be inclined to go for a second one - unless you don't care if it becomes unavailable at random times. MS will apply security patches and will reboot your instances without asking so the second instance will prevent your site being unavailable as they'll update them separately.
It doesn't look particularly challenging to upgrade to bigger instances anyway should it become a sell-out.
If you've got an MSDN subscription (premium level, I think) then you get enough free hours to run 2 small instances.
From what little I've seen I don't think there's any real 'con' in adding the background processor. After all you're paying for a whole machine so you might as well make it work for it's money. That was kind of the impression they gave at the recent tech days thingy I saw.
Try it and see...