Starting out with Windows Azure, but how do I know which is better to handle web-traffic and a background processor. Would 2x extra small instances be better or a single small instance.
If I were to use a small instance, I would make the background processor in the web-role, what are the cons of doing it this way?
In future this would also apply where multiple small instances or fewer big instances.
Is there some sort of tool to help decide which way I'll be able to get the best bang for my buck etc.?
I know that for the Microsoft's SLA to be met 2x instances need to be running.
It is better to have 2 extra-small rather that 1 small instance as far service availability is concerned. That being said there are multiple gotchas:
You need to put your 2 VMs into 2 distinct upgrade domains (done in role definition file).
Your app needs to support multi-VM, aka not rely on non-shared session state.
Better availability does not mean better performance, in particular, local cache is basically halved.
Size of cache and overall difficulties at spreading an app over many small VMs typically explain why most dev stick to a single but larger VM until they reach a point that really calls for scaling out (which is likely to never happen for most apps anyway).
For SLA purposes, you need at least two instances, as Joannes alluded to when talking about service availability. Other things to consider:
It's easy to handle background tasks in a web role - you get the same OnStart() and Run() as a worker role.
When scaling, remember that, if you combine functionality into a single role, it all scales together as a single unit. So, if your background processing is being starved out due to excessive web traffic, you'll want to consider splitting them into separate roles
Extra Small instances have shared CPU. More importantly, they're going to have less network throughput. A Small instance has approx. 100Mbps. An Extra Small instance is a fraction of that (I'll need to look up the number). And... memory is 768MB vs. 1.75GB for a Small.
If you have an MSDN subscription, the included Windows Azure subscription comes with 1,500 CPU-hours monthly. But... that excludes Extra Small instances. You'll pay for those. Be sure to use Small when using your MSDN-supplied account. Edit: MSDN allowances are now friendly to extra-small instances.
I guess we can't really know without usage figures and more info and even then I think only time will tell but...
Why not sign up for one of the free trial accounts that gives you an extra small instance. See if your app copes well enough then when it goes live get a second one for load balancing, SLA etc.
If it doesn't cope then get a bigger one - but I'd still be inclined to go for a second one - unless you don't care if it becomes unavailable at random times. MS will apply security patches and will reboot your instances without asking so the second instance will prevent your site being unavailable as they'll update them separately.
It doesn't look particularly challenging to upgrade to bigger instances anyway should it become a sell-out.
If you've got an MSDN subscription (premium level, I think) then you get enough free hours to run 2 small instances.
From what little I've seen I don't think there's any real 'con' in adding the background processor. After all you're paying for a whole machine so you might as well make it work for it's money. That was kind of the impression they gave at the recent tech days thingy I saw.
Try it and see...
Related
first of all would like to say a big thank you to all who is reading my questions, I really appreciate your time and hope to be able contribute back, second, I did see part of my question already asked in another thread but it does not answer it plus my questions have a bit different angle, so here it goes:
Does not elasticity already include scalability? I see scalability and elasticity go as two separate features of the cloud in service promotions, is there a technical difference or is it just marketing play of terminology?
I have similar confusion about compute and networking, does not compute power already include networking, I saw it being briefly displayed as two separate advantages of cloud service
I will give it a try :) But it will largely be my understanding and less citing of provider documentation.
Elasticity vs. Scalability
I interprete elasticity as the capability to react to more or less daily variation in resource needs. Unlike reserved instances or your own server hardware "in the basement" the cloud provider offer both the resources and the managment tools to let you use varying amounts of compute, network , ... resources from hour to hour or day to day.
So elasticity (in my mind) solves the business need to react / adapt to changing demand that might follow a pattern like day / night or season / off-season but might be relatively stable from year to year or even week to week.
Scalability in my mind is more than everything the ability of these "hyper-scalers" to allow customers / you to grow your system continously and almost with no upper limit. So I would say the average (i.e.) weekly usage can go up every week for months after months and you wouldn't run out of upgrade options with the cloud providers to help you serve more and more requests.
Compute and Networking
The cloud uses "software defined networking" which abstracts all that hardware stuff like switches and routers from you as a user and offers connectivity options that would be hard to realize on your own / with traditional networking. So the networking capabilities of a major cloud provider are a feature set of their own, with lots of room for system improvements and capabilities. Therefor it is designed, serviced, billed... separately from other service classes like compute or storage.
A simple example or illustration of that might be a virtual machine (or multiple) that on their own as stand alone compute resources might have a network interface and a public ip attached to them. You can reach that machine, the machine can reach to internet (if you configure it that way) and you can install stuff on it. That's it - you have compute power.
But when you group virtual machines in i.e. application security groups and use these groups as objects in resources that allow, deny, redirect traffic internally or externally and maybe tunnel traffic to these compute resources to your on-premise resources (like in many cases Active Directory Domain Services) you start to use advanced networking capabilites. But obviously there's much more and networking can be one of the hardest parts of certification exams on cloud topics.
I'm interested in using Azure Functions for a piece of serverless code, but I would like to ensure that I am always within the free tier, so as to not incur any expenses (I'm okay with potential downtime, not really critical). How do I achieve this?
My function is limited to some domains I control, and possibly a resource used in GitHub readme (like a tracking pixel). How do I combat potential DDOS, and massive bill spikes?
I've seen other questions on how to manage fanout, scale etc, but none on setting hard limits. I'm still a student, so I'd rather stay exclusively in the free tier.
Note, by 'Free tier', I mean the 'Always Free' offering.
You cannont (as far as I know) set a hard limit.
What you can do is to reduce your functions ability to scale. So that it can just process a single request at a time, then depending on how long the request takes you will stay within the free tier.
https://nullable.online/2018/11/20/how-to-throttle-a-azure-function-hosted-on-a-consumption-plan/
I've come across many clients who aren't really able to provide real production data about a website's peak usage. I often do not get peak pageviews per hour, etc.
In these circumstances, besides just guessing or going with what "feels right" (i.e. making it all up), how exactly does one come up with a realistic workload model with an appropriate # of virtual users and a good pacing value?
I use Loadrunner for my performance/load testing.
Ask for the logs for a month.
Find the stats for session duration, then count the number of distinct IP's blocked by session duration.
Once you have the high volume hour, count the number of page instances. Business processes will typically have a termination page which is distinct and allows you to understand how many times a particular action takes place, such as request new password, update profile, business process 1, etc...
With this you will have a measurement of users and actions. You will want your stakeholder to take ownership of this data. As Quality assurance, we should not own both the requirement and the test against it. We should own one, but not both. If your client will not own the requirement, cascading it down to rest of the organization, assume you will be left out in the cold with a result they do not like....i.e., defects that need to be addressed before deployment to production.
Now comes your largest challenge, which is something that needs to be fixed with a process issue by your client.....You are about to test using requirements that no other part of the organization, architecture, development, platform engineering, had when they built the solution. Even if your requirements are a perfect recovery, plus some amount for growth, any defects you find will be challenged aggressively.
Your test will not match any assumptions or requirements used by any other portion of the organization.
And, in a sense, these other orgs will be correct in aggressively challenging your results. It really isn't fair to hold their designed solution to a set of requirements which were not in place when they made decisions which impacted scalability and response times for the system. You would be wise to call this out with your clients before the first execution of any performance test.
You can buy yourself some time. If the client does have a demand for a particular response time, such as an adoption of the Google RAIL model, then you can implement a gate before accepting any code for multi-user performance testing that the code SHALL BE compliant for a single user. It is not going to get any faster for two or more users. Implementing this hard gate will solve about 80% of your performance issues, for the changes required to bring code into compliance for a single user most often will have benefits on the multi-user front.
You can buy yourself some time in a second way as well. Take a look at their current site using tools such as Google Lighthouse and GTMetrix. Most of us are creatures of habit, that includes architect, developers, and ops personnel. We design, build, deploy to patterns we know and are comfortable with....usually the same ones over and over again until we are forced to make a change. It is highly likely that the performance antipatterns pulled from Lighthouse and GTMetrix will be carried forward into a future release unless they are called out for mitigation. Begin citing defects directly off of these tools before you even run a performance test. You will need management support, but you might consider not even accepting a build for multi-user performance testing until GTMetrix scores at least a B across the board and Lighthouse a score of 90 or better.
This should leave edge cases when you do get to multi-user performance testing, such as too early allocation of a resource, holding onto resources too long, too large of a resource allocation, hitting something too often, lock contention on a shared resource. An architectural review might pick up on these, where someone might say, "we are pre-allocating this because.....," or "Marketing says we need to hold the cart for 30 minutes before de-allocation," or "...." Well, you get the idea.
Don't forget to have the database profiler running while functional testing is going on. You are likely to pick up a few missing indexes or high cost queries here which should be addressed before multi-user performance testing as well.
You are probably wondering why am I pointing out all of these things before your performance test takes place. Darnit, you are hired to engage in a performance test! The test you are about to conduct is very high risk politically. Even if it finds something ugly, because the other parts of the organization did not benefit from the requirements, the result is likely to be rejected until the issue shows up in production. By shifting the focus to objective measures even before you need to run two users in anger together there are many avenues to finding and fixing performance issues which are far less politically volatile. Food for thought.
For small app they are no problem.
But for apps with traffic you can hit limits easily.
Http protocol is req-res driven. Just because your backend is stuck with limit, you can't really wait to send respond back until rate limit allows you to resume making your api calls.
What do you do?
I can think of several scenarios:
Wait it out: while it sucks, but sometimes it's easy fix, as you don't need to do anything.
Queue it: this a lot of work oppose to making just api call. This requires that first you store it in database, then have background task go through database and do the task. Also user would be told "it is processing" not "it's done"
Use lot of apis: very hacky... and lot of trouble to manage. Say you are using amazon, now you would have to create, verify, validate like 10 accounts. Not even possible for where you need to verify with say domain name. Since amazon would know account abc already owns it.
To expand on what your queueing options are:
Unless you can design the problem of hitting this rate limit out of existence as #Hammerbot walks through, I would go with some implementation of queue. The solution can scale in complexity and robustness according to what loads you're facing and how many rate limited APIs you're dealing with.
Recommended
You use some library to take care of this for you. Node-rate-limiter looks promising. It still appears you would have to worry about how you handle your user interaction (make them wait, write to a db/cache-service and notify them later).
"Simplest case" - not recommended
You can implement a minimally functioning queue and back it with a database or cache. I've done this before and it was fine, initially. Just remember you'll run into needing to implement your own retry logic, will have to worry about things like queue starvation **. Basically, the caveats of rolling your own < insert thing whose implementation someone already worried about > should be taken into consideration.
**(e.g. your calls keep failing for some reason and all of a sudden your background process is endlessly retrying large numbers of failing queue work elements and your app runs out of memory).
Complex case:
You have a bunch of API calls that all get rate-limited and those calls are all made at volumes that make you start considering decoupling your architecture so that your user-facing app doesn't have to worry about handling this asynchronous background processing.
High-level architecture:
Your user-facing server pushes work units of different type onto different queues. Each of these queues corresponds to a differently rate-limited processing (e.g. 10 queries per hour, 1000 queries per day). You then have a "rate-limit service" that acts as a gate to consuming work units off the different queues. Horizontally distributed workers then only consume items from the queues if and only if the rate limit service says they can. The results of these workers could then be written to a database and you could have some background process to then notify your users of the result of the asynchronous work you had to perform.
Of course, in this case you're wading into a whole world of infrastructure concerns.
For further reading, you could use Lyft's rate-limiting service (which I think implements the token bucket algorithm to handle rate limiting). You could use Amazon's simple queueing service for the queues and Amazon lambda as the queue consumers.
There are two reasons why rate limits may cause you problems.
Chronic: (that is, a sustained situation). You are hitting rate limits because your sustained demand exceeds your allowance.
In this case, consider a local cache, so you don't ask for the same thing twice. Hopefully the API you are using has a reliable "last-modified" date so you can detect when your cache is stale. With this approach, your API calling is to refresh your cache, and you serve requests from your cache.
If that can't help, you need higher rate limits
Acute: your application makes bursts of calls that exceed the rate limit, but on average your demand is under the limit. So you have a short term problem. I have settled on a brute-force solution for this ("shoot first, ask permission later"). I burst until I hit the rate limit, then I use retry logic, which is easy as my preferred tool is python, which supports this easily. The returned error is trapped and retry handling takes over. I think every mature library would have something like this.
https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html
The default retry logic is to backoff in increasingly big steps of time.
This has a starvation risk, I think. That is, if there are multiple clients using the same API, they share the same rate limit as a pool. On your nth retry, your backoff may be so long that newer clients with shorter backoff times are stealing your slots ... by the time your long backoff time expires, the rate limit has already been consumed by a younger competitor, so you now retry even longer, making the problem worse,although at the limit, this just means the same as the chronic situation: the real problem is your total rate limit is insufficient, but you might not be sharing fairly among jobs due to starvation. An improvement is to provide a less naive algorithm, it's the same locking problem that you do in computer science (introducing randomisation is a big improvement). Once again, a mature library is aware of this and should help with built-in retry options.
I think that this depends on which API you want to call and for what data.
For example, Facebook limits their API call to 200 requests per hour and per user. So if your app grows, and you are using their OAuth implementation correctly, you shouldn't be limited here.
Now, what data do you need? Do you really need to make all these calls? Is the information you call somewhat storable on any of your server?
Let's imagine that you need to display an Instagram feed on a website. So at each visitor request, you reach Instagram to get the pictures you need. And when your app grows, you reach the API limit because you have more visitors than what the Instagram API allows. In this case, you should definitely store the data on your server once per hour, and let your users reach your database rather than Instagram's one.
Now let's say that you need specific information for every user at each request. Isn't it possible to let that user handle his connection to the API's? Either by implementing the OAuth 2 flow of the API or by asking the user their API informations (not very secure I think...)?
Finally, if you really can't change the way you are working now, I don't see any other options that the ones you listed here.
EDIT: And Finally, as #Eric Stein stated in his comment, some APIs allow you to rise your API limit by paying (a lot of SaaS do that), so if your app grows, you should afford to pay for those services (they are delivering value to you, it's fair to pay them back)
I'm designing an architecture for leveraging Azure Search for multiple tenants. Since each tenant will have a slightly different schema my solution will require 1 index per tenant. This is easy enough to set up and I'm really liking what Microsoft has put together. However now that I am starting to think about on-boarding new tenants, monthly costs and scaling up the service I am starting to hit a few walls and wondering what my "best" option is.
Has anyone encountered this situation that can shed some light onto best practices? Here are the options as I see them now:
Option 1:
Spin up a new BASIC plan for every 5 tenants at a cost of $38/m for every 5 tenants ($7.60 per tenant per month).
Pros: Cheap to start.
Cons: Tenants are crippled by the limited performance and storage capabilities, I'll have to manage X number of services and ClientQueryKeys once I get past 5 indexes/tenants.
Option 2:
Spin up a new STANDARD S1 plan for every 50 tenants at a cost of $250/m for every 50 tenants ($5 per tenant per month).
Pros: Better performance, less services to manage as tenant counts increase
Cons: Much higher costs to start, still need to manage tenant-to-service relation once system has greater than 50 tenants, I'll have to manage X number of services and ClientQueryKeys once I get past 50 indexes/tenants.
Option 3:
Spin up a single STANDARD S2 plan that can be used for ALL tenants (assuming no cap on index count)
Pros: Better performance, no need to manage multiple services/client keys as tenant counts increase
Cons: Much higher costs to start, very little documentation on costs and limitations.
In all scenarios (aside from option 3, I'm assuming?) I would have to manage client keys across multiple services. So obviously having only one service with an infinite index count is ideal. However I am a startup (yes I am using BizSpark already) and the costs for search a very daunting when I may only have 1-5 tenants to start.
I've read that there is no way to easily migrate data between plans (without doing it manually or writing a script) so my first choice is likely to be my last. I would also prefer to only have to manage one service with one plan for all my tenant. Therefore I am leaning to option 3.
If option 3 is the best option:
Can I start on BASIC and scale up to S1 then S2 as needed, or is this not possible?
If BASIC cannot scale to S1 is it at least possible to scale from STANDARD S1 to S2 once I go past 50 tenants or will I need to manually manage this or start at S2?
What are my startup costs and/or costs per index/tenant on Standard S2?
Is my index limit infinite on S2?
If not, what is the index cap?
Are there any other options or caveats that I should consider?
S2 services work much better in multi-tenant scenarios. Not only they can fit more indexes (up to 200), but they also have more overall capacity so assuming exponential distribution of index sizes and loads, you get a better typical experience for your customers.
You're right that the cost of entry is higher.
Regarding the cons of S2, soon we're going to publish proper documentation and other supporting materials for it. In the meanwhile, feel free to contact me directly (Pablo DOT Castro AT the usual Microsoft domain) for more details.
If you think you'll have lots of indexes in the future (many 100s), we're also working on options for better multi-tenant support. We're not ready to announce the details yet but I'm happy to discuss if you get in touch with us.
Answering your specific questions:
1.Can I start on BASIC and scale up to S1 then S2 as needed, or is this not possible?
We don't currently support this. You'd have to create a new search service and migrate the indexes.
2.If BASIC cannot scale to S1 is it at least possible to scale from STANDARD S1 to S2 once I go past 50 tenants or will I need to manually manage this or start at S2?
No, it's not. We want to do this, just have not gotten to it yet.
3.What are my startup costs and/or costs per index/tenant on Standard S2?
Please get in touch with us and we can discuss pricing.
4.Is my index limit infinite on S2?
5.If not, what is the index cap?
No, S2 services are limited to 200 indexes/service.
6.Are there any other options or caveats that I should consider?
You've done a good analysis, I think you're on the right track. One thing you may want to consider is fairness. All indexes in the same service share the capacity you've provisioned for the service. If there's risk of unfair loads you might want to consider per-tenant throttling.