Description:
In our Performance Test environment, we have used a combination of following Azure VMs:
D4s_v3, Standard F8s , D32s_v3.
But our Production environment has a different set of high performant VMs like Standard_DS5_v2.
With the current VMs available in Performance Test environment, I want to carry out test runs and then use these results to see if Production env. can perform properly or not.
To compare different VM types, I tried using this URL as a reference.
Query:
My question is, will it be correct to use the average score as a scaling factor to extrapolate the results for production environment? If not, what other scaling factor can I use to extrapolate the results.
You cannot extrapolate the test results and cannot predict what would be system performance in production by running a test against a scaled down environment.
What you can do is:
Run an integration test to ensure that the system is doing what it is supposed to be doing
Run a load test to see if the scaled down system can support anticipated number of users
Run a stress test to identify the first bottleneck, it might be the case it is not connected with lack of resources like CPU or RAM or whatever
Run a soak test to identify possible memory leaks
Related
I'm running three MEAN stack programmes. Each application receives over 10,000 monthly users. Could you please assist me in finding an EC2 instance for my apps?
I've been using a "t3.large" instance with two vCPUs and eight gigabytes of RAM, but it costs $62 to $64 per month.
I need help deciding which EC2 instance to use for three Nodejs applications.
First check CloudWatch metrics for the current instances. Is CPU and memory usage consistent over time? Analysing the metrics could help you to decide whether you should select a smaller/bigger instance or not.
One way to avoid too unnecessary costs is to use auto scaling groups and load balancers. By using them and finding and applying proper settings, you could have always right amount of computing power for your applications.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html
https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html
Depends on your applications. If your apps need more compute power or more memory or more storage? Deciding a server is similar to installing an app on system. Check what are basic requirements for it & then proceed to choose server.
If you have 10k+ monthly customers, think about using ALB so that traffic gets distributed evenly. Try caching to server some content if possible. Use unlimited burst mode of t3 servers if CPU keeps hitting 100%. Also, try to optimize code so that fewer resources are consumed. Once you are comfortable with ec2 choice, try to purchase saving plans or RIs for less cost.
Also, do monitor the servers & traffic using Cloudwatch agent, internet monitor etc features.
Azure Load Testing (Preview) has a limit of 250 threads per Engine Instance. Each test can only run up to 45 Engines, so the theoretical maximum number of concurrent sessions that can be run is 11,250.
I need to simulate 120k concurrent users. Is there a suitable workaround to do this with ALT?
I think it's better to contact Azure Support for the details.
In the mean time be informed that you don't need 3rd-party providers to run a JMeter test without any limits, given you have a subscription you can
kick off several virtual machines
install JMeter on these machines
run your test in distributed mode
which is pretty much what Azure Load Testing is doing under the hood
I need to compare network latency for 2 clients half the globe away from each other and visualize the response time history for each location the request comes from side-by-side. I've been researching load and performance testing tools and found Locust to be very convenient. Is there a way I can achieve my goal with Locust in a quick/standard/non-hacky way?
Whether you mean you need 2 clients in different locations running the same task to compare or 2 clients in 2 different locations running 2 different tasks, Locust can handle either of those scenarios.
Check out the Tasks section of the documentation for details about how to write tasks. You can write a single task that does both of what you need, two different tasks that are randomly selected for each user that starts, write two tasks and have them be weighted in their randomness (including making one weighted at 0 so it's never run, effectively turning one task off so only the other one is run at that time), and many other options. Which method is best will depend on exactly what you need and how you want to do it. It may take some experimentation to determine what's best.
As for running in multiple locations, you can run the test separately in different places and compare results, or Locust can run distributed so you can have workers in multiple locations running at the same time. You may also want to look at using Docker, which in some ways can make running in different locations easier if you use AWS, Azure, GCP or whatever other cloud provider to spin up instances to run on.
I built an app on node.js using Docker and I'm not sure how to scale it on a Kubernetes cluster so that I take the most out of my cluster hardware.
From a performance perspective which of the following is better:
clusterize my node app and run as many containers as needed
or
just run as many containers as needed without clustering ?
When I say clustering I mean this https://nodejs.org/api/cluster.html
My app is a simple CRUD Api backed by mongoDB. We estimate that it will have 1000 concurrent users. Our cluster has 3 nodes.
The NodeJS cluster mechanism is useful to allow NodeJS to more effectively use greater than a single core, so depending on your code it may benefit you, but it's highly dependent on your code and the various dependencies and how well they work (or not) with clustering.
As a general practice, if you can break your containers down into nicely parallelized efforts that can be run as pods within kubernetes, then I'd recommend the following as a process to see what works for you:
set up a single pod with your code in it, and run a load test against it. Use the data that Kubernetes has from cAdvisor to characterize how much resources (cpu & memory) your pod likes to have.
set a resource limit for cpu and memory based on what you see above.
run a load test to validate what your single pod handles in terms of scale
And from there, you have a baseline where you can use Kubernetes to scale this horizontally to validate the 1000 user concurrent baseline you want to achieve.
There's a good talk on this process from the 2017 Kubecon called Load Testing Kubernetes: How to optimize your cluster resource allocation in production
Once you have a baseline, you can run a prototype out leveraging the clustering in your code, and then compare against the non-clustered version. If you do this, I'd double-check that any limits you set are > 1 core for CPU, or you'll be self-limiting outside of the NodeJS runtime to get access to multiple cores, which would defeat the purpose of using clustering.
Depending on what you're doing in your code, there may be significant re-work needed to enable clustering, as it wants to leverage its own worker concept, and it's not clear what frameworks you're using and if they'll fit reasonably into that structure.
The JMeter manual says
Your hardware's capabilities will limit the number of threads you can effectively run with JMeter. It will also depend on how fast your server is (a faster server makes JMeter work harder since it returns request quicker). The more JMeter works, the less accurate its timing information may become.
The question I want to ask is How many threads can I run from a single desktop machine and still get accurate enough results? However, I realize that's going to depend on what we define modern hardware as, or how fast my application/site is, etc.
So, the better (but harder to answer) question is, how to I profile JMeter to know when I've gone beyond the thread/user count that it's reasonable for a single machine to handle? Accurate deterministic methods are preferred, but anecdotal/rules-of-thumb are welcome.
I first suggest you follow best-practices for building JMeter test plans and running them:
http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
http://jmeter.apache.org/usermanual/best-practices.html
Then once your test plan is built, baseline it on the JMeter machine:
Monitor CPU (don't exceed 50%), swap (ensure no swap in/out at all)
Check GC for no long pauses
And don't forget issues which make Test wrong can come from lot of factors:
Networks issue between injector and application
TCP stack issues on JMeter injector
Components between the Injector and Application (Firewall, Load Balancer ...)