I am developing a website (basically a public facing site).
How can I simulate multiple users are surfing my site and doing various activities so that I can understand how my site will behave in a real time environment?
I am using Apache server and PHP.
As mentioned in previous posts, you will need a load test tool. The good news is that there are many tools and services in this field: open source load test ones like JMeter and Gatling, commercial one like Loadrunner and Neoload, the bad news is that you have to answer some questions and make some decisions.
One key decision you need to make is whether to test your application in the lab or in the cloud.
cloud based testing: Blazemeter, soasta, neustar ...
in-lab testing: JMeter, Gatling, Neoload, loadrunner, webperformer ....
In addition, you need to answer the following questioins:
how many virtual clients you want to emulate to stress the server
how much budget do you have
how complex is the web application
how much skills do the tester(s) have.
If you have high budget, complex web application and testers with good skills (like a developer), you can consider Loadrunner, NeoLoad.
If you have low budget but your tester(s) have good skills, you can consider Jmeter and Gating.
If you need to emulate lots of virtual clients (say 10000) to stress your complex web site and your tester(s) don't have the skill of a developer/programer, you may want to consider NetGend. There is a blog site where you can find out how complex performance testing can be (like filling HTML forms, extract values from JSON messages etc) and how easy it is on NetGend platform. By the way, you don't need high budget for NetGend.
Good lucky in your load/performance testing!
What you want is a load testing tool, there are several but id check out Neoload. You could also use Selenium and the various way to run selenium tests automatically
As this is your first time engaging in this task you would be well advised to find someone who has been there, done that and developed the battle scars from this activity. It is not a trivial effort to performance test a piece of software. If you listen to the traditional software vendors they will tell you that "any business analyst can use this tool and be effective" as if the tool is 85-95% of the skills you will need to be able to successfully performance test an application or a site. This is marketing foo to remove barriers to a sale.
In actuality the tool you select is anywhere from 5-15% of the total skill set you will need to be successful. Also, if the financial risk of failure is sufficiently high to warrant a performance test then it almost matters not which tool you pick, for the cost of the tool and the expertise will be dwarfed by your financial risk of not scaling.
If you don't have time to develop the skills or enough lead time to get a solid performance tester then you may want to consider some of the managed services offerings in the market, such as SOASTA, which can provide the expertise and the tool bundled within the deal. Here are some things you will want to look at in advance of any test (common issues)
Load Balancer misconfiguration resulting in distorted load to one node
Not appropriately managing your cache age for your static resources (.jpg, .css, ...) resulting in higher than expected load
All of your lookup queries to the database should be index optimized. Use a database profiler to check this
Holding onto resources too long. If your 95th percentile page to page request delay is five minutes then don't set your timeout at 30 minutes or 90 minutes for the HTTP session. This will hold onto resources far too long for the dead session. I use a rule of thumb of 95th percentile value times 1.5.
if this is a shopping site then don't hand out a default cart to everyone who shows up. Make sure they are on the revenue path before you hand them a cart, such as looking at the cart or placing something in the cart. Otherwise you have just built a 1:1 relationship with every customer and just about every piece of your architecture from web server to app server to the database server where the cart is created and managed
Also on the cart front, implement a 100x100 rule. If someone has a 100 items in the cart pick up the phone and call them to personalize the sale. If you have persistent carts which never expire then consider implementing a 100 day rule for evacuating from the cart items of that age or killing a cart altogether which hasn't been touched in that period. These people are clearly not on the revenue path.
Consider your design for ecommerce. Every step between add to cart and checkout is an opportunity to abandon the sale. The fewer the steps the greater the conversion rate: This is the genius behind the Amazon one click checkout. Minimize your number of steps and you will see a higher revenue flow as a result.
I would recommend gatling for load testings. It's scala based but a recorder is provided to generate workload test cases.
=> http://gatling-tool.org/
It's important to set some dimensions for test tool assessment to simulate user traffic:
SLA details (performance goals such as pages per second (PPS), http request per second (HPS), throughput, CPU usage etc.)
Virtual user size to simulate (You will need this info to decide number of
slave PCs/VMs, in other words that's called load generators, according to virtual user number
range)
Maintenance cost of scripts over changes
Effort to develop and execute test scripts
Number of test scenarios
Scheduling needs (You may want to schedule tool in regular basis or
execute test on-demand if needed)
Budget for test tool licenses and ROI (return of investment) calculations (Price, tool
expertise cost, utilizaton of test tool on other web applications etc)
Metrics provided by test tools
Monitoring requirements of network, servers and client
Integration with current test infrastructure (If HP ALM exist, you may be interested in Loadrunner)
If you are in hurry and don't have time to evaluate which tool to select, you may start with JMeter.
Selenium can be used for test automation of regression tests, I would like to highlight it's not effective for performance test due to its API. Sahi is another option for test automation.
I think that you are definitely looking for a load testing tool like Blazemeter. I recently discover this webinar which shows you how to do load testing on your application using a PaaS provider as a development and runtime environment, where you can deploy your application to run the load tests. They combine Blazemeter with a monitoring tool, New Relic in this case, to see the way in which you detect the new users surfing on your website. It is really cool and very interesting since you can know what is the performance of your application with a specific infrastructure.
Simple.
Set up server / application monitoring - New Relic is the easiest and most powerful. Free for 14 days.
Record a typical user's activity - Use JMeter to set up a proxy on your laptop and route web requests, mobile app usage etc through it. Sounds difficult but it's really easy. JMeter can act as the man-in-the-middle and capture all the requests sent by the browser/app to the server(s).
Now you "clone" the above user as many times as you need/could and blast the server. Initially you'd run the load test from your dev machine. Mine can take up to 80 concurrent users before cpu/ram runs out. Beyond this level explore BlazeMeter (free 50 concurrent users), jmeter-ec2 script (free), flood.io etc. Upload your script and blast away at your server. Ideally you should run incremental stress tests at your server. 10 users, 50 users, 100 users, 200 users etc.
Analyse, fix issues & ramp up the stress - In between each stress blast, go over your new relic. How is the application & server performing? What's failing? How are the alerts working?
If you are also searching for UI-Testing, you should check out Sahi Pro 6 automated testing tool, it also can be integrated with Jenkins.
==> http://sahipro.com/
It is really easy to record user actions with it on any browser and just playback the recorded scripts.
You can run scripts simultaneously on multiple browsers, thus simulate multiple users browsing your page:
https://sahipro.com/docs/using-sahi/playback-desktop.html#Distributed%20Runs%20-%20More%20Information
Related
I want to know how to perform Load Testing.
Requirement is
Total number of concurrent users at any given time are 5000
Request a session once every 40 seconds on average
Each session requires the web application consume an average 12Kb download bandwidth.
Note:
We need to perform load testing for a newly developed web application to assess the data rate required for the hosting webserver.
Your question is too broad because the implementation will be different depending on the load testing tool choice, I would recommend taking the following steps:
Get familiar with the web applications performance testing general concept, i.e. Performance Testing Guidance for Web Applications
If time allows it would be good to read ISTQB Performance Testing Specialist Syllabus as well
Then you need to choose a load testing tool, if there is no standard one in your organization you need to either acquire a commercial license or go for free and open source solution, an example comparison of the most outstanding/popular free and open source load testing tools: Open Source Load Testing Tools: Which One Should You Use?
Once you choose the tool then consult its documentation with regards to implementing your scenario or ask a tool-specific question using relevant SO tags
I am currently working on multiple Load testing tool such as Jmeter, LoadRunner and Gatling.
All above tool works upon protocol level user load testing except TrueClient protocol offered by LoadRunner. Now something like real browser testing is in place which is definitely high on resources consumption tools such as LoadNinja and Flood.IO works on this novel concept.
I have few queries in this regards
What will be the scenario where real browser based load testing fits perfectly?
What real browser testing offers which is not possible in protocol based load testing?
I know, we can use Jmeter to Mimic browser behavior for load testing but is there anything different that real browser testing has to offer?
....this novel concept.....
You're showing your age a bit here. Full client testing was state of the art in 1996 before companies shifted en masse to protocol based testing because it's more efficient in terms of resources. (Mercury, HP, Microfocus) LoadRunner, and (Segue, Borland, Microfocus) Silk, and (Rational, IBM) Robot, have retained the ability to use full GUI virtual users (run full clients using functional automation tools) since this time. TruClient is a recent addition which runs a full client, but simply does not write the output to the screen, so you get 99% of the benefits and the measurements
What is the benefit. Well, historically two tier client server clients were thick. Lots of application processing going on. So having a GUI Virtual user in a small quantity combined with protocol virtual users allowed you to measure the cost/weight of the client. The flows to the server might take two seconds, but with the transform and present in the client it might take an addtional 10 seconds. You now know where the bottleneck is/was in the user experience.
Well, welcome to the days of future past. The web, once super thin as a presentation later, has become just as thick as the classical two tier client server applications. I might argue thicker as the modern browser interpreting JavaScript is more of a resource hog than the two tier compiled apps of years past. It is simply universally available and based upon a common client-server protocol - HTTP.
Now that the web is thick, there is value in understanding the delta between arrival and presentation. You can also observe much of this data inside of the performance tab of Chrome. We also have great w3c in browser metrics which can provide insight into the cost/weight of the local code execution.
Shifting the logic to the client also has resulted in a challenge on trying to reproduce the logic and flow of the JavaScript frameworks for producing the protocol level dataflows back and forth. Here's where the old client-server interfaces has a distinct advantage, the protocols were highly structured in terms of data representation. So, even with a complex thick client it became easy to represent and modify the dataflows at the protocol level (think database as an example, rows, columns....). HTML/HTTP is very much unstructured. Your developer can send and receive virtually anything as long as the carrier is HTTP and you can transform it to be used in JavaScript.
To make it easier and more time efficient for script creation with complex JavaScript frameworks the GUI virtual user has come back into vogue. Instead of running a full functional testing tool driving a browser, where we can have 1 browser and 1 copy of the test tool per OS instance, we now have something that scale a bit more efficiently, Truclient, where multiple can be run per OS instance. There is no getting around the high resource cost of the underlying browser instance however.
Let me try to answer your questions below:
What will be the scenario where real browser based load testing fits perfectly?
What real browser testing offers which is not possible in protocol based load testing?
Some companies do real browser based load testing. However, as you rightly concluded that it is extremely costly to simulate such scenarios. Fintech Companies mostly do that if the load is pretty less (say 100 users) and application they want to test is extremely critical and such applications cannot be tested using the standard api load tests as these are mostly legacy applications.
I know, we can use JMeter to Mimic browser behaviour for load testing but is there anything different that real browser testing has to offer?
Yes, real Browsers have JavaScript. Sometimes if implementation is poor on the front end (website), you cannot catch these issues using service level load tests. It makes sense to load test if you want to see how well the JS written by the developers or other logic is affecting page load times.
It is important to understand that performance testing is not limited to APIs alone but the entire user experience as well.
Hope this helps.
There are 2 types of test you need to consider:
Backend performance test: simulating X real users which are concurrently accessing the web application. The goal is to determine relationship between increasing number of virtual users and response time/throughput (number of requests per second), identify saturation point, first bottleneck, etc.
Frontend performance test: protocol-based load testing tools don't actually render the page therefore even if response from the server came quickly it might be the case that due to a bug in client-side JavaScript rendering will take a lot of time therefore you might want to use real browser (1-2 instances) in order to collect browser performance metrics
Well-behaved performance test should check both scenarios, it's better to conduct main load using protocol-based tools and at the same time access the application with the real browser in order to perform client-side measurements.
I'm a web developer and completely new to Loadrunner suite.
Our client has already provided us with some Loadrunner actions, that I need to run them to test a site that is hosted on the company's intranet that I'm currently working.
The computer I'm using can not handle more than 7 vusers, therefore I was requested to use Amazon EC2 for load generators.
Before I request my company to be charged with Amazon services I need to know, would I be able to test our internal page from my computer exactly as I do with the load generator on my localhost, or the page that will be tested needs to be publicly accessible from the internet?
Any feedback will be appreciated. Thanks.
Please read carefully what James wrote. You said you are a web developer so the task that was given to you is roughly equivalent to "write a new DB access layer".
You didn't mention which protocol you are using but I will assume TruClient (based on the 7 vUsers per machine). I will also assume you are using the latest version of LoadRunner or at least something from the 12.6X family.
1) You already have a solution for AWS out of the box in the form of StormRunner (https://www.microfocus.com/en-us/products/stormrunner-load-agile-cloud-testing/overview). If you want to test if the solution works for you please request a couple of execution hours from the sales team and try it. If your company has a valid license for LoadRunner I don't think this will be an issue.
2) You have a simple integration into the controller application for EC2 and alike. In the controller go to Tools->Manage cloud accounts. If you run a small test the cost should not be too great I assume.
3) If you are a developer, we have a new offering called TruWeb which is a transport level protocol which should be more developer friendly. It will be able to run much more users per machine so you will be able to use it to test on EC2 micro machine (free tier). The caveat is that you will have to write some JavaScript code and not be able to reuse the actions given to you. You can download TruWeb from here - https://marketplace.microfocus.com/appdelivery/content/truweb and it comes with the LoadRunner installation out of the box since 12.58. If you need further assistance with TruWeb feel free to email us to - truweb_forum#microfocus.com
I hope this will give you some directions.
a) You need training. This is not a discipline that someone is socially promoted to and finds success
b) Expect that it will take at least six months to begin delivering value in this field, longer if you are not working with a mentor
c) This is a question of application communication architecture. Architecture is one of the foundation skills for a performance tester/engineer/architect.
d) It is not recommended that you use the controller as a load generator. It is not recommended that you use just one load generator. Both of which will cause your test to fail an audit from a more mature testing firm. Go with a minimum of three, two for primary load, one for a control set of a single virtual user of each type. Design your tests to allow for the examination of Control timing records compared to the global set to understand if you have an application issue or a load generator issue as part of your test design.
e) You will need to coordinate with your network team for two reasons. One, you may need to open outbound ports (covered in documentation) to allow your controller to communicate with your load generators. Two, you absolutely will have to coordinate a tunnel from the outside internet to your internal applications under test. Expect that security will be paramount only our requests and no other requests through the tunnel. There are many mechanisms to address this, from a custom HTTP header to certificates. Speak with your network security professionals for the setup and configuration which you will be able to implement.
The self paced training for loadrunner is available for download. It takes about three days to go through. This is the absolute minimum before you pick up this tool in anger. Ideally, you would go through training with a certified instructor and be paired with a mentor for a period. The length of time for the mentor is directly related to the number of foundation skills which you bring to the table.
I have to test a bulk emailing website (like MailChimp). I am a manual tester. Now how can I perform a performance test so that I can know when the website will be crash/down ?
Web applications performance testing basically assumes simulating normal usage of the website by real users. The overall process should look something like:
Consider one of the load testing tools. If you don't have a corporate standard or ready solution there is quite a number of free and open source load testing tools. If you are uncertain which one to choose check out Open Source Load Testing Tools: Which One Should You Use? article which describes and compares the most popular and advanced solutions which are currently available.
The majority of load testing tools come with record-and-replay functionality so the next step would be recording your use cases / test scenarios to build up a load test "skeleton"
After that most probably you will need to perform correlation - the process of identifying dynamic parameters, cookies, headers, etc. and handling them by extracting changing parts from the previous response and replacing recorded hard-coded values
Normally you will also require parameterization, i.e. configure your test to use different credentials for different virtual users.
Once your test is ready you can run it with 1-2 users and iterations and inspect request and response details to ensure that your test works as expected.
When you're happy with your test behavior you can start **gradually* adding more and more virtual users unless response time exceeds acceptable maximum or errors start occurring or website crashes, whatever comes the first.
When you discover the maximum amount of virtual users and collect associated performance metrics it would also be good to identify the bottleneck
You need to be very explicit about what you are testing. Are you testing the scheduling engine? Are you testing the SMTP relay? Are you testing the SMTP relay with specific max out of spec conditions delays in connecting to a downstream relay? Are you testing re-queues before your SMTP throughput drops to zero? It makes a significant difference in your test setup.
Also, it would be worthwhile for you to read the RFCs on SMTP before this test. Email is designed to be resilient across less-than-perfect connections, but this would result in a slowed SMTP relay over time.
< soapbox >
Also, as a manual tester you should never be asked to do this by your management unless they have committed to your training and a mentor to assist you in the effort. Any other path and they are interested in a check box or billing, but not a reduction in risk
< /soapbox >
Say I have a bunch of webservers each serving 100's of requests/s, and I want to see real time stats like:
Request rate over last 5s, 60s, 5 min etc
Number of unique users seen again per time window
Or in general for a bunch of timestamped events, I want to see real-time derived statistics - what's the best way to go about it?
I've considered having each GET request update a global counter somewhere, then sampling that at various intervals, but at the event rates I'm seeing it's hard to get a distributed counter that's fast enough.
Any ideas welcome!
Added: Servers are Linux running Apache/mod_wsgi, with a Python (Django) stack.
Added: To give a sense of the event rates I want to track stats for, they're coming in at over 10K events/s. Even incrementing a distributed counter at that rate is a challenge.
You might like to help us try out the beta of our agent for application performance monitoring in Python web applications.
http://newrelic.com
It delves more into the application performance rather than just the web server, but since any bottlenecks aren't generate going to be the web server, but your application then that is going to be more useful anyway.
Disclaimer. I work for New Relic and this is the project I am working on. It is a paid product, but the beta means it is free for now with all features. Later when that changes, if you didn't want to pay for it, their is still a Lite subscription level which is free and which gives you basic web metrics reporting which still covers some of what you are after. Anyway, right now would be a great opportunity to make use of it to debug your performance while you can.
Virtually all good servers provide this kind of functionality out of the box. For example, Apache has the mod_status module and Glassfish supports JMX. Furthermore, there are many commercial packages for monitoring clusters, such as Hyperic and Zenoss.
What web or application server are you using? It is difficult to provide a solution without that information.
Look at using WebSockets, their overhead is much smaller than a HTTP request, they are very well suited to real-time web applications. See: http://nodeknockout.com/ for Node based websocket examples.
http://en.wikipedia.org/wiki/WebSocket
You will need to run a daemon if you want to run it on your apache server.
Also take a look at:
http://kaazing.com/ if you wan't less hassle, but are willing to fork out some cash.
On the Windows side, Perfmonance monitor is the tool you should investigate.
As Jared O'Connor said, you should precise what kind of web server you want to monitor.