Anything otherthan Jmeter tool
In absolute majority of cases you don't need to test the serverless application itself as it is being run by 3rd-party provider, i.e. Microsoft or Amazon
The only thing it makes sense to do is to perform analysis of the "function" implementation in order to identify that it is well developed from code perspective:
uses optimal algorithms
doesn't have obvious memory leaks
uses underlying infrastructure efficiently from multithreading perspectice
All this can be done using source code analysis and profiling tools
If your question is broader and you're looking for a load testing tool other than JMeter - check out Open Source Load Testing Tools: Which One Should You Use? article which describes alternatives including but not limited to:
Grinder
Gatling
Tsung
However "classic" load testing tools are not applicable to FAAS applications
Related
I was wondering if there are any tools worth of trying to test dotnet core WebAPI performance during high load.
In the past I used jMeter of Apache, but configuring that alongside with TeamCity and dotnet core builds is a bit of pain.
I am looking for something that could deliver statistics, so automated run of tests can give me information if recent changes have or haven't decreased performance etc.
I also did a quick google, VisualStudio has something on board, but first of all it requires Enterprise edition of software, and I am not convinced if that tool is good enough.
Thank you
Take a look at https://github.com/dotnet/BenchmarkDotNet - it's a widely used and respected library.
If your goal is to test WebAPI you can create a HttpClient and use it for making calls to the API, wrap it in a benchmark and run it on TeamCity as a simple dotnet step.
The tool also can create reports in HTML which then can be added to TeamCity as a custom tab.
If it's not enough then you can extract performance metrics e.g. from PlainExporter and integrate it with TeamCity built-in Service Messages and create Custom Charts from those stats so TeamCity will help you track performance trends.
You could even act on those measures and e.g. fail the build if there is a significant performance degradation.
I want to be clear: I am asking about the case where I am not using any concurrency in my own implementation. I just want to know if the framework within which my backend will be invoked (ie google app engine) itself imposes thread-safety requirements on the code running on it.
Thank you!
P.S. as a related but separate question, is there any guidance on how to do multithreading in our own backend code (which then obviously needs to be appropriately thread-safe). Specifically, can we use java's standard executor services / thread pools, or there is some google-approved API? Thanks.
So google app engine or any other platform won't ensure thread safety for you because all the critical sections which your concurrency control techniques are trying to make thread safe are defined by the developer (you) and there is no way for the OS to know when reads and writes should occur. For the JVM, the two main concurrency libraries which are heavily used are the Guava libraries (made by google! ;-) ) and the Akka framework.
Both are great libraries, I've used both and have had a pleasure from their learning curves. I would also recommend looking into the Play Framework, they support the akka framework, and building reactive web apps is their main focus. If you're interested in learning some cool aspects of production frameworks, I would highly recommend learning and implementing Dropwizard (Google app engine doesn't support it, but the take aways you get from it definitely outweigh that con).
Please let me know if you have any questions!
I want to write one web application with node.js and MongoDB and I have got task to even test it. I would like to know if there are any tools like JMeter or anything else for load/stress testing of Node.js?
EDIT
My application is going to be information extraction kind of application and client expects extraction should not take more than 10 seconds for one document. Currently I have same application written in C# but its not scaling upto client's expectations. Then I came across this beautiful and fast Node.js. I think Node.js can help me alot.
Please enlighten !!!
Try nodeload: it's a collection of node.js modules for load testing HTTP services.
As a developer, you should be able to write load tests and get
informative reports without having to learn another framework. You
should be able to build by example and selectively use the parts of a
tool that fit your task. Being a library means that you can use as
much or as little of nodeload as makes sense, and you can create load
tests with the power of a full programming language. For example, if
you need to execute some function at a given rate, just use the
'nodeload/loop' module, and write the rest yourself
Just found out that this package is no longer under development so here are some active forks:
https://github.com/gamechanger/nodeload
https://github.com/Samuel29/NodeStressSuite
Why couldn't you test a node server with JMeter? For most load tests it doesn't matter what language your server is, you're just hitting it with a bunch of requests.
In any case, you could try loadtest which is implement in node.
Runs a load test on the selected HTTP or WebSockets URL. The API allows for easy integration in your own tests.
Edit:
This answer provides more options:
NodeJs stress testing tools/methods [closed]
Try artillery. Here are its features, the description of which is taken from the documentation:
Multiple protocols: Load test HTTP, WebSocket, Socket.io, Kinesis, HLS and more.
Scenarios: Support for complex scenarios to test multi-step interactions in your API or web app (great for ecommerce, transactional APIs, game servers etc).
Load testing & Functional testing: reuse the same scenario definitions to run performance tests or functional tests on your API or backend.
Performance metrics: get detailed performance metrics (latency, requests per second, concurrency, throughput).
Scriptable: write custom logic in JS, using any of the thousands of useful npm modules.
Integrations: statsd support out of the box for real-time reporting (integrate with Datadog, Librato, InfluxDB etc).
Extensible: write custom reporters, custom plugins, custom protocol engines etc.
and more! HTML reports, nice CLI, parameterization with CSV files.
What is the difference between the next terms, it can help a lot in interviews and general understanding.
Framerwork
Library
IDE
API
Framework
Some predefined architecture that a developer has chosen and which dictates how the application will be written. It usually already includes many concepts which helps the developer to concentrate on the domain of the application instead of the plumbing. This plumbing is provided by the framework. For example the .NET framework provides out-of-the-box tools that would allow you to talk to web servers, without even knowing the internals of the TCP/IP protocol (actually it helps knowing the internals but you get the point).
Library
A reusable compiled unit that can be redistributed and reused across various projects. Well not necessary compiled in case of dynamic languages.
IDE
It's the development environment where you create the other three parts (usually text editor), it might also include compiler and the possibility to execute, debug and see the output of the program in order to speed up the development process.
API
Application Programming Interface. This could mean many things but usually it is a set of functions given to the disposition of the developer and which perform specific tasks and work only in a specific context.
IDE is a tool for fast, easy and flexible development
An API is provided for an existing software. Using these third party applications can interact with main/primary application.
A framework or library are typically same. They are a common set of functionality for other software to use.
Ref: wiki for Framework, API
Framework: a collection of libraries and programming practices to provide general functionality for a program, so that it doesn't have to be rewritten. Typically a framework for an application program will handle user display and input, among other things. The intent is usually to hide the more complex functionality of an application, and to encourage a certain style.
Library: A piece of software to provide certain functionality to other programs that call it. Typically designed to be reusable and modular, so that a library can be distributed and be useful without its source code.
Integrated Development Environment: A integrated set of tools to write programs and turn them into finished products, usually including at least an editor, compiler, linker, and debugger. IDEs sometimes provide support for frameworks.
Application Programming Interface: A set of function calls and sometimes variable accesses available to a program, typically being the public interface of one or more libraries.
I just started getting into BizTalk at work and would love to keep using everything I've learned about DDD, TDD, etc. Is this even possible or am I always going to have to use the Visio like editors when creating things like pipelines and orchestrations?
You can certainly apply a lot of the concepts of TDD and DDD to BizTalk development.
You can design and develop around the concept of domain objects (although in BizTalk and integration development I often find interface objects or contract first design to be a more useful way of thinking - what messages get passed around at my interfaces). And you can also follow the 'Build the simplest possible thing that will work' and 'only build things that make tests pass' philosophies of TDD.
However, your question sounds like you are asking more about the code-centric sides of these design and development approaches.
Am I right that you would like to be able to follow the test driven development approach of first writing a unti test that exercises a requirement and fails, then writing a method that fulfils the requirement and causes the test to pass - all within a traditional programing language like C#?
For that, unfortunately, the answer is no. The majority of BizTalk artifacts (pipelines, maps, orchestrations...) can only really be built using the Visual Studio BizTalk plugins. There are ways of viewing the underlying c# code, but one would never want to try and directly develop this code.
There are two tools BizUnit and BizUnit Extensions that give some ability to control the execution of BizTalk applications and test them but this really only gets you to the point of performing more controled and more test driven integration tests.
The shapes that you drag onto the Orchestration design surface will largely just do their thing as one opaque unit of execution. And Orchestrations, pipelines, maps etc... all these things are largely intended to be executed (and tested) within an entire BizTalk solution.
Good design practices (taking pointers from approaches like TDD) will lead to breaking BizTalk solutions into smaller, more modular and testable chunks, and are there are ways of testing things like pipelines in isolation.
But the detailed specifics of TDD and DDD in code sadly don't translate.
For some related discussion that may be useful see this question:
Mocking WebService consumed by a Biztalk Request-Response port
If you often make use of pipelines and custom pipeline components in BizTalk, you might find my own PipelineTesting library useful. It allows you to use NUnit (or whatever other testing framework you prefer) to create automated tests for complete pipelines, specific pipeline components or even schemas (such as flat file schemas).
It's pretty useful if you use this kind of functionality, if I may say so myself (I make heavy use of it on my own projects).
You can find an introduction to the library here, and the full code on github. There's also some more detailed documentation on its wiki.
I agree with the comments by CKarras. Many people have cited that as their reason for not liking the BizUnit framework. But take a look at BizUnit 3.0. It has an object model that allows you to write the entire test step in C#/VB instead of XML. BizUnitExtensions is being upgraded to the new object model as well.
The advantages of the XML based system is that it is easier to generate test steps and there is no need to recompile when you update the steps. In my own Extensions library, I found the XmlPokeStep (inspired by NAnt) to be very useful. My team could update test step xml on the fly. For example, lets say we had to call a webservice that created a customer record and then checked a database for that same record. Now if the webservice returned the ID (dynamically generated), we could update the test step for the next step on the fly (not in the same xml file of course) and then use that to check the database.
From a coding perspective, the intellisense should be addressed now in BizUnit 3.0. The lack of an XSD did make things difficult in the past. I'm hoping to get an XSD out that will aid in the intellisense. There were some snippets as well for an old version of BizUnit but those havent been updated, maybe if theres time I'll give that a go.
But coming back to the TDD issue, if you take some of the intent behind TDD - the specification or behavior driven element, then you can apply it to some extent to Biztalk development as well because BizTalk is based heavily on contract driven development. So you can specify your interfaces first and create stub orchestrations etc to handle them and then build out the core. You could write the BizUnit tests at that time. I wish there were some tools that could automate this process but right now there arent.
Using frameworks such as the ESB guidance can also help give you a base platform to work off so you can implement the major use cases through your system iteratively.
Just a few thoughts. Hope this helps. I think its worth blogging about more extensively.
This is a good topic to discuss.Do ping me if you have any questions or we can always discuss more over here.
Rgds
Benjy
You could use BizUnit to create and reuse generic test cases both in code and excel(for functional scenarios)
http://www.codeplex.com/bizunit
BizTalk Server 2009 is expected to have more IDE integrated testability.
Cheers
Hemil.
BizUnit is really a pain to use because all the tests are written in XML instead of a programming language.
In our projects, we have "ported" parts of BizUnit to a plain old C# test framework. This allows us to use BizUnit's library of steps directly in C# NUnit/MSTest code. This makes tests that are easier to write (using VS Intellisense), more flexible, and most important, easier to debug in case of a test failure. The main drawback of this approach is that we have forked from the main BizUnit source.
Another interesting option I would consider for future projects is BooUnit, which is a Boo wrapper on top of BizUnit. It has advantages similar to our BizUnit "port", but also has the advantage of still using BizUnit instead of forking from it.