How do you test pages with single sign-on (SSO) login during integration tests (for instance by using caybara or cucumber)? For a normal login, you would write a method which visits the login page, fills out the form, and submits it. This is a bit difficult if the login form comes from an external SSO server like Shibboleth or OpenAM/OpenSSO. How is it possible to write integration tests for pages protected by SSO?
A similar problem is integration testing with a separate search server (Solr or Sphinx). You would probably solve it by using some form of mocks or stubs. Can someone give a good example how to mock or stub a SSO for cucumber or capybara? If this is too difficult, then a comparable example for a search server would be helpful, too.
Integration testing of a SSO application is a
special case of a more general problem: testing
distributed applications. This is a difficult
problem and there does not seem to be a magic
bullet for it. There are various ways to combine
a set of different servers or services and test them
as a whole. The two extremes are
a) Test an instance of the whole system. You don't
need any mocks or stubs then, but you need
a complete, full-fledged setup of the entire stack. This includes
a running instance of every server involved.
For each test, setup the entire application stack,
and test the whole stack, i.e. test the
entire distributed system as a whole with all
the components involved, which is difficult
in general. This whole thing works only if each
components and all connections are working well.
b) Write an integration test for each component,
treat it as a black box, and cover the
missing connections by mocks and stubs.
In practice, this approach is more common for
unit testing, one writes tests for each
MVC layer: model, view, and controller
(view and controller often together).
In both cases, you have not considered
broken connections. In principle one
has to check for each external server/service
the following possibilities
is down
is up and behaves well
is up and and replies wrong
is up, but you send it wrong data
Basically, testing of distributed apps is difficult.
This is one reason why distributed applications are hard to develop.
The more parts and servers a distributed application has, the more difficult it is to setup many full-fledged environments like production, staging, test and development.
The larger the system, the more difficult the
integration testing becomes. In practice,
one uses the first approach and creates a small
but complete version of the whole application.
A typical simple setup would be App Server + DB Server + Search Server.
On your development machine, you would have
two different versions of a complete system:
One DB Server with multiple databases (development and test)
One Search Server with multiple indexes (development and test)
The common Ruby plugins for search servers (Thinking Sphinx for Sphinx
or Sunspot for Solr) support cucumber and integration
tests. They "turn on" the search server for certain portions of
your tests. For the code that does not use the search server,
they "stub" the server or mock out the connection to avoid unneeded
indexing.
For RSpec tests, it is possible
to stub out the authentication methods,
for example for a controller test by
before :each do
#current_user = Factory(:user)
controller.stub!(:current_user).and_return(#current_user)
controller.stub!(:logged_in?).and_return(:true)
end
It also works for helper and view tests, but
not for RSpec request or integration tests.
For cucumber tests, it is possible to stub
out the search server by replacing the connection
to the search server with a stub (for Sunspot and
Solr this can be done by replacing the Sunspot.session,
which encapsulates a connection to Solr).
This all sounds well, unfortunately it is a bit hard to
transfer this solution for a SSO Server. A typical minimal
setup would be App Server + DB Server + SSO Server.
A complete integration test would mean we have to setup one SSO Server with
multiple user data stores (development and test).
Setting up a SSO Server is already difficult enough,
setting up a SSO Server with multiple user data
stores is probably not a very good idea.
One possible solution to the problem is maybe somewhere in the
direction of Fakeweb. FakeWeb is a Ruby library written by
Blaine Cook for faking web requests. It lets you decouple
your test environment from live services. Faking the response
of a SSO server is unfortunately a bit hard.
Another possible solution I ended up using is to use a fake login, i.e.
add a fake login method that can be called within the integration
test. This fake login is a dynamic method only added during the
test (through a form of monkey patching). This is a bit messy, but
it seems to work.
Related
I want to set up a CI/CD workflow which includes testing my server as a whole.
Using node I have a couple options such as
Jest,
Mocha,
Chai
or a different approach such as Postman/Newman.
I have tried out Jest and found it very difficult to adjust my databases such as Postgres and Redis to the mock environment. In fact, I didn't get it to work at all, presumably because of all the technologies involved.
Is there is a tool, maybe one of the others mentioned, that takes the server as it is, without any additional configuration, calls the endpoints of the Server and gives me results, which I can then use to further take action (push/rollback/discard changes)?
I have a question about how best to do end-to-end/integration testing without having to mock an entire API. I've not found any great solutions and I'm starting to wonder if I'm thinking about this the wrong way.
My situation
I have a web client that I want to test which relies heavily on a REST API that is part of our larger system. For the most part, testing against an instance of the real API service seems like the right thing to do in our end-to-end testing scenario, but in some cases (e.g. provoking errors, empty lists et c) it would be easier to mock portions of the API than to go in and change the actual state of the API service.
In other cases, I don't need to mock the API, but I do need to confirm that requests were made as expected, essentially a spy, but on an API.
My idea for a solution
I imagined the best way to solve this would be an HTTP proxy controlled by the test suite, sitting between the system-under-test and the API service. The test suite would then:
Configure the system-under-test to use the API proxy
For each test
Set up mocks only on relevant endpoints (the rest are proxied to the API service)
Exercise the system-under-test
Make assertions by reading spies from the proxy
Tear down/reset the mocks and spies afterwards
I have multiple use cases for this, one being Cypress end-to-end testing inside a browser, where monkeypatching or dependency injection is not possible, because significant portions of the system under test is not executed directly by the test-suite.
I found MockServer which seems to solve what I'm looking to achieve, but in a massive Java program that adds a lot of requirements (e.g. Java, which translates to CI costs) to an otherwise node-based environment.
EDIT: This image from MockServer illustrates one usage which matches my scenario:
My question(s)
Is what I'm considering a good approach to this type of situation? What is a good way (e.g. existing software) to solve this within the Node eco-system?
We're using ServiceStack for a client project with several distinct problem domains, which we'd prefer to keep separated. We've developed a testing framework that spins up an AppHostHttpListener and sets up an in-memory test database using SQLite and DbUp - but, as you know, a test session's AppDomain can only have one AppHost at a time.
On the other hand, we have two different AppHosts that we want to deploy, let's call them Foo and Bar. Foo accepts requests and passes them to Bar, so Foo -> Bar, and Bar is standalone.
We want to be able to write end-to-end integration tests that exercise instances of both Foo and Bar. With ServiceStack's limitation of one AppHost per AppDomain, we seem to have the following options:
Spin up a new AppDomain for each AppHost inside the test session and control their lifetime across a MarshallByRef boundary. Not sure how this would perform sharing 'test connections' between AppHosts though.
Mock out the external service. This is the textbook answer, but these systems are critical enough that we'd like to see when changes to one service break the other.
Make the endpoints pluggable so that they can be loaded in the same AppHost for testing, but under different sub-URLs. The way I see it, this would require the endpoints to share AuthFeature, IDbConnectionFactory etc, so we would lose that flexibility.
My questions to you are:
Which option would you go with?
Can you recommend another approach that would enable us to test integration of multiple ServiceStack endpoints in memory?
The only way to test multiple Services in memory is to combine them in the same Test AppHost which will only need the to register the dependencies the integration tests are testing. In memory Integration tests normally have a Custom AppHost built to task, the AppHost isn't part of the test.
The alternative is to use IIS Express and start instances of the endpoints used in the integration test before running them.
I want to write one web application with node.js and MongoDB and I have got task to even test it. I would like to know if there are any tools like JMeter or anything else for load/stress testing of Node.js?
EDIT
My application is going to be information extraction kind of application and client expects extraction should not take more than 10 seconds for one document. Currently I have same application written in C# but its not scaling upto client's expectations. Then I came across this beautiful and fast Node.js. I think Node.js can help me alot.
Please enlighten !!!
Try nodeload: it's a collection of node.js modules for load testing HTTP services.
As a developer, you should be able to write load tests and get
informative reports without having to learn another framework. You
should be able to build by example and selectively use the parts of a
tool that fit your task. Being a library means that you can use as
much or as little of nodeload as makes sense, and you can create load
tests with the power of a full programming language. For example, if
you need to execute some function at a given rate, just use the
'nodeload/loop' module, and write the rest yourself
Just found out that this package is no longer under development so here are some active forks:
https://github.com/gamechanger/nodeload
https://github.com/Samuel29/NodeStressSuite
Why couldn't you test a node server with JMeter? For most load tests it doesn't matter what language your server is, you're just hitting it with a bunch of requests.
In any case, you could try loadtest which is implement in node.
Runs a load test on the selected HTTP or WebSockets URL. The API allows for easy integration in your own tests.
Edit:
This answer provides more options:
NodeJs stress testing tools/methods [closed]
Try artillery. Here are its features, the description of which is taken from the documentation:
Multiple protocols: Load test HTTP, WebSocket, Socket.io, Kinesis, HLS and more.
Scenarios: Support for complex scenarios to test multi-step interactions in your API or web app (great for ecommerce, transactional APIs, game servers etc).
Load testing & Functional testing: reuse the same scenario definitions to run performance tests or functional tests on your API or backend.
Performance metrics: get detailed performance metrics (latency, requests per second, concurrency, throughput).
Scriptable: write custom logic in JS, using any of the thousands of useful npm modules.
Integrations: statsd support out of the box for real-time reporting (integrate with Datadog, Librato, InfluxDB etc).
Extensible: write custom reporters, custom plugins, custom protocol engines etc.
and more! HTML reports, nice CLI, parameterization with CSV files.
I have a mixed UI (Win App, WPF App, and soon an ASP.NET MVC App) setup, so far I'm using Client Application Services for security. I know how to programmatically get a user authenticated and doing so is working beautifully. However...
I want to implement some cross cutting that basically checks to see if the user is authenticated all the time. Since everything will be accessing web services I want to enable this as a standard execution for pretty much everything the UI does. So far I'm thinking the PIAB - Policy Injection Application Block - will serve that function. What I'm wondering is two things;
1 Will the PIAB cover that needed functionality? Verifying authentication at every practical step if used against the UI?
...and...
2 Are there alternatives out there besides the PIAB? I'm curious to do a comparison of aspect oriented policy injection frameworks.
I'm not really familiar with Client Application Services but from my experience, most AOP frameworks wrap interfaces in order to implement the cross-cutting functionality. If CAS uses interfaces, you could probably just wrap them with what ever functionality you require.
Alternative AOP frameworks:
Spring.NET
Castle Dynamic Proxy
Spring.NET and Dynamic proxy seem to work in much the same way and have much the same performance in my Hello World type tests (about half-way between direct calls and invoking through reflection). PIAB is significantly slower than both these frameworks and I found bit more verbose. It does have the ability to be configurable via xml and I'm not sure if that's a good thing or not. Not sure if the other frameworks provide that. It does of course have the MS stamp of approval though :P.