I'm using pytest to run a few thousands of tests against an API.
The need is now to not only use multiprocessing (pytest-xdist) and multithreading (pytest-parallel) but to also have them run on multiple machines (still keeping the multi process and threading capabilities).
This is the current state, the need is to basically duplicate this chart.
https://i.imgur.com/AKj2nmL.jpg
Our last resort will be to develop a test runner service that will be deployed on as many machines as needed, and use an sqs so these machines can pull work from there.
Are there any better way of achieving this? Using pytest or maybe combined with Jenkins.
Related
I'm moving my first steps with prefect, and I'm trying to see what its degrees of freedom are. To this end, I'm investigating whether prefect supports running different tasks on different schedules in the same python process. For example, Task A might have to run every 5 minutes, while Task B might run twice a day with a Cron scheduler.
It seems to me that schedules are associated with a Flow, not with a task, so to do the above, one would have to create two distinct one-task Flows, each with its own schedule. But even as that, given that running a flow is a blocking operation, I can't see how to "start" both flows concurrently (or pseudo-concurrently, I'm perfectly aware the flows won't execute on separate threads).
Is there a built-in way of getting the tasks running on their independent schedules? I'm under the impression that there is a way to achieve this, but given my limited experience with prefect, I'm completely missing it.
Many thanks in advance for any pointers.
You are right that schedules are associated with Flows and not Tasks, so the only place to add a schedule is a Flow. Running a Flow is a blocking operation if you are using the open source Prefect core only. For production use cases, it's recommended running your Flows against Prefect Cloud or Prefect Server. Cloud is the managed offering and Server is when you host it yourself. Note that Cloud has a very generous free tier.
When using a backend, you will use an agent that will kick off the flow run in a new process. This will not be blocking.
To start with using a backend, you can check the docs here
This Prefect Discourse topic discusses a very similar problem and shows how you could solve it using a flow-of-flows orchestrator pattern.
One way to approach it is to leverage Caching to avoid recomputation of certain tasks that require lower-frequency scheduling than the main flow.
We are using the following tech stack
NodeJS
PostgreSQL
Cypress
We have many integration tests and we are getting to the point where we should parallelize the test cases. When I used to work with Python, we had multiprocessing library that allowed us to easily dispatch things to multiple cores in a pool. For example, you would loop through all the things you need to dispatch and if you've exhausted all cores you would wait until the next one is available.
Please use this package for parallel-testing
https://www.npmjs.com/package/mocha-parallel-tests
I need to compare network latency for 2 clients half the globe away from each other and visualize the response time history for each location the request comes from side-by-side. I've been researching load and performance testing tools and found Locust to be very convenient. Is there a way I can achieve my goal with Locust in a quick/standard/non-hacky way?
Whether you mean you need 2 clients in different locations running the same task to compare or 2 clients in 2 different locations running 2 different tasks, Locust can handle either of those scenarios.
Check out the Tasks section of the documentation for details about how to write tasks. You can write a single task that does both of what you need, two different tasks that are randomly selected for each user that starts, write two tasks and have them be weighted in their randomness (including making one weighted at 0 so it's never run, effectively turning one task off so only the other one is run at that time), and many other options. Which method is best will depend on exactly what you need and how you want to do it. It may take some experimentation to determine what's best.
As for running in multiple locations, you can run the test separately in different places and compare results, or Locust can run distributed so you can have workers in multiple locations running at the same time. You may also want to look at using Docker, which in some ways can make running in different locations easier if you use AWS, Azure, GCP or whatever other cloud provider to spin up instances to run on.
I am building a small utility that packages Locust - performance testing tool (https://locust.io/) and deploys it on azure functions. Just a fun side project to get some hands on with the serverless craze.
Here's the git repo: https://github.com/amanvirmundra/locust-serverless.
Now I am thinking that it would be great to run locust test in distributed mode on serverless architecture (azure functions consumption plan). Locust supports distributed mode but it needs the slaves to communicate with master using it's IP. That's the problem!!
I can provision multiple functions but I am not quite sure how I can make them talk to each other on the fly(without manual intervention).
Thinking out loud:
Somehow get the IP of the master function and pass it on to the slave functions. Not sure if that's possible in Azure functions, but some people have figured a way to get an IP of azure function using .net libraries. Mine is a python version but I am sure if it can be done using .net then there would be a python way as well.
Create some sort of a VPN and map a function to a private IP. Not sure if this sort of mapping is possible in azure.
Some has done this using AWS Lambdas (https://github.com/FutureSharks/invokust). Ask that person or try to understand the code.
Need advice in figuring out what's possible at the same time keeping things serverless. Open to ideas and/or code contributions :)
Update
This is the current setup:
The performance test session is triggered by an http request, which takes in number of requests to make, the base url, and no. of concurrent users to simulate.
Locustfile define the test setup and orchestration.
Run.py triggers the tests.
What I want to do now, is to have master/slave setup (cluster) for a massive scale perf test.
I would imagine that the master function is triggered by an http request, with a similar payload.
The master will in turn trigger slaves.
When the slaves join the cluster, the performance session would start.
What you describe doesn't sounds like a good use-case for Azure Functions.
Functions are supposed to be:
Triggered by an event
Short running (max 10 minutes)
Stateless and ephemeral
But indeed, Functions are good to do load testing, but the setup should be different:
You define a trigger for your Function (e.g. HTTP, or Event Hub)
Each function execution makes a given amount of requests, in parallel or sequentially, and then quits
There is an orchestrator somewhere (e.g. just a console app), who sends "commands" (HTTP call or Event) to trigger the Function
So, Functions are "multiplying" the load as per schedule defined by the orchestrator. You rely on Consumption Plan scalability to make sure that enough executions are provisioned at any given time.
The biggest difference is that function executions don't talk to each other, so they don't need IPs.
I think the mentioned example based on AWS Lambda is just calling Lambdas too, it does not setup master-client lambdas talking to each other.
I guess my point is that you might not need that Locust framework at all, and instead leverage the built-in capabilities of autoscaled FaaS.
Lets say we have 100 functional Coded UI scripts. What kind of infrastructure is needed to run these tests in parallel? I'm afraid it would take down Visual Studio Server.
Because coded UI interacts with a UI, only one instance can run on a machine at a time. If you set up a lab, or VM instances all running parts of your test your limit is # of instances and how much your application under test can handle.