I would like some comments on the possibility or alternatives to the following workflow.
The players:
JimBob - Tester
APP_Test - Virtual machine configured with Lab, Build and Test agent. (shows testing and workflow capabilities active in the environment viewer)
Manual Test Environment - Environment consisting of one virtual machine "APP_Test"
Functional Tests - A test plan configured with the Test Environment for Manual runs being "Manual test Environment"
Build-Deploy - A TFS Build definition that restores a clean checkpoint and then deploys the latest version of the software to the "APP_Test" virtual machine
Build-Deploy-Test - A TFS Build definition that restores a clean checkpoint and then deploys the latest version of the software to the "APP_Test" virtual machine and then runs coded ui tests contained in the solution.
John - Tester/Dev
The scenario:
JimBob uses the TFS web interface to queue a new workflow build of the software ("Build-Deploy"). As part of the build the latest release of the software is deployed to the virtual machine "APP_Test". JimBob opens Microsoft Test Manager 2010 navigates to the Testing Centre, chooses the Test tab and highlights a Test from a previously defined Test Suite in the test plan "Functional Tests". When JimBob Runs the test the environment viewer opens viewing the "APP_Test" virtual machine. Jim performs the test and creates an action recording on the "APP_Test" virtual machine.
John then takes the action recording and creates a coded ui test, adds the test to the solution and when he queues the "Build-Deploy-Test" build it runs the coded ui test in the same environment that the original action recording was created in.
Questions:
I have created the environments, VM, Test plan and Test Suites outlined in the above scenario. What I can't seem to get working is that when a manual test is run it always runs the test on the Local machine not in a VM environment. This will not give me the ability to deploy so that my environment is always in a known state with no interference. I guess my question boils down to: Is it possible to use test management to create action recordings on virtual machines that belong to an environment? I have had success running tests on machines in environments but not making the initial recording. It seems to me that my testers should be able to always perform their tests (automated and manual) in a snapshotted known environment. What am I missing?
Maybe I didn't understand well, but since you want to perform recording you are talking about CodedUI I guess. I believe that the tester can record in his own workstation and perform any tweaks needed in the code and then deliver you an automated test case. Then you can run this automated test case in any virtual environment that satisfies your preconditions.
You cannot do action recording on a remote machine in the environment, see this link
Instead I installed MTM on the VM so I have a clean environment to build my tests in.
Related
After Azure => Pipelines, I end up with two published artifacts: one containing a .NET Core Console Application (myDrop), another containing the corresponding Testing library (written with xUnit) (myTestDrop). Then I move on to Azure => Releases to create a new Release Pipeline as below:
I have a Windows Virtual Machine (VM), which is already installed with all necessary libraries, e.g. .NET Core; and I would like to carry out the Integration Testing (the 2nd Stage above) in that machine. Specifically,
Copy both myDrop and myTestDrop to that VM.
Set some Environment Variable: the path to, let's say, MyConsole.exe in myDrop.
Then run the Integration Test: dotnet vstest "MyConsole.Tests.dll" --logger:trx --ResultsDirectory:"c:/Somewhere" /TestCaseFilter:"Category=IntegrationTest"
If the test is successful, the returned code from dotnet.exe is 0 (otherwise, 1).
The 3rd Stage only runs in case the 2nd Stage is successful.
There should be a way to read *.trx generated from the Integration Test above, especially in case there are some test failures.
My experience with Azure DevOps is limited. I have searched around but most Azure Release examples involve Web Application (IIS, SQL...), not a normal Console Application with Test on a specific VM. Feel free to suggest other alternatives or best practices, given the scenario above.
Any advice, suggestions are appreciated.
How to carry out a Test Stage at a specific virtual machine?
You can install and use a self-hosted agent in that machine. Please refer to this document to install Self-hosted Windows agents. You need to add an agent job to the release pipeline first. Then, choose the agent pool with self-hosted agent installed.
Copy both myDrop and myTestDrop to that VM.
Since your agent is installed on the VM, it will automatically download artifacts to its local folder.
The 3rd Stage only runs in case the 2nd Stage is successful.
You can select “After Stage” trigger in pre-deployment conditions. For example, if the test stage in your screenshot fails, the deployment stage will not deploy.
There should be a way to read *.trx generated from the Integration
Test above, especially in case there are some test failures.
You can check the test results in the Tests tab of the release result page. You can also download the *.trx file on this page.
I have downloaded and installed in a windows server 2016 a vs test agent, which was installed by a tool called Test Agent Configuration Tool. I then followed this guide to configure the agent to run tests that interact with desktop - I want to run CodedUI Tests for web projects from whithin my release pipeline.
This is the output from the Test Agent Configuration Tool
When I run my release pipeline I get this error during the test phase:
Error calling Initialization method for test class
Captura.PDV.Web.Test.CodedUI.CodedUITest1:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To
run tests that interact with the desktop, you must set up the test
agent to run as an interactive process. For more information, see "How
to: Set Up Your Test Agent to Run Tests That Interact with the
Desktop" (https://go.microsoft.com/fwlink/?LinkId=255012)
If you are
running the tests as part of your team build, you must also set up the
build agent to run as an interactive process. For more information,
see "How to: Configure and Run Scheduled Tests After Building Your
Application" (https://go.microsoft.com/fwlink/?LinkId=254735)
Here is my test task configuration:
Some notes and questions:
I can run the tests calling vstest.console.exe manually from within the server with success
My release pipeline runs on VSTS
I am not using Microsoft Test Manager - Is it really necessary?
Do I need to set up a test lab enviroment?
Do I need to have a controller for the agent?
I just want to run my tests in first place to see how it goes and then proceed to more advanced tests scenarios using MTM
You're mixed up between the old way of doing things and the new way.
You don't need to use MTM -- MTM is deprecated.
You don't need to set up a lab environment. Lab environments are deprecated.
You don't need special test controllers or agents.
All you should need to do in order to run UI tests is:
Set up a build/release agent that runs interactively.
Run the tests on that agent with the Visual Studio Test task with the "Test Mix Contains UI tests" check box checked.
That's it.
I'd also recommend against using Coded UI and instead use Selenium (or Appium for desktop applications); Microsoft is no longer investing in the Coded UI platform and recommends using Selenium as an alternative.
I'm evaluating gitlab community edition as a self hosted version. Everything is fine with the product except anyone who has access to pipelines (master or admin) can run deployment on production.
I saw their issue board and i see that this is a feature that will not be coming to gitlab anytime soon.
See https://gitlab.com/gitlab-org/gitlab-ce/issues/20261
For now, I plan on deploying my spring boot applications using the following strategy.
A separate runner installed on the production server
There will be an install script with instructions some where in production server
gitlab-runner user will only have permissions to run the above specific script (Somehow).
Have validations in the script for GITLAB_USER_NAME variable.
But I also see that there are disadvantages in this approach.
GITLAB_USER_NAME is an environment variable which can be overridden easily thus compromising the validation.
Things are complicated when introducing new prod servers.
Too many changes apart from .gitlab-ci.yml... CI/CD was supposed to be simple not painful...
Do I have any alternate approaches or hacks for this...?
I've got an ASP.NET MVC web app with a set of Selenium UI integration tests for it. I want to put this in Azure, and apply continuous delivery like this:
I commit a change to the Git repository
Azure notices the commit, gets the latest code, builds it, and deploys it to some test environment
Azure runs my unit tests on the code, and fails the build if any tests fail
Azure runs my integration tests on the test environment, and fails the build if any tests fail
If all tests pass, promote the code to production
This seems like a logical pipeline to me but I've struggled to get it working.
I've managed to get to step 3. by creating two web apps, e.g. Muppet and MuppetStaging. The MuppetStaging environment is automatically updated on each commit, so that's a great start. I edited the deploy.cmd to perform unit tests. This fails the build when the unit tests fail. That's great too.
The integration testing is the hard bit. Before the deployment has been completed, I can't integration test the current build - because it's not been deployed yet. The best I could do was integration test the previous version... but not the pending version. And even if I could, how would I then promote a successful build to production?
I know there is a "staging slot" concept but I don't want to have to press a "Swap" button to get my change into production, that would defeat the concept of CD. There is also an "Auto Swap" but again I don't think this takes integration testing into account.
I've been looking at tutorials relating to this kind of issue but no-one seems to have my requirements exactly... so either it's not possible or I'm doing it all wrong.
let me see if i can help and answer some of your question.
The integration testing is the hard bit. Before the deployment has
been completed, I can't integration test the current build - because
it's not been deployed yet.
There is post deployment hook that trigger after everything is deployed but still within the deployment process.
https://github.com/projectkudu/kudu/wiki/Post-Deployment-Action-Hooks
There is also an "Auto Swap" but again I don't think this takes
integration testing into account.
if you do testing with post deployment hook, if test failed which will fail the deployment operation, "Auto Swap" shouldn`t be trigger. if it does please do file a bug in Project Kudu
Microsoft Release Management tools seems to apply to your problem perfectly. If you haven't tried these already you should. They are now available as a service and also works great with Azure. Use Visual Studio Team Services (previously known as Visual Studio Online) to use Release Management as a Service and use Git as version control system.
Check these resources:
1. Official RM Page
2. Visual Studio Team Services
3. Continuous Integration via VSTS
For creating a Build-Deploy-Test Workflow for my Wpf application, I ma using TFS Build definition and MTM to create Environment.
I am using following Steps:
I am Using LabDefaultTemplate.11.xaml in Build Definition
In the Lab Process Settings, I have given the required details as:
Environment: I have created an environment using MTM on the VM
in the environment I have added a Desktop Client(the same VM only) in the machines List
Please note that my Test Controller, agent, MTM all are on the VM only
Build: I have selected an existing build definition and it's working fine.
Deploy: I have specidfied command for application installation(running of msi file)
also, In the deploy section only, at the end I have specified a batch file where I am calling my CodedUI tests via command line(using VSTEST.CONSOLE.EXE)
Test: Not using it i.e. Not checked the "Run these Tests in Environment"
When I run this build definition, build creation, and installation are working fine but failes to run the codedui tests. Getting following error:
Error Message:
Error calling Initialization method for test class CodedUITestProject1.AISTest: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
If you are running the tests as part of your team build, you must also set up the build agent to run as an interactive process. For more information, see "How to: Configure and Run Scheduled Tests After Building Your Application" (http://go.microsoft.com/fwlink/?LinkId=254735)
Stack Trace:
at Microsoft.VisualStudio.TestTools.UITesting.Playback.Initialize()
at Microsoft.VisualStudio.TestTools.UITesting.CodedUITestExtensionExecution.BeforeTestInitialize(Object sender, BeforeTestInitializeEventArgs e)
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecution.RaiseBeforeTestInitialize(BeforeTestInitializeEventArgs args)
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.RunInitializeMethod()
Test Run Failed.
Also, I have made sure that test agent is configured as an interactive process.
Any Help would be highly appreciated.
The issue got resolved after setting the build controller also to run as a interactive process.