I have a suite of working Coded UI tests, written in C#, all marked as [TestMethod], written in VS 2013 and running on a set of VMWare machines, using the MS Test Agent (version 12.0).
I am building a release pipeline in Azure DevOps and use the following tasks:
Visual Studio Test Platform Installer (selected version 15.9.2)
VsTest - Test Assemblies (selected Visual Studio 2015 as this is the lowest version)
All of the test DLLs are at a specific folder on the VM I'm running and they are all copied from the deployment folder of an actual working VMWare machine. The test directories of the tasks are pointing to that folder.
I tried running .orderedtests file in the test assemblies section of the VsTest, but got an error that no tests were found.
I then switched to a *.dll so I can just see some tests recognized and running.
At this point I just want to see tests running. I don't care for the results.
In this scenario - letting the VsTest task discover the available tests, I get the following error:
##[error]The slice of type 'Discovery' is 'Aborted' because of the error : System.Exception: No test is available in [some
folder]\Some-File.dll. Make sure that test discoverer & executors are
registered and platform & framework version settings are appropriate
and try again.
What am I missing and what is needed to be able to run those 2013 (12.0) tests, which are working perfectly on local VMs, on the Azure DevOps pipeline?
Thanks!
The slice of type 'Discovery' is 'Aborted' because of the error : System.Exception: No test is available in [some folder]\Some-File.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Based on the error message, you can try to can specify the appropriate framework and platform in the vstest task.
For example:
/Platform: x64 /Framework: framework40
For more info, you could refer to this doc: VSTest.Console.exe command-line options
On the other hand, since you can run the test on your local machine successfully, you can also try to use the self-hosted agent.
Related
Where can I see the system capabilities of hosted agents? I went here and it only shows user capabilities:
https://dev.azure.com/{myOrg}/{myProj}/_settings/agentqueues?agentId=8&queueId=91&view=capabilities
Trying to find versions of msbuild and paths to it on the computer. I need to use version 16
You can check the tools and softwares installed on the Microsoft hosted agent here. Click the Link shown as below screenshot to see the included softwares.
Unfortunately msbuild seems not installed separately, as it cannot be found in above hosted agents. But you can directly use Msbuild task and specify the msbuildVersion to 16.0 in your pipeline. The pipeline will automatically download the specified version msbuild from the cloud.
And also you can build your own self-hosted agent to run your pipeline. and manually download msbuild version 16 to your on-premise machine. So that you can specify the msbuild location in msbuild task to your on-premise msbuild location.
They have actually open sourced the Hosted Agents and you can find out all the information required from the repositories:
https://github.com/MicrosoftDocs/vsts-docs/blob/master/docs/pipelines/agents/hosted.md
The have links for each of the Images that are available at this link:
https://github.com/MicrosoftDocs/vsts-docs/blob/master/docs/pipelines/agents/hosted.md#software
Windows2019-Agent
https://github.com/actions/virtual-environments/blob/master/images/win/Windows2019-Readme.md
You can also add Demands to Agents:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml
This can be used to ensure the agent has your the specific software needed for your builds.
My company is searching for a test management and test automation tool. Since we already use TFS 2015 for version control of our internal documents and product documentation, the question arose whether or not TFS is a suitable candidate for our testing needs.
Our situation is the following: We produce a software system that runs on our custom Linux-based OS, so all test systems are Linux machines. Each member of our test center has a personal Windows computer as well. We want test automation primarily for our functional tests, since our unit tests are already integrated into our build process.
I have done some research into this matter and found out that test management and manual testing should work quite fine with TFS via the Web Access. However, I cannot find reliable statements on the question whether or not the test automation via Microsoft Test Manager works with (remote) Linux machines.
So this is my question: Is there a way to use TFS and the Microsoft Test Manager to automate tests under Linux? If TFS itself cannot do this, are there integrations with external test automation frameworks?
For my research so far, I have used the following resources:
the official documentation on testing with TFS
https://www.visualstudio.com/en-us/docs/test/overview
this official site on how to automate a test case with Microsoft Test Manager
https://msdn.microsoft.com/en-us/library/dd380741.aspx
I would appreciate any kind of help provided.
Afraid not, Install and Deploy test agent is not support in Linux System.
What system requirements do I need to install my test test agents?
Windows 10
Windows 8, Windows 8.1
Windows 7 Service Pack 1
Windows XP Service Pack 3
Windows Server 2012, Windows Server 2012 R2
Windows Server 2008 Release 2, Service Pack 1
Source Link: Install and configure test agents
Features such as diagnostic data collection and functional test would not work on these machines since the test agent is not supported on those OSes.
More detailed info about TFS, MTM Test Management, Automation test, you can refer my answer in this question: Test Automation tools shipped with Visual Studio 2015 Enterprise?
Per my view, we don’t have to run automation tests via TFS. You can use command line to run test. I guess you just want to reflect the test result in the TFS/build. If that’s the case, you can
Run your test under linux which will generate trx file
Publish the test result to the TFS/build. See
https://learn.microsoft.com/en-us/previous-versions/ms243151(v%3dvs.140)
I am having difficultly trying to come up with a working automated build and automated testing solution for testing several applications remotely.
I have several different applications which install onto Windows test machine VMs (Win 7) and an automated test solution which is built using a test settings file and test case filter and run automated tests against this application.
I have two groups of VMs, one group to build the automated tests solution which then sends the tests remotely to the second group which has the application installed on.
Currently this is done by a default template build definition (VS 2012) which builds the test solution on a build VM and sends the tests remotely to a test VM (has application needed manually installed prior to starting build), defining a test controller in the test settings file (there is a test settings file per test VM / build definition) the tests are sent to that test controller. Each test VM has a test controller and test agent on the same VM to prevent the tests going to multiple machines, the tests need to be sent to one machine with that specific application installed.
I am wanting to scale this process and allow for complete automation, so I can just kick off a specific application build which will install the required application on a free test VM, build the test solution code on a build VM and send the tests remotely to the test VM which now has the application needed installed on it, run the tests and send the results back.
I am having issues with doing this between the build and test VMs and having the test settings file on the build machine updated with the test controller name of the free test VM.
Is something like this possible to do and if so what would be the best way to go about this?
Any help or feedback would be greatly appreciated.
Setup:-
TFS / Visual Studio 2012 and Windows 7 build / test machines
Reference links:-
How can TFS build process be configured to execute tests on Test Agents through a Test Controller?
Ok, so you need to look at Lab Management for configuring environments. TFS has a built in method of saying that a set of machines go together and that you want to run tests against it. You can then use MTM to push groups of automated tests to environments and record the data.
In addition you should look at Release Management for VS 2013 (works with 2012) for pushing out the bits to your environments.
With those two tools you can maintain release pipelines where not only can you push tests but they pull tests as part of the release process.
http://nakedalm.com/execute-tests-release-management-visual-studio-2013/
You can have a build kick off the process of build->deploy->test and repeat the process easily...
I'm trying to run unit tests in the build server of Visual Studio. Unfortunately, I get this message:
No test found. Make sure that installed test discoverers & executors,
platform & framework version settings are appropriate and try again.
I gave it a test setting files to run, specified the run settings, set the MSBUILD to x86 and the target platform to x86:
Emulator WVGA
Tried to specify the regex pattern to find the test DLLs. Still no luck. Am I doing something wrong ?
I'm using MSTest.
Edit: Apparently, I believe I mistakenly thought that Windows Phone Unit Test project is different from a Normal Unit Test Project. It was able to see the test sources when I just use normal unit test project. Correct me if I'm wrong. Question now is what's the point of the Windows Phone Unit Test App?
If you are using a third party test framework you will need to install it on the controller server.
Check this http://www.visualstudio.com/get-started/create-and-run-unit-tests-vs#frameworks
Some weeks ago, I used the following steps to publish our developers' application to our testing Web site.
Get Latest Version (Recursive) in Solution Explorer.
Run Update-Database in Package Manager Console.
"Publish..." in Solution Explorer.
This worked but we decided to automatize this with Continuos Integration in a build definition. I put the following MSBuild arguments into the build definition
/p:DeployOnBuild=True
/p:DeployTarget=MsDeployPublish
/p:MSDeployPublishMethod=WMSvc
/p:CreatePackageOnPublish=True
/p:DeployIisAppPath="MyProject-testing"
/p:MsDeployServiceUrl=tfs.mydomain
/p:username=Administrator
/p:password=MYPassw!
/p:AllowUntrustedCertificate=True
/p:SkipExtraFilesOnServer=True
After some actions on the server tfs.mydomain, the executables and many other files of the application started appearing on the server automatically via the build definition. All except the database started being refreshed on the server automatically via the build definition. The application worked if there were no need to update database. But the updates are needed sometimes.
How can I modify the mentioned build definition to make the database up-to-date automatically what I did manually with Update-Database in Package Manager Console?
Thank you if you answer.