My company is searching for a test management and test automation tool. Since we already use TFS 2015 for version control of our internal documents and product documentation, the question arose whether or not TFS is a suitable candidate for our testing needs.
Our situation is the following: We produce a software system that runs on our custom Linux-based OS, so all test systems are Linux machines. Each member of our test center has a personal Windows computer as well. We want test automation primarily for our functional tests, since our unit tests are already integrated into our build process.
I have done some research into this matter and found out that test management and manual testing should work quite fine with TFS via the Web Access. However, I cannot find reliable statements on the question whether or not the test automation via Microsoft Test Manager works with (remote) Linux machines.
So this is my question: Is there a way to use TFS and the Microsoft Test Manager to automate tests under Linux? If TFS itself cannot do this, are there integrations with external test automation frameworks?
For my research so far, I have used the following resources:
the official documentation on testing with TFS
https://www.visualstudio.com/en-us/docs/test/overview
this official site on how to automate a test case with Microsoft Test Manager
https://msdn.microsoft.com/en-us/library/dd380741.aspx
I would appreciate any kind of help provided.
Afraid not, Install and Deploy test agent is not support in Linux System.
What system requirements do I need to install my test test agents?
Windows 10
Windows 8, Windows 8.1
Windows 7 Service Pack 1
Windows XP Service Pack 3
Windows Server 2012, Windows Server 2012 R2
Windows Server 2008 Release 2, Service Pack 1
Source Link: Install and configure test agents
Features such as diagnostic data collection and functional test would not work on these machines since the test agent is not supported on those OSes.
More detailed info about TFS, MTM Test Management, Automation test, you can refer my answer in this question: Test Automation tools shipped with Visual Studio 2015 Enterprise?
Per my view, we don’t have to run automation tests via TFS. You can use command line to run test. I guess you just want to reflect the test result in the TFS/build. If that’s the case, you can
Run your test under linux which will generate trx file
Publish the test result to the TFS/build. See
https://learn.microsoft.com/en-us/previous-versions/ms243151(v%3dvs.140)
Related
Should I install the TFS Test Controller on the same machine as the Build Controller?
and also let it launch the browser on this same machine?
What is the best practice?
We have a TFS 2013 Build Controller on a local machine.
We have CodedUI tests which launch a browser which we want to get going again (worked on tfs2010).
Also where exactly do I get the installer for the Test Controller? I presume its inside the TFS 2013 server ISO.
Yes, I think you can install test controller and build controller on the same machine. You can consider installing build agent on a different machine than TFS Application Tier for performance consideration especially you would like to set up some Gated Check-in or CI builds.
In addition, as you would like to run Coded Ui test, you should configure the test agent to interact with the Desktop. Check https://msdn.microsoft.com/en-us/library/Ee291332(v=vs.120).aspx
In addition, you can download the Test Controller here. (Both test controller and test agent included) https://www.microsoft.com/en-us/download/details.aspx?id=40750
I am having difficultly trying to come up with a working automated build and automated testing solution for testing several applications remotely.
I have several different applications which install onto Windows test machine VMs (Win 7) and an automated test solution which is built using a test settings file and test case filter and run automated tests against this application.
I have two groups of VMs, one group to build the automated tests solution which then sends the tests remotely to the second group which has the application installed on.
Currently this is done by a default template build definition (VS 2012) which builds the test solution on a build VM and sends the tests remotely to a test VM (has application needed manually installed prior to starting build), defining a test controller in the test settings file (there is a test settings file per test VM / build definition) the tests are sent to that test controller. Each test VM has a test controller and test agent on the same VM to prevent the tests going to multiple machines, the tests need to be sent to one machine with that specific application installed.
I am wanting to scale this process and allow for complete automation, so I can just kick off a specific application build which will install the required application on a free test VM, build the test solution code on a build VM and send the tests remotely to the test VM which now has the application needed installed on it, run the tests and send the results back.
I am having issues with doing this between the build and test VMs and having the test settings file on the build machine updated with the test controller name of the free test VM.
Is something like this possible to do and if so what would be the best way to go about this?
Any help or feedback would be greatly appreciated.
Setup:-
TFS / Visual Studio 2012 and Windows 7 build / test machines
Reference links:-
How can TFS build process be configured to execute tests on Test Agents through a Test Controller?
Ok, so you need to look at Lab Management for configuring environments. TFS has a built in method of saying that a set of machines go together and that you want to run tests against it. You can then use MTM to push groups of automated tests to environments and record the data.
In addition you should look at Release Management for VS 2013 (works with 2012) for pushing out the bits to your environments.
With those two tools you can maintain release pipelines where not only can you push tests but they pull tests as part of the release process.
http://nakedalm.com/execute-tests-release-management-visual-studio-2013/
You can have a build kick off the process of build->deploy->test and repeat the process easily...
I am trying to determine if I can configure a development machine for Sharepoint 2013 Enterprise using Visual Studio Pro 2013 and Sharepoint Foundation 2013.
I do not wish to install VS on the production server and run it from there - that is just asking for trouble.
However the budget will not withstand another SP2013 Enterprise license.
I have searched this site and others, but have not found any instance that specifically addresses my question:
can I install SP2013 Foundation (which is free) on my development machine with VS2013 Pro, and use that to build solutions for deployment to my production SP2013 Enterprise server?
The short answer is yes you could do this, but it may not let you do what you need...
So my first addendum to 'yes,' is that the dev machine needs to be running Windows Server 2008 R2 or later and meet all the other hardware and software requirements.
My second is probably a bit more important -- if you develop on SP foundation instead of enterprise you'll be unable to develop and test many of the features that your organization (or your customer, whichever is appropriate) probably purchased the enterprise edition for in the first place.
Something that's also important to note, you wouldn't need to purchase another full enterprise license; you can get an MSDN subscription (platforms or premium for access to SharePoint) that will give you access to software to use specifically for development and testing purposes.
We are having compilations problems in a TFS server and it's because the server lacks several libraries built in the default VS2012 Premium installation (Microsoft Fakes in this case).
I'm unsure of going ahead installing a full instance of VS, but first I want to know what is the best practice in this regard?
What is recommended?
Since we are talking a sandbox, do whatever and don't worry about it. If we are talking best practices, it's not a good idea to put your build tier on the app tier / data tier. Any developer could check in code that gets run on the server during the compile and trash your entire environment.
Have you looked at Visual Studio Online? It's a hosted TFS service and you can use their hosted build controller or configure your own. That makes for a very good sandbox IMO.
I don't see any issue installing VS on the TFS server(I assume you run your builds on that server too and that's when you are seeing the problem. Ideally tfs server and build box should be separate but some people use the same box.)
I have used Visual Studio on the build box several times to debug issues with builds. You just need to make sure you close the VS instance (if it has a solution open) once you are done with debugging otherwise your builds can fail when they try to clean up the project directory at the start of the build.
We run a single server TFS instance which has everything - sql, SharePoint and tfs - running on it. It is also a build server so it has to have VS 2010 and 2012 installed. We've done this with all versions since 2005 and have had no issues with it at all.
I am in a situation where the corporation has just recently upgraded to TFS 2008. They have no intention of upgrading to TFS 2010 at this time. As a development group, we've moved to Visual Studio 2010 this week. As with any large corporation, we cannot get our own environment created to install TFS 2010. Steps on too many toes, and isn't corporate standard. Etc.
I want to take full advantage of the new testing features in relation to the new UI Testing and other features. This appears to require TFS 2010. So my "dream" is to do my daily work at the office and write tests, but at night, have my code synchronized with my TFS 2010 server at home and run automated builds with the full testing capabilities enabled.
So is there is best practice for this? I've read up on the Workspace theory and the binding issues that are involved and that sounds the biggest hurdle to overcome.
Possible Solution - Create two workspaces $/WorkProject and $/WorkProject-Mirror and use a custom application using FileSystemWatcher to kick off a job that synchronizes code changes and a custom rewrite of the bindings. Use job on work laptop and home machine to allow bi-directional binding.
Research to see if TFS Integration Platform will help with this
You are correct the new testing UI (Test Manager 2010) requires TFS 2010, you are also correct that you can use the TFS Integration Platform between a TFS2008 & TFS2010 server. Then use test manager on the 2010 server.
All the above should be easy, the tough part will be the bindings in the solution file. I would suggest you have a second one created that points to your TFS2010 server so that you can open the correct solution file for the correct environment without stepping on your co-workers toes.
I think the two workspace route is overkill, it's just a solution file you need.
I wonder if you could use a read-only account to perform a get from TFS2008 and then do a check-in to your TFS2010 with a more-privileged account. I'm sure those two things and a little clever PowerShell scripting could get you what you're looking for.
I would encourage you to write a second utility to monitor that this script continues to work and to notify you if it detects a failure or something.