I have created a number of (standalone) automated test-cases captured using CodedUITest in Visual Studio 2013, to test webpages.
They work fine within Visual Studio, individually; and when several of them are put into a single project as a solution, to create a kind of playlist.
However, I'm trying to use Microsoft Test Manager as a 'front-end' - in order to be able to select which tests, create play-lists, decide how many times a specific test-case should be run etc, with the results stored.
TFS is being used to both store my (individual) test-cases, and, where I'd like to deposit the resultant output of pass/fail etc.
Trouble is, even though the test-automation part functions very well within Visual Studio, getting Microsoft Test Manager to function with what I have, and its associated environment is proving a COMPLETE NIGHTMARE.
My system is simple; I have a virtual machine setup with the testing environment which allows (within VS) for me to run these automated tests.
Why is this proving so difficult to work with MTM? It should be easy - I should simply be having to point MTM at the folder my test-cases are stored, and use its GUI to tell it what and how many tests I want to run.
Anyone else have a similar problem, or a similar setup?
All MS do is point me to (countless) pages which I've already read - for which the whole lot seems to be much deeper than it needs to be.
You cant just point MTM at a folder and tell it to run the tests as it would have no idea where to put the data. The results of each codedui running are associated with a corresponding test case in mtm.
You also need to have an automated build create the output (your assemblies) for you. Idealy everything goes together with your application. As your application changes, so will your tests.
You should add your CodedUI projects to the same solution that is used to build the application that you are testing. Then when the automated build for that application kicks off your code is picked up to. Both things, test and application, end up in a drop location. It is that drop location that MTM will use to find your test assemblies.
If, while you have the main solution open, you open/create a Test Case you can go to the automation tab in Visual Studio and associate one of your CodedUI tests with that Test Case. The test case will then show up as Automated in MTM.
Now that we have the versioned bits and know where to find them MTM needs somewhere to run them. If you open MTM and switch to the Lab center you can create a Standard environment to run your tests. This will automatically go install the agents required so you wikk need admin on those boxes.
Now that we have both versioned tests and an environment you can find the Test Case in MTM and see that it is "automated". If you right-click on it and say run you will get a box requesting an environment, and which version of the bits to go run. It will then go off and run those tests against that environment and feed the result back into the Test Case.
Does that help?
You'll need to setup Test controller and Test agent and associate you build with a Test Plan in MTM.In Test Lab you need to associate Test Agent to the Test Controller.
Once done you'll need to allow File & Printer sharing exception
And then you will have to add Test Controllers and Agents to Appropriate Groups.
Control Panel->All Control Panel Items->Administrative Tools->Computer
Management
Related
We have a project with a single code base which we build both on Windows and Linux. And we want to run Klocwork code analysis on both Windows and Linux. Currently our approach is:
We have set up one KW project in the web UI
Inject and build on Linux, push the results to the server, save the report
Inject and build on Windows, push the results to the server, save the report
It somehow works, but the problem is that latter scan effectively overwrites results of the first one. If we save report directly after push, then we can still have a saved copy, but if developers want to triage/analyze the hit which is present only in the first build (i.e. some Linux-specific code), then it's almost impossible because KW has already marked this hit as "obsolete" (because it was not present in Windows scan)
Having two projects is not really an option, because 90% of the code is shared and it will cause huge overhead of developers to triage the same hits twice.
There are multiple options to achieve your goal.
Option 1: There is a tab under projects called Builds. That can give you the build chain report. here you can see the reports of previous builds.
Option 2: Are you using Klocwork desktop tools (Plugins/kwcheck)? If yes, developer will be notified automatically about the new defects/issues that he has produced at his machine. So there may be no question of reviewing Klocwork portal by a developer just to see what are all the issues he has created.
Option 3: I see you have mentioned 90% of the code has been shared. Is that meant, your project needs windows dlls and Linux libraries together to build your project?.
If the Answer is YES, please do let me know. i will think about some possible workarounds.
If the Answer is NO, then creating kwtables is one time job and from second time onwards Klocwork can perform incremental analysis (kwbuildproject ....... --incremental)
Option 4: Creating multiple project is not a bad option. Existing project settings can be replicated and the issue status can be sync. When you push the results to Klocwork server, the results will be pushed from Build Machine to Klocwork web/database server and it creates /projects_root/My_Project/builds/My_Build_Name/ directory. So, maintaining two Klocwork projects wont make much of a difference.
Option 5: Schedule a call with Klocwork support team. They will be happy to assist you with the best possible way.
I hope this helps.
Using the Test hub in the Web Portal why is the "Associated Automation" tab completely empty, greyed out and prevented from being used?
What do I need to do to enable all this and run a coded ui test? I have hand coded, Coded UI tests that are build and in dlls but need to run the tests in those dlls in my test suite.
For background, I can run the tests in the dlls using a batch file and have done so successfully but struggling to get started running them from my test plan! My Microsoft "Test Manager" has the same problem, all greyed out but as Test Manager is being deprecated I am less concerned about that.
EDIT:
Another way of viewing this problem is to ask why does the Automation Status not show the option "Automated"? See picture below ...
It is not possible per MSDN documentation.
In the visual studio development environment, try right clicking on a test in test explorer, clicking something such as "Associate to Test Case", then entering in the test case number and see if that works. I believe it is greyed out because there is no valid input options through the web right now.
I really don't know how to better explain this situation I'm running into but I'll try my best.
This is what I'm trying to do...
I currently have a batch file that does some set of actions such as downloading files from FTP, Folder creation, modifying some text files etc. This was taking us 45 minutes to do manually and with the batch script automation it's easier.
The next step I want to do is, launch our .net Windows application, login to it and do some actions in it and then log out of it. This is actually a regression test case which I've automated using VS Coded UI on another machine. The problem I'm running into is, there is a separate support team who will need to do that 45mins of job which I've already automated followed by some actions after logging into the application. That support team's machines will not have VS or Coded UI installed in it.
So, how do I go about it? Any idea, please?
You can execute the CodedUI tests you've written without Visual Studio/CodedUI being installed on the machines. Remember that you can run tests through controllers and agents in TFS or Microsoft Test Manager. You can take those principles to run them manually, even if it's a strange corner case. This takes two steps, if I recall correctly:
Design your CodedUI tests to reference the CodedUI .dlls as part of the solution, rather than the GAC. By this I mean copy and paste the required .dlls into a solution folder and replace the existing references with ones that point to the .dlls in the folder. When you distribute your tests, be sure to include this soln folder of course. (UPDATE: After some more experience wit this, I've found it much easier to use NuGet Packages instead. Project level references are an absolute NIGHTMARE)
Install the "Test Agent" software that Microsoft provides for free on the tester's machine. This will install the other testing .dlls your tester will probably need in their GAC. You could do step 1 with these as well but to be honest I think this is less trouble. In addition, it installs the necessary mstest executables.
Your testers will then have to use mstest.exe (UPDATE: Once you've installed agents, you can/should use vstest.console.exe as an alternative) in the console to run your CodedUI tests. Alternatively you canprobably use powershell or your batch files to execute your tests and your other tests in one neat package.
Please let me know if this gave you a potential solution for your problem.
I'm trying to create a worker process for Azure and I'd like to test it via nUnit. I need to write a process which connect to storage and uploads data to it. The problem is I can't find any references to how you actually set up the testing framework to start dev fabric etc.
I've created the Work Processor role and then created a testing project, wired up nunit to start and for the project to pick up the test project dll. This all works fine with nUnit opening, I'm able to see the test dll and run the test.
My issue is "How do I get the dev fabric to start up when running the project through nUnit?" I'm presuming that I need to put something into the setup routine within the test project but I can't see what I should put in.
If anyone has any tips, experience etc., links to how to etc. I'd be very grateful. I'm sure that I can't be the first person who want to put Azure projects under test.
Haven't done this myself, but the DevFabric (compute and store) has a command-line interface.
You would just invoke the commdn from your test stand-up routine and it should be good to go.
CSrun.exe is the command, please see MSDN for usage reference:
http://msdn.microsoft.com/en-us/library/gg433001.aspx
I want to automate the process of running coded UI tests.
Do i need to have user logged in to the system?
Or is it possible to run it using XYNT service?
We are able to run our coded ui tests in an automated fashion using windows scheduler and the command line MSTest.exe. Take a look at the following MSDN article on how to accomplish this:
http://msdn.microsoft.com/en-us/library/ms182486.aspx
I believe you can use Microsoft's Test Manager to accomplish this as well but we don't have TFS (yet) so I can't vouch for it.
As far as I know, CodedUI Tests require an active desktop session, so running them under a service account without an open desktop session may work, but not without a session.
I used the mstest to run the coded ui test and use Windows TaskScheduler to schedule the test.
You can find detail in MSTest.exe command-line documentation.
When you want to run it automatically, there are two things you should pay attention:
1. Set auto log on
2. Close the screen saver
You need to take the snapshot of your enviroment with user logged in and then you can run your test through MS Test Manager or TFS (nightly build process)
Write the test's in Visual Studio 2010/2012, this means you can manually edit the test's the code, then you should run them in Test Manager,
BTW you need TFS and Test Manager to be the same year otherwise it won't work.
I've being doing a lot of these recently if you have any questions post it online and send me a message, I'll be happy to help.