I'm trying to create a worker process for Azure and I'd like to test it via nUnit. I need to write a process which connect to storage and uploads data to it. The problem is I can't find any references to how you actually set up the testing framework to start dev fabric etc.
I've created the Work Processor role and then created a testing project, wired up nunit to start and for the project to pick up the test project dll. This all works fine with nUnit opening, I'm able to see the test dll and run the test.
My issue is "How do I get the dev fabric to start up when running the project through nUnit?" I'm presuming that I need to put something into the setup routine within the test project but I can't see what I should put in.
If anyone has any tips, experience etc., links to how to etc. I'd be very grateful. I'm sure that I can't be the first person who want to put Azure projects under test.
Haven't done this myself, but the DevFabric (compute and store) has a command-line interface.
You would just invoke the commdn from your test stand-up routine and it should be good to go.
CSrun.exe is the command, please see MSDN for usage reference:
http://msdn.microsoft.com/en-us/library/gg433001.aspx
Related
We are trying to generate code coverage on performing manual tests for application hosted in Azure app services.
I have a trail account in portal.azure.com with 'user' privilege.
Tried the below options:
A. Use VS tools in Azure console and local
B. Use OpenCover in Azure console
Both the above mentioned options using VS tools and Opencover has been used in the localhost (IIS server) to successfully generate code coverage on runnign manual tests.
A. Steps for code coverage using VS tools in Azure:
1. cd D:\Program Files (x86)\Microsoft Visual Studio 15.0\Team Tools\Performance Tools
2. vsinstr /coverage D:\home\site\wwwroot\bin\abc.dll
3. vsperfcmd /start:coverage /output:D:\home\LogFiles\Testing123.coverage
4. vsperfcmd -shutdown
Issue1: The generated code coverage file has error: Empty results generated: No binaries were instrumented. Make sure the tests ran, required binaries were loaded, had matching symbol files, and were not excluded through custom settings. For more information see https://go.microsoft.com/fwlink/?LinkID=253731
However I'm able to generate code coverage on the same application which has been hosted in my local IIS server. The same 4 steps mentioned above is working. The issue is that the instrumented dll doesn't log the transactions performed on the hosted application.
Reference: https://gurunadhduvvuru.wordpress.com/2014/08/01/generating-functional-testing-code-coverage/
PS: We have tried starting the coverage process before and after publish the application.
B. Steps for code coverage using Open Cover in Azure
1. OpenCover.Console.exe -target:C:\Windows\System32\inetsrv\w3wp.exe -targetargs:"-debug -s 1" -targetdir:C:\inetpub\wwwroot\bin\ -filter:+[] -register:user -output:C:\Users\coverage2.xml
Issue 2: Access denied is displayed when the application is not running.
Issue 3: The generated coverage file is empty and doesn't has the same error mentioned in Issue 1, on executing the opencover command when the application is running.
Reference: https://automationrhapsody.com/code-coverage-manual-automated-tests-opencover-net-applications/
Any help on the reason for the generated coverage file to be empty ? or any reference to suffice my requirement would be much appreciated?
It appears that it is a limitation of Azure WebApp Sandbox, all Azure Web Apps run in a secure environment called a sandbox. Each app runs inside its own sandbox, isolating its execution from other instances on the same machine as well as providing an additional degree of security and privacy which would otherwise not be available.
You cannot install arbitrary software when running on Azure App Service. Generally, you cannot make any machine level changes (e.g. registry changes).
Reference: https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox. If your requirement fits, try this on an Azure VM running IIS, where you have greater control on managing the VM.
I really don't know how to better explain this situation I'm running into but I'll try my best.
This is what I'm trying to do...
I currently have a batch file that does some set of actions such as downloading files from FTP, Folder creation, modifying some text files etc. This was taking us 45 minutes to do manually and with the batch script automation it's easier.
The next step I want to do is, launch our .net Windows application, login to it and do some actions in it and then log out of it. This is actually a regression test case which I've automated using VS Coded UI on another machine. The problem I'm running into is, there is a separate support team who will need to do that 45mins of job which I've already automated followed by some actions after logging into the application. That support team's machines will not have VS or Coded UI installed in it.
So, how do I go about it? Any idea, please?
You can execute the CodedUI tests you've written without Visual Studio/CodedUI being installed on the machines. Remember that you can run tests through controllers and agents in TFS or Microsoft Test Manager. You can take those principles to run them manually, even if it's a strange corner case. This takes two steps, if I recall correctly:
Design your CodedUI tests to reference the CodedUI .dlls as part of the solution, rather than the GAC. By this I mean copy and paste the required .dlls into a solution folder and replace the existing references with ones that point to the .dlls in the folder. When you distribute your tests, be sure to include this soln folder of course. (UPDATE: After some more experience wit this, I've found it much easier to use NuGet Packages instead. Project level references are an absolute NIGHTMARE)
Install the "Test Agent" software that Microsoft provides for free on the tester's machine. This will install the other testing .dlls your tester will probably need in their GAC. You could do step 1 with these as well but to be honest I think this is less trouble. In addition, it installs the necessary mstest executables.
Your testers will then have to use mstest.exe (UPDATE: Once you've installed agents, you can/should use vstest.console.exe as an alternative) in the console to run your CodedUI tests. Alternatively you canprobably use powershell or your batch files to execute your tests and your other tests in one neat package.
Please let me know if this gave you a potential solution for your problem.
I have created a number of (standalone) automated test-cases captured using CodedUITest in Visual Studio 2013, to test webpages.
They work fine within Visual Studio, individually; and when several of them are put into a single project as a solution, to create a kind of playlist.
However, I'm trying to use Microsoft Test Manager as a 'front-end' - in order to be able to select which tests, create play-lists, decide how many times a specific test-case should be run etc, with the results stored.
TFS is being used to both store my (individual) test-cases, and, where I'd like to deposit the resultant output of pass/fail etc.
Trouble is, even though the test-automation part functions very well within Visual Studio, getting Microsoft Test Manager to function with what I have, and its associated environment is proving a COMPLETE NIGHTMARE.
My system is simple; I have a virtual machine setup with the testing environment which allows (within VS) for me to run these automated tests.
Why is this proving so difficult to work with MTM? It should be easy - I should simply be having to point MTM at the folder my test-cases are stored, and use its GUI to tell it what and how many tests I want to run.
Anyone else have a similar problem, or a similar setup?
All MS do is point me to (countless) pages which I've already read - for which the whole lot seems to be much deeper than it needs to be.
You cant just point MTM at a folder and tell it to run the tests as it would have no idea where to put the data. The results of each codedui running are associated with a corresponding test case in mtm.
You also need to have an automated build create the output (your assemblies) for you. Idealy everything goes together with your application. As your application changes, so will your tests.
You should add your CodedUI projects to the same solution that is used to build the application that you are testing. Then when the automated build for that application kicks off your code is picked up to. Both things, test and application, end up in a drop location. It is that drop location that MTM will use to find your test assemblies.
If, while you have the main solution open, you open/create a Test Case you can go to the automation tab in Visual Studio and associate one of your CodedUI tests with that Test Case. The test case will then show up as Automated in MTM.
Now that we have the versioned bits and know where to find them MTM needs somewhere to run them. If you open MTM and switch to the Lab center you can create a Standard environment to run your tests. This will automatically go install the agents required so you wikk need admin on those boxes.
Now that we have both versioned tests and an environment you can find the Test Case in MTM and see that it is "automated". If you right-click on it and say run you will get a box requesting an environment, and which version of the bits to go run. It will then go off and run those tests against that environment and feed the result back into the Test Case.
Does that help?
You'll need to setup Test controller and Test agent and associate you build with a Test Plan in MTM.In Test Lab you need to associate Test Agent to the Test Controller.
Once done you'll need to allow File & Printer sharing exception
And then you will have to add Test Controllers and Agents to Appropriate Groups.
Control Panel->All Control Panel Items->Administrative Tools->Computer
Management
Environment: TeamCity 6.5.1 on Win2k3, BuildAgent(s) on Win2k3, Visual Studio 2k10, .NET v4, Nant 0.91
I'm completing the setup of TeamCity and am trying to lock down the BuildAgent account on the build machine(s) per our security guidelines. The build is crashing the first time "devenv.exe /build" is called via the Nant script:
Faulting application devenv.exe, version 10.0.30319.1, stamp 4ba1fab3,
faulting module msenv.dll, version 10.0.30319.1, stamp 4ba1fd94,
debug? 0, fault address 0x0000c36b.
I had no luck googling that message. However, if I change the BuildAgent Service from the Local Network Account to the Administrator account, things work. However, if I use another domain account, it fails. Also fails if I add that domain account to the local Administrators group.
Any ideas on what I'm missing? Is there a specific privilege you need to have in order for a "DevEnv /build" to work without crashing?
Yuck, I just went through this recently. First, use devenv.com, not devenv.exe. The devenv with the com extension can build a solution and send all output to the console, without using the gui. As the TeamCity agent is a service, it may not be allowed to interact with the gui at all.
Second, and I realize that this might not be possible for you (especially if you are building an MSI), but consider doing whatever you need to do to use the built in Visual Studio build runner that comes with TeamCity. It does utilize MSBuild to do its work. If you go this route and you still need devenv, then go find MSBuild Extensions Pack, which has already solved a lot of these issues with their own devenv build task.
Honestly, I ended up replacing Microsoft's installation projects with alternatives (InstallShield or WiX), and never looked back.
I want to automate the process of running coded UI tests.
Do i need to have user logged in to the system?
Or is it possible to run it using XYNT service?
We are able to run our coded ui tests in an automated fashion using windows scheduler and the command line MSTest.exe. Take a look at the following MSDN article on how to accomplish this:
http://msdn.microsoft.com/en-us/library/ms182486.aspx
I believe you can use Microsoft's Test Manager to accomplish this as well but we don't have TFS (yet) so I can't vouch for it.
As far as I know, CodedUI Tests require an active desktop session, so running them under a service account without an open desktop session may work, but not without a session.
I used the mstest to run the coded ui test and use Windows TaskScheduler to schedule the test.
You can find detail in MSTest.exe command-line documentation.
When you want to run it automatically, there are two things you should pay attention:
1. Set auto log on
2. Close the screen saver
You need to take the snapshot of your enviroment with user logged in and then you can run your test through MS Test Manager or TFS (nightly build process)
Write the test's in Visual Studio 2010/2012, this means you can manually edit the test's the code, then you should run them in Test Manager,
BTW you need TFS and Test Manager to be the same year otherwise it won't work.
I've being doing a lot of these recently if you have any questions post it online and send me a message, I'll be happy to help.