15
I am working on creating a Reports Dashboard for automated tests that run once in a day. I am using WebdriverIO and so far allure has been quite helpful in building detailed reports.
I am using the allure command line command:
allure generate allure-results && allure open
to generate the reports, however every time I run my automated test the previous reports gets cleared and new report for current run is getting is generated.
Is there any way I can see consolidated reports for all the runs including current and previous ?
I needed a small help. Whenever I run the pipeline, the data being displayed in testtab is incorrect and it displays the data which I ran on my local desktop rather than the job ran on Agent.
You can run tests in test tabs using different options. You can use any to solve your problem:
Automatically inferred test result: for some popular test runners' pipeline automatically infer the test output.
This is done with the help of describing the error logs generated during build operation.
Publish Test result task: Publishes test result to azure pipeline you choose the runner for tests execution available in
any formats.
Test execution tasks: Built -in test execution tasks such as visual studio test that automatically publish tests results to the pipeline
Azure Devops also provide other way to surface the test information You can use this.
Test analytics provide rich insights into test result over a period of time, identify problematic area in your test.
The dashboard provide track of your work's progress add widgets that surface test related information.
Requirement quality
Test result trend
Deployment status
Reference: https://learn.microsoft.com/en-us/azure/devops/pipelines/test/review-continuous-test-results-after-build?view=azure-devops
SO far i can run the pipeline manually and it runs my testcafe tests with the "node myTests.js" command.
my pipeline run
My file myTests.js looks like this:
myTests.js
I followed this tutorial: https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
I tried to associated an test to my Testplan via the REST API, i guess theres some problem. because i can name it whatever i want and it just runs it without errors:my testcase association
When i run a testcase it says it found an automated test and it runs it without errors, the VsTest Job runs but with a warning:
2021-05-18T09:16:32.7619103Z Source filter: *test.dll,!*TestAdapter.dll,!\obj*
2021-05-18T09:16:32.7879061Z ##[warning]No test sources found matching the given filter '*,!\obj**'
Any ideas what im doing wrong? I just want to run my pipeline or my test when im running a testcase in my testplans.
Azure DevOps: How can i connect Testcafe-tests to a testcase?
I am afraid you could not connect the Testcafe-tests to a testcase at this moment.
According to the document Run automated tests from test plans:
You will need:
A Team Build pipeline that generates builds containing the test
binaries.
That is the reason why you get the error No test sources found matching the given filter '*test.dll,!*TestAdapter.dll,!\obj*.
We could build and test the Testcafe-tests in the azure devops pipeline.
Please check the document Integrate TestCafe with Azure DevOps for some more details.
I am trying to publish allure report on azure devops but i got that error on generate the report task :
##[warning]Failed to publish test run data: System.ArgumentException: Unknown test runner
In my baseconf.js the runner: 'local',
Where is my mistake. should i change the runner to remote and how to do it?
I can reproduce this issue And when I click on the Allure report page, I get the error 404.
This should be a issue with the extension Allure Test Reports, we recommend that you raise this issue to here, they will investigate the issue and help us solve it.
In addition, It seems that they are working on a second version, you could also check this page.
I am running the NUnit tests (project in .Net Framework 4.5), as part of azure devops build pipeline.
- task: VSTest#2
inputs:
testAssemblyVer2: 'tests/**/*.Tests.dll'
pathtoCustomTestAdapters: '$(Build.SourcesDirectory)/packages'
codeCoverageEnabled: true
displayName: 'NUnit Testing'
- task: PublishCodeCoverageResults#1
inputs:
codeCoverageTool: JaCoCo
summaryFileLocation: '$(Common.TestResultsDirectory)/**/*.xml'
displayName: 'Publish Code Coverage'
// summaryFileLocation: '$(Common.TestResultsDirectory)/**/*.coverage'
But I am not able to see the coverage report, all I see the download link for coverage results...
How can I convert the .coverage report to JaCoCo format? OR generate the report directly in JaCoCo format?
I have seen some solution for .Net Core (link), but none for .Net framework
Update:
As per the release to Azure Devops for Sprint 150
When publishing code coverage reports, you no longer need to specify HTML files.
Therefore, the script in my illustration no longer needs to use the report generator tool directly to create the html report, and when publishing the coverage results, the directory containing those html reports doesn't need to be specified.
Edit:
The trick I've found for getting the coverage results from a .Net Framework project to show up on the Code Coverage tab is in the same line of thought to your linked article.
Don't run tests with the VS Test Task in Azure
Install the Report Generator and Coverlet tools directly
Use dotnet-vstest command for running tests through Coverlet
Publish reports generated with Report Generator and Cobertura format coverage results
Don't use the VS Test Task
Running this task will allow you to collect coverage with a simple checkbox, but you then surrender your opportunity to provide the content for the Code Coverage Tab
Install tools directly
Use a Powershell task (or similar) to install the Coverlet and Report Generator tools directly. This allows you to use them on projects that are not .Net Core.
"install tools:"
&dotnet tool install dotnet-reportgenerator-globaltool --tool-path . --version 4.0.12
&dotnet tool install coverlet.console --tool-path . --version 1.4.1
Use dotnet vstest through coverlet
It's my understanding that dotnet test doesn't play nice with .Net Framework projects/assemblies. However, we can still use the dotnet command, which we know will be on the agent machine, but we need to use it as a mechanism to get to the vstest.console.exe.
The Coverlet tool, as mentioned in the article you linked, will output coverage results in Cobertura format if you tell it to do so.
&$coverlet $unitTestFile.FullName --target "dotnet" --targetargs "vstest $($unitTestFile.FullName) --logger:trx" --format "cobertura"
Publish results
Complete script sample
note: this script is pretty rough, so use it as a thought exercise for your individual situation.
"install tools:"
&dotnet tool install dotnet-reportgenerator-globaltool --tool-path . --version 4.0.12
&dotnet tool install coverlet.console --tool-path . --version 1.4.1
"`nmake reports dir:"
mkdir .\reports
"`nrun tests:"
$unitTestFile = gci -Recurse | ?{ $_.FullName -like "*bin\*UnitTestProject2.dll" }
Write-Host "`$unitTestFile value: $unitTestFile"
$coverlet = "$pwd\coverlet.exe"
"calling $coverlet for $($unitTestFile.FullName)"
&$coverlet $unitTestFile.FullName --target "dotnet" --targetargs "vstest $($unitTestFile.FullName) --logger:trx" --format "cobertura"
"`ngenerate report(s)"
gci -Recurse |
?{ $_.Name -eq "coverage.cobertura.xml" } |
%{ &"$pwd\reportgenerator.exe" "-reports:$($_.FullName)" "-targetdir:reports" "-reporttypes:HTMLInline;HTMLChart" }
If you're struggling to figure out the escaping of quotes and such with the Coverlet command, YOU ARE NOT ALONE. I used the echoargs commandlet from PSCX more times than I care to admit so I could see what was actually getting provided to the .exe calls I was making.
The Results!!
...because that's really what matters
Original Answer:
Because of the way the linked article you mentioned is installing and using the report generator global tool I would think you can still follow those guidelines for creating the HTML inline and chart report types.
I'm not sure what is meant or how it works when the article says,
The point is the reporttypes: Use HTMLInLine for enabling the output on the Azure DevOps page. Azure DevOps Coverage page show index.html on the web.
I'm understanding that you can use the tool to create the HTML report from the .xml coverage results, and then publish the coverage results and report together with the Publish Code Coverage task.
So it seems all you need is to have an .xml format of the .coverage tool.
I didn't get it working in straight powershell, but you could follow the instructions from the Report Generator documentation to write a C# utility to access the Coverage.Analysis library.
For anyone looking for code coverage in Azure Devops (using classic editor, without Yaml), in current .NET (core) 5, with xUnit tests:
In your xUnit test project, add following (it generally comes by default in .NET 5, xUnit template now):
<PackageReference Include="coverlet.collector" Version="3.0.3" />
Keep checking for new version.
Head to Azure devops, create pipeline using classic editor. Do the restore, build steps. (Or you can choose dotnet core template as below):
In the test command of dotnet core task, add argument - --collect:"XPlat Code Coverage". Remember "XPlat Code Coverage" is friendly name and case sensitive. Your test command would look like:
Check this checkbox if you want to publish your test results: Publish test results and code coverage, but it won't publish code coverage. The functionality is not yet working (at least not in non-windows).
Next add - Publish code coverage results task. Choose "Code coverage tool" as "Cobertura" and in "Summary file" field, add $(Agent.TempDirectory)/**/coverage.cobertura.xml. Looks like this:
Save and Queue (in any agent, I use Ubuntu) and see the result once pipeline run completes:
Coverage report tab:
HTML Coverage reports and coverage cobertura xml are published as artifacts:
You can use Publish Code Coverage Results task in azure devops pipeline to see code coverage result in Jacoco format.
for further information about setup and configuration , please check the blog in MSDN
https://learn.microsoft.com/hi-in/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=tfs-2015#q--a
Hope it helps.