I build and test NET 6 app in docker which works well - I see test output in console. When I try to publish test results I encounter an error. Full task output:
Starting: Publish test results
============================================================================== Task : Publish Test Results Description : Publish test
results to Azure Pipelines Version : 2.170.1 Author :
Microsoft Corporation Help :
https://learn.microsoft.com/azure/devops/pipelines/tasks/test/publish-test-results
============================================================================== /usr/bin/dotnet --version
3.1.418 Async Command Start: Publish test results Publishing test results to test run '22493988' Test results remaining: 65. Test run
id: 22493988
> ##[warning]Failed to publish test results: Cannot insert duplicate key row in object 'TestResult.tbl_TestCaseReference' with unique index
'ix_TestCaseReference2'. The duplicate key value is (1, 3559, 0, 0, 0,
0x2b12334e709060df7748bee70b2472ff22c0b671ae3e3b997395858711e5eef5,
0x89d7d75814a51ce876ec351fcb9aec22313feeece07fb1d5a3023a3e7a1f634d,
UnitTest, 0, 0, ). Async Command End: Publish test results Finishing:
Publish test results
Copy test results and publish steps
- pwsh: |
$id=docker images --filter "label=test=$(Build.BuildId)" -q | Select-Object -First 1
docker create --name testcontainer $id
docker cp testcontainer:/testresults ./testresults
docker rm testcontainer
displayName: 'Copy test results'
- task: PublishTestResults#2
inputs:
testResultsFormat: 'VSTest'
testResultsFiles: '**/*.trx'
searchFolder: '$(System.DefaultWorkingDirectory)/testresults'
displayName: 'Publish test results'
Copy tests results without errors:
Starting: Copy test results
============================================================================== Task : PowerShell Description : Run a PowerShell script on
Linux, macOS, or Windows Version : 2.170.1 Author :
Microsoft Corporation Help :
https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
============================================================================== Generating script.
========================== Starting Command Output =========================== /usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/vsts/agent/_work/_temp/20d795dd-e081-4216-b9b1-327d257bbcb2.ps1'
69ba144bfd471794f0e11047a2b214bd7190b96b5b47616957927125c740f266
testcontainer
Finishing: Copy test results
I took code from tutorial: https://www.programmingwithwolfgang.com/run-xUnit-inside-docker-during-ci-build
The xunit test publisher may be an issue here.
Try resolving the error by t by changing Xunit to VSTest.
And also, you don't necessarily need a separate Publish Test Results job in the pipeline because built-in tasks like the Visual Studio Test task automatically publish test results to the pipeline.
Please refer this MSFT doc Publish Test Results task for more information.
I contacted my colleagues and turns out it's a bug in Azure. I replaced VSTest publisher and format with XUnit publisher and XML format - the XML was successfully uploaded. Another issue was that XML was only for one project in solution, so I ended up dropping that step all together because I don't really need this information anyway (if test failed, so fails build).
Related
I am using
coverlet.msbuild v3.2.0
coverlet.collector v3.2.0
and dotnet sdk v6.0.402
When I run this test command in powershell (restore and build ran before that)
dotnet test --no-build --no-restore --collect:"XPlat Code Coverage" /p:Configuration=$Cfg /p:CollectCoverage=true /p:CoverletOutput=.\CodeCoverage\ --% /p:CoverletOutputFormat=\"cobertura,opencover\"
The report files
coverage.cobertura.xml
coverage.opencover.xml
are created and the powershell output is this:
Test run for C:\Users\MyUser\repos\My.Project\Specs\bin\Release\net472\My.Project.Specs.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Attachments:
C:\Users\MyUser\repos\My.Project\Specs\TestResults\dbe8cb53-3ec5-4227-a231-3bfedf94694f\coverage.cobertura.xml
Passed! - Failed: 0, Passed: 14, Skipped: 0, Total: 14, Duration: 910 ms - My.Project.Specs.dll (net472)
Calculating coverage result...
Generating report '.\CodeCoverage\coverage.cobertura.xml'
Generating report '.\CodeCoverage\coverage.opencover.xml'
+----------------------+--------+--------+--------+
| Module | Line | Branch | Method |
+----------------------+--------+--------+--------+
| My.Project | 20.23% | 18.53% | 20.09% |
+----------------------+--------+--------+--------+
+---------+--------+--------+--------+
| | Line | Branch | Method |
+---------+--------+--------+--------+
| Total | 20.23% | 18.53% | 20.09% |
+---------+--------+--------+--------+
| Average | 20.23% | 18.53% | 20.09% |
+---------+--------+--------+--------+
I have this azure-pipeline task:
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
arguments: '--no-restore --no-build --collect:"XPlat Code Coverage" /p:Configuration=$(Build.Configuration) /p:CollectCoverage=true /p:CoverletOutput=$(Build.SourcesDirectory)\CodeCoverage --% /p:CoverletOutputFormat=\"cobertura,opencover\"'
publishTestResults: true
Which executes this command:
C:\agent\_work\_tool\dotnet\dotnet.exe test --logger trx --results-directory C:\agent\_work\_temp --no-restore --no-build "--collect:XPlat Code Coverage" /p:Configuration=Release /p:CollectCoverage=true /p:CoverletOutput=C:\agent\_work\9\s\CodeCoverage --% "/p:CoverletOutputFormat=\cobertura,opencover\""
And has this output
Test run for C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\My.Project.Specs.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\LivingDoc.SpecFlowPlugin.dll
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
-> Loading plugin C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\a4203b08-db42-449b-86d7-55cb48c54fc4\a4203b08-db42-449b-86d7-55cb48c54fc4\assembly\dl3\e501d05b\fee426f7_ddf4d801\My.Project.Specs.dll
-> Using specflow.json
-> LivingDocPlugin: Output generated in: C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TestExecution.json
Results File: C:\agent\_work\_temp\BUILDMACHINE01$_BUILDMACHINE01_2022-11-10_09_25_37.trx
Passed! - Failed: 0, Passed: 14, Skipped: 0, Total: 14, Duration: 985 ms - My.Project.Specs.dll (net472)
Attachments:
C:\agent\_work\_temp\faa5dbb6-5931-43fe-880e-a37576815c1c\coverage.cobertura.xml
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
Async Command Start: Publish test results
Publishing test results to test run '1436028'.
TestResults To Publish 12, Test run id:1436028
Test results publishing 12, remaining: 0. Test run id: 1436028
Published Test Run : https://dev.azure.com/orgteamservices/My.Project/_TestManagement/Runs?runId=1436028&_a=runCharts
Async Command End: Publish test results
Finishing: Test
No reports are generated on the on-prem agent machine in the expected directory.
The agent does create this though
C:\agent\_work\_temp\faa5dbb6-5931-43fe-880e-a37576815c1c\coverage.cobertura.xml
Why is it not creating both reports in the expected directory?
The problem:
Since Visual Studio Build tools 17.4.0 any arguments (/p:) passed to coverlet are ignored.
The workaround:
Specifying publishTestResults: true as input parameter for DotNetCoreCLI#2 task (or no publishTestResults input at all) causes the following arguments to be added: --logger trx --results-directory $(Agent.TempDirectory). This is normal behaviour btw.
This in turn causes the coverage results to be placed in the agents temp directory as the coverlet arguments are ignored.
So set publishTestResults to false
Add the arguments which are now omitted yourself:
--logger trx --results-directory CodeCoverage
Pass any additional runsettings either by file or by adding the following runsettings arguments:
--DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover,cobertura
This should mitigate the problem for now.
This is what my entire yml task looks like:
- task: DotNetCoreCLI#2
displayName: 'Run acceptance tests'
inputs:
command: 'test'
projects: './src/Service.Requirements/Service.Requirements.csproj'
workingDirectory: './src/Service.Requirements'
publishTestResults: false
arguments: '--logger trx --results-directory CodeCoverage --configuration $(buildConfiguration) --no-build --collect "XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Exclude=[Service.Data]*'
Ok, found a solution.
I have created a coverlet.runsettings file
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
<RunConfiguration>
<ResultsDirectory>./CodeCoverage/</ResultsDirectory>
</RunConfiguration>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="XPlat code coverage">
<Configuration>
<Format>cobertura,opencover</Format>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
and changed the task like this
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
- sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/CodeCoverage/coverage.opencover.xml
+ sonar.cs.opencover.reportsPaths=$(Agent.TempDirectory)/**/coverage.opencover.xml
- arguments: --no-restore --no-build --collect:"XPlat Code Coverage" /p:Configuration=$(Build.Configuration) /p:CollectCoverage=true /p:CoverletOutput=$(Build.SourcesDirectory)\CodeCoverage\ --% /p:CoverletOutputFormat=\"cobertura,opencover\"
+ arguments: --no-restore --no-build --collect:"XPlat Code Coverage" --settings ./My.Project.Specs/coverlet.runsettings /p:Configuration=$(Build.Configuration) /p:CollectCoverage=true /p:CoverletOutput=$(Agent.TempDirectory)\ --% /p:CoverletOutputFormat=\"cobertura,opencover\"
- summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/coverage.cobertura.xml'
+ summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
I specify the paths to the report files using the agents temp directory and a glob pattern.
This task creates this output now (abbreviated):
C:\agent\_work\_tool\dotnet\dotnet.exe test --logger trx --results-directory C:\agent\_work\_temp --no-restore --no-build "--collect:XPlat Code Coverage" --settings ./My.Project.Specs/coverlet.runsettings /p:Configuration=Release /p:CollectCoverage=true /p:CoverletOutput=C:\agent\_work\_temp\ --% "/p:CoverletOutputFormat=\cobertura,opencover\""
Test run for C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\My.Project.Specs.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\LivingDoc.SpecFlowPlugin.dll
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
-> Loading plugin C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\fa03db93-d250-4d1a-acfc-4ec086feac88\fa03db93-d250-4d1a-acfc-4ec086feac88\assembly\dl3\6a2f2724\d56f5937_94f5d801\My.Project.Specs.dll
-> Using specflow.json
-> LivingDocPlugin: Output generated in: C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TestExecution.json
Results File: C:\agent\_work\_temp\BUILDMACHINE01$_BUILDMACHINE01_2022-11-11_07_10_08.trx
Passed! - Failed: 0, Passed: 14, Skipped: 0, Total: 14, Duration: 876 ms - My.Project.Specs.dll (net472)
Attachments:
C:\agent\_work\_temp\c68cb66e-3fd4-498d-b3e3-218ed75f68a8\coverage.cobertura.xml
C:\agent\_work\_temp\c68cb66e-3fd4-498d-b3e3-218ed75f68a8\coverage.opencover.xml
It still ignores all the directories I provide but creates both reports in the temp directory where I can access them.
Please include below steps:
It worked for me.
steps:
- task: UseDotNet#2
displayName: Install .NET Core 3.1 SDK
inputs:
version: '3.1.x'
packageType: sdk
- task: DotNetCoreCLI#2
displayName: 'Run Unit Tests'
condition: succeeded()
inputs:
projects: 'tests/**/*.csproj'
arguments: '--logger trx --configuration Release /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:ExcludeByFile="**/*.cshtml" --collect "Code Coverage"'
command: test
publishTestResults: true
I want the pipeline to fail if the threshold of failing test is more than 70%, how should I implement it in Azure Devops Pipeline. The test are running with mvn install command and test script.
Update:
Based on your reply, you need to fail the pipeline based on the number of failed tests.
There is no out-of-box method can achieve it in Azure Pipeline.
I suggest that you can check the result based on the Test Result file(XML file) created by Maven test.
You can use the Maven task to run the test and generate the test xml file.
Then you can use PowerShell task to determine if the pipeline can pass.
Here is an example:
steps:
- task: Maven#3
displayName: 'Maven pom.xml'
inputs:
goals: test command
testResultsFiles: '$(build.artifactstagingdirectory/surefire-reports/TEST.xml'
continueOnError: true
- powershell: |
[xml]$xml= Get-Content $(build.artifactstagingdirectory)\surefire-reports\Test.xml
$failures= $xml.testsuite.failures
If($failures -gt xx)
{
Write-host "##vso[task.complete result=failed;]DONE"
}
displayName: 'PowerShell Script'
I am trying to run a series of Postman test in my Azure build pipeline but keep getting errors that Newman is not installed, I have checked by going to the exact location and running the Newman commands without any issue. My screenshots show I have implemented them and the errors.
I see you have used commands as call newman run.
Instead use:
newman run collection.json -e environment_file.json --reporters cli,junit,htmlextra --reporter-junit-export junitReport.xml
It works for me from Azure Pipelines.
This is what mine looks like in YAML - if you click on 'View YAML' on the top right you can see the difference. I think you have this 'call' word that shouldn't be there.
- task: CmdLine#2
displayName: Run newman tests
inputs:
script: 'newman run "$(System.DefaultWorkingDirectory)/${{ parameters.e2eCollectionPath }}" -e "$(System.DefaultWorkingDirectory)/${{ parameters.e2eEnvironmentPath }}" --reporters cli,junit --reporter-junit-export $(System.DefaultWorkingDirectory)/report.xml'
In our .NET Web API project, we tried to build API project in Azure DevOps and publish the artifact to a folder with the pipeline task below:
- task: DotNetCoreCLI#2
displayName: Publish web API artifact
inputs:
command: publish
publishWebProjects: false
arguments: '$(Build.SourcesDirectory)\XYZ.Research.API\XYZ.Research.API.csproj --configuration $(BuildConfiguration) --output testpath'
zipAfterPublish: true
modifyOutputPath: true
But I am not sure which folder the artifact is kept. Below is the log from this step:
2020-07-31T12:04:23.6282186Z ##[section]Starting: Publish web API artifact
2020-07-31T12:04:23.6590490Z ==============================================================================
2020-07-31T12:04:23.6591051Z Task : .NET Core
2020-07-31T12:04:23.6591393Z Description : Build, test, package, or publish a dotnet application, or run a custom dotnet command
2020-07-31T12:04:23.6591740Z Version : 2.172.2
2020-07-31T12:04:23.6591974Z Author : Microsoft Corporation
2020-07-31T12:04:23.6592357Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
2020-07-31T12:04:23.6592942Z ==============================================================================
2020-07-31T12:04:25.5581194Z [command]C:\windows\system32\chcp.com 65001
2020-07-31T12:04:25.5581889Z Active code page: 65001
2020-07-31T12:04:25.5583746Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.
2020-07-31T12:04:25.5588792Z [command]C:\hostedtoolcache\windows\dotnet\dotnet.exe publish d:\a\1\s\XYZ.Research.API\XYZ.Research.API.csproj --configuration Release --output testpath
.....
some warning message ignored
.....
2020-07-31T12:04:38.0843543Z XYZ.Research.API -> d:\a\1\s\XYZ.Research.API\bin\Release\netcoreapp3.0\XYZ.Research.API.dll
2020-07-31T12:04:38.9127845Z XYZ.Research.API -> d:\a\1\s\testpath\
2020-07-31T12:04:46.0295716Z Info: Azure Pipelines hosted agents have been updated to contain .Net Core 3.x (3.1) SDK/Runtime along with 2.1. Unless you have locked down a SDK version for your project(s), 3.x SDK might be picked up which might have breaking behavior as compared to previous versions.
2020-07-31T12:04:46.0296632Z Some commonly encountered changes are:
2020-07-31T12:04:46.0297619Z If you're using `Publish` command with -o or --Output argument, you will see that the output folder is now being created at root directory rather than Project File's directory. To learn about more such changes and troubleshoot, refer here: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
2020-07-31T12:04:46.0442329Z ##[section]Finishing: Publish web API artifact
Because we will need the file location in next step (deployment), I tried
d:\a\1\s\testpath\XYZ.Reserch.API.zip
d:\a\1\s\testpath\XYZ.Reserch.API\XYZ.Reserch.API.zip
but none of these location has the artifact file.
Did anyone see this issue before? Any help would be appreciated.
------------------- update -------------------------------
As #Source Code suggested, I used task "PowerShell#2" and find that the artifact file are actually in "D:\a\1\s\testpath\testpath.zip". That means the 'testpath' sub-folder are created in $(Build.SourceDirectory) and the artifact file are also renamed to 'test.zip'.
I would recommend that you add a PowerShell/Bash/Cmd task after your DotNetCoreCLI#2 task and run a inline script with the 'ls' command that should list all the items to the results for you. This will allow you to see what is actually there after the task.
If on a Windows agent:
- task: PowerShell#2
displayName: List Files Post Publish
inputs:
targetType: inline
script: Get-ChildItem
If on Linux or Mac
- task: Bash#3
displayName: List Files Post Publish
inputs:
targetType: inline
script: ls
Additionally I noticed you're providing your csproj file via the arguments parameter. There is a parameter named projects which should be used for this. Also you may consider using the the artifacts staging directory as your output directory. The task would look like this:
- task: DotNetCoreCLI#2
displayName: Publish web API artifact
inputs:
command: publish
projects: '$(Build.SourcesDirectory)\XYZ.Research.API\XYZ.Research.API.csproj'
publishWebProjects: false
arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
zipAfterPublish: true
modifyOutputPath: true
One important thing to note is if you do change the output directory ensure that you change the working directory for the PowerShell or Bash tasks so you output the contents of the correct directory. It defaults to the $(Build.SourcesDirectory) so ensure that you change this if needed.
I'm running into a weird issue when running Newman on Azure DevOps Pipeline. Here's a summary of what's happening:
Postman tests run fine locally
Pipeline tests fail only on the first test
Post
Test A
POST XXXXX [500 Internal Server Error, 442B, 8.6s]
1⠄ JSONError in test-script
Test A Copy
POST XXX [200 OK, 692B, 8.9s]
√ Is Successful
√ Status Code
√ Status Message
---
# failure detail
1. JSONError
No data, empty input at 1:1
^
at test-script
It doesn't seem to matter what the exact test is, it always fails if it's the first. As a way to demonstrate this I've copied the test that was failing so that now I had
Test A
Test A Copy
Test B
Test ...
And suddenly Test A Copy works. So it's not the contents of the test but rather the first test to be tested. All of these tests are POST's
Test A Contents:
var jsonData = pm.response.json();
pm.test("Is Successful", function() {
pm.expect(jsonData.IsSuccessful).to.be.true;
})
pm.test("Status Code", function() {
pm.response.to.have.status(200);
})
pm.test("Status Message", function() {
pm.expect(jsonData.StatusMessage).eql("Document insert successful.");
})
Nothing too fancy, so why would this fail on the first run (TEST A) but not the second (TEST A Copy). It doesn't matter which test it is, if I were to run TEST B first this would be the one to fail.
It almost looks like the first request is what's waking up the server and then everything is okay.
I run the Azure Devops Rest API in Postman and use the export json file to run the Postman test in Pipeline
Here are my steps to run newman in azure pipeline, you can refer to them.
Step1: Export the Collection in PostMan.
Step2: Upload the Json file(e.g. APITEST.postman_collection.json) to Azure Repo.
Step3: Create a pipeline and add the install Newman step, run Postman test step.
Example:
steps:
- script: |
npm install -g newman#5.1.2
workingDirectory: '$(System.DefaultWorkingDirectory)'
displayName: 'Command Line Script'
- script: 'newman run TEST.postman_collection.json --reporters cli,junit --reporter-junit-export Results\junitReport.xml '
workingDirectory: '$(build.sourcesdirectory)'
displayName: 'Command Line Script
or run with the Newman the cli Companion for Postman task(This is an extension task).
steps:
- script: |
npm install -g newman#4.6.1
workingDirectory: '$(System.DefaultWorkingDirectory)'
displayName: 'Command Line Script'
- task: NewmanPostman#4
displayName: 'Newman - Postman'
inputs:
collectionFileSource: 'TEST.postman_collection.json'
environmentSourceType: none
ignoreRedirect: false
bail: false
sslInsecure: false
htmlExtraDarkTheme: false
htmlExtraLogs: false
htmlExtraTestPaging: false
Step4: Run the pipeline and it can display the same api results as in postman.