I have been having issues with sending build artifacts to my feed and can't figure out where my issue is at.
I forked over this repository from an Azure document since I am new to this and learning to create a CI/CD pipeline (https://github.com/Azure-Samples/python-docs-hello-world).
With the twine or universal package publishing setup guides there are steps for connecting to the feed such as creating a .piyrc file in your home directory but is that done locally or somewhere within the pipeline YAML?
Universal Publishing YAML
steps:
- task: UniversalPackages#0
displayName: 'Universal publish'
inputs:
command: publish
vstsFeed: 'cd75ead1-7beb-42f9-9477-e958501bb986'
publishDirectory: '$(Pipeline.Workspace)'
vstsFeedPublish: 'cd75ead1-7beb-42f9-9477-e958501bb986'
vstsFeedPackagePublish: drop
Twine Method
twine upload -r {Feed} --config-file $(PYPIRC_PATH) $(Pipeline.Workspace)
With Universal Publishing I receive an error about the path provided as being invalid.
With Twine I get an error about InvalidDistribution: Cannot find file (or expand pattern)
The $(Pipeline.Workspace) that I have written above was created as a path in my build pipeline to copy all files over from an Archive step. I see the artifact being made in the build pipeline and then downloaded in the first step of the release pipeline so I'm not sure what is going on or if it's something as simple as using the incorrect path.
With Twine I get an error about InvalidDistribution: Cannot find file (or expand pattern)
You need to specify the specific artifacts path instead of using the $(Pipeline.Workspace).
The $(pipeline.workspcae) is equal to the $(Agent.BuildDirectory). You could refer to this doc.
From the Github link, it seems that you want to publish a python package to feed.
You could refer to the following steps to create CI\CD.
In CI , you could Build sdist and publish the artifact to pipeline.
Here is the sample:
steps:
- task: UsePythonVersion#0
displayName: 'Use Python 3.6'
inputs:
versionSpec: 3.6
- script: 'python setup.py sdist'
displayName: 'Build sdist'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: dist'
inputs:
PathtoPublish: dist
ArtifactName: dist
In CD , you could set the build artifacts as resource and use twine to upload the python package to feed.
Here is an example:
twine upload -r AzureTest23 --config-file $(PYPIRC_PATH) D:\a\r1\a\{Source alias}\dist\*
The twine authenticate task could give the $(PYPIRC_PATH) variable.
If you want to determine your correct path, you can find it in the release log.
Note: If there are spaces or special characters in the path, they need to be escaped in cmd, otherwise they will not be recognized.
The name relates with the source alias, you could change it in artifacts source.
By the way, if you use the Universal Publish task, you also need to give the correct path.
Related
I want to publish a java jar file to azure devops artifacts section in azure devops. Once it is there, I want to be able to setup my gradle so it can download jars from that location. I think I understand how to do the second step, but how to do the first step is not clear to me. I can use an azure pipelines task to build the jar and publish the artifact. However the published artifact does not end up in the location I want. It ends up in an artifacts section of the pipeline run, it does not end up in the artifacts section of ADO. Here is the code:
steps:
- checkout: self
path: 'my-repo/'
- task: Gradle#2
displayName: Test and build jar
inputs:
gradleWrapperFile: 'gradlew'
tasks: 'build'
publishJUnitResults: false
javaHomeOption: 'JDKVersion'
- task: PublishBuildArtifacts#1
inputs:
pathToPublish: '$(System.DefaultWorkingDirectory)/build/libs/commons.jar'
You can see in this image. The green arrow points to where the artifact ends up after successful execution of the above code. However when I click on the artifacts section indicated by the blue arrow it is not there. How do I get it to publish over there so it can be consumed by my build process on my local machine? Thanks in advance.
PublishBuildArtifacts#1 publishes 'Build artifacts' which are later available to be downloaded by DownloadBuildArtifacts#0 in later pipeline stages or in another pipelines. (And it will be only accessible where your green arrow points)
'Build artifact' is not the same thing as 'Artifacts Feed'
'Artifacts Feed' serves as source of packages which can be downloaded for example by NuGet if you configure in nuget.exe as a source your 'Artifacts Feed'
For example you can add NuGet package there using NuGet push(with firstly configured Source)
https://learn.microsoft.com/en-us/azure/devops/artifacts/get-started-nuget?view=azure-devops&tabs=windows#publish-a-nuget-package-by-using-the-command-line
To publish Universal Packages there is already task UniversalPackages#0
https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/universal-packages?toc=%2Fazure%2Fdevops%2Fartifacts%2Ftoc.json&bc=%2Fazure%2Fdevops%2Fartifacts%2Fbreadcrumb%2Ftoc.json&view=azure-devops&tabs=yaml
I am trying to use AutoIT in my automated tests for the project. Locally I am able to register the COM Library using regsvr32 but when I try to do the same from my azure pipeline, the script runs continuously.
I have my azure pipeline yml as following:
- job: Tests
displayName: Automated Tests
pool:
vmImage: "windows-latest"
steps:
- task: NuGetToolInstaller#1
- task: DotNetCoreCLI#2
displayName: Restore Packages
inputs:
command: 'restore'
projects: 'Free.Automation/Free.Automation.csproj'
feedsToUse: 'config'
nugetConfigPath: 'Free.Automation/nuget.config'
- task: BatchScript#1
displayName: Register AutoIT
inputs:
filename: 'Free.Automation/autoit.bat'
- task: MSBuild#1
inputs:
solution: "Free.Automation/Free.Automation.sln"
And this is the bat file I am using:
cd c:\windows\system32
regsvr32 C:\Users\%USERNAME%\.nuget\packages\autoitx.dotnet\3.3.14.5\build\AutoItX3.dll
I verified that the path of azure pipeline space is something D:\1\a\s but not sure how the directory works.
Could anyone help me registering the COM lib on azure hosted pipeline space?
With Azure DevOps Microsoft-hosted agent, you can't get your local files directly. So if you want to use COM DLLs, you need to include them in your source code files.
I recommend that you have a lib folder to store your DLLs in. Please make sure that your DLLs are referenced correctly as a relative path in .csproj.
I verified that the path of azure pipeline space is something D:\1\a\s but not sure how the directory works.
In Azure DevOps, you can use the predefined variable $(System.DefaultWorkingDirectory) to get the local path on the agent where your source code files are downloaded. That's the "azure pipeline space D:\1\a\s" you mentioned.
In our .NET Web API project, we tried to build API project in Azure DevOps and publish the artifact to a folder with the pipeline task below:
- task: DotNetCoreCLI#2
displayName: Publish web API artifact
inputs:
command: publish
publishWebProjects: false
arguments: '$(Build.SourcesDirectory)\XYZ.Research.API\XYZ.Research.API.csproj --configuration $(BuildConfiguration) --output testpath'
zipAfterPublish: true
modifyOutputPath: true
But I am not sure which folder the artifact is kept. Below is the log from this step:
2020-07-31T12:04:23.6282186Z ##[section]Starting: Publish web API artifact
2020-07-31T12:04:23.6590490Z ==============================================================================
2020-07-31T12:04:23.6591051Z Task : .NET Core
2020-07-31T12:04:23.6591393Z Description : Build, test, package, or publish a dotnet application, or run a custom dotnet command
2020-07-31T12:04:23.6591740Z Version : 2.172.2
2020-07-31T12:04:23.6591974Z Author : Microsoft Corporation
2020-07-31T12:04:23.6592357Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
2020-07-31T12:04:23.6592942Z ==============================================================================
2020-07-31T12:04:25.5581194Z [command]C:\windows\system32\chcp.com 65001
2020-07-31T12:04:25.5581889Z Active code page: 65001
2020-07-31T12:04:25.5583746Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.
2020-07-31T12:04:25.5588792Z [command]C:\hostedtoolcache\windows\dotnet\dotnet.exe publish d:\a\1\s\XYZ.Research.API\XYZ.Research.API.csproj --configuration Release --output testpath
.....
some warning message ignored
.....
2020-07-31T12:04:38.0843543Z XYZ.Research.API -> d:\a\1\s\XYZ.Research.API\bin\Release\netcoreapp3.0\XYZ.Research.API.dll
2020-07-31T12:04:38.9127845Z XYZ.Research.API -> d:\a\1\s\testpath\
2020-07-31T12:04:46.0295716Z Info: Azure Pipelines hosted agents have been updated to contain .Net Core 3.x (3.1) SDK/Runtime along with 2.1. Unless you have locked down a SDK version for your project(s), 3.x SDK might be picked up which might have breaking behavior as compared to previous versions.
2020-07-31T12:04:46.0296632Z Some commonly encountered changes are:
2020-07-31T12:04:46.0297619Z If you're using `Publish` command with -o or --Output argument, you will see that the output folder is now being created at root directory rather than Project File's directory. To learn about more such changes and troubleshoot, refer here: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
2020-07-31T12:04:46.0442329Z ##[section]Finishing: Publish web API artifact
Because we will need the file location in next step (deployment), I tried
d:\a\1\s\testpath\XYZ.Reserch.API.zip
d:\a\1\s\testpath\XYZ.Reserch.API\XYZ.Reserch.API.zip
but none of these location has the artifact file.
Did anyone see this issue before? Any help would be appreciated.
------------------- update -------------------------------
As #Source Code suggested, I used task "PowerShell#2" and find that the artifact file are actually in "D:\a\1\s\testpath\testpath.zip". That means the 'testpath' sub-folder are created in $(Build.SourceDirectory) and the artifact file are also renamed to 'test.zip'.
I would recommend that you add a PowerShell/Bash/Cmd task after your DotNetCoreCLI#2 task and run a inline script with the 'ls' command that should list all the items to the results for you. This will allow you to see what is actually there after the task.
If on a Windows agent:
- task: PowerShell#2
displayName: List Files Post Publish
inputs:
targetType: inline
script: Get-ChildItem
If on Linux or Mac
- task: Bash#3
displayName: List Files Post Publish
inputs:
targetType: inline
script: ls
Additionally I noticed you're providing your csproj file via the arguments parameter. There is a parameter named projects which should be used for this. Also you may consider using the the artifacts staging directory as your output directory. The task would look like this:
- task: DotNetCoreCLI#2
displayName: Publish web API artifact
inputs:
command: publish
projects: '$(Build.SourcesDirectory)\XYZ.Research.API\XYZ.Research.API.csproj'
publishWebProjects: false
arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
zipAfterPublish: true
modifyOutputPath: true
One important thing to note is if you do change the output directory ensure that you change the working directory for the PowerShell or Bash tasks so you output the contents of the correct directory. It defaults to the $(Build.SourcesDirectory) so ensure that you change this if needed.
Recently just built an Azure Pipeline where in one stage there are different zip files in the artifact staging directory. What I'm trying to achieve is publish to the drop folder all the zip files from the staging folder with PublishPipelineArtifact task.
I have 2 archived zip files in artifact staging directory:
$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
$(Build.ArtifactStagingDirectory)/cli_scripts_$(Build.BuildId).zip
In my azure-pipelines.yml file please find the publish task:
- task: PublishPipelineArtifact#0
displayName: 'Publish pipeline artifacts'
inputs:
targetPath: $(Build.ArtifactStagingDirectory)/**
This gives the following error:
[error] Path does not exist: d:\a\1\a**
I have already tried with the following as well but none of them working:
$(Build.ArtifactStagingDirectory)/**
$(Build.ArtifactStagingDirectory)/**/*.zip
$(Build.ArtifactStagingDirectory)/*.zip
Question:
What is the pattern for targetPath to move all the zip files from that folder?
Any help is appreciated!
What finally resolved the issue is including a pattern with archiveFilePatterns in the task and not combining with the targetPath as I originally tried.
The solution which worked well is the following:
- task: PublishPipelineArtifact#0
displayName: 'Publish pipeline artifacts'
inputs:
targetPath: $(Build.ArtifactStagingDirectory)/
archiveFilePatterns: '**/*.zip'
The official documentation does not really states this but it was giving the idea using the pattern attribute: Publish and download artifacts
I hope that helps someone in the future.
You can use the .artifactignore file to filter what the PublishPipelineArtifact task can see. Make sure the file is in the folder that you're publishing, as mentioned here:
I recently moved my sources to azure devOps to use cd/ci and all the other cool stuff.
Now i created my first build pipeline to build the android part of my Xamarin project. But I end up getting an error message, that a resource of a referenced project could not be found and i shall do a package restore and try again.
Now, since i have azure hosted build agents and not self hosted, i have no ways of setting the agent up properly before doing the build.
But i guess there should be some way to properly configure the build pipeline to do all the necessary stuff.
Its just that i have no clue what i should add to my yaml file in order to fix this stuff.
This is the error message i got:
##[error]C:\Program Files\dotnet\sdk\2.2.105\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(208,5): Error NETSDK1004: Assets file 'd:\a\1\s\*****\*****\*****\*****\obj\project.assets.json' not found. Run a NuGet package restore to generate this file.
The problem is, that this file should be generated by compiling a referenced project and is not part of a nuget package.
Here is my build pipeline as far as i figured it out by myself.
# Xamarin.Android
# Build a Xamarin.Android project.
# Add steps that test, sign, and distribute an app, save build artifacts, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/xamarin
trigger:
- Share/main
pool:
vmImage: 'VS2017-Win2016'
variables:
buildConfiguration: 'Debug'
outputDirectory: '$(build.binariesDirectory)/$(buildConfiguration)'
steps:
- task: NuGetToolInstaller#1
inputs:
versionSpec: 5.1.0
- task: NuGetCommand#2
displayName: 'Restore NuGet Packages'
inputs:
command: restore
restoreSolution: '**/*.sln'
- task: XamarinAndroid#1
inputs:
projectFile: 'Mobile4/Droid/Mobile4.Droid.csproj'
outputDirectory: '$(outputDirectory)'
configuration: '$(buildConfiguration)'
- task: AndroidSigning#3
inputs:
apksign: false
zipalign: false
apkFiles: '$(outputDirectory)/*.apk'
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: '$(outputDirectory)'
The build always breaks on step XamarinAndroid
I hope you can help me.
The solution must be out there somewhere, i just cannot see it right now.
Thx in Advance.
Mav
[error]C:\Program Files\dotnet\sdk\2.2.105\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(208,5):
Error NETSDK1004: Assets file
'd:\a\1\s********************\obj\project.assets.json' not found.
Run a NuGet package restore to generate this file.
According to this error message, the project is .NetCore and its SDK used is 2.2.105. For the file "....\obj\project.assets.json", whether the project.assets.json exists is determined by package restore step. Now, it prompt this could not be found, it means the package restore does not restore this file successfully.
As I mentioned previously, it is a .NetCore project. So you should use dotnet restore instead of nuget restore. For .NetCore project, the obj folder restored by nuget restore does not contain project.assets.json in it.
So, to solve the issue you meet, you should replace the task Nuget restore as dotnet restore: dotnet.
- task: DotNetCoreCLI#2
displayName: 'dotnet restore'
inputs:
command: restore
projects: '**/*.csproj'
vstsFeed: 'e157d03d-******-fc06f9e13177'