Datafactory npm validation error (yml ci/cd pipelines) - azure

I've been followings microsoft's documentation for the new yml ci-cd datafactory deployment. However, I keep getting a validation error when trying to implement CI/CD pipelines for Azure Data Factory with a yml file, after the installation of npm.
I get the following error message:
Error: Command failed: node /home/vsts/work/1/s/DataFactories/joaov1/build/downloads/main.js validate /home/vsts/work/1/s DataFactories/joaov1/subscriptions/000000-0000-0000-0000-864d437bd294/resourceGroups/rg-hi-joaov1-dev/providers/Microsoft.DataFactory/factories/adf-client-joaov1-dev
Execution finished....
Found npm debug log, make sure the path matches with the one in npm's output: /home/vsts/.npm/_logs/2021-12-01T21_20_55_974Z-debug.log
I'm quite confident that it is a an issue with the path, but after hours of research online I still can't get it to work...
My file structure is the following:
And my code is as follows:
# Sample YAML file to validate and export an ARM template into a build artifact
# Requires a package.json file located in the target repository
trigger:
- master
# parameters:
# - name: 'dev'
# - name: 'test'
# - name: 'prod'
pool:
vmImage: 'ubuntu-latest'
steps:
# Installs Node and the npm packages saved in your package.json file in the build
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- task: Npm#1
inputs:
command: 'install'
workingDir: '$(Build.Repository.LocalPath)/DataFactories/joaov1/build' #replace with the package.json folder
verbose: true
displayName: 'Install npm package'
# Validates all of the Data Factory resources in the repository. You'll get the same validation errors as when "Validate All" is selected.
# Enter the appropriate subscription and name for the source factory.
- task: Npm#1
inputs:
command: 'custom'
workingDir: '$(Build.Repository.LocalPath)/DataFactories/joaov1/build' #replace with the package.json folder
customCommand: 'run build validate $(Build.Repository.LocalPath)/DataFactories/joaov1/subscriptions/000000-0000-0000-974d-864d437bd294/resourceGroups/rg-hi-joaov1-dev/providers/Microsoft.DataFactory/factories/adf-client-joaov1-dev'
displayName: 'Validate'
# Validate and then generate the ARM template into the destination folder, which is the same as selecting "Publish" from the UX.
# The ARM template generated isn't published to the live version of the factory. Deployment should be done by using a CI/CD pipeline.
- task: Npm#1
inputs:
command: 'custom'
workingDir: '$(Build.Repository.LocalPath)/DataFactories/joaov1/build' #replace with the package.json folder
customCommand: 'run build export $(Build.Repository.LocalPath)/DataFactories/joaov1/subscriptions/000000-0000-4aab-974d-864d437bd294/resourceGroups/rg-hi-joaov1-dev/providers/Microsoft.DataFactory/factories/adf-client-joaov1-dev "ArmTemplate"'
displayName: 'Validate and Generate ARM template'
# Publish the artifact to be used as a source for a release pipeline.
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Build.Repository.LocalPath)/DataFactories/joaov1/build/ArmTemplate' #replace with the package.json folder
artifact: 'ArmTemplates'
publishLocation: 'pipeline'
Any help would be great,
Thank you so much,
Joao

You ADF Build tasks Node version should be changed from 10.X to 14.X.
- task: NodeTool#0
inputs:
versionSpec: '14.x'
displayName: 'Install Node.js'

Related

[error]Bash exited with code '1' - bad substitution in Azure YAML pipeline

I am trying to integrate storybook/chromatic into my azure pipeline, however i am getting the following error:
/home/vsts/work/_temp/60ae6370-8818-4ab7-a2d8-f4a0443fadf9.sh: line 1: --project-token=${project_token}: bad substitution
I followed this https://www.chromatic.com/docs/azure-pipelines
I did obviously put my project token where it was needed.
Based on the error message, it seems that this issue comes from a script format or symbol.
I have tested the YAML sample from the link you shared and it can work fine.
Here are my steps, you can refer to them.
Step1 : Set the variable CHROMATIC_PROJECT_TOKEN
Step2: Run the YAML sample:
stages:
- stage: Test
displayName: Chromatic Testing
# Job list
jobs:
- job: Chromatic_Deploy
displayName: Install packages and publishes to Chromatic
steps:
- task: NodeTool#0
inputs:
versionSpec: '12.x'
displayName: 'Install Node.js'
- task: Npm#1
inputs:
command: 'install'
workingDir: '$(build.sourcesdirectory)'
- task: CmdLine#2
displayName: Publish to Chromatic
inputs:
script: npx chromatic --project-token=${CHROMATIC_PROJECT_TOKEN}
Here is my result:
You can also try to use --project-token=$(CHROMATIC_PROJECT_TOKEN) in the YAML sample.

PublishCucumberReport - Can't find output path of cucumber report

The plugin can't find files and generate HTML report on Azure DevOps
Azure Devops plugin - https://marketplace.visualstudio.com/items?itemName=MaciejMaciejewski.azure-pipelines-cucumber
azure-pipelines.yml
jobs:
# Build Electron
- job: UserAcceptanceTest
displayName: E2E-Tests
pool:
name: ado-win-pool
timeoutInMinutes: 120
steps:
- task: CopyFiles#2
inputs:
sourceFolder: $(Build.SourcesDirectory)
targetFolder: $(Build.ArtifactStagingDirectory)
- task: PublishPipelineArtifact#1
displayName: 'Publishing build artifacts'
inputs:
targetPath: $(Build.ArtifactStagingDirectory)
- task: NodeTool#0
displayName: 'Install Node 12.x'
inputs:
versionSpec: 12.x
- task: PublishCucumberReport#1
displayName: 'Publish Cucumber Report'
inputs:
jsonDir: target/results/cucumber/
outputPath: target/results/cucumber/
Actual:
Found 0 matching C:/agent/_work/13/s/target/results/cucumber pattern
##[warning]Error: Not found outputPath: C:\agent\_work\13\s\target\results\cucumber
Finishing: Publish Cucumber Report
Expected:
Found cucumber JSON file.
In your case, the problem is in the path provided for the cucumber report.
It should be a path to the folder with cucumber reports in json format, but not to some particular json.
So, the correct snippet in yaml would be:
- task: PublishCucumberReport#1
displayName: 'Publish Cucumber Report'
inputs:
jsonDir: target/results/cucumber/
outputPath: target/results/cucumber/
In above yaml pipeline, you didnot have the step to run your cucumber test to generate the cucumber JSON file.
If the cucumber JSON file is already existing in your repo. Then the error from PublishCucumberReport task is because the directory target/results/cucumber doesnot exist in your repo.
Then You need to check where the cucumber JSON file is located in your repo and specify the correct path for PublishCucumberReport task.
If there is no cucumber JSON fileexisting in your repo. You should add steps in the yaml pipeline to run your tests.
If you have your test scripts configured in the package.json file, like below(report folder must exist in the repo):
You can just run the npm test to execute your tests and generate the json report in the report folder. See below;
steps:
- task: NodeTool#0
displayName: 'Install Node 12.x'
inputs:
versionSpec: 12.x
- script: |
npm install
npm test
displayName: 'Run tests'
- task: PublishCucumberReport#1
inputs:
jsonDir: report
outputPath: report
If there is no test script defined in your package.json file. You can run the cucumber-js command in the yaml pipeline to generate the json file. See below:
- script: |
#npm install cucumber
npm install
./node_modules/.bin/cucumber-js features -f json:report/cucumber_report.json
displayName: 'Run tests'

Azure DevOps First Time Web App Deployment Always Gives Error

I am facing this odd situation. I have the following setup.
React JS app created using create-react-app
Code on GitHub
Azure Web App - Create from the Azure Portal.
Azure DevOps + Pipelines + YAML
I have been doing a bunch of deployments this week and I noticed this situation. Every 1st deployment, always ends up with the following situation inside the wwwroot folder.
But, every subsequent deployment, works just fine.
To reproduce
Create a new app with create-react-app
Put it on GitHub
Link GitHub repository with Azure DevOps Pipelines
Target the Pipeline to deploy to a Azure Web App, which has been created brand new.
The initial deployment, will always fail, with the above image. And, please, remember, Subsequent deployments dont fail.
So, the question is, is there something I can do to prevent this?
I am planning to automate resource creation on Azure, along with automating deployment.
Update 1
as per the comment, including my YAML here. I use this in all of my react JS deployments, which show the same issue as above.
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run build
displayName: 'npm install and build'
- task: CopyFiles#2
inputs:
Contents: 'build/**' # Pull the build directory (React)
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory) # dist or build files
ArtifactName: 'www' # output artifact named www
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.ArtifactStagingDirectory)/build/'
includeRootFolder: false
- task: AzureWebApp#1
inputs:
azureSubscription: 'ReactJSRecipeAppConnection'
appName: 'ReactJSRecipeAppSep232020'
package: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
FAILED TO INITIALIZE RUN FROM PACKAGE usually means the zip file is corrupted or not deflateable. Please check whether the WEBSITE_RUN_FROM_PACKAGE flag under App Service -> Application settings was set.
In addition, try to use task Azure App Service deploy v4, instead of task Azure Web App v1.
Taking off from the discussion about with #Cece Dong, here is what eventually worked for me.
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run build
displayName: 'npm install and build'
- task: CopyFiles#2
inputs:
Contents: 'build/**' # Pull the build directory (React)
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory) # dist or build files
ArtifactName: 'www' # output artifact named www
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.ArtifactStagingDirectory)/build/'
includeRootFolder: false
- task: AzureRMWebAppDeployment#4
inputs:
appType: webApp
azureSubscription: 'RandomStuffReactJSConnection'
WebAppName: 'randomstuffreactjsappsept24'
package: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
The above, I can confirm, no longer requires a second trigger for a proper deployment.
specifically, the usage of 'AzureRMWebAppDeployment#4' is what solved the original issue.

What path should I point the AzureWebApp task to?

I'm trying to deploy a react web app. Here is my current yaml file. I've been following this tutorial.
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
variables:
azureSubscription: <myServiceConnection>
appName: <myAppName>
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
#so this calls the build object in package.json. Theoretically creates a /build/ directory
- script: |
npm install
npm run build
displayName: 'npm install and build'
#this should copy contents of build into artifactstagingdirectory.
- task: CopyFiles#2
inputs:
Contents: 'build/**' # Pull the build directory (React)
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory) # dist or build files
ArtifactName: 'www' # output artifact named www
- task: AzureWebApp#1
inputs:
azureSubscription: <myServiceConnection>
appName: <myAppName>
appType: webAppLinux
package: $(Build.ArtifactStagingDirectory)/**/www/build
customWebConfig: '-handler iisnode -NodeStartFile server.js -appType node'
Basically, the issue is that I dont understand where the PublishBuildArtifact task publishes the files, and so I don't know where to point the package of AzureWebApp to.
So I turned on system.debug and I got the following information
npm install build uses the following directory: /home/vsts/work/1/s
copy files copies the build folder over. So it goes /home/vsts/work/1/s/build to /home/vsts/work/1/a/build. So this means artifactstagingdirectory is /home/vsts/work/1/a
Publish Build artifacts takes the artifactstagingdirectory and publishes it to some folder www/build. The following output is provided: Upload '/home/vsts/work/1/a' to file container: '#/8995596/www'
The current AzureWebApp task is looking in /home/vsts/work/1/a/**/www/build
So I think it should be the right directory. However, it is not providing the whole path in step 3. So it looks like I'm wrong. Where is this www folder being created?
Also since it doesn't seem like building/publishing the react app will create a zip file, do I just point it to the build folder or the www folder?
try
**/wwwroot/
instead of **/www/
Check following blog from Microsoft with deployment details outlined.
https://learn.microsoft.com/en-us/azure/app-service/faq-deployment

How to link build result of "plain React"/node app with release pipeline

To try to make it short, I have a React application that's build on Azure DevOps Build Pipeline like so
trigger:
- Release
queue:
name: Hosted
demands: npm
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm ci
npm run build
displayName: 'Do npm ci and build'
This does what the build would do locally too. Locally the results of build go to build directory (unzipped).
Now when I try to create release pipeline there isn't an artifact to be found using Azure Deploy Web Service task.
If I try to add PublishPipelineArtifact#0 to the build pipeline to create a publication, the YAML editor just tells string does not match pattern... and doesn't let save the definition.
I suppose I should zip the contents of the generated build directory, but what would be the right way? Also, Is using the Azure App Service Deploy task the right way to deploy Azure WebApp? It works for ASP.NET Core apps so it finds the code drop artefact (zipped) and deploys it.
<edit: Adding
- task: PublishPipelineArtifact#0
inputs:
artifactName: 'drop'
targetPath: '$(Build.ArtifactStagingDirectory)/build'
Can actually be saved and the build run. Though it errors with
2019-01-25T22:42:27.6896518Z ##[section]Starting:
PublishPipelineArtifact 2019-01-25T22:42:27.6898909Z
============================================================================== 2019-01-25T22:42:27.6898962Z Task : Publish Pipeline Artifact
2019-01-25T22:42:27.6899006Z Description : Publish Pipeline Artifact
2019-01-25T22:42:27.6899034Z Version : 0.139.0
2019-01-25T22:42:27.6899062Z Author : Microsoft Corporation
2019-01-25T22:42:27.6899090Z Help : Publish a local directory
or file as a named artifact for the current pipeline.
2019-01-25T22:42:27.6899137Z
============================================================================== 2019-01-25T22:42:28.0499917Z ##[error]Path not exist: D:\a\1\a\build
2019-01-25T22:42:28.0708878Z ##[section]Finishing:
PublishPipelineArtifact
< edit 2: It appears removing the /build does the trick. Feels a tad odd since this is what's produced locally. It doesn't produce a zip file which is expected by the release job, namely Azure App Service Deploy.
Have to examine this problem later today (two o'clock at night here).
There was another problem: The script ran only npm ci and didn't run the building part. Separating it to two different steps made the difference. It appears PublishPipelineArtifact#0 in this case isn't an ideal option if I'd like to have the results as zipped to the staging area.
A currently working solution seem to be
resources:
- repo: self
trigger:
- Release
queue:
name: Hosted
demands: npm
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm ci
displayName: 'npm ci'
- script: |
npm run build
displayName: 'npm run build'
- task: ArchiveFiles#2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(System.DefaultWorkingDirectory)/build'
includeRootFolder: false
archiveType: 'zip'
archiveFile: '$(Build.ArtifactStagingDirectory)/artifact.zip'
replaceExistingArchive: true
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact'
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
Maybe this can be simplified, but works for now and is flexible, it feels.

Resources