I am trying to run a series of Postman test in my Azure build pipeline but keep getting errors that Newman is not installed, I have checked by going to the exact location and running the Newman commands without any issue. My screenshots show I have implemented them and the errors.
I see you have used commands as call newman run.
Instead use:
newman run collection.json -e environment_file.json --reporters cli,junit,htmlextra --reporter-junit-export junitReport.xml
It works for me from Azure Pipelines.
This is what mine looks like in YAML - if you click on 'View YAML' on the top right you can see the difference. I think you have this 'call' word that shouldn't be there.
- task: CmdLine#2
displayName: Run newman tests
inputs:
script: 'newman run "$(System.DefaultWorkingDirectory)/${{ parameters.e2eCollectionPath }}" -e "$(System.DefaultWorkingDirectory)/${{ parameters.e2eEnvironmentPath }}" --reporters cli,junit --reporter-junit-export $(System.DefaultWorkingDirectory)/report.xml'
Related
I am currently migrating some build components to Azure Pipelines and am attempting to set some environment variables for all Golang related processes. I wish to execute the following command within the pipeline:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build [...]
When utilizing the provided Golang integrations, it is easy to add arguments for Go related processes, but setting an environment variable for all (or for every individual) Go process does not seem possible. Neither GoTool or the default Go task seem to support it, and performing a script task with a shell execution in it do not seem to be supported either.
I have also tried adding an environment variable to the entire pipelines process that defines the desired flags, but these appear to be ignored by the Go task provided by Azure Pipelines itself.
Would there be a way to do add these flags to each (or a single) go process, such as how I do it in the following code block (in which the flags input line was made-up by me)?
- task: Go#0
inputs:
flags: 'CGO_ENABLED=0 GOOS=linux GOARCH=amd64'
command: 'build'
arguments: '[...]'
workingDirectory: '$(System.DefaultWorkingDirectory)'
displayName: 'Build the application'
Based on the information I was able to find and many hours of debugging, I ended up using a workaround in which I ran the golang commands in a CmdLine#2 task, instead. Due to how GoTool#0 sets up the pipeline and environment, this is possible.
Thus, the code snippet below worked for my purposes.
steps:
- task: GoTool#0
inputs:
version: '1.19.0'
- task: CmdLine#2
inputs:
script: 'CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build'
workingDirectory: '$(System.DefaultWorkingDirectory)'
Azure documentation (https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/build/docker?view=azure-devops) does not specify how to run a docker container in Azure pipeline.
We can use the Docker#2 task to build / push docker images but it does not have a command to run a container. By looking at source code of older versions of Docker task I can see there has been a run command, but those are now deprecated and there is no documentation to be found.
I also followed the doc: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops
With following yaml I was able to pull a docker image which was previously pushed to ACR.
(my-acr is a service connection I added via project settings)
pool:
vmImage: 'ubuntu-16.04'
container:
image: somerepo/rnd-hello:latest
endpoint: my-acr
steps:
- script: printenv
But I cannot get the container to run.
Apparently the configuration mentioned in the question will pull the image and run the step (in this case printenv command in the script) inside the container. A temporary working directory will be mounted automatically and it will run inside that dir.
However this will not run the container itself. (CMD command defined in the Dockerfile will not be executed)
In order to run the container itself we have to login to docker registry with Docker#2 inbuilt task and then manually execute the docker run as a script. Here is an example,
trigger: none
jobs:
- job: RunTest
workspace:
clean: all
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Docker#2
displayName: Login to ACR
inputs:
command: login
containerRegistry: my-acr
- script: |
docker run my-registry.azurecr.io/somerepo/rnd-hello:latest
If you want, you can simply use a shell command to execute docker run and simply rely on that for all the further steps in your pipeline. You don't need to use Docker tasks in Pipelines to be able to communicate with the daemon.
Another solution would be using Azure Container Registry for running a container, but that seems like the last resort in case something went wrong with Pipelines.
I am using Cypress.io (Version 5.1.0) for testing my project.
My project is in azure DevOps. Now i want to include my cypress tests in Azure DevOps so my tests will run automatically.
I set up the JUnit reporter on my Cypress project:
into my “package.json” file i added
"cypress-multi-reporters": "^1.2.4",
"mocha-junit-reporter": "^1.23.3"
then run
npm install
than added
"scripts": {
"scripts": "cypress run",
"test": "npm run scripts"
}
Into my “cypress.json” file i added
"reporter": "mocha-junit-reporter",
"reporterOptions": {
"mochaFile": "cypress/reports/junit/test-results.[hash].xml",
"testsuitesTitle": false
}
After this I created a new Pipeline using Azure Repos in Azure DevOps.
For Pipeline Configuration i selected Node.js.
Now I have a YAML file. Here i removed npm build from the first script.
Now I picked npm from the assisstant. On the npm configurations, I selected custom and write the command run test . Now I Select the result format as “JUnit” and set Test results files to “*.xml”
At last I selected the option "Merge test results".
Now I saved and run the pipeline.
This is what my Job does:
Pool: Azure Pipelines
Image: ubuntu-latest
Agent: Hosted Agent
Started: Yesterday at 17:31
Expanded: Object
Result: True
Evaluating: not(containsValue(job['steps']['*']['task']['id'], '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Expanded: not(containsValue(Object, '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Result: True
Evaluating: resources['repositories']['self']['checkoutOptions']
Result: Object
Finished evaluating template 'system-pre-steps.yml'
********************************************************************************
Template and static variable resolution complete. Final runtime YAML document:
steps:
- task: 6d15af64-176c-496d-b583-fd2ae21d4df4#1
inputs:
repository: self
MaxConcurrency: 0
What is wrong with my automation? How can I fix this?
Update:
Thats my yml file:
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
displayName: 'npm install'
- task: Npm#1
inputs:
command: 'custom'
customCommand: 'run test'
continueOnError: true
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/cypress/reports/junit'
mergeTestResults: true
testRunTitle: 'Publish Test Results'
I got an email with this Details
Job 1 error(s), 1 warning(s) Error: Npm failed with return code: 254
The issue may be due to the agent rather than your code and scripts.
You can try the following solutions:
Change your agent image. As you are currently using the ubuntu-latest, it is recommanded to try the ubuntu-20.04 or ubuntu-16.04.
Use a self-hosted agent. If you don't have a self-hosted agent, click Self-hosted Linux agent for detailed steps.
Change the orgnization. Choose another organization that can run the build correctly, and just in case, it is better to create a new organization. Then create a new project and try your tests.
As stated already, the problem most likely lies with the Azure environment. Cypress has a dependency on a browser (electron, chrome) in order to execute. For example, if you are using docker, they provide an official image called cypress/browsers:node14.7.0-chrome84 that has everything you need out of the box. The Dockerfile also has useful info on the environment needed. Make sure to provide a headless configuration as well, something like:
cypress run --headless --browser chrome --spec yourSpecHere.js
I'm running into a weird issue when running Newman on Azure DevOps Pipeline. Here's a summary of what's happening:
Postman tests run fine locally
Pipeline tests fail only on the first test
Post
Test A
POST XXXXX [500 Internal Server Error, 442B, 8.6s]
1⠄ JSONError in test-script
Test A Copy
POST XXX [200 OK, 692B, 8.9s]
√ Is Successful
√ Status Code
√ Status Message
---
# failure detail
1. JSONError
No data, empty input at 1:1
^
at test-script
It doesn't seem to matter what the exact test is, it always fails if it's the first. As a way to demonstrate this I've copied the test that was failing so that now I had
Test A
Test A Copy
Test B
Test ...
And suddenly Test A Copy works. So it's not the contents of the test but rather the first test to be tested. All of these tests are POST's
Test A Contents:
var jsonData = pm.response.json();
pm.test("Is Successful", function() {
pm.expect(jsonData.IsSuccessful).to.be.true;
})
pm.test("Status Code", function() {
pm.response.to.have.status(200);
})
pm.test("Status Message", function() {
pm.expect(jsonData.StatusMessage).eql("Document insert successful.");
})
Nothing too fancy, so why would this fail on the first run (TEST A) but not the second (TEST A Copy). It doesn't matter which test it is, if I were to run TEST B first this would be the one to fail.
It almost looks like the first request is what's waking up the server and then everything is okay.
I run the Azure Devops Rest API in Postman and use the export json file to run the Postman test in Pipeline
Here are my steps to run newman in azure pipeline, you can refer to them.
Step1: Export the Collection in PostMan.
Step2: Upload the Json file(e.g. APITEST.postman_collection.json) to Azure Repo.
Step3: Create a pipeline and add the install Newman step, run Postman test step.
Example:
steps:
- script: |
npm install -g newman#5.1.2
workingDirectory: '$(System.DefaultWorkingDirectory)'
displayName: 'Command Line Script'
- script: 'newman run TEST.postman_collection.json --reporters cli,junit --reporter-junit-export Results\junitReport.xml '
workingDirectory: '$(build.sourcesdirectory)'
displayName: 'Command Line Script
or run with the Newman the cli Companion for Postman task(This is an extension task).
steps:
- script: |
npm install -g newman#4.6.1
workingDirectory: '$(System.DefaultWorkingDirectory)'
displayName: 'Command Line Script'
- task: NewmanPostman#4
displayName: 'Newman - Postman'
inputs:
collectionFileSource: 'TEST.postman_collection.json'
environmentSourceType: none
ignoreRedirect: false
bail: false
sslInsecure: false
htmlExtraDarkTheme: false
htmlExtraLogs: false
htmlExtraTestPaging: false
Step4: Run the pipeline and it can display the same api results as in postman.
I need to run tests in my gitlab CI pipeline. This is how my YAML document looks like:
before_script:
- docker info
build:
script:
- docker build -t $CI_REGISTRY_IMAGE:$VERSION .
- docker push $CI_REGISTRY_IMAGE:$VERSION
I'm using a shell runner. And now I want to check for correct eslint, which would look like:
eslint .
In a second step I want to do some unit testing using mocha
meteor test --driver-package practicalmeteor:mocha
How can I do this in the pipeline using the already build container/image? How should I implement this into the YAML file?
If I understand correctly, you want to run eslint in your newly built container. You could do it by adding the following to your .gitlab-ci.yml
lint:
script:
# This step is probably optional
- docker pull $CI_REGISTRY_IMAGE:$VERSION
# This will run eslint inside the container
- docker run -it $CI_REGISTRY_IMAGE:$VERSION eslint .
If there are any linting errors, the exit code should be non-zero which will make the job fail (which is probably what you want).