I am looking to fetch the subfolders and files inside jfrog artifactory repo and for that I am running the below script which I am running in Groovy
def test = sh(script: "curl -u uname:password -X POST -k https://artifactory.xxxx.com/artifactory/api/search/aql -d 'items.find({\"type\" : \"file\",\"\$or\":[{\"repo\" : {\"\$match\" : \"war*\"}, \"repo\" : {\"\$match\" : \"web*\"} }]}).include(\"name\",\"repo\",\"path\",\"size\").sort({\"\$desc\": [\"size\"]}).limit(10)'", returnStdout: true).trim()
echo "The list is ${test}"
But its not returning any value.
Any solution would be helpful.
Thanks
You can use api/storage get the children of a artifact path.
For example, your Artifactory has repository: maven-prerelease-local for maven, you can open
https://artifactory.xxxx.com/maven-prerelease-local in browser, it will list file and folders under it.
By adding api/storage in URL, it will return a JSON response.
def test = sh(script: """
curl -u uname:password -X GET -k \
"https://artifactory.xxxx.com/api/storage/maven-prerelease-local/com/xxx/xxx/"
""", returnStdout: true).trim()
echo "The list is ${test}"
To get detailed information about the existing subfolders under a specific directory/repository, you can use the following format of execution.
$ jfrog rt search --spec=test.aql
[Info] Searching artifacts...
[Info] Found 1 artifact.
[
{
"path": "delta-generic-local/alpha/beta",
"type": "folder",
"created": "2022-08-04T13:53:36.173Z",
"modified": "2022-08-04T13:53:36.173Z"
}
]
& the spec file includes the following content.
$ cat test.aql
{
"files":
[
{
"aql":
{
"items.find" :
{
"type":"folder",
"repo":{"$eq":"delta-generic-local"},
"path":{"$eq":"alpha"}
}
}
}
]
}
I am guessing you are in escape special character hell. Put your query in a *.aql file and then point to it. See below.
// Create the aql file and write the query to it
writeFile file: 'sizeQuery.aql', text: 'items.find({"type":"file"}).sort({"$desc":["size"]}).limit(10)'
// Pass the aql file to your curl command
sh 'curl -u uname:password -H "Content-Type: text/plain" -X POST -d #sizeQuery.aql "https://artifactory.xxxx.com/artifactory/api/search/aql"'
Related
I am rewriting my pipline in node, I need to understand how to perform a step with a gait in node now an error is coming from stage('Deploy')
node {
checkout scm
def customImage = docker.build("python-web-tests:${env.BUILD_ID}")
customImage.inside {
sh "python ${env.CMD_PARAMS}"
}
stage('Deploy') {
post {
always {
allure([
includeProperties: false,
jdk: '',
properties: [],
reportBuildPolicy: 'ALWAYS',
results: [[path: 'report']]
])
cleanWs()
}
}
}
and this is the old pipeline
pipeline {
agent {label "slave_first"}
stages {
stage("Создание контейнера image") {
steps {
catchError {
script {
docker.build("python-web-tests:${env.BUILD_ID}", "-f Dockerfile .")
}
}
}
}
stage("Running and debugging the test") {
steps {
sh 'ls'
sh 'docker run --rm -e REGION=${REGION} -e DATA=${DATA} -e BUILD_DESCRIPTION=${BUILD_URL} -v ${WORKSPACE}:/tmp python-web-tests:${BUILD_ID} /bin/bash -c "python ${CMD_PARAMS} || exit_code=$?; chmod -R 777 /tmp; exit $exit_code"'
}
}
}
post {
always {
allure([
includeProperties: false,
jdk: '',
properties: [],
reportBuildPolicy: 'ALWAYS',
results: [[path: 'report']]
])
cleanWs()
}
}
}
I tried to transfer the method of creating an allure report, but nothing worked, I use the version above, almost everything turned out, you can still add environment variables to the build, for example, those that are specified -e DATA=${DATA} how do I add it
I don't recommend to switch from declarative to scriptive pipeline.
You are losing possibility to use multiple tooling connected with declarative approach like syntax checkers.
If you still want to use scriptive approach try this:
node('slave_first') {
stage('Build') {
checkout scm
def customImage = docker.build("python-web-tests:${env.BUILD_ID}")
customImage.inside {
sh "python ${env.CMD_PARAMS}"
}
}
stage('Deploy') {
allure([
includeProperties: false,
jdk: '',
properties: [],
reportBuildPolicy: 'ALWAYS',
results: [[path: 'report']]])
cleanWs()
}
}
There is no post and always directive in scriptive pipelines. It's on your head to catch all exceptions and set status of the job. I guess you were using this page: https://www.jenkins.io/doc/book/pipeline/syntax/, but it's a mistake.
This page only refers to declarative approach and in few cases you have hidden scriptive code as examples.
Also i don't know if you have default agent label set in your Jenkins config, but by looking at your declarative one I think you missed 'slave_first' arg in node object.
those that are specified -e DATA=${DATA} how do I add it
That's a docker question not a Jenkins. If you want to launch docker image and then also have access to some reports located in this container you should mount workspace/file where those output files landed. You should also pass location of those files to allure.
I suggest you to try this:
mount some subfolder in workspace to docker container
cat test report file if it's visible
add allure report with passing this file location to allure step
Trying to make a curl request to gitlab.com api for linting .gitlab-ci.yaml file but receiving bad request response: {"status":400,"error":"Bad Request"}
#!/usr/bin/env bash
PAYLOAD=$( cat << JSON
{ "content":
$(<$PWD/../.gitlab-ci.yml)
JSON
)
echo "Payload is $PAYLOAD"
curl --include --show-error --request POST --header "Content-Type: application/json" --header "Accept: application/json" "https://gitlab.com/api/v4/ci/lint" --data-binary "$PAYLOAD"
Has anyone managed to successfully lint a .gitlab-ci.yml via a bash script? Also tried wrapping the content payload in braces and receive same response.
Update
I think what is happening is that the GitLab CI endpoint expects the contents of the .gitlab-ci yaml file to be converted to json for the POST request. See here
Modifed the script to use ruby to convert yaml to json before sending and this works for simple .gitlab-ci.yml. However when using the yaml file for my project it gives an error: {"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 221"]}% When I use the gitlab web page for linting the file is valid.
{"content": "{ \"stages\": [ \"build\", \"test\", \"pages\", \"release\" ], \"variables\": { \"DOCKER_DRIVER\": \"overlay2\" }, \"services\": [ \"docker:19.03.11-dind\" ], \"build:plugin\": { \"image\": \"docker:19.03.11\", \"stage\": \"build\", \"before_script\": [ \"echo \"$CI_JOB_TOKEN\" | docker login -u gitlab-ci-token --password-stdin \"$CI_REGISTRY\"\" ].....
Column 221 is \"image\": \"docker:19.03.11\" in the above json extract, specifically at the closing escaped quote. Think it is a problem with incorrectly escaped quotes??
#!/usr/bin/env bash
json=$(ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))' < .gitlab-ci.yml)
# escape quotes
json_content=$(echo $json | perl -pe 's/(?<!\\)"/\\"/g')
# Add object contect for GitLab linter
json_content='{"content": "'${json_content}'"}'
echo "${json_content}"
curl --include --show-error --request POST \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
"https://gitlab.com/api/v4/ci/lint" \
--data-binary "$json_content"
Second Update
Using the above bash script this yaml file:
stages:
- test
test:
stage: test
script:
- echo "test"
gets converted to this json:
{"content": "{ \"stages\": [ \"test\" ], \"test\": { \"stage\": \"test\", \"script\": [ \"echo \"test\"\" ] } }"}
When this is sent to the api receive the following json error response:
{"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 62"]}%
Got it working finally using the following script:
#!/usr/bin/env bash
json=$(ruby -ryaml -rjson -e 'puts(YAML.load(ARGF.read).to_json)' custom_hooks/valid.yml)
# escape quotes
json_content=$(echo $json | python -c 'import json,sys; print(json.dumps(sys.stdin.read()))')
echo $json_content
# Add object contect for GitLab linter
json_content="{\"content\": ${json_content}}"
# Output escaped content to file
echo $json_content > custom_hooks/input.json
echo "Escaped json content written to file input.json"
curl --include --show-error --request POST \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
"https://gitlab.com/api/v4/ci/lint" \
--data-binary "$json_content"
N.B will be tweaking script to read file from system args rather than the fixed file location custom_hooks/valid.yml. Also the JSON response needs parsing using jq or python / ruby command shell. Including this script on the offchance that it will help others.
The problem was that initially I was sending YAML contents of the file directly to the api:
{ "content": { <contents of .gitlab-yml> } }
It looks as though GitLab accepts YAML converted to an escaped JSON string in their API. So used ruby to convert the yaml to JSON and then used python to escape the resulting JSON produced by ruby. Finally was able to use curl to send the escaped JSON string to the GitLab API for validating.....
Not sure if Ruby has something equivalent to python's json.dumps .... but this solution allows me to validate gitlab-ci....Next stage hookup to git pre-commit hooks / server side pre-receive (if possible!) to prevent invalid .gitlab-ci.yml files breaking CI pipeline.
Newbie to ruby...since posting original answer have had a go at creating a ruby script that can be used from pre-commit hooks etc. Now only require bash and ruby:
#!/usr/bin/env ruby
require 'json'
require 'net/http'
require 'optparse'
require 'yaml'
=begin
POST to GitLab api for linting ci yaml
Params:
+url+ :: Api url
+yaml+ :: Yaml payload for linting
Returns:
Json validation result from API for HTTP response Success
Aborts with HTTP Message for all other status codes
=end
def call_api(url, yaml)
uri = URI.parse(url)
req = Net::HTTP::Post.new(uri)
req.content_type='application/json'
req['Accept']='application/json'
req.body = JSON.dump({"content" => yaml.to_json})
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
https.verify_mode = OpenSSL::SSL::VERIFY_PEER
response = https.request(req)
case response
when Net::HTTPSuccess
puts "request successful"
return JSON.parse response.body
when Net::HTTPUnauthorized
abort("#{response.message}: invalid token in api request?")
when Net::HTTPServerError
abort('error' => "#{response.message}: server error, try again later?")
when Net::HTTPBadRequest
puts "Bad request..." + request.body
abort("#{response.message}: bad api request?")
when Net::HTTPNotFound
abort("#{response.message}: api request not found?")
else
puts "Failed validation\nJSON payload :: #{request.body}\nHTTP Response: #{response.message}"
abort("#{response.message}: failed api request?")
end
end
=begin
Display exit report and raise the appropriate system exit code
Params:
+status+ :: Validation status string. Legal values are valid or invalid
+errors+ :: String array storing errors if yaml was reported as invalid
Returns:
Exits with 0 when successful
Exits with 1 on validation errors or fails to parse legal status value
=end
def exit_report(status, errors)
case status
when "valid"
puts ".gitlab-ci.yml is valid"
exit(0)
when "invalid"
abort(".gitlab-ci.yml is invalid with errors:\n\n" + errors.join("\n"))
else
abort("A problem was encountered parsing status : " + status)
end
end
=begin
Load yaml file from path and return contents
Params:
+path+ :: Absolute or relative path to .gitlab-ci.yml file
=end
def load_yaml(path)
begin
YAML.load_file(path)
rescue Errno::ENOENT
abort("Failed to load .gitlab-ci.yml")
end
end
=begin
Parse command line options
Returns:
Hash containing keys: {:yaml_file,:url}
=end
def read_args()
options = {}
OptionParser.new do |opt|
opt.on('-f', '--yaml YAML-PATH', 'Path to .gitlab-ci.yml') { |o| options[:yaml_file] = o }
opt.on('-l', '--url GitLab url', 'GitLab API url') { |o| options[:url] = o }
end.parse!
options
end
=begin
Load yaml to send to GitLab API for linting
Display report of linting retrieved from api
Returns:
Exits with 0 upon success and 1 when errors encountered
=end
def main()
# try and parse the arguments
options = read_args()
unless !options.has_key?(:yaml_file) || !options.has_key?(:url)
# try and load the yaml from path
puts "Loading file #{options[:yaml_file]}"
yaml = load_yaml(options[:yaml_file])
# make lint request to api
puts "Making POST request to #{options[:url]}"
response_data=call_api(options[:url], yaml)
# display exit report and raise appropriate exit code
unless !response_data.has_key?("status") || !response_data.has_key?("errors")
exit_report response_data["status"], response_data["errors"]
else
puts "Something went wrong parsing the json response " + response_data
end
else
abort("Missing required arguments yaml_file and url, use -h for usage")
end
end
# start
main
Is it possible to use a here-document from within a multiline command in .gitlab-ci.yml
I am having difficulty using a here-document within a command that expands multiple lines in a .gitlab-ci.yml job. For example I have created a short test job below, that uses a here-document to assign a multiline string to an environment variable PAYLOAD that is then used in a curl request to POST data to a url:
example:
image: node:12-stretch-slim
stage: mystage
script:
- >
PAYLOAD=$(cat << 'JSON'
'{
"branch": "master",
"commit_message": "some commit message",
"actions": [
{
"action": "create",
"file_path": "foo/bar",
"content": "some content"
}
]
}'
JSON
)
- >
curl -X POST https://requestbin.io/1f84by61
--header 'Content-Type: application/json; charset=utf-8'
--data-binary "$PAYLOAD"
when: manual
only:
- /^release-.*$/
The following script fails with the following message on gitlab.com CI server:
$ PAYLOAD=$(cat << 'JSON' '{ # collapsed multi-line command
[551](<private url>) /bin/bash: line 140: warning: here-document at line 140 delimited by end-of-file (wanted `JSON')
[552](<private url>) /bin/bash: line 139: warning: here-document at line 139 delimited by end-of-file (wanted `JSON')
[553](<private url>) cat: '{'$'\n'' "branch": "master",'$'\n'' "commit_message": "some commit message",'$'\n'' "actions": ['$'\n'' {'$'\n'' "action": "create",'$'\n'' "file_path": "foo/bar",'$'\n'' "content": "some content"'$'\n'' }'$'\n'' ]'$'\n''}': No such file or directory
[554](<private url>) cat: JSON: No such file or directory
Can anyone help?
Solved it after reading this
Using |- preserves newlines within the command and does not append a newline at the end of the command string. Beforehand I was using > which replaces newlines in the command string with spaces.
release:
image: node:12-stretch-slim
stage: release
before_script:
- apt-get update && apt-get install -y curl git jq
script:
- |-
PAYLOAD=$(cat << JSON
{
"branch": "master",
"commit_message": "some commit message",
"actions": [
{
"action": "create",
"file_path": "foo/bar",
"content": "some content"
}
]
}
JSON
)
- >
curl -X POST https://requestbin.io/1f84by61
--header 'Content-Type: application/json; charset=utf-8'
--data-binary "$PAYLOAD"
when: manual
only:
- /^release-.*$/
Hoping someone can help. I am experiencing difficulty making a curl POST request with JSON data from a gitlab CI job.
The curl request works fine in a local terminal session, (N.B I did not use double quotes in terminal session). If I do not escape double quotes in the gitlab CI yaml I get the error curl: (3) [globbing] nested brace in column 112
If I escape the double quotes in the GitLab CI job, as shown below I get the error:
curl: (3) [globbing] unmatched brace in column 1
In all cases I get the error /bin/bash: line 134: warning: here-document at line 134 delimited by end-of-file (wanted `EOF')
Is it possible to POST JSON data using here-documents from a GitLab CI job?
.gitlab-ci.yml job extract
release:
image: node:12-stretch-slim
stage: release
before_script:
- apt-get update && apt-get install -y curl git jq
script:
- version=$(git describe --tags | cut -d'-' -f 1 | sed 's/^v*//')
- echo "generating release for version ${version}"
- npm pack
# - >
# url=$(curl
# --header "Private-Token: $API_TOKEN"
# -F "file=#${CI_PROJECT_DIR}/objectdetection-plugin-${version}.tgz" "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/uploads"
# |
# jq '.url')
- url="http://www.example.com"
- echo "Retrieved url from file upload as ${url}"
- echo "The full url would be ${CI_PROJECT_URL}/${url}"
- >
curl -X POST https://requestbin.io/1f84by61
--header 'Content-Type: application/json; charset=utf-8'
--data-binary #- << EOF
{
\"name\": \"Release v${version}\",
\"tag_name\": \"v${version}\",
\"ref\": \"v${version}\",
\"description\": \"Test\",
\"assets\": {
\"links\": [
{
\"name\": \"objectdetection-plugin-source\",
\"url\": \"CI_PROJECT_URL/${url}\",
\"filepath\": \"${url}\",
\"link_type\": \"other\"
}
]
}
}
EOF
when: manual
only:
- /^release-.*$/
Solved it after reading this
Using |- preserves newlines within the command and does not append a newline at the end of the command string. Used this principle to save the JSON data to a variable and then referenced the variable in the subsequent curl command.
Below I have included the script:
release:
image: node:12-stretch-slim
stage: release
before_script:
- apt-get update && apt-get install -y curl git jq
script:
- git fetch --prune --unshallow
- version=$(git describe --tags | cut -d'-' -f 1 | sed 's/^v*//')
- npm pack
- >
url=$(curl --silent --show-error
--request POST
--header "Private-Token: $API_TOKEN"
-F "file=#${CI_PROJECT_DIR}/objectdetection-plugin-${version}.tgz" "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/uploads"
|
jq --raw-output --monochrome-output '.url')
- |-
PAYLOAD=$(cat << JSON
{
"name": "Release v$version",
"tag_name": "v$version",
"ref": "v$version",
"description": "$(sed -zE 's/\r\n|\n/\\n/g' < CHANGELOG.md)",
"assets": {
"links": [
{
"name": "objectdetection-plugin-source",
"url": "$CI_PROJECT_URL$url",
"filepath": "$url",
"link_type": "other"
}
]
}
}
JSON
)
- echo "$PAYLOAD"
- >
http_response=$(curl --silent --show-error --write-out "%{http_code}" -o response.txt
--request POST "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/releases"
--header 'Content-Type: application/json'
--header 'Accept: application/json'
--header "Private-Token: ${API_TOKEN}"
--data-binary "${PAYLOAD}")
- |-
if [ $http_response != "201" ]; then
exit 1
else
echo "Server returned:"
cat response.txt
fi
when: manual
allow_failure: false
only:
- /^release-.*$/
I was trying to send a message to my Google Chat, WebHook URL, and the above solution didn't work.
So I used this instead
'curl --silent --show-error --location --request POST ${GCHAT_DEVOPS_WEBHOOK} --header ''Content-Type: application/json; charset=UTF-8'' --data-raw ''{"text":"Hey <users/all>: A new build has been deployed to *''${APP_ENVIRONMENT}''* via ''${CI_JOB_NAME}'' - *''${APP_BRANCH}''*"}'''
I was trying to use child_process.exec to call curl with a long command in order to send some data to an API. Something similar to the following example:
exec('git log --oneline | wc -l', function(error, stdin, stderr) {
if (stdin > 1) {
exec('curl -H "Content-Type: application/json" -X POST -d \'{"value1": "\'"$arg"\'"}\' https://https://maker.ifttt.com/trigger/{event}/with/key/<my-key>', { "env" : {"arg": stdin } });
}
})
So if a git repo includes more than one line in its git log output, then you execute a POST request to some API (here, a simple webhook in ifttt.com), in which you're passing some variable (arg) in the process.
Notice that this is the best attempt, but in general, I was struggling quite a bit to escape single and double quotes. In this particular case, the HTTP request was not sent correctly because the body includes a line break:
POST / HTTP/1.1
Host: <some-host>
User-Agent: curl/7.50.1
Accept: */*
Content-Type: application/json
Content-Length: 16
{"value1": "2
"}
At the end, I had to use an external bash script:
exec('./send_request.sh $arg', { "env": {"arg": stdin } });
but I'm still very curious on how to make it work within the same js file.
If it helps, I'm running node 6.11.0 and curl 7.52.1.
Try:
exec('git log --oneline | wc -l', function(error, stdin, stderr) {
if (stdin > 1) {
exec('curl -H "Content-Type: application/json" -X POST -d \'{"value1": "\'"$arg"\'"}\' https://https://maker.ifttt.com/trigger/{event}/with/key/<my-key>', { "env" : {"arg": stdin.replace(/\n/g, '') } });
}
})
What is happening is that your variable 'stdin' (you should rename it to 'stdout') has a \n at the end of it.