Gitlab YAML API linter gives no output - gitlab

Issue
I am trying to use the Gitlab yaml API linting tool on an enterprise instance of Gitlab. However, I am getting an empty response (not just an empty json object, like absolutely zero output).
Steps to duplicate
I am using a stripped version of the sample .yaml file shown on the gitlab CI/CD tutorial page. The file is shown here:
build-job:
stage: build
script:
- echo "Hello, $GITLAB_USER_LOGIN!"
deploy-prod:
stage: deploy
script:
- echo "This job deploys something from the $CI_COMMIT_BRANCH branch."
I am using the 1 line curl command as shown on the CI Linting API page.
If I use the command as given (replacing only the filename), I get
$ jq --null-input --arg yaml "$(<.gitlab-ci.yml)" '.content=$yaml' \
| curl "https://gitlab.mycompany.com/api/v4/ci/lint?include_merged_yaml=true" \
--header 'Content-Type: application/json' \
--data #-
I get the output {"message":"401 Unauthorized"}, which is to be expected as the API call requires an API key. I generate an API key in my profile and try again:
$ export TOKEN='xxxxxxxxxx'
$ jq --null-input --arg yaml "$(<.gitlab-ci.yml)" '.content=$yaml' \
| curl "https://gitlab.mycompany.com/api/v4/ci/lint?include_merged_yaml=true" \
--header "Content-Type: application/json PRIVATE-TOKEN=${TOKEN}" \
--data #-
When I run this, the output shows nothing. This is confirmed by a pipe to wc -c which outputs 0.
My expected output should be:
{
"status": "valid",
"errors": [],
"warnings": []
}
Questions:
Why is no response a result of me using my valid API key (This is with a newly generated key)?
How can I fix this, and receive the expected output shown above?

Make sure your token as the api scope, as illustrated here.
Without that scope, you would get a 401 Unauthorized, which would not be parsed parsed by jq at all.

Related

windows command prompt curl POST to run Azure DevOps pipeline

I am trying to run an Azure DevOps pipeline from the windows command prompt using curl.
Based on Microsoft documentation ( Runs - Run Pipeline ) I should be able to run a pipeline by posting:
https://dev.azure.com/{organization}/{project}/_apis/pipelines/{pipelineId}/runs?api-version=6.0-preview.1
I am able to GET using the command:
curl -u :<PAT> https://dev.azure.com/{organization}/{project}/_apis/pipelines/{pipelineId}/runs?api-version=6.0-preview.1
However, I can't figure out how to do a POST for DevOps using curl to run the pipeline.
I have tried the following:
curl -s -X POST -L https://dev.azure.com/{organization}/{project}/_apis/pipelines/{pipelineId}/runs?api-version=6.0-preview.1 -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization: Basic <PAT>"
But this returns the error
HTTP Error 411. The request must be chunked or have a content length
I managed to solve my problem which consisted of numerous steps:
Log the API call from the DevOps website to get the correct format of the json-body ( See this question ).
Format the json-body with extra double quotation signs.
Use the {project}-id instead of the ascii-name since it included a special character which the command prompt misinterpret.
Hence the complete curl command was:
curl -X POST -u :<PAT> "https://dev.azure.com/{organization}/{project}/_apis/pipelines/{pipelineId}/runs?api-version=6.0-preview.1" -H "Content-Type: application/json" -d "{""stagesToSkip"": ""[]"", ""resources"": {""repositories"": {""self"": {""refName"": ""refs/heads/master""}}}}"

Using Github api to create a file in postman

I am new to GitHub API and trying to create a file in my repo using API.
I am using postman to check the API first.
PUT Method:
URL:https://api.github.com/repos/KaranS-hexaware/Rapidx_Documentation/contents/text.txt
Body:{
"message":"my commit message",
"content":"bXkgbmV3IGZpbGUgY29udGVudHM="
}
Header
Authorization:Bearer ****************************************************
Content-Type:application/vnd.api+json
The response is:
{
"message": "Not Found",
"documentation_url": "https://docs.github.com/rest/reference/repos#create-or-update-file-contents"
}
Can I get some input on what I am missing here?
The create file contents API example is:
curl \
-X PUT \
-H "Accept: application/vnd.github+json" \
-H "Authorization: token <TOKEN>" \
https://api.github.com/repos/OWNER/REPO/contents/PATH \
-d '{"message":"my commit message","committer":{"name":"Monalisa Octocat","email":"octocat#github.com"},"content":"bXkgbmV3IGZpbGUgY29udGVudHM="}'
Check if those headers (which are slightly different than yours) make a difference and allow your call to succeed.

syntax use variable in curl script json data in gitlab-ci.yml

I am running out of ideas to have this to work in my gitlab-ci.yml.
I have a variable called TARGET, and I want to successfully inject its value inside the json payload of my http POST request in form of curl bash command.
job:
stage: deploy
script:
- curl -k --request POST ${URL} --header "Authorization:Basic ${TOKEN}" --header 'Content-Type:application/json' --data "{"extra_vars":{"target":${TARGET}}}"
variables:
TARGET: host1
the pipeline output keeps complaining with error message:
{"detail":"JSON parse error - Expecting property name enclosed in double quotes: line 1 column 2 (char 1)\nPossible cause: trailing comma."}
I also tried escaping some special characters with \ as below, but not working either:
curl -k --request POST ${URL} --header "Authorization:Basic ${TOKEN}" --header 'Content-Type:application/json' --data "\{"extra_vars":\{"target":${TARGET}\}\}"
your helps are very appreciated.
There are two issues here. You are missing double quotes around the value for target inside the JSON since TARGET is a string value.
You are also wrapping your data argument with double quotes hence you should not use them inside the payload without either escaping them or switching to single quotes.
Keeping that in mind the payload may look something like this:
'{"extra_vars":{"target":"'${TARGET}'"}}'
Here's a quick tip on how to test the output using httpbin.org:
TARGET=host1 && curl -v https://httpbin.org/post -H "Content-Type: application/json" -d '{"extra_vars":{"target":"'${TARGET}'"}}'
Which will respond with the sent payload (which is ofcourse escaped as it is also wrapped inside JSON:
"data": "{\"extra_vars\":{\"target\":\"host1\"}}",

How to get Gitlab runner registration token from command line?

I'm trying to deploy a Gitlab instance and runners ready with Terraform. The script creates both Gitlab and runners without any problem, but I don't know how to register the runners automatically after the creation.
Is there any way to get the registration token from command line? If it's possible I can register just calling external data source using Terraform.
The projects API endpoint response contains the runners_token key. You can use this to automatically fetch the runner tokens for any project.
You can then use that in a few ways. One way would be to have your runner registration script fetch the runner token itself such as with this example:
curl --fail --silent --header "Private-Token: ${GITLAB_API_TOKEN}" "https://$GITLAB_URL/api/v4/projects/${PROJECT}"
Or you could use the Gitlab Terraform provider's gitlab_project data source to fetch this from whatever is running Terraform and then inject it into the thing that runs the registration script such as a templated file:
data "gitlab_project" "example" {
id = 30
}
locals {
runner_config = {
runner_token = data.gitlab_project.example.runners_token
}
}
output "example" {
value = templatefile("${path.module}/register-runners.sh.tpl", local.runner_config)
}
Yes, you can.
The command has to be run on the server hosting your Gitlab instance. The line below will output the current shared runner token.
sudo gitlab-rails runner -e production "puts Gitlab::CurrentSettings.current_application_settings.runners_registration_token"
As others have mentioned, there is not API endpoint that currently allows this (there has been discussion over this for quite some time here. However, I find this solution satisfactory for my needs.
Credits for this answer go to MxNxPx. This script used to work (for me) two days ago:
GITUSER="root"
GITURL="http://127.0.0.1"
GITROOTPWD="mysupersecretgitlabrootuserpassword"
# 1. curl for the login page to get a session cookie and the sources with the auth tokens
body_header=$(curl -k -c gitlab-cookies.txt -i "${GITURL}/users/sign_in" -sS)
# grep the auth token for the user login for
# not sure whether another token on the page will work, too - there are 3 of them
csrf_token=$(echo $body_header | perl -ne 'print "$1\n" if /new_user.*?authenticity_token"[[:blank:]]value="(.+?)"/' | sed -n 1p)
# 2. send login credentials with curl, using cookies and token from previous request
curl -sS -k -b gitlab-cookies.txt -c gitlab-cookies.txt "${GITURL}/users/sign_in" \
--data "user[login]=${GITUSER}&user[password]=${GITROOTPWD}" \
--data-urlencode "authenticity_token=${csrf_token}" -o /dev/null
# 3. send curl GET request to gitlab runners page to get registration token
body_header=$(curl -sS -k -H 'user-agent: curl' -b gitlab-cookies.txt "${GITURL}/admin/runners" -o gitlab-header.txt)
reg_token=$(cat gitlab-header.txt | perl -ne 'print "$1\n" if /code id="registration_token">(.+?)</' | sed -n 1p)
echo $reg_token
However, as of today it stopped working. I noticed the second body_header variable is empty. Upon inspecting the gitlab-header.txt file, I noticed it contained:
You are being redirected.
Whereas I would expect it to be signed in at that point, with a gitlab-header.txt file that contains the respective runner registration token. I expect I am doing something wrong, however, perhaps there has been an update to the gitlab/gitlab-ce:latest package such that a change to the script is required.
Disclaimer, I am involved in creating that code
Here is a horrible but working Python boiler plate code that gets the runner token and exports it to a parent repository: https://github.com/a-t-0/get-gitlab-runner-registration-token.
Independent usage
It requires a few manual steps to set up, and then gets the GitLab runner registration token automatically (from the CLI with:). It requires Conda and Python however, and downloads a browser controller. So it is most likely wiser to look a bit better into the curl commands instead.
Integrated in parent [bash] repository
First install the conda environment, then activate it. After that, you can execute the function below automatically from the CLI (if you put that function in a file at path parent_repo/src/get_gitlab_server_runner_token.sh, assuming you have the credentials etc as specified in the Readme), with:
cd parent_repo
source src/get_gitlab_server_runner_token.sh && get_registration_token_with_python
This bash function gets the token:
get_registration_token_with_python() {
# delete the runner registration token file if it exist
if [ -f "$RUNNER_REGISTRATION_TOKEN_FILEPATH" ] ; then
rm "$RUNNER_REGISTRATION_TOKEN_FILEPATH"
fi
git clone https://github.com/a-t-0/get-gitlab-runner-registration-token.git &&
set +e
cd get-gitlab-runner-registration-token && python -m code.project1.src
cd ..
}
And here is a BATS test that verifies the token is retrieved:
#!./test/libs/bats/bin/bats
load 'libs/bats-support/load'
load 'libs/bats-assert/load'
load 'libs/bats-file/load'
source src/get_gitlab_server_runner_token.sh
source src/hardcoded_variables.txt
#test "Checking if the gitlab runner registration token is obtained correctly." {
get_registration_token_with_python
actual_result=$(cat $RUNNER_REGISTRATION_TOKEN_FILEPATH)
EXPECTED_OUTPUT="somecode"
assert_file_exist $RUNNER_REGISTRATION_TOKEN_FILEPATH
assert_equal ${#actual_result} 20
}

How does urllib.request differ from curl or httpx in behaviour? Getting a 401 in a request to the Google Container Registry

I am currently working on some code to interact with images on the Google Container Registry. I have working code both using plain curl and also httpx. I am trying to build a package without 3rd party dependencies. My curiosity is around a particular endpoint from which I get a successful response in curl and httpx but a 401 Unauthorized using urllib.request.
The bash script that demonstrates what I'm trying to achieve is the following. It retrieves an access token from the registry API, then uses that token to verify that the API indeed runs version 2 and tries to access a particular Docker image configuration. I'm afraid that in order to test this, you will need access to a private GCR image and a digest for one of the tags.
#!/usr/bin/env bash
set -eu
token=$(gcloud auth print-access-token)
image=...
digest=sha256:...
get_token() {
curl -sSL \
-G \
--http1.1 \
-H "Authorization: Bearer ${token}" \
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
--data-urlencode "scope=repository:$1:pull" \
--data-urlencode "service=gcr.io" \
"https://gcr.io/v2/token" | jq -r '.token'
}
echo "---"
echo "Retrieving access token."
access_token=$(get_token ${image})
echo
echo "---"
echo "Testing version 2 capability with access token."
curl -sSL \
--http1.1 \
-o /dev/null \
-w "%{http_code}" \
-H "Authorization: Bearer ${access_token}" \
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
https://gcr.io/v2/
echo
echo "---"
echo "Retrieving image configuration with access token."
curl -vL \
--http1.1 \
-o /dev/null \
-w "%{http_code}" \
-H "Authorization: Bearer ${access_token}" \
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
"https://gcr.io/v2/${image}/blobs/${digest}"
I additionally created two Jupyter notebooks demonstrating my solutions in httpx and bare urllib.request. The httpx one works perfectly while somehow urllib fails on the image configuration request. I'm running out of ideas trying to spot the difference. If you run the notebook yourself, you will see that the called URL contains a token as a query parameter (is this a security issue?). When I open that link I can actually successfully download the data myself. Maybe urllib still passes along the Authorization header with the Bearer token making that last call fail with 401 Unauthorized?
Any insights are greatly appreciated.
I did some investigation and I believe the difference is that the last call to "https://gcr.io/v2/${image}/blobs/${digest}" actually contains a redirect. Inspecting the curl and httpx calls showed me that both do not include the Authorization header in the second, redirected request, whereas in the way that I set up the urllib.request in the notebook, this header is always included. It's a bit odd that this leads to a 401 but now I know how to address it.
Edit: I can now confirm that by building a urllib.request.Request instance and unlike in the linked notebook, add the authorization header with the request's add_unredirected_header method, everything works as expected.

Resources