GitLab API get commit count of a branch - gitlab

Is there a GitLab API to get the commit count of a specific branch?
I can get the commits of the branch using following curl command, but not the commit count.
curl -X GET -H "PRIVATE-TOKEN: <my_private_token>" "http://<my_locally_hosted_web_server>/api/v4/projects/2/repository/commits/?ref_name=master"

Get the number of pages (see also gitlab pagination) and iterate over pages counting json array elements using jq :
TOTAL_PAGES=$(curl -Ss -k --head --header "PRIVATE-TOKEN: <my_private_token>" "http://<my_locally_hosted_web_server>/api/v4/projects/2/repository/commits/?ref_name=master" | grep x-total-pages | cut -d':' -f2 )
for ((i=1;i<=TOTAL_PAGES;i++)); do
SUM=$(($SUM + $(curl -Ss -k --request GET --header "PRIVATE-TOKEN: <my_private_token>" "http://<my_locally_hosted_web_server>/api/v4/projects/2/repository/commits/?ref_name=master&per_page=100&page=$i" | jq -r '. | length')));
done;
echo $SUM

You can simply use the headers informations 'X-Total' :
commits_count=$(curl -Ss -k --head --header "PRIVATE-TOKEN: <my_private_token>" "http://<my_locally_hosted_web_server>/api/v4/projects/:id/merge_requests/:merge_request_iid/commits?per_page=5" | grep -m 1 X-Total | cut -d':' -f2 )

PHP-CURL example using iteration through contributors
function commitStatistics()
{
$projectId = 100;
$projectPrivateToken = "secretToken123";
$url = $this->projectUrl . "/api/v4/projects/$projectId/repository/contributors";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('PRIVATE-TOKEN:' . $projectPrivateToken));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);
$total_counter = 0;
foreach (json_decode($result) as $contributor) {
$total_counter += $contributor->commits;
}
return $total_counter;
}

Related

bash script variable in curl

bash get-token.sh | grep x-subject-token | awk '{print $2}' >> token.txt
token=`cat token.txt`
echo $token this is true
data=`curl -s -H "X-Auth-Token: ${token}" -X GET "https://url"`
echo $token = correct token but $data is invalid.
When I write the token in curl and run it manually I get the correct data.
How can I use this token in my script?

How to use GitHub api to add a multiline comment

I'm finding a method to post a multiline comment through environment variable with curl and GitHub api.
In GitHub actions, I used npm run unit-test-ci-coverage >> $GITHUB_STEP_SUMMARY to redirect the result to $GITHUB_STEP_SUMMARY. The result would be like these
> my-repo#1.0.0 unit-test-ci-coverage
> node --inspect ./node_modules/.bin/jest --coverage --config ./jest.config.js
---------------|---------|----------|---------|---------|---------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
---------------|---------|----------|---------|---------|---------------------
All files | 80 | 80 | 80 | 80 |
Then I used environment variable $RES to store the part that I want to add to the comment.
echo "RES<<EOF" >> $GITHUB_ENV
sed -n 5,7p $GITHUB_STEP_SUMMARY >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
$RES could get the part (File, % Stmts...) successfully. I wanted to create the comment containing the step summary url and the result, so $SUMMARY_URL is the url, and $RES is the result.
However, I used the following commands
curl \
-X POST \
-H "Accept: application/vnd.github.v3+json" \
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
https://api.github.com/repos/$REPO/issues/$PR_NUMBER/comments \
-d '{"body":"## Result\n '$SUMMARY_URL' \n '$RES'"}'
It showed
curl: option ---------------|---------|----------|---------|---------|---------------------: is unknown
It seemed that curl didn't get the correct values through $RES. How to pass it to curl?

Curl function cannot parse proxy coming from a variable in bash

I have a proxy txt file with the format:
102.129.249.120:3128
102.129.249.120:8080
101.4.136.34:8080
103.228.117.244:8080
etc
and I am trying to create a bash script that will do (for example): curl -x "$IP" google.com.
Unfortunately, curl is giving me an unsupported proxy syntax for all the proxies.
Any ideas?
BTW, I really doubt this question has been repeated as I have tried everything else to no avail.
My script:
Number=$(wc -l < ProxyList.txt)
for ((i=1;i<=$Number;++i)) do
ip=$(head -n ${i} ProxyList.txt | tail -n +${i})
curl -p -x "$ip" 'webpage' -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6
done
A small sample of my proxy list:
102.129.249.120:3128
102.129.249.120:8080
101.4.136.34:8080
103.228.117.244:8080
103.253.27.108:80
104.45.188.43:3128
104.250.34.179:80
105.27.238.161:80
104.154.143.77:3128
110.243.20.2:9999
111.68.26.237:8080
106.104.151.142:58198
113.252.95.19:8197
115.231.31.130:80
118.69.50.154:80
118.69.50.154:443
119.81.189.194:80
119.81.189.194:8123
119.81.199.81:8123
119.81.199.83:8123
119.81.199.80:8123
12.139.101.100:80
12.139.101.101:80
119.81.199.85:31288
119.81.199.86:8123
119.81.199.87:8123
12.139.101.102:80
124.156.98.172:443
13.228.91.252:3128
138.197.157.32:3128
138.197.157.32:8080
138.68.240.218:8080
138.68.240.218:3128
138.68.60.8:8080
138.68.60.8:3128
Your input file has carriage return characters at the end of each line.
Each line in your input file ends with \r\n instead of just \n.
You can check with od:
$ head -1 ProxyList.txt | od -c
0000000 1 0 2 . 1 2 9 . 2 4 9 . 1 2 0 :
0000020 3 1 2 8 \r \n
0000026
So in your script, $ip has in fact a value of 102.129.249.120:3128\r.
You can remove the \r characters with tr for example:
while read proxy; do
curl -p -x $proxy $webpage
done < <( tr -d '\r' < ProxyList.txt )
try this:
for ip in $(cat ProxyList.txt)
do
curl -p -x "$ip" 'webpage' -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6
done
but the problem with curl might be, that should set the environment variables http_proxy and https_proxy like this:
export http_proxy=http://1.2.3.4:3128/
export https_proxy=http://1.2.3.4:3128/
As per the curl man page, the -x (or --proxy) switch can be prefixed with the protocol in front of the argument (if omitted, I assume that it defaults to http://):
-x, --proxy [protocol://]host[:port]
A simple bash script with xargs would look like:
#!/bin/bash
webpage=${1:-http://google.com}
cat ProxyList.txt \
| xargs -n1 -I{} curl -p -x http://{} "$webpage" -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6

Bash shell script to find Robots meta tag value

I've found this bash script to check status of URLs from text file and print the destination URL when having redirections :
#!/bin/bash
while read url
do
dt=$(date '+%H:%M:%S');
urlstatus=$(curl -kH 'Cache-Control: no-cache' -o /dev/null --silent --head --write-out '%{http_code} %{redirect_url}' "$url" )
echo "$url $urlstatus $dt" >> urlstatus.txt
done < $1
I'm not that good in bash : I'd like to add - for each url - the value of its Robots meta tag (if is exists)
Actually I'd really suggest a DOM parser (e.g. Nokogiri, hxselect, etc.),
but you can do this for instance (Handles lines starting with <meta and "extracts" the value of the robots' attribute content):
curl -s "$url" | sed -n '/\<meta/s/\<meta[[:space:]][[:space:]]*name="*robots"*[[:space:]][[:space:]]*content="*\([^"]*\)"*\>/\1/p'
This will print the value of the attribute or the empty string if not available.
Do you need a pure Bash solution? Or do you have sed?
You can add a line to extract the meta header for robots from the source code of the page and modify the line with echo to show its value:
#!/bin/bash
while read url
do
dt=$(date '+%H:%M:%S');
urlstatus=$(curl -kH 'Cache-Control: no-cache' -o /dev/null --silent --head --write-out '%{http_code} %{redirect_url}' "$url" )
metarobotsheader=$(curl -kH 'Cache-Control: no-cache' --silent "$url" | grep -P -i "<meta.+robots" )
echo "$url $urlstatus $dt $metarobotsheader" >> urlstatus.txt
done < $1
This example records the original line with the meta header for robots.
If you want to put a mark "-" when the page has no meta header for robots, you can change the metarobotsheader line, and put this one:
metarobotsheader=$(curl -kH 'Cache-Control: no-cache' --silent "$url" | grep -P -i "<meta.+robots" || echo "-")
If you want to extract the exact value of the attribute, you can change that line:
metarobotsheader="$(curl -kH 'Cache-Control: no-cache' --silent "$url" | grep -P -i "<meta.+robots" | perl -e '$line = <STDIN>; if ( $line =~ m#content=[\x27"]?(\w+)[\x27"]?#i) { print "$1"; } else {print "no_meta_robots";}')"
When the URL doesn't contain any meta header for robots, it will show no_meta_robots.

Curl bash script truncated/ merging

My aim is to have bash scripts get up to get a token from a WEB API and parse that token into a post command which will then create an new account (based on user input) on that WEB API. It succesffuly gets the token but when I reference the variable in my POST command, it either cuts it off or merges it into the tab. Has anyone seen this happen before?
I have read a similar issue where the issue was the content header, I have removed that but it hasnt fixed the issue.
A warning...I am a learner so my code isn't the best!
My bash script:
#!/bin/bash
curl --insecure -X POST \
https://192.168.XX.XXX:8443/application/token \
-H 'Accept: */*' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'Cache-Control: no-cache' \
-H 'Connection: keep-alive' \
-H 'Content-Type: application/xml' \
-H 'Host: 192.168.XX.XX:8443' \
-H 'cache-control: no-cache' \
-H 'client-key: 123' \
-H 'client-name: 1234' \
-H 'client-secret: 123456' \
-H 'grant_type: OAuth' \
-d '<request>
<username>johndoe</username>
<password>secret</password>
</request>' > tokenNewAcc.xml
grep 'access_token' tokenNewAcc.xml | sed 's/.*://' | sed s/,// | tr -d '"' > /home/oracle/tokenNewAcc.txt
rm tokenNewAcc.xml
value=`cat /home/oracle/tokenNewAcc.txt`
MYVAR=`cat /home/oracle/testaccounts.txt`
ID=`cut -d' ' -f1 /home/oracle/testaccounts.txt`
FNAME=`cut -d' ' -f2 /home/oracle/testaccounts.txt`
LNAME=`cut -d' ' -f3 /home/oracle/testaccounts.txt`
EMAIL=`cut -d' ' -f4 /home/oracle/testaccounts.txt`L
\
curl -k -X POST \
https://192.168.XX.XXX:8443/application/accounts \
-H 'Accept: */*' \
-H 'Content-Type: application/xml' \
-H 'Host: 192.168.XX.XXX:8443' \
-H "auth: "$value"" \
-H 'cache-control: no-cache' \
-d '<request>
<userid'$ID'</userid>
<firstname>'$FNAME'</firstname>
<lastname>'$LNAME'</lastname>
<email>'$EMAIL'</email>
</request>'
OUTPUT of POST where it has trouble:
curl -k -X POST https://192.168.36.XXX:8443/cloudminder/accounts -H 'Accept: */*' -H 'Content-Type: application/xml' -H 'Host: 192.168.36.1' -H 'cache-control: no-cache' -d '<request>ad9f23295b6
<useridconnectortest</userid>
<firstname>conn</firstname>
<lastname>connect</lastname>
<email>connectorL</email>
</request>'
As you can see, the host url is cut off, the beginning of the token (ad9f23295b6) merges into the request.

Resources