How to get total time spend in Gitlab? - gitlab

Is there a way to get the total time spend on all issues that a user have spend with the time tracking /spend slash command?
Time tracking stats with the API gets just a small amount of data: https://docs.gitlab.com/ce/api/issues.html#get-time-tracking-stats
Gitlab CE 9.1.4

As I see it is possible to parse comments from API v3 and calculate total.
For example,
https://gitlab.com/api/v3/projects/:id/issues/:issue_id/notes?private_token=your_token
{
id: 73113225,
body: "added 1h of time spent at 2018-05-15",
attachment: null,
author: {
...
username: "mnvxxx",
},
...
}
More info:
https://docs.gitlab.com/ee/api/notes.html
UPDATE
Currently I has created instrument for calculating spent time by every contributor. I hope it will be helpful:
https://github.com/zubroide/gitpab

Here is a very simple Python script that works with API v4:
import requests
API_KEY = "" # Enter your API key here
BASE_URL = "https://{{enter gitlab url here}}/api/v4/"
item_counter = 0
total_seconds = 0
for i in range(1, 57): # manually set range of issues here. All issues doesn't work well.
issue = requests.get(BASE_URL + 'projects/2/issues/' + str(i) + '/time_stats')
total_seconds += issue.json()['total_time_spent']
item_counter += 1
print("Hours on all issues: %.2f" % float((total_seconds / 60) / 60))
print("Total issues: " + str(item_counter))
I'm posting to this thread because this is the first answer that comes up on Google, and there isn't really any other ready-made solutions to be found.

Building on top of what #josh-harkema has provided, this is a version that lists all closed issues that were assigned to a specific username that have been updated in a given time period (and do not have the label 'paid' set):
import requests
import os
username = os.environ.get('GITLAB_REPORTING_USERNAME')
project_id = os.environ.get('GITLAB_REPORTING_PROJECTID') # in the top of your project page
access_token = os.environ.get('GITLAB_REPORTING_TOKEN') # https://gitlab.com/profile/personal_access_tokens
base_url = "https://gitlab.com/api/v4"
updated_after = "2019-06-01T00:00:00.00Z"
updated_before = "2019-07-01T00:00:00.00Z"
item_counter = 0
total_seconds = 0
headers = { 'Private-Token': access_token }
url_template = "{base_url}/projects/{project_id}/issues?" \
"state=closed&assignee_username={username}&updated_after={updated_after}&updated_before={updated_before}"
url = url_template.format(base_url=base_url, project_id=project_id, username=username,
updated_after=updated_after, updated_before=updated_before)
# call API
issues = requests.get(url, headers = headers)
total_seconds = 0
issues_to_pay = []
line_template = "id: {id} closed: {closed_at} time spent: {time}\ttitle: {title}\turl: {url}"
print("Issue statistics for {u} from {f} to {t}:\n".format(u=username,f=updated_after, t=updated_before))
for issue in issues.json():
time_val = issue['time_stats']['human_total_time_spent']
already_paid = u'paid' in issue['labels'] # you can put a label 'paid' to exclude an issue
if already_paid:
time_val = time_val + " *"
else:
# if the issue has been paid, already, don't add the time, and don't list as to be paid
total_seconds += issue['time_stats']['total_time_spent']
issues_to_pay.append(str(issue['id']))
line = line_template.format(
id=issue['id'],
closed_at=issue['closed_at'],
title=issue['title'],
time=time_val,
url=issue['web_url']
)
print(line)
print("")
print("Hours to pay on all issues: %.2f" % float((float(total_seconds) / 60) / 60))
print("")
print("* = issue has been paid for, already")
print("All issue to pay: {issues}".format(issues=",".join(issues_to_pay)))
Note: You need to have environment variables set for the GITLAB_REPORTING_USERNAME, the GITLAB_REPORTING_PROJECTID, as well as the GITLAB_REPORTING_TOKEN.
We use this to pay contractors. Hope this helps!

I was myself looking for the same and after more searching, I found this excellent CLI tool called gitlab-time-tracker. It generates comprehensive reports of tracked-time that you can customise by multiple options and can print them even as PDFs!
For keeping this answer relevant to OP's question, you can print (in your terminal) total time spent by a user using following command**:
gtt report "namespace/project" --user username --closed --from="2017-03-01" --to="2017-04-01"
** This assumes that you've installed this tool (gtt) and setup your Gitlab PAT (with "api" scope activated) in its config file.

You'll have to use both the Gitlab Commits API and the GraphQL API to achieve it. Below is some code that's been shorted for brevity.
You'll need to specify a Gitlab instance flag and your personal token.
Say you have a function used to capture all users in your Gitlab instance called "GetUsers":
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"time"
"github.com/sirupsen/logrus"
"github.com/xanzy/go-gitlab"
"gopkg.in/alecthomas/kingpin.v2"
)
var (
address = kingpin.Flag("address", "The Gitlab URL to use").Required().String()
token = kingpin.Flag("token", "Project token").String()
)
func main () {
timeTracker()
}
func timeTracker(git *gitlab.Client) {
names := GetUsers(git)
for _, name := range names {
jsonData := map[string]interface{}{
"query": `
query($user: String!) {
timelogs(username: $user ) {
edges {
node {
id
user {
id
username
}
timeSpent
issue{
labels{
nodes{
title
}
}
}
}
}
}
}
`, "variables": fmt.Sprintf(`{"user":"%s"}`, name),
}
jsonValue, _ := json.Marshal(jsonData)
request, err := http.NewRequest("POST", "https://" +*address + "/api/graphql", bytes.NewBuffer(jsonValue))
if err != nil {
logrus.Error(err)
}
request.Header.Set("Authorization", "Bearer "+*token)
request.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: time.Second * 10}
response, err := client.Do(request)
response.Body.Close()
if err != nil {
fmt.Printf("The HTTP request failed with error %s\n", err)
}
logrus.Print(ioutil.ReadAll(response.Body))
Result (when decode with JSONDecoder):
INFO[0000] User: user1, Time spent: 300 (s)
INFO[0000] User: user2, Time spent: 120 (s)
You can then take this data and decode it into a struct (I would copy and paste the post request to an autogenerator for sanity's sake), then do what you want with it. Or change the POST request to capture users by Project if you're more interested in something granular.
Source: https://docs.gitlab.com/ee/api/graphql/getting_started.html

Related

How can I scrape multiple followers with Tweepy?

The Twitter API limit lets me only scrape 15-20 pages with 200 followers. After a 15 min break limit is set.
If I run my code new I don't want to start again at the beginning of a follower list, how can I step into where I left before the limitation?
My code looks like this with a Twitter user example who has 4.8 Mill follower.
def fetch_followers_to_df():
try:
screen_name = "JeffBezos"
# fetching the user
user = api.get_user(screen_name=screen_name)
num_followers = user.followers_count
print(user.id)
print(user.followers_count)
cnt_followers = input()
if cnt_followers == "":
cnt_pages = float(num_followers) // 200
print(cnt_pages)
df_followers = pd.DataFrame(columns=["User", "ID", "Bio", "Location"])
for page in tweepy.Cursor(
api.get_followers, screen_name=screen_name, count=200
).pages(cnt_pages):
for user in page:
temp_dict = {
"User": user.screen_name,
"ID": user.id,
"Bio": user.description,
"Location": user.location,
}
temp_df = pd.DataFrame(data=temp_dict, index=[1])
df_followers = pd.concat([df_followers, temp_df])
print(len(page))
df_followers.to_csv("raw_followers.csv", index=False)
except:
print("Wait for new API Limit!")
df_followers = pd.read_csv("raw_followers.csv")
return df_followers
Set the wait_on_rate_limit argument to True when you initialize API (see here).
On a side note, you don't need the cnt_pages argument for pages (default is inf), and you would go much faster by using the methods of the Twitter API V2.

How to create Wiki Subpages in Azure Devops thru Python?

My Azure devops page will look like :
I have 4 pandas dataframes.
I need to create 4 sub pages in Azure devops wiki from each dataframe.
Say, Sub1 from first dataframe, Sub2 from second dataframe and so on.
My result should be in tab. The result should look like :
Is it possible to create subpages thru API?
I have referenced the following docs. But I am unable to make any sense. Any inputs will be helpful. Thanks.
https://github.com/microsoft/azure-devops-python-samples/blob/main/API%20Samples.ipynb
https://learn.microsoft.com/en-us/rest/api/azure/devops/wiki/pages/create%20or%20update?view=azure-devops-rest-6.0
To create a wiki subpage, you should use Pages - Create Or Update api, and specify the path to pagename/subpagename. Regarding how to use the api in Python, you could use Azure DevOps Python API and refer to the sample below:
def create_or_update_page(self, parameters, project, wiki_identifier, path, version, comment=None, version_descriptor=None):
"""CreateOrUpdatePage.
[Preview API] Creates or edits a wiki page.
:param :class:`<WikiPageCreateOrUpdateParameters> <azure.devops.v6_0.wiki.models.WikiPageCreateOrUpdateParameters>` parameters: Wiki create or update operation parameters.
:param str project: Project ID or project name
:param str wiki_identifier: Wiki ID or wiki name.
:param str path: Wiki page path.
:param String version: Version of the page on which the change is to be made. Mandatory for `Edit` scenario. To be populated in the If-Match header of the request.
:param str comment: Comment to be associated with the page operation.
:param :class:`<GitVersionDescriptor> <azure.devops.v6_0.wiki.models.GitVersionDescriptor>` version_descriptor: GitVersionDescriptor for the page. (Optional in case of ProjectWiki).
:rtype: :class:`<WikiPageResponse> <azure.devops.v6_0.wiki.models.WikiPageResponse>`
"""
route_values = {}
if project is not None:
route_values['project'] = self._serialize.url('project', project, 'str')
if wiki_identifier is not None:
route_values['wikiIdentifier'] = self._serialize.url('wiki_identifier', wiki_identifier, 'str')
query_parameters = {}
if path is not None:
query_parameters['path'] = self._serialize.query('path', path, 'str')
if comment is not None:
query_parameters['comment'] = self._serialize.query('comment', comment, 'str')
if version_descriptor is not None:
if version_descriptor.version_type is not None:
query_parameters['versionDescriptor.versionType'] = version_descriptor.version_type
if version_descriptor.version is not None:
query_parameters['versionDescriptor.version'] = version_descriptor.version
if version_descriptor.version_options is not None:
query_parameters['versionDescriptor.versionOptions'] = version_descriptor.version_options
additional_headers = {}
if version is not None:
additional_headers['If-Match'] = version
content = self._serialize.body(parameters, 'WikiPageCreateOrUpdateParameters')
response = self._send(http_method='PUT',
location_id='25d3fbc7-fe3d-46cb-b5a5-0b6f79caf27b',
version='6.0-preview.1',
route_values=route_values,
query_parameters=query_parameters,
additional_headers=additional_headers,
content=content)
response_object = models.WikiPageResponse()
response_object.page = self._deserialize('WikiPage', response)
response_object.eTag = response.headers.get('ETag')
return response_object
More details, you could refer to the link below:
https://github.com/microsoft/azure-devops-python-api/blob/451cade4c475482792cbe9e522c1fee32393139e/azure-devops/azure/devops/v6_0/wiki/wiki_client.py#L107
Able to achieve with rest api
import requests
import base64
import pandas as pd
pat = 'TO BE FILLED BY YOU' #CONFIDENTIAL
authorization = str(base64.b64encode(bytes(':'+pat, 'ascii')), 'ascii')
headers = {
'Accept': 'application/json',
'Authorization': 'Basic '+authorization
}
df = pd.read_csv('sf_metadata.csv') #METADATA OF 3 TABLES
df.set_index('TABLE_NAME', inplace=True,drop=True)
df_test1 = df.loc['CURRENCY']
x1 = df_test1.to_html() # CONVERTING TO HTML TO PRESERVE THE TABULAR STRUCTURE
#JSON FOR PUT REQUEST
SamplePage1 = {
"content": x1
}
#API CALLS TO AZURE DEVOPS WIKI
response = requests.put(
url="https://dev.azure.com/xxx/yyy/_apis/wiki/wikis/yyy.wiki/pages?path=SamplePag2&api-version=6.0", headers=headers,json=SamplePage1)
print(response.text)
Based on #usr_lal123's answer, here is a function that can update a wiki page or create it if it doesn't:
import requests
import base64
pat = '' # Personal Access Token to be created by you
authorization = str(base64.b64encode(bytes(':'+pat, 'ascii')), 'ascii')
def update_or_create_wiki_page(organization, project, wikiIdentifier, path):
# Check if page exists by performing a Get
headers = {
'Accept': 'application/json',
'Authorization': 'Basic '+authorization
}
response = requests.get(url=f"https://dev.azure.com/{organization}/{project}/_apis/wiki/wikis/{wikiIdentifier}/pages?path={path}&api-version=6.0", headers=headers)
# Existing page will return an ETag in their response, which is required when updating a page
version = ''
if response.ok:
version = response.headers['ETag']
# Modify the headers
headers['If-Match'] = version
pageContent = {
"content": "[[_TOC_]] \n ## Section 1 \n normal text"
+ "\n ## Section 2 \n [ADO link](https://azure.microsoft.com/en-us/products/devops/)"
}
response = requests.put(
url=f"https://dev.azure.com/{organization}/{project}/_apis/wiki/wikis/{wikiIdentifier}/pages?path={path}&api-version=6.0", headers=headers,json=pageContent)
print("response.text: ", response.text)

Locust load testing For API calls

I have written a code for locust load testing for my case, where i can do a token call and then will make feature calls as per below code.
This is working fine with single token and 'n' number of users mentioned in master
I made a token call outside class and sending it as parameter to ENV. The userclass is reading the single token and using the same for all users.
I do not want to make token call inside the class, as it generates new token in every execution.
I'm looking if there is anyway like making token calls based on number of users mentioned on master -u and using only those tokens in User class.
Please suggest me if there is any documentation pointer which i can refer for this usecase
#! /usr/bin/python3.6
import json
from locust import HttpUser, task, constant, tag, events
from locust.log import setup_logging
import os
from datetime import datetime
import requests
setup_logging("INFO", None)
#events.init_command_line_parser.add_listener
def init_parser(parser):
parser.add_argument("--resp-delay", type=str, env_var="LOCUST_RESP_DELAY", default="", help="It's working")
parser.add_argument("--resp-size", type=str, env_var="LOCUST_RESP_SIZE", default="", help="It's working")
parser.add_argument("--env-endpoint", type=str, env_var="LOCUST_ENV_ENDPOINT", default="", help="It's working")
#events.init.add_listener
def _(environment, **kw):
os.environ['resp-delay'] = environment.parsed_options.resp_delay
os.environ['resp-size'] = environment.parsed_options.resp_size
os.environ['env-endpoint'] = environment.parsed_options.env_endpoint
with open("resources/data/" + environment.parsed_options.env_endpoint + '/data.json') as f:
data = json.load(f)
cal_transaction_id = datetime.now().strftime('%Y%m%dT%H%M%S')
#retrive cliend id and client secret from bitbucket repo
dict_car_app_all = data["data"]
print("env-endpoint:" + os.environ.get("env-endpoint"))
token_url = "https://ENDPOINT/auth/token"
#retrive cliend id and client secret from bitbucket repo
token_payload = "client_id=" + dict_car_app_all[0]["client_id"] + "&client_secret=" + dict_car_app_all[0]["client_secret"]
token_headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
response = requests.request("POST", token_url, data=token_payload, headers=token_headers,
cert=( 'resources/certs/' + environment.parsed_options.env_endpoint + '/MyCERT.pem',
'resources/certs/' + environment.parsed_options.env_endpoint + '/MYCERT.key'))
result = json.loads(response.text)
token = result["access_token"]
os.environ['access_token'] = token
os.environ['cal_transaction_id'] = cal_transaction_id
#class User_1(User):
class User_0(HttpUser):
wait_time = constant(1)
host = "host.com"
#tag('tag1')
#task
def load_test_api_tag1(self):
token_0 = os.environ.get('access_token')
cal_transaction_id = os.environ.get('cal_transaction_id')
env_endpoint = os.environ.get('env-endpoint')
resp_delay = os.environ.get("resp-delay")
resp_size = os.environ.get("resp-size")
feature_headers = {
'Authorization': "Bearer " + str(token_0),
'sm_transactionID': cal_transaction_id
}
url = "https://ENDPOINT/SERVICE/mytestservice/first_test"
querystring = {"response_delay": resp_delay, "data_size": resp_size}
self.client.request("GET", url, headers=feature_headers, params=querystring,
cert = ('resources/certs/' + env_endpoint + '/MyCERT.pem',
'resources/certs/' + env_endpoint + '/MyCERT.key'))
You can generate tokens in User class's on_start method so each user generates a new token when spawning.
class MyUser(User):
def on_start(self):
#generate token here and assign an instance variable like self.token=abc
super().on_start()
there is a drawback to this though, if your user count is more than your token generating service can handle some users will not be able to spawn, the way I do in my tests is if token generating part is not a part of the system I am testing, I generate tokens beforehand and write it in some file or some external db and read them from there.

How to Poll a request for certain interval and determine Pass or Failure

I've the API request as below :
* def reqCreate = read('classpath:integration/create-request.json')
* def resCreate = read('classpath:integration/create-response.json')
* def personId = 12
Given path '/person/' + personId
And header Authorization = 'Bearer ' + accessToken
When method get
Then status 200
Then match response == resCreate
I need to check the response after every 5 seconds till one minute.During one minute or till one minutes any any moment, if response assertion gives true then final result is true else should return false after specified duration.
This is explained clearly in the documentation : https://github.com/intuit/karate#retry-until
* configure retry = { count: 12, interval: 5000 }
Given url demoBaseUrl
And path 'greeting'
And retry until response.id > 3
When method get
Then status 200
In your case :
Given path '/person/' + personId
And configure retry = { count: 12, interval: 5000 }
And header Authorization = 'Bearer ' + accessToken
And retry until response == resCreate
When method get
Then status 200
Please don't forget to mark your previous questions as answered, you have a few that you left open.

How to create a random string everytime a test runs in karate dsl

The json request I am sending is:
Given url applicationURL
And path 'applications'
And header Authorization = subscribeToken
And request:
{
"throttlingTier": "Unlimited",
"description": "sample app description",
"name": "TestbyKarate",
"callbackUrl": "https:/apistore-dev-dev-a878-14-ams10-nonp.qcpaws.qantas.com.au/callback"
}
When method post
Then status 201
* def applicationId = response.applicationId
* print 'applicationId is ', applicationId
I am sending the name in my request as TestbyKarate but I want to send a unique value every time my test runs.
Is there any way to do it?
Can you please read the docs once. It will really benefit you.
https://github.com/intuit/karate#commonly-needed-utilities
So in the Background or common feature, you have:
* def now = function(){ return java.lang.System.currentTimeMillis() }
Then you can do this:
* def name = 'Test-' + now()
* request { name: '#(name)' }
check below example
Feature: Create User Name
Scenario: UserName with unique Name
def getDate =
"""
function(){ return java.lang.System.nanoTime() }
"""
def time = getDate()
def userName = 'createUser' + time + '_Test'

Resources