I am working on GitHub and Google Cloud Build integration for CI/CD.
I am able to link repository to Google Cloud build installation in GitHub using below code.
import requests
header = dict()
header['Authorization'] = 'token %s' % personal_token
header['Accept'] = 'application/vnd.github.machine-man-preview+json'
# Fetching repository id
url = f'https://api.github.com/repos/{organization_name}/{repo_name}'
output = requests.get(url, headers=header)
repo_id = output.json()['id']
# Adding repository to Google Cloud build installation
url = f'https://api.github.com/user/installations/{gcb_installation_id}/repositories/{repo_id}'
output = requests.put(url, headers=header)
When I try to create trigger with below code, I am getting an error google.api_core.exceptions.FailedPrecondition: 400 Repository mapping does not exist
from google.cloud.devtools import cloudbuild_v1
client = cloudbuild_v1.CloudBuildClient()
build_trigger_template = cloudbuild_v1.types.BuildTrigger()
build_trigger_template.description = 'test to create trigger'
build_trigger_template.name = 'github-cloudbuild-trigger1'
build_trigger_template.github.name = 'github-cloudbuild'
build_trigger_template.github.pull_request.branch = 'master'
build_trigger_template.filename = 'cloudbuild.yaml'
response = client.create_build_trigger('dev',
build_trigger_template)
Can anyone please help me on how to connect (map) a GitHub repository to Google Cloud build project using python or any other automated process. I am able to manually map repository but need to automate.
Thanks,
Raghunath.
Related
I try to request the number of stars and commits of a public software repository in GitLab using its Python client. However, I keep getting GitlabHttpError 503 if executing the following script.
import gitlab
import requests
url = 'https://gitlab.com/juliensimon/huggingface-demos'
private_token = 'xxxxxxxx'
gl = gitlab.Gitlab(url, private_token=private_token)
all_projects = gl.projects.list(all=True)
I read the previous posts but none of them works for me: [1], [2], and [3]. People mentioned:
Retrying later usually works [I tried in different periods but still got the same error.]
Setting an environment variable for no_proxy [Not sure what it means for me? I do not set the proxy explicitly.]
The 503 response is telling you something - your base URL is off. You only need to provide the base GitLab URL so the client makes requests against its api/v4/ endpoint.
Either use https://gitlab.com only, so that the client will correctly call https://gitlab.com/api/v4 endpoints (instead of trying https://gitlab.com/juliensimon/huggingface-demos/api/v4 as it does now), or skip it entirely when using GitLab.com if you're on python-gitlab 3.0.0 or later.
# Explicit gitlab.com
url = 'https://gitlab.com'
gl = gitlab.Gitlab(url, private_token=private_token)
# Or just use the default for GitLab.com (python-gitlab 3.0.0+ required)
gl = gitlab.Gitlab(private_token=private_token)
Edit: The original question was about the 503, but the comment to my answer is a follow-up on how to get project details. Here's the full snippet, which also fetches the token from the environment instead:
import os
import gitlab
private_token = os.getenv("GITLAB_PRIVATE_TOKEN")
gl = gitlab.Gitlab(private_token=private_token)
project = gl.projects.get("juliensimon/huggingface-demos")
print(project.forks_count)
print(project.star_count)
Scenario: I need to get when was the latest commit done in the repo and by whom, and to which branch did that user do the commit on?
Solution:
I'm using python azure.devops module. and here is my code:
cm_search_criteria = models.GitQueryCommitsCriteria(history_mode='firstParent', top=10)
commits = git_client.get_commits(repo.id, search_criteria=cm_search_criteria, project=project.name)
for i in commits:
datetimeobj = datetime.strptime(i.committer.date.strftime("%x"), '%m/%d/%y')
last_commit_on = datetimeobj.date()
last_commit_by = i.committer.email
break
Now how do I get the branch name to which the user had committed the code? In the UI we can see the branch name... how can i get the same data using Azure Devops REST API ?
enter image description here
you may need to use Stats - List to retrieve statistics about all branches within a repository, then evaluate the latest commit, you also need to consider argument of baseVersionDescriptor.versionOptions of firstParent
I'm not sure if Python wrapper module support this seems like the github project is achieved now.
I am using google cloud build for CI/CD purpose, in which I need to give access for specific repositories (can't use All repositories option). Is there any way to provide access by repository wise through python code. If not possible through python is there any alternative to this requirement.
Thanks,
Raghunath.
When I checked with GitHub support, they have shared me below links.
To get repository id -- https://developer.github.com/v3/repos/#get-a-repository
To get installations details -- https://developer.github.com/v3/orgs/#list-installations-for-an-organization
To add repository to an installation -- https://developer.github.com/v3/apps/installations/#add-repository-to-installation
I used these links to create below mentioned code which has helped me to implement the desired requirement.
# Header to use in request
header = dict()
header['Authorization'] = 'token %s' % personal_token
header['Accept'] = 'application/vnd.github.machine-man-preview+json'
# Fetching repository id
url = f'https://api.github.com/repos/{organization_name}/{repo_name}'
output = requests.get(url, headers=header)
repo_id = output.json()['id']
# Adding repository to Google Cloud build installation
url = f'https://api.github.com/user/installations/{gcb_installation_id}/repositories/{repo_id}'
output = requests.put(url, headers=header)
I have set up multiple Azure DevOps build pipelines that are triggered by submitting pull requests to a GitHub repo. Whenever a build happens, I can navigate to that build's page and see its build steps and whether the build was successful. How do I pull that information out programmatically?
You can get a specific build status by restful api
https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}?api-version=5.1
But I guess you probably want a build status badge to be added to your git repo.
First go to the Pipelines page in your Azure Devops, select the pipeline of your git repo.
On the right top corner, click on the eclipse and choose status badge
Copy the sample markdown link.
Then go to your github repo, and paste the status markdown link in the read.me file and commit the change to your repo. The build status will be shown as below in your github repo
And aslo you need go to the Project Settings of your azure devops. Go to the Settings under Pipelines. Make sure Allow anonymous access to badges checkbox under General
To pull information out of Azure DevOps programmatically, I can use the Azure DevOps Python API. This SDK provides multiple clients to pull various data from Azure DevOps. As #Levi Lu-MSFT's answer hinted, the data I wanted can be obtained from the build client.
I installed the SDK in a conda environment with this YAML:
name: DevOpsData
channels:
- conda-forge
dependencies:
- python=3.6
- pip==19.2.3
- nb_conda_kernels==2.2.1
- papermill==1.0.1
- pandas==0.23.4
- scikit-learn==0.20.0
- lightgbm==2.2.1
- pip:
- prompt_toolkit==2.0.9
- azure-cli==2.0.69
- azure-devops
I pulled what I needed using this script:
# Copyright (C) Microsoft Corporation. All rights reserved.
from azure.devops.connection import Connection
from msrest.authentication import BasicAuthentication
import pprint
import datetime
if __name__ == "__main__":
# Fill in with your personal access token and org URL
personal_access_token = "MY_ACDESS_TOKEN"
organization_url = "https://dev.azure.com/MY_ORGANIZATION"
# Create a connection to the org
credentials = BasicAuthentication("", personal_access_token)
connection = Connection(base_url=organization_url, creds=credentials)
# Get a client (the "core" client provides access to projects, teams, etc)
build_client = connection.clients.get_build_client()
# Get the first page of builds
project = "MY_PROJECT_NAME"
get_builds_response = build_client.get_builds(project=project)
index = 0
while get_builds_response is not None:
for build in get_builds_response.value:
duration = build.finish_time - build.start_time
seconds = duration.days*(24*60*60)+duration.seconds
print("[{}]\t{}\t{}\t{}\t{:,} seconds".format(
index, build.build_number, build.source_branch, build.result,
seconds
))
index += 1
if (get_builds_response.continuation_token is not None
and get_builds_response.continuation_token != ""):
# Get the next page of builds
get_builds_response = build_client.get_builds(
continuation_token=get_builds_response.continuation_token)
else:
# All builds have been retrieved
get_builds_response = None
I'm making a python app that launches a batch.
I want, via user inputs, to create a pool.
For simplicity, I'll just add all the applications present in the batch account to the pool.
I'm not able to get the list of available application packages.
This is the portion of code:
import azure.batch.batch_service_client as batch
from azure.common.credentials import ServicePrincipalCredentials
credentials = ServicePrincipalCredentials(
client_id='xxxxx',
secret='xxxxx',
tenant='xxxx',
resource="https://batch.core.windows.net/"
)
batch_client = batch.BatchServiceClient(
credentials,
base_url=self.AppData['CloudSettings'][0]['BatchAccountURL'])
# Get list of applications
batchApps = batch_client.application.list()
I can create a pool, so credentials are good and there are applications but the returned list is empty.
Can anybody help me with this?
Thank you,
Guido
Update:
I tried:
import azure.batch.batch_service_client as batch
batchApps = batch.ApplicationOperations.list(batch_client)
and
import azure.batch.operations as batch_operations
batchApps = batch_operations.ApplicationOperations.list(batch_client)
but they don't seem to work. batchApps is always empty.
I don't think it's an authentication issue since I'd get an error otherwise.
At this point I wonder if it just a bug in the python SDK?
The SDK version I'm using is:
azure.batch: 4.1.3
azure: 4.0.0
This is a screenshot of the empty batchApps var:
Is this the link you are looking for:
Understanding the application package concept here: https://learn.microsoft.com/en-us/azure/batch/batch-application-packages
Since its python SDK in action here: https://learn.microsoft.com/en-us/python/api/azure-batch/azure.batch.operations.applicationoperations?view=azure-python
list operation and here is get
hope this helps.
I haven't tried lately using the Azure Python SDK but the way I solved this was to use the Azure REST API:
https://learn.microsoft.com/en-us/rest/api/batchservice/application/list
For the authorization, I had to create an application and give it access to the Batch services and the I programmatically generated the token with the following request:
data = {'grant_type': 'client_credentials',
'client_id': clientId,
'client_secret': clientSecret,
'resource': 'https://batch.core.windows.net/'}
postReply = requests.post('https://login.microsoftonline.com/' + tenantId + '/oauth2/token', data)