find (and remove) member from all GitLab projects (that I own) - gitlab

Is there a way to search all GitLab groups and associated projects (that I own or administer) for a specific user (with the intent of removing that user)? I know how if they are a member of the group itself, but if the group has many projects and some projects have users added individually, then it's tedious to search each project one-by-one.

I have not found such functionality in GitLab UI, so I developed python script using GitLab API.
Here is the link to repo with source code of my script and docs: https://gitlab.com/CVisionLab/gitlab-tools/
Just download the gitlab-delete-members.py script and run in bash command line. Usage example:
$ gitlab-delete-members.py --query johndoe123 --token Abcdefg123
💬 Auth at https://gitlab.com
Project "MyGroupSpam / my-project-foo" (id=11111111) : no users to delete
Project "MyGroupSpam / my-project-bar" (id=11111112) : delete johndoe123 (id=3333333) : 🟢 ok
Project "NotMyGroupEggs / not-my-project-baz" (id=11111113) : delete johndoe123 (id=3333333) : 🔴 fail
💬 1 members deleted in 2 repositories
Just in case I provide full source of the script here:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Handy script for searching and deleting members from Gitlab projects.
Built on top of GitLab Python API: https://python-gitlab.readthedocs.io/
"""
import argparse
import gitlab
def parse_args():
parser = argparse.ArgumentParser(
description='Delete member from all accessible gitlab projects.')
parser.add_argument('--query', required=True,
help='Gitlab query string that defines users '
'to be deleted (username is recommended)')
parser.add_argument('--token', required=True,
help='Gitlab token of your user')
parser.add_argument('--url', default='https://gitlab.com',
help='Gitlab URL')
parser.add_argument('--visibility', default='private',
help='Gitlab projects visibility')
parser.add_argument('--dry', action='store_true',
help='dry run')
return parser.parse_args()
def print_ok():
print('🟢 ok')
def print_fail():
print('🔴 fail')
def print_skip():
print('🟡 skip')
def main():
# Greeting and args parsing.
args = parse_args()
# Initialize Gitlab API.
print(f'💬 Auth to {args.url} : ', end='')
gl = gitlab.Gitlab(args.url, private_token=args.token)
try:
gl.auth()
print_ok()
except: # noqa
print_fail()
return
# Iterate over projects.
projects = gl.projects.list(all=True, visibility=args.visibility)
del_members_count = 0
del_projects_count = 0
for p in projects:
print(f'Project "{p.name_with_namespace}" (id={p.id}) :',
end='')
# Query members.
members = p.members.list(query=args.query)
# Delete members.
if len(members) == 0:
print(' no users to delete')
else:
del_projects_count += 1
for m in members:
print(f' delete {m.username} (id={m.id}) : ', end='')
if not args.dry:
try:
m.delete()
print_ok()
del_members_count += 1
except: # noqa
print_fail()
else:
print_skip()
# Statistics.
print(f'💬 {del_members_count} members deleted '
f'in {del_projects_count} repositories')
if __name__ == '__main__':
main()
If you want to know which GitLab API functions are used, here is the short list:
gl = gitlab.Gitlab(...) – create API object
gl.auth() - authenticate
projects = gl.projects.list(...) - get list of projects
members = p.members.list(...) - list members of the certain project that matches query
m.delete() - delete member from project

if the group has many projects and some projects have users added individually, then it's tedious to search each project one-by-one.
Yet, this would be the one option, considering it is easy to script:
The Groups API can list projects (a bit as in this gist)
The Projects API can list users
If you are an admin, you can also lists the project of a User
Removing a user from a project through API was requested in issue 7687 and resolved in GitLAb 8.7.
DEL /projects/#{project.id}/members/#{user.id}

Related

400 Caller's project doesn't match parent project

I have this block of code that basically translates text from one language to another using the cloud translate API. The problem is that this code always throws the error: "Caller's project doesn't match parent project". What could be the problem?
translation_separator = "translated_text: "
language_separator = "detected_language_code: "
translate_client = translate.TranslationServiceClient()
# parent = translate_client.location_path(
# self.translate_project_id, self.translate_location
# )
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = (
os.getcwd()
+ "/translator_credentials.json"
)
# Text can also be a sequence of strings, in which case this method
# will return a sequence of results for each text.
try:
result = str(
translate_client.translate_text(
request={
"contents": [text],
"target_language_code": self.target_language_code,
"parent": f'projects/{self.translate_project_id}/'
f'locations/{self.translate_location}',
"model": self.translate_model
}
)
)
print(result)
except Exception as e:
print("error here>>>>>", e)
Your issue seems to be related to the authentication method that you are using on your application, please follow the guide for authention methods with the translate API. If you are trying to pass the credentials using code, you can explicitly point to your service account file in code with:
def explicit():
from google.cloud import storage
# Explicitly use service account credentials by specifying the private key
# file.
storage_client = storage.Client.from_service_account_json(
'service_account.json')
Also, there is a codelab for getting started with the translation API with Python, this is a great step by step getting started guide for running the translate API with Python.
If the issue persists, you can try creating a Public Issue Tracker for Google Support

How to get the list of source repositories of a user/organisation on GitHub?

I wanna get the list of repositories of a user or an organisation on GitHub, but excluding forked, mirrored or archived repos.
Using PyGithub you can do the following to filter public repos:
from github import Github
g = Github()
user = g.get_user("<username>") # target user
repos = user.get_repos()
non_forks = []
for repo in user.get_repos():
if repo.fork is False:
non_forks.append(repo.name)
print(non_forks)
https://github.com/PyGithub/PyGithub
is a python library for interacting with the Github API. From the readme:
from github import Github
# First create a Github instance:
# using username and password
g = Github("user", "password")
# or using an access token
g = Github("access_token")
# Github Enterprise with custom hostname
g = Github(base_url="https://{hostname}/api/v3", login_or_token="access_token")
# Then play with your Github objects:
for repo in g.get_user().get_repos():
print(repo.name)
The Repository object has properties that allow checking whether the repo is archived, a fork, or a mirror:
repo.archived // is repo archived?
repo.fork // is repo a fork?
repo.mirror_url // return the url of the mirrored repo, if applicable
I've written a small script that does the job perfectly
#!/usr/bin/env python3.9
#-*- coding: utf-8 -*-
import readline, sys, os, requests
import click
from github import Github
g = Github()
def userexists(username):
addr = "https://api.github.com/users/" + username
response = requests.get(addr)
if response.status_code == 404:
return False
else:
if response.status_code == 200:
return True
def printrepos(repos):
original_repos = []
for repo in repos:
if repo.fork is False and repo.archived is False:
print(repo.clone_url)
#click.command()
#click.argument('username')
def main(username):
if userexists(username):
user = g.get_user(username)
repos = user.get_repos()
printrepos(repos)
else:
print("Username doesn't exist")
if __name__ == "__main__":
main()

Google Slides API: no "client_secret.json"

I'm new to Google Slides API and am trying to build a slide deck for daily news headlines by replacing image and text placeholders (for your reference, see https://www.youtube.com/watch?v=8LSUbKZq4ZY and http://wescpy.blogspot.com/2016/11/using-google-slides-api-with-python.html).
But when I try to run my modified program, I get an error message that says no file or directory exists called "client_secret.json" (which is included in the API tutorial's code). The tutorial code is from 2 years ago so I'm not sure if there's been any updates in the Google Slides API, but I'd really appreciate help on navigating this issue. Below is my code (note: "scraped list" is a list of dictionaries, with each dictionary containing a value for keys "headline" and "imgURL".)
from __future__ import print_function
from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools
from datetime import date
from scrapef2 import scrape
scrapedlist = scrape()
TMPLFILE = 'CrimsonTemplate' # use your own!
SCOPES = (
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/presentations',
)
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
creds = tools.run_flow(flow, store)
HTTP = creds.authorize(Http())
DRIVE = discovery.build('drive', 'v3', http=HTTP)
SLIDES = discovery.build('slides', 'v1', http=HTTP)
rsp = DRIVE.files().list(q="name='%s'" % TMPLFILE).execute().get('files')[0]
DATA = {'name': '[DN] '+ str(date.today())}
print('** Copying template %r as %r' % (rsp['name'], DATA['name']))
DECK_ID = DRIVE.files().copy(body=DATA, fileId=rsp['id']).execute().get('id') # TO DO: How to copy into a specific folder
for i in range(3):
print('** Get slide objects, search for image placeholder')
slide = SLIDES.presentations().get(presentationId=DECK_ID,
fields='slides').execute().get('slides')[i]
obj = None
for obj in slide['pageElements']:
if obj['shape']['shapeType'] == 'RECTANGLE':
break
print('** Replacing placeholder text and icon')
reqs = [
{'replaceAllText': {
'containsText': {'text': '{{Headline}}'},
'replaceText': scrapedlist[i]["headline"]
}},
{'createImage': {
'url': scrapedlist[i]["imgURL"],
'elementProperties': {
'pageObjectId': slide['objectId'],
'size': obj['size'],
'transform': obj['transform'],
}
}},
{'deleteObject': {'objectId': obj['objectId']}},
]
SLIDES.presentations().batchUpdate(body={'requests': reqs},
presentationId=DECK_ID).execute()
print('DONE')
Never used python google api but error indicates that you dont have your 'client_secret.json' file or it is in wrong place.
Scenario 1 - you dont have 'client_secret.json' file
This file is used by API to automatically verify that you are you. With this all API calls are made by your behalf. To get this file:
go to Google API console
open your project (or create new one)
click "Enable APIs and services" to find and enable Google Slides API
click "Credentials" in left menu, and then "Create credentials" -> "oAuth client ID"
choose Web application, accept all windows
now you should see new credentials on list, you can click on them and there will be button on top menu named "download JSON", there you will obtain your credentials (which by name are secret so keep them somewhere safe)
Scenario 2 - your 'client_secret.json' file is in wrong place
In this case I can't be very helpful, just try to inspect library to know where it looks for file and put it there (library directory, project root directory, hard to tell).
Let me know if it worked, as Google APIs and their libraries sometimes acts unexpectedly.

How to disable Create Project permission for users by default in GitLab?

I am using the Omnibus GitLab CE system with LDAP authentication.
Because of LDAP authentication, anyone in my company can sign in to GitLab and a new GitLab user account associated with this user is created (according to my understanding).
I want to modify it so that by default this new user (who can automatically sign in based on his LDAP credentials) cannot create new projects.
Then, I as the admin, will probably handle most new project creation.
I might give the Create Project permission to a few special users.
In newer versions of GitLab >= v7.8 …
This is not a setting in config/gitlab.yml but rather in the GUI for admins.
Simply navigate to https://___[your GitLab URL]___/admin/application_settings/general#js-account-settings, and set Default projects limit to 0.
You can then access individual users's project limit at https://___[your GitLab URL]___/admin/users.
See GitLab's update docs for more settings changed between v7.7 and v7.8.
git diff origin/7-7-stable:config/gitlab.yml.example origin/7-8-stable:config/gitlab.yml.example
For all new users:
Refer to Nick Merrill answer.
For all existing users:
This is the best and quick method to make changes to projects limits:
$ gitlab-rails runner "User.where(projects_limit: 10).each { |u| u.projects_limit = 0; u.save }"
( Update: This applies to versions <= 7.7:)
The default permissions are set in gitlab.yml
In omnibus, that is /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
Look for
## User settings
default_projects_limit: 10
# default_can_create_group: false # default: true
Setting default_projects_limit to zero, and default_can_create_group to false may be what you want.
Then an admin can change the limits for individual users.
Update:
This setting was included in the admin GUI in version 7.8 (see answer by #Nick M). At least with Omnibus on Centos7 an upgrade retains the setting.
Note that the setting default_can_create_group is still in gitlab.yml.
Here's my quick-and-dirty Python script which you can use in case you already have some users created and want to change all your existing users to make them unable to create projects on their own:
#!/usr/bin/env python
import requests
import json
gitlab_url = "https://<your_gitlab_host_and_domain>/api/v3"
headers = {'PRIVATE-TOKEN': '<private_token_of_a_user_with_admin_rights>'}
def set_user_projects_limit_to_zero (user):
user_id = str(user['id'])
put = requests.put(gitlab_url + "/users/" + user_id + "?projects_limit=0", headers=headers)
if put.status_code != 200:
print "!!! change failed with user id=%s, status code=%s" % (user_id, put.status_code)
exit(1)
else:
print "user with id=%s changed!" % user_id
users_processed = 0
page_no = 1
total_pages = 1
print "processing 1st page of users..."
while page_no <= total_pages:
users = requests.get(gitlab_url + "/users?page=" + str(page_no), headers=headers)
total_pages = int(users.headers['X-Total-Pages'])
for user in users.json():
set_user_projects_limit_to_zero(user)
users_processed = users_processed + 1
print "processed page %s/%s..." % (page_no, total_pages)
page_no = page_no + 1
print "no of processed users=%s" % users_processed
Tested & working with GitLab CE 8.4.1 052b38d, YMMV.

Jenkins Python API returns HTML

I'm trying to write a Python script to talk to my instance of Jenkins. I am using the newest version of the jenkinsapi module and querying Jenkins 1.509.3.
I can get a job list like follows:
l=j.get_jobs_list()
where j is an instance of jenkinsapi.Jenkins (I used the requester from jenkinsapi.utils.requester to skip ssl verification)
However, when I try to get more information on an individual job with
j.get_job(l[0])
it fails with this error: Inappropriate content found at [some_address] and what is returned is a bunch of HTML (that looks like the starting page for my instance, the one you see when you log in) instead of anything that should look like the response. Pasting [some_address] into the browser gives me what I expect as a response.
While I can get some information on the Jenkins instance, what I am really interested in is info on individual jobs. Any ideas how to fix it and get the job info?
Using python 3.6, python-jenkins 1.0.1 and Jenkins 2.121.1, following works nicely:
import pprint
import jenkins
IP = 'localhost'
USERNAME = 'my_username'
PW = 'my_password'
def get_version(server):
user = server.get_whoami()
version = server.get_version()
print('Hello %s from Jenkins %s' % (user['fullName'], version))
def get_jobs(server):
jobs = server.get_jobs() # List[dict]
print("Here are top 5 jobs")
pprint(jobs[:5])
return jobs
def get_job(server, job_name):
job_config = server.get_job_config(job_name) # XML
job_info = server.get_job_info(job_name) # dict
print("\n --- JOB CONFIG --- ")
print(job_config)
print("\n --- JOB INFO --- ")
pprint(job_info)
if __name__ == "__main__":
_server = jenkins.Jenkins(IP, username=USERNAME, password=PW)
get_version(_server)
_jobs = get_jobs(_server)
get_job(_server, _jobs[0]['name'])
Jenkins API I was using is documented here: https://python-jenkins.readthedocs.io/en/latest/index.html

Resources