How to get subfolders and files using gitlab api - gitlab

I am using gitlab api to get the files and folders and succeded,
But I can able to get only directory names, not its subfolders and files.
So, how can i get full tree of my repository.
Please let me know.
Thanks in advance,
Mallikarjuna

According to the API, we can use
GET /projects/:id/repository/tree
to list files and directories in a project. But we can only get the files and directories in top-level of the repo in this way, and sub-directories of directories in top-level with param path.
If you wanna get directories of script/js/components, for example, you can use
GET /projects/:id/repository/tree?path=script/js/components

Rest API
You can use the recursive option to get the full tree using /projects/:id/repository/tree?recursive=true
For example : https://your_gitlab_host/api/v4/projects/:id/repository/tree?recursive=true&per_page=100
GraphQL API
You can also use the recently released Gitlab GraphQL API to get the trees in a recursive way :
{
project(fullPath: "project_name_here") {
repository {
tree(ref: "master", recursive: true){
blobs{
nodes {
name
type
flatPath
}
}
}
}
}
}
You can go to the following URL : https://$gitlab_url/-/graphql-explorer and past the above query
The Graphql endpoint is a POST on "https://$gitlab_url/api/graphql"
An example using curl & jq :
gitlab_url=<your gitlab host>
access_token=<your access token>
project_name=<your project name>
branch=master
curl -s -H "Authorization: Bearer $access_token" \
-H "Content-Type:application/json" \
-d '{
"query": "{ project(fullPath: \"'$project_name'\") { repository { tree(ref: \"'$branch'\", recursive: true){ blobs{ nodes { name type flatPath }}}}}}"
}' "https://$gitlab_url/api/graphql" | jq '.'

you should do url encoding to the full path of the file. for example lest assume that the path to file under your repository is: javascript/header.js
then you could use:
curl --head --header "PRIVATE-TOKEN: <your_access_token>" "https://<>/api/v4/projects//repository/files/javascript%2Fheader%2Ejs"

Of course, as mentioned in other responses, you have missed the path attribute of the gitlab repositories API which lets you browse the file hierarchy.
In addition, for simplicity, the python gitlab project exposes it through the projects API. Example:
# list the content of the root directory for the default branch
items = project.repository_tree()
# list the content of a subdirectory on a specific branch
items = project.repository_tree(path='docs', ref='branch1')

For getting the whole tree with sub-directories and files, you can pass a parameter called "recursive" to true
By default it's false
Api - {Gitlab_URl}/api/v4/projects/{Project_id}/repository/tree?recursive=true
Thanks!

Related

Create new git project: Is not valid HTTP Git repository

I try create new git projects with files from the existing project: https://docs.gitlab.com/ee/api/projects.html#create-project
curl --request POST --header "PRIVATE-TOKEN: <your-token>" \
--header "Content-Type: application/json" --data '{
"name": "new_project", "description": "New Project", "path": "new_project",
"import_url": "https://user:token#gitlab.com/<path>.git",
"namespace_id": "42", "initialize_with_readme": "false"}' \
--url 'https://gitlab.example.com/api/v4/projects/'
But I got the error:
{"message":"https://user:token#gitlab.com/<path>.git is not a valid HTTP Git repository"}
I was found the issue https://gitlab.com/gitlab-org/gitlab/-/issues/360266 with same error, but I set "initialize_with_readme": "false"
Gitlab version is 14.9.2
Also I tried use a broken url as import_url and got this error too
The issue you mentioned used:
https://user:token#gitlab.example.com/templates/someproject.git
^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
| |- replace with your project/repo name
|
|- needed if the import project is private
and need authentication to be accessed
That type of URL might work better in your case.
It this is somehow not supported, as in this thread (Apr. 2022) mentioned by the OP, the workaround is:
In any case we just worked around it by creating a blank project in the new GitLab, then pulling and pushing the project from the old one and into the new one.
This coding works well.
gitLabApi.getProjectApi().createProject(pro,"https://username:password#host/group/front-template.git");

Updating a file for a quick-pull using github cli

Currently in the github UI, a user can edit a file and create a new branch in a single action. This can also be done through the github api using something like this:
curl 'https://github.com/<my_org>/<my_repo>/tree-save/master/<path_to_file>' \
-H 'content-type: application/x-www-form-urlencoded' \
--data-raw 'authenticity_token=<generated_token>&filename=<filename>&new_filename=<filename>&content_changed=true&value=<new_contents_of_file>&message=Updated+file+in+my+repo&placeholder_message=Update+<filename>&description=&commit-choice=quick-pull&target_branch=<new_branch_name>&quick_pull=master&guidance_task=&commit=<target_commit_checksum>&same_repo=1&pr='
What I would like to be able to do, is perform the same action using the github cli* (gh). I have tried using the following commands:
gh api <my_org>/<my_repo>/tree-save/master/<path_to_file> -F "filename=<filename>" -F ...
and
gh api repos/<my_org>/<my_repo>/contents/<path_to_file> -F "filename=<filename>" -F ...
For both cases (and many variations on these options), I'm getting a 404** back. Any ideas what I'm doing wrong? Does the github cli even allow the functionality allowed in the above curl?
* For those curious, I want to use the CLI because of how it handles auth and it's statelessness. I can't generate a token to use, like in the curl above. And, due to multiple issues, I also can't clone the repo locally.
** I'm able to retrieve the file just fine using the simple GET command (the second command above without the '-F' flags)
After reading documentation, and then verifying by altering credentials, it appears to be a permissions issue. Evidently, for security reasons, if a token is used that does not meet the required permissions, a 404 is returned instead of a 403.
Interesting that I can still use the curl above through the browser. So, now i need to figure out why the gh cli token does not have the same permissions as my user.

Gitlab API to upload whole project from local file system

I need to upload using API whole project folder which contains multiple text files, image files, source files, etc.
I had gone through Gitlab Commits and upload API's but none of them matches requirement.
Is there any API to upload full project folder form API?
I used below API to create a project
POST : https://repo.**.com/api/v4/projects
{
"name": "test",
"default_branch": "master"
}
You can use repository files API to create files.
https://docs.gitlab.com/ee/api/repository_files.html#create-new-file-in-repository
POST /projects/:id/repository/files/:file_path
curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' --header "Content-Type: application/json" \
--data '{"branch": "master", "author_email": "author#example.com", "author_name": "Firstname Lastname", \
"content": "some content", "commit_message": "create a new file"}' \
'https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb'
After creating new project with gitlab api, e.g.:
curl --fail -d '{"name":"test","default_branch": "master"}' -H "Content-Type: application/json" -X POST https://repo.**.com/api/v4/projects?access_token=<your access token> --output project_details.json
You could assume your new project url based on post data name -> https://repo.**.com/<gitlab user or gitlab group>/test.git or you could parse response json with some tool, e.g. python:
http_url_to_repo=$(python -c "import json; print(json.load(open('project_details.json'))['http_url_to_repo'])")
Then you could follow new project instruction for existing folder:
#Push an existing folder
cd existing_folder
git init
git remote add origin $http_url_to_repo
git add .
git commit -m "Initial commit"
git push -u origin master
Where existing_folder is folder with your source files which you want to upload as a commit to new gitlab project.
Had the same problem and ended up walking the folder like this:
import os
for (dirpath, dirnames, files) in os.walk(my_folder):
for file in files:
# do the api call to commit for dirpath, file
It works nicely...

wget a raw file from Github from a private repo

I am trying to get a raw file from github private project using wget. Usually if my project is public it is very simple
For Public Repo This is my repo url (you don't have to click on it to answer this question)
https://github.com/samirtendulkar/profile_rest_api/blob/master/deploy/server_setup.sh
I click raw
After I lick raw My URL looks like this
https://raw.githubusercontent.com/samirtendulkar/profile_rest_api/master/deploy/server_setup.sh (Notice only the word "raw" is added to the URL)
which is awesome I then do
ubuntu#ip-172-31-39-47:~$ wget https://raw.githubusercontent.com/samirtendulkar/profile_rest_api/master/deploy/server_setup.sh
when I do ls it shows that the file has been downloaded
ubuntu#ip-172-31-39-47:~$ ls
'server_setup.sh'
For a Private repo The raw file comes with a token
https://github.com/samirtendulkar/my_project/blob/master/deploy/server_setup.sh
So far so good Now when I click Raw (see image above) My URL changes and has a token in it along with the "raw" prefix
https://raw.githubusercontent.com/samirtendulkar/my_project/master/deploy/server_setup.sh?token=AkSv7SycSHacUNlSEZamo6hpMAI6ZhsLks5b4uFuwA%3D%3D
The url has these extra parameters ?token=AkSv7SycSHacUNlSEZamo6hpMAI6ZhsLks5b4uFuwA%3D%3D
My wget does not work. How Do I fix this issue. By the way when I say it does not work I mean instead of the ls showing
ubuntu#ip-172-31-39-47:~$ ls
'server_setup.sh'
It shows as below
which is not making me run futher commands like
ubuntu#ip-172-31-39-47:~$ chmod +x server_setup.sh
and
ubuntu#ip-172-31-39-47:~$ sudo ./server_setup.sh
which I need to get the project on to AWS
The token is from the Personal Access Tokens section that you can find the details in Github.
With Personal Access Tokens, you can create one and pick the first option "repo" to get access control over the private repos for the token.
following line solved my problem which was not being able to download the file.
Hope this will help
wget --header 'Authorization: token PERSONAL_ACCESS_TOKEN_HERE' https://raw.githubusercontent.com/repoOwner/repoName/master/folder/filename
You can use wget's -O option when you're downloading just one file at a time:
wget -O server_setup.sh https://raw.githubusercontent.com/samirtendulkar/my_project/master/deploy/server_setup.sh?token=AkSv7SycSHacUNlSEZamo6hpMAI6ZhsLks5b4uFuwA%3D%3D
The downside is that you have to know the output file name, but I think that's OK if I understand your question well.

IBM Object Storage-How to make a File or a Container public accesible one?

I have uploaded a file from local to the IBM Storage container using Node js
In the response I have got the baseResourceUrl (https://dal.objectstorage.open.softlayer.com:443/v1/AUTH_bf7
d89eb565f4c638d7b6f7b15e73538/testContainer/test.png)
The URL is not accesible in the browser.It says Unauthorized
In my usecase I need to access publicly anywhere.Is that possible? If possible how to give public access to files.
I am new to IBM Bluemix environment and Object Storage service.Kindly share some solutions
After uploading the file you'll need to modify the ACLs. You'll end up creating a Read ACL for the container: testContainer.
I was able to test this using the Swift CLI, however you can use the curl examples to figure out what you'll need to write using Node.
Swift Example (assumes you have ENV variables exported):
swift post testContainer --read-acl ".r:*,.rlistings"
Curl with AUTH_TOKEN:
$ curl -X PUT -i \
> -H "X-Auth-Token: AUTH_TOKEN" \
> -H "X-Container-Read: .r:*,.rlistings" \
> https://dal.objectstorage.open.softlayer.com/v1/AUTH_bf7d89eb565f4c638d7b6f7b15e73538/testContainer
https://www.swiftstack.com/docs/cookbooks/swift_usage/container_acl.html#setting-container-read-acl
If we go back to the swift client we can view the current ACLs in place:
swift stat -v testContainer
Which will return data about the container:
Read ACL: .r:*,.rlistings
Write ACL:

Resources