I am currently trying to read through the (GitHub API)[http://developer.github.com/v3/repos/contents/] to figure out how I can programmatically retrieve all versions of a specific file in a single repository.
I see that one can get the list of commits, and the current version of a single file easily. But, is there a way to list all the commits relevant for a specific file and then iterate through all the versions of that file?
To get the list of commits relevant for a specific file, use this API endpoint and specify the path parameter:
GET https://api.github.com/repos/:owner/:repo/commits?path=FILE_PATH
You'll get back an array of commit objects, each of which has a sha attribute.
Now that you have all the commit SHAs, you can fetch all the different versions of the file using this API endpoint and by specifying the ref query parameter to set the SHA. So, for each commit SHA, make a request to:
GET https://api.github.com/repos/:owner/:repo/contents/:FILE_PATH?ref=SHA
and read the content attribute. Notice that the content is Base64 encoded, but you can also request a raw version by setting the relevant Accept HTTP header.
Related
I need to pull BRANCH-1 to BRANCH-2 using the GitLab API. I don't see any similar features in the API. Merge Request only allows you to merge MR with the ability to compare SHA with HEAD. Rebase has no parameters for selecting a branch. Is there any way to do this?
If you're trying to overwrite BRANCH-2 so it exactly matches BRANCH-1, then you can do that by deleting BRANCH-2 and then recreating BRANCH-2 from BRANCH-1.
This is obviously a destructive operation, so don't do it if BRANCH-2 has unique changes that you need to keep. With that warning in mind, here's how you do it.
Delete BRANCH-2 with this API request:
DELETE /projects/:id/repository/branches/:branch
where :id is the ID or URL-encoded path of the project and :branch is the name of BRANCH-2.
Then recreate BRANCH-2 with this API request:
POST /projects/:id/repository/branches?branch=:branch&ref=:ref
where :id is the ID or URL-encoded path of the project, :branch is the name of BRANCH-2, and :ref is the name of BRANCH-1.
I was working on uploading and downloading a file to S3 bucket using pre-signed URLs.I came across these two methods generate_presigned_url('put_object') and generate_presigned_post.
What is the difference between these two methods?
# upload a file to a bucket with generate_presigned_url with put object
s3_client.generate_presigned_url('put_object', Params= {'Bucket': "BUCKET_NAME",
"Key":"OBJECT_KEY"},
ExpiresIn=3600)
# upload a file to a bucket using presigned post
s3_client.generate_presigned_post(Bucket="BUCKET_NAME", Key="OBJECT_PATH",
ExpiresIn=3600)
Could someone please explain the difference between both?
If we have generate_presigned_post why was there a generate_presigned_url method with put_object for uploading in the first place.
Note : I know that generate_presigned_post is the recommended method for file uploads and I have used the same. However, there is no clear documentation on the difference between these methods.
This is an extended version of #jellycsc's comment. I had posted the same query to aws support as well. I got the below answer from them.
More detailed explanation is given here
Posting here as it could be useful for someone.
What is the difference between these two methods?
generate_presigned_post() is more powerful because of the POST Policy feature. The POST Policy is simply conditions you set when creating the presigned POST. Using it, you can allow certain MIME types and file extensions, allow multiple files to be uploaded with a given prefix, restrict the file size, and more, which is not possible in generate_presigned_url()
Please note that both the methods can be used to fulfill the same goal, i.e provide controlled way for users to upload files directly to S3 buckets. The process is also the same for both as the backend needs to sign the request after validating that the user is authorized then the browser sends the file directly to S3.
Differences:
URLStructure:
PUT URLs encode everything in the URL itself as there is nothing else communicated back to the client. This means fewer variables can be customized.
POST URLs use multiple fields for different kinds of information. The signing algorithm returns a list of fields along with the URL itself and the client must send those to S3 as well while accessing the presigned URL.
While PUT URLs provide a destination to upload files without any other required parts, POST URLs are made for forms that can send multiple fields. However, their usage is not limited to forms.
Content Type
For PUT URLs the signing must be done for a specific content type. That means you either hardcode a content type on the backend, for example, application/xml if you want to allow users to upload XML documents, or the client must send the desired content type as part of the signing request.
For POST URLs the policy supports a prefix constraint as well as an exact match.
Content-Length:
In case of PUT URLs, you have no control over the size of the uploaded file.
For POST URLs you can set an allowed range in the policy.
Sample presigned post in python:
response = s3_client.generate_presigned_post(Bucket="BUCKET_NAME",
Key="S3KEY",
Fields={"Content-Type": "image/jpg"},
Conditions=["starts-with", "$Content-Type", "image/"],
ExpiresIn=3600)
Is it possible to somehow get information about whole repository using one request to github api or we have to recursively asking API for specific level?
According to this request: https://api.github.com/repos/microsoft/terminal/contents?ref=master
we are only getting one level.
You can get the tree recursively using :
/repos/:owner/:repo/git/trees/:tree_sha?recursive=1
In your case it would be :
https://api.github.com/repos/microsoft/terminal/git/trees/master?recursive=true
You can make a request to here to see details about your branch:
https://api.github.com/repos/{owner}/{repo}/branches/{default_branch}
In the response you will see "commit" field and inside of it there is "sha". Take that one put it in this request:
https://api.github.com/repos/{owner}/{repo}/git/trees/YOUR_SHA_GOES_HERE?recursive=1
You will receive every file under your repo like this.
Cheers!
It's possible to generate a link to raw content of the file in GitHub, is it possible to do with VSTS/DevOps?
Even after reading the existing answers, I still struggled with this a bit, so I wanted to leave a bit more of a thorough response.
As others have said, the pattern is (query split onto separate lines for ease of reading):
https://dev.azure.com/{{organization}}/{{project}}/_apis/sourceProviders/{{providerName}}/filecontents
?repository={{repository}}
&path={{path}}
&commitOrBranch={{commitOrBranch}}
&api-version=5.0-preview.1
But how do you find the values for these variables? If you go into your Azure DevOps, choose Repos > Files from the left navigation, and select a particular file, your current url should look something like this:
https://dev.azure.com/{{organization}}/{{project}}/_git/{{repository}}?path=%2Fpackage.json
You should use those values for organization, project, and repository. For path, you'll see an HTTP encoded version of the unix file path. %2F is the HTTP encoding for /, so that path is actually just /package.json (a tool like Postman will do that encoding for you).
Commit or branch is pretty self explanatory; you either know what you want for this value or you should use master. I have "hard-coded" the api version in the above url because that's what the documentation currently points to.
For the last variable, you need providerName. In short, you should probably use TfsGit. I got this value from looking through the list of source providers and looking for one with a value of true for supportedCapabilities.queryFileContents.
However, if you just request this URL you'll get a "203 Non-Authoritative Information" response back because you still need to authenticate yourself. Referring again to the same documentation, it says to use Basic auth with any value for the username and a personal access token for the password. You can create a personal access token at https://dev.azure.com/{{organization}}/_usersSettings/tokens; ensure that it has the Token Administration - Read & Manage permission.
If you're unfamiliar with this sort of thing, again Postman is super helpful with getting these requests working before you get into the code.
So if you have a repository with a src directory at the root, and you're trying to get the file contents of src/package.json, your URL should look something like:
https://dev.azure.com/{{organization}}/{{project}}/_apis/sourceProviders/TfsGit/filecontents?repository={{repository}}&commitOrBranch=master&api-version={{api-version}}&path=src%2Fpackage.json
And don't forget the basic auth!
Sure, here's the rests call needed:
GET https://feeds.dev.azure.com/{organization}/_apis/packaging/Feeds/{feedId}/packages/{packageId}?includeAllVersions={includeAllVersions}&includeUrls={includeUrls}&isListed={isListed}&isRelease={isRelease}&includeDeleted={includeDeleted}&includeDescription={includeDescription}&api-version=5.0-preview.1
https://learn.microsoft.com/en-us/rest/api/azure/devops/artifacts/artifact%20%20details/get%20package?view=azure-devops-rest-5.0#package
I was able to get the raw contents of a file using this URL.
GET https://dev.azure.com/{organization}/{project}/_apis/sourceProviders/{providerName}/filecontents?serviceEndpointId={serviceEndpointId}&repository={repository}&commitOrBranch={commitOrBranch}&path={path}&api-version=5.0-preview.1
I got this from here.
https://learn.microsoft.com/en-us/rest/api/azure/devops/build/source%20providers/get%20file%20contents?view=azure-devops-rest-5.0
You can obtain the raw URL using chrome.
Turn on Developer tools and view the Network tab.
Navigate to view the required file in the DevOps portal (Content panel). Once the content view is visible check the network tab again and find the URL which starts with "Items?Path", this is json response which contains the required "url:" element.
Drag the filename from the attachments windows and drop it in to any other MS application to get the raw URL or linked filename.
Most answers address this well, but in context of a public repo with anonymous access the api is different. Here is the one that works in such a scenario:
https://dev.azure.com/{{your_user_name}}/{{project_name}}/_apis/git/repositories/{{repo_name_encoded}}/items?scopePath={{path_to_your_file}}&api-version=6.0
This is the exact equivalent of the "raw" url provided by Github.
Another way that may be helpful if you want to quickly get the raw URL for a specific file that you are browsing:
install the browser extension named "Undisposition"
from the dot menu (top right) choose "Download": the file will open in a new browser tab from which you can copy the URL
(edit: unfortunately this will only work for file types that the browser knows how to open, otherwise it will still offer to download it...)
I am fairly new to this and had an issue accessing a raw file in an Azure DevOps Repo. It's straightforward in Github.
I wanted to download a file in CMD and BASH using Curl.
First I browsed to the file contents in the browser make a note of the bold sections:
https://dev.azure.com/**myOrg**/_git/**myProjectName**?path=%2F**MyFileName.ps1**
I then constructed the URL similar to what #Zach posted above.
https://dev.azure.com/**myOrg**/**myProjectName**/_apis/sourceProviders/TfsGit/filecontents?repository=**myProjectName**&commitOrBranch=**master**&api-version=5.0-preview.1&path=%2F**MyFileName.ps1**
Now when I paste the above URL in the browser it displays the content in RAW form similar to GitHub.
The difference was I had to setup a PAT (Personal Access Token) in My Azure DevOps account then authenticate the URL in DOS/BASH example below:
curl -u "<username>:<password>" "https://dev.azure.com/myOrg/myProjectName/_apis/sourceProviders/TfsGit/filecontents?repository=myProjectName&commitOrBranch=master&api-version=5.0-preview.1&path=%2FMyFileName.ps1" -# -L -o MyFileName.ps1
I've created a fork of a github repository via the github API. Now, later on, I want to pull any updates from the origin repository into the fork. This should always be a fast-forward in my use case. I have read access to the origin repository and read-write to the fork.
I thought of maybe creating a Pull Request then accepting (both of which you can do via the API) but this creates noise (Pull Requests being created and destroyed) and just doesn't seem right.
Is there any way to do this via the API?
I don't have the inside scoop on this, so this might be a miss-feature that will be removed at some point. Until then:
Github makes available all commits across (I assume) the entire fork network; So APIs that accept commit hashes will be happy to work on hashes from the upstream, or across other forks (This is explicitly documented for repos/commits/compare and creating a pull requst).
So there are a couple of ways to update via APIs only:
Using Git data api: This will usually be the best option, if you don't change your fork's master.
Get upstream ref /repos/upstream/repo/git/refs/heads/master, and get the hash from it
Update your fork PATCH /repos/my/repo/git/refs/heads/master with the same hash.
Using a higher-level merge api: This will create a merge commit, which some people like.
Get the upstream ref like before
Create a merge to branch master in your repo.
Pull-request to yourself and merge it via api: This will end up creating not only a merge commit, but a PR as well.
Create PR: POST to /repos/your/repo/pulls with head = "upstream:master"
Get the PR url from the response,
Merge it: PUT to /repos/your/repo/pulls/number/merge
It's possible that the "upstream:master" notation would also work for options 1 & 2, saving an API call.
Not possible currently, but I've gone ahead and added that to our API wishlist. :)
This that work for me, because I needed update from upstream but without a merge request commit. My ref is master.
Create a pull request POST /repos/:myUsername/:myRepo/pulls
INPUT: {title, head: 'ownerFromUpStream:master', base: 'master', ...}
Get sha from pull request (ex. response.data.head.sha)
PATCH /repos/:myUsername/:myRepo/git/refs/master
PARAMS: {sha: shaFromPullRequest}
DOC.
Update ref
Create pull request
This is now possible in the GitHub API; documentation here, and announcement here.
In summary, make a POST request to /repos/{owner}/{repo}/merge-upstream with the proper authentication and the payload of { "branch": "branch-name" }.