I am new to Azure Data Factory and i am trying to copy data from JIRA to Azure Storage using Data Factory.
I am trying the JIRA REST API to fetch the data, but unfortunately i am struggling with pagination. Can Someone help me how to use the pagination in this case.
Request URL:
https://jira.abc.com/rest/api/3/search?jql=&startAt=0&maxResult=10
Sample response from JIRA:
{
"startAt" : 0,
"maxResults" : 10,
"total": 200,
"isLast": false,
"values": [
{ /* result 0 */ },
{ /* result 1 */ },
{ /* result 2 */ }
]
}
Click on the Copy Data activity (the block) in your pipeline
Click on
the Source tab
Scroll down to Pagination rules and click the plus
sign to add a rule
As highlighted below, put AbsoluteURL in the first box
and then, for JIRA it appears (according to their docs) that you need to put $._links.next instead of $.next_page in my screenshot. (This is the reference to the part of the JSON doc returned by JIRA containing the URL of the next page.)
Please comment if it works for you.
Refs:
MS data factory docs
JIRA API docs
Related
I'm using Contentful API to get the details of all the entries in a Space, is there a way to fetch the name of 'linked entry/entries' to a particular entry? In this case, i'm fetching all the entries with the content_type = HorizontalImageCard and then that each entry has link to other entries(could be a link to an Asset or link to different Contentul entry), and i would like to get the name of such entries, attaching the screenshot of what i'm looking for, in the screenshot there's a Linked Entry called 'Jennifer Test', how do i fetch that name?
let fetchRes = fetch("https://preview.contentful.com/spaces/{space_id}/environments/ca/entries/{entry_id}?access_token=vOKxCDyhU8gLUsU4FR5tgdwQVe3arwfJzFTCloi2sjM&content_type=HorizontalImageCard");
Did you try with the GraphQL API? I didn't get a chance to try it out with the REST API, but I was able to get the result with the GraphQL API. Here's an example query:
{
userCollection(preview:true) {
items {
name
linkedFrom {
blogArticleCollection {
items {
title
}
}
}
}
}
}
The User content type is linked in the Blog Article content type. This method allows me to list all those references. Hope this helps :)
I am working on creating a workbook that provides an umbrella view over multiple app insight instances. Our solution has many microservices (Azure functions) each having its own app insight instance. Aim of this workbook is to provide a health status for the whole app by surfacing up errors across app insight instances in to a single view.
I have used the "Failure Analysis" template to set this up. User is able to select different app insight instances at the top and the views will filter based on that. There's a view that shows exception counts with trends like this. Each error may belong to different App Insight instances.
When you click on a line item all instances of that error will be shown in a following view like this
I use the following query to load it
let row = dynamic({Row});
let req = requests
| where '{Row}' == '{}' or (row.Kind == 'Application' and row.Id
== appName) or (row.Kind == 'Request' and row.Id ==
strcat(appName, "::", name))
| where success == "False";
let errors = exceptions
| where appName == appName
| where timestamp between({TimeRange:start}..{TimeRange:end});
errors
| join req on operation_Id
| project operation_Id, itemId, timestamp,requestName=name,
exception=type, method, outerMessage, innermostMessage,
details, appName
As mentioned in this question Get exception details from a Azure Monitor Workbook, the itemId is available and I try to link it to the "Exception Details" view. Please note that these errors can come from one of many App Insight instances based on what gets selected at the previous view.
I have configured the item Id and appName columns as follows using Link renderer and Automatic renderer.
However the link always directs to one specific app insight instance (Not the one associated with the error) and hence the error won't get loaded. Is it possible to load the "Detail Views" across App Insight instances using this technique? If not what could be other avenues?
When the application insights is workspace based, each telemetry item has a field _ResourceId that contains a link to the resource, for example
/subscriptions/c8vfbeab-a5a67-4272-aa6e-4c9f4142e962/resourcegroups/rg-my-resource-group/providers/microsoft.insights/components/my-ai-resource
You can use this a part of the url to create a deep link to the details page of the Application Insights resource of the telemetry with the specified item id. Take this query for example:
exceptions
| take 1
| extend portalUrl = strcat("https://portal.azure.com/#blade/AppInsightsExtension/BladeRedirect/BladeName/searchV1/ResourceId/", url_encode(_ResourceId), "/BladeInputs/%7B%22tables%22%3A%5B%22availabilityResults%22%2C%22requests%22%2C%22exceptions%22%2C%22pageViews%22%2C%22traces%22%2C%22customEvents%22%2C%22dependencies%22%5D%2C%22timeContextWhereClause%22%3A%22%7C%20where%20timestamp%20%3E%20datetime(%5C%222022-02-12T12%3A55%3A02.739Z%5C%22)%20and%20timestamp%20%3C%20datetime(%5C%222022-03-14T12%3A55%3A02.739Z%5C%22)%22%2C%22filterWhereClause%22%3A%22%7C%20where%20*%20has%20%5C%22a1a20ad1a12ff348a852288a4d9953a5%5C%22%7C%20order%20by%20timestamp%20desc%22%2C%22originalParams%22%3A%7B%22eventTypes%22%3A%5B%7B%22value%22%3A%22availabilityResult%22%2C%22tableName%22%3A%22availabilityResults%22%2C%22label%22%3A%22Availability%22%7D%2C%7B%22value%22%3A%22request%22%2C%22tableName%22%3A%22requests%22%2C%22label%22%3A%22Request%22%7D%2C%7B%22value%22%3A%22exception%22%2C%22tableName%22%3A%22exceptions%22%2C%22label%22%3A%22Exception%22%7D%2C%7B%22value%22%3A%22pageView%22%2C%22tableName%22%3A%22pageViews%22%2C%22label%22%3A%22Page%20View%22%7D%2C%7B%22value%22%3A%22trace%22%2C%22tableName%22%3A%22traces%22%2C%22label%22%3A%22Trace%22%7D%2C%7B%22value%22%3A%22customEvent%22%2C%22tableName%22%3A%22customEvents%22%2C%22label%22%3A%22Custom%20Event%22%7D%2C%7B%22value%22%3A%22dependency%22%2C%22tableName%22%3A%22dependencies%22%2C%22label%22%3A%22Dependency%22%7D%5D%2C%22timeContext%22%3A%7B%22durationMs%22%3A2592000000%7D%2C%22filter%22%3A%5B%5D%2C%22searchPhrase%22%3A%7B%22originalPhrase%22%3A%22", itemId,"%22%2C%22_tokens%22%3A%5B%7B%22conjunction%22%3A%22and%22%2C%22value%22%3A%22a1a20ad1a12ff348a852288a4d9953a5%22%2C%22isNot%22%3Afalse%2C%22kql%22%3A%22%20*%20has%20%5C%22a1a20ad1a12ff348a852288a4d9953a5%5C%22%22%7D%5D%7D%2C%22sort%22%3A%22desc%22%7D%7D")
| project timestamp, problemId, itemId, portalUrl
If you create a workbook and render a table based on the query above you have to modify the itemId column to be a link like this:
Clicking the itemId column it should open the correct Application Insights resource details page:
Now, I hope this give you enough clues to extend your own queries by including the deep link url in your query output.
For completeness, this is the full workbook Gallery Template:
{
"version": "Notebook/1.0",
"items": [
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "exceptions\n| take 1\n| extend portalUrl = strcat(\"https://portal.azure.com/#blade/AppInsightsExtension/BladeRedirect/BladeName/searchV1/ResourceId/\", url_encode(_ResourceId), \"/BladeInputs/%7B%22tables%22%3A%5B%22availabilityResults%22%2C%22requests%22%2C%22exceptions%22%2C%22pageViews%22%2C%22traces%22%2C%22customEvents%22%2C%22dependencies%22%5D%2C%22timeContextWhereClause%22%3A%22%7C%20where%20timestamp%20%3E%20datetime(%5C%222022-02-12T12%3A55%3A02.739Z%5C%22)%20and%20timestamp%20%3C%20datetime(%5C%222022-03-14T12%3A55%3A02.739Z%5C%22)%22%2C%22filterWhereClause%22%3A%22%7C%20where%20*%20has%20%5C%22a1a20ad1a12ff348a852288a4d9953a5%5C%22%7C%20order%20by%20timestamp%20desc%22%2C%22originalParams%22%3A%7B%22eventTypes%22%3A%5B%7B%22value%22%3A%22availabilityResult%22%2C%22tableName%22%3A%22availabilityResults%22%2C%22label%22%3A%22Availability%22%7D%2C%7B%22value%22%3A%22request%22%2C%22tableName%22%3A%22requests%22%2C%22label%22%3A%22Request%22%7D%2C%7B%22value%22%3A%22exception%22%2C%22tableName%22%3A%22exceptions%22%2C%22label%22%3A%22Exception%22%7D%2C%7B%22value%22%3A%22pageView%22%2C%22tableName%22%3A%22pageViews%22%2C%22label%22%3A%22Page%20View%22%7D%2C%7B%22value%22%3A%22trace%22%2C%22tableName%22%3A%22traces%22%2C%22label%22%3A%22Trace%22%7D%2C%7B%22value%22%3A%22customEvent%22%2C%22tableName%22%3A%22customEvents%22%2C%22label%22%3A%22Custom%20Event%22%7D%2C%7B%22value%22%3A%22dependency%22%2C%22tableName%22%3A%22dependencies%22%2C%22label%22%3A%22Dependency%22%7D%5D%2C%22timeContext%22%3A%7B%22durationMs%22%3A2592000000%7D%2C%22filter%22%3A%5B%5D%2C%22searchPhrase%22%3A%7B%22originalPhrase%22%3A%22\", itemId,\"%22%2C%22_tokens%22%3A%5B%7B%22conjunction%22%3A%22and%22%2C%22value%22%3A%22a1a20ad1a12ff348a852288a4d9953a5%22%2C%22isNot%22%3Afalse%2C%22kql%22%3A%22%20*%20has%20%5C%22a1a20ad1a12ff348a852288a4d9953a5%5C%22%22%7D%5D%7D%2C%22sort%22%3A%22desc%22%7D%7D\")\n| project timestamp, problemId, itemId, portalUrl",
"size": 1,
"timeContext": {
"durationMs": 86400000
},
"queryType": 0,
"resourceType": "microsoft.insights/components",
"gridSettings": {
"formatters": [
{
"columnMatch": "problemId",
"formatter": 1,
"formatOptions": {
"linkColumn": "portalUrl",
"linkTarget": "Url"
}
}
]
}
},
"name": "query - 2"
}
],
"fallbackResourceIds": [
"/subscriptions/4547474-1a67-4272-aa6e-4c9f4142e269/resourceGroups/rg-resourcegroup-prod/providers/microsoft.insights/components/appi-demo-prod"
],
"$schema": "https://github.com/Microsoft/Application-Insights-Workbooks/blob/master/schema/workbook.json"
}
It looks like from a quick perusal of the code, that if you also have an appName column in the row where itemId is, it will try to look in the resource list used in the query to try to find a resource with that same name, and if not, just take the first one it can find?
but i see that you have appName there, but i'm not sure the rest of the configuration from your step? are you also using all of the resources in the query?
I'm working on an automation task where I need a group to have a set of permissions on Repos, Pipelines, and releases, etc. I'm looking for a Rest API that can manage the permissions for this group.
For Example:
At Cross repo policies, how do I manage/set the permissions for the group "PROJECT ADMINISTRATORS" to allow the "Bypass policies when pushing", "Bypass policies when pushing", etc using a Rest API.
Thank you in Advance.
Based on your requirement, you could use the Rest API: Access Control Entries - Set Access Control Entries
POST https://dev.azure.com/{organization}/_apis/accesscontrolentries/{securityNamespaceId}?api-version=6.0
Request Body:
{
"token": "repoV2/{ProjectID}/{RepoID(If you want to set the permission for a single repo)}",
"merge": true,
"accessControlEntries": [
{
"descriptor": "Microsoft.TeamFoundation.Identity;S-....",
"allow": 32768,
"deny": 0,
"extendedinfo": {}
}
]
}
You can get the parameter values needed in the Rest API through the following methods：
securityNamespaceId：
GET https://dev.azure.com/{OrganizationName}/_apis/securitynamespaces?api-version=6.0
In the Response Body: you could search for the Git Repositories.
Then you could get the namespaceid and Parameter values corresponding to permissions.
For example:
To get the Group Identity(S-...), there is no directly Rest API to get it. You use the following method to get it:
1.Get the descriptor:
GET https://vssps.dev.azure.com/{org name}/_apis/graph/users?api-version=5.1-preview.1
2.Using the following C# code to convert it:
public static string Base64Decode(string base64EncodedData)
{
var lengthMod4 = base64EncodedData.Length % 4;
if (lengthMod4 != 0)
{
//fix Invalid length for a Base-64 char array or string
base64EncodedData += new string('=', 4 - lengthMod4);
}
var base64EncodedBytes = System.Convert.FromBase64String(base64EncodedData);
return System.Text.Encoding.UTF8.GetString(base64EncodedBytes);
}
public static string Base64Encode(string plainText)
{
var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
return System.Convert.ToBase64String(plainTextBytes);
}
Here has a blog which written by our Azure Identity Team engineer, you could refer to it for more detailed information.
In addition, you can also obtain the values of all parameters directly through the browser F12 record.
If you look at: http://developer.github.com/v3/pulls/ it shows you how to get pull requests for a given repository.
How do we get "my pull requests" from the GitHub API similar to the data displayed on the GitHub dashboard?
I asked Github directly. A rep told me to use the search endpoint. Search for issues owned by you that are open and of type pr.
https://api.github.com/search/issues?q=state%3Aopen+author%3Adavidxia+type%3Apr
If you're using a python client lib like Pygithub you can do
issues = gh.search_issues('', state='open', author='davidxia', type='pr')
You can also use GraphQL API v4 to get all your pull requests :
{
user(login: "bertrandmartel") {
pullRequests(first: 100, states: OPEN) {
totalCount
nodes {
createdAt
number
title
}
pageInfo {
hasNextPage
endCursor
}
}
}
}
Try it in the explorer
or using viewer :
{
viewer {
pullRequests(first: 100, states: OPEN) {
totalCount
nodes {
createdAt
number
title
}
pageInfo {
hasNextPage
endCursor
}
}
}
}
First you have to realize that you must authenticate using either Basic Authentication or a token. Next you have to realize that there is no simple way to do this so you will have to be clever.
To be specific, if you probe https://api.github.com/issues, you'll notice that the issues there have a hash called pull_request which should have 3 URLs: html, diff, and patch. All three will be non-null if the issue is also a Pull Request. (Pro-tip: They're the same thing as far as GitHub is concerned…sort of.)
If you iterate over your issues and filter for ones where those attributes are not null, then you'll have your pull requests.
I'm displaying a list of articles in a page that are fetched using the Ember Data RESTAdapter. I need to implement a bootstrap'esque paginator (see: http://twitter.github.com/bootstrap/components.html#pagination) and cant seem to find a sane pattern for returning pagination data such as, page count, article count, current page, within a single request.
For example, I'd like the API to return something like:
{
articles: [{...}, {...}],
page: 3,
article_count: 4525,
per_page: 20
}
One idea was to add an App.Paginator DS.Model so the response could look like:
{
articles: [{...}, {...}],
paginator: {
page: 3,
article_count: 4525,
per_page: 20
}
}
But this seems like overkill to hack together for something so trivial. Has anyone solved this problem or found a particular pattern they like? Is there a simple way to manage the RESTAdapter mappings to account for scenarios such as this?
Try to use Ember Pagination Support Mixin and provide your own implementation of the following method. Instead of loading all the content, you can fetch the required content when the user is navigating the pages. All what you need initially is the total account of your records.
didRequestRange: function(rangeStart, rangeStop) {
var content = this.get('fullContent').slice(rangeStart, rangeStop);
this.replace(0, this.get('length'), content);
}
With ember-data-beta3 you can pass a meta-property in your result. The default RESTSerializer looks for that property and stores it.
You can access the meta-data like this:
var meta = this.get("store").metadataFor("post");
If you are not able to change the JSON returned from the server you could override the extractMeta-hook on the ApplicationSerializer (or any other Model-specific serializer).
App.ApplicationSerializer = DS.RESTSerializer.extend({
extractMeta: function(store, type, payload) {
if (payload && payload.total) {
store.metaForType(type, { total: payload.total }); // sets the metadata for "post"
delete payload.total; // keeps ember data from trying to parse "total" as a record
}
}
});
Read more about meta-data here