I'm using GitHub API and PowerShell to try to do a query on my repos to filter the ones that contains topics with value equals "cloud".
The code I'm doing is like this:
$repoNames = gh api -X GET "/orgs/$org/repos?type=private" --paginate --jq '.[].name' | sort
foreach ($name in $repoNames) {
gh api -X GET "/repos/$org/$name/topics"
}
Is there a way to do the query to filter the repos by topic on the first gh cmd?
Or is there a way to filter topics by value in the second gh cmd?
I tried this code and nothing works to query topics so far.
Related
I'm trying to visualize the browser statistics of our app hosted in Azure.
For that I'm using the nginx logs and run an Azure Log Analytics query like this:
ContainerLog
| where LogEntrySource == "stdout" and LogEntry has "nginx"
| extend logEntry=parse_json(LogEntry)
| extend userAgent=parse_user_agent(logEntry.nginx.http_user_agent, "browser")
| extend browser=parse_json(userAgent)
| summarize count=count() by tostring(browser.Browser.Family)
| sort by ['count']
| render piechart with (legend=hidden)
Then I'm getting this diagram, which is exactly what I want:
But the query is very very slow. If I set the time range to more than just the last few hours it takes several minutes or doesn't work at all.
My solution is to use a search job like this:
ContainerLog
| where LogEntrySource == "stdout" and LogEntry has "nginx"
| extend d=parse_json(LogEntry)
| extend user_agent=parse_user_agent(d.nginx.http_user_agent, "browser")
| extend browser=parse_json(user_agent)
It creates a new table BrowserStats_SRCH on which I can do this search query:
BrowserStats_SRCH
| summarize count=count() by tostring(browser.Browser.Family)
| sort by ['count']
| render piechart with (legend=hidden)
This is much faster now and only takes some seconds.
But my problem is, how can I keep this up-to-date? Preferably this search job would run once a day automatically and refreshed the BrowserStats_SRCH table so that new queries on that table run always on the most recent logs. Is this possible? Right now I can't even trigger the search job manually again, because then I get the error "A destination table with this name already exists".
In the end I would like to have a deeplink to the pie chart with the browser stats without the need to do any further click. Any help would be appreciated.
But my problem is, how can I keep this up-to-date? Preferably this search job would run once a day automatically and refreshed the BrowserStats_SRCH table so that new queries on that table run always on the most recent logs. Is this possible?
You can leverage the api to create a search job. Then use a timer triggered azure function or logic app to call that api on a schedule.
PUT https://management.azure.com/subscriptions/00000000-0000-0000-0000-00000000000/resourcegroups/testRG/providers/Microsoft.OperationalInsights/workspaces/testWS/tables/Syslog_suspected_SRCH?api-version=2021-12-01-preview
with a request body containing the query
{
"properties": {
"searchResults": {
"query": "Syslog | where * has 'suspected.exe'",
"limit": 1000,
"startSearchTime": "2020-01-01T00:00:00Z",
"endSearchTime": "2020-01-31T00:00:00Z"
}
}
}
Or you can use the Azure CLI:
az monitor log-analytics workspace table search-job create --subscription ContosoSID --resource-group ContosoRG --workspace-name ContosoWorkspace --name HeartbeatByIp_SRCH --search-query 'Heartbeat | where ComputerIP has "00.000.00.000"' --limit 1500 --start-search-time "2022-01-01T00:00:00.000Z" --end-search-time "2022-01-08T00:00:00.000Z" --no-wait
Right now I can't even trigger the search job manually again, because then I get the error "A destination table with this name already exists".
Before you start the job as described above, remove the old result table using an api call:
DELETE https://management.azure.com/subscriptions/{subscriptionId}/resourcegroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/tables/{tableName}?api-version=2021-12-01-preview
Optionally, you could check the status of the job using this api before you delete it to make sure it is not InProgress or Deleting
On the gcloud cli, when listing the pubsub subscriptions of a project, it is possible to filter results by using the --filter flag. Here is an example:
gcloud --project=my-project pubsub subscriptions list --filter=my-filter-string --format='value(name)'
I did not manage to find out how to do this with the python library and its list_subscription method.
It seems to only basically accept a project string and to return all subscriptions in the project. This means I would need to get all the subscriptions in the project and then loop through them to filter them, as follows:
from google.cloud import pubsub_v1
subscriber_client = pubsub_v1.SubscriberClient()
filter = "my-filter-string"
with subscriber_client:
page_result = subscriber_client.list_subscriptions(
project="projects/my-project",
)
filtered_subscriptions = [
subscription.name
for subscription in page_result
if filter in subscription.name.split("/")[-1]
]
for subscription_name in filtered_subscriptions:
print(subscription_name)
Is there a more efficient way to do that ?
I have been trying to do this with the metadata: Sequence[Tuple[str, str]] argument on the method, but could not find examples of how to do it.
Neither the REST nor RPC API provide a way to filter on the server side, so no there is no more efficient way to do this.
I imagine the gcloud code to do the filter is conceptually similar to what you wrote.
I am building a new corpus using Watson Concept-Insights. I've created about 100 documents so far using nodeJS. If I use curl to list the documents, I can find all of them. However when I nodeJS to list the same set of documents, it consistently ignores the limit value and returns the default of 20 documents. Help!!
Essential code follows (account key replaced with 'myAccount'):
var watson = require('watson-developer-cloud');
var concept_insights = watson.concept_insights({ yada yada... this all works }
params = { 'corpus': '/corpora/myAccount/theAdviser', 'limit': 200 };
concept_insights.corpora.listDocuments(params, function(err,_res) {
if (err) { console.log(err); }
else { console.log(JSON.stringify(_res, null, 2));
res.send(JSON.stringify(_res, null, 2)); }
});
No matter what value is entered for the limit option, I always get 20 results. CURL, on the other hand, returns the full list or a subset based on the specified limit.
The equivalent working curl statement is:
curl -u "{userID}":"{password}" "https://gateway.watsonplatform.net/concept-insights-beta/api/v2/corpora/myAccount/theAdviser/documents?limit=200"
It looks like this was an oversight in the npm module. I just added support for the limit param, it should be released as v1.9.1 once the CI loop finishes.
Unfortunately, this does not seem to be reproducible for the corpora I have access to. For example, this curl:
curl -s -u username:password \
"https://gateway.watsonplatform.net/concept-insights/api/v2/corpora/public/TEDTalks/documents?limit=100"
Produces a list of 100 documents for me. If you have jq installed you can verify:
curl -s -u username:password \
"https://gateway.watsonplatform.net/concept-insights/api/v2/corpora/public/TEDTalks/documents?limit=100" \
| jq '.[] | length'
100
Another way to try to look at your corpus, is checking the "Concept Insights Dashboard" available in Bluemix by clicking on your service instance tile (the icon that is currently in use by your application). The first page of the dashboard allows you to select the corpus, and it reports a high-level summary of the corpus (including number of documents).
I just want to query the PRs from a specific user.
Do I need to use the head param? But how?
https://api.github.com/repos/org-a/repo-b/pulls?state=open&per_page=1&head=owner:user-c does not work.
API Refer: https://developer.github.com/v3/pulls/#list-pull-requests
Use the Search API and specify repo:REPO, author:USER and is:pr filters:
https://developer.github.com/v3/search/#search-issues-and-pull-requests
For example, here are the pull requests from rails/rails from aderyabin:
https://api.github.com/search/issues?q=is:pr+repo:rails/rails+author:aderyabin
Here is the GraphQL API syntax to query the pull requests in the repo rails/rails from the author aderyabin:
{
search(query: "is:pr author:aderyabin repo:rails/rails", type: ISSUE, first: 100) {
issueCount
edges {
node {
... on PullRequest {
title
createdAt
url
}
}
}
}
}
The PullRequest object docs lists the fields available to query.
It doesn't seem to be possible at the moment to filter a user's PR for a given repo (or a repo's PR for a given user), but the same result can be achieved with the search query above.
I'm trying to use the JIRA REST API (2.0.alpha1), to query and get all issues in a certain project at "Awaiting Release".
I can get a query to get me all issues in a certain project "TST" (with this string):
http://hostname.com/jira/rest/api/2.0.alpha1/search?jql=project=TST
However, I want to filter this further by only getting the issues with a certain status.
I've created a filter in JIRA using their JQL language, and it looks like this (if this helps):
project = TST AND issuetype in (Bug, "User Story") AND status = "Awaiting Release"
Also, I am using 'node-jira' (https://npmjs.org/package/jira). If you are familiar, here is my call. It always returns a 500 for some reason.
jira.searchJira('project=TST', {}, function (err, issue) {
console.log(err);
console.log(issue);
});
Here is the documentation: https://docs.atlassian.com/jira/REST/latest/#d2e1291
Here is the example page: https://developer.atlassian.com/display/JIRADEV/JIRA+REST+API+Example+-+Query+issues#JIRARESTAPIExample-Queryissues-Request.4
Okay, I've got the following URL query that works:
http://hostname.com/jira/rest/api/2.0.alpha1/search?jql=project=TST+AND+status=%22Awaiting%20Release%22+AND+issuetype+in%20(Bug,%20%22User%20Story%22)
However, I still cannot figure out why that node-jira is returning a 500 on the result. Please post if you have any input as well!