I want to get overall datastore Statistics for my Dashboard Application. I went through the docs there are no tutorials about how to get the statistics using datastore. Somehow to get the status details there is GQL QuerySELECT * FROM __Stat_Total__ which displays
builtin_index_bytes,
builtin_index_count,
bytes,
composite_index_bytes,
composite_index_count,
count,
entity_bytes,
timestamp.
I want to display all these details through Python API Client.
I tried a few examples which didn't work out.
def get_summary_statistics(self):
#[START getting the Summary Statistics]
stats = self.client.query(kind= self.kind_name)
overall_stats = stats.__Stat_Total__ ()
return overall_stats
How do I get all datastore Statistics?
The Cloud Datastore NDB administration documentation has some information about __Stat_Total__ and other stat entities, along with a small example script that queries Datastore stats:
from google.appengine.ext.ndb import stats
global_stat = stats.GlobalStat.query().get()
print 'Total bytes stored: %d' % global_stat.bytes
print 'Total entities stored: %d' % global_stat.count
Related
I have a tasklist with repeating tasks inside. I managed to retrieve the tasks via google tasks API, but only for the current day. From the documentation I find it impossible to retrieve all the history of reccuring tasks. Am I missing something?
The idea is to create a script to keep a count of how successful a person is in doing his dailies, keeping a streak, maybe make a chart to see ratio of completed/not completed etc...
I used the google quickstart code and modified it a bit, but as much as I try, I can only get the reccuring tasks from that day... Any help?
# Call the Tasks API
results = service.tasks().list(tasklist='b09LaUd1MzUzY3RrOGlSUg', showCompleted=True,
showDeleted=False, showHidden=True).execute()
items = results.get('items', [])
if not items:
print('No task lists found.')
return
print('Task lists:')
for item in items:
print(item)
Iam implementing a search endpoint for a particular condition using oracle r2dbc to provide a paged output. I need total number of records so that the user can fetch the results accordingly. Below is the code that am using. However it fails the performance test and throws up 503 error. Any help will be appreciated.
{
return responseMapper.map(planogramRepository.findPlanogramsByDepartmentAndSubgroup(filter.getDepartment_id(),filter.getSubgroup(),pageRequest))
.collectList()
.zipWith(this.planogramRepository.countPlanogramsByDepartmentAndSubgroup(filter.getDepartment_id(),filter.getSubgroup()))
.flatMap(tuple2 -> Mono.just(SearchResponse.builder().totalCount(tuple2.getT2()).planograms(tuple2.getT1()).build()));
}
I had a script that ran for about ~1hr.
df is a data frame created by rows I queried from my own Postgres database.
I'm iterating over that to get the artist_name value and make the API call.
There are ~47k records in that table.
After an hour or so, the script stopped giving me any responses. There are no errors.
The line that breaks is results = artist_albums(...)
Putting a breakpoint() before works but once that line runs, It stops. No status errors, 429, etc...
Did I hit the Spotipy API too much?
for idx, val in enumerate(df['artist_name']):
#get albums for each artist
results = artist_albums('spotify:artist:'+df['artist_name'], album_type='album')
albums = results['items']
while results['next']:
results = spotify.next(results)
albums.extend(results['items'])
sleep(0.5)
for album in albums:
print(album['name'])
try:
q = (album['name'],
album['id'],
album['uri'],
album['release_date'],
album['release_date_precision'],
album['total_tracks'],
album['artists'][0]['id'],
album['type'],
album['external_urls']['spotify']
)
)
cur.execute("""insert into schema.table values (
%s, %s, %s, %s, %s,
%s, %s, %s, %s)""", q)
conn.commit()
```
You have probably hit Spotiy API's rate limits, which works in a 30-seconds rolling window.
If your app makes a lot of Web API requests in a short period of time
then it may receive a 429 error response from Spotify. This indicates
that your app has reached our Web API rate limit. The Web API has rate
limits in order to keep our API reliable and to help third-party
developers use the API in a responsible way.
Spotify’s API rate limit is calculated based on the number of calls that your app makes to Spotify in a rolling 30 second window.
A way to avoid this would be to introduce some waiting time between API calls, for example using time.sleep, i.e.:
import time
time.sleep(10) # sleeps for 10 seconds
The python code which I have written, it generates some global and BigQuery related logs on stackdriver logging window. Further I am trying to create a metric manually and then send some alerts. I wanted to know whether we can create a single metric for both global and bigquery logs on stackdriver?
In Advanced filter query I tried:
resource.type="bigquery_resource" AND resource.type="global"
severity=ERROR
but its give error: "Invalid request: Request contains an invalid argument"
Then I tried:
resource.type="bigquery_resource", "global" AND
severity=ERROR
again it gives error: "Invalid request: Unparseable filter: syntax error at line 1, column 33, token ','"
import logging
from google.cloud import logging as gcp_logging
client = gcp_logging.Client()
client.setup_logging()
try:
if row_count_before.num_rows == row_count_after.num_rows:
logging.error("Received empty query result")
else:
newly_added_rows = row_count_after.num_rows - row_count_before.num_rows
logging.info("{} rows are found as query result".format(newly_added_rows))
except RuntimeError:
logging.error("Exception occured {}".format(client1.report_exception()))
Looking for a approach where I can have single metric for multiple resource type. Thank you.
I think you want
resource.type="bigquery_resource" OR resource.type="global"
severity=ERROR
i want to search-tweet related 'data' and count more than 100
this is python grammer
from twython import Twython
twitter= Twython(app_key=APP_KEY,app_secret=APP_SECRET)
for status in twitter.search(q='"data"',count =10000)["statuses"]:
user =status["user"]["screen_name"].encode('utf-8')
text =status["text"]
data = "{0} {1} {2}".format(user ,text,'\n\n')
print(data)
f.writelines(data)
So what you're trying to do uses the Twitter API. Specifically the GET search/tweets endpoint.
In the docs for this endpoint:
https://dev.twitter.com/rest/reference/get/search/tweets
We can see that count has a maximum value of 100:
So even though you specify 10000, it only returns 100 because that's the max.
I've not tried either, but you can likely use the until or max_id parameters also mentioned in the docs to get more results/the next 100 results.
Keep in mind: "that the search index has a 7-day limit. In other words, no tweets will be found for a date older than one week" - the docs
Hope this helps!
You can use the field next_token of the response to get more tweets.
Refer to these articles:
https://lixinjack.com/how-to-collect-more-than-100-tweets-when-using-twitter-api-v2/
https://developer.twitter.com/en/docs/twitter-api/tweets/search/integrate/paginate
The max_id parameter is the key and it is further explained here:
To use max_id correctly, an application’s first request to a timeline
endpoint should only specify a count. When processing this and
subsequent responses, keep track of the lowest ID received. This ID
should be passed as the value of the max_id parameter for the next
request, which will only return Tweets with IDs lower than or equal to
the value of the max_id parameter.
https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines
In other words, using the lowest id retrieved from a search, you can access the older tweets. As mentioned by Tyler, the non-commercial version is limited to 7-day, but the commercial version can search up to 30 days.