The GitHub API provides functionality to get stared items sorted by created date. But, I can't find a possibility to get this date.
Only three date-type fields are present in the response:
created_at: '2013-06-13T21:10:36Z',
updated_at: '2013-06-18T07:45:56Z',
pushed_at: '2013-06-17T16:52:44Z',
create_at here is always that date the repository created, not star. updated_at does not give the right information either.
On the GitHub site, if you check your stars, data is sorted properly and the correct time span is rendered. But it looks like the API misses that information.
That gist clearly show, that data returned "sorted", but created_at is wrong.
This can be done in the GitHub API v3 by adding the header:
Accept: application/vnd.github.v3.star+json
see here:
https://developer.github.com/v3/activity/starring/#alternative-response-with-star-creation-timestamps-1
https://developer.github.com/v3/activity/starring/#alternative-response-with-star-creation-timestamps
There is absolutely no way to get this information from the API. Your best chance of getting it is going through events either on the repository or each user that has starred the repository and that could take tons of parsing because the events could be in the thousands. The number of API calls to do so would be immense.
Related
I'm using Google Photos API to access albums.
Users can have in an album anything from a single photo to thousands of photos.
Is there a way to get something like an ETag to know if an album has changed since a previous known state?
Currently, the only way I could find is to iterate over all the images, and having to do that in 100 photos each time can take a lot of calls just to find out at the end that nothing has changed.
You want to know whether the album of Google Photos has been changed.
I could understand like above. Unfortunately, in the current stage, there are no metadata like the modified date when the contents in the album were changed. So in my case, I use the following workaround. Please think of this as just one of several workarounds.
Workaround:
If the number of photos in the album is changed, it can be known by the property of mediaItemsCount.
If the cover photo is changed, it can be known by the property of coverPhotoBaseUrl.
If several photos in a album are changed without increasing and decreasing the number of photos, unfortunately, in the current stage, I think that it is required to confirm the creationTime, filename and so on using the method of mediaItems.list.
By confirming from above in order, I can know about the change of the album.
References:
Method: albums.list
Method: albums.get
Method: mediaItems.list
So I have some code that's been running for a good ~6 months or so, and I don't think I've changed anything relevant to this (famous last developer words), but Foursquare has started returning what I'd consider to be incorrect responses.
I'm trying to hit the checkins endpoint for yourself: i.e., /v2/users/self/checkins. My options params are as follows:
params = {
v: '20160622',
afterTimestamp: start_time,
sort: 'oldestfirst',
limit: 250
}
This all seems to be fine and dandy, but Foursquare is only seemingly sub-sorting newest-first. In other words, I'm asking Foursquare for "show me all of the checkins, sorted by the oldest" and it's returning stuff from today. Especially with afterTimestamp: 0, I expect that it should show me the oldest checkin associated with my account: namely, my very first checkin of all time.
So, to recap: how do I query the API so that I can find my oldest checkins first? From what I can tell, this should be properly returning the correct checkins.
Cheers!
I was playing around with this endpoint and it appears to sort results after they're extracted from "most recent". So in your case it's taking the most recent 250 checkins then sorting them by oldest first.
To get your first checkin ever you can look at the count of checkins that gets returned then offset by that amount. After you've got your first checkin you can work forwards with the afterTimestamp parameter.
When I pull media using either the user or tag endpoint, the received media not always contains all tags for a post. This seems (we are still in sandbox mode) to be the case when the tags are contained in comments BUT not always.
E.g: Requesting tag "arkiromantix" (https://api.instagram.com/v1/tags/arkiromantix?access_token=[access_token]).
Returned data contains among others post with id 1192868766714482828_38257867 which lists the requested tag under tags and post with id 1218885626018570787_210152727 which does not.
But with both posts the tag in question was added in a comment not in the caption.
My questions would be:
Is there a way to assure getting all tags (from caption and authors comments) when pulling media from the Instagram API? (For now I am requesting the comments for each media, which increases the amount of requests the app uses considerably). And if not:
Is there a rule by which Instagram decides when they return all tags (from caption and comments) and when not? (So i can at least reduce the amount of requests)
It seems that tags in comments count as tags on the gram itself if and only if the comment was written by the user that posted the image. This SO question deals with a related issue (i.e. retrieving recent grams with a given hashtag doesn't actually sort by post date of the gram if the hashtag is in a comment made by the gram's poster, but instead by post date of that comment, in that case). See in particular this comment.
I can confirm this (undocumented!) sorting quirk from personal experience from the past two days ... this sort of thing really makes work interesting.
I'm working on an instagram scraper for something and I'm trying to figure out if it's possible to get all photos for a tag that have an id or timestamp later than the last one I have.
The instagram API docs are useless in that they don't have any real info on pagination (which I presume I'll have to abuse).
Does anyone have any ideas?
I've been slogging through the Instagram API for the last couple of days so here's my 2 cents worth:
As far as I can see it if you call the api with /tags/tag-name/media/recent it only return a list if items. If the amount exceeds about 25 you have to make another request with the pagination value returned in the previous request.
In order to gain some control I am initially iterating through all images and storing the results (just the URL not the actual image) to a database. Now I can manipulate however I want. When I feel like updating (I'm doing it manually now but could be a cron job or use the real-time api) I re-read all the images, compare to what I have in my DB and add possible new images. My app then reads out the url and info from my DB (which btw is a heck of a lot faster than going through the instagram api, which will only return about 25 images per request - regardless of any 'count' parameter value you put in the request url) and displays it.
I am developing this for a client who is afraid of people posting nsfw or whatever pics using their dedicated hashtag (for a contest) - with the above set up I can offer them an interface where they can check and mark images that are then displayed in the app.
One thing to watch out for is when a user deletes his picture; you will have to find a way to check for this. Currently (since I'm lazy) I load all images and use jquery to check for an error loading the image. If there is one I delete the image from the DB (via ajax).
I'm not sure the pagination is going to help you: as far as I can see the pagination response has no relation to the id's of the actual image objects on each page - so theoretically a pagination id that jumps to a certain page (i.e. date) might not work tomorrow if enough images have been deleted in the mean time.
to get all images instead of latest 20, just append &count=-1 to your api call - it's that simple.
In either case, there is a timestamp on each json object - or if you prefer, you can use max_tag_id
check out my post here: there any way to show more than 20 photos of the instagram API?
* Update April 2014: count=-1 is no longer available.
I'm doing a notification system for my website.
A notification systeme like facebook. Or stackoverflow.
I have 2 problems.
How store in db ? I can store ALL notifications in the user document ? or in a document apart (because i think monogdb is limited for size in a document ?) Or, store intelligently ? (using inc, or a value (see: true/false) in db, with query sophisticated)
How do for brought at the page ? For exemple, when i click in a link in my inbox for stackoverflow, i'm redirect to the page. But me, i have a system that is multipage for exemple: I have 100 friends. There are listed 30 per page. So when i click on the notification i can't redirect to the because it's impossible to know the good page (users can be removed).
Thank you very much !
And if you have another ideas, tell me. Thanks.
EDIT:
(sorry for my english, i'm french)
For the first problem, i realize that i have to wait the time comes to choose my structure. Because my notification is .. a little complicated, so advance to the feeling.
For the second, i solved the problem. I explain:
(I take the exemple of friends because it's easy to undestand.)
I stored my data like this:
{
friends: [
{_id: xxxxx, ts: xxxx},
{_id: xxxxx, ts: xxxx}
]
}
Imagine i display all friends: 30 per page.
The problems are:
when i want to display all friends i cant sort using mongo. (a little problem)
If i want to lead a user to this list (30 per page) at a special friend, always keeping the sort by ts. I can't know the page. The uniq solution is to take all document.
But: veryyy bad in performance.
So, i store like this:
{
friends: {
xxxx: {ts:xxx},
xxxx: {ts:xxx}
}
}
Know i can sort the document, with use skip and limit.
So if i want a portion, i do not need to take all documents.
To know the page, i just do the number of < or > to the ts, i have for exemple 11 friends who are > to the ts of the friends that i want, and do a count for all friends (ex: 50 friends) with 50 and 11, i can guess the page.
Is this solution is good ?
- i need a count
- a query to know the number of > or <
and i can take the page where is listed the friend, keeping the sort ts
You can don't understand why i use a count. I need because they are not store in the same docment.
2 EDIT:
The problem with this solution, is that i need to make query object and update object outside of the mongo query (ex: for do friends.xxxxxx: {$exists:true}
ps: And what advantages are to use ts instead of date for mongodb ?
I'm using ts but i think i will store date, and no ts.
3 EDIT
I will do like Sammaye. Store in separate document. Take a look at: http://mongly.com/Multiple-Collections-Versus-Embedded-Documents/#1 and http://openmymind.net/2012/1/30/MongoDB-Embedded-Documents-vs-Multiple-Collections/
#Stennie make a pretty complete answer.
However recently I did a similar thing in PHP for my website. The first thing to understand is whether you are doing a notification system or a wall (the two are very different), it seems unclear to me and I am not sure what you mean by:
How do for brought at the page ? For exemple, when i click in a link
in my inbox for stackoverflow, i'm redirect to the page. But me, i
have a system that is multipage for exemple: I have 100 friends. There
are listed 30 per page. So when i click on the notification i can't
redirect to the because it's impossible to know the good page (users
can be removed).
That is not very good English and is very confusing when I read it. If you can expand on that I am sure people can answer better.
For a notification system I found that a large collection of notification objects also worked. So I had a schema like:
{
_id: {},
to_user: ObjectId{},
user_id: ObjectId{}, // Originating user
custom_text: "has posted a new comment on your wall post",
read: false,
ts: MongoDate()
}
And this would literally be the document I have to produce notifications. Each time a user commits an action that generates a notification it writes a new row to the DB with to_user being populated each time with each user needing to be notified. As for multiple users commiting the same action I actually convert the user_id field in a list of OjbectId's so I can say:
Sam, Dan and Mike all commented on your wall post
I then query by ts storing the last ts that the user looked at in their row allowing me to do a range based query on the newest notifications each time. This works quite well for sharding and querying in my personal experience.
Hope it helps,
Whether to embed or link is a common question for data modelling in MongoDB. If your number of notifications is going to be unbounded, you are likely going to be better saving these in a separate collection.
The current 16Mb document limit actually isn't as much as an issue as some other considerations:
A performance issue you may encounter by including all notifications in a single document is that fast-growing documents may also need to be relocated in the database more frequently (see Padding Factor).
You may want to be applying multiple updates to a document (such as setting a "read" flag on notifications) in a very short period of time, which means more contention for updating the same document (see Atomic Operations).
In order to implement paging you can use limit() in combination with a range query or skip(). A range query (eg. based on an indexed notificationDate) will make more effective use of indexes and perform better than skip() as your collection grows.