I'm trying to find a way to check if my rdfa-parser (written in nodejs) is working.
So I have an rdfa-parser, which should print all triples, found in a file or url (with rdfa-syntax).
So far I know, that there are testsuits for RDFa-parsing (http://rdfa.info/test-suite/rdfa1.1/html5/manifest), but I'm not sure how to use them.
Is there a good webpage, where this is described? Or can anyone help me in another way?
There should be some information at the rdfa.info/tests site. Basically, you need a service that will accept a GET request, where the "uri" query parameter points to the input file. The service then parses the file, and returns some other form of RDF, typically N-Triples. More information on the Github page: https://github.com/rdfa/rdfa-website/blob/master/README.md
Lately some of the articles have imbedded videos, but they are not listed in the JSON responses when querying the API; not even in a "related videos" type of attribute. Only the normal videos are available via the API.
Unfortunately I don't think there is. For our older articles, which are just a blob of HTML, /api/v1/articles/<id> will give you a htmlContent field will presumably contain some iframe. For our newer articles, the perseusContent field has the data we use to render it; I don't know that we document the format anywhere but the code that renders it is open-source so you could take a look at that, or just look at some actual articles with videos and see what they look like. Sorry we don't have a more direct way to do this!
I need an alternative to Google Custom Search for a website I look after, it has to be something that will crawl a website, index it, allow fiddling of priorities, and then allow search queries via REST or something similar and return XML or JSON etc. It needs to run on a Windows Server instance.
So, I'm up and running with http://www.opensearchserver.com/ and it seems to do the trick, but can't, for the life of me, work out how to get thumbnail images in the results? I've searched the documentation and read everything I could, but can't find out how to do this (or how to get my head around it).
I'm crawling standard web pages and they all have thumbnail meta data, which I'm assuming should be able to be parsed somehow for results and included in the JSON results?
Any pointers at all would be very helpful, thanks!
I figured this out, in case anyone else is struggling, here's how I did it. The answer is in the documentations, it's just not that simple.
Read: http://www.opensearchserver.com/documentation/faq/crawling/how_to_extract_specific_information_from_web_pages.md - it contains the method
Assume you set up a 'web crawler' index.
Assuming you're using a meta thumbnail like this:
<meta name="thumbnail" content="http://my_cdn.com/news/images/29637.jpg">
Go into Schema / Fields. Add a new field called 'thumbnail' with index no, store yes, vector no, analyser Text, copy of blank. Save that.
Now go to schema / parser list, edit HTML parser. Go to 'field mapping', now add a new regex for the thumbnail in the html. We map from the 'htmlSource' to the thumbnail' with the matching regex.
My imperfect regex (that works though) is:
htmlSource -> linked in: thumbnail -> captured by:
(?s)<meta name="thumbnail" content="(.*?)">
Now SAVE this and go to crawl/manual crawl, enter a url that has a thumbnail and then check if the field now appears in the list below when it's read. If not check your regex, and check you actually saved the HTML Parser changes.
To get the thumb in your results, simply add the fieldname to the JSON you send with the query:
"returnedFields": [ "
"url",
"thumbnail"
],
http://www.theverge.com/2015/7/6/8899807/instagram-1080p-pictures-photo-upload
The Verge recently made known the latest app versions of Instagram are uploading photos in 1080x1080 resolution. These photos can be easily sourced via HTML when browsing the site's source code. Can these be accessed via the official Instagram API? And if so, how?
[Update] After Mar 23, 2018
I used Ekrem Gurdal's answer to make it work:
The IG API gives you "link" attribute, e.g. https://www.instagram.com/p/BTduOwSAwN6/
Then you can append ?__a=1 to this e.g. https://www.instagram.com/p/BTduOwSAwN6/?__a=1
And you can extract from the json result where the high resolution photo is stored: graphsql.shortcode_media.display_url
E.g. https://instagram.fmnl2-1.fna.fbcdn.net/vp/f8f54b9fcb2bca69d7c73d9ffb232d64/5C324A1C/t51.2885-15/e35/18161146_397673830625587_2385146908864675840_n.jpg
Hope this helps. Maybe IG folks will also find a way to disable this in the future. Why do they not just include the hi-res photo in the API response, smh
[Update] As of Mar 23, 2018, workaround below doesn't work anymore.
Just to update anyone who's looking at this in 2018. The solution that worked for me is to remove the "vp/" on top of replacing the resolution to 1080x1080.
E.g.
https://scontent.cdninstagram.com/vp/41d5aee08758061976d26e48f45e673a/5ADA338E/t51.2885-15/s320x320/e35/20065394_1785212551507111_5192354829363576832_n.jpg
to
https://scontent.cdninstagram.com/41d5aee08758061976d26e48f45e673a/5ADA338E/t51.2885-15/s1080x1080/e35/20065394_1785212551507111_5192354829363576832_n.jpg
Not sure why you are getting down-voted, but no, that resolution is not returned by the API at the moment. However, it can be "guessed" from the other resolutions returned by the API, but that could change at any time. For example, given the standard resolution JSON:
"standard_resolution": {
"url": "https://scontent.cdninstagram.com/hphotos-xfa1/t51.2885-15/s640x640/e35/sh0.08/11370971_837216559694773_771634899_n.jpg",
"width": 640,
"height": 640
}
Remove the size part (s640x640 here) and the two paths that follows (e35/sh0.08 for this picture, likely different for yours) to end up with the URL to the 1080x1080:
https://scontent.cdninstagram.com/hphotos-xfa1/t51.2885-15/11370971_837216559694773_771634899_n.jpg
This has worked since that resolution was activated on my account. But again, I would just wait for the API.
As of 26th March 2018, Instagram changed their links.
Here's a working solution to get 1080x1080 resolution images from API.
Get the user_Id from https://www.instagram.com/{username}/?__a=1.
Paste the user_id in this link https://i.instagram.com/api/v1/users/{user_id}/info/.
Get the hd url link from hd_profile_pic_url_info. :)
I noticed that simply removing the size part, the biggest image is returned:
https://scontent.cdninstagram.com/hphotos-xfa1/t51.2885-15/s640x640/e35/sh0.08/11370971_837216559694773_771634899_n.jpg
to
https://scontent.cdninstagram.com/hphotos-xfa1/t51.2885-15/e35/sh0.08/11370971_837216559694773_771634899_n.jpg
Hence this should do the trick:
var hightUrl = data.images.standard_resolution.url.replace("/s640x640/","/");
You can simply fetch 1080p image if available by using this url:
https://www.instagram.com/p/['IMAGE-CODE']/?__a=1
The problem with instagram image URL's is that the URL signatures provided in the json responses expire after some time. My workaround for this problem is as follows:
Steps:
1. Get the instagram post link. Eg : https://www.instagram.com/p/Bo-Jru-g7Wa/
or if you don't have the link, the instagram api provides you with a permalink(image-code) option in the result array which for the above link is Bo-Jru-g7Wa
Now just add media?size=l after the url ie.,
Result: High quality image url:
https://www.instagram.com/p/Bo-Jru-g7Wa/media?size=l
you can see it in action here: https://jsfiddle.net/nmj1z7wo/fiddle URL
This link can be considered as a shorthand code to instagram image URL's which are very much bigger. Supported values for size are t (thumbnail), m (medium), l (large). Defaults to m.
This works as of 20th March 2018.
Remove the vp and the size parts.
So if we have a 150x150 image here
https://scontent.cdninstagram.com/vp/6f28b299faab04675a936073c94d0fde/5B2E3498/t51.2885-15/s150x150/e35/c135.0.810.810/28152308_875888592619569_7505395194448052224_n.jpg
Remove /vp and /s150x150 to get
https://scontent.cdninstagram.com/6f28b299faab04675a936073c94d0fde/5B2E3498/t51.2885-15/e35/c135.0.810.810/28152308_875888592619569_7505395194448052224_n.jpg
And we can get the 1080 image
here
You can even just leave the last two segments
https://scontent.cdninstagram.com/e35/28152308_875888592619569_7505395194448052224_n.jpg
which can be seen here
Hope this helps
This worked for me today (15 August 2018) for most public images. I think, because people use different apps to upload images, the API may show different sizes but the links are to the same default resolution file.
Go to the user's page: https://www.instagram.com/username
Click on the image you want and the URL bar will change to: https://www.instagram.com/p/image-code/?taken-by=username
Copy the image-code from the URL
Go to: https://www.instagram.com/p/image-code/?__a=1
Scan through the JSON file (it helps to have a JSON viewer extension enabled) and click on the highest resolution image to download it. For example:
I'm unsure whether it's possible to get a 1080p resolution image. But here is one way to get an image with 320x320 resolution.
1) Enter the username of the user in the following URL. Here I'm using simonsterstrength as username.
https://www.instagram.com/simonsterstrength/?__a=1
2) Search for "profile_pic_url_hd" and copy the URL and access the picture that IG calls it as HD.
Note: As of June-2020, the response from IG CDN is OK.
You received
"standard_resolution": {
"url": "https://scontent.cdninstagram.com/hphotos-xfa1/t51.2885-15/s640x640/e35/sh0.08/11370971_837216559694773_771634899_n.jpg",
"width": 640,
"height": 640
}
It's possible to replace s640x640 with s1080x1080 or 2048x2048. Both this request return image with resolut 1080x1080
var hightUrl = data.images.standard_resolution.url.replace("s640x640","s1080x1080");
I've noticed something weird and I was thinking that maybe I'm missing something. I noticed that if you use the tag/tagname/media/recent endpoint a lot of the result show they contain the given tag. However if you actually open the url and you see that they either lack that tag or the photo doesn't have any tags at all. Then when doublechecking by querying by specific media id the results are the same. Any idea why this is happening?
The endpoint returns media where the tag is also in any of the comments. The tag you were searching for might be assigned in any of the comments for that media. I don't believe that all comments are returned for each image in this endpoint, so it might be assigned to a comment that you don't see in your results.