Bulk async calls to FB and G+ count endpoints fail - node.js

my app has a controller that outputs the total social count for various pages: http://pastebin.com/MLBTb3mi
..it works fine when i'm making a few calls at a time but say when there is a website of 1000 urls and i want to update the social count for each of the urls, it breaks when its hit with volume calls asynchronously - particularly g+ and fb break. Here's the console error I get as the response to the call made to fb's graph id (this is the 'body' response - please refer to line 74 of the controller: http://pastebin.com/MLBTb3mi
facebook body is {"id":"970371719648388","created_time":"2015-04-02T07:43:09+0000","is_scrap
ed":false,"type":"website","updated_time":"2015-04-02T07:43:09+0000","url":"http:\/\/www.zappos.com\/womens-clothing\/"}
facebook result is undefined
Does anyone know how to solve this issue and to make the controller work smoothly?
I tried making bulk calls using sharedcount.com and i'm able to do 10s of thousand of simultaneous calls without any problem. How can I write my controller to handle such bulk operations without using external services like sharedcount?

You should examine the status code of the response. My guess is that you're getting rate-limited. See the Facebook page on rate-limiting for details; it doesn't discuss specific numbers on what will get you blocked, but it provides details on what to look for to indicate that you're being throttled. The solution is to throttle requests on your end so that they don't go out too fast; you can use something as simple as Lodash's _.throttle for this.

Related

Make multiple API request parallel and show the data as soon as we receive any one response in asp.net

I am building a hotel search engine where I have 20 xml API integration. Now facing a problem of time laps in receiving all 20 xml responses from all API before binding the result page. I am looking for a solution where we can call all API parallel in multithreading and I want to show the api response as soon as I receive 1st response from any API and then merge next response from previous one and update the binding page. In this way I can show result of 1st few hotels starting from very 1st API response.
Please suggest me the way I can achieve my objective in asp.net c#. This is my 1st question in this forum and very eager to have feedback.
You would have to use javascript to display the responses fluently. Basically, have ASP.NET load an empty page then use javascript to call the 20 API endpoints. As the responses come in, update the display. Single page application (SPA) frameworks, such as Angular, are fantastic for these situations.

facebook open graph debugger returning PARTIAL_FILE or operation_timeouted

I'm trying to setup a facebook share on https://donate.mozilla.org/en-US/thunderbird/share/
The og:url points to just /thunderbird which is the url I would want shared. Best I can tell the og tags are all there.
When I try to update the data on https://developers.facebook.com/tools/debug/og/object/
When I fetch new scrape information I get one of two errors. Initially, it'll take a long time then respond with a Curl Error : OPERATION_TIMEOUTED Operation timed out after 10000 milliseconds with {some number less than 10000} bytes received then subsequent fetch attempts respond with Curl Error : PARTIAL_FILE transfer closed with 17071 bytes remaining to read
We're using AWS Cloudfront and nodejs with hapijs
It responds with a 206 partial content, which, should be fine. The og tags are all in the beginning of the file.
I found this: docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RangeGETs.html
There it says a range request is used to get the file in chunks, not to get just the part of the file and give up. So maybe that's causing unexpected behavior. Maybe cloudfront is sending it back in chunks, and facebook stops listening after the first response? I dunno. Just trying to find a theory that fits the facts.
We already have a working share for donate.mozilla.org/en-US/share/ but that might be old data from when we were not using hapijs and instead using expressjs which I don't think was supporting range requests and would instead return a 200.
I'm mostly a front end dev, so a lot of this is out of my comfort zone but I have already learned a lot :)
Edit: I also want to point out we use Heroku for hosting, and if I setup a test with just heroku and without cloudfront: donate.mofostaging.net/en-US/thunderbird/ it fetches the tags successfully. So I suspect it's a bug when facebook and hapijs interact with cloudfront.

Disable direct requests to REST API

I'm making a REST backend for private use of our frontend, they will both be in the same server.
The problem is that I'm worried about security issues, I don't want a attacker to use the API directly, either by JS or by using other REST client.
Lets take this service as an example
http://myserver:8080/something/webresources/film
That's a service that allows to GET, PUT, POST, DELETE I want that only the frontend be able to use it, otherwise since anyone can see client-code it would be easy to get the endpoint and start putting or getting data. I do have BASIC AUTH so they would have to register and if they did something wrong I would be able to see who did it, but that doesn't solve the problem.
I could implement access control so that a user only could get/update/delete their own movies, but I would have to do that for every REST service(I have over 100 by now), plus I might need to actually get others movies
This is my first public project, I am really lost.
You can do it through your web server. I'm using Nginx. I have an if statement that checks the $http_referer. If it returns nothing, or the value returned is not my application/frontend page (meaning someone is trying to hit the api directly), it'll return a 403 forbidden page.
If your application doesn't send out emails to your users don't worry about the following: I added a block to allow access to my static images as the only exception, as my application sends out emails with images and I don't want them to break.
That's it. Problem solved. No one has access to my api except my frontend page/application, unless they can forge the $http_referer to match my domain which if they can do that then they deserve to break in.
Your only option to call the REST API from server side. You cannot hide from the users what's going on in their browser... You can have a layered application, so the frontend layer can call the backend layer on the server while the client can see only the frontend. (Check the layered system constraint.)

avoiding get limit when request map route on Nokia Here API

I'm new using this API. I want to request for a route using "Here API". The route xml/json is what I get easyly (I have the Maneuver values with calculateroute method), but I want to show users the map with the route. I have been able to show it using http://image.maps.cit.api.here.com/mia/1.6/route?, but this method only supports GET requests, and many times I have to make requests larger than 2500 characters supported by GET. Is there any other method to get an image with the route from Madrid to Berlin, for example?
Any reason why you could not use the JS-Maps API as well for showing the route, then you could simply use the waypoints as arguments. You can find example at: https://developer.here.com/javascript-apis/documentation/maps/topics/routing.html

Get Photos and Detailed Info for Many Foursquare Venues in one call

I am working on an iPhone app which allows users to search Foursquare. I am using the venues/explore endpoint for the search which works great, but the results don't include the images for a place or the priceTier.
Right now I am calling /venues/VENUE_ID for each of the returned results, which is generating a lot of API calls. Is there a better way to get this info in a single call?
Related question: If I use the multi endpoint to batch these requests, does that count as a single request towards the limit or as multiple requests?
Sounds like you're worried about limits more than network latency? If you're worried that making the extra call to details will make you hit rate limits faster, this is actually why we generally ask developers to cache details such as prices or photos :) A single multi request is not a single API call; it counts as how many requests are bundled into one.
There is a little help with photos though—if you pass in the venuePhotos=1 param as part of an explore request, you ought to get back photos in the response.

Resources