I want to create giveaways which require the participants to follow the twitter account of the giveaway creator.
My first idea was to use the Twitter API (endpoint: "/2/users/:id/followers"). This works fine for me however I always run into rating limits. The API allows me to send 15 requests every 15 minutes and returns a maximum of 1000 users per request. Since many accounts have more then 15000 followers and since many request happen at the same time (many users want to participate in a giveaway) this solution is not suitable for me.
My secound idea was to use web scraping instead (e.g Node Fetch). I was following along this tutoria: However doing so I always run into the issue that Twitter uses random strings to name their html elements. You can see in the picture there is no defined class to grap the elements.
So my main question is how can I access these element ?
Random Follower of my Twitter Account
I also have a follow up question regarding the effictivness of this method. Assuming I have multiple people who want to particpate in a short amount of time (e.g 10 people in 5 minutes) and they all need to follow a big twitter account (e.g 100k followers).
Is it efficent to scrape all 100k followers each time or should I instead try to fetch the 100k followers once, safe them to my database and use my database to check for each user later ?
As a side note, I am using node.js and node-fetch, however I have no problems to switch the framework. In addition I think the grabbing of the element as well as the performance should be universal.
Thanks for your help :)
They're going to detect your servers excessive calls. There is a Twitter Developer Portal where you can request elevated access which may raise the limits for you.
https://developer.twitter.com
Related
I'm looking for a way in Node.js, preferably in the twitter-lite package, to find out how many API requests I have left in the current time-window because I am automating a bot that attempts to check the followers of 4 of my personal Twitter pages.
Is there a way to do this at all? I didn't find any information about whether Twitter API reports such a counter.
Instagram API permission is strict now and I want to know whether the tool my company will make can get the permission from Instagram or not, because it may not suit with the 3 valid use case.
We are an offshore company which develops application.
One clients ( PR or Advertisement company) asked us to make an app which finds influencers(the user who has many followers) on Instagram so that the clients can ask the influencer to advertise their products.
We want to make a influencer searching tool. Lets, say if you search with #Chocolate, the list of users who have posted photos with #Chocolate will show up, and we want to sort out the list according to the number of followers.
Referring to the valid use case for Instagram, which case may be suitable for the app we want to make?
Also, by one-off project, what does it mean? We are an offshore company and get orders from a client which want us to make a tool for them. If we just make it, and will not sell or use the application again, would it be "one-off" project?
In order to avoid one-off project, do we need to keep selling or offering the app ?
There is no direct API call to make this. You have to make a series of API calls to get all the posts for hashtag, this will have user_id and name. Then for each user you have to make another API call to get followers count. And then you have sort the results and make query for >3000
For example #chocolate has 1 million posts, so to get all 1 million posts you have make about 50000 API calls (u get 20 post per API call), and then you have to make 1 million user_info API calls to get follower counts, so 1050000 API calls total and you have 5000/hr API calls limit. So to complete this operation it will take you 9 days.
This will be a one off project if u do for a client use, you have to make a platform or service for this feature and have your client (and others) login and use the service.
In any case, the project is unrealistic to implement due to the limitation of APIs
Im trying to add a feature to my website that involves the typical "invite your friends" with help from a contact importer (cloudsponge). Its a pretty popular and gets the job done but I need something faster..
The problem with cloudsponge is that they request all contacts in one call, this could mean a long wait time for someone with alot of contacts.
I looked at their rest calls and there doesnt seem to be a way to load contacts in pieces. Do any of these contact importing services allow you to pull in a few contacts at a time (lets say 50) so that we can show our user the first 50 contacts and load the rest / updating the view. So they dont have to wait forever for all the contacts to be pulled?
Ive looked at other apis like context io but cant seem to find a solution to this one.
I built the CloudSponge API.
Early on, we decided to support imports across a variety of providers while exposing a simple and consistent interface. Pagination and rolling or real-time access to contacts were things that were excluded in order to do that. To provide end-user feedback on the progress of the import, we added the /events endpoint.
So far import speed hasn't been a major issue for a couple reasons:
In general, end users with an address book of 10000+ contacts are rare (although this may not be the case for certain niches).
End users who do have this many contacts in their address book usually understand that it will take a while to import.
Having said that, the speed is something that we can definitely improve upon. Here's a few ideas:
We can allow for returning only a subset of all contacts by default. For example, we currently return all contacts for Gmail, which is usually a much larger number of contacts than are actually stored in 'my contacts'.
We can implement parallel paginated imports on the server side. This will make our server process work harder and faster to download the user's contacts from, say, Gmail. This adds complexity on our side but keeps the API untouched.
We can implement your suggestion: add a rolling or real-time access to contacts in our API, either in an extended endpoint or a new version of our interface.
I'm happy to work with you on exploring these to improve our service. Send us an email: support#cloudsponge.com
Graeme
is there a way to grab instagram users based on a specific hashtag ?
I run contests based on re posting photos with specified hashtag then randomly pick a winner, i need a tool that can grab the usernames of those who reposted that photo and used that hashtag.
You can query instagram using the API. There are official clients for both python and ruby.
You didn't specify what language/platform you are using, so I'll give you the generic approach.
Query instagram using the Tag Recent Media endpoint.
In the response, you will receive a user object that has the user's username, id, profile url, and so on. This should be enough to do what you are describing.
As far as tools, there aren't great options to probably do things exactly how you want. If you just want a simple contest, you could use statigram, but it's not free.
If you roll your own solution, I highly recommend you also do the following:
Implement a rate limiting mechanism such as a task queue so you don't exceed your API calls (5000 per hour for most calls). Also useful for failures/network hicups, etc.
Have users authenticate so you can use OAuth to extend your API calls to 5000/per user/hour to get around #1.
Try the subscribe API if there won't be many items. You can subscribe to a specific tag as well, and you will get a change notification. At that point though you need to retrieve the actual media item(s), and this can cost a lot of API calls depending on how frequent and what volume these changes occur.
If your users don't have much photos/relatively small/known in advance, you can actually query the user's recent media instead and filter in your own code by hash tag.
I created an iOS app that using Instagram API and have a critical issue posting likes to Instagram.
Using endpoint – POST /media/media-id/likes, with user-specific access token. I request scope – likes from users.
Instagram API works fine for all requests except saving likes. We can only save/post a couple hundred likes to Instagram, and then response is "We've taken extra measures to protect the Instagram API from abnormal activity. Since you have recently created your API application, please contact apidevelopers#instagram.com to receive whitelisting for a higher rate limit."
There is other developers with same issue with no support or any feedback waiting for months.
Please, let me know if there is limits and how it works, or how to become whitelisted
I don’t want to create many Instagram API clients and assign users among them. It seems like bad acting for me, if there is official API with strict limits I want to work with it.
Instagram has new API limits: 100 likes/hr 60 follows/unfollows/hr per access token. That is only if you authenticate serverside and sign your headers. Check out the new authentication: http://instagram.com/developer/authentication/
I'm using it for my new site that automates likes/follows/unfollows, haven't had any problems works great. Check it out: http://instapromobiz.com
Instagram takes increased measures to limit your API requests based on IP address. This is not common knowledge, it's just something I've come to know from working with the API extensively.
They've built an API that is FAR from perfect, but have been upgrading it to be "smart". Smart enough to know that a newer client, pinging the API many times just for one user, may very well be a script kiddie trying to bot the system. For this reason, they do not give you a rate limit error, they politely say "we have taken extra measures to protect our API from being spammed by bots".
The good news is, other users should still be able to access your liking mechanism. Just not you, from that IP.