Github 304 responses seem to count against rate limit - github-api

I'm running into an issue with the github api counting requests that yeild a 304 response against the rate limit
GET /repos/******/***/contents/***?ref=master HTTP/1.1
Host: api.github.com
Connection: keep-alive
Cache-Control: max-age=0
Sec-Fetch-Dest: empty
If-None-Match: W/"b15846765021dc2483e5f3110b53a69f210ececa"
If-Modified-Since: Thu, 19 Mar 2020 01:33:51 GMT
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36
Accept: */*
Origin: *******
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: cors
Referer: ******
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
this request will return with a 304 and no body, but X-Ratelimit-Remaining decrements.
Conditional Github Requests
Note: Making a conditional request and receiving a 304 response does not count against your Rate Limit, so we encourage you to use it whenever possible.
I'm only setting the If-None-Match and If-Modified-Since headers, the rest are set by the browser (using fetch api)
Request
This can be replicated with just a browser by navigating to https://api.github.com/repos/angular/angular/contents/?ref=master and making sure disable cache is off in the developer console
Responses
On first call
If-None-Match and If-Modified-Since are not set
HTTP/1.1 200 OK
date: Wed, 01 Apr 2020 12:43:09 GMT
content-type: application/json; charset=utf-8
server: GitHub.com
status: 200 OK
cache-control: public, max-age=60, s-maxage=60
vary: Accept, Accept-Encoding, Accept, X-Requested-With
etag: W/"b15846765021dc2483e5f3110b53a69f210ececa"
last-modified: Thu, 19 Mar 2020 01:33:51 GMT
x-github-media-type: github.v3; format=json
access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
access-control-allow-origin: *
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'
content-encoding: gzip
X-Ratelimit-Limit: 60
X-Ratelimit-Remaining: 59
X-Ratelimit-Reset: 1585748589
Accept-Ranges: bytes
Content-Length: 1696
X-GitHub-Request-Id: B298:74D9:10D7B8:177665:5E848C5D
followed by the requested data
Subsequent Calls:
If-None-Match and If-Modified-Since are set to etag and last-modified header from previous request
HTTP/1.1 304 Not Modified
date: Wed, 01 Apr 2020 12:44:51 GMT
content-type: application/json; charset=utf-8
server: GitHub.com
status: 200 OK
cache-control: public, max-age=60, s-maxage=60
vary: Accept, Accept-Encoding, Accept, X-Requested-With
etag: W/"b15846765021dc2483e5f3110b53a69f210ececa"
last-modified: Thu, 19 Mar 2020 01:33:51 GMT
x-github-media-type: github.v3; format=json
access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
access-control-allow-origin: *
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'
content-encoding: gzip
X-Ratelimit-Limit: 60
X-Ratelimit-Remaining: 58
X-Ratelimit-Reset: 1585748589
X-GitHub-Request-Id: FC10:2036:26E528:346432:5E848CC3
no data is returned.
According to the documentation X-Ratelimit-Remaining should have stayed at 59
Is there something wrong with my request or is there a bug with the ratelimit on githubs end?

got the same problem. and resolved it. i've added to headers 'Authorization' key with string value. i'm using python and requests for humans. so in my console looks this like that:
>>> response = requests.get('http://api.github.com/repos/zappycode/zappycode-django/commits', headers={'Authorization': 'JustaString', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0', 'If-Modified-Since': 'Sat, 18 Jul 2020 00:15:14 GMT'})
>>> response.headers
{'Date': 'Tue, 21 Jul 2020 09:06:05 GMT', 'Server': 'GitHub.com', 'Status': '304 Not Modified', 'X-RateLimit-Limit': '60', 'X-RateLimit-Remaining': '59', 'X-RateLimit-Reset': '1595323695', 'Cache-Control': 'public, max-age=60, s-maxage=60', 'Vary': 'Accept, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding', 'ETag': '"03e2ffed23748e4e7fee925076e63b47"', 'Last-Modified': 'Sat, 18 Jul 2020 00:15:14 GMT', 'Access-Control-Expose-Headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Referrer-Policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'Content-Security-Policy': "default-src 'none'", 'X-GitHub-Request-Id': 'B6AC:EC41:82A5D:B3030:5F16AFFD'}
>>> response_2 = requests.get('http://api.github.com/repos/zappycode/zappycode-django/commits', headers={'Authorization': 'JustaString', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0', 'If-Modified-Since': 'Sat, 18 Jul 2020 00:15:14 GMT'})
>>> response_2.headers
{'Date': 'Tue, 21 Jul 2020 09:06:44 GMT', 'Server': 'GitHub.com', 'Status': '304 Not Modified', 'X-RateLimit-Limit': '60', 'X-RateLimit-Remaining': '59', 'X-RateLimit-Reset': '1595323695', 'Cache-Control': 'public, max-age=60, s-maxage=60', 'Vary': 'Accept, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding', 'ETag': '"03e2ffed23748e4e7fee925076e63b47"', 'Last-Modified': 'Sat, 18 Jul 2020 00:15:14 GMT', 'Access-Control-Expose-Headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Referrer-Policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'Content-Security-Policy': "default-src 'none'", 'X-GitHub-Request-Id': 'A962:EC4D:11CCB9AF:15D06EA5:5F16B023'}
>>> response_3 = requests.get('http://api.github.com/repos/zappycode/zappycode-django/commits', headers={'Authorization': 'JustaString', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0', 'If-Modified-Since': 'Sat, 18 Jul 2020 00:15:14 GMT'})
>>> response_3.headers
{'Date': 'Tue, 21 Jul 2020 09:07:39 GMT', 'Server': 'GitHub.com', 'Status': '304 Not Modified', 'X-RateLimit-Limit': '60', 'X-RateLimit-Remaining': '59', 'X-RateLimit-Reset': '1595323695', 'Cache-Control': 'public, max-age=60, s-maxage=60', 'Vary': 'Accept, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding', 'ETag': '"03e2ffed23748e4e7fee925076e63b47"', 'Last-Modified': 'Sat, 18 Jul 2020 00:15:14 GMT', 'Access-Control-Expose-Headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Referrer-Policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'Content-Security-Policy': "default-src 'none'", 'X-GitHub-Request-Id': 'A968:69B0:11A84908:159BBA46:5F16B05B'}
hope it's gonna work for u :)

According to the documentation X-Ratelimit-Remaining should have stayed at 59
This 59 here suggests that you're performing unauthenticated requests, which might be key here. I wonder if this is related to the rate limits for unauthenticated requests not being measured in the same way as authenticated requests, and this situation may not be considered a "conditional" request.
This case is not documented explicitly, but GitHub highly recommends using authenticated requests as these rate limits are tracked per-account, not per-IP address.

Related

Twitter API works on one IP but not the other

I am currently building a Twitter bot in Nodejs with twitter-api-v2 library. The below error is from the likes API that I use to like people's tweets.
When I first finished the app and deployed it to the server, it was working fine until it wasn't. It started to give me 429 Too Many Requests error despite that I have available rate limit left (As shown in the error below).
What is more interesting is that while it isn't working in my server, the app works fine on my localhost and it only throws 429 when the rate limit is used up. (I am NOT using docker)
The only difference I noticed about these two error responses is that on my server, it is reaching server tsa_p, but on my localhost it is reaching server tsa_b
What could this mean? Does Twitter API have hidden restriction on the amount of requests you can make per IP? Why can server tsa_b handle my request but tsa_p couldn't?
ApiResponseError: Request failed with code 429
at RequestHandlerHelper.createResponseError (/srv/twitterbots/auto_like_bot/node_modules/twitter-api-v2/dist/cjs/client-mixins/request-handler.helper.js:103:16)
at RequestHandlerHelper.onResponseEndHandler (/srv/twitterbots/auto_like_bot/node_modules/twitter-api-v2/dist/cjs/client-mixins/request-handler.helper.js:252:25)
at Gunzip.emit (node:events:513:28)
at endReadableNT (node:internal/streams/readable:1359:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
error: true,
type: 'response',
code: 429,
headers: {
date: 'Sat, 21 Jan 2023 03:08:24 UTC',
perf: '7626143928',
server: 'tsa_p',
'set-cookie': [
'guest_id_marketing=v1%3A167427050431236487; Max-Age=63072000; Expires=Mon, 20 Jan 2025 03:08:24 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None',
'guest_id_ads=v1%3A167427050431236487; Max-Age=63072000; Expires=Mon, 20 Jan 2025 03:08:24 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None',
'personalization_id="v1_vOZZLJVrhS4Q78smd3/U9g=="; Max-Age=63072000; Expires=Mon, 20 Jan 2025 03:08:24 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None',
'guest_id=v1%3A167427050431236487; Max-Age=63072000; Expires=Mon, 20 Jan 2025 03:08:24 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None'
],
'api-version': '2.61',
'content-type': 'application/json; charset=utf-8',
'cache-control': 'no-cache, no-store, max-age=0',
'content-length': '94',
'x-access-level': 'read-write',
'x-frame-options': 'SAMEORIGIN',
'content-encoding': 'gzip',
'x-transaction-id': '32cdfb3dfd64d27a',
'x-xss-protection': '0',
'x-rate-limit-limit': '50',
'x-rate-limit-reset': '1674271132',
'content-disposition': 'attachment; filename=json.json',
'x-content-type-options': 'nosniff',
'x-rate-limit-remaining': '46',
'strict-transport-security': 'max-age=631138519',
'x-response-time': '51',
'x-connection-hash': '5dbb0f4de195912f76293e11543a8685a6b1c296b5067ace43de9f665bbba298',
connection: 'close'
},
rateLimit: { limit: 50, remaining: 46, reset: 1674271132 },
data: {
title: 'Too Many Requests',
detail: 'Too Many Requests',
type: 'about:blank',
status: 429
}
}

Puppeteer code doesn't work on Google Cloud Virtual Machines but works perfectly on local machine

I recently started studying puppeteer, and to practice it I tried scraping www.myntra.com and the code was working fine on my local machine. I then decided to try and run it on a Virtual Machine on Google Cloud and initially it was working fine. Now after a couple of weeks, when I try running the code again on VM it doesn't load the website but the code still works as expected locally.
This image is captured using page.screenshot command when being run on the VM
const myntra_url = 'https://www.myntra.com/'
async function openBrowser()
{
//open browser
browser = await puppeteer.launch({
ignoreHTTPSErrors: true,
headless: true,
defaultViewport: null,
args: ['--no-sandbox']
});
page = await browser.newPage();
await page.setUserAgent(
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36'
)
await page.goto(myntra_url);
await page.screenshot({path: 'img.png', fullPage: true });
}
The above code snippet is what I used to capture the screenshot to check if the website was loading on Virtual Machines or not.
I also tried changing the IP of the Virtual Machine and even tried different providers like DigitalOcean and Hostinger
Edit: This is the HTTP response from the server while navigating using page.goto()
{ 'mime-version': '1.0',
'content-length': '483',
'cache-control': 'no-cache, no-store, must-revalidate',
pragma: 'no-cache',
expires: '0',
'content-type': 'text/html',
server: 'Apache/2.4.1 (Unix)',
date: 'Tue, 21 Dec 2021 05:53:33 GMT',
'set-cookie':
'AKA_A2=A; expires=Tue, 21-Dec-2021 06:53:33 GMT; path=/; domain=myntra.com; secure; HttpOnly\nakaas_myntra_SegmentationLabel=1642658013~rv=33~id=0e014616c2288ab0cbb4054d9cf995bb~rn=PWA; path=/; Expires=Thu, 20 Jan 2022 05:53:33 GMT; Secure; SameSite=None',
'server-timing': 'cdn-cache; desc=HIT\nedge; dur=1',
'strict-transport-security': 'max-age=15768000 ; includeSubDomains' }
Edit 2: This is the HTTP response while navigating using page.goto() when the code is run on my laptop
{
'content-type': 'text/html; charset=utf-8',
etag: 'W/"2325a-ra/cTi3uexrVThqYp3VqutNXiQU"',
'x-response-time': '21.313ms',
vary: 'Accept-Encoding',
'content-encoding': 'gzip',
'x-frame-options': 'SAMEORIGIN',
'x-akamai-transformed': '9 - 0 pmb=mTOE,4mRUM,3',
date: 'Thu, 23 Dec 2021 07:10:00 GMT',
'set-cookie': '_d_id=b4998f50-5039-4f36-b284-8dbdd6250308; Domain=.myntra.com; Path=/; Expires=Wed, 18 Sep 2024 07:10:00 GMT\n' +
'_mxab_=checkout.selective%3Denabled%3Bconfig.bucket%3Dregular%3Bpdp.desktop.savedAddress%3Denabled%3Bcheckout.couponUpsell%3Denabled%3Bcheckout.cartLoginNudge%3Denabled%3Bcheckout.attachedProducts%3Denabled%3Bcheckout.cartMessaging%3Denabled%3Bcheckout.donation%3Denabled%3Bpdp.expiry.date%3Denabled%3Bcheckout.addressOnCart%3Denabled; Domain=.myntra.com; Path=/; Expires=Thu, 23 Dec 2021 07:40:00 GMT\n' +
'_pv=default; Domain=.myntra.com; Path=/; Expires=Thu, 23 Dec 2021 07:15:00 GMT; HttpOnly\n' +
'dp=d; Domain=.myntra.com; Path=/; Expires=Thu, 23 Dec 2021 07:16:00 GMT; HttpOnly\n' +
'at=ZXlKaGJHY2lPaUpJVXpJMU5pSXNJbXRwWkNJNklqRWlMQ0owZVhBaU9pSktWMVFpZlEuZXlKdWFXUjRJam9pTlRkalpUQmtOekl0TmpOaVppMHhNV1ZqTFdJM1ltWXRNREF3WkROaFpqSTRZVFUzSWl3aVkybGtlQ0k2SW0xNWJuUnlZUzB3TW1RM1pHVmpOUzA0WVRBd0xUUmpOelF0T1dObU55MDVaRFl5WkdKbFlUVmxOakVpTENKaGNIQk9ZVzFsSWpvaWJYbHVkSEpoSWl3aWMzUnZjbVZKWkNJNklqSXlPVGNpTENKbGVIQWlPakUyTlRVM09UVTBNREFzSW1semN5STZJa2xFUlVFaWZRLmJCNmVNRlY0LUVTMVJuU1lpNDE4a2hodVN1M2V5d1lPYnlFZFotVjhBNE0=; Domain=.myntra.com; Path=/; Expires=Mon, 27 Jun 2022 07:10:00 GMT; HttpOnly\n' +
'bc=true; Domain=.myntra.com; Path=/\n' +
'microsessid=573; Domain=.myntra.com; Path=/; Expires=Thu, 23 Dec 2021 07:40:00 GMT; HttpOnly\n' +
'utm_track_v1=%7B%22utm_source%22%3A%22direct%22%2C%22utm_medium%22%3A%22direct%22%2C%22trackstart%22%3A1640243400%2C%22trackend%22%3A1640243460%7D; Domain=.myntra.com; Path=/; Expires=Thu, 23 Dec 2021 07:11:00 GMT; HttpOnly\n' +
'lt_timeout=1; Domain=.myntra.com; Path=/; Expires=Thu, 23 Dec 2021 07:40:00 GMT; HttpOnly\n' +
'lt_session=1; Domain=.myntra.com; Path=/; Expires=Thu, 23 Dec 2021 07:40:00 GMT; HttpOnly\n' +
'utrid=uuid-16402434004707; Domain=.myntra.com; Path=/; Expires=Tue, 21 Jun 2022 07:10:00 GMT; HttpOnly\n' +
'user_session=-1J5ucA1ht4_PXossxy10g.-CMVetdEuZC-KGhpMzXY0gOJ7DR6jOXF9-rwK4YyzK65vBn0_7ISlEFLXJfAJv9qbLiOBwwZRJXoJ5HUZZgcv-aNSiD_LsY1FxrXFJnaApMRquj8rRSxdXXpYPOkKYGiGiONQSyWwf-4tEzMsnfolw.1640243400468.86400000.73IBy-RLTmo1rot2RsOfgTEIr5tJLeeT1_FyRMFRAqg; path=/; expires=Thu, 23 Dec 2021 07:20:00 GMT; domain=.myntra.com; httponly\n' +
'AKA_A2=A; expires=Thu, 23-Dec-2021 08:10:00 GMT; path=/; domain=myntra.com; secure; HttpOnly\n' +
'akaas_myntra_SegmentationLabel=1642835400~rv=60~id=0dc3e9a01677da83cf8b1359fe95e1b8~rn=PWA; path=/; Expires=Sat, 22 Jan 2022 07:10:00 GMT; Secure; SameSite=None\n' +
'bm_mi=13A16C8EAD963EAA2178C4F10B835209~8jOw4j5/LNPx+Q64n4D1NoWnn3ANCmd7Z4xMfHjDxlyoP/0VmwnxNb8I9nFsRMJva9YZAPyEF1aU1RINWxx2r1u74EQK56d5hdLrTPpHAlpZ3ZBqyUfV9C3cGvIQAQlxzM1dK9yaq5zPp5JsEXcH9JLNWC/Bc++MAqDMi8jRsShFobMIBPVvpYwhVtLhb8xBg4PRZ69Y4I0ObztcinLxeQfilVz/zF/cekSRzHz46ZlCouyk5uwYJj/a3v2u5zNl; Domain=.myntra.com; Path=/; Max-Age=0; HttpOnly\n' +
'_abck=5FFD2B9DAA2A989A9E57E7E539BB9DE3~-1~YAAQpP3UF9mC2Mp9AQAAOQ8g5gfphOk+bERcC6JoVEJoqW71i3fgcSJL3yjDaOCCz45XCMVwKR04u27A3fhtRwARgb27lGA3/QM+wgjLqiVT+5zd7jsgyFLsy8S8Rve1bF6/VexmGC7Fn0QwqPJWBArDQJDZF0fyUoyNvpr3Guitxotxole3CD/MIkXU8IOgxuceFUADNoUIyGVqZ1XQh0RI8/b1sWw4hzRGU5rDB5QueIDisubwy9WRdsGilEuAfIDRBuetDb1GyOQ8/rwda1xNbv3Y4BDrAPuEawInhOcSoJbfksBLWXPU32dIGGbuk/HjmK08D6CO++iL5xIVKkj+Z6xDfeAjeJDrDc3mXdPFP+PUKO8LKSBfTbA=~-1~-1~-1; Domain=.myntra.com; Path=/; Expires=Fri, 23 Dec 2022 07:10:00 GMT; Max-Age=31536000; Secure\n' +
'ak_bmsc=87F76D36C1615D7A6A6C701D5CBF81B2~000000000000000000000000000000~YAAQpP3UF9qC2Mp9AQAAOQ8g5g6AXDGBK/udhQzN/jsXcOO/7g40KWGKMpeEfsBdLQLDk0O+nglUDKyLFmRwL79g7NRjo8QxCwHuUWSpk40pU1wVXuVnAshCXisFS3pa6HiIMBsUAj9qnwIKySTRl/C1HObUQreKDM0lFc0KMa/acDpZwj00mcz6IKbHw5sphcPSK7H0AFHPYf51HlcFrT42if6E/4tzdLp9tqSZglUIfZtwQpyUZK4PeuodafZWYNZ1oP+WQOIO36O6BkljvCxNoIYbzL6reY9MM1WAQKhQb5DscMMsnASqqSj6vB3gA2OVJzURiBNuODP+W85DKFK6kndlPIt/UaKMXiiwKC+9Xwy04HwmtQcktKYmyXsAlBIw33vG0Dg0; Domain=.myntra.com; Path=/; Expires=Thu, 23 Dec 2021 09:10:00 GMT; Max-Age=7200; HttpOnly\n' +
'bm_sz=65769455C92599BAC6D7C6AA3AA996B6~YAAQpP3UF9uC2Mp9AQAAOQ8g5g4htUDLSyu98f+YExGJg9JPORO/I76qfr6lTo0eHrtHrC0rFq/hjeibhI1rKH2G4h58mEs/7cJotRBmtvPSaT1CDSz5B6kK+uiKumyeBhCzBOscwj/CCYsZojnQkHfVoZJ/bmV8cJ0M2fmjhY7Jzw7M6P9G90qMjxh5/+OOmlD0QcqOg4a6gxBFT3MFEFYQyHF6pDtL92DHqgAoKQJrblUZ0oFQLbu5L9TkD6atkzAZI3Zhqn5lAqT/K7AWKA3LRFXlWaatlvU6HPmTNO8O5yA=~3553605~4338245; Domain=.myntra.com; Path=/; Expires=Thu, 23 Dec 2021 11:10:00 GMT; Max-Age=14400',
'server-timing': 'cdn-cache; desc=MISS\nedge; dur=61\norigin; dur=30',
'strict-transport-security': 'max-age=15768000 ; includeSubDomains'
}

mongodb-runner does not work(Response code 403)

So, I wanted to setup my mongoDB + Parse Server on my Kali Linux server.
I followed the instructions on this Git-Repo: https://github.com/parse-community/parse-server
When typing npm install -g parse-server mongodb-runner
and mongodb-runner start I get the following log:
TPError: Response code 403 (Forbidden)
at EventEmitter.ee.on.res (/usr/local/lib/node_modules/mongodb-runner/node_modules/got/index.js:250:24)
at EventEmitter.emit (events.js:182:13)
at Immediate.setImmediate (/usr/local/lib/node_modules/mongodb-runner/node_modules/got/index.js:99:8)
at runCallback (timers.js:694:18)
at tryOnImmediate (timers.js:665:5)
at processImmediate (timers.js:647:5)
name: 'HTTPError',
host: 'fastdl.mongodb.org',
hostname: 'fastdl.mongodb.org',
method: 'GET',
path: '/linux/mongodb-linux-x86_64-4.2.8.tgz',
protocol: 'https:',
url:
'https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.2.8.tgz',
statusCode: 403,
statusMessage: 'Forbidden',
headers:
{ 'content-type': 'application/xml',
'transfer-encoding': 'chunked',
connection: 'close',
date: 'Mon, 15 Jun 2020 10:17:34 GMT',
server: 'AmazonS3',
'x-cache': 'Error from cloudfront',
via:
'1.1 6b4954a8411e7b2a232537f8000c5c9d.cloudfront.net (CloudFront)',
'x-amz-cf-pop': 'FRA50-C1',
'x-amz-cf-id': 'af_MHWQpIoBDL77-FBQHI5BED9OTX2E_kOGfHNg-LF9ujCGiCUGW2g==' } }
{ HTTPError: Response code 403 (Forbidden)
at EventEmitter.ee.on.res (/usr/local/lib/node_modules/mongodb-runner/node_modules/got/index.js:250:24)
at EventEmitter.emit (events.js:182:13)
at Immediate.setImmediate (/usr/local/lib/node_modules/mongodb-runner/node_modules/got/index.js:99:8)
at runCallback (timers.js:694:18)
at tryOnImmediate (timers.js:665:5)
at processImmediate (timers.js:647:5)
name: 'HTTPError',
host: 'fastdl.mongodb.org',
hostname: 'fastdl.mongodb.org',
method: 'GET',
path: '/linux/mongodb-linux-x86_64-4.2.8.tgz',
protocol: 'https:',
url:
'https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.2.8.tgz',
statusCode: 403,
statusMessage: 'Forbidden',
headers:
{ 'content-type': 'application/xml',
'transfer-encoding': 'chunked',
connection: 'close',
date: 'Mon, 15 Jun 2020 10:17:34 GMT',
server: 'AmazonS3',
'x-cache': 'Error from cloudfront',
via:
'1.1 6b4954a8411e7b2a232537f8000c5c9d.cloudfront.net (CloudFront)',
'x-amz-cf-pop': 'FRA50-C1',
'x-amz-cf-id': 'af_MHWQpIoBDL77-FBQHI5BED9OTX2E_kOGfHNg-LF9ujCGiCUGW2g==' } }
I dont know what to do. Did I install any wrong packages or something like that?

Premium requests against Foursquare API fail with 429 and "quota_exceeded" error

I have a problem making "premium" requests against Foursquare API. There's an app that has been doing requests for a while now, but ceased to do so a few days ago.
Here's how it looks like on a sample venue (from API explorer):
In [47]: requests.get('https://api.foursquare.com/v2/venues/49eeaf08f964a52078681fe3', params={'client_id': settings.FOURSQUARE_CLIENT_ID, 'client_secret': settings.FOURSQUARE_CLIENT_SECRET, 'v': '20180611'})
2018-06-11 14:37:03,221:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.foursquare.com
2018-06-11 14:37:03,616:DEBUG:urllib3.connectionpool:https://api.foursquare.com:443 "GET /v2/venues/49eeaf08f964a52078681fe3?client_secret=<SNIP>&client_id=<SNIP>&v=20180611 HTTP/1.1" 429 134
Out[47]: <Response [429]>
In [49]: dict(_47.headers)
Out[49]:
{'Accept-Ranges': 'bytes',
'Access-Control-Allow-Origin': '*',
'Connection': 'keep-alive',
'Content-Length': '134',
'Content-Type': 'application/json; charset=utf-8',
'Date': 'Mon, 11 Jun 2018 14:37:03 GMT',
'Server': 'nginx',
'Tracer-Time': '1',
'Via': '1.1 varnish',
'X-Cache': 'MISS',
'X-Cache-Hits': '0',
'X-RateLimit-Limit': '0',
'X-RateLimit-Path': '/v2/venues/X',
'X-RateLimit-Remaining': '0',
'X-Served-By': 'cache-hhn1536-HHN'}
In [50]: _47.json()
Out[50]:
{u'meta': {u'code': 429,
u'errorDetail': u'Quota exceeded',
u'errorType': u'quota_exceeded',
u'requestId': u'5b1e890f6a60717ee10bf7cd'},
u'response': {}}
The response is clearly related to rate limits, but it doesn't follow the description outlined in https://foursquare.com/dev/overview/ratelimits or https://developer.foursquare.com/docs/api/troubleshooting/rate-limits. More importantly, it shows 'X-RateLimit-Limit': '0' suggesting there's either no limit or that it's entirely forbidden.
User-based access with tokens created by users of our applications fail in the same manner.
User-based access with a token copied from API explorer surprisingly works (but displays the same 'X-RateLimit-Limit': '0' as the failing response above):
In [56]: requests.get('https://api.foursquare.com/v2/venues/49eeaf08f964a52078681fe3', params={'oauth_token': sample_oauth_token, 'v': '20180611'})
2018-06-11 14:35:00,417:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.foursquare.com
2018-06-11 14:35:01,066:DEBUG:urllib3.connectionpool:https://api.foursquare.com:443 "GET /v2/venues/49eeaf08f964a52078681fe3?oauth_token=<SNIP>&v=20180611 HTTP/1.1" 200 None
Out[56]: <Response [200]>
In [57]: dict(_57.headers)
Out[57]:
{'Accept-Ranges': 'bytes',
'Access-Control-Allow-Origin': '*',
'Connection': 'keep-alive',
'Content-Encoding': 'gzip',
'Content-Type': 'application/json; charset=utf-8',
'Date': 'Mon, 11 Jun 2018 14:35:01 GMT',
'Server': 'nginx',
'Strict-Transport-Security': 'max-age=31536000',
'Tracer-Time': '257',
'Transfer-Encoding': 'chunked',
'Vary': 'Accept-Encoding,User-Agent,Accept-Language',
'Via': '1.1 varnish',
'X-Cache': 'MISS',
'X-Cache-Hits': '0',
'X-Rate-Limit-Key': '77.49.114.206',
'X-RateLimit-Limit': '0',
'X-RateLimit-Path': '/v2/venues/X',
'X-RateLimit-Remaining': '0',
'X-Served-By': 'cache-hhn1530-HHN',
'X-ex': 'fastly_cdn'}
Finally, non-premium requests work with our application credentials without a problem:
In [59]: requests.get('https://api.foursquare.com/v2/venues/49eeaf08f964a52078681fe3/similar', params={'client_id': settings.FOURSQUARE_CLIENT_ID, 'client_secret': settings.FOURSQUARE_CLIENT_SECRET, 'v': '20180611'})
2018-06-11 14:50:33,378:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.foursquare.com
2018-06-11 14:50:33,790:DEBUG:urllib3.connectionpool:https://api.foursquare.com:443 "GET /v2/venues/49eeaf08f964a52078681fe3/similar?client_secret=<SNIP>&client_id=<SNIP>&v=20180611 HTTP/1.1" 200 None
Out[59]: <Response [200]>
In [60]: dict(_59.headers)
Out[60]:
{'Accept-Ranges': 'bytes',
'Access-Control-Allow-Origin': '*',
'Connection': 'keep-alive',
'Content-Encoding': 'gzip',
'Content-Type': 'application/json; charset=utf-8',
'Date': 'Mon, 11 Jun 2018 14:50:33 GMT',
'Server': 'nginx',
'Strict-Transport-Security': 'max-age=31536000',
'Tracer-Time': '18',
'Transfer-Encoding': 'chunked',
'Vary': 'Accept-Encoding,User-Agent,Accept-Language',
'Via': '1.1 varnish',
'X-Cache': 'MISS',
'X-Cache-Hits': '0',
'X-Rate-Limit-Key': '77.49.114.206',
'X-RateLimit-Limit': '10000',
'X-RateLimit-Path': '/v2/venues/X/similar',
'X-RateLimit-Remaining': '9993',
'X-Served-By': 'cache-hhn1529-HHN',
'X-ex': 'fastly_cdn'}
Any idea what is the quota/rate limit problem here and how to fix it?

Socket.io handshake address value changes

This is peculiar. Socket.io version ~1.3
io.sockets.on('connection', function (socket) {
console.log('Client connected from: ' + socket.handshake.address);
}
Returns
Client connected from: ::1
However
io.sockets.on('connection', function (socket) {
console.log(socket.handshake);
console.log('Client connected from: ' + socket.handshake.address);
}
Returns
{ headers:
{ host: 'localhost:8000',
connection: 'keep-alive',
origin: 'http://localhost:3000',
'user-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTM
L, like Gecko) Chrome/43.0.2357.130 Safari/537.36',
accept: '*/*',
dnt: '1',
referer: 'http://localhost:3000/dev.html',
'accept-encoding': 'gzip, deflate, sdch',
'accept-language': 'en-US;q=0.8,en;q=0.6,ko;q=0.4,de;q=0.2,ru;q=0.2,fr;q=0.2,ja;q=0.2,it;q=0.2',
cookie: 'io=yhyuAabou3GufhzNAAAA' },
time: 'Wed Jun 24 2015 22:50:19 GMT+0200 (Central European Daylight Time)',
address: '::ffff:127.0.0.1',
xdomain: true,
secure: false,
issued: 1435179019584,
url: '/socket.io/?EIO=3&transport=polling&t=1435179017804-3',
query: { EIO: '3', transport: 'polling', t: '1435179017804-3' } }
Client connected from: ::ffff:127.0.0.1
Why? Is there some ES6 proxy in the way? I thought maybe some weird JS conversion magic was in place, but it doesn't seem like it.
::ffff:127.0.0.1 is an IPv6 version of 127.0.0.1 and ::1 is an IPv6 shortcut for both.
See Express.js req.ip is returning ::ffff:127.0.0.1 for a similar question.

Resources