API GET request not reflecting changes in DB, delayed by 5min - node.js

I am trying to make an web-app that notifies when new vaccine slots arrive on government portal using provided public APIs.
What i need is to call the API every minute and check if the slots have been added to the database. But the response I am getting is stale as the new sessions detected by my app(also in Chrome) were about 5 minutes old, I know this because some telegram channels are showing update earlier than my app.
Also, when I try to hit the same API with Postman, the response I am getting is fresh.
Issue is - Chorme/myApp response is not reflecting the updated database... but postman is showing the updated one... chrome is getting the updated response 5 mins after its showing in postman.
Public API: https://cdn-api.co-vin.in/api/v2/appointment/sessions/public/calendarByDistrict?district_id=141&date=06-07-2021
let response = await fetch(`https://cdn-api.co-vin.in/api/v2/appointment/sessions/public/calendarByDistrict?district_id=${id}&date=${today}`, {
method: 'GET',
headers: {
'Content-Type': 'application/json',
'Connection': 'keep-alive',
},
})
Do I need to change some headers or anything else in my get requests?... or anything else???
Help, me fix it...

So couple of things.
First, use Find by district API instead of Calendar by district API. Thats more accurate.
https://cdn-api.co-vin.in/api/v2/appointment/sessions/public/findByDistrict?district_id=512&date=31-03-2021
Second, pass the user agent. This is in PHP, but you can always update to other language.
$header = array(
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Pragma: no-cache",
"Cache-Control: no-cache",
"Accept-Language: en-us",
"User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15",
"Upgrade-Insecure-Requests: 1"
);

Related

Request blocked if it is sent by node,js axios

I am using axios and a API (cowin api https://apisetu.gov.in/public/marketplace/api/cowin/cowin-public-v2) which has strong kind of protection against the web requests.
When I was getting error 403 on my dev machine (Windows) then, I solve it by just adding a header 'User-Agent'.
When I have deployed it to heroku I am still getting the same error.
const { data } = await axios.get(url, {
headers: {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36',
},
})
Using a fake user-agent in your headers can help with this problem, but there are other variables you may want to consider.
For example, if you are making multiple HTTP requests you may want to have multiple fake user-agents to and then randomize the user-agent for every request made. This can help limit the changes of your scraper being detected.
If that still doesn't work you may want to consider optimizing your headers further. Other than sending HTTP requests with a randomized user-agent, you can further imitate a browser's request Headers by adding more Headers than just the "user-agent"- then ensuring that the user-agent that is selected is consistent with the information sent from the rest of the headers.
You can check out here for more information.
On the site it will not only provide information on how to optimize your headers consistently with the user-agent, but also provide more solutions in case the above mentioned still was unsuccessful.
In my situation, it was the case that I had to bypass cloudflare. You can determine if this is your situation as well if you log your error to the terminal and then check if under the "server" key it says "cloudflare". In which case you can use this documentation for further assistance.

Getting 403 forbidden status through python requests

I am trying to scrape a website content and getting 403 Forbidden status. I have tried solutions like using sessions for cookies and mocking browser through a 'User-Agent' header. Here is the code I have been using
session = requests.Session()
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36',
}
page = session.get('https://www.sizeofficial.nl/product/zwart-new-balance-992/343646_sizenl/', headers = headers)
Note that this approach works on other websites, it is just this one which does not seem to work. I have even tried using other headers which my browser is sending them, and it does not seem to work. Another approach I have tried is to first create a session cookie and then pass that cookie to session.get, still doesn't work for me. Is it not allowed to scrape the website or am I still missing something?
I am using python 3.8 requests to achieve this purpose.

Powershell Invoke-WebRequest difference with NodeJS fetch API

I've been trying to extract some data from a website, but the only way I'm able to get something useful is through Powershell.
The script I'm running from Powershell is:
Invoke-WebRequest -Uri "https://www.pelispedia.tv/api/iframes.php?id=18471?nocache" -Headers #{"method"="GET"; "authority"="www.pelispedia.tv"; "scheme"="https"; "path"="/api/iframes.php?id=18471?nocache"; "upgrade-insecure-requests"="1"; "user-agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36"; "accept"="text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"; "referer"="https://www.pelispedia.tv/pelicula/el-nino-que-domo-el-viento/"; "accept-encoding"="gzip, deflate, br"; "accept-language"="es,en;q=0.9"} | Select-Object -Expand Content
I got it from Chromes's Network tab inside the DevTools while watching this site load: https://www.pelispedia.tv/pelicula/el-nino-que-domo-el-viento/
Devtools Screenshot - also includes cURL and fetch
The response is a full HTML site, which I want to use later.
The fetch script is:
fetch("https://www.pelispedia.tv/api/iframes.php?id=18471?nocache", {
"credentials": "include",
"headers": {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"accept-language": "es,en;q=0.9",
"upgrade-insecure-requests": "1"
},
"referrer": "https://www.pelispedia.tv/pelicula/el-nino-que-domo-el-viento/",
"referrerPolicy": "no-referrer-when-downgrade",
"body": null,
"method": "GET",
"mode": "cors"
})
.then(res => res.text())
.then(body => console.log(body));
I tried using multiple NodeJS packages like node-fetch, axios and request to get the same result as in Powershell, but I simply get an HTML with the line "ERROR".
This approach does not work in NodeJS, but if I run it from within Chrome's console, while I'm from the site, it works.
I would like to know what Powershell is doing to get the correct response and how to recreate it in Node or any other language/runtime (Java, Python, PHP...).
Using fetch form chrome dev tools and using fetch from node or using Powershell are completely different things.
fetch form chrome dev tools has all the headers and other thing attached to the request as the browser does so it is essentially your browser making the request as perceived by the server of the website.
But in case of PowerShell or nodejs request or fetch, all those headers, referer, and many other things are stripped off. So the server rejects the request considering you a bot.

Granting READER role access to a Subscription in Azure works fine in Postman, but not via Angular. Why?

OK, I might be missing something simple here in Angular, but I could really use some help. I am trying to grant a Service Principal READER role to a Subscription programmatically. If I use PostMan, it works fine. However, when I send the same PUT request via Angular6 I get a 400 error from Azure that says:
The content of your request was not valid, and the original object
could not be deserialized. Exception message: 'Required property
'permissions' not found in JSON. Path 'properties', line 1, position
231.'
The JSON being sent in both cases is:
{
"properties":
{
"roleDefinitionId":"/subscriptions/{some_subscription_guid}/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7",
"principalId":"{some_service_provider_guid}"
}
}
I've captured traffic from both requests, and they show as application/json payloads on the PUT. So I am at a loss of what is deserializing incorrectly through Azure that is causing this error. I am trying to follow the REST instructions documented here: https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-rest
Any ideas what I am missing?
UPDATE
Adding the RAW REQUEST per request. I have replaced any sensitive data (access token, GUIDs etc) without changing anything else from the Fiddler output.
PUT https://management.azure.com/subscriptions/<VALID_SUBSCRIPTION_WAS_HERE>/providers/Microsoft.Authorization/roleDefinitions/7ec2aca1-e4f2-4152-aee2-68991e8b48ad?api-version=2015-07-01 HTTP/1.1
Host: management.azure.com
Connection: keep-alive
Content-Length: 233
Accept: application/json, text/plain, */*
Origin: http://localhost:4200
Authorization: Bearer <VALID_TOKEN_WAS_HERE>
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/json
Referer: http://localhost:4200/token/<VALID_DOMAIN_WAS_HERE>.onmicrosoft.com/graph
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
{"properties": { "roleDefinitionId":"/subscriptions/<VALID_SUBSCRIPTION_GUID_HERE>/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7", "principalId":"<VALID_OBJECTID_HERE>" }}
Alright, I finally figured out what was going on here. It appears that I was posting to the wrong endpoint. I need to be posting to roleAssignment and not roleDefinitions.
So why did it work in PostMan? It seems there is a fallback from a previous version of the API that supported both when using legacy clients, which for some reason PostMan fell under. However, when posting via Angular it was actively rejecting it.
End result... send to "/Microsoft.Authorization/roleassignments/" with an API version later than "api-version=2015-07-01". All will work.

how to using cookie with request (request , tough-cookie , node.js)

I'm wondering to know how to using cookie with request (https://github.com/mikeal/request)
I need to set a cookie which able to be fetched for every sub domains from request,
something like
*.examples.com
and the path is for every page, something like
/
then server-side able to fetch the data from cookie correctly, something like
test=1234
I found the cookies which setup from response was working fine,
I added a custom jar to save the cookies, something like
var theJar = request.jar();
var theRequest = request.defaults({
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36'
}
, jar: theJar
});
but the cookies which I setup from request, only able to be fetched in same domain,
and I can't find a method to setup cookie in more options
for now if I want one cookie which able to be fetched in three sub domains,
I have to setup like this way:
theJar.setCookie('test=1234', 'http://www.examples.com/', {"ignoreError":true});
theJar.setCookie('test=1234', 'http://member.examples.com/', {"ignoreError":true});
theJar.setCookie('test=1234', 'http://api.examples.com/', {"ignoreError":true});
Is here any advance ways to setup a cookie from request,
made it able to be fetched in every sub domains ???
I just found the solution ....
theJar.setCookie('test=1234; path=/; domain=examples.com', 'http://examples.com/');
hm...I have to say, the document which for request is not so good..., lol

Resources