Is it possible to identify the client / library which sent a HTTP request?
I am trying to fetch some data via an API and it is possible to query the API via cURL and python, but when I try to use node (doesn't matter which library, axios requests, unirest, native, ...) or wget I get a proprietary error back from the backend.
Now I am wondering, if the backend is able to identify, which library I am using?
More information:
The requests are exactly the same, so no way to distinguish them
The user-agent header field is set and overwritten for all requests
I already tried to monitor the traffic in wireshark, but couldn't find any differences with the packets on HTTP layer (only the order of the header fields is different, that according to the standard this shouldn't make a difference)
It turns out that the problem was TLS fingerprinting.
See: https://httptoolkit.tech/blog/tls-fingerprinting-node-js/
Nodejs uses google V8 JS engine, V8 based http request clients will not allow you to override headers that would compromise 'web safety', so for example if you are setting "Origin, Host, Referrer" headers, node might refuse to do so. I had the same issue previously.
Un-opinionated http clients, such as the ones written in C++(curl) and python won't 'web safety' check your requests, so that is what is causing the difference in behavior.
In my case I used a C++ library that I called from javascript to make my 'unsafe' requests and the problem was solved.
Related
I’ve to make a post request to a service (not implemented with ServiceStack). From the docs, please correct me if I am wrong, I have to use HTTPUtils nuget package (v. 6.0.2), but if I make a request using its extensions the service returns a 400 bad request.
The same request done using RestSharp (v.105.0) works.
However, I noticed that I had to use an old version compared to the available version of RestSharp.(nothing changes if I downgrade ServiceStack).
Could it be that the service implementation is not compatible with the latest versions of RestSharp and ServiceStack?Is it correct to use HTTPUtils for a service that I don't know if it's implemented with ServiceStack?
Does ServiceStack add some extra wrapper to the .NET framework HTTP client?
Thanks in advance
Here are the docs for ServiceStack's HTTP Utils which can be used for calling generic HTTP APIs, which are extension methods in the ServiceStack.Text NuGet package.
Receiving a 400 Bad Request response suggests that you're sending an invalid request.
Whenever you're investigating issues calling HTTP APIs you should be inspecting the HTTP Traffic with a HTTP tool like WireShark or Fiddler so you can verify that it's sending the HTTP Request you want to send, whilst Postman is a useful tool for quickly working out the HTTP Request you want to send.
If you want help with using a tool you'll need to post the C# source code you're using, the HTTP Request/Response it's sending and the HTTP Request you want to send. Typically the HTTP Response should contain information on why your request is invalid.
My current setup for the project I'm working on is:
Next.js
Wordpress backend with GraphQL plugin enabled
They live on two different servers
I would like to be able to make a request from a Next.js page that proxy via an api-route to the Wordpress backend. I want the GraphQL query to be passed along and I would like to be able to modify the request (for example add header, set a cookie etc) before it reaches the Wordpress backend.
I first tried to achieve this using this module: https://github.com/http-party/node-http-proxy and using the .web() request. It almost worked except I got back a response from Wordpress that I wasn't able to decode (tried with Buffer etc, but no success).
So my current way to do this is to make an axios-request from my api route and pass along the req.body in that request, and that setup works.
However, is this way to proxy OK or should I try to make it work with node-http-proxy? Don't know about what possible benefits there are.
Thank you
if you use Axios you will make an extra request when you retrieve data from the source. This will decrease performance. On the other side if you use proxy you will forward the incoming request and this way you will have improved performance.
I'm looking to validate URLs by whitelisting a certain list which are allowed. I am using a Node.js server with express.js.
I've looked into using the Node.js library "validator", but this means I need access to the full URL that the request was sent from. However, if the user searches for https://localhost:8080/index.html I have found no way of handling that request, so that I can pass it through the validator library and thus accept or reject it.
You can do so using the NPM package 'express-validator' which contains a whitelist sanitation chain API.
https://github.com/validatorjs/validator.js#sanitizers
I'm currently using a framework in Node.js ( the botbuilder module from Microsoft Bot Framework) which uses the request[2] module to make HTTP requests.
I'm encountering a problem : this framework seems to send a malformed JSON to Microsoft's servers, but I fail to see why and what is this JSON message made of.
So I'm looking for a way to log those messages, to take a peek at this malformed JSON, as I don't have access to the request object (unless I heavily alter the framework code, which is not something one shall do)
So far, I'm able to log the response body (by adding request to the NODE_DEBUG environment variable), but not the original request body. I did try a tcpdump on our server but since it's all HTTPS there's nothing I can use there.
Any idea / tool that might help ?
Thanks for your time.
Use Node.js middleware to log all your requests. For example, you could use the module request-debug.
Another popular request logging middleware worth knowing about is Morgan, from the Express.js server team.
I have developed a front-end interface using Aja(AngularJS) and HTML5. Right now, I send an HTTP get request to my backend server which returns some data based on the GET parameters.
Since the URL is exposed in the Javascript file, I believe anyone could just use the URL to create there own API to fetch the data. How can I prevent such things ?
One way I could think of is that now instead of directly sending the request to the backend server, an application server could be used (hosting the HTML as well). The Ajax request would then be sent to this server (PHP script ?) which would in turn forward the request to the backend server and return the result to the UI. To prevent 3rd party services, I can disable cross origin requests on my application server.
Is this the correct way to solve my problem or are there better ways to do this? I am concerned that this would unnecessarily create another hop (internal though) for requests.
Note: The backend is running Apache Tomcat
In APIs that are not open to the world the user has to authenticate first in order to use it, see for example https://stripe.com/docs/api#authentication or http://dev.maxmind.com/geoip/geoip2/web-services/ -> Authorization