I have created a REST web service using Web API 2.2 on a Windows Server 2008 R2 box running IIS 7.5. The problem that I'm having is that the web service is returning a compressed response (Content-Encoding: gzip) when I make the request through the Google Chrome Postman application. But when I make the same request using the .NET 4.5.1 HttpClient, the server does not return a compressed response (the Content-Encoding header is blank). Here is my C# code:
var handler = new HttpClientHandler();
handler.UseProxy = false;
handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
handler.Credentials = CredentialCache.DefaultNetworkCredentials;
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.AcceptCharset.Add(new StringWithQualityHeaderValue("utf-8"));
client.DefaultRequestHeaders.AcceptLanguage.Add(new StringWithQualityHeaderValue("en-US"));
client.DefaultRequestHeaders.CacheControl = new CacheControlHeaderValue { NoCache = true };
client.DefaultRequestHeaders.Connection.Add("keep-alive");
var response = await client.GetAsync("https://localhost/mywebsite");
Note: I'm using an SSL connection. I can confirm that the Web API web service is receiving the Accept-Encoding: gzip header from both the Postman application request and the HttpClient request. In fact, the request headers are exactly the same for both, except that the Connection: keep-alive header seems to be stripped from the HttpClient request. Does anyone have any idea why the web service won't serve a compressed response to the HttpClient?
So, I monitored the HTTP traffic using Fiddler and lo and behold the server response was in fact compressed when using the HttpClient (the number bytes received by Postman was the same as the number of bytes received by HttpClient) and had sent the corresponding Content-Encoding: gzip header! I guess that the HttpClient is trying to be smart by removing the Content-Encoding: gzip header when it is in automatic decompression mode. Is this documented anywhere?
you fiddler decode auto,you can disable fiddler auto decode
Related
General context:
I am working on an IoT application where I upload images from an ESP32 connected to an SBC.
The uploading is done through an API provided by a third-party backend developer.
The upload API works through other mediums (such as Postman, python requests library, python http client library)
The ESP32 is connected to the SBC through UART.
I construct/generate the HTTP request on the SBC and send it as bytes. I have written a function on ESP32 that can send the bytes as a generic HTTP request, to the URL specified.
Then it sends the response string back to the SBC.
All of this works. For small requests, I am facing no issues. I am able to download images, etc.
However, when uploading an image, I don't get a response and I end up timing out after 30s. I checked without timeout FYI, but no response.
I checked from the server-side. It appears my request has succeeded and the server is sending me 200 with the URL of the image. Using that URL, I was able to verify that the image was uploaded successfully.
However, I do not receive this response on the microcontroller.
Not sure what the issue is. Any suggestions as to what I can do?
I can't give out the code but I'll send a general structure:
ESP32
-> Receives URL, port, length of request
-> Connects to server and reads the request from UART and writes to server
-> Wait for response after response is sent
Python raw http
POST (server path) HTTP/1.1
Host: (url)
correlation-id: test5
Content-Type: multipart/form-data; boundary=WebKitFormBoundary7MA4YWxkTrZu0gW
Authorization: Bearer (access token)
Content-Length: 268
--WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="portraits"; filename="name"
Content-Type: image/jpeg
(data)
--WebKitFormBoundary7MA4YWxkTrZu0gW--
Edit 1:
So, turns out it is not only "upload image", some other requests are also behaving similarly. Our server has many microservices. The services written in nodeJS which have more than 1 redirects are not working...?
I figured out what the issue is and hopefully, it will help anyone else facing the same issue. Also, some of my requests to the backend server which used a different authentication method worked
I had been generating the raw HTTP using postman code generation but it turns out Postman doesn't add a few headers which are needed for communicating with more complex servers.
What I mean is that if I host a local server, the above code will work. I had already tested it that way
What solved my problem is adding these headers:
POST (server path) HTTP/1.1
Host: (server URL)
User-Agent: ESP32
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
correlation-id: test
Authorization: Bearer (access_token)
Content-Length: 146360
Content-Type: multipart/form-data; boundary=af59ef02d60cd0efefb7bc03db1f4ffc
--af59ef02d60cd0efefb7bc03db1f4ffc
Content-Disposition: form-data; name="portraits"; filename="(name)"
Content-Type: image/jpeg
(data)
--af59ef02d60cd0efefb7bc03db1f4ffc--
I have setup an express/serverless application to retrieve a pdf file on a GET request. But I just retrieve a corrupted repsonse pdf response. I just wondering If my settings are correct to achieve a correct response.
I'm using aws-serverless-express and want to return my pdf buffer to the client browser (it should open in the browser)
My code:
status = 200;
let fileName = "demo.pdf";
res.setHeader('Content-disposition', 'inline; filename="' + fileName + '"');
res.setHeader('Content-type', 'application/pdf');
res.setHeader('isBase64Encoded', true);//isBase64Encoded: true
let pdf = pdfBuffer.toString('base64');
res.status(status).send(pdf);
so I'm sending a base64 encoded string to APIGW. I'm not actually sure if I can set the isBase64Encoded flag via header. I read this before but I'm not so certain about that
I have done this whole procedure before, but didn't make use of aws-serverless-express (where I Could set the isBase64Encoded flag easily)
I'm also using serverless-apigw-binary to automatically setup APIGW for the correct decoding of the base64 encoded data
lambda is automatically encoding to base64, so I had to remove it and directly send the buffer.
I came across similar problem while using serverless Express. AWS Gateway needs a http response like this:
{
"header":{....}
"body": "/9j/4AAQSkZ...",
"isBase64Encoded": true
}
It's useless to put isBase64Encoded in the Http Response Header, AWS API Gateway only checks if the isBase64Encoded is outside the Http header. If it's true, then decode the body before sending it to a HTTP client.
Express (its response object) doesn't seem to allow to add something outside HTTP Header in a Http Response.
If you don't want to give up using Express, the workaround is to do the decoding in the browser, which is pretty easy:
var decodedData = Buffer.from(body,'base64')
I'm using security headers middleware in a web app to add security headers to all outgoing http requests. Security headers seem to get added to all network requests to internal resources - that is resources that make up the web app such as the javascript scripts and the images used in the web app and the css and html files. However the security headers do not get added to any external http requests such as to an API that I made that the web app uses to get json data. How do I make it just add security headers to everything, rather than just to the web apps own resources?
Below is some of the relevant code that adds security headers middleware
startup.cs
private ILogger<SecurityHeadersBuilder> _logger;
private readonly SecurityHeadersPolicy _policy = new SecurityHeadersPolicy();
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory, ISecurityHeadersBuilder securityHeadersBuilder)
{...
app.UseSecurityHeadersMiddleware(
securityHeadersBuilder.AddDefaultSecurePolicy()
);
securityHeadersBuilder.cs
public SecurityHeadersBuilder AddDefaultSecurePolicy()
{
AddFrameOptionsDeny();
AddXssProtectionBlock();
AddContentTypeOptionsNoSniff();
AddNoCache();
AddStrictTransportSecurityMaxAgeIncludeSubDomains();
AddContentSecurityPolicyAllContentFromSelfAndGoogle();
RemoveServerHeader();
return this;
}
public SecurityHeadersBuilder AddFrameOptionsDeny()
{
_policy.SetHeaders[FrameOptionsConstants.Header] = FrameOptionsConstants.Deny;
_logger.LogInformation(string.Format("setting {0} http header value to {1}", FrameOptionsConstants.Header, FrameOptionsConstants.Deny));
return this;
}
There are two type of headers: request headers and _response headers.
The server sets response headers to instruct the browser how to handle a response (block iframing for example).
Therefore it would not make sense to do a request with (for example) the header X-Frame-Options : Deny. Because the client application could alter the value and ignore the security restriction. The server will not handle the value of the header anyway, the user-agent of the browser will use this response header.
If you do a call to an (external) API you should manually add request headers to an HttpClient and make the call. The API in turn can return the (security) response headers.
All the headers that you have in the example code are response headers and should not be set as request headers.
In java, cookies (present as a part of request headers) can be accessed using cookie store:
HttpClient httpClient = new DefaultHttpClient();
// execute get/post/put or whatever
httpClient.doGetPostPutOrWhatever();
// get cookieStore
CookieStore cookieStore = httpClient.getCookieStore();
// get Cookies
List<Cookie> cookies = cookieStore.getCookies();
How to access cookie store or request headers in node.js application?
I am using XMLHttpRequest in the node.js based application. However, it does not support anything like getCookieStore(). xhr.getResponseHeader('Cookie') does not work as cookies are not present as part of response.
When using node.js as a client, is it possible to connect to a server using Windows integrated authentication (e.g. when connecting to IIS)?
My searches for this only turn up results where node.js is used as a server.
2015 Update: There are now some modules that implement Windows-integrated authentication. node-sspi uses SSPI (the Windows security API) to handle the server side of things, but does not do client auth. There are several client implementations such as http-ntlm, but they are not truly integrated since they require the user password -- they do not use SSPI to do transparent auth.
2019 Update: It appears to be possible to use the kerberos library to do true Windows-integrated HTTP auth using SSPI (ie, use the node process' token to do transparent auth). See kerberos-agent. Obviously this uses Kerberos rather than NTLM/Negotiate, so this may or may not work depending on your exact situation.
"Windows integrated authentication" is what's known as NTLM authentication. When you receive a HTTP 401 from IIS with a WWW-Authenticate header containing NTLM, you now have the fun of implementing the NTLM authentication protocol. Quoting from this document about the NTLM authentication protocol:
The client requests a protected resource from the server:
GET /index.html HTTP/1.1
The server responds with a 401 status, indicating that the client must authenticate. NTLM is presented as a supported authentication mechanism via the WWW-Authenticate header. Typically, the server closes the connection at this time:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: NTLM
Connection: close
Note that Internet Explorer will only select NTLM if it is the first mechanism offered; this is at odds with RFC 2616, which states that the client must select the strongest supported authentication scheme.
The client resubmits the request with an Authorization header containing a Type 1 message parameter. The Type 1 message is Base-64 encoded for transmission. From this point forward, the connection is kept open; closing the connection requires reauthentication of subsequent requests. This implies that the server and client must support persistent connections, via either the HTTP 1.0-style "Keep-Alive" header or HTTP 1.1 (in which persistent connections are employed by default). The relevant request headers appear as follows:
GET /index.html HTTP/1.1
Authorization: NTLM TlRMTVNTUAABAAAABzIAAAYABgArAAAACwALACAAAABXT1JLU1RBVElPTkRPTUFJTg==
The server replies with a 401 status containing a Type 2 message in the WWW-Authenticate header (again, Base-64 encoded). This is shown below.
HTTP/1.1 401 Unauthorized
WWW-Authenticate: NTLM TlRMTVNTUAACAAAADAAMADAAAAABAoEAASNFZ4mrze8AAAAAAAAAAGIAYgA8AAAARABPAE0AQQBJAE4AAgAMAEQATwBNAEEASQBOAAEADABTAEUAUgBWAEUAUgAEABQAZABvAG0AYQBpAG4ALgBjAG8AbQADACIAcwBlAHIAdgBlAHIALgBkAG8AbQBhAGkAbgAuAGMAbwBtAAAAAAA=
The client responds to the Type 2 message by resubmitting the request with an Authorization header containing a Base-64 encoded Type 3 message:
GET /index.html HTTP/1.1
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAGoAAAAYABgAggAAAAwADABAAAAACAAIAEwAAAAWABYAVAAAAAAAAACaAAAAAQIAAEQATwBNAEEASQBOAHUAcwBlAHIAVwBPAFIASwBTAFQAQQBUAEkATwBOAMM3zVy9RPyXgqZnr21CfG3mfCDC0+d8ViWpjBwx6BhHRmspst9GgPOZWPuMITqcxg==
Finally, the server validates the responses in the client's Type 3 message and allows access to the resource.
HTTP/1.1 200 OK
You'll have to figure out how you'll reply to the Type 2 message's challenge, where the user's password is MD4 hashed and used to create DES keys to encrypt the challenge data.
I'm not sure how you'd get access to the logged in user's credential data which would allow you to accomplish this, though I'm sure it would involve writing a native C++ addon so you could talk to the necessary Windows API. Or, I suppose you could just ask for the user's password.
Alternatively, you could proxy your Node requests through software that handles the NTLM mess for you.
For Kerberos:
node-sspi
Just on windows
No client side node
Supports NTLM too
passport-negotiate
Needs python on the server
it's a passportJs strategy
For NTLM
node-sspi
Just on windows
No client side node
Supports Kerberos too
httpntlm
express-ntlm
request-ntlm
ntlm
experimental project!
ntlm-auth
experimental!
passport-ntlm
supports SMB protocol
it's a passportJs strategy
I chose passport-negotiate for Kerberos and express-ntlm for NTLM
For client side, what works is to use node-libcurl to do REST / HTTP calls.
here's sample code:
var endpoint = urlString;
var url = require("url");
var endpointUrl = url.parse(endpoint);
var Curl = require( 'node-libcurl' ).Curl;
var curl = new Curl();
curl.setOpt( 'USERNAME', '' );
//curl.setOpt( 'VERBOSE', 1 );
curl.setOpt( 'URL', endpoint );
curl.setOpt( 'HTTPAUTH', Curl.auth.NEGOTIATE );
curl.setOpt( 'NOPROXY', endpointUrl.hostname );
curl.on( 'end', function( statusCode, body, headers ) {
if (statusCode === 200) {
console.log(body);
cb(null, { statusCode, body, headers } );
} else {
cb(new Error(), { statusCode, body, headers } );
}
this.close();
});
curl.on( 'error', curl.close.bind( curl ) );
curl.perform();