Error while making http request from spark - apache-spark

I am using internal service which uses org.asynchttpclient library for HTTP request. From my Spark app when a remote http call is made I am seeing Http 408. The request goes through haproxy, and looks like the request fails to complete with following error
java.lang.IllegalAccessError: tried to access method io.netty.channel.DefaultChannelId.<init>()V from class org.asynchttpclient.netty.channel.Channels
at org.asynchttpclient.netty.channel.Channels.initChannelId(Channels.java:67)
at org.asynchttpclient.netty.request.NettyChannelConnector$1.onSuccess(NettyChannelConnector.java:90)
at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:26)
at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)

Related

Azure Functions Python BadHttpRequestException - Reading the request body timed out

We have deployed an Azure Function App with the Python 3.8 stack. For some requests, we have received a BadHttpRequestException
Exception while executing function: Functions.***
Exception binding parameter 'req' Reading the request body timed out
due to data arriving too slowly. See MinRequestBodyDataRate.
But this exception is thrown from ASP .Net core . We are not sure what is causing this exception, network connectivity seems to fine.
Is it something to do with request payload size, if it is big and it resulting in slowness.
In Python Http Trigger Azure function where do set the MinRequestBodyDataRate

httpClient throws Connection reset while invoking https connection from azure reverse proxy

Actually I'am having saml application that's configured to be redirect to keycloak
when first I configured the application with HTTP URLs(httpClient 3.1,jdk1.8) I didn't receive any problem but when using the reverse proxy (WAF azure application gateway reverse proxy) for HTTPS connection an exception is appeared
**Caused by**: org.opensaml.saml2.metadata.provider.MetadataProviderException: Error retrieving metadata from https://example.com/auth/realms/realmName/protocol/saml/descriptor
at org.opensaml.saml2.metadata.provider.HTTPMetadataProvider.fetchMetadata (HTTPMetadataProvider.java:274)
at org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider.refresh (AbstractReloadingMetadataProvider.java:255)
... 9 more
**Caused by**: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.opensaml.saml2.metadata.provider.HTTPMetadataProvider.fetchMetadata(HTTPMetadataProvider.java:250)
Knowing that I have already added the certificate at the JVM level and the URL I just called (https://example.com/auth/realms/realmName/protocol/saml/descripto)is accessible through the curl command or from the browser
I was able to solve the same problem by upgrading my jdk version

Getting HTTP error 400- Header too long error

I am getting HTTP 400- Header too long error, I have tried most of solution over internet, but none of them seemed to be working (solutions are like adding reg entries, etc).
I have a sample web application which is calling a web api on my system, but I am getting below error in HTTPERR log file:
2019-11-16 16:58:59 ::1%0 1213 ::1%0 80 HTTP/1.1 GET
/WebApplication2/api/values - 400 - RequestLength -
Here is my code which is calling the web API:
HttpWebRequest GETRequest = (HttpWebRequest)WebRequest.Create(URL);
GETRequest.Method = "GET";
GETRequest.ContentType = "application/json";
GETRequest.Headers.Add("Authorization", "<Something Big value>");
WebResponse GETResponse = await GETRequest.GetResponseAsync();
It is recommended to capture ETL log via
https://blogs.msdn.microsoft.com/wndp/2007/01/18/event-tracing-in-http-sys-part-1-capturing-a-trace/
Then you could use network monitor to analyze both request payload and etl log.
It will help you find the root cause.
Besides, can you find the 400 error in IIS log? Now that the 400 error appear in httperr log. It looks like http.sys blocked the request before it reach IIS server.
If you can find the 400 error in IIS , then you could try to modify the limitation in system.web/httpRuntime and request filter.
If the 400 error only can be found in httperr log. Then you may have to try to make some change in http.sys registry.
https://support.microsoft.com/en-us/help/820129/http-sys-registry-settings-for-windows
Please remember to reboot server to activate these registry!

Http Inbound Gateway takes 2x timeout to send timeout response to client

We're using an http inbound gateway to process http requests. Lately , we wanted to set specific http response code that will be sent to user after a timeout.
We upgraded to spring 4.2.1 and set
the reply-timeout-status-code-expression attribute
and it's working now.
But there is a problem : if we set the timeout to 10 seconds , a ReplyTimeoutException is raised in the web application after exactly 10 seconds but the response arrives to the client after 20 seconds ( exactly the double ).
We changed the timeout several times but it always take 2x timeout for the response to be received by client.
This is the xml config for the http inbound gateway :
<int-http:inbound-gateway id="inboundRequest" request-channel="inbound-channel"
supported-methods="POST" path="/Request" error-channel="errorChannel" reply-channel="gateway_response"
reply-timeout="5000" request-payload-type="java.lang.String" reply-timeout-status-code-expression="'504'" >
A Help would be really appreciated
I see you have an error-channel - if the error flow returns no response, the reply timer starts again when the error flow ends (we're waiting for a reply from the error flow).
Instead of returning nothing from the error flow, you need to throw a MessageTimeoutException.
Turning on DEBUG logging and following the message flow is the best way to diagnose issues like this.
If that's not the problem, you need to show the rest of your configuration (edit the question, don't try to post it in a comment).

Spring Integration Http Inbound Gateway Request Logging

Hi I am currently using Spring Integration 4.0.3.I have a requirement of logging the actual request which are coming to the Http Inbound Gateways.The request are to be logged before they are converted into a message by the Http Inbound Messaging Gateway and sent to the gateway request channel.This is how I am trying to achieve this .
Configured a aop pointcut on the HttpRequestHandlingMessagingGateway class handleRequest method.
Configured a custom method Interceptor as an advice and configured the pointcut to execute this advice in the aop config.
In the MethodInterceptor I am retreiving the Request and Response object and logging the request.
But it seems whenever I am applying the aop on handleRequest method the dispatcher servlet is unable to find the handler for any of the url mapping used by all of my http inbound gateways.
I know the same can be achieved through configuring filters in web.xml but I am just curious why this is happening....Please help ...
Actually you can't proxy HttpRequestHandlingMessagingGateway#handleRequest because it is final.
From other side there is no reason to do so complex AOP work for that logging stuff.
HttpRequestHandlingMessagingGateway populates RequestContextHolder.currentRequestAttributes() as EvaluatationContext variable, so you can put it to MessageHeader and use anywhere in downstream flow:
<int-http:inbound-gateway path="/foo" request-channel="requestChannel">
<int-http:header name="requestAttributes" expression="#requestAttributes"/>
</int-http:inbound-gateway>
<publish-subscribe-channel id="requestChannel"/>
<logging-channel-adapter channel="requestChannel"
expression="headers.requestAttributes.request"/>

Resources