RestTemplate.exchange: Locking threads - multithreading

Question

I am using RestTemplate to talk to APIs. Recently i encountered this issue, where almost 90% of the threads are in waiting state.
p685934-2328" #2328 prio=5 os_prio=0 tid=0x00000000023b6000 nid=0x2f1f waiting on condition [0x00007f6df703f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ab0eaae8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:380)
at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
- locked <0x00000000e728c1e8> (a org.apache.http.pool.AbstractConnPool$2)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:723)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:680)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600)
RestTemplate is defined like below.
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory();
clientHttpRequestFactory.setConnectTimeout( 3000 );
clientHttpRequestFactory.setReadTimeout( 3000 );
SSLContext sslContext = SSLContexts.custom()
.loadTrustMaterial((X509Certificate[] chain, String authType) -> true)
.build();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
sslContext, (String hostname, SSLSession session) -> true);
clientHttpRequestFactory.setHttpClient(HttpClients.custom()
.setSSLSocketFactory(sslsf).build());
RestTemplate restTemplate = new RestTemplate(clientHttpRequestFactory);
Additional Information
Spring boot version : 2.0.3.RELEASE
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.5</version>
</dependency>
More than 500 requests per minute will go from RestTemplate.
Any help on what went wrong with this API.
will RestTemplate do not close the socket in any scenario?
Thanks,
Hari

Answer 1

I think the reason 90% of the threads are in a locking state is that the threads are blocked / waiting for the API to respond. The RestTemplate is blocking or synchronous per the documentation:
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/client/RestTemplate.html
You could switch to the asynchronous or non-blocking WebClient and you should get much better throughput per thread:
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/reactive/function/client/WebClient.html
Another option would be to spawn a thread per connection if that is possible but I think the asynchronous client will scale better if you have the ability to port your code.
Cheers,
Kyle