javax.net.ssl.SSLHandshakeException: error:10000438:SSL routines:OPENSSL_internal:TLSV1_ALERT_INTERNAL_ERROR - apache-pulsar

Question

I'm unable to connect to my Pulsar test cluster from my local environment, but it was working fine yesterday. Nothing has changed on the client-side or server-side. When I try to connect from my app, I get this exception:
2021-08-24T17:07:55,776 [pulsar-io-23-15] WARN org.apache.pulsar.broker.service.ServerCnx - [/10.16.13.41:23586] Got exception io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000438:SSL routines:OPENSSL_internal:TLSV1_ALERT_INTERNAL_ERROR
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: error:10000438:SSL routines:OPENSSL_internal:TLSV1_ALERT_INTERNAL_ERROR
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1007)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1271)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1225)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1296)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1339)
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1340)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
... 15 more```
Restarting the app does not resolve the issue.
We're using Pulsar token-auth with a pretty vanilla configuration:
client = PulsarClient.builder().serviceUrl(serviceUrl)
.tlsTrustCertsFilePath(serverCertificateFilePath.toString())
.enableTlsHostnameVerification(false)
.allowTlsInsecureConnection(false)
.authentication(AuthenticationFactory.token(authToken))
.build();
The consumer is also pretty standard (though it has a DLQ) and looks like this:
pulsarClient.newConsumer(Schema.STRING)
.consumerName(String.format("%s:%s", consumerProperties.getSubscriptionName(), UUID.randomUUID()))
.topic(consumerProperties.getDeadLetterTopic())
.subscriptionName(consumerProperties.getSubscriptionName())
.subscriptionType(SubscriptionType.Shared)
.messageListener(deadLetterTopicMessageListener)
.ackTimeout(consumerProperties.getAcknowledgeTimeout(), TimeUnit.SECONDS)
.subscribe();
We've double-checked that the token and certificates are all correct.
Also, the broker tlsProtocol includes v1, as follows:
tlsProtocols=TLSv1.2,TLSv1.1,TLSv1
How do I resolve this issue?

Answer 1

We've seen some edge cases where client VPN issues result in this error message.
Some users have reported that this issue is resolved after restarting, clearing their docker cache, and re-connecting to the VPN.
One user reported that it went away after multiple restarts, so it's possible that something times out eventually.