I have written a client which tries to connect to Azure service bus. As soon as the server starts up i get the below errors and i receive no messages present at the queue. I tried replacing the sb protocol with amqpwss, but it dint help.
2020-05-25 21:23:11 [ReactorThreadeebf108d-444b-4acd-935f-c2c2c135451d] INFO c.m.a.s.p.RequestResponseLink - Internal send link 'RequestResponseLink-Sender_0480eb_c31e1cc239bf471e811e53a30adc6488_G51' of requestresponselink to '$cbs' encountered error.
com.microsoft.azure.servicebus.primitives.ServiceBusException: com.microsoft.azure.servicebus.amqp.AmqpException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'LinkTracker'. TrackingId:c31e1cc239bf471e811e53a30adc6488_G51, SystemTracker:gateway7, Timestamp:2020-05-25T21:23:10
at com.microsoft.azure.servicebus.primitives.ExceptionUtil.toException(ExceptionUtil.java:55)
at com.microsoft.azure.servicebus.primitives.RequestResponseLink$InternalSender.onClose(RequestResponseLink.java:759)
at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.processOnClose(BaseLinkHandler.java:66)
at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:42)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
at com.microsoft.azure.servicebus.primitives.MessagingFactory$RunReactor.run(MessagingFactory.java:491)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.azure.servicebus.amqp.AmqpException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'LinkTracker'. TrackingId:c31e1cc239bf471e811e53a30adc6488_G51, SystemTracker:gateway7, Timestamp:2020-05-25T21:23:10
... 10 common frames omitted
There is a similar issue opened in GitHub
what you posted here is the trace, not the error. Yes, the service
closes idle connections are 10 minutes. The client traces it and
reopens the connection. It is seamless, doesn't throw any exceptions
to the application. That can't be your problem. If your sends are
failing means there may be another problem, but not this one.
As i see the second line it is about the timeout of 6 secs, can you check the troubleshoot page if it helps. Also this.
we recommend adding "sync-publish=true" to the connection url
Related
I have a basic Pulsar app, and when I try to connect to Pulsar, I get this exception:
2021-03-10 14:38:26.107 WARN 7 --- [r-client-io-1-1]
o.a.pulsar.client.impl.ConnectionPool : Failed to open connection
to my-pulsar-server-ms-tls.domain.com:6651 :
io.netty.channel.ConnectTimeoutException: connection timed out:
my-pulsar-server-ms-tls.domain.com/10.80.13.38:6651 2021-03-10
14:38:26.212 WARN 7 --- [al-listener-3-1]
o.a.pulsar.client.impl.PulsarClientImpl : [topic:
persistent://myTenant/myNamespace/myTopic]
Could not get connection while getPartitionedTopicMetadata -- Will try
again in 100 ms
My Pulsar client is pretty basic:
PulsarClient.builder()
.serviceUrl(serviceUrl)
.authentication(AuthenticationFactory.token(authToken))
.tlsTrustCertsFilePath(serverCertificateFilePath.toString())
.enableTlsHostnameVerification(false)
.allowTlsInsecureConnection(false)
.build();
The producer is also pretty basic and looks like this:
pulsarClient.newProducer(Schema.STRING)
.topic(topic)
.create();
I've verified that the token and TLS cert are correct. I've also tried connecting a consumer from this same environment and got a similar exception, and I know that others with the same code are able to connect to the same Pulsar cluster from other environments. What is the issue?
Your connection is getting blocked by a firewall or network issue.
Verify that you can establish a connection to your endpoint my-pulsar-server-ms-tls.domain.com:6651 from your environment.
If you're able to run a network packet dump (like tcpdump), that should make it obvious if you're not able to establish a connection.
You can also try running curl my-pulsar-server-ms-tls.domain.com:6651, and if you get back some html, that means you were able to reach the server. However, if you get Could not resolve host, then you were blocked by the network configuration (such as a missing route) or firewall.
I have a Blazor server side app published on IIS 10.
When browsing to an arbitrary page and just letting it idle after a minute or so (sometimes only 45 sec, sometimes something between 1 and two minutes) the modal
Attempting to reconnect to server ...
appears for a couple of seconds.
In the browser console the logging shows either
Error: Connection disconnected with error 'Error: Server timeout
elapsed without receiving a message from the server.'.
or
Information: Connection disconnected.
Since this seems to be a timeout problem I added the following options to ConfigureServices in my startup.cs
services.AddServerSideBlazor()
.AddHubOptions(options =>
{
options.ClientTimeoutInterval = TimeSpan.FromMinutes(10);
options.KeepAliveInterval = TimeSpan.FromSeconds(3);
options.HandshakeTimeout = TimeSpan.FromMinutes(10);
});
This does not solve the problem though.
I also went to the advanced settings of my site in IIS and increased the connection timeout from the default 120 sec to 600 sec. This did not help either.
Those frequent disconnections only happen on the live site hosted on IIS 10.
If I start the app locally with Visual Studio the connection is stable.
Any hints of what I'm missing would be appreciated!
Update:
As suggested by #agua from mars in comment below I changed transport type like this
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
endpoints.MapBlazorHub(options => { options.Transports = HttpTransportType.LongPolling; });
endpoints.MapFallbackToPage("/_Host");
});
With this change the connection is still closed. The console log shows
Information: (LongPolling transport) Poll terminated by server.
I also tried HttpTransportType.ServerSentEvents which does not work at all but gives this error
Error: Failed to start the connection: Error: Unable to connect to the
server with any of the available transports. ServerSentEvents failed:
Error: 'ServerSentEvents' does not support Binary.
Update 2:
The IIS is configured to use HTTP 1.1
I tried changing to HTTP/2 but this did not change anything regarding the disconnections.
This is related to application pool recycling in IIS as stated by #Programmer. You can reproduce this by going into the application pool, right click the pool and choose recycle to force it. Your blazor app will get the "reconnect modal screen".
For me, I did not want to disable pool recycle, so I added js in the _Hosts.cshtml file as
<script>Blazor.defaultReconnectionHandler._reconnectCallback = function (d) {document.location.reload();}</script>
to automatically reconnect when the server comes back up.
Try this out..
app.UseEndpoints(endpoints =>
{
//other settings
.
.
endpoints.MapBlazorHub(options => options.WebSockets.CloseTimeout = new TimeSpan(1, 1, 1));
//other settings
.
.
});
This could be related to IIS application pool recycling. Try disabling the recycling to see if that's casing the disconnection.
I suffer the same problem on my Blazor server too: Myspector.com
I am sure this comes from network of data provider. I use Othello in Germany with 4G and see disconnection in 5 sec . When I am with wifi with t online on same target server no disconnection at all.
I Think some operators are incompatible with Blazor server/websoscket....
My recent experience especially on a shared server, increase the pool memory. Connectivity issues went away when we bumped 256MB up to 1GB for a small user base.
I have some code written to communicate with an azure service bus. It sends messages to a queue. It's in a project targeting .net standard 2.0
When I run it from a .net core terminal app it runs fine. But, when the same code is called from a .net framework 4.7.2 project then the first attempt to send a message results in the following exception after 30 to 90 seconds:
"The remote party closed the WebSocket connection without completing the close handshake."
But any further messages will be sent without problem.
// This is using Microsoft.Azure.ServiceBus, if that makes any difference...
MessageSender MessageSender = new MessageSender(ConnectionString, SendQueueName;
try
{
await MessageSender.SendAsync(new Message(Encoding.UTF8.GetBytes("Test that won't work")));
}
catch(Exception e)
{
// Error will be caught here:
// "The remote party closed the WebSocket connection without completing the close handshake."
}
await MessageSender.SendAsync(new Message(Encoding.UTF8.GetBytes("Test that will work")));
Does anybody know why the first call fails? And how to make it not fail? Or fail quicker? I've tried changing the OperationTimeout and RetryPolicy but they don'e seem to have any effect.
These first connections are via port 5671/56712, which Trend antivirus intercepts. Once these have timed out then the framework falls back to using 443, which works fine.
We tried turning Trend off and running testing the connection and its pretty much instantaneous.
My app just disappears from the list of consumers in RabbitMQ Admin after working just fine for like 30-40 mins. AMQP lib used: node-amqp. Here's the connection:
const con = amqp.createConnection(options,{defaultExchangeName: 'amq.topic', reconnect: true})
The following event handlers are configured too: connect, ready, close, tag.change, error
The worst part is that i don't get any errors or close events, app just disconnects and logs nothing...
It just seems that connection is terminated out of being 'idle' for a while...
Has anyone had something similar? How did you deal with it?
Perhaps this helps someone. To resolve the issue we have to put heartbeat field to options and specify the interval in seconds the connection has to be checked and refreshed.
The heartbeat is doesn't have any default values, so if it is not explicitly added, amqp won't use it.
i have a problem on startup of BAM server.
My machine has the IP 1.33.33.127 and hostname "srv-lc-presen".
I it have configurated using this document:
Monitoring and statistics.
I have modified the at carbon.xml. I have it set to 1.
I've modified the master-datasources.xml and set
WSO2BAM_CASSANDRA_DATASOURCE url = jdbc:cassandra://srv-lc-presen:9161/EVENT_KS
WSO2BAM_UTIL_DATASOURCE url = jdbc:cassandra://srv-lc-presen:9161/BAM_UTIL_KS
I have tried with localhost, 1.33.33.127 and srv-lc-presen.
I always get the same error:
ERROR {me.prettyprint.cassandra.connection.HConnectionManager} - Could not start connection pool for host srv-lc-presen(1.33.33.127):9161
[2014-05-07 12:04:24,983] WARN {me.prettyprint.cassandra.connection.CassandraHostRetryService} - Downed srv-lc-presen(1.33.33.127):9161 host still appears to be down: Unable to open transport to srv-lc-presen(1.33.33.127):9161 , java.net.ConnectException: Connection refused
[2014-05-07 12:04:24,987] ERROR {org.wso2.carbon.bam.notification.task.internal.NotificationDispatchComponent} - All host pools marked down. Retry burden pushed out to client.
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:393)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249)
at me.prettyprint.cassandra.service.ThriftCluster.addKeyspace(ThriftCluster.java:168)
at org.wso2.carbon.bam.datasource.utils.DataSourceUtils.createKeyspaceIfNotExist(DataSourceUtils.java:80)
at org.wso2.carbon.bam.datasource.utils.DataSourceUtils.getClusterKeyspaceFromRDBMSConfig(DataSourceUtils.java:92)
at org.wso2.carbon.bam.datasource.utils.DataSourceUtils.getClusterKeyspaceFromRDBMSDataSource(DataSourceUtils.java:96)
NEW information
i have tried to reconfigure and i don't find the problem.
I see in BAM console this error
[2014-05-08 09:10:57,531] ERROR {me.prettyprint.cassandra.connection.HConnectionManager} - Could not start connection pool for host 1.33.33.127(1.33.33.127):9161
[2014-05-08 09:10:57,564] ERROR {org.wso2.carbon.bam.notification.task.internal.NotificationDispatchComponent} - All host pools marked down. Retry burden pushed out to client.
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:393)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249)
at me.prettyprint.cassandra.service.ThriftCluster.addKeyspace(ThriftCluster.java:168)
at org.wso2.carbon.bam.datasource.utils.DataSourceUtils.createKeyspaceIfNotExist(DataSourceUtils.java:80)
at org.wso2.carbon.bam.datasource.utils.DataSourceUtils.getClusterKeyspaceFromRDBMSConfig(DataSourceUtils.java:92)
at org.wso2.carbon.bam.datasource.utils.DataSourceUtils.getClusterKeyspaceFromRDBMSDataSource(DataSourceUtils.java:96)
at org.wso2.carbon.bam.notification.task.internal.NotificationDispatchComponent.initRecordStore(NotificationDispatchComponent.java:72)
at org.wso2.carbon.bam.notification.task.internal.NotificationDispatchComponent.activate(NotificationDispatchComponent.java:64)
And in API Manager console this
[2014-05-08 09:14:52,096] ERROR - ReceiverGroup No receiver is reachable at reconnection, can't publish the events
[2014-05-08 09:14:55,102] ERROR - AsyncDataPublisher Reconnection failed for for tcp://1.33.33.127:7612/
Please use this command at startup or edit wso2server.sh if you are not using notification feature sh wso2server.sh -Ddisable.notification.task
https://docs.wso2.org/display/BAM240/Notifications