I am running Mirth 3.7.1 on a VM within Azure. The Mirth database is on a SQL Server managed instance within the same Azure subscription. I have several channels which consume ADT/ORM messages which seem to be working as expected, however, I also have a File Reader channel which reads PDF files from disk and sends them as MDM messages. This channel is intermittently erroring (see stack traces below) in what appears to me to be with its connection to the Mirth DB. I am assuming that this is due to the fact that it is attempting to save out the larger file data as it moves through the steps in the channel since the ADT/ORM channels are not having the same issue. We had this same channel running in a traditional environment and we did not see this same problem. Any thoughts on how to resolve this issue?
Also, I have alerts configured to send email when an error occurs. I am recieving these when the error is within the channel, but I am not being notified of these internal Mirth errors. Is there any way that I can be notified?
Mike
com.mirth.connect.donkey.server.channel.ChannelException: com.mirth.connect.donkey.server.data.DonkeyDaoException: java.sql.SQLException: I/O Error: Connection reset
at com.mirth.connect.donkey.server.channel.Channel.dispatchRawMessage(Channel.java:1213)
at com.mirth.connect.donkey.server.channel.SourceConnector.dispatchRawMessage(SourceConnector.java:192)
at com.mirth.connect.donkey.server.channel.SourceConnector.dispatchRawMessage(SourceConnector.java:170)
at com.mirth.connect.connectors.file.FileReceiver.processFile(FileReceiver.java:354)
at com.mirth.connect.connectors.file.FileReceiver.processFiles(FileReceiver.java:247)
at com.mirth.connect.connectors.file.FileReceiver.poll(FileReceiver.java:203)
at com.mirth.connect.donkey.server.channel.PollConnectorJob.execute(PollConnectorJob.java:49)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)Caused by: com.mirth.connect.donkey.server.data.DonkeyDaoException: java.sql.SQLException: I/O Error: Connection reset
at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.insertContent(JdbcDao.java:274)
at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.insertMessageContent(JdbcDao.java:193)
at com.mirth.connect.donkey.server.data.buffered.BufferedDao.executeTasks(BufferedDao.java:110)
at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:85)
at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:72)
at com.mirth.connect.donkey.server.channel.Channel.dispatchRawMessage(Channel.java:1185)
... 8 moreCaused by: java.sql.SQLException: I/O Error: Connection reset
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1093)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:563)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:727)
at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.insertContent(JdbcDao.java:271)
... 13 moreCaused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.DataInputStream.readFully(Unknown Source)
at java.io.DataInputStream.readFully(Unknown Source)
at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:850)
at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:731)
at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:477)
at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:114)
at net.sourceforge.jtds.jdbc.ResponseStream.peek(ResponseStream.java:99)
at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:4127)
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1086)
... 16 more
Azure Monitor has capabilities to monitor Azure VM health but only the following perf counters are included for Windows VMs and Linux VMs.
However, Using Azure Monitor for VMs (preview) Map to understand application components there is an ability to create application maps that allow you to monitor specific aspects of an application environment and trigger alerts. For example, you can set-up a map for failed connections for both processes and connections.
Related
During heavy load my consumption azure functions are timing out with the following errors-
1.System.InvalidOperationException : An exception has been raised that is likely due to a transient failure. Consider enabling transient error resiliency by adding 'EnableRetryOnFailure()' to the 'UseSqlServer' call. ---> System.Data.SqlClient.SqlException : The client was unable to establish a connection because of an error during connection initialization process before login. Possible causes include the following: the client tried to connect to an unsupported version of SQL Server; the server was too busy to accept new connections; or there was a resource limitation (insufficient memory or maximum allowed connections) on the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) ---> System.ComponentModel.Win32Exception : An existing connection was forcibly closed by the remote host.
2.System.InvalidOperationException : Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
We are using Azure SQL database P4 with 500 DTUs. My initial thought was that due to less available worker threads it might be failing. But, they are well within limit with max at 12%.
We know that some of out LINQ queries are slow and are not performing well, but that would require business logic change.
Is there any solution on Azure Infrastructure side or any logs I can look into to?
We had first problem couple of month ago, we just added EnableRetryOnFailure() on database configuration, it resolved the first issue. Sample code given below
var optionsBuilder = new DbContextOptionsBuilder<KasDbContext>();
optionsBuilder.UseSqlServer(getConnectionString(), options =>
{
options.EnableRetryOnFailure(maxRetryCount: Constants.MaxRetryCountOnDbTransientFailure, maxRetryDelay: TimeSpan.FromSeconds(Constants.MaxDelaySecondsOnDbTransientFailure), errorNumbersToAdd: null);
});
return new DbContext(optionsBuilder.Options);
I installed Apache pulsar standalone. Pulsar get timeout sometimes. It's not related to high throuput neither to a particular topic (following log). Pulsar-admin brokers healthcheck returns OK or timeout also. How to investigate on it ?
10:46:46.365 [pulsar-ordered-OrderedExecutor-7-0] WARN org.apache.pulsar.broker.service.BrokerService - Got exception when reading persistence policy for persistent://nnx/agent_ns/action_up-53da8177-b4b9-4b92-8f75-efe94dc2309d: null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) ~[?:1.8.0_232]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[?:1.8.0_232]
at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0]
at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$32(BrokerService.java:922) ~[org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
at org.apache.bookkeeper.mledger.util.SafeRun$2.safeRun(SafeRun.java:49) [org.apache.pulsar-managed-ledger-2.5.0.jar:2.5.0]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
I am glad you were able to resolve the issue my adding more cores. The issue was a connection timeout while trying to access some topic metadata that is stored inside of ZookKeeper as indicated by the following line in the stack trace:
at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0]
Increasing the cores must of freed up enough threads to allow the ZK node to respond to this request.
You can check the connection to the server looks like connection issue if you are using any TLScertificate file path check if you have the right certificate.
The Problem is we don't have lot of solutions in internet for apache pulsar but if you are following the apache pulsar doc might help and also we have apache pulsar git hub and sample projects.
I am deploying a batch spark job on Kubernetes on GKE.
Job tries to fetch some data from MySQL (Google Cloud SQL) but it gives connection link failure.
I tried manually connecting to mysql by installing mysql client from pod and it connected fine.
is there any additional thing which I need to configure?
Exception:
Exception in thread "main" com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:590)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:57)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:1606)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:633)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:347)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:219)
at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:63)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:54)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:56)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:210)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
Issue was actually with the firewall rules in GCP. Working fine now.
When i ran my test I got the below error no the response section of View Result Tree.
I was testing a IIS server.
Is this Connection reset created by JMeter of was it done IIS server.
How do you interpret this error message.
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:475)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:295)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1105)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1094)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:429)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257)
at java.lang.Thread.run(Unknown Source)
There are at least 3 possibles reasons:
Most probable cause : Your server (meaning web servers handling request and any components after them) is not handling the load correctly and slowing down, monitor the system and check
You have exhausted your injector ephemeral ports , you need to adjust your OS TCP settings to increase port range
You're running load test in GUI mode with a View Results Tree in test, this is bad practice as GC will happen frequently possibly triggering Stop The World leading to this. As per best-practices use NON GUI mode:
https://jmeter.apache.org/usermanual/best-practices.html
https://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
While opening IBM Cognos Administration you can receive error "CNC-MON-0024 Monitor Service paging error: The Network Adapter could not establish the connection."
The issue was resolved when the cognos connection was restarted.
I didn't understand why the error had occurred. Can someone please explain why this error had occurred.
Thank you
There can be many reason why it's happening:
Getting error CNC-MON-0024 Monitor Service paging error: No suitable driver
CNC-MON-0024 Monitor Service paging error
Launching the Cognos Administration Page displays an error
From my expirience, usually it's usually either incorect configuration or corrupted notification DB.
In order to troubleshoot it, i suggest you split the content store db and the notification db as a first step:
Change the Connection Properties for the Notification Database