In my Android project I am using Retrofit/OKHTTP to connect to several API endpoints. I am observing that the initial network call will always be delayed by several seconds. This will happen if only one API call is made or multiple at the same time, it does not matter. This could be 30-40 seconds with absolutely no activity at all.
Any subsequent requests sent to any API Endpoint will fire immediately without delay. If airplane mode is toggled, again the next network call will be delayed by several seconds.
I can replicate the issue on both Ubuntu and Windows, so I don't think its an environment or configuration issue. The issue does not occur when I test using a physical device, making me suspect the Emulator is at fault.
I found:
https://square.github.io/okhttp/changelogs/upgrading_to_okhttp_4/#advanced-profiling
Which I suspect might have a role to play however I cannot disable "Advanced Profiling" in my Studio versions Run Configurations (Android Studio Chipmunk | 2021.2.1 Patch 1)
I created a OKHTTP EventListener to dump out a log of all the requests:
2022-08-19 13:51:06.397 11142-11142/com.traktapp D/OkHttpPerformanceEventL: [RoundTrip] *** callStart https://api.trakt.tv/users/xxx/ratings/episodes?limit=50&page=1
2022-08-19 **13:51:06.397** 11142-11142/com.traktapp D/OkHttpPerformanceEventL: [RoundTrip] *** resetMetricData
2022-08-19 **13:51:34.094** 11142-11417/com.traktapp D/OkHttpPerformanceEventL: [RoundTrip] *** responseBodyStart https://api.trakt.tv/users/xxx/ratings/episodes?limit=50&page=1. URL:
2022-08-19 13:51:34.099 11142-11417/com.traktapp D/OkHttpPerformanceEventL: [RoundTrip] *** responseBodyEnd responseBodyStartingTime: 1660913494094 , responseBodyTimeMS: 5 , responseBodyPayload: 2369. URL: https://api.trakt.tv/users/xxx/ratings/episodes?limit=50&page=1
2022-08-19 13:51:34.101 11142-11417/com.traktapp D/OkHttpPerformanceEventL: [RoundTrip] *** callEnd https://api.trakt.tv/users/xxx/ratings/episodes?limit=50&page=1
2022-08-19 13:51:34.191 11142-11142/com.traktapp D/OkHttpPerformanceEventL: [RoundTrip] *** callStart https://api.trakt.tv/users/xxx/ratings/episodes?limit=50&page=2
Here is the network activity graph where the delay is evident:
As can be seen from the logging, the initial network call was started 13:51:06.397, but the response body was returned sometime later 13:51:34.094.
Issue is not replicated using Postman or using a physical device to test. Has anyone any ideas or come across this problem before? It is not a computer specification/bottlenecking issue (Ryzen 5 CPU, 32GB RAM and Nvidia 1070TI GPU).
Just an update, I took a thread dump of the point where the connection delay occurs:
"OkHttp https://api.trakt.tv/...#25189" prio=5 tid=0x45 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at libcore.io.Linux.poll(Linux.java:-1)
at libcore.io.ForwardingOs.poll(ForwardingOs.java:577)
at libcore.io.BlockGuardOs.poll(BlockGuardOs.java:282)
at libcore.io.ForwardingOs.poll(ForwardingOs.java:577)
at libcore.io.IoBridge.isConnected(IoBridge.java:328)
at libcore.io.IoBridge.connectErrno(IoBridge.java:238)
at libcore.io.IoBridge.connect(IoBridge.java:180)
at java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:142)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:390)
- locked <0x62a7> (a java.net.SocksSocketImpl)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:230)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:212)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:436)
at java.net.Socket.connect(Socket.java:621)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at com.uwetrottmann.trakt5.TraktV2Interceptor.handleIntercept(TraktV2Interceptor.java:60)
at com.uwetrottmann.trakt5.TraktV2Interceptor.intercept(TraktV2Interceptor.java:26)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:221)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:920)
It's not helped much in identifying the issue, however.
This seems to be an issue with the emulator 31.3.10. There's a bug opened, but doesn't have much information: https://issuetracker.google.com/issues/243053479
To fix it, revert to 31.2.10.
Download it from here: https://developer.android.com/studio/emulator_archive
And very important: You have to keep package.xml from the previous emulator installation, otherwise Android Studio can't find it.
Related
I have written a client which tries to connect to Azure service bus. As soon as the server starts up i get the below errors and i receive no messages present at the queue. I tried replacing the sb protocol with amqpwss, but it dint help.
2020-05-25 21:23:11 [ReactorThreadeebf108d-444b-4acd-935f-c2c2c135451d] INFO c.m.a.s.p.RequestResponseLink - Internal send link 'RequestResponseLink-Sender_0480eb_c31e1cc239bf471e811e53a30adc6488_G51' of requestresponselink to '$cbs' encountered error.
com.microsoft.azure.servicebus.primitives.ServiceBusException: com.microsoft.azure.servicebus.amqp.AmqpException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'LinkTracker'. TrackingId:c31e1cc239bf471e811e53a30adc6488_G51, SystemTracker:gateway7, Timestamp:2020-05-25T21:23:10
at com.microsoft.azure.servicebus.primitives.ExceptionUtil.toException(ExceptionUtil.java:55)
at com.microsoft.azure.servicebus.primitives.RequestResponseLink$InternalSender.onClose(RequestResponseLink.java:759)
at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.processOnClose(BaseLinkHandler.java:66)
at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:42)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
at com.microsoft.azure.servicebus.primitives.MessagingFactory$RunReactor.run(MessagingFactory.java:491)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.azure.servicebus.amqp.AmqpException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'LinkTracker'. TrackingId:c31e1cc239bf471e811e53a30adc6488_G51, SystemTracker:gateway7, Timestamp:2020-05-25T21:23:10
... 10 common frames omitted
There is a similar issue opened in GitHub
what you posted here is the trace, not the error. Yes, the service
closes idle connections are 10 minutes. The client traces it and
reopens the connection. It is seamless, doesn't throw any exceptions
to the application. That can't be your problem. If your sends are
failing means there may be another problem, but not this one.
As i see the second line it is about the timeout of 6 secs, can you check the troubleshoot page if it helps. Also this.
we recommend adding "sync-publish=true" to the connection url
I am running Cypress UI tests in AzureDevOps CI/CD and some how most of the UI test are getting failed. All of the tests were running fine few days back.
It is throwing a Timed out waiting for the browser to connect. Retrying. error: Any advise on how to resolve the issue.
Environment Details:
Cypress version: 3.4.1,
Node: 10.x,
Azure DevOps CI/CD
Running: report/send-report.spec.js... (12 of 14)
2019-10-10T00:47:31.0294852Z
2019-10-10T00:47:31.0295427Z Warning: Cypress can only record videos when using the built in 'electron' browser.
2019-10-10T00:47:31.0295707Z
2019-10-10T00:47:31.0296579Z You have set the browser to: 'chrome'
2019-10-10T00:47:31.0296837Z
2019-10-10T00:47:31.0297613Z A video will not be recorded when using this browser.
2019-10-10T00:47:31.0313740Z (node:4030) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 end listeners added. Use emitter.setMaxListeners() to increase limit
2019-10-10T00:48:01.0316223Z
2019-10-10T00:48:01.0592004Z Timed out waiting for the browser to connect. Retrying...
2019-10-10T00:48:31.0587550Z
2019-10-10T00:48:31.0839142Z Timed out waiting for the browser to connect. Retrying again...
2019-10-10T00:49:01.0877330Z
2019-10-10T00:49:01.1241198Z The browser never connected. Something is wrong. The tests cannot run. Aborting...
I have noticed that you have set retries value as 2 to enable immediately retry on failure instead of moving on to the next test. So I recommend you to change the value and check if the error still occur.
And you can try another workaround, to change numTestsKeptInMemory down from 50 to something sane like 1 or 0. Here is the offical documentation. https://docs.cypress.io/guides/references/configuration.html#Global
In addition, it seems like an occasional error. Because some users failed on the first pipeline, but succeed on the second pipeline. And this should be a problem with cypress itself or your system's memory, you can report this problem to cypress directly.
Here is the link about cypress-io/cypress. https://github.com/cypress-io/cypress/issues/
And here is the link about the same error message.https://github.com/cypress-io/cypress/issues/1305
My application deployed in Weblogic instance is getting too slow sometimes. At that time, it's hitting the error related to Stuck Thread Time in Managed server log. Initially, when I noticed this, I did some research and increased the value of Max Stuck Thread Time to 800 seconds in place of 600 seconds. But, this didn't fix the issue. I got the following error again.
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
WatchData: MESSAGE = [STUCK] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "812" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl#42f38088[POST /****/faces/index.jsf]
", which is more than the configured time (StuckThreadMaxTime) of "800" seconds in "server-failure-trigger". Stack trace:
oracle.jbo.pcoll.PCollNode.objectAt(PCollNode.java:1753)
oracle.jbo.pcoll.PCollNode.objectAt(PCollNode.java:1753)
oracle.jbo.pcoll.PCollection.elementAt(PCollection.java:839)
oracle.jbo.server.QueryCollection.get(QueryCollection.java:2556)
oracle.jbo.server.ViewRowSetImpl.getRow(ViewRowSetImpl.java:5540)
oracle.jbo.server.ViewRowSetIteratorImpl.getRangeIndexOf(ViewRowSetIteratorImpl.java:1179)
oracle.jbo.server.ViewRowSetIteratorImpl.notifyRowUpdated(ViewRowSetIteratorImpl.java:3491)
We are using
ADF 12c for application development
Weblogic Version: 12.2.1 in windows server
Database : Oracle 11g
Jdk version : 1.8-65
Can anyone please advise me on the reason and possible solution for this issue?
Thanks in advance.
Increasing the stuck threads time will not resolve your issue. From the stack trace it looks like your application may be reading some file. There can be different reason for slowness
1) May be the file which you are reading is very large
2) May be slow network
3) May be IO latency
If possible you can introduce some debugging messages in your application. This will provide the details on the process which your application is running.
If you are running your application on linux/unix environment then you can monitor process activity inside the /var/proc directory
I am running an azure webjobs SDK console application (continuous) with the recommended setup:
public static void ProcessQueueMessage([QueueTrigger("logqueue")] string logMessage, TextWriter logger)
The azure queue I am running against has ~6000 messages in it and I am running the web-job locally, as a console application.
The problem I'm having is that the processing randomly stops after processing between zero and ~30 messages. The console stays open, but no more console messages are displayed.
For example, it might just process 2 messages:
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
And then, nothing. There doesn't seem to be anything wrong with my internet connection and I can't trace the issues down to any particular messages.
Has anyone else had issues with this SDK?
Update:
I made sure that I was using the right versions of all of the dependencies by removing the nuget packages and then re-running install-package Microsoft.Axure.Webjobs. I am now using webjobs version 1.1.0 which has pulled in version 4.3 of azure storage.
As recommended by Matthew, I have pulled down the source code for azure webjobs to determine where the process is freezing up. Once the freez-up occurs, I pause execution and checked the running threads for what I believe is the culprit within Microsoft.Azure.WebJobs.Host.CompositeTraceWriter
protected virtual void InvokeTextWriter(TraceEvent traceEvent)
{
if (_innerTextWriter != null)
{
string message = traceEvent.Message;
if (!string.IsNullOrEmpty(message) &&
message.EndsWith("\r\n", StringComparison.OrdinalIgnoreCase))
{
// remove any terminating return+line feed, since we're
// calling WriteLine below
message = message.Substring(0, message.Length - 2);
}
_innerTextWriter.WriteLine(message);
if (traceEvent.Exception != null)
{
_innerTextWriter.WriteLine(traceEvent.Exception.ToDetails());
}
}
}
The line it freezes on is line 66 : _innerTextWriter.WriteLine(message);
_innerTextWriter is an instance of System.IO.TextWriter.SyncTextWriter
Is it possible there is some deadlock issue with this class or the way it is being used?
Some notes:
I am running in the debugger, so in this case I believe the textwriter is forwarding to the console internally
I have my batchsize set to 1 via config.Queues.BatchSize = 1;, not sure if that could matter
I'm currently working on setting up an environment on another computer so that I can see if it is reproducible somewhere other than this machine (surface book).
Update
The issue was me not understanding how the new windows 10 command prompt works. Any time you click on the command window, it goes into "select" mode which completely pauses execution of the process.
Basically: https://superuser.com/questions/419717/windows-command-prompt-freezing-randomly?newreg=ece53f5584254346be68f85d1fd2f18d
You can tell it is in this state because it will prefix the window title with the word "Select":
You have to press enter or click again to get it going once again.
So, two final comments:
1) What an incredibly confusing and un-intuitive behavior for a command window!
2) I hope some admin will come take pity on the shame I have brought upon myself and my family by deleting this question.
To get rid of this strange behavior, you can disable QuickEdit mode:
Strange. When it is in this stuck state, can you try adding a new queue message to the queue and see if that triggers? Are you sure your function isn't hanging internally? What version of the SDK are you using? You might also try upgrading to v1.1.0 which we just released last week. If there are really a bunch of messages in the queue waiting to be processed, I can't think of anything that would cause this. The queue listener in the SDK should chug along, reading batches of messages in parallel and dispatching them to your function. Have you changed any of the JobHostConfiguration.Queues configuration knobs? You haven't force updated the version of the Azure SDK have you to something higher than the WebJobs SDK supports?
Another option if you can't figure this out might be to clone the SDK, build it and debug it locally. The repo is here. The main queue processing loop is here.
I'm managing Autonomy/Interwoven OpenDeploy Base Server.
OS:RHEL5.3
TSVer.:TeamSite6.7.2
ODVer:OpenDeploy7.1
OpenDeploy Base Server process sometimes hangs up when starting deployment.
following message are out in [od.log]
* glibc detected /var/od-home/OpenDeployNG/jre/bin/java: double free or corruption (!prev): 0xb01721d8 **
does anyone know about this problem?
Teamsite 6.7.1+ doesn't support Vintela (VAS) for users or groups on Linux.
I've seen same error in the iwtrace.log, the fix is to make sure that VAS (Vintela) is not used for authentication of users and groups.