Hello currently I am developing an Arquillian extension for Moco framework (https://github.com/dreamhead/moco). Moco is used for testing RESTful services and relies on Netty for dealing with communication. Currently Moco is using Netty 4.0.18.Final.
But I have found a problem when running Moco (and Netty server) inside a container (Arquillian runs tests within the container) and is that it starts correctly but when the application is undeployed and server is shutdown next log error messages are printed:
SEVERE: The web application [/ba32e781-3a18-44b3-9547-7c26787f3fe7] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.
abr 08, 2014 10:29:06 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/ba32e781-3a18-44b3-9547-7c26787f3fe7] created a ThreadLocal with key of type [io.netty.util.internal.ThreadLocalRandom$2] (value [io.netty.util.internal.ThreadLocalRandom$2#77468cae]) and a value of type [io.netty.util.internal.ThreadLocalRandom] (value [io.netty.util.internal.ThreadLocalRandom#6cd3851]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Basically it seems that there are some threads that are not closed yet when the server tries to shutdown.
From the point of view of Arquillian extension when the application is deployed into the server the start method of Moco is called and before undeploying the application the stop method from Moco is called.
But let me show you the code of Moco:
public int start(final int port, ChannelHandler pipelineFactory) {
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.childHandler(pipelineFactory);
try {
future = bootstrap.bind(port).sync();
SocketAddress socketAddress = future.channel().localAddress();
address = (InetSocketAddress) socketAddress;
return address.getPort();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
and the stop method looks like:
private void doStop() {
if (future != null) {
future.channel().close().syncUninterruptibly();
future = null;
}
So it seems that the close method returns before killing all the threads and for this reason containers warns you about possible memory leaks.
Because I have never used Netty I was wondering if there is a way to ensure that the whole Netty runtime is closed.
Thank you so much for your help.
I am new to Netty as well (and unfamiliar with Arquillian), but based on the Netty Docs examples I believe you might not be shutting down the EventLoopGroups you created (bossGroup, workerGroup). From the Netty 4.0 User Guide:
Shutting down a Netty application is usually as simple as shutting down all EventLoopGroups you created via shutdownGracefully(). It returns a Future that notifies you when the EventLoopGroup has been terminated completely and all Channels that belong to the group have been closed.
So your doStop() method might look like:
private void doStop() {
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
}
An example in the Netty docs: Http Static File Server Example
Related
I am using Spring Integration and Spring Boot for some development on my location machine based on the Spring Guides. I am using Gradle to build and run the application. The following code is used to bootstrap Spring and I can terminate the application by pressing the enter key.
public class Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext ctx = new SpringApplication(Application.class).run(args);
System.out.println("Hit Enter to terminate");
System.in.read();
ctx.close();
}
}
This works fine but when I introduce a ThreadPoolTaskExecutor into the integration flow, the application never terminates. I have to use ^C to kill the application. The code I am using is as follows.
...
channel(MessageChannels.executor(myTaskExecutor()))
...
#Bean
public ThreadPoolTaskExecutor myTaskExecutor() {
ThreadPoolTaskExecutor pool = new ThreadPoolTaskExecutor();
pool.setCorePoolSize(10);
pool.setMaxPoolSize(20);
pool.setWaitForTasksToCompleteOnShutdown(true);
pool.setAwaitTerminationSeconds(1);
pool.initialize();
return pool;
}
I have:
Tried to shutdown the executor (using the shutdown()) method) before and after the context is closed.
Tried the above code also within the onApplicationEvent(ContextClosedEvent event) method.
Temporarily commenting the code which is run in the thread to make sure that is not holding on to the thread in any way.
Is there any anything else I need to do?
ctx.close() will shutdown any executor beans (by calling the destroy() method), so it is likely you have a thread "stuck" somewhere.
Take a Thread dump (jstack) to see what the executor thread is doing.
I have a requirement of executing parent task which may or maynot have child task. Each parent and child task should be run in thread. If something goes wrong in parent or child execution the transaction of both parent and child task must be rollback. I am using hibernate4.
If I got it, the parent and the child task will run in differents threads.
According to me it's a very bad idea that does not worth considering.
While it may be possible using jta transaction, it's clearly not the case using hibernate transaction management delegation to underlying jdbc connection (you have one connection per session and MUST NOT share an hibernate session between threads).
Using jta you will have to handle connection retrieval and transactions yourself and can't so take advantages of connection pooling and container managed transaction (spring or java ee ones). It may be overcomplicated for about no performance improvments as sharing the database connection between two threads will just probably move the bottleneck one level below.
See how to share one transaction between multi threads
According to OP expectation here is a pseudo code for Hibernate 4 standalone session management with jdbc transaction (I personnaly advise to go with a container (Java ee or spring) and JTA container managed transaction)
In hibernate.cfg.xml
<property name="hibernate.current_session_context_class">thread</property>
SessionFactory :
Configuration configuration = new Configuration();
configuration.configure("hibernate.cfg.xml");
StandardServiceRegistryBuilder builder = new StandardServiceRegistryBuilder().applySettings(configuration.getProperties());
SessionFactory sessionFactory = configuration.buildSessionFactory(builder.build());
The session factory should be exposed using a singleton (any way you choose you must have only one instance for the whole app)
public void executeParentTask() {
try {
sessionFactory.getCurrentSession().beginTransaction();
sessionFactory.getCurrentSession().persist(someEntity);
myChildTask.execute();
sessionFactory.getCurrentSession().getTransaction().commit();
}
catch (RuntimeException e) {
sessionFactory .getCurrentSession().getTransaction().rollback();
throw e; // or display error message
}
}
getCurrentSession() will return the session bound to the current thread. If you manage the thread execution yourself you should create the session at the beginning of the thread execution and close it at the end.
the child task will retrieve the same session than the parent one using sessionFactory.getCurrentSession()
See https://docs.jboss.org/hibernate/orm/4.3/manual/en-US/html/ch03.html#configuration-sessionfactory
http://docs.jboss.org/hibernate/orm/4.3/manual/en-US/html_single/#transactions-demarcation-nonmanaged
You may find this interesting too : How to configure and get session in Hibernate 4.3.4.Final?
Context
I have a RedisMqServer configured to handle a single message on my ServiceStack web service. The messages on that MQ originate from another application and show up in the .inq with all the correct properties. Everything is on 4.0.38.
My configuration in MyAppHost.cs:
public override void Configure(Container container)
{
var redisFactory = new PooledRedisClientManager(0, "etc:etc");
redisFactory.ConnectTimeout = 5;
redisFactory.IdleTimeOutSecs = 30;
redisFactory.PoolTimeout = 3;
container.Register<IRedisClientsManager>(redisFactory);
//Plugins, Filters, other Registrations omitted
var mqHost = new RedisMqServer(redisFactory, retryCount: 2);
mqHost.DisablePublishingResponses = true;
mqHost.RegisterHandler<CreateVisitor>(ServiceController.ExecuteMessage);
mqHost.Start();
}
And then in Global.asax.cs:
void Application_Start(object sender, EventArgs e)
{
new MyAppHost().Init();
}
Problem
The messages are not consistently handled when I deploy this elsewhere. They wait in the .inq until whenever. Nothing is lost, just delayed for an indeterminate duration.
As of this moment, the only things that come to mind are:
I'm using IIS Express locally, and the server is using IIS.
Application_Start needs to happen before it can handle messages.
I've tried initializing the service by making other API calls over HTTP, before and after queuing messages, with more failure than success. Sometimes the service starts to handle them, but I am unable to identify and thus influence when this happens.
Note
I do have several other console applications and windows services that listen on other MQs and handle messages placed by other applications, and those have always worked flawlessly. This is the first time I've tried this from within an existing web service, however.
Hard to know what the issue from this description (are messages getting lost or just delayed?) but this sounds like it's due to ASP.NET AppDomain recycling in which case you can disable AppDomain recycling or setup up a continuous ping route to hit your ASP.NET Web Application to keep the AppDomain alive.
If the ASP.NET Service is available on the Internet you can use services like https://uptimerobot.com or https://www.pingdom.com to configure it to ping your Service at different intervals (e.g. 5-10 minutes) otherwise if this is an internal Service you can use a Scheduled Task.
I have below application:
Its windows console .NET 3.0 application
I'm creating 20 workloads and assigning them to threadpool to process.
Each thread in ThreadPool creates WCF Client and calls service with request created using workload assigned.
Sometimes on production servers[12 core machines], I get following exception:
There was an error reflecting type 'xyz' while invoking operation using WCF client. This starts appearing in all threads. After sometime it suddenly disappears and starts appearing again.
Code:
Pseudo Code:
for(int i=0;i<20;i++)
{
MultiThreadedProcess proc =new MultThreadedProcess(someData[i]);
ThreadPool.QueueUserWorkItem(proc.CallBack,i);
}
In Class MultiThreadedProcess, I do something like this:
public void Callback(object index)
{
MyServiceClient client = new MyServiceClient();
MyServiceResponse response =client.SomeOperation(new MyServiceRequest(proc.SomeData));
client.close();
//Process Response
}
Can anyone suggest some resolutions for this problem?
If you can turn on diagnostic, appears to me serialization issue, there might be chance that certain data members/values are not able to de-serialized properly for operation call.
I have a very long running query that takes too long to keep my client connected. I want to make a call into my DomainService, create a new worker thread, then return from the service so that my client can then begin polling to see if the long running query is complete.
The problem I am running into is that since my calling thread is exiting right away, I am getting exceptions thrown when my worker tries to access any entities since the ObjectContext gets disposed when the original thread ends.
Here is how I create the new context and call from my Silverlight client:
MyDomainContext context = new MyDomainContext();
context.SearchAndStore(_myParm, SearchQuery,
p => {
if (p.HasError) { // Do some work and return to start
} // polling the server for completion...
}, null);
The entry method on the server:
[Invoke]
public int SearchAndStore(object parm)
{
Thread t = new Thread(new ParameterizedThreadStart(SearchThread));
t.Start(parms);
return 0;
// Once this method returns, I get ObjectContext already Disposed Exceptions
}
Here is the WorkerProc method that gets called with the new Thread. As soon as I try to iterate through my query1 object, I get the ObjectContext already Disposed exception.
private void WorkerProc(object o)
{
HashSet<long> excludeList = new HashSet<long>();
var query1 = from doc in this.ObjectContext.Documents
join filters in this.ObjectContext.AppliedGlobalFilters
.Where(f => f.FilterId == 1)
on doc.FileExtension equals filters.FilterValue
select doc.FileId;
foreach (long fileId in query1) // Here occurs the exception because the
{ // Object Context is already disposed of.
excludeList.Add(fileId);
}
}
How can I prevent this from happening? Is there a way to create a new context for the new thread? I'm really stuck on this one.
Thanks.
Since you're using WCF RIA. I have to assume that you're implementing two parts:
A WCF Web Service
A Silverlight client which consumes the WCF Service.
So, this means that you have two applications. The service running on IIS, and the Silverlight running on the web browser. These applications have different life cycles.
The silverlight application starts living when it's loaded in the web page, and it dies when the page is closed (or an exception happens). On the other hand (at server side), the WCF Web Service life is quite sort. You application starts living when the service is requested and it dies once the request has finished.
In your case your the server request finishes when the SearchAndStore method finishes. Thus, when this particular method starts ,you create an Thread which starts running on background (in the server), and your method continues the execution, which is more likely to finishes in a couple of lines.
If I'm right, you don't need to do this. You can call your method without using a thread, in theory it does not matter if it takes awhile to respond. this is because the Silvelight application (on the client) won't be waiting. In Silverlight all the operations are asynchronous (this means that they're running in their own thread). Therefore, when you call the service method from the client, you only have to wait until the callback is invoked.
If it's really taking long time, you are more likely to look for a mechanism to keep the connection between your silverlight client and your web server alive for longer. I think by modifying the service configuration.
Here is a sample of what I'm saying:
https://github.com/hmadrigal/CodeSamples/tree/master/wcfria/SampleWebApplication01
In the sample you can see the different times on client and server side. You click the button and have to wait 30 seconds to receive a response from the server.
I hope this helps,
Best regards,
Herber