My hazelcast client got disconnected because of possible memory outbreak. After the client reconnects, it is not getting getting messages from listeners on maps and topics. Is there a way to make the client active to the listeners again ?
This is how I am initiating a listener in my client/service:
IMap liveMap = hazelcastCacheClientService.getMap("MYMAP");
MyListener myMapListener = new MyListener();
liveMap.addEntryListener(myMapListener,true);
This is how my listener looks like:
public class MyListener implements EntryListener {
#Override
public void entryAdded(EntryEvent entryEvent) {
//do something
}
#Override
public void entryEvicted(EntryEvent entryEvent) {
}
#Override
public void entryRemoved(EntryEvent entryEvent) {
}
#Override
public void entryUpdated(EntryEvent entryEvent) {
//do something
}
#Override
public void mapCleared(MapEvent mapEvent) {
}
#Override
public void mapEvicted(MapEvent mapEvent) {
}
}
I have multiple other services, who put entries into the map.
If your version is 3.9 or older it could be a bug. We improved the client reconnect logic in 3.10 and further improved it in 3.11. Here are the 3.11 documentation for client reconnect configuration: https://docs.hazelcast.org/docs/3.11/manual/html-single/index.html#configuring-client-connection-retry
Now having said that I always recommend an additional circuit breaker pattern such as the one from netflix: https://github.com/Netflix/Hystrix/wiki/How-it-Works
In order to restore the client listeners it is best to destroy the hazelcastinstance and create a new one. This forces all the sockets to perform an init during startup. Hazelcast is a socket application so if you suffer from 1/2 open sockets then there is not much Hazelcast can do about restoring the connections.
Hope this helps,
Related
I'm having trouble grasping the concepts behind multithreading in netty, EventLoopGroup (MultithreadEventLoopGroup), MultithreadEventExecutorGroup, DefaultEventExecutorGroup
I am trying to understand how the server handles multiple clients simultaneously sending requests that will execute some business logic and CRUD operations that add to RTT. Below is my netty server code which works, but I am trying to understand exactly how it will work with concurrent users and multiple open channels.
I have a simple ServerBootstrap
#Component
#RequiredArgsConstructor
public class SocketServer {
private final ContextAwareLogger logger;
private final ServerInitializer serverInitializer;
private final NioEventLoopGroup bossGroup;
private final NioEventLoopGroup workerGroup;
private Channel mainChannel;
#PostConstruct
public void start() {
try {
ServerBootstrap bootstrap = init();
mainChannel = bootstrap.bind(8484).sync().channel(); // save the main channel so we can cleanly close it when app is shutdown
logger.info("Netty server started...");
} catch (Exception e) {
e.printStackTrace();
}
}
#PreDestroy
public void stop() throws InterruptedException {
logger.info("Shutting down Netty server");
bossGroup.shutdownGracefully().sync();
workerGroup.shutdownGracefully().sync();
mainChannel.closeFuture().sync();
logger.info("Netty Server shutdown complete.");
}
private ServerBootstrap init() {
return new ServerBootstrap()
.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.option(ChannelOption.SO_BACKLOG, 5000)
.option(ChannelOption.TCP_NODELAY, true)
.option(ChannelOption.SO_KEEPALIVE, true)
.childHandler(serverInitializer);
}
}
ChannelInitializer:
#Component
#RequiredArgsConstructor
public class ServerInitializer extends ChannelInitializer<SocketChannel> {
private final PacketDecoder packetDecoder;
private final ServerHandler serverHandler;
private final PacketEncoder packetEncoder;
#Override
protected void initChannel(SocketChannel socketChannel) throws Exception {
socketChannel.pipeline()
.addLast("decoder", packetDecoder) // ByteArrayDecoder
.addLast("encoder", packetEncoder) // ByteArrayEncoder
.addLast("inbound", serverHandler); // ChannelInboundHandlerAdapter
}
}
ChannelInboundHandlerAdapter:
#Component
#Sharable
public class ServerHandler extends ChannelInboundHandlerAdapter {
#Autowired
private SomeService someService;
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
// contains db access
byte[] accept = someService.validateClient(ctx.channel());
ctx.channel().writeAndFlush(accept);
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
// may contain db access
byte[] response = someService.processPacket(ctx.channel(), msg));
ctx.channel().writeAndFlush(response);
}
}
Now when a client connects, I understand that a new Channel will opened and the handlers will be reused. Requirement is each client request/response needs to process immediately without waiting for some other client's CRUD operations to finish.
Are my channelRead and channelActive, etc, async because I am using NioEventLoopGroup (ie will each client's channel operations be run independent of each other)?
If a single client sends multiple requests in series, are they guaranteed to be handled in the same order?
Do I need to specific DefaultEventExecutorGroup for my inbound handler? (https://stackoverflow.com/a/28305019/1738539)
You either would need to use a DefaultEventExecutorGroup for your ServerHandler or dispatch the validateClient(...) / processPacket(...) to your own ThreadPool. Failing todo so will cause the EventLoop thread to block and so no other IO can be processed for this EventLoop until the blocking operation completes.
I have developed own idGenerator based on Hazelcast IdGenerator class (with storing each last_used_id into db). Now I want to run hazelcast cluster as a single java application and my web-application as other app (web-application restart shouldn't move id values to next block). I move MyIdGeneratorProxy and MyIdGeneratorService to new application, run it, run web-application as a hazelcast-client and get
IllegalArgumentException: No factory registered for service: ecs:impl:idGeneratorService
It was okay when client and server were the same application.
It seems it's unable to process without some clientProxy. I have compared IdGeneratorProxy and ClientIdGeneratorProxy and it looks the same. What is the idea? How to write client proxy for services? I have found no documentation yet. Is direction of investigations correct? I thought it is possible to divide hazelcast inner services (like a id generator service) and my business-processes. Should I store custom ClientProxy (for custom spi) in my web-application?
This is a demo how to create a client proxy, the missing part CustomClientProxy function call, is quit complicated(more like a server proxy,here is called ReadRequest, the server is called Operation), you can find a how AtomicLong implement.For every client proxy method you have to make a request.
#Test
public void client() throws InterruptedException, IOException
{
ClientConfig cfg = new XmlClientConfigBuilder("hazelcast-client.xml").build();
ServiceConfig serviceConfig = new ServiceConfig();
serviceConfig.setName(ConnectorService.NAME)
.setClassName(ConnectorService.class.getCanonicalName())
.setEnabled(true);
ProxyFactoryConfig proxyFactoryConfig = new ProxyFactoryConfig();
proxyFactoryConfig.setService(ConnectorService.NAME);
proxyFactoryConfig.setClassName(CustomProxyFactory.class.getName());
cfg.addProxyFactoryConfig(proxyFactoryConfig);
HazelcastInstance hz = HazelcastClient.newHazelcastClient(cfg);
Thread.sleep(1000);
for (int i = 0; i < 10; i++)
{
Connector c = hz.getDistributedObject(ConnectorService.NAME, "Connector:" + ThreadLocalRandom.current()
.nextInt(10000));
System.out.println(c.snapshot());
}
}
private static class CustomProxyFactory implements ClientProxyFactory
{
#Override
public ClientProxy create(String id)
{
return new CustomClientProxy(ConnectorService.NAME, id);
}
}
private static class CustomClientProxy extends ClientProxy implements Connector
{
protected CustomClientProxy(String serviceName, String objectName)
{
super(serviceName, objectName);
}
#Override
public ConnectorState snapshot()
{
return null;
}
#Override
public void loadState(ConnectorState state)
{
}
#Override
public boolean reconnect(HostNode node)
{
return false;
}
#Override
public boolean connect()
{
return false;
}
}
EDIT
In hazelcast the IdGenerate is implemented as a wrapper for AtomicLong, you should implement you IdGenerate by you own, instead of extend IdGenerate.
So you have to implement these(more like a todo list XD):
API
interface MyIdGenerate
Server
MyIdGenerateService
MyIdGenerateProxy
MyIdGenerateXXXOperation
Client
ClientMyIdGenerateFactory
ClientMyIdGenerateProxy
MyIdGenerateXXXRequest
I also made a sequence(same as IdGenerate) here, this is backed by zookeeper or redis,also it's easy to add a db backend,too.I will integrate to hazelcast if I got time.
I am confused with Async feature introduced in Servlet 3.0 spec
From Oracle site (http://docs.oracle.com/javaee/7/tutorial/doc/servlets012.htm):
To create scalable web applications, you must ensure that no threads
associated with a request are sitting idle, so the container can use
them to process new requests.
There are two common scenarios in which a thread associated with a
request can be sitting idle.
1- The thread needs to wait for a resource to become available or process data before building the response. For example, an application
may need to query a database or access data from a remote web service
before generating the response.
2- The thread needs to wait for an event before generating the response. For example, an application may have to wait for a JMS
message, new information from another client, or new data available in
a queue before generating the response.
The first item happens a lot (nearly always, we always query db or call a remote webservice to get some data). And calling an external resource will always consume some time.
Does it mean that we should ALWAYS use servelt async feature for ALL our servelts and filter ?!
I can ask this way too, if I write all my servelts and filters async, will I lose anything (performance)?!
If above is correct the skeleton of ALL our servlets will be:
public class Work implements ServletContextListener {
private static final BlockingQueue queue = new LinkedBlockingQueue();
private volatile Thread thread;
#Override
public void contextInitialized(ServletContextEvent servletContextEvent) {
thread = new Thread(new Runnable() {
#Override
public void run() {
while (true) {
try {
ServiceFecade.doBusiness();
AsyncContext context;
while ((context = queue.poll()) != null) {
try {
ServletResponse response = context.getResponse();
PrintWriter out = response.getWriter();
out.printf("Bussiness done");
out.flush();
} catch (Exception e) {
throw new RuntimeException(e.getMessage(), e);
} finally {
context.complete();
}
}
} catch (InterruptedException e) {
return;
}
}
}
});
thread.start();
}
public static void add(AsyncContext c) {
queue.add(c);
}
#Override
public void contextDestroyed(ServletContextEvent servletContextEvent) {
thread.interrupt();
}
}
I am using HornetQ in distributed transaction environment with MDBs. I read from the JMS documentation that we should not create Connection instance frequently, rather we should reuse the connection and create JMS sessions as and when required. So I wrote a code which creates JMS connection and then reuse it. But I have encountered the following exception while reusing the JMS connection object.
Could not create a session: Only allowed one session per connection.
See the J2EE spec, e.g. J2EE1.4 Section 6.6
I read few blogs on this but they all are specific to seam framework.
Here is my code
public class DefaultService implements IMessageService {
private static final long serialVersionUID = 1L;
private static final Logger logger = LogManager.getLogger(DefaultService.class);
private static final String connectionFactoryJndiName = "java:/JmsXA";
private static volatile Connection connection = null;
private Session session = null;
#Override
public void sendMessage(String destinationStr, Serializable object) {
try {
Destination destination = jmsServiceLocator.getDestination(destinationStr);
ObjectMessage message = session.createObjectMessage();
message.setObject(object);
MessageProducer messageProducer = session.createProducer(destination);
messageProducer.send(destination, message);
messageProducer.close();
logger.trace("Sent JMS Messagae for: " + object.getClass().getName());
}
catch (NamingException e) {
throw new RuntimeException("Couldn't send jms message", e);
}
catch (JMSException e) {
throw new RuntimeException("Couldn't send jms message", e);
}
}
#Override
public void close() {
try {
if (session != null) {
session.close();
}
}
catch (Exception e) {
logger.error("Couldn't close session", e);
}
}
}
I am using JBoss EAP 6.
Did I miss any settings here?
On JCA connection (i.e. connection where you used the PooledConnectionFactory) you are supposed to create one Session only per connection. That is part of the EE specification. (It has always been).
This is because these connections are pooled and it would be impossible to put them back on the pool if you were using more than one session per connection.
If you switch for non pooled connection factories (the ones that are meant for remote clients) you would have it working the way you wanted but then you would miss pooling from the application server. EE components are usually short lived and opening / closing JMS Connections (any connection to be more precise) it's an expensive operation.
I'am playing around with WebSocketServlet (tomcat) and I have some question about doing it properly without race condition problems.
I have an instance variable (so non thread-safe) that will keep track of all the websocket connections
HashMap<String,MyServers> myNonThreadSafeVariable = HashMap<String,MyServers>
This is what the HashMap will contain (roughly...)
private final class MyServers extends MessageInbound {
final Set<MyClients> clients = new CopyOnWriteArraySet<MyClients>();
private String serverName;
#Override
protected void onOpen(WsOutbound outbound) {}
#Override
protected void onClose(WsOutbound outbound) {}
#Override
protected void onText(WsOutbound outbound) {}
}
private final class Clients extends MessageInbound {
private int clientID;
#Override
protected void onOpen(WsOutbound outbound) {}
#Override
protected void onClose(WsOutbound outbound) {}
#Override
protected void onText(WsOutbound outbound) {}
}
So now.. during my servlet life time, I am looping through myNonThreadSafeVariable and then maybe also will loop through myNonThreadSafeVariable.clients and then maybe also modified or add a clients or server etc...
For example when a server connect, in his onOpen there will be something like
myNonThreadSafeVariable.put(key,this);
or When a client connects in his onOpen (quit concern about this one)
server = myNonThreadSafeVariable,get(key);
sever.clients.add(this);
Or sometimes when I have to ping all the clients of all the servers:
for (Entry<String, MyServers> entry : myNonThreadSafeVariable.entrySet()) {
MyServers server = entry.getValue();
server.sendMessage("ping||");
for (MyClients member : entry.getValue().clients) {
client.sendMessage("")
}
}
So If I undertand correctly as myNonThreadSafeVariable is global so will myNonThreadSafeVariable.clients etc..
So my question is what is a good practice to avoid race condition in this scenario ?
Using mutex and synchronized on them when access ether the myNonThreadSafeVariable and myNonThreadSafeVariable.clients ? Or should I avoid using an instance variable at all ? But how ?
thanks !
You could use a ReadWriteLock: you block readers and writers when writing, you block only writers when reading:
private final ReadWriteLock lock = new ReentrantReadWriteLock();
private final Lock readLock = lock.readLock();
private final Lock writeLock = lock.writeLock();
...
writeLock.lock();
try
{
myNonThreadSafeVariable.put(key,this);
}
finally
{
writeLock.unlock();
}
...
writeLock.lock();
try
{
server = myNonThreadSafeVariable,get(key);
sever.clients.add(this);
}
finally
{
writeLock.unlock();
}
...
readLock.lock();
try
{
for (Entry<String, MyServers> entry : myNonThreadSafeVariable.entrySet())
{
MyServers server = entry.getValue();
server.sendMessage("ping||");
for (MyClients member : entry.getValue().clients)
{
client.sendMessage("")
}
}
}
finally
{
readLock.unlock();
}
Moreover if you want to avoid the read lock you can copy the whole collection and scan the copy, letting the possibility for the original collection to be changed while notifying.