We've been advised to try Ganglia as a monitoring tool for our cluster.
The installation was pretty smooth but I have a problem with connectivity between gmond and gmetad.
Meta node is able to see(on web) only local gmond host (itself).
The configuration of gmetad (10.45.11.26 is gmetad localhost):
data_source "hbase" 10.45.11.26
The configuration of gmond(10.45.11.27 is gmond localhost):
cluster {
name = "hbase"
owner = "unspecified"
latlong = "unspecified"
url = "unspecified"
}
udp_send_channel {
host=10.45.11.26
port = 8649
ttl = 1
}
udp_recv_channel {
port = 8649
bind = 10.45.11.27
}
tcp_accept_channel {
port = 8649
}
Telnet from gmetad to gmond on port 8649 returns xml.
I can see udp traffic coming from gmond on gmetad node (tcpdump)
What I'm missing here?
I don't know if you still need help, but it can help to add
globals {
(......)
send_metadata_interval = 60 /*gmond heartbeats in secs */
}
in gmond.conf
In this case, you may have to wait 60 seconds after node startup before it is seen.
Related
Currently I am working on an external vendor integration via Spring Integration TCP where I am sending to different ports of 2 IPs (IP 1, port 1, 2 ,3 and IP 2 port 1 , 2 & 3) by creating different #ServiceActivator for unique IP/Port combination.
All is working fine in a normal messages load, however, when I increase the load to 40-50 transactions/sec, then I am seeing the messages are getting timeout by waiting for the response.
Below is one of the Service Activator and others are setup in the similar way as well. I am using FailoverClientConnectionFactory for failover purpose if my primary ip is not available. I am also using CachingClientConnectionFactory so, that every request I don't have to create a connection which is expensive.
#Bean
#ServiceActivator(inputChannel = "toRequest")
public MessageHandler serviceActivatorOne() {
TcpOutboundGateway tcpOutputGateway = new TcpOutboundGateway();
List<ServerNode> nodes = properties.getFailOver().getServers().subList(0,2);
tcpOutputGateway.setConnectionFactory(createFailOverConnectionFactory(nodes));
tcpOutputGateway.setReplyChannelName("bytesToObjectChannel");
tcpOutputGateway.setRemoteTimeout(5000);
return tcpOutputGateway;
}
...
...
private AbstractClientConnectionFactory createFailOverConnectionFactory(List<ServerNode> serverNodeList) {
FailoverClientConnectionFactory connectionFactory =
new FailoverClientConnectionFactory(createConnectionFactories(serverNodeList));
connectionFactory.setSingleUse(true);
connectionFactory.afterPropertiesSet();
return connectionFactory;
}
private AbstractClientConnectionFactory createDefaultConnectionFactory(String url, int port, String name) {
TcpNetClientConnectionFactory connFactory = new TcpNetClientConnectionFactory(url, port);
connFactory.setSerializer(byteArrayLengthHeaderSerializer());
connFactory.setDeserializer(byteArrayLengthHeaderSerializer());
connFactory.setSoTimeout(properties.getSocketTimeOut());
connFactory.setSoKeepAlive(true);
connFactory.setLookupHost(false);
connFactory.setComponentName(name);
connFactory.setSingleUse(true);
if(properties.getPoolMaxSize() <= 0) {
return connFactory;
} else {
CachingClientConnectionFactory cachingConnFactory =
new CachingClientConnectionFactory(connFactory, properties.getPoolMaxSize());
cachingConnFactory.setConnectionWaitTimeout(properties.getPoolMaxWait());
return cachingConnFactory;
}
}
Exception details:
[DEBUG]-[http-nio-9090-exec-103]-[11-Nov-2019;22:01:14.403]-[TcpOutboundGateway:handleRequestMessage:162]-Remote Timeout on 1980e8d4-3167-4610-b8ac-8fb0d20eb92a:1
[DEBUG]-[http-nio-9090-exec-103]-[11-Nov-2019;22:01:14.403]-[SimplePool:releaseItem:226]-Releasing TcpNetConnection:216.76.27.251:20303:54340:d9c78e03-ecbb-45af-b502-6aff9a9b0036 back to the pool
[ERROR]-[http-nio-9090-exec-103]-[11-Nov-2019;22:01:14.404]-[TcpOutboundGateway:handleRequestMessage:174]-Tcp Gateway exception
org.springframework.integration.MessageTimeoutException: Timed out waiting for response
at org.springframework.integration.ip.tcp.TcpOutboundGateway.handleRequestMessage(TcpOutboundGateway.java:166)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:123)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:162)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:144)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:105)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:453)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:401)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:4
I tried to find out some details from Spring reference on TCP but didn't succeed so far. I am not using shared socket as setSingleUse is set as false. Any help/clue is appreciated.
It looks like the server you connect cannot answer in time for that 5 seconds timeout under heavy load.
It seems the default remoteTimeout is 10s and I am setting to 5s which seems the server can't response with in 5s. I can tune the remoteTimeout and check. Do you have any suggestions where I can change any of the spring-tcp integration settings or is it solely depends on slow responsive nature from the server?
Right, it does depend on server. I imagine a situation when with some huge load even those 10 seconds won’t be enough. You should consult with your server vendor to determine their throughput.
Trying to cluster Socket.io using net.createServer. All examples are using IP to split what connection goes to witch node. However I'm using 4 servers with a load balancer that points ip;s to the different servers.
So in node cluster I would like to use an unique id to point the connection to a specific cluster.
Figure that each user that wants to connect can add a parameter to the connection url ws://localhost/socket.io?id=xxyyzz
How can I get the connection url in net.createServer
todays code for ip:
var server = net.createServer({ pauseOnConnect: true }, function(connection) {
// We received a connection and need to pass it to the appropriate
// worker. Get the worker for this connection's source IP and pass
// it the connection.
var remote = connection.remoteAddress;
var local = connection.localAddress;
var ip = (remote+local).match( /[0-9]+/g )[0].replace(/,/g, '');
var wIndex = ip % num_processes;
var worker = workers[wIndex];
worker.send('sticky-session:connection', connection);
});
I was wondering if there is an example usage of the AddressResolver interface in apache ignite.
I was trying to 'bind' my local IP addresses (e.g. 192.168.10.101) to my external IP address using the AddressResolver interface, but without luck.
When I do that the Ignite server just hangs (no output from the debug either)
My code for starting the server is:
TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(ipaddresses);
spi.setIpFinder(ipFinder);
spi.setAddressResolver(new IotAddressResolver());
IgniteConfiguration cfg = new IgniteConfiguration();
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
System.setProperty("IGNITE_QUIET", "false");
// Start Ignite node.
ignite = Ignition.start(cfg);
My implementation for AddressResolver is:
public class IotAddressResolver implements AddressResolver {
#Override
public Collection<InetSocketAddress> getExternalAddresses(
InetSocketAddress internalAddresses) throws IgniteCheckedException {
String host = "XX.XX.XX.XX";
Collection<InetSocketAddress> result = new ArrayList<InetSocketAddress>();
result.add(new InetSocketAddress(host, internalAddresses.getPort()));
return result;
}
}
The last line of the ignite debug log is:
WARNING: Timed out waiting for message to be read (most probably, the reason is in long GC pauses on remote node) [curTimeout=9989]
I will appreciate any help. Thank you
Can you provide more details about your deployment and what you're trying to achieve with the help of address resolver? How many physical hosts and Ignite nodes do you have? Are they located in different networks with the router between them?
I dont know if this is the best way to handle this but I managed to start igntie as local server. I am setting my local ip and port like this:
System.setProperty("IGNITE_QUIET", "false");
TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();
// Set initial IP addresses.
ipFinder.setAddresses(ipaddresses);
spi.setIpFinder(ipFinder);
// Override local port.
commSpi.setLocalPort(47501);
commSpi.setLocalAddress("XX.XX.XX.XX");
commSpi.setLocalPortRange(50);
IgniteConfiguration cfg = new IgniteConfiguration();
// Override default communication SPI.
cfg.setCommunicationSpi(commSpi);
cfg.setDiscoverySpi(spi);
cfg.setAddressResolver(new IotAddressResolver());
cfg.setClientMode(true);
// Start Ignite node
ignite = Ignition.start(cfg);
Where XX.XX.XX.XX is my local IP address
I am trying to develop a Video Client/functionality that captures video using webcam and transfers to other servent (server-client) somewhere on the internet. I am using UDPCLient Class to do that.
I want my application to be able to listen and tarnsmit video captured from webcam. The capturing, transmission and receiving works fine when i do that on local network.
But when i test the application from behind router (across two differnt networks/internet) after forwarding respective ports, the internet connectivity is lost on both routers (They hang up or something) and i need to restart the routers or switch to an alternate connection. The configuration is as follows:
Servent 1 <--> Router1 <--> Internet Connection#01
Servent 02 <---> Router2 <---> Internet Connection#02
Both connections are on separate DSL Line. One of the routers is ZTE brand and the other is of Netgear.
Code for listenning/transmission is as follows:
private void StartSockets()
{
//For testing across internet i use IPAddress obtained via different function
var IPAddress = getMyIpAddress();
this.udpSender = new UdpClient(IpAddress, 4000);
this.udpListener = new UdpClient(4000);
}
private IPAddress getMyIpAddress()
{
IPAddress localIP ;//= AddressAr[0];
localIP = IPAddress.Parse(GetPublicIP());
return localIP;
}
public string GetPublicIP()
{
String direction = "";
WebRequest request = WebRequest.Create("http://checkip.dyndns.org/");
using (WebResponse response = request.GetResponse())
{
using (StreamReader stream = new StreamReader(response.GetResponseStream()))
{
direction = stream.ReadToEnd();
}
}
//Search for the ip in the html
int first = direction.IndexOf("Address: ") + 9;
int last = direction.LastIndexOf("</body>");
direction = direction.Substring(first, last - first);
return direction;
}
Code for receiving response is as follows:
private void ReceiveData()
{
//For testing across internet i use IPAddress obtained via different function
var IPAddress = getMyIpAddress();
IPEndPoint ep = new IPEndPoint(IPAddress, myPort);
try
{
byte[] receiveBytes = this.udpListener.Receive(ref ep);
this.OnReadImage(new ImageEventArgs(this.ByteToImage(receiveBytes)));
}
catch (Exception)
{
}
}
If i test on local network , i use DNSHostname to get ip address (private ip addresses) and video works fine on local network. That does not work over internet so i switch to live Ip Address and thus i use the method of getPublicIpAddress().
I know there is something seriously wrong with my approach? What would be right approach?
Should i switch to TCP Listenner? I intend to have multiple receiver of same video in future. So would that affect?
Can UDP clients cause routers to crash, hang up and restart? How can i avoid that?
Lastly, if were to avoid port-forwarding what would be the best strategy?
Please help.
Thanks
Steve
hello all
I looked at the at the redis-node-client source (relevant part is shown bellow) and I see that it connects to redis via the 'net' package, which is TCP based.
line 370
exports.createClient = function (port, host, options) {
var port = port || exports.DEFAULT_PORT;
var host = host || exports.DEFAULT_HOST;
var client = new Client(net.createConnection(port, host), options);
client.port = port;
client.host = host;
return client;
};
I was wondering if there's a more direct client for redis, preferably via domain-sockets or something of that sort. Im using redis localy, as cache, without going over the wire so its unnecessary to encode/decode messages with TCP headers...
thank you
Unix Domain Socket support appears to have landed in Redis as of Nov 4th.
http://code.google.com/p/redis/issues/detail?id=231
To connect to a Unix Domain Socket, you need to supply the pathname to net.createConnection. Maybe something like this in redis-node-client:
exports.createSocketClient = function (path, options) {
var client = new Client(net.createConnection(path), options);
client.path = path;
return client;
};