Trying to send some logs to logstash server. Using logstash forwarder to forward the logs to logstash
But its getting timed out:
2015/03/04 08:19:15.266955 Started harvester at end of file (current offset now 10659): /apps/azuga-dds/logs/amqData.log
2015/03/04 08:19:15.267089 Setting trusted CA from file: /etc/logstash-forwarder/logstash-forwarder.crt
2015/03/04 08:19:15.290016 Connecting to [10.90.9.242]:5000 (ec2-54-70-33-51.us-west-2.compute.amazonaws.com)
2015/03/04 08:19:20.290259 Failure connecting to 10.90.9.242: dial tcp 10.90.9.242:5000: i/o timeout
2015/03/04 08:19:21.291691 Connecting to [10.90.9.242]:5000 (ec2-54-70-33-51.us-west-2.compute.amazonaws.com)
2015/03/04 08:19:26.291903 Failure connecting to 10.90.9.242: dial tcp 10.90.9.242:5000: i/o timeout
2015/03/04 08:19:27.293218 Connecting to [10.90.9.242]:5000 (ec2-54-70-33-51.us-west-2.compute.amazonaws.com)
Any idea how to resolve this issue.
You may have some problems with SSL cert, sometimes checking cert may help. And be sure that you are using same version of JVM on logstash-forwarder, logstash and elasticsearch. Generate cert with your logstash-server IP, log says that you try to connect to host with IP, that not listed in cerificate.
Try
openssl s_client -showcerts -connect host:port
Try generating a new ssl cert in the logsatsh server (10.90.9.242) with the IP-SAN alternate name which means editing the /etc/ssl/openssl.cnf and adding:
subjectAltName = IP:10.90.9.242
under the [v3_ca] section.
and only afterwards generating the cert and key by running:
openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt -days 3650
Don't forget to reset the logstash and move the crt and key to the correct path of the logstash-forwarder (written in the config file).
Related
I have on Windows server certificate which is valid and active and WinRM listener is active as well on port 5986 (telnet works) for WinRM connection that needs to be established from the Linux server.
I didn't copy that certificate anywhere on the linux server as I do not know where that should be or how it should be configured.
If I try to establish WinRM connection I am getting this error on the Linux server.
openssl s_client -connect 10.7.147.210:5986
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 1367 bytes and written 447 bytes
Verification error: unable to verify the first certificate
I tried by referencing CAfile and CApath and cert option but without success:
openssl s_client -cert winrmcert.pem -key winrmcert.key -CApath . -connect 10.7.147.210:5986
openssl s_client -CAfile winrmcert.pem -connect 10.7.147.210:5986
Can you please help me what I need to do and configure on Linux server for certificate generated on Windows server for WinRM connection? I am not the expert for this topic so I would appreciate all useful instructions. Thank you
We currently run a multi region cassandra cluster in AWS. It runs in four regions, 12 nodes per region. It runs without node to node encryption (or client encryption either). We are trying to enable inter datacenter node to node encryption. However, when we flip encryption over we get an exception that nodes are unable to gossip with any peers.
It could possibly be that we didn't build our jks keystore/truststores correctly (more on how we built these files below). But, we additionally do not see intra datacenter communication working (which should be set to unencrypted communication). Additionally, cqlsh cannot connect to the node either; even though we have (by default) client_auth_required set to false.
ERROR [main] 2019-08-15 18:46:32,241 CassandraDaemon.java:749 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [apache-cassandra-3.11.4.jar:3.11.4]
INFO [main] 2019-08-15 18:47:07,384 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml
Something to note is that this error message occurs after a few minutes of the node being up. (i.e. there is a delay between start up before this exception is thrown).
Information about our cassandra setup
cassandra version: 3.11.4
JDK version: openjdk-8.
Linux: Ubuntu 18.04 (bionic).
cassandra.yaml
endpoint_snitch: Ec2MultiRegionSnitch
server_encryption_options:
internode_encryption: dc
keystore: <omitted>
keystore_password: <omitted>
truststore: <omitted>
truststore_password: <omitted>
client_encryption_options:
enabled: false
cassandra-rackdc.properties
prefer_local=true
No obvious errors with SSH output
When starting cassandra with JVM_OPTS="$JVM_OPTS -Djavax.net.debug=ssl" added to cassandra-env.sh we see SSL logs printed to stdout (Note: Subject and Issuer were omitted on purpose).
found key for : cassy-us-west-2
adding as trusted cert:
Subject: ...
Issuer: ...
Algorithm: RSA; Serial number: 0xdad28d843fc73325d4c1a75207d4e74
Valid from Fri May 27 00:00:00 UTC 2016 until Tue May 26 23:59:59 UTC 2026
...
trigger seeding of SecureRandom
done seeding SecureRandom
Looking at Java SE SSL/TLS connection debugging, this looks correct. But to note, we see this series of messages (along with the RSA key signature output) repeated several times in rapid fire. We never observe any messages about the trust store being added; however that might be something that occurs only on client initiation (?)
Additionally, we do see cassandra report that the Encrypted Messaging service has been started.
INFO [main] 2019-08-15 18:45:31,022 MessagingService.java:704 - Starting Encrypted Messaging Service on SSL port 7001
Doesn't appear to be a cassandra.yaml configuration problem
We can bring the node back online by simply configuring internode_encryption: none. This action seems to rule out a broadcast_address or rpc_address configuration problem.
How we built our keystore/truststores
We followed the basic template datastax docs for preparing SSL certificates. One minor difference was that our private key and CSRs were generated using openssl. One per each region (we plan to share key/signed certs across nodes in regions). This was created using a command template as:
openssl req -new -newkey rsa:2048 -out cassy-<region>.csr -keyout cassy-<region>.key -config cassy-<region>.conf -subj "..." -nodes -sha256
The generated CSR was then signed by an internal root CA. Because we generated our files using openssl, we had to build our jks files by importing our certs into them.
Commands to generate truststore
We distribute this one file to all nodes.
keytool -importcert
-keystore generic-server-truststore.jks
-alias rootCa
-file rootCa.crt
-noprompt
-keypass omitted
-storepass omitted
Commands to generate keystore
This was done one per region; but essentially we created a keystore with keytool, then deleted the key entry and then imported our key entry using keytool from a pkcs12 file.
keytool -genkeypair -keyalg RSA -alias cassy-${region} -keystore cassy-${region}.jks -storepass omitted -keypass omitted -validity 365 -keysize 2048 -dname "..."
keytool -delete -alias cassy-${region} -keystore cassy-${region}.jks -storepass omitted
openssl pkcs12 -export -in signed_certs/${region}.pem -inkey keys/cassandra.${region}.key -name cassy-${region} -out ${region}.p12
keytool -importkeystore -deststorepass omitted -destkeystore cassy-${region}.jks -srckeystore ${region}.p12 -srcstoretype PKCS12
keytool -importcert -keystore cassy-${region}.jks -alias rootCa -file ca.crt -noprompt -keypass omitted -storepass omitted
Looking back at this, I don't remember why we used keytool to generate a keypair/keystore, then deleted and imported. I think it was because the keytool importkeystore command refused to run if the keystore didn't already exist.
ca.crt and pem file
The ca.crt file contains the root certificate and the intermediate certificate that was used to sign the CSR. The pem file contains the signed CSR returned to us, the intermediate cert, and the root CA (in that order).
openssl verify ca.crt and pem
openssl verify -CAfile ca.crt us-west-2.pem
signed_certs/us-west-2.pem: OK
Command output after enabling encryption
nodetool status (output truncated)
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
?N 52.44.11.221 ? 256 25.4% null 1c
...
?N 52.204.232.195 ? 256 23.2% null 1d
Datacenter: us-west-2
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
?N 34.209.2.144 ? 256 26.5% null 2c
UN 52.40.32.177 105.99 GiB 256 23.7% null 2c
?N 34.210.109.203 ? 256 24.7% null 2a
...
With the online node being the node with encryption set.
cqlsh to localhost
cassy-node6:~$ cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
cqlsh to remote node
Remote node is a node with encryption enabled
cassy-node6:~$ cqlsh 10.0.2.7
Connection error: ('Unable to connect to any servers', {'10.0.2.7': error(111, "Tried connecting to [('10.0.2.7', 9042)]. Last error: Connection refused")})
Behavior we expected
We expected for the node to report that the other regions were all down, as they need to be handled over encryption. So that works as expected; however, cqlsh and intra datacenter peers being reported as unreachable is unexpected.
Specifically, we expected the node to still show peer nodes within the same datacenter as up and normal, regardless if there is a cert issue/error. We also expected cqlsh to continue to work.
Lastly, we are also trying to figure out if we have a jks certificate problem.
I'm trying to implement an ELK stack while gathering information with Filebeat and MetricBeat on clients.
Installation went well, filebeat is runing OK on the client and sending information to logstash (thanks to 'insecure: true).
Metricbeats don't want to connect at all and show:
2017-02-08T15:57:36+01:00 ERR Connecting error publishing events (retrying): x509: cannot validate certificate for xxx.xxx.xxx.xxx because it doesn't contain any IP SANs
2017-02-08T15:57:37+01:00 ERR Connecting error publishing events (retrying): x509: cannot validate certificate for xxx.xxx.xxx.xxx because it doesn't contain any IP SANs
I looked around and tried:
1.using FQDN for creating the ssl certs:
sudo openssl req -subj '/CN=ec2xxxxeu-west-1.compute.amazonaws.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
This shows :
2017-02-08T15:47:22+01:00 ERR Connecting error publishing events (retrying): x509: certificate is valid for , not ec2-34-249-172-152.eu-west-1.compute.amazonaws.com
2.using ip certs by putting 'subjectAltName = IP: ELK_server_private_IP' in openssl configuration then using:
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
I followed this tutorial for ELK / filebeat installation
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-16-04
That show the error msg in the title about SAN ips.
Do someone have an explanation of what's going on, do metrisbeats have a insecure:true to make it work?
I have 2 servers:
logstash server
app server
If I run a logstash-forwarder on a logstash server - it's ok.
But if I run logstash-forwarder on app server i get an error:
Failed to tls handshake with <ip> x509: certificate has expired or is not yet valid
And when I check the certificate by command:
openssl x509 -in logstash-forwarder.crt -noout -text
I take a valid certificate:
...
Validity
Not Before: Jun 28 17:33:36 2015 GMT
Not After : Jun 27 17:33:36 2016 GMT
...
X509v3 extensions:
X509v3 Subject Alternative Name:
IP Address:<ip>
Date on app serve:
root#app:/etc/pki/logstash# date
Sun Jun 28 20:53:30 MSK 2015
What's wrong?
I faced the similar kind of issue and fixed it. Looks like you didn't have any DNS set for you logstash index server. In this case you have to set SAN to openssl.cnf and regenerate the cert along with key.
How to set SAN?
sudo vi /etc/pki/tls/openssl.cnf
Find the [ v3_ca ] section in the file, and add this line under it (substituting in the Logstash Index Server's IP address):
subjectAltName = IP: logstash__index_server_private_ip
After that try regenerating your cert and key
cd /etc/pki/tls
sudo openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
And change your config file to point the cert and key from both end(index and forwarder) accordingly. Hope this will workout for you.
In relation to this question, I am trying to start a TLS server in node.js to reflect the one I created in OpenSSL. I have tested the client and server using OpenSSL from the command line and they successfully make a connection. When I try to port the server to node.js (and still connect to it with an OpenSSL client), I receive a 'no shared cipher' error. I am wondering if there is something special I need to do when using the passphrase option with tls.createServer()
Below are my successful OpenSSL commands for server and client respectively, note that the passphrase.txt file contains a single line that is the passphrase:
$ openssl s_server -accept 8888 -cert server.cert -key server.key -pass file:passphrase.txt -CAfile ca.cert
$ openssl s_client -connect 127.0.0.1:8888 -cert client.cert -key client.key -pass file:passphrase.txt -CAfile ca.cert
I can also make a successful connection if I specify a cipher for the client and/or server with the additional argument of -cipher 'ECDHE-ECDSA-AES128-GCM-SHA256'. I am using elliptic curve keys generated with openssl ecparam and signed with a CA created using openssl ca as discussed in my previous question.
The server code written in node.js looks like this:
var tls = require('tls');
var fs = require('fs');
var msg = '***********\n\nHello there secure client!\n\n***********';
var port = 8888;
var host = 'localhost';
var options = {
cert : fs.readFileSync('server.cert'),
key : fs.readFileSync('server.key'),
passphrase : (fs.readFileSync('passphrase.txt')).toString(),
ca : fs.readFileSync('ca.cert'),
// ciphers: 'ECDHE-ECDSA-AES128-GCM-SHA256',
// requestCert : true,
// rejectUnauthorized : true
};
tls.createServer(options, function(cleartextStream) {
if (cleartextStream.authorized) {
console.log('Server-side connection authorized by a Certificate Authority.');
} else {
// TODO this code does not appear to get executed even on failed connections
console.log('Server-side connection not authorized: ' + cleartextStream.authorizationError);
}
// send the server message to the client
cleartextStream.write(msg);
cleartextStream.setEncoding('utf8');
cleartextStream.pipe(cleartextStream);
}).listen(port, function() {
console.log('Server started on port: ' + port);
}).on('clientError', function(err){
console.log('A failed client connection attempt occurred.');
console.error(err);
console.log();
});
After calling the above code with node tlsServer.js and attempting to connect with an OpenSSL client on the command line, I receive the following messages.
SERVER:
$ node tlsServer.js
Server started on port: 8888
<< client started here >>
A failed client connection attempt occurred.
[Error: 6396:error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher:openssl\ssl\s3_srvr.c:1132:
]
CLIENT:
$ openssl s_client -connect 127.0.0.1:8888 -cert client.cert -key client.key -pass file:passphrase.txt -CAfile ca.cert
CONNECTED(00000003)
2674688:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
I am using node v0.6.15. And the errors do not change when I uncomment the ciphers, requestCert, and rejectUnauthorized in the options list sent to tls.createServer(). I also have a node.js cersion of the client, and I get a socket hang up code ECONNRESET when I attempt to connect to the node server, and the following error when trying to connect to a OpenSSL server:
Connection to localhost:8888 could not be made.
[Error: 6968:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:openssl\ssl\s23_clnt.c:602:
]
Thanks in advance for your help and ideas!
If the passphrase is wrong, maybe. Try removing it.
However, no_shared_cipher is an error raised when the client can't agree on a cipher suit with the server . Try first removing the cipher suit restriction on the server and seeing what it negotiates to use to isolate the problem. If this works, then place one with the client and see what happens.
Also, can you check that node is using the same openssl library as the openssl command.