Certificate verify failed for self signed on Linux 7.2 - linux

Question

I have read endless posts on certification error that basically say either turn off validation or fix you certificate. In our project however we really what to use the certificate, but we can't get it to work. The SA and 2 programmers have been try everything we can think of and nothing is working. So clearly we don't know what we are doing.
First, This is the error we get on a simple connection and get perl program. WEBHOSTNAME replace real web hostname.
perl -MIO::Socket::SSL=debug30 testerTut2.pl
DEBUG: .../IO/Socket/SSL.pm:1784: new ctx 46260896
DEBUG: .../IO/Socket/SSL.pm:446: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:448: socket connected
DEBUG: .../IO/Socket/SSL.pm:466: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:501: using SNI with hostname WEBHOSTNAME
DEBUG: .../IO/Socket/SSL.pm:524: set socket to non-blocking to enforce timeout=10
DEBUG: .../IO/Socket/SSL.pm:537: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:547: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:557: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:577: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:1772: ok=0 cert=46303216
DEBUG: .../IO/Socket/SSL.pm:537: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1408: SSL connect attempt failed with unknown error
DEBUG: .../IO/Socket/SSL.pm:543: fatal SSL error: SSL connect attempt failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
DEBUG: .../IO/Socket/SSL.pm:1821: free ctx 46260896 open=46260896
DEBUG: .../IO/Socket/SSL.pm:1826: free ctx 46260896 callback
DEBUG: .../IO/Socket/SSL.pm:1829: OK free ctx 46260896
500 Can't connect to WEBHOSTNAME:443 at testerTut2.pl line 34.
This is the perl program:
#!/bin/perl
require LWP::UserAgent;
use LWP::Protocol::https;
#note USERNAME is where the real account name goes
my $ua = LWP::UserAgent->new( ssl_opts => { verify_hostname => 1, SSL_ca_file => '/home/USERNAME/ca-bundle.crt'});
$ua->timeout(10);
$ua->env_proxy;
#note hostname is where the real web host name goes
my $response = $ua->get('https://hostname/tut/ops/data/newtut.dat');
if ($response->is_success)
{
print $response->content;
print $response->decoded_content; # or whatever
}
else
{
die $response->status_line;
}
The SA has made a signed certificate on the web server. Again webhostname would be replaced with real web hostname.
openssl s_client -connect lomweb01:443 -showcerts
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
CONNECTED(00000003)
depth=0 C = US, ST = MD, L = Greenbelt, O = NASA, OU = MMS, CN = webhostname.edtfmmslinux.ecs.nasa.gov
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = MD, L = Greenbelt, O = NASA, OU = MMS, CN = webhostname.edtfmmslinux.ecs.nasa.gov
verify return:1
---
Certificate chain
0 s:/C=US/ST=MD/L=Greenbelt/O=NASA/OU=MMS/CN=webhostname .edtfmmslinux.ecs.nasa.gov
i:/C=US/ST=MD/L=Greenbelt/O=NASA/OU=MMS/CN=webhostname .edtfmmslinux.ecs.nasa.gov
-----BEGIN CERTIFICATE-----
MIIDbDCCAlQC etc etc k7Pr1nRQG
3/NKQVqaSITGHGZBtlKjpw==
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=MD/L=Greenbelt/O=NASA/OU=MMS/CN=webhostname .edtfmmslinux.ecs.nasa.gov
issuer=/C=US/ST=MD/L=Greenbelt/O=NASA/OU=MMS/CN=webhostname .edtfmmslinux.ecs.nasa.gov
---
No client certificate CA names sent
---
SSL handshake has read 1639 bytes and written 711 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 921941EFFB19FA3158C751D155C012D5A418425BFAE94FEA1D99231A3CEF5D3C
Session-ID-ctx:
Master-Key: 13BFF7BE2B8ED18056BA54415026FC1ED133F47BADE2683A5EB3A2FED15F34ABD3F837985A498404A948B7F5B1F4774B
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - d6 99 6e 28 8c 86 5e 9b-e2 e8 etc. etc.
00b0 - 20 96 ea 05 9
Start Time: 1466432096
Timeout : 300 (sec)
Verify return code: 18 (self signed certificate)
---
On the client box I created a local ca-bundle.crt file in the USERNAME home directory. I "cp /etc/pki/tls/certs/WEBSERVER.crt ~/ca-bundle.crt" and had the perl script set the SSL_ca_file value to its path.
The Apache configuration file was update to use the /etc/pki/tls/cert/WEBSERVER.crt file and restarted.
And it still doesn't work. We have tried different web host name patterns but there is no change. We have no idea why the certificate is not working, but we think we are following the instructions correctly. Firefox is happy after we accept the certificate but perl is not. So what are we doing wrong?

Answer 1

After stracing the perl script it turned out to be sybase. The sybase client comes with an incompatible version of openssl. Net::SSLeay is an interface to openssl. IO::Socket::SSL uses Net::SSLLeay, LWP::UserAgent uses IO::Socket::SSL. When the perl script is run, the PATH has sybase directories in the front. sybase has its own CA files that are not compatible/out of date with the CA RHEL 7 and it reads the sybase CAs first messing everything up by the time it gets to ca.pem. To fix the problem we changed the perl script to strip out the sybase directories from PATH and then the SSL_ca_file option then works fine.
my $ua = LWP::UserAgent->new( ssl_opts => { verify_hostname => 1, SSL_ca_file => '/home/<server account>/certs/ca.pem'});
This is the second sybase side effect we have had since going to RHEL 7. The other one was sybase in increasing file descriptors limits to 4096 vs. the standard 1024 causing a boundary overflow in an array that was hard coded to 1024.