Check for errors in TLS implementation - security

Question

I'm developing a small TLS client, which is used together with SMTP. The handshake is working well until my client sends the encrypted finished message. So Client Hello, Server Hello, Certificate, Server Hello Done, Client Key Exchange and Change Cipher Spec are working. As chiper suite I'm using TLS_RSA_WITH_AES_256_CBC_SHA256.
My problem is, that I receive the "Bad Record MAC" alert from the server after sending the finished message. But I have no idea where I can start to search for the error. I've double checked all my functions and readed the RFC twice.
In my opionion one of the following points can cause the "Bad Record MAC" alert:
The master-secret is wrong.
The client_write_MAC_key is wrong.
The client_write_encryption_key is wrong.
The P_hash function is wrong.
The PRF function is wrong.
The MAC function is wrong.
The AES encryption doesn't work correctly.
The hash of the Finished Message is wrong.
Has anyone an idea what I can do to find the issue. Are there any tools to check if the master-secret, MAC and key calculations are correct? Or is it possible to decrypt the content with Wireshark? Note that I'm not having the private key of the server.

Answer 1

I've setted up an local test server with OpenSSL as Steffen Ullrich suggested. I've done a detailed analytics of the "debug output" which was provided by Wireshark.
I was able to solve a problem with my padding. But I'm still getting a Bad Record MAC alert. Like the debug output says, the MAC is incorrect (message ssl_decrypt_record: mac failed).
Some more information about the connection and the debug output:
The pre-master-secret, master-secret and all the keys (client write MAC key, server write MAC key, client write key, server write key, client write IV, server write IV) are correct. I've compared these from my software with them of the debug output. So everything until the Encrypted Finished Message is correct.
The server / Wireshark can decrypt the Encrypted Finished Message successfuly.
The server / Wireshark detects the padding correctly.
The server / Wireshark "skips" the IV and only shows the Finished Message and the MAC in the line Plaintext[64]:
In my opinion only two things can cause this error:
The MAC is calculated wrong.
The hash (verify_data) from the handshake messages are wrong.
My Encrypted Finished Message has a full length of 80 bytes and the following structure:
struct
{
// TLS record
ContentType type;
ProtocolVersion version;
uint16 length;
// TLS handshake and content (encrypted)
uint8 IV[16];
struct
{
HandshakeType msg_type;
uint24 length;
uint8 verify_data[12];
} content;
uint8 MAC[32];
uint8 padding[15];
uint8 paddingLength;
} FinishedMessage;
The content looks for example like the following:
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
0000 16 03 03 00 50 XX XX XX XX XX XX XX XX XX XX XX
0010 XX XX XX XX XX 14 00 00 0C YY YY YY YY YY YY YY
0020 YY YY YY YY YY ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ
0030 ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ
0040 ZZ ZZ ZZ ZZ ZZ 0F 0F 0F 0F 0F 0F 0F 0F 0F 0F 0F
0050 0F 0F 0F 0F 0F
Where XX represents the randomly generated IV and YY the verify_data and ZZ the MAC of the message like described in the RFC.
The verify_data contains the first 12 bytes of the SHA-256 hash. The hash is computed by the messages Client Hello, Server Hello, Certificate (from Server), Server Hello Done and Client Key Exchange. For hashing the full messages without the record header (first 5 bytes) are used.
The MAC is computed by the client write MAC key and the message. As sequence number the number 0 is used. The fragment which is used for calculating the MAC only contains the msg_type, length and verify_data (see structure content above).
Has anyone an idea how I can find what is wrong with my MAC?