base64 digest- invalid input? - linux

Question

I have here the following data for which I have to find the sha1 digest using openssl.
data:
AwAIAOwIAAABABwAgAIAABYAAAAAAAAAAAAAAHQAAAAAAAAAAAAAAAgAAAAkAAAAQgAAAFQAAABsAAAAhgAAAJgAAACuAAAAwgAAAM4AAADsAAAAAgEAAAwBAAAoAQAARgEAAFgBAACwAQAAtAEAANABAADkAQAA+gEAAAIAaQBkAAAADABsAGEAeQBvAHUAdABfAHcAaQBkAHQAaAAAAA0AbABhAHkAbwB1AHQAXwBoAGUAaQBnAGgAdAAAAAcAZwByAGEAdgBpAHQAeQAAAAoAYgBhAGMAawBnAHIAbwB1AG4AZAAAAAsAbwByAGkAZQBuAHQAYQB0AGkAbwBuAAAABwBwAGEAZABkAGkAbgBnAAAACQB0AGUAeAB0AEMAbwBsAG8AcgAAAAgAdABlAHgAdABTAGkAegBlAAAABAB0AGUAeAB0AAAADQBwAGEAZABkAGkAbgBnAEIAbwB0AHQAbwBtAAAACQBzAGMAYQBsAGUAVAB5AHAAZQAAAAMAcwByAGMAAAAMAHAAYQBkAGQAaQBuAGcAUgBpAGcAaAB0AAAADQBsAGEAeQBvAHUAdABfAHcAZQBpAGcAaAB0AAAABwBhAG4AZAByAG8AaQBkAAAAKgBoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBhAG4AZAByAG8AaQBkAC4AYwBvAG0ALwBhAHAAawAvAHIAZQBzAC8AYQBuAGQAcgBvAGkAZAAAAAAAAAAMAEwAaQBuAGUAYQByAEwAYQB5AG8AdQB0AAAACABUAGUAeAB0AFYAaQBlAHcAAAAJAEkAbQBhAGcAZQBWAGkAZQB3AAAABgBCAHUAdAB0AG8AbgAAAAAAgAEIAEQAAADQAAEB9AABAfUAAQGvAAEB1AABAcQAAQHVAAEBmAABAZUAAQFPAQEB2QABAR0BAQEZAQEB2AABAYEBAQEAARAAGAAAABEAAAD/////DwAAABAAAAACARAAsAAAABEAAAD//////////xIAAAAUABQABwAAAAAAAAAQAAAAAwAAAP////8IAAAREQAAABAAAAAFAAAA/////wgAABABAAAAEAAAAAAAAAD/////CAAAAR0AB38QAAAABAAAAP////8IAAABEQAGfxAAAAAGAAAA/////wgAAAUBEAAAEAAAAAEAAAD/////CAAAEP////8QAAAAAgAAAP////8IAAAQ/////wIBEACcAAAAGgAAAP//////////EwAAABQAFAAGAAAAAAAAABAAAAAIAAAA/////wgAAAUCEgAAEAAAAAcAAAD/////CAAAARAABn8QAAAACgAAAP////8IAAAFARgAABAAAAABAAAA/////wgAABD/////EAAAAAIAAAD/////CAAAEP7///8QAAAACQAAAP////8IAAABRwAIfwMBEAAYAAAAIAAAAP//////////EwAAAAIBEAB0AAAAIgAAAP//////////EgAAABQAFAAEAAAAAAAAABAAAAAFAAAA/////wgAABAAAAAAEAAAAAQAAAD/////CAAAAREABn8QAAAAAQAAAP////8IAAAQ/////xAAAAACAAAA/////wgAABD+////AgEQAIgAAAAoAAAA//////////8UAAAAFAAUAAUAAAAAAAAAEAAAAA0AAAD/////CAAABQEYAAAQAAAAAQAAAP////8IAAAQ/v///xAAAAACAAAA/////wgAABD+////EAAAAAwAAAD/////CAAAAQEAAn8QAAAACwAAAP////8IAAAQBQAAAAMBEAAYAAAALQAAAP//////////FAAAAAIBEAB0AAAALwAAAP//////////EgAAABQAFAAEAAAAAAAAABAAAAAFAAAA/////wgAABABAAAAEAAAAAEAAAD/////CAAABQEAAAAQAAAAAgAAAP////8IAAAQ/v///xAAAAAOAAAA/////wgAAAQAAIA/AgEQAHQAAAA1AAAA//////////8VAAAAFAAUAAQAAAAAAAAAEAAAAAAAAAD/////CAAAASgAB38QAAAAAQAAAP////8IAAAQ/////xAAAAACAAAA/////wgAABD+////EAAAAAkAAAD/////CAAAARUACH8DARAAGAAAADgAAAD//////////xUAAAACARAAdAAAADoAAAD//////////xUAAAAUABQABAAAAAAAAAAQAAAAAAAAAP////8IAAABKgAHfxAAAAABAAAA/////wgAABD/////EAAAAAIAAAD/////CAAAEP7///8QAAAACQAAAP////8IAAABGgAIfwMBEAAYAAAAPQAAAP//////////FQAAAAMBEAAYAAAAPwAAAP//////////EgAAAAIBEAB0AAAAQQAAAP//////////EgAAABQAFAAEAAAAAAAAABAAAAAFAAAA/////wgAABABAAAAEAAAAAEAAAD/////CAAABQEAAAAQAAAAAgAAAP////8IAAAQ/v///xAAAAAOAAAA/////wgAAAQAAIA/AgEQAHQAAABHAAAA//////////8VAAAAFAAUAAQAAAAAAAAAEAAAAAAAAAD/////CAAAASkAB38QAAAAAQAAAP////8IAAAQ/////xAAAAACAAAA/////wgAABD+////EAAAAAkAAAD/////CAAAARYACH8DARAAGAAAAEoAAAD//////////xUAAAACARAAdAAAAEwAAAD//////////xUAAAAUABQABAAAAAAAAAAQAAAAAAAAAP////8IAAABKwAHfxAAAAABAAAA/////wgAABD/////EAAAAAIAAAD/////CAAAEP7///8QAAAACQAAAP////8IAAABGQAIfwMBEAAYAAAATwAAAP//////////FQAAAAMBEAAYAAAAUQAAAP//////////EgAAAAMBEAAYAAAAUwAAAP//////////EgAAAAMBEAAYAAAAVQAAAP//////////EgAAAAEBEAAYAAAAVQAAAP////8PAAAAEAAAABgAAAA9AAAA//////////8fAAAAAgEQAGAAAAA/AAAA//////////8eAAAAFAAUAAMAAAAAAAAAGQAAAAUAAAD/////CAAAEAAAAAAZAAAAAAAAAP////8IAAAQ/v///xkAAAABAAAA/////wgAABD+////AgEQAMQAAABEAAAA//////////8gAAAAFAAUAAgAAAAAAAAAGQAAABIAAAD/////CAAABQIOAAAZAAAAEQAAAP////8IAAARAQAAABkAAAAQAAAA/////wgAAAEGAAZ/GQAAAAIAAAD/////CAAAARIAB38ZAAAAEwAAAP////8IAAAFAQQAABkAAAAAAAAA/////wgAABD+////GQAAAAEAAAD/////CAAAEP7///8ZAAAADwAAAP////8IAAABMwAIfwMBEAAYAAAASwAAAP//////////IAAAAAIBEACIAAAATQAAAP//////////IAAAABQAFAAFAAAAAAAAABkAAAASAAAA/////wgAAAUCDgAAGQAAABAAAAD/////CAAAAQYABn8ZAAAAAgAAAP////8IAAABEwAHfxkAAAAAAAAA/////wgAABD+////GQAAAAEAAAD/////CAAAEP7///8DARAAGAAAAFEAAAD//////////yAAAAADARAAGAAAAFMAAAD//////////x4AAAACARAAYAAAAFUAAAD//////////x4AAAAUABQAAwAAAAAAAAAZAAAABQAAAP////8IAAAQAAAAABkAAAAAAAAA/////wgAABD+////GQAAAAEAAAD/////CAAAEP7///8CARAAxAAAAFoAAAD//////////yAAAAAUABQACAAAAAAAAAAZAAAAEgAAAP////8IAAAFAg4AABkAAAARAAAA/////wgAABEBAAAAGQAAABAAAAD/////CAAAAQYABn8ZAAAAAgAAAP////8IAAABFAAHfxkAAAATAAAA/////wgAAA
The digest as given to me is:
Wk2pJnOErEHsElMw4TMX+rjHsQQ=
But when I use(f1= file where I copied the above data):
base64 -d f1.txt | openssl dgst -sha1 -binary | base64
I get a "base64: invalid input" error and the following digest which seems completely different :(
BaRlDid73RYBFMgqveC8G+gFBBU=
Can somebody confirm and explain if there is some mistake??
UPDATED:
Scenario: Client's binary file is base64 encoded and sent to server. Server decodes this and computes the sha1 digest. Since I have client's base64 encoded sha1 digest, the server also encodes the digest to base64. Now these two should match. And it doesn't! I receive all data. I have rechecked it. I shall present part of the code here:
//RCVBUFSIZE = 1024 (defined)
void HandleClient(int clntSocket)
{
char echoBuffer[RCVBUFSIZE] ; /* Buffer for echo string */
memset(echoBuffer, 0, RCVBUFSIZE);
char inBuffer; /* Buffer for first string */
char recv_data;
int recvMsgSize = 0; /* Size of received message */
char replyBuffer[32];
int bytes_received = 0;
int rv = 0;
int connected = clntSocket;
int len= 0;
int i = 0;
EVP_MD_CTX md_ctx;
const EVP_MD *md;
unsigned char md_value[EVP_MAX_MD_SIZE];
unsigned int md_len;
OpenSSL_add_all_digests();
md = EVP_get_digestbyname("sha1");
EVP_MD_CTX_init(&md_ctx);
EVP_DigestInit_ex(&md_ctx, md, NULL);
/* Receive message from client */
while (((bytes_received = recv(connected,&inBuffer,1,0)) > 0) && (inBuffer != '\n')){
/* Send received string and receive again until end of transmission */
if (bytes_received > 0) /* zero indicates end of transmission */
{
printf("Message received from Client is : %c\n", inBuffer);
char n = inBuffer;
int indicator = 0;
int current = 0;
unsigned long fileLen;
if(n =='6'){
if ((recvMsgSize = recv(connected, echoBuffer, RCVBUFSIZE, 0)) < 0)
DieWithError("recv() failed");
printf("no. of bytes got : %d\n", recvMsgSize);
if (recvMsgSize > 0)
echoBuffer[recvMsgSize] = '\0';
len= atoi(echoBuffer);
char *data =NULL;
printf("length of following message : %d\n", len);
if(len>0){
for( i = RCVBUFSIZE; i < (len+RCVBUFSIZE); i=i+RCVBUFSIZE){
if(i>len)
recvMsgSize = recv(connected, echoBuffer, (len - (i-RCVBUFSIZE)), 0);
else
recvMsgSize = recv(connected, echoBuffer, RCVBUFSIZE, 0);
echoBuffer[recvMsgSize] = '\0';
decode(echoBuffer, recvMsgSize, "file_out");
data = readFileBuffer("file_out");
EVP_DigestUpdate(&md_ctx, data, strlen(data));
}
}
len = 0;
memset(echoBuffer, 0, RCVBUFSIZE);
recvMsgSize = 0;
}
if (n =='5'){
printf("Update Digest Over- Calculate Final Dgst!!!!! \n");
n= 0;
EVP_DigestFinal_ex(&md_ctx, md_value, &md_len); //retrieve digest from ctx unto md_value and #bytes written is copied into md_len
EVP_MD_CTX_cleanup(&md_ctx);
FILE *f;
f = fopen("file_sha1", "w");
printf("\n");
printf("******************************************************\n ");
printf("Digest is: ");
for(i = 0; i < md_len; i++){
if ( f !=NULL){
fputc(md_value[i], f);
}
printf("%02x", md_value[i]);
}
printf("\n");
printf("******************************************************\n ");
fclose(f);
}
printf("socket closing\n");
close(connected); /* Close client socket */
}
}
char *readFileBuffer(char *name)
{
FILE *file;
char *buffer = NULL;
unsigned long fileLen;
//Open file
file = fopen(name, "rb");
if (!file)
{
fprintf(stderr, "Unable to open file %s", name);
return;
}
//Get file length
fseek(file, 0, SEEK_END);
fileLen=ftell(file);
printf("file length = %ld\n", fileLen);
fseek(file, 0, SEEK_SET);
//printf("Allocate memory\n");
buffer=(char *)malloc(fileLen+1);
printf("length of write buffer = %d\n", strlen(buffer));
if (!buffer)
{
fprintf(stderr, "Memory error!");
}
long int n = fread(buffer,1, fileLen,file);
buffer[n] = '\0';
printf("Read no. of bytes = %ld into buffer \n", n);
printf("len of buffer %d \n", strlen(buffer));
if (!buffer)
{
fprintf(stderr, "Memory error!");
fclose(file);
}
fclose(file);
//free(name);
return buffer;
}
// reads b64 encoded msg (ReadBuffer) and writes to WriiteFile.
void decode(char *ReadBuffer, int Length, char *WriteFile)
{
char *msg = (char *)malloc(Length);
memset(msg, 0x00, Length);
int readbytes = -1;
printf("buffer write file %s\n", WriteFile);
// the decode msg is written to this bio
BIO *fileWrBIO = BIO_new_file(WriteFile, "w");
BIO *b64 = BIO_new(BIO_f_base64());
BIO *bio = BIO_new_mem_buf(ReadBuffer, Length);
bio = BIO_push(b64, bio);
BIO_set_flags(bio,BIO_FLAGS_BASE64_NO_NL);
while ((readbytes = BIO_read(bio, msg, Length)) > 0)
{
printf("readbytes: %d\n", readbytes);
BIO_write(fileWrBIO, msg, readbytes);
BIO_flush(fileWrBIO);
memset(msg, 0x00, sizeof(msg));
}
free(msg);
BIO_free_all(bio);
BIO_free_all(fileWrBIO);
}

Answer 1

FWIW...
There are implementations where base64 cannot read its own output.
# base64 ssh_host_rsa_key | base64 -d
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA7qHASF1Jgbase64: invalid input
This is on a CentOS 5 machine.
Reason is that base64 produces output with line breaks, which are garbage chars for the decoder.
Solution is to either produce the base64 without linebreaks (-w 0) or make the decoder ignore garbage chars (-i).

Answer 2

Your data is invalid, probably partial. A valid base64 encoded string should have a length multiple of 4. So the different digest output is expected.

Answer 3

You can encrypt using this command
base64 -w 0 < id_rsa

Answer 4

Well, the data doesn't seem to be a valid base64 string. You might be missing some characters.

Answer 5

Just hit this, also on CentOS 5. Both -w 0 and -i were required, -i didn't work alone. e.g.:
tar -cf - /home/backup | gzip | base64 -w 0
base64 -d -i | gunzip | tar -xvf - -C /
worked fine to move a small home directory via copy&paste.