How to sign on a javacard applet and return the signature to the host application - digital-signature

Question

I have the following function in the javacard applet that is supposed to receive a challenge from the host application, sign it, and return it to the host via command-response apdu communication.
private void sign(APDU apdu) {
if(!pin.isValidated())ISOException.throwIt(SW_PIN_VERIFICATION_REQUIRED);
else{
byte[] buffer = apdu.getBuffer();
byte [] output = new byte [20];
short length = 20;
short x =0;
Signature signature =Signature.getInstance(Signature.ALG_RSA_SHA_PKCS1, false);
signature.init(privKey, Signature.MODE_SIGN);
short sigLength = signature.sign(buffer, offset,length, output, x);
//This sequence of three methods sends the data contained in
//'serial' with offset '0' and length 'serial.length'
//to the host application.
apdu.setOutgoing();
apdu.setOutgoingLength((short)output.length);
apdu.sendBytesLong(output,(short)0,(short)output.length);
}
}
The host computes the challenge as follows and sends it to the javacard applet for signing:
//produce challenge
SecureRandom random = SecureRandom . getInstance( "SHA1PRNG" ) ;
byte [ ]bytes = new byte [ 20 ] ;
random . nextBytes ( bytes) ;
CommandAPDU challenge;
ResponseAPDU resp3;
challenge = new CommandAPDU(IDENTITY_CARD_CLA,SIGN_CHALLENGE, 0x00, 0x00,bytes ,20 );
resp3= c.transmit(challenge);
if(resp3.getSW()==0x9000) {
card_signature = resp2.getData();
String s = new String(card_signature);
System.out.println("signature " + s);
}else System.out.println("Challenge signature error: " + resp3.getSW());
As you can see, I check for both succesful and unsuccesful signing but I get the following printed out:
Challenge signature error:28416
Where exactly do I go wrong? Is it possible I retrieve the challenge in a faulty way with `byte[] buffer = apdu.getBuffer(); or is my signature all wrong?

Answer 1

You are trying to sign using an RSA key. However, the signature size of an RSA generated signature is identical to the key size (the modulus size) encoded in a minimum number of bytes. So e.g. a 2048 bit key results in a signature with size ceil(2028D / 8D) = 256 bytes (the maximum response size, unless you use extended length APDU's).
You should never create byte arrays in Java except when creating the class or when personalizing the applet. Any array created in persistent memory using new byte[] will likely remain until the garbage collector is run, and it may wear out the EEPROM or flash. And for signatures you don't need persistent memory.
If you look at the Signature.sign method:
The input and output buffer data may overlap.
So you can just generate the signature into the APDU buffer instead. Otherwise you can generate it in a JCSystem.makeTransientByteArray created buffer, but if you want to communicate it to the client you'll have to copy it into the APDU buffer anyway.
Please don't ever do the following:
String s = new String(card_signature);
A signature is almost indistinguishable from random bytes, so printing this out will generate just garbage. If you need text output try hexadecimals or base 64 encoding of the signature. Or print it as decimal number (but note that this may lead to loss of leading bytes with value 00).