Denis,
 
I do not know why I was looking at PKCS#11.  THe RSA spec is PKCS#1.  In that document (ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-1/pkcs-1v2-1.pdf) on page 25 it says:
 
"Two signature schemes with appendix are specified in this document: RSASSA-PSS and RSASSA-PKCS1-v1_5."
 
I implemented v1_5.  It may be that Java is using PSS.  I may have implemented v1.5 wrong.  The signature creation and verification algorithms start on page 30.  The encoding is on 35.
 
Rob

From: Rob Withers
Sent: Tuesday, September 21, 2010 12:06 PM
To: The general-purpose Squeak developers list
Cc: Squeak Crypto
Subject: Re: [squeak-dev] Crypto RSAWithSHA1 sign

Hi Denis,
 
I originally wrote the v15 signature methods in April of 2007.  I am currently trying to download the PKCS#11 V2.30 doc to verify, but IIRC there are more than one signature algorithm defined for RSA.  I don't recall why I chose v15.  Perhaps Java is using another RSA signature function.
 
There are no explicit tests for this signature.  There is a test inside of the CryptoX509Test  (#verifySHA1WithRSAEncryptionFromParentCertificate: ), but it isn't used as the certificate that exposed it has expired and so is failing.  I removed that certificate test.
 
Let's talk bytes...the way this works in Squeak is that the RSA pads the SHA1 hashed message and encrypts it.
 
v15SignMessage: aMessage
 
 ^ self v15SignMessageHash: (SHA1 hashMessage: aMessage).
and
 
v15SignMessageHash: encodedMsg
 
 | padded toBeSigned |
 padded := ByteArray new: (256 - encodedMsg size - 3) withAll: 255.
 toBeSigned := #(0) asByteArray, #(1) asByteArray, padded, #(0) asByteArray, encodedMsg.
 ^ (self crypt: toBeSigned asInteger) asByteArray.
Presumably the #crypt: function will be the same in Java and Squeak given the same key.  So if there are 2 different signature functions in RSA, I would suspect that the padding would be different.
 
Still trying to download the spec....
 
What do you think?
 
Cheers,
Rob

From: Denis Kudriashov
Sent: Tuesday, September 21, 2010 11:21 AM
To: The general-purpose Squeak developers list
Subject: [squeak-dev] Crypto RSAWithSHA1 sign

Hello

Is somebody use Cryptography for RSA with SHA1 digital signature?

I try do same result as I hava in java programm
I have rsa private key as smalltalk object. It has same values as java private key object.

But code

privateKey v15SignMessage: message asByteArray  .

returns me wrong result. Its differ from java working test