[squeak-dev] [Vm-dev] SHA512 squeak implementation?

Robert robert.withers at pm.me
Tue Mar 10 22:06:53 UTC 2020

Hi Levente,

Here is a new release of CryptographyHashing-rww.15. It is not linked up through dependencies, so load it after. It supports SHA512WithPrimitive and SHA512NonPrimitive and passes all tests. CryptoGreen for SHA512, with the shiny, new SHA2Plugin and without. Find plugins here, for linux64x64: https://www.dropbox.com/home/Callisto%20House/squeak-crypto-plugins.

Here is this working implementation of SHA512. The naming ought to be without prefix for th ecore classes. I have no problem whatsoever if we were to rebase your work as the defining implementation for all of thosew funcrtions, using one plugin. That's something wonderful. We should use you hashFunction and rename without prefix. Tests separate, that's fashionable. We can figure out the mc config later.

publish your work on, then I will link your solution into dependencies.


CryptographyHashing-rww.15 (Release)

File:	[CryptographyHashing-rww.15.mcz](http://www.squeaksource.com/Cryptography/CryptographyHashing-rww.15.mcz)
Author:	[Robert Withers](http://www.squeaksource.com/@x4F_VDyHPTW39OmM/1dY6OOdf?24)
Timestamp:	10 March 2020 9:57:39 pm
UUID:	b7df722e-ab05-4465-97ef-deeffb0212d0
Ancestors:	[CryptographyHashing-rww.14](http://www.squeaksource.com/@x4F_VDyHPTW39OmM/1dY6OOdf?25)
Dependencies:	[CryptographyCore-rww.5](http://www.squeaksource.com/@x4F_VDyHPTW39OmM/1dY6OOdf?26)
Release:	This is a release that can be read by anybody.
Message:	adapt to new #primCopyoubleWords:intoBytes:.
CryptoGreen for SHA512, with the shiny, new SHA2Plugin and without. Find plugins here, for linux64x64: https://www.dropbox.com/home/Callisto%20House/squeak-crypto-plugins.

rttyk, r

On 3/10/20 4:48 PM, Levente Uzonyi wrote:

> Hi Robert,
> On Mon, 9 Mar 2020, Robert wrote:
>> Hi Levente,
>> On 3/9/20 5:14 PM, Levente Uzonyi wrote:
>>> Hi Robert,
>>> On Mon, 9 Mar 2020, Robert wrote:
>>>> Hi Levente,
>>>> Have you seen my other posts on ProCrypto and the work I did to the
>>>> Hashing package? After I ported your code, the size went up to 730 kb,
>>>> while the rest of Cryptography was about 270 kb. So there was a lot of
>>> Yes, I've seen your messages.
>> Too many...I ride like a storm. Always have done so. I do hope that you
>> are not upset about my ripping SHA512 into a Crypto package. I feel very
>> strongly that a commitment by provider implementers doing Crypto to
>> subclass the Root classes: BlockCipher, HashFunction, BlockCipherMode,
>> and importantly adopt the common protocol inherited from those classes.
>> Then all Crypto implemeters will be able to easily interoperate.
>> Producing a lookup interface, much like Java's provider lookup, will be
>> a step we could take forwards with the shared and interoperable Crypto
>> library packages.
> Are there any other "Crypo implementers" having hash functions? I doubt
> it. So, I don't think we need anything like this. We need simple code
> that works.
>>> As I mentioned in my previous email, the size of the package is large
>>> because of the test strings. I split the package up into Core and Tests.
>>> If you have a look at the Hasher repository, you'll find that
>>> HAHasher-Core.1 is 24.6 kB, while HAHasher-Tests.1 is 650kB.
>> I see, alright, that's fair dinkum. I also feel strongly that a minimum
>> of tests be carried within the primary package for some code. If you
>> look at the Crypto packages, you'll see Hashing tests in hashing, Random
>> tests in Random, Cipher tests in Cipher. This way it is easy to run
>> them, off a CI server or what have you. The tests come with the code, is
> As others wrote, this is not a good idea. You don't need those tests in
> your production images.
>> a principle I try to follow. I suspect that maybe we could reduce the
>> tests, for SHA512, to a minimum. I will look into this. I also feel like
>> having a separate package with more robust testing is fair dinkum. I
>> think that would make a good solution here, where you could have a
>> couple of tests with the package and a separate Tests package for the
>> rest of them.
>>> By bloat I didn't mean that the size of the package is too large. I meant
>>> that there are too many methods and classes are added to the environment
>>> in a non-isolated way (no class prefixes, extension methods). And when you
>>> only need a small chunk of those, you've got no way to load just that.
>>>> bloat. As we already have solutions for SHA1 and SHA256, I really only
>>>> needed SHA512. So I removed all the other classes and tests and ported
>>> I think community needs come first, so having all SHA2 hash functions is
>>> better than just having SHA512.
>> Alright, how does this proposed approach grab you? I have a Hashing
>> package as a part of the ProCrypto offering. Dependencies are setup in
>> Monticello for Random to pull in that Hashing package, which has all our
>> traditional functions and now has SHA512, thanks to your offering. Here
>> is my further thought, If you interface with the common Crypto
>> superclasses (reclass HAHashFunction under HashFunction) and adopt the
>> Crypto protocol, your package as a provider package that could be loaded
>> side by side with Crypto's Hashing package. You would naturally extend
>> our offerings. This is my approach with SSL and SSH, as the port
>> progresses so slooowly, running as a specialization of the ParrotTalk
>> code, so it just fits in and works as an option.  I really see a
>> potential explosion of such solutions, if colleges and universities get
>> onboard. *dreaming about a better future*
>> Would this solution work well for you, Levente?
> I did't clean up and restructure HashFunction in Hasher just to
> reintroduce the stuff I removed/replaced.
> Hasher is a replacement for the whole HashFunction hierarchy.
>>>> SHA512 onto the HashFunction defined in CryptoCore. I removed all the
>>>> RGThirtyTwoBitRegisters and I plan to move this ported SHA512 off of
>>>> using a RGSixtyFourBitRegister, to eliminate that code. The size is
>>> I see no value in doing that. The code is there, complete, ready for
>>> integration.
>> As an alternate Crypto package, folks could choose your solution. This
>> way we can align our efforts together and mix and match solution
>> packages. That seems like the exact approach to managing Cryptography
>> library and how we conducts our affairs.
> Our community is not large enough for that.
>>>> currently 204 kb for Hashing. I do not know how this plays with your
>>>> plugin, but there should just be a SHA512Plugin.
>>>> Thanks for digging in and writing your plugin! Can it be changed to just
>>>> doing SHA512, easily?
>>> It is designed to support all SHA2 functions. You may be able to remove
>>> the SHA224/SHA256 stuff, but I see no value in that. This plugin is better
>>> than SHA256Plugin.
>> I must tell you, that I am totally unfamiliar with the SHA256Plugin,
>> someone else wrote it over a decade ago! As such, I cannot evaluate the
>> quality of one over the other, so I will trust your views.
> I wrote SHA256Plugin and MD5Plugin. I wrote the image side code for
> SHA256, and I added plugin support to MD5 on the image side too.
> Other than these hashes, the Cryptography package has SHA1, which is
> buggy (try SHA1 hashMessage: ''), MD2 and MD4, which are obsolete, and
> nothing else.
> So, again, I suggest replacing the HashFunction hierarchy in Cryptography
> with Hasher.
> Levente
>> K, r
>>> Levente
>>>> Kindly,
>>>> Robert
>>>> On 3/9/20 3:11 PM, Levente Uzonyi wrote:
>>>>> Hi Robert,
>>>>> During the weekend I wrote a new plugin, SHA2Plugin, which covers all
>>>>> cases of SHA2, therefore obsoletes SHA256Plugin. The code is available on
>>>>> SqueakSource:
>>>>> http://www.squeaksource.com/Cryptography/CryptographyPlugins-ul.18.mcz
>>>>> I've updated the HAHasher package with hash variants using the new plugin
>>>>> (and the old MD5 and SHA256 plugins), and split it up to Core and Tests,
>>>>> so that the tests with the huge strings can be loaded optionally.
>>>>> After building a VM with the new plugin, load:
>>>>> Installer ss
>>>>>        project: 'Registers';
>>>>>        install: 'Registers';
>>>>>        project: 'Hasher';
>>>>>        install: 'HAHasher-Core';
>>>>>        install: 'HAHasher-Tests'.
>>>>> However, SHA2Plugin is not complete yet. There's an issue with the VM's
>>>>> code generator affecting 32-bit platforms, and I want to change the
>>>>> methods swapping endianness before the plugin is added to the VMs.
>>>>> On Sat, 7 Mar 2020, Robert wrote:
>>>>>> Hi Levente,
>>>>>> Between this and your HashFunctions and my email in response to Jakob, I
>>>>>> have merged your work into Cryptography, published as:
>>>>>> Name: Cryptography-v5.3-rww.121
>>>>>> Author: rww
>>>>>> Time: 7 March 2020, 12:13:00.184627 pm
>>>>>> UUID: 2b69a713-eac3-4680-9d00-8104eea5a7da
>>>>>> Ancestors: Cryptography-v5.3-rww.120
>>>>>> Ported in the Registers & HAHasher packages and merged into
>>>>>> Cryptography. CryptoGreen!
>>>>> I suppose my point didn't get through. In my opinion, the code in
>>>>> Hasher/HAHasher package should replace Cryptography's HashFunction and
>>>>> subclasses. Why?
>>>>> - it's a single package already (except for its dependency on Registers,
>>>>> but I see no problem with Cryptography having dependencies)
>>>>> - slimmer: fewer, simpler methods per class
>>>>> - faster: uses the highly optimized Registers package when plugins are not
>>>>> available; has plugin support for all implemented hashes (SHA2Plugin);
>>>>> highly optimized in general
>>>>> - way more hashes: supports all SHA-2 variants except for SHA-512/t for
>>>>> arbitrary t other than 224, 256, 384, but that can be implemented as well
>>>>> if there's need for it.
>>>>> - has more comments and tests (e.g. check out SHA1 hashMessage: '' in
>>>>> Cryptography)
>>>>> - no bad tricks (e.g. check out SHA256 new class, or MD5 new class)
>>>>>> I am now looking to recategorize to fragment Cryptography. Future work
>>>>>> should be released to the whole enchilada Cryptography package and
>>>>>> copied to the correct fragment. For example reorging, rebasing and
>>>>>> renaming HASHA512 to SHA512. As I fragment I will be adding required
>>>>>> packages. It seems like we are unable to specify a version to the
>>>>>> required package. Is this the case?
>>>>>> QUESTION TO CRYPTOGRAPHY TEAM: How do y'all feel about fragmenting
>>>>>> Cryptography? Pros? Cons?
>>>>>> ---
>>>>>> Here is a profile of Cryptography and I see many leaves for the various
>>>>>> Registers, so there may be some optimization that could occur.
>>>>> Registers is a highly optimized package, which should perform better
>>>>> than ThirtyTwoBitRegister, especially on 64-bit platforms. If you think
>>>>> you can do even better in pure Smalltalk, I'm all ears.
>>>>> Levente
>>>>>> - 6250 tallies, 6325 msec.
>>>>>> **Leaves**
>>>>>> 7.5% {476ms} RGSixtyFourBitRegister64>>loadFrom:
>>>>>> 7.5% {472ms} Random>>nextBytes:into:startingAt:
>>>>>> 6.8% {427ms} LargePositiveInteger(Integer)>>bitShift:
>>>>>> 5.2% {328ms} [] SystemProgressMorph(Morph)>>updateDropShadowCache
>>>>>> 4.4% {280ms} RGSixtyFourBitRegister64>>bitXor:
>>>>>> 3.5% {220ms} Random>>generateStates
>>>>>> 3.0% {192ms} RGThirtyTwoBitRegisterTest64(TestCase)>>timeout:after:
>>>>>> 2.9% {181ms} RGSixtyFourBitRegister64>>+=
>>>>>> 2.8% {178ms} HASHA256Inlined64>>processBuffer
>>>>>> 2.7% {170ms} RGThirtyTwoBitRegister64>>loadFrom:
>>>>>> 2.2% {140ms} RGThirtyTwoBitRegister64>>+=
>>>>>> 2.1% {134ms} RGSixtyFourBitRegisterTest64(TestCase)>>assert:description:
>>>>>> 2.1% {133ms} Point>>=
>>>>>> 1.7% {109ms} RGThirtyTwoBitRegister64>>bitXor:
>>>>>> 1.7% {107ms} SHA1>>hashStream:
>>>>>> 1.5% {96ms} RGSixtyFourBitRegister64>>leftRotateBy:
>>>>>> 1.4% {89ms} RGSixtyFourBitRegister64>>load:
>>>>>> 1.4% {89ms} RGSixtyFourBitRegister32>>load:
>>>>>> 1.4% {89ms} SmallInteger(Number)>>negative
>>>>>> 1.4% {88ms} []
>>>>>> RGThirtyTwoBitRegisterTest64(RGRegisterTest)>>testLeftRotateBy
>>>>>> 1.2% {77ms} LargePositiveInteger>>*
>>>>>> 1.2% {76ms} GrafPort>>fillRoundRect:radius:
>>>>>> 1.1% {70ms} GrafPort>>copyBits
>>>>>> 1.1% {67ms} DisplayScreen(Form)>>depth
>>>>>> **Memory**
>>>>>>       old            +0 bytes
>>>>>>       young        +2,071,872 bytes
>>>>>>       used        +2,071,872 bytes
>>>>>>       free        -2,071,872 bytes
>>>>>> **GCs**
>>>>>>       full            1 totalling 74 ms (1.17% uptime), avg 74 ms
>>>>>>       incr            223 totalling 43 ms (0.7% uptime), avg 0.2 ms
>>>>>>       tenures        5,503 (avg 0 GCs/tenure)
>>>>>>       root table    0 overflows
>>>>>> K, r
>>>>>> On 3/7/20 9:39 AM, Robert wrote:
>>>>>>> Hi Levente,
>>>>>>> Thanks for your comments, I totally understand your views. I have some
>>>>>>> response, but the bottom line is that I or others in Cryptography
>>>>>>> (looking for new college age recruits...) will harvest some from your
>>>>>>> packages. It may only be SHA512 that we port over to round out the hash
>>>>>>> function offerings. I appreciate your willingness to allow us to do that.
>>>>>>> Your main argument is that Cryptography is too bloated, both with
>>>>>>> obsolete algorithms as well as combining the various facets into one
>>>>>>> package. The prime function of the Cryptography package is to have one
>>>>>>> package with all the Crypto we have, for single click loading. This also
>>>>>>> supports the Crypto toolbox/sandbox approach providing an educational
>>>>>>> environment to learning Crypto. Crypto is a learning package, as well as
>>>>>>> being fully functional.
>>>>>>> Cryptography, while larger than most packages, comes in at 300 kb, with
>>>>>>> the recent addition of Blowfish. That is loaded into an image file that
>>>>>>> is 47 MB! This means that Cryptography is just 0.6% the size of the
>>>>>>> image. This does not register to me as too bloated. But I see your
>>>>>>> perspective with outdated Crypto being present as well as the
>>>>>>> unnecessary add-ons, like X09 and ASN1; ciphers and hashes.
>>>>>>> When looking to do end to end encryption, we would load all of
>>>>>>> Cryptography (304 kb), ParrotTalk (93 kb), SSL (350 kb), Telnet (101 kb)
>>>>>>> & SSH (49 kb), for a total of 900 kb. That is only 1.9 % of the total
>>>>>>> image. Note that as I migrate SSL & SSH over onto the ParrotTalk
>>>>>>> framework, I expect to see the sizes of those two packages drastically
>>>>>>> reduce, through adoption of reuse.
>>>>>>> 1) having other hashes completes the scope of Crypto even if not utilized.
>>>>>>> 2) I look forward to diving deeper and understand your point here as
>>>>>>> well as see how your instances are created.
>>>>>>> 3) I would port Registers, as well.
>>>>>>> The absence of an ability to easily link dependencies provides a
>>>>>>> challenge to breaking Cryptography up. If only Monticello allowed for
>>>>>>> dependencies,  It is doable to break up Cryptography.
>>>>>>> So let us imagine a future where Cryptography stabilizes again. Before
>>>>>>> declaring Crypto stable, I would include adding the Signal encryption
>>>>>>> needs into the Crypto package. This would include the double ratchet
>>>>>>> block cipher mode. As that point of stability is reached, we could
>>>>>>> preserve the total package for one click loading and also reduce and
>>>>>>> break it up into pieces. How would it sound to you if old obsolete
>>>>>>> functions & ciphers are removed, then the ASN1 and X509 is split off. At
>>>>>>> this point to your point let us consider splitting off the ciphers and
>>>>>>> the hash functions and the randomizers, leaving a Crypto base. Then a
>>>>>>> pro user would load Crypto-base, Randomizers, Hash Functions (No MD2...)
>>>>>>> and Ciphers (No DES...).
>>>>>>> I tried to use your example of use of the Installer to load Cryptography
>>>>>>> and it did not work. It could not find the versions.
>>>>>>> Installer ss
>>>>>>>	project: 'Cryptography';
>>>>>>>	install: 'Cryptography-v5.3'.
>>>>>>> If this could be made to work, then small scripts could load this split up Crypto pro solution. Optionally loading the different pieces.
>>>>>>> Installer ss
>>>>>>>	project: 'Cryptography';
>>>>>>>	install: 'Cryptography-base-v5.3';
>>>>>>>	install: 'Cryptography-core-v5.3';
>>>>>>>	install: 'Cryptography-hash-v5.3';
>>>>>>>	install: 'Cryptography-cipher-v5.3';
>>>>>>>	install: 'Cryptography-ASN1-v5.3';
>>>>>>>	install: 'Cryptography-X509-v5.3';
>>>>>>>	install: 'ParrotTalk';
>>>>>>>	install: 'SSL';
>>>>>>>	install: 'Telnet';
>>>>>>>	install: 'SSH';
>>>>>>>	install: 'Signal'.
>>>>>>> Kindly,
>>>>>>> Robert
>>>>>>> On 3/7/20 5:20 AM, Levente Uzonyi wrote:
>>>>>>>> Hi Robert,
>>>>>>>> On Thu, 5 Mar 2020, Robert wrote:
>>>>>>>>> Oh yes, Levente, I recall speaking with you about it. I would like to
>>>>>>>>> make a proposal. Do you think you could fold all those hash functions,
>>>>>>>>> without the HA, into the Cryptography library? We have a HashFunction
>>>>>>>>> class in there, I do not know how different they may be in their public
>>>>>>>>> interface. I think it would be valuable to combine them. To support TLS
>>>>>>>>> 1.3, we would also need elliptical Diffie-Hellmans, I think.
>>>>>>>>> Levente, would you be willing to fold your work into Cryptography?
>>>>>>>> The reason why I created a separate package was that I found the
>>>>>>>> Cryptography package too bloated. Cryptographic hash functions seem to be
>>>>>>>> more commonly needed than ciphers, CSPRNGS, ASN1, etc.
>>>>>>>> It is possible to replace HashFunction and subclasses from Cryptography
>>>>>>>> with those in Hasher, but there would be some consequences:
>>>>>>>> 1) Hasher doesn't have MD2 or MD4, but those are obsolete and broken. I
>>>>>>>> see little to no value rewriting them to satisfy Hasher's HashFunction
>>>>>>>> requirements, but it shouldn't be too hard to do that.
>>>>>>>> 2) the way instances are created differ. I didn't want to do it the way
>>>>>>>> it's done in Cryptography's MD5, SHA1, SHA256, where class side #new may
>>>>>>>> return an object that is not of that class but a subclass. So, I added
>>>>>>>> instance creation methods to Hasher's HashFunction which return an
>>>>>>>> instance optimized for the current platform. So, a few methods need to
>>>>>>>> be changed in Cryptography to use the optimized hash functions.
>>>>>>>> 3) Cryptography would depend on the Registers package.
>>>>>>>> Levente
>>>>>>>>> Kindly,
>>>>>>>>> Robert
>>>>>>>>> On 3/5/20 12:43 PM, Levente Uzonyi wrote:
>>>>>>>>>> Hi Robert,
>>>>>>>>>> The mail you are looking for is here:
>>>>>>>>>> http://lists.squeakfoundation.org/pipermail/vm-dev/2020-March/032986.html
>>>>>>>>>> Since that email, to make life easier to those who have the Cryptography
>>>>>>>>>> package loaded in their images, I've uploaded another variant of
>>>>>>>>>> Hasher: HAHasher. It's the same as the Hasher package but all class
>>>>>>>>>> names are prefixed with HA.
>>>>>>>>>> To load that, evaluate:
>>>>>>>>>> Installer ss
>>>>>>>>>>	project: 'Registers';
>>>>>>>>>>	install: 'Registers';
>>>>>>>>>>	project: 'Hasher';
>>>>>>>>>>	install: 'HAHasher'.
>>>>>>>>>> And then you can write
>>>>>>>>>> HAHashFunction newSHA512 hashMessage: 'test'.
>>>>>>>>>> Levente
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20200310/7f705fe0/attachment-0001.html>

More information about the Squeak-dev mailing list