Hi all,
I want to spread an algorithm over several images. For that I need to transfer some objects.
For other purposes I already read and write objects via JSON files. To change this to UDP is straightforward. But the conversion to JSON and back takes considerable time, biggest chunk of data is an array with floats.
Is there a fast and easy way to transfer objects binary between images? I can transfer only the array of floats (maybe a float array) if that saves time.
Thanks,
Herbert
I think ASN1 inside of the Cryptography package may have some speed and now supports Floats, ScaledDecimals and Fractions, I think. I would love to hear your real world comparison to json, as I am starting to adapt STON to my needs.
On 10/20/2017 06:18 AM, Herbert König wrote:
Hi all,
I want to spread an algorithm over several images. For that I need to transfer some objects.
For other purposes I already read and write objects via JSON files. To change this to UDP is straightforward. But the conversion to JSON and back takes considerable time, biggest chunk of data is an array with floats.
Is there a fast and easy way to transfer objects binary between images? I can transfer only the array of floats (maybe a float array) if that saves time.
Thanks,
Herbert
Yes. Serialized ReferenceStream .obj files (see attached) You could transfer huge complex objects (complete 6000+ .html pages of swiki) between Squeak and Cuis and with some extra work to Pharo too Or you could convert your JSON to .obj with the attached .mcz and see if more convenient tranfer JSON or .obj yourself
On 10/20/17, 07:18, "Herbert König" herbertkoenig@gmx.net wrote:
Hi all,
I want to spread an algorithm over several images. For that I need to
transfer some objects.
For other purposes I already read and write objects
via JSON files. To
change this to UDP is straightforward. But the conversion
to JSON and
back takes considerable time, biggest chunk of data is an array
with floats.
Is there a fast and easy way to transfer objects binary between
images?
I can transfer only the array of floats (maybe a float array) if that
saves time.
Thanks,
Herbert
On Fri, Oct 20, 2017 at 06:33:28PM +0000, Kjell Godo wrote:
What about Fuel?
I thought I had already suggested that, but I guess it was in a different discussion:
http://lists.squeakfoundation.org/pipermail/squeak-dev/2017-October/date.htm...
Yes, Fuel is a good choice for serializing between images. ReferenceStream works well, but some things do not work with non-positionable streams, whereas Fuel seems to do a good job with any kind of stream (such as streaming on an OS pipe between two images).
The Magma serializer is another candidate.
Dave
Thanks all,
first I'll try Edgar's way. Nothing new to load/learn and I used ReferenceStream earlier.
Cheers,
Herbert
Am 20.10.2017 um 12:18 schrieb Herbert König:
Hi all,
I want to spread an algorithm over several images. For that I need to transfer some objects.
For other purposes I already read and write objects via JSON files. To change this to UDP is straightforward. But the conversion to JSON and back takes considerable time, biggest chunk of data is an array with floats.
Is there a fast and easy way to transfer objects binary between images? I can transfer only the array of floats (maybe a float array) if that saves time.
Thanks,
Herbert
And noteworthy in addition is that in case you do not have cycles in your net of objects a DataStream is just fine.
http://wiki.squeak.org/squeak/4310
--Hannes
On 10/21/17, Herbert König herbertkoenig@gmx.net wrote:
Thanks all,
first I'll try Edgar's way. Nothing new to load/learn and I used ReferenceStream earlier.
Cheers,
Herbert
Am 20.10.2017 um 12:18 schrieb Herbert König:
Hi all,
I want to spread an algorithm over several images. For that I need to transfer some objects.
For other purposes I already read and write objects via JSON files. To change this to UDP is straightforward. But the conversion to JSON and back takes considerable time, biggest chunk of data is an array with floats.
Is there a fast and easy way to transfer objects binary between images? I can transfer only the array of floats (maybe a float array) if that saves time.
Thanks,
Herbert
Hi Hannes,
Am Sat, 21 Oct 2017 15:05:19 +0200 schrieb "H. Hirzel" hannes.hirzel@gmail.com:
And noteworthy in addition is that in case you do not have cycles in your net of objects a DataStream is just fine.
Which I don't have so maybe this is even faster because it doesn't employ any logic to deal with cycles.
Thanks,
Herbert
--Hannes
On 10/21/17, Herbert König herbertkoenig@gmx.net wrote:
Thanks all,
first I'll try Edgar's way. Nothing new to load/learn and I used ReferenceStream earlier.
Cheers,
Herbert
Am 20.10.2017 um 12:18 schrieb Herbert König:
Hi all,
I want to spread an algorithm over several images. For that I need to transfer some objects.
For other purposes I already read and write objects via JSON files. To change this to UDP is straightforward. But the conversion to JSON and back takes considerable time, biggest chunk of data is an array with floats.
Is there a fast and easy way to transfer objects binary between images? I can transfer only the array of floats (maybe a float array) if that saves time.
Thanks,
Herbert
There reason I suggested ASN1 is that it is crosee-platofrm, if that is your requirement. I believe it is effectively binary, for your needs of lots of Floats and I just added support for them too. If you need that...
On 10/21/2017 11:38 AM, Herbert König wrote:
Hi Hannes,
Am Sat, 21 Oct 2017 15:05:19 +0200 schrieb "H. Hirzel" hannes.hirzel@gmail.com:
And noteworthy in addition is that in case you do not have cycles in your net of objects a DataStream is just fine.
Which I don't have so maybe this is even faster because it doesn't employ any logic to deal with cycles.
Thanks,
Herbert
--Hannes
On 10/21/17, Herbert König herbertkoenig@gmx.net wrote:
Thanks all,
first I'll try Edgar's way. Nothing new to load/learn and I used ReferenceStream earlier.
Cheers,
Herbert
Am 20.10.2017 um 12:18 schrieb Herbert König:
Hi all,
I want to spread an algorithm over several images. For that I need to transfer some objects.
For other purposes I already read and write objects via JSON files. To change this to UDP is straightforward. But the conversion to JSON and back takes considerable time, biggest chunk of data is an array with floats.
Is there a fast and easy way to transfer objects binary between images? I can transfer only the array of floats (maybe a float array) if that saves time.
Thanks,
Herbert
Is JSON the preferred way to transfer data between different Smalltalk platforms? like Dolphin S8 visualWorks GnuSmalltalk Smalltalk/X Amber etc? i would like a way for all the different things to talk to each other why should they all be seperate?
I am assuming JSON would be best for going between languages like Smalltalk - Lisp ( it is the easiest to get access to everything has a Package for it ).
Are there any Packages that are designed to link different languages together? like Smalltalk - Racket - CommonLisp - SWIProlog etc? i am assuming that sockets + JSON are the easiest quickest way or DCom or what was that Linux d-Bus is d-Bus also on Windows?easy? is COM also on Linux?does anyone use it?hard to use? Dolphin and VisialWork do COM is anyone doing this sort of thing?who would just know?
On Sat, Oct 21, 2017 at 04:33:21PM +0000, Kjell Godo wrote:
Is JSON the preferred way to transfer data between different Smalltalk platforms? like Dolphin S8 visualWorks GnuSmalltalk Smalltalk/X Amber etc? i would like a way for all the different things to talk to each other why should they all be seperate?
I am assuming JSON would be best for going between languages like Smalltalk - Lisp ( it is the easiest to get access to everything has a Package for it ).
Are there any Packages that are designed to link different languages together? like Smalltalk - Racket - CommonLisp - SWIProlog etc? i am assuming that sockets + JSON are the easiest quickest way or DCom or what was that Linux d-Bus is d-Bus also on Windows?easy? is COM also on Linux?does anyone use it?hard to use? Dolphin and VisialWork do COM is anyone doing this sort of thing?who would just know?
There is a SOAP implementation by Masashi Umezawa:
http://wiki.squeak.org/squeak/1399
You can find it in the SqueakMap package loader (SoapCore, SoapCoreClient, SoapCoreServer, SoapOpera).
I believe that the home page is here:
http://www.mars.dti.ne.jp/~umejava/smalltalk/soapOpera/soapCore.html
I do not know if it still works in current Squeak, or if the package is still being maintained. But I know that it used to work, so it might be worth a look.
Dave
Hi,
I used JSON with the addition of carrying the class (JSON being Javascript born has no concept of classes) because it requires only very little work after loading the JSON package. (And because I've been doing some web application where I needed JSON anyway.)
Each class needs on the class side:
constructFromJson: j ^ self new constructFromJson: j
and on the instance side:
jsonWriteOn: aStream Json renderInstanceVariables: (self class allInstVarNames collect: [:ea| ea asSymbol]) of: self on: aStream
and:
constructFromJson: j j keysAndValuesDo: [:key :value| self instVarNamed: key put: value]
Simple, stupid and good enough for several of my applications.
This should work across dialects if the destination image has the required classes in the same shape. The @Classname designator is a Smalltalk specific extension to JSON. I think it is not good enough to transfer object trees with cyclic dependencies. Other languages with a different ideas of how a class is constructed it might need more work.
JSON excerpt:
[@IndividualGenom{"halfNum":77,"fitness":2.364011719201257,"fitnesses":null,"geneValues":[-0.7875619210225705,-1.1191086825115324,2.2987547782235476,-2.1996847933965746,3.3254889274350594,-1.7481778033261461 ......
Cheers,
Herbert
Am 21.10.2017 um 18:33 schrieb Kjell Godo:
Is JSON the preferred way to transfer data between different Smalltalk platforms? like Dolphin S8 visualWorks GnuSmalltalk Smalltalk/X Amber etc? i would like a way for all the different things to talk to each other why should they all be seperate?
I am assuming JSON would be best for going between languages like Smalltalk - Lisp ( it is the easiest to get access to everything has a Package for it ).
Are there any Packages that are designed to link different languages together? like Smalltalk - Racket - CommonLisp - SWIProlog etc? i am assuming that sockets + JSON are the easiest quickest way or DCom or what was that Linux d-Bus is d-Bus also on Windows?easy? is COM also on Linux?does anyone use it?hard to use? Dolphin and VisialWork do COM is anyone doing this sort of thing?who would just know?
Hi,
DataStream it is. On an Array of objects
JSON write avg. 6 s read avg. 1s for 1500 KB
ReferenceStream write avg. 1.9 s read avg. 0.4 s for 689 KB
DataStream write avg. 1.55 s read avg. 0.13s for 698 KB
Cheers,
Herbert
Am 21.10.2017 um 17:38 schrieb Herbert König:
Hi Hannes,
Am Sat, 21 Oct 2017 15:05:19 +0200 schrieb "H. Hirzel" hannes.hirzel@gmail.com:
And noteworthy in addition is that in case you do not have cycles in your net of objects a DataStream is just fine.
Which I don't have so maybe this is even faster because it doesn't employ any logic to deal with cycles.
Thanks,
Herbert
--Hannes
On 10/21/17, Herbert König herbertkoenig@gmx.net wrote:
Thanks all,
first I'll try Edgar's way. Nothing new to load/learn and I used ReferenceStream earlier.
Cheers,
Herbert
Am 20.10.2017 um 12:18 schrieb Herbert König:
Hi all,
I want to spread an algorithm over several images. For that I need to transfer some objects.
For other purposes I already read and write objects via JSON files. To change this to UDP is straightforward. But the conversion to JSON and back takes considerable time, biggest chunk of data is an array with floats.
Is there a fast and easy way to transfer objects binary between images? I can transfer only the array of floats (maybe a float array) if that saves time.
Thanks,
Herbert
By the way, this would be a great library to port for serialization and interfacing to other programming languages: https://avro.apache.org/docs/current/index.html
It gets quite a bit of praise in the book Designing Data-Intensive Applications (https://dataintensive.net) which I can highly recommend.
Cheers, Bernhard
Am 22.10.2017 um 17:11 schrieb Herbert König herbertkoenig@gmx.net:
Hi,
DataStream it is. On an Array of objects
JSON write avg. 6 s read avg. 1s for 1500 KB
ReferenceStream write avg. 1.9 s read avg. 0.4 s for 689 KB
DataStream write avg. 1.55 s read avg. 0.13s for 698 KB
Cheers,
Herbert
Am 21.10.2017 um 17:38 schrieb Herbert König:
Hi Hannes,
Am Sat, 21 Oct 2017 15:05:19 +0200 schrieb "H. Hirzel" hannes.hirzel@gmail.com:
And noteworthy in addition is that in case you do not have cycles in your net of objects a DataStream is just fine.
Which I don't have so maybe this is even faster because it doesn't employ any logic to deal with cycles.
Thanks,
Herbert
--Hannes
On 10/21/17, Herbert König herbertkoenig@gmx.net wrote:
Thanks all,
first I'll try Edgar's way. Nothing new to load/learn and I used ReferenceStream earlier.
Cheers,
Herbert
Am 20.10.2017 um 12:18 schrieb Herbert König:
Hi all,
I want to spread an algorithm over several images. For that I need to transfer some objects.
For other purposes I already read and write objects via JSON files. To change this to UDP is straightforward. But the conversion to JSON and back takes considerable time, biggest chunk of data is an array with floats.
Is there a fast and easy way to transfer objects binary between images? I can transfer only the array of floats (maybe a float array) if that saves time.
Thanks,
Herbert
squeak-dev@lists.squeakfoundation.org