hi all,
Last month as an introduction to Squeak I did some performance testing for the creation of morphs, and a comparison between derivatives of Squeak. An advance copy of one of the graphs was posted previously, but I have since updated the graphs and discussion based on feedback at that time. I was going to tweak them some more but have been distracted by other priorities, and so I thought I would just post them now. I hope they are of some interest. I'll look at updating the graphs later, unless someone beats me too it in the comments.
http://blog.openinworld.com/2011/03/morphic-performance/ http://blog.openinworld.com/2011/03/morphic-flavour-performance/ http://blog.openinworld.com/2011/04/performance-testing-spreadsheet/
Its my first go at a blog (I hear all the cool kids are doin' it) - so any and all feedback on style and content is welcome.
Cheers, Ben
On 4/28/11 8:52 AM, Ben Coman wrote:
http://blog.openinworld.com/2011/03/morphic-performance/ http://blog.openinworld.com/2011/03/morphic-flavour-performance/ http://blog.openinworld.com/2011/04/performance-testing-spreadsheet/
Hi Ben.. nice posts!
I'll add you to planet.squeak.org if that's ok, the whole blog unless you prefer to tag your squeak-related posts.
On 28.04.2011, at 17:52, Ben Coman wrote:
hi all,
Last month as an introduction to Squeak I did some performance testing for the creation of morphs, and a comparison between derivatives of Squeak. An advance copy of one of the graphs was posted previously, but I have since updated the graphs and discussion based on feedback at that time. I was going to tweak them some more but have been distracted by other priorities, and so I thought I would just post them now. I hope they are of some interest. I'll look at updating the graphs later, unless someone beats me too it in the comments.
http://blog.openinworld.com/2011/03/morphic-performance/ http://blog.openinworld.com/2011/03/morphic-flavour-performance/ http://blog.openinworld.com/2011/04/performance-testing-spreadsheet/
Its my first go at a blog (I hear all the cool kids are doin' it) - so any and all feedback on style and content is welcome.
Cheers, Ben
Nice work. However, the VM (Interpreter or Cog) is independent of the image (Squeak / Cuis / Pharo). To get comparable results you should run either image on both VMs (or, since you just care about performance, use Cog for all three).
- Bert -
On Thu, 28 Apr 2011, Bert Freudenberg wrote:
On 28.04.2011, at 17:52, Ben Coman wrote:
hi all,
Last month as an introduction to Squeak I did some performance testing for the creation of morphs, and a comparison between derivatives of Squeak. An advance copy of one of the graphs was posted previously, but I have since updated the graphs and discussion based on feedback at that time. I was going to tweak them some more but have been distracted by other priorities, and so I thought I would just post them now. I hope they are of some interest. I'll look at updating the graphs later, unless someone beats me too it in the comments.
http://blog.openinworld.com/2011/03/morphic-performance/ http://blog.openinworld.com/2011/03/morphic-flavour-performance/ http://blog.openinworld.com/2011/04/performance-testing-spreadsheet/
Its my first go at a blog (I hear all the cool kids are doin' it) - so any and all feedback on style and content is welcome.
Cheers, Ben
Nice work. However, the VM (Interpreter or Cog) is independent of the image (Squeak / Cuis / Pharo). To get comparable results you should run either image on both VMs (or, since you just care about performance, use Cog for all three).
Since this isn't the first time people do benchmarks, but don't download VMs independently, therefore I think we have to package VMs with the releases in the future. IIRC someone even concluded that Pharo is faster than Squeak, but he just used what he got from the website (Cog for Pharo and the interpreter vm for Squeak).
Levente
- Bert -
I agree. Does this imply that two all-in-one packages are needed, a Cog and an non-cog?
--Hannes
On 4/28/11, Levente Uzonyi leves@elte.hu wrote:
On Thu, 28 Apr 2011, Bert Freudenberg wrote:
On 28.04.2011, at 17:52, Ben Coman wrote:
hi all,
Last month as an introduction to Squeak I did some performance testing for the creation of morphs, and a comparison between derivatives of Squeak. An advance copy of one of the graphs was posted previously, but I have since updated the graphs and discussion based on feedback at that time. I was going to tweak them some more but have been distracted by other priorities, and so I thought I would just post them now. I hope they are of some interest. I'll look at updating the graphs later, unless someone beats me too it in the comments.
http://blog.openinworld.com/2011/03/morphic-performance/ http://blog.openinworld.com/2011/03/morphic-flavour-performance/ http://blog.openinworld.com/2011/04/performance-testing-spreadsheet/
Its my first go at a blog (I hear all the cool kids are doin' it) - so any and all feedback on style and content is welcome.
Cheers, Ben
Nice work. However, the VM (Interpreter or Cog) is independent of the image (Squeak / Cuis / Pharo). To get comparable results you should run either image on both VMs (or, since you just care about performance, use Cog for all three).
Since this isn't the first time people do benchmarks, but don't download VMs independently, therefore I think we have to package VMs with the releases in the future. IIRC someone even concluded that Pharo is faster than Squeak, but he just used what he got from the website (Cog for Pharo and the interpreter vm for Squeak).
Levente
- Bert -
Hannes Hirzel wrote:
I agree. Does this imply that two all-in-one packages are needed, a Cog and an non-cog?
--Hannes
On 4/28/11, Levente Uzonyi leves@elte.hu wrote:
Since this isn't the first time people do benchmarks, but don't download VMs independently, therefore I think we have to package VMs with the releases in the future. IIRC someone even concluded that Pharo is faster than Squeak, but he just used what he got from the website (Cog for Pharo and the interpreter vm for Squeak).
Levente
Is the difference between a Cog and non-Cog VM just the a different exe (for windows)? How compatible are the images for Cog and non-Cog? Could two executables be included in a single All-In-One package - ie named like SqueakClassic.exe & SqueakCog.exe ?
To have Squeak with Cog I replaced the old Squeak VM with a Cog one from http://www.mirandabanda.org/files/Cog/VM/ (MSWindows)
--Hannes
On 4/29/11, Ben Coman btc@openinworld.com wrote:
Hannes Hirzel wrote:
I agree. Does this imply that two all-in-one packages are needed, a Cog and an non-cog?
--Hannes
On 4/28/11, Levente Uzonyi leves@elte.hu wrote:
Since this isn't the first time people do benchmarks, but don't download VMs independently, therefore I think we have to package VMs with the releases in the future. IIRC someone even concluded that Pharo is faster than Squeak, but he just used what he got from the website (Cog for Pharo and the interpreter vm for Squeak).
Levente
Is the difference between a Cog and non-Cog VM just the a different exe (for windows)? How compatible are the images for Cog and non-Cog? Could two executables be included in a single All-In-One package - ie named like SqueakClassic.exe & SqueakCog.exe ?
Yes, unzip.
You do not need to rename the file. croquet.exe. Just double click and then it looks for the image file and launches it.
But I think you cannot go back to squeak.exe after you have started and saved the image with croquet.exe. (Others please correct me)
--Hannes
On 4/29/11, Ben Coman btc@openinworld.com wrote:
Do you mean extract cogwin.zip into the Squeak-All-In-One folder and then rename croquet.exe to squeak.exe ?
Hannes Hirzel wrote:
To have Squeak with Cog I replaced the old Squeak VM with a Cog one from http://www.mirandabanda.org/files/Cog/VM/ (MSWindows)
--Hannes
On 4/29/11, Ben Coman btc@openinworld.com wrote:
Hannes Hirzel wrote:
I agree. Does this imply that two all-in-one packages are needed, a Cog and an non-cog?
--Hannes
On 4/28/11, Levente Uzonyi leves@elte.hu wrote:
Since this isn't the first time people do benchmarks, but don't download VMs independently, therefore I think we have to package VMs with the releases in the future. IIRC someone even concluded that Pharo is faster than Squeak, but he just used what he got from the website (Cog for Pharo and the interpreter vm for Squeak).
Levente
Is the difference between a Cog and non-Cog VM just the a different exe (for windows)? How compatible are the images for Cog and non-Cog? Could two executables be included in a single All-In-One package - ie named like SqueakClassic.exe & SqueakCog.exe ?
On Fri, 29 Apr 2011, Hannes Hirzel wrote:
Yes, unzip.
You do not need to rename the file. croquet.exe. Just double click and then it looks for the image file and launches it.
But I think you cannot go back to squeak.exe after you have started and saved the image with croquet.exe. (Others please correct me)
Squeak images saved by the CogVM will have a new image format. The current release (4.1.1) of SqueakVM for windows can't read such images. I built a VM which can be used to convert your images back to the old formatl. It's available here: http://leves.web.elte.hu/squeak/SqueakVM-Win32-4.4.9-2358-non-official-bin.z...
Levente
--Hannes
On 4/29/11, Ben Coman btc@openinworld.com wrote:
Do you mean extract cogwin.zip into the Squeak-All-In-One folder and then rename croquet.exe to squeak.exe ?
Hannes Hirzel wrote:
To have Squeak with Cog I replaced the old Squeak VM with a Cog one from http://www.mirandabanda.org/files/Cog/VM/ (MSWindows)
--Hannes
On 4/29/11, Ben Coman btc@openinworld.com wrote:
Hannes Hirzel wrote:
I agree. Does this imply that two all-in-one packages are needed, a Cog and an non-cog?
--Hannes
On 4/28/11, Levente Uzonyi leves@elte.hu wrote:
Since this isn't the first time people do benchmarks, but don't download VMs independently, therefore I think we have to package VMs with the releases in the future. IIRC someone even concluded that Pharo is faster than Squeak, but he just used what he got from the website (Cog for Pharo and the interpreter vm for Squeak).
Levente
Is the difference between a Cog and non-Cog VM just the a different exe (for windows)? How compatible are the images for Cog and non-Cog? Could two executables be included in a single All-In-One package - ie named like SqueakClassic.exe & SqueakCog.exe ?
On Fri, Apr 29, 2011 at 8:19 AM, Levente Uzonyi leves@elte.hu wrote:
On Fri, 29 Apr 2011, Hannes Hirzel wrote:
Yes, unzip.
You do not need to rename the file. croquet.exe. Just double click and then it looks for the image file and launches it.
But I think you cannot go back to squeak.exe after you have started and saved the image with croquet.exe. (Others please correct me)
Squeak images saved by the CogVM will have a new image format. The current release (4.1.1) of SqueakVM for windows can't read such images. I built a VM which can be used to convert your images back to the old formatl. It's available here: http://leves.web.elte.hu/squeak/SqueakVM-Win32-4.4.9-2358-non-official-bin.z...
Have these changes been folded back into VMMaker trunk?
Levente
--Hannes
On 4/29/11, Ben Coman btc@openinworld.com wrote:
Do you mean extract cogwin.zip into the Squeak-All-In-One folder and then rename croquet.exe to squeak.exe ?
Hannes Hirzel wrote:
To have Squeak with Cog I replaced the old Squeak VM with a Cog one from http://www.mirandabanda.org/files/Cog/VM/ (MSWindows)
--Hannes
On 4/29/11, Ben Coman btc@openinworld.com wrote:
Hannes Hirzel wrote:
I agree. Does this imply that two all-in-one packages are needed, a Cog and an non-cog?
--Hannes
On 4/28/11, Levente Uzonyi leves@elte.hu wrote:
> Since this isn't the first time people do benchmarks, but don't > download > VMs independently, therefore I think we have to package VMs with the > releases in the future. IIRC someone even concluded that Pharo is > faster > than Squeak, but he just used what he got from the website (Cog for > Pharo > and the interpreter vm for Squeak). > > > Levente > >
Is the difference between a Cog and non-Cog VM just the a different exe (for windows)? How compatible are the images for Cog and non-Cog? Could two executables be included in a single All-In-One package - ie named like SqueakClassic.exe & SqueakCog.exe ?
On Sat, Apr 30, 2011 at 12:48:31AM +0200, Levente Uzonyi wrote:
On Fri, 29 Apr 2011, Eliot Miranda wrote:
Have these changes been folded back into VMMaker trunk?
There are no changes, this is a VM built from the Trunk.
Levente
Support for Cog images (float word ordering) was added in these two VMMaker updates, and is present in any recent VM build (but the Windows VM has not been updated):
Name: VMMaker-dtl.189 Author: dtl Time: 6 October 2010, 12:17:37 pm UUID: 29192b6e-491d-4bc6-a209-5ba1c8625da0 Ancestors: VMMaker-dtl.188
VMMaker 4.3.3
Allow a standard interpreter VM to load and run an image that was saved from a Cog or StackInterpreter VM. On image load, the storage format of Float objects is returned to normalized word ordering (different from platform word order for little endian platforms) and unrecognized header flags are cleared. The image will be saved in standard interpreter format (image format 6504 for a 32-bit image), which may subsequently be loaded by a Cog or StackInterpreter VM.
Works for 32-bit Cog images (image format 6505) on both 32-bit and 64-bit host (compile -m32 or -m64). Support for 64-bit image formats is not yet implemented, see comment in #normalizeFloatOrderingInImage.
Name: VMMaker-dtl.190 Author: dtl Time: 6 October 2010, 8:39:07 am UUID: 4798e721-0a2b-4dea-b924-7492012a75f1 Ancestors: VMMaker-dtl.189
VMMaker 4.3.4
Fix #normalizeFloatOrderingInImage to work on 64-bit Squeak images. Float data storage in a 64-bit image is the same as for a 32 bit image, with the two 32-bit halves stored in the next two 32-bit words beginning at accessibleObjectAfter: floatOop (in the case of 64-bit memory, the two 32-bit halves occupy one object memory word, while a 32-bit object memory uses two). Therefore use the same logic in #normalizeFloatOrderingInImage for a 64-bit image.
On Sat, 30 Apr 2011, Levente Uzonyi wrote:
On Fri, 29 Apr 2011, Eliot Miranda wrote:
Have these changes been folded back into VMMaker trunk?
There are no changes, this is a VM built from the Trunk.
Whoops, I forgot that I copied the source files for FilePlugin from Cog's branch, so this VM has stdio support.
Levente
Levente
On Fri, 29 Apr 2011, Ben Coman wrote:
Hannes Hirzel wrote:
I agree. Does this imply that two all-in-one packages are needed, a Cog and an non-cog?
--Hannes
On 4/28/11, Levente Uzonyi leves@elte.hu wrote:
Since this isn't the first time people do benchmarks, but don't download VMs independently, therefore I think we have to package VMs with the releases in the future. IIRC someone even concluded that Pharo is faster than Squeak, but he just used what he got from the website (Cog for Pharo and the interpreter vm for Squeak).
Levente
Is the difference between a Cog and non-Cog VM just the a different exe (for windows)?
Basically yes, some plugins are also different, but from the user's point of view it's just a different executable on all platforms.
How compatible are the images for Cog and non-Cog? Could two executables be
In case of released images, only Squeak 4.2 is Cog compatible, because Cog was released shortly after the release of Squeak 4.1. But applying a few changes to a Squeak 4.1 image (that's what Pharo developers did to Pharo 1.1 to release Pharo 1.1.1) will make it Cog compatible. This intermediate release seemed unnecessary for Squeak, because with a change of a preference and a single click (update) one could make her/his image Cog compatible.
included in a single All-In-One package - ie named like SqueakClassic.exe & SqueakCog.exe ?
Yes, it's possible, Cobalt is already packaged this way.
Levente
On Fri, 29 Apr 2011, Ben Coman wrote:
Bert Freudenberg wrote:
On 28.04.2011, at 17:52, Ben Coman wrote:
http://blog.openinworld.com/2011/03/morphic-performance/ http://blog.openinworld.com/2011/03/morphic-flavour-performance/ http://blog.openinworld.com/2011/04/performance-testing-spreadsheet/
Nice work. However, the VM (Interpreter or Cog) is independent of the image (Squeak / Cuis / Pharo). To get comparable results you should run either image on both VMs (or, since you just care about performance, use Cog for all three).
- Bert -
Thanks Bert. I only used the All-In-One-Click packages and am not familiar with how mix & match VMs and images. It is not something I have looked into yet and a pointer to to some information on that would be useful. Also, Cog seemed to be only newly arrived for Squeak and I didn't want to hit complications. I was hoping it was fair to compare Squeak-Plain with Pharo-Plain and them to Pharo-Cog.
To a comment on the morphic-flavour-performance post I have attached a graph of Levente's results for CogVM with recent images.
Thanks. I was too lazy to convert from the output to the format used by the spreadsheet.
Levente
On Fri, 29 Apr 2011, Ben Coman wrote:
Levente Uzonyi wrote: On Fri, 29 Apr 2011, Ben Coman wrote:
Bert Freudenberg wrote: On 28.04.2011, at 17:52, Ben Coman wrote: http://blog.openinworld.com/2011/03/morphic-performance/ http://blog.openinworld.com/2011/03/morphic-flavour-performance/ http://blog.openinworld.com/2011/04/performance-testing-spreadsheet/ Nice work. However, the VM (Interpreter or Cog) is independent of the image (Squeak / Cuis / Pharo). To get comparable results you should run either image on both VMs (or, since you just care about performance, use Cog for all three). - Bert - Thanks Bert. I only used the All-In-One-Click packages and am not familiar with how mix & match VMs and images. It is not something I have looked into yet and a pointer to to some information on that would be useful. Also, Cog seemed to be only newly arrived for Squeak and I didn't want to hit complications. I was hoping it was fair to compare Squeak-Plain with Pharo-Plain and them to Pharo-Cog. To a comment on the morphic-flavour-performance post I have attached a graph of Levente's results for CogVM with recent images. Thanks. I was too lazy to convert from the output to the format used by the spreadsheet. LeventeI notice a big difference in your results between Pharo and Squeak that may be related to an enlightening comment left by Henrik on the morphic-flavour-performance article
The intersects check used for culling really dislikes floating point bounding box though, as it has to convert them to Fractions before comparing
with integers.
Would you feel inclined to rerun your tests using "rounded" ?
I've got no time for that right now, maybe later this evening.
OR... if you could direct me to instructions on how to update my images from the one-clicks to the versions you used, then I will re-run on my hardware to get a direct comparison to my previous results. I've been googling terms like "squeak updating to trunk" but there is a lot of fluff that isn't what I need.
Using another VM is pretty easy. Just download this http://www.mirandabanda.org/files/Cog/VM/VM.r2382/cogwin.zip , unzip it to any folder, then double click on Croquet.exe, select the image you want to open, and that's it. If the sources file for your image is not on the path, then the image will be unhappy about it, but that shouldn't be a problem. You can always put the sources file to the same folder where Croquet.exe is to fix this though.
Note that the Pharo and Cuis versions I used have changed their Transcript implementations, so you'll have to adjust your code accordingly, otherwise you won't be able to copy the results from Transcript.
Levente
On Fri, 29 Apr 2011, Levente Uzonyi wrote:
I've got no time for that right now, maybe later this evening.
Done:
Morphic Performance Test Code v3.0 - Squeak 4.3-11361 amount, createTime, hideTime, showTime, deleteTime 1, 2, 19, 21, 68 10, 1, 22, 20, 68 100, 57, 51, 56, 52 200, 68, 50, 63, 52 300, 77, 55, 75, 55 400, 90, 54, 81, 55 500, 98, 53, 87, 52 600, 98, 53, 97, 54 700, 112, 55, 100, 57 800, 118, 55, 109, 52 900, 131, 57, 116, 53 1000, 134, 58, 125, 55 2000, 214, 66, 197, 51 3000, 293, 70, 273, 57 4000, 373, 73, 358, 59 5000, 531, 80, 420, 61 6000, 540, 84, 500, 61 7000, 692, 183, 561, 58 8000, 704, 97, 685, 63 9000, 838, 102, 712, 65 10000, 1005, 203, 787, 66 30000, 2774, 209, 2994, 94 100000, 10310, 1093, 10142, 415
Morphic Performance Test Code v3.0 - Pharo 1.3-13176 amount, createTime, hideTime, showTime, deleteTime 1, 3, 18, 20, 33 10, 9, 21, 21, 31 100, 30, 25, 25, 23 200, 34, 12, 42, 22 300, 44, 22, 42, 22 400, 55, 26, 50, 10 500, 72, 27, 56, 24 600, 76, 15, 65, 21 700, 78, 25, 72, 23 800, 89, 25, 82, 10 900, 107, 25, 88, 23 1000, 102, 28, 94, 13 2000, 196, 26, 169, 13 3000, 268, 27, 238, 23 4000, 362, 30, 321, 21 5000, 450, 34, 391, 23 6000, 530, 37, 525, 24 7000, 611, 39, 536, 23 8000, 695, 45, 616, 27 9000, 794, 47, 689, 27 10000, 880, 50, 768, 25 30000, 2739, 119, 2296, 48 100000, 9681, 393, 8895, 166
Morphic Performance Test Code v3.0 - Cuis 3.2-0914 amount, createTime, hideTime, showTime, deleteTime 1, 1, 21, 21, 49 10, 2, 22, 26, 51 100, 40, 30, 41, 31 200, 59, 32, 55, 30 300, 69, 31, 73, 31 400, 82, 33, 84, 30 500, 96, 33, 97, 30 600, 118, 33, 116, 30 700, 138, 32, 138, 34 800, 152, 31, 162, 30 900, 180, 32, 184, 31 1000, 211, 33, 222, 31 2000, 354, 37, 365, 31 3000, 475, 41, 483, 32 4000, 613, 45, 628, 32 5000, 740, 45, 735, 33 6000, 859, 52, 870, 34 7000, 966, 50, 992, 35 8000, 1090, 61, 1082, 37 9000, 1171, 61, 1161, 35 10000, 1288, 60, 1298, 36 30000, 3585, 129, 3466, 53 100000, 18824, 469, 11742, 115
Levente
Here's the script I used for these measurements:
rand := Random new. output := (String new: 100) writeStream. creating := [ | m | m := EllipseMorph new. ellipses add: m. m color: Color random. m position: (ActiveWorld extent * (rand next @ rand next)) rounded ]. deleting := [ World removeAllMorphsIn: ellipses. ]. hiding := [ ellipses do: [ :item | item hide ] ]. showing := [ ellipses do: [ :item | item show ] ].
morphs := World submorphs. World removeAllMorphs. World doOneCycle. Smalltalk garbageCollect. output nextPutAll: 'Morphic Performance Test Code v3.0 - ##FLAVOUR## '; cr; nextPutAll: 'amount, createTime, hideTime, showTime, deleteTime'; cr. #(1 10 100 200 300 400 500 600 700 800 900 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 30000 100000) do: [ :amount | ellipses := OrderedCollection new. createTime := [ amount timesRepeat: creating. World addAllMorphs: ellipses. ActiveWorld doOneCycle ] timeToRun. hideTime := [ hiding value. ActiveWorld doOneCycle. ] timeToRun . showTime := [ showing value. ActiveWorld doOneCycle. ] timeToRun. deleteTime := [ deleting value. ActiveWorld doOneCycle. ] timeToRun. output print: amount; nextPutAll: ', '; print: createTime; nextPutAll: ', '; print: hideTime; nextPutAll: ', '; print: showTime; nextPutAll: ', '; print: deleteTime; cr ]. World addAllMorphs: morphs. output contents explore
Levente
On Sat, 30 Apr 2011, Levente Uzonyi wrote:
On Fri, 29 Apr 2011, Levente Uzonyi wrote:
I've got no time for that right now, maybe later this evening.
Done:
Morphic Performance Test Code v3.0 - Squeak 4.3-11361 amount, createTime, hideTime, showTime, deleteTime 1, 2, 19, 21, 68 10, 1, 22, 20, 68 100, 57, 51, 56, 52 200, 68, 50, 63, 52 300, 77, 55, 75, 55 400, 90, 54, 81, 55 500, 98, 53, 87, 52 600, 98, 53, 97, 54 700, 112, 55, 100, 57 800, 118, 55, 109, 52 900, 131, 57, 116, 53 1000, 134, 58, 125, 55 2000, 214, 66, 197, 51 3000, 293, 70, 273, 57 4000, 373, 73, 358, 59 5000, 531, 80, 420, 61 6000, 540, 84, 500, 61 7000, 692, 183, 561, 58 8000, 704, 97, 685, 63 9000, 838, 102, 712, 65 10000, 1005, 203, 787, 66 30000, 2774, 209, 2994, 94 100000, 10310, 1093, 10142, 415
Morphic Performance Test Code v3.0 - Pharo 1.3-13176 amount, createTime, hideTime, showTime, deleteTime 1, 3, 18, 20, 33 10, 9, 21, 21, 31 100, 30, 25, 25, 23 200, 34, 12, 42, 22 300, 44, 22, 42, 22 400, 55, 26, 50, 10 500, 72, 27, 56, 24 600, 76, 15, 65, 21 700, 78, 25, 72, 23 800, 89, 25, 82, 10 900, 107, 25, 88, 23 1000, 102, 28, 94, 13 2000, 196, 26, 169, 13 3000, 268, 27, 238, 23 4000, 362, 30, 321, 21 5000, 450, 34, 391, 23 6000, 530, 37, 525, 24 7000, 611, 39, 536, 23 8000, 695, 45, 616, 27 9000, 794, 47, 689, 27 10000, 880, 50, 768, 25 30000, 2739, 119, 2296, 48 100000, 9681, 393, 8895, 166
Morphic Performance Test Code v3.0 - Cuis 3.2-0914 amount, createTime, hideTime, showTime, deleteTime 1, 1, 21, 21, 49 10, 2, 22, 26, 51 100, 40, 30, 41, 31 200, 59, 32, 55, 30 300, 69, 31, 73, 31 400, 82, 33, 84, 30 500, 96, 33, 97, 30 600, 118, 33, 116, 30 700, 138, 32, 138, 34 800, 152, 31, 162, 30 900, 180, 32, 184, 31 1000, 211, 33, 222, 31 2000, 354, 37, 365, 31 3000, 475, 41, 483, 32 4000, 613, 45, 628, 32 5000, 740, 45, 735, 33 6000, 859, 52, 870, 34 7000, 966, 50, 992, 35 8000, 1090, 61, 1082, 37 9000, 1171, 61, 1161, 35 10000, 1288, 60, 1298, 36 30000, 3585, 129, 3466, 53 100000, 18824, 469, 11742, 115
Levente
On Thu, 28 Apr 2011, Ben Coman wrote:
hi all,
Last month as an introduction to Squeak I did some performance testing for the creation of morphs, and a comparison between derivatives of Squeak. An advance copy of one of the graphs was posted previously, but I have since updated the graphs and discussion based on feedback at that time. I was going to tweak them some more but have been distracted by other priorities, and so I thought I would just post them now. I hope they are of some interest. I'll look at updating the graphs later, unless someone beats me too it in the comments.
http://blog.openinworld.com/2011/03/morphic-performance/ http://blog.openinworld.com/2011/03/morphic-flavour-performance/ http://blog.openinworld.com/2011/04/performance-testing-spreadsheet/
Its my first go at a blog (I hear all the cool kids are doin' it) - so any and all feedback on style and content is welcome.
Interesting benchmark. For better comparison I ran it on my pc using the same VM (Cog r2382) and recent images.
Morphic Performance Test Code v3.0 - Squeak 4.3-11353, CogVM r2382 amount, createTime, hideTime, showTime, deleteTime 1, 51, 3, 19, 101 10, 12, 16, 23, 64 100, 66, 53, 61, 54 200, 71, 58, 68, 55 300, 82, 58, 76, 54 400, 88, 56, 83, 59 500, 95, 56, 92, 55 600, 102, 59, 98, 55 700, 111, 58, 104, 56 800, 118, 57, 113, 57 900, 125, 62, 117, 55 1000, 138, 60, 133, 57 2000, 215, 67, 195, 57 3000, 314, 154, 265, 58 4000, 375, 76, 336, 62 5000, 454, 83, 443, 60 6000, 607, 82, 482, 59 7000, 621, 94, 702, 64 8000, 768, 99, 623, 62 9000, 847, 99, 719, 64 10000, 944, 105, 809, 65 30000, 2713, 307, 2740, 98 100000, 9475, 1061, 10158, 415
Morphic Performance Test Code v3.0 - Pharo 1.3-13171, CogVM r2382 amount, createTime, hideTime, showTime, deleteTime 1, 17, 4, 20, 40 10, 14, 10, 21, 39 100, 50, 41, 44, 23 200, 57, 42, 57, 22 300, 81, 57, 80, 21 400, 93, 68, 97, 20 500, 123, 85, 120, 22 600, 137, 90, 143, 23 700, 159, 111, 160, 22 800, 170, 115, 175, 22 900, 200, 136, 200, 23 1000, 215, 142, 219, 22 2000, 411, 275, 425, 22 3000, 605, 396, 632, 26 4000, 808, 524, 830, 24 5000, 1076, 664, 1034, 25 6000, 1191, 776, 1233, 26 7000, 1386, 911, 1446, 26 8000, 1587, 1044, 1651, 29 9000, 1787, 1159, 1893, 36 10000, 2140, 1334, 2048, 29 30000, 6037, 3906, 6335, 56 100000, 21994, 13990, 22803, 167
Morphic Performance Test Code v3.0 - Cuis 3.2-0914, CogVM r2382 amount, createTime, hideTime, showTime, deleteTime 1, 49, 22, 20, 55 10, 4, 21, 26, 57 100, 11, 76, 46, 36 200, 56, 36, 54, 36 300, 65, 37, 61, 35 400, 74, 37, 75, 34 500, 94, 35, 94, 35 600, 98, 38, 106, 36 700, 112, 37, 124, 34 800, 143, 39, 151, 35 900, 160, 38, 172, 36 1000, 187, 41, 198, 37 2000, 286, 50, 314, 34 3000, 414, 46, 430, 36 4000, 523, 53, 547, 38 5000, 640, 59, 643, 38 6000, 733, 62, 745, 42 7000, 837, 66, 824, 40 8000, 926, 74, 921, 41 9000, 1010, 72, 1011, 42 10000, 1140, 83, 1143, 40 30000, 6495, 257, 3027, 58 100000, 14997, 700, 10333, 338
Levente
Cheers, Ben
Nice! :)
I think you should run several passes for each value of "amount" and visualize the average value for each "amount" to get more meaningful results. There are several peaks in your graphs that, I think, do not represent the average use case but the influence of another hidden variable (e.g., garbage collector) you want to block out in your results.
As far as I could see, you did not considered that in your benchmark code. Correct me, if I am wrong. :)
Marcel
-- View this message in context: http://forum.world.st/Morphic-Performance-Graphs-tp3481429p3483122.html Sent from the Squeak - Dev mailing list archive at Nabble.com.
Marcel Taeumel wrote:
Nice! :)
I think you should run several passes for each value of "amount" and visualize the average value for each "amount" to get more meaningful results. There are several peaks in your graphs that, I think, do not represent the average use case but the influence of another hidden variable (e.g., garbage collector) you want to block out in your results.
As far as I could see, you did not considered that in your benchmark code. Correct me, if I am wrong. :)
Marcel
-- View this message in context: http://forum.world.st/Morphic-Performance-Graphs-tp3481429p3483122.html Sent from the Squeak - Dev mailing list archive at Nabble.com.
You are correct that the results are only one-shot and not averaged across multiple runs. Early on I did implement multiple-runs but it was an added complication for the blog entry. Also I find the peaks enlightening without hiding the general rising trend.
What I did find really interesting is the way the peaks disappeared in Figure 2 of morphic-flavour-performance. This was still just one-shot results. It almost seems like "ActiveWorld doOneCycle" schedules a garbage collection after it completes, such that GC no longer occurs at random times.
On Fri, 29 Apr 2011, Marcel Taeumel wrote:
Nice! :)
I think you should run several passes for each value of "amount" and visualize the average value for each "amount" to get more meaningful results. There are several peaks in your graphs that, I think, do not represent the average use case but the influence of another hidden variable (e.g., garbage collector) you want to block out in your results.
As far as I could see, you did not considered that in your benchmark code. Correct me, if I am wrong. :)
GC is part of the execution. Ignoring it for benchmarks with high memory usage yields false results. A single GC before each run is useful though, but I guess it's effect is not that significant for this benchmark. Note that the cost of finalization is also different among the used images.
Levente
Marcel
-- View this message in context: http://forum.world.st/Morphic-Performance-Graphs-tp3481429p3483122.html Sent from the Squeak - Dev mailing list archive at Nabble.com.
It's not about ignoring GC, but to add the influence of GC correctly. Some runs need more time for GC, some don't - even for the same "amount" value. This means, that not GC itself is a variable to block out, but another hidden one that influences the time for GC. What you do not want, is to have "peaks" in your graph that happended by chance, because that doesn't mean anything for the general morphic performance.
Peaks which occur very often for several runs - not only a few times - would be VERY interesting though. :)
Marcel-- View this message in context: http://forum.world.st/Morphic-Performance-Graphs-tp3481429p3487398.html Sent from the Squeak - Dev mailing list archive at Nabble.com.
squeak-dev@lists.squeakfoundation.org
-
Ben Coman -
Bert Freudenberg -
David T. Lewis -
Eliot Miranda -
Hannes Hirzel -
Levente Uzonyi -
Marcel Taeumel -
Simon Michael