Do block closures work right in 3.9? 3.10?
I am struggling with local variables used across nested blocks (Magma #commit: within Seaside #callback: or Scriptaculous #triggerInPlaceEditor etc.) and am finding it quite frustrating. I had some #fixTemps but am totally guessing at when and why to use it.
Thanks - Sophie
2007/12/18, itsme213 itsme213@hotmail.com:
Do block closures work right in 3.9? 3.10?
No, block closures are broken in every Squeak. You have to load the NewCompiler and activate two compiler settings to get working block closures. You'll have to live with some other issues though.
Cheers Philippe
I am struggling with local variables used across nested blocks (Magma #commit: within Seaside #callback: or Scriptaculous #triggerInPlaceEditor etc.) and am finding it quite frustrating. I had some #fixTemps but am totally guessing at when and why to use it.
Thanks - Sophie
Would you mind sharing what details you know about the current state of closures and what the trade-offs are using the NewCompiler?
And, do you know of any energy focused on improving the existing alternatives?
On Dec 18, 2007 2:08 PM, Philippe Marschall philippe.marschall@gmail.com wrote:
2007/12/18, itsme213 itsme213@hotmail.com:
Do block closures work right in 3.9? 3.10?
No, block closures are broken in every Squeak. You have to load the NewCompiler and activate two compiler settings to get working block closures. You'll have to live with some other issues though.
Cheers Philippe
I am struggling with local variables used across nested blocks (Magma #commit: within Seaside #callback: or Scriptaculous #triggerInPlaceEditor etc.) and am finding it quite frustrating. I had some #fixTemps but am totally guessing at when and why to use it.
Thanks - Sophie
2007/12/19, Chris Muller asqueaker@gmail.com:
Would you mind sharing what details you know about the current state of closures and what the trade-offs are using the NewCompiler?
The compiler itself works and can be loaded into 3.9 without problems. Error messages are much improved and unlike your standard SmaCC error messages. Performance is lower. There is a lot of work going on in the decompiler and I don't know the exact state of this.
You should ask Marcus about what the current status exactly is. Or Klaus.
And, do you know of any energy focused on improving the existing alternatives?
What alternatives?
Cheers Philippe
On Dec 18, 2007 2:08 PM, Philippe Marschall philippe.marschall@gmail.com wrote:
2007/12/18, itsme213 itsme213@hotmail.com:
Do block closures work right in 3.9? 3.10?
No, block closures are broken in every Squeak. You have to load the NewCompiler and activate two compiler settings to get working block closures. You'll have to live with some other issues though.
Cheers Philippe
I am struggling with local variables used across nested blocks (Magma #commit: within Seaside #callback: or Scriptaculous #triggerInPlaceEditor etc.) and am finding it quite frustrating. I had some #fixTemps but am totally guessing at when and why to use it.
Thanks - Sophie
Philippe Marschall wrote:
The compiler itself works and can be loaded into 3.9 without problems.
Unfortunately, I can't confirm the "working without problems" part in 3.9. For testing the speed I was copying class Object and it barfed halfways through in #copyTwoLevel in a perfectly valid expression:
class isVariable ifTrue:[ index _ self basicSize. [index > 0] whileTrue:[ newObject basicAt: index put: (self basicAt: index) shallowCopy. index _ index - 1]"xxxxx Argument expected ->xxxxx"].
(note: you can just select the expression in a workspace and doIt if you have compileUseNewCompiler enabled; it'll complain right away) This is very strange and the error message makes little sense to me - I tried adding a period because perhaps it was expecting a message to the block but that doesn't do it. Has anyone tried to simply recompile the entirety of 3.9 to see if that works or not? I've been using 3.9-final-7067 for my test with the latest published Squeakmap releases for AST and NewCompiler. If there is a better version to use, please let me know.
[BTW, there are many other obscure errors that happen when I simply remove the offending code and continue to compile. Ultimately it appears to die somewhere unrelated at first glance but since this only happens when using compileUseNewCompiler I'm pretty sure it's related to some NewCompiler issues]
Error messages are much improved and unlike your standard SmaCC error messages. Performance is lower. There is a lot of work going on in the decompiler and I don't know the exact state of this.
Nice to hear about the error messages. I hadn't tried the compiler in a while but the above really shows that error messages really have greatly improved.
Concerning speed: When doing an [Object2 compileAll] timeToRun (Object2 being the partial copy of class Object that NewCompiler was able to hand me) I ended up with 1444 msecs for Compiler and 4020 msecs for NewCompiler. Unless I'm mistaken this is an improvement over the last version I tried which was roughly 4x slower (this is still 2.8x slower but it's a definitive improvement).
You should ask Marcus about what the current status exactly is. Or Klaus.
It would be a big step if it were possible to recompile Squeak successfully and perhaps even bytecode-identically. Being able to generate an identical version of the method would allow people to trivially verify the correct functioning of NewCompiler.
And, do you know of any energy focused on improving the existing alternatives?
What alternatives?
The compiler that's actually being used. The one that's 3x faster ;-) If NewCompiler worked properly I'd be actually quite tempted to just adopt the closure solution to the current compiler since it's smaller, faster, and well-established.
Cheers, - Andreas
2007/12/19, Andreas Raab andreas.raab@gmx.de:
Philippe Marschall wrote:
The compiler itself works and can be loaded into 3.9 without problems.
Unfortunately, I can't confirm the "working without problems" part in 3.9. For testing the speed I was copying class Object and it barfed halfways through in #copyTwoLevel in a perfectly valid expression:
class isVariable ifTrue:[ index _ self basicSize. [index > 0] whileTrue:[ newObject basicAt: index put: (self basicAt: index) shallowCopy. index _ index - 1]"xxxxx Argument expected ->xxxxx"].
(note: you can just select the expression in a workspace and doIt if you have compileUseNewCompiler enabled; it'll complain right away) This is very strange and the error message makes little sense to me - I tried adding a period because perhaps it was expecting a message to the block but that doesn't do it. Has anyone tried to simply recompile the entirety of 3.9 to see if that works or not? I've been using 3.9-final-7067 for my test with the latest published Squeakmap releases for AST and NewCompiler. If there is a better version to use, please let me know.
I'll have a look at it. It can be that the ones on SqueakSource are newer.
[BTW, there are many other obscure errors that happen when I simply remove the offending code and continue to compile. Ultimately it appears to die somewhere unrelated at first glance but since this only happens when using compileUseNewCompiler I'm pretty sure it's related to some NewCompiler issues]
Error messages are much improved and unlike your standard SmaCC error messages. Performance is lower. There is a lot of work going on in the decompiler and I don't know the exact state of this.
Nice to hear about the error messages. I hadn't tried the compiler in a while but the above really shows that error messages really have greatly improved.
Concerning speed: When doing an [Object2 compileAll] timeToRun (Object2 being the partial copy of class Object that NewCompiler was able to hand me) I ended up with 1444 msecs for Compiler and 4020 msecs for NewCompiler. Unless I'm mistaken this is an improvement over the last version I tried which was roughly 4x slower (this is still 2.8x slower but it's a definitive improvement).
I wasn't talking about compilation speed. What's really limiting for bigger projects is not the speed of the compiler but PackageInfo which makes Monticello slow. I was talking about execution speed.
You should ask Marcus about what the current status exactly is. Or Klaus.
It would be a big step if it were possible to recompile Squeak successfully
The NewCompiler images from Marcus do this since a long time ago. See the Recompiler class.
and perhaps even bytecode-identically.
Nah, NewCompiler in certain circumstances generates marginally more efficient bytecode.
Being able to generate an identical version of the method would allow people to trivially verify the correct functioning of NewCompiler.
And, do you know of any energy focused on improving the existing alternatives?
What alternatives?
The compiler that's actually being used. The one that's 3x faster ;-)
The one that's causing so much trouble in Seaside? The that gives you this cool C++ feeling bugs where you spend two hours trying to find out why your block arguments are all messed up and you think you went crazy? And then suddenly you add a lot of #fixTemps everywhere until it works?
Cheers Philippe
If NewCompiler worked properly I'd be actually quite tempted to just adopt the closure solution to the current compiler since it's smaller, faster, and well-established.
Cheers,
- Andreas
Philippe Marschall wrote:
Concerning speed: When doing an [Object2 compileAll] timeToRun (Object2 being the partial copy of class Object that NewCompiler was able to hand me) I ended up with 1444 msecs for Compiler and 4020 msecs for NewCompiler. Unless I'm mistaken this is an improvement over the last version I tried which was roughly 4x slower (this is still 2.8x slower but it's a definitive improvement).
I wasn't talking about compilation speed. What's really limiting for bigger projects is not the speed of the compiler but PackageInfo which makes Monticello slow. I was talking about execution speed.
I see. Has anyone measured the performance of the micro and macro benchmarks with the entire system compiled for using closures? I'm particularly interested in the macro benchmarks.
You should ask Marcus about what the current status exactly is. Or Klaus.
It would be a big step if it were possible to recompile Squeak successfully
The NewCompiler images from Marcus do this since a long time ago. See the Recompiler class.
Ah, good.
and perhaps even bytecode-identically.
Nah, NewCompiler in certain circumstances generates marginally more efficient bytecode.
This may actually be worthwhile "undoing" simply for being able to verify correctness.
And, do you know of any energy focused on improving the existing alternatives?
What alternatives?
The compiler that's actually being used. The one that's 3x faster ;-)
The one that's causing so much trouble in Seaside? The that gives you this cool C++ feeling bugs where you spend two hours trying to find out why your block arguments are all messed up and you think you went crazy? And then suddenly you add a lot of #fixTemps everywhere until it works?
Am I missing a smiley somewhere? You are describing properties that apply to NewCompiler just as well if you disable closure compilation and there is really no reason whatsoever why a working closure solution couldn't be adopted to the current compiler (which would mitigate the risk factors of adoption dramatically). Just like I was saying in the part you'd snipped away I'd be quite interested in adopting a working solution (which doesn't strike me as hard given the simplistic nature of the current compiler).
BTW, since you phrase it as such a big deal, is anyone using NewCompiler with Seaside? If it is really as problematic as you make it above I'd expect to see users of Seaside using it. Do they? If so, what's the practical experience?
Cheers, - Andreas
Hi
On Dec 19, 2007, at 9:12 AM, Andreas Raab wrote:
Philippe Marschall wrote:
Concerning speed: When doing an [Object2 compileAll] timeToRun (Object2 being the partial copy of class Object that NewCompiler was able to hand me) I ended up with 1444 msecs for Compiler and 4020 msecs for NewCompiler. Unless I'm mistaken this is an improvement over the last version I tried which was roughly 4x slower (this is still 2.8x slower but it's a definitive improvement).
I wasn't talking about compilation speed. What's really limiting for bigger projects is not the speed of the compiler but PackageInfo which makes Monticello slow. I was talking about execution speed.
I see. Has anyone measured the performance of the micro and macro benchmarks with the entire system compiled for using closures? I'm particularly interested in the macro benchmarks.
You should ask Marcus about what the current status exactly is. Or Klaus.
It would be a big step if it were possible to recompile Squeak successfully
The NewCompiler images from Marcus do this since a long time ago. See the Recompiler class.
Ah, good.
and perhaps even bytecode-identically.
Nah, NewCompiler in certain circumstances generates marginally more efficient bytecode.
This may actually be worthwhile "undoing" simply for being able to verify correctness.
What we do with Markus is recompiling the complete image with the newcompiler. The image remain stable so we are quite sure that the compiler give correct bytecode.
And, do you know of any energy focused on improving the existing alternatives?
What alternatives?
The compiler that's actually being used. The one that's 3x faster ;-)
The one that's causing so much trouble in Seaside? The that gives you this cool C++ feeling bugs where you spend two hours trying to find out why your block arguments are all messed up and you think you went crazy? And then suddenly you add a lot of #fixTemps everywhere until it works?
Am I missing a smiley somewhere? You are describing properties that apply to NewCompiler just as well if you disable closure compilation and there is really no reason whatsoever why a working closure solution couldn't be adopted to the current compiler (which would mitigate the risk factors of adoption dramatically). Just like I was saying in the part you'd snipped away I'd be quite interested in adopting a working solution (which doesn't strike me as hard given the simplistic nature of the current compiler).
BTW, since you phrase it as such a big deal, is anyone using NewCompiler with Seaside? If it is really as problematic as you make it above I'd expect to see users of Seaside using it. Do they? If so, what's the practical experience?
Cheers,
- Andreas
Mth
2007/12/19, Andreas Raab andreas.raab@gmx.de:
Philippe Marschall wrote:
Concerning speed: When doing an [Object2 compileAll] timeToRun (Object2 being the partial copy of class Object that NewCompiler was able to hand me) I ended up with 1444 msecs for Compiler and 4020 msecs for NewCompiler. Unless I'm mistaken this is an improvement over the last version I tried which was roughly 4x slower (this is still 2.8x slower but it's a definitive improvement).
I wasn't talking about compilation speed. What's really limiting for bigger projects is not the speed of the compiler but PackageInfo which makes Monticello slow. I was talking about execution speed.
I see. Has anyone measured the performance of the micro and macro benchmarks with the entire system compiled for using closures? I'm particularly interested in the macro benchmarks.
You should ask Marcus about what the current status exactly is. Or Klaus.
It would be a big step if it were possible to recompile Squeak successfully
The NewCompiler images from Marcus do this since a long time ago. See the Recompiler class.
Ah, good.
and perhaps even bytecode-identically.
Nah, NewCompiler in certain circumstances generates marginally more efficient bytecode.
This may actually be worthwhile "undoing" simply for being able to verify correctness.
And, do you know of any energy focused on improving the existing alternatives?
What alternatives?
The compiler that's actually being used. The one that's 3x faster ;-)
The one that's causing so much trouble in Seaside? The that gives you this cool C++ feeling bugs where you spend two hours trying to find out why your block arguments are all messed up and you think you went crazy? And then suddenly you add a lot of #fixTemps everywhere until it works?
Am I missing a smiley somewhere?
No, there's nothing funny about it.
You are describing properties that apply to NewCompiler just as well if you disable closure compilation
Yes.
and there is really no reason whatsoever why a working closure solution couldn't be adopted to the current compiler (which would mitigate the risk factors of adoption dramatically).
They why was the closure solution of the new compiler rejected?
Just like I was saying in the part you'd snipped away I'd be quite interested in adopting a working solution (which doesn't strike me as hard given the simplistic nature of the current compiler).
BTW, since you phrase it as such a big deal, is anyone using NewCompiler with Seaside? If it is really as problematic as you make it above I'd expect to see users of Seaside using it. Do they? If so, what's the practical experience?
Seaside used to send #fixTemps to every block (!) now it has a more sophisticated hack: #fixCallbackTemps (that's right a web frameworks includes hacks to work around block bugs). That works in most situations. Problems with blocks are rare, however when they happen you spend a lot of time figuring out what went wrong. With time you get a certain feeling of what constructs trigger block bugs and try to avoid them.
Cheers Philippe
Philippe Marschall wrote:
there is really no reason whatsoever why a working closure solution couldn't be adopted to the current compiler (which would mitigate the risk factors of adoption dramatically).
They why was the closure solution of the new compiler rejected?
"The closure solution rejected"? Not that I know of. I haven't even seen a thorough discussion about its merits (which should include benchmarks etc) - what I did see was some talk that led me to believe that there is still a ways to go here. To the best of my knowledge this solution isn't "final" by any means.
Or do you mean the compiler itself? In which case all I can say it's hard to recommend a compiler which can't even recompile Object ;-)
Cheers, - Andreas
The newcompiler can recompile the complete image. I do not which version you are using.
Stef
On 19 déc. 07, at 18:12, Andreas Raab wrote:
Philippe Marschall wrote:
there is really no reason whatsoever why a working closure solution couldn't be adopted to the current compiler (which would mitigate the risk factors of adoption dramatically).
They why was the closure solution of the new compiler rejected?
"The closure solution rejected"? Not that I know of. I haven't even seen a thorough discussion about its merits (which should include benchmarks etc) - what I did see was some talk that led me to believe that there is still a ways to go here. To the best of my knowledge this solution isn't "final" by any means.
Or do you mean the compiler itself? In which case all I can say it's hard to recommend a compiler which can't even recompile Object ;-)
Cheers,
- Andreas
stephane ducasse wrote:
The newcompiler can recompile the complete image. I do not which version you are using.
* Start with 3.9.7067-final * From Squeakmap load the latest published version of AST * From Squeakmap load the latest published version of NewCompiler * Preferences enable: #compileUseNewCompiler * Object compileAll * Boom.
Besides, when I tried to load NewCompiler yesterday evening via Universes in 3.10 it failed to load entirely. Not sure why (and I wasn't in a mood of trying to find out).
Cheers, - Andreas
On Dec 19, 2007, at 9:48 PM, Andreas Raab wrote:
stephane ducasse wrote:
The newcompiler can recompile the complete image. I do not which version you are using.
- Start with 3.9.7067-final
- From Squeakmap load the latest published version of AST
- From Squeakmap load the latest published version of NewCompiler
- Preferences enable: #compileUseNewCompiler
- Object compileAll
- Boom.
Besides, when I tried to load NewCompiler yesterday evening via Universes in 3.10 it failed to load entirely. Not sure why (and I wasn't in a mood of trying to find out).
Cheers,
- Andreas
Sorry I made a mistake in the universe browser. It is now fix
Mth
Mathieu Suen wrote:
Sorry I made a mistake in the universe browser. It is now fix
Indeed much better now. I'm getting halfways through now with recompiling everything but now it dies in the middle of recompiling Preferences (you can try this by executing "Preferences compileAll"). Any ideas?
Cheers, - Andreas
On Dec 20, 2007, at 1:55 AM, Andreas Raab wrote:
Mathieu Suen wrote:
Sorry I made a mistake in the universe browser. It is now fix
Indeed much better now. I'm getting halfways through now with recompiling everything but now it dies in the middle of recompiling Preferences (you can try this by executing "Preferences compileAll"). Any ideas?
Cheers,
- Andreas
It work for me which version do you use?
Mth
El 12/19/07 5:48 PM, "Andreas Raab" andreas.raab@gmx.de escribió:
- Start with 3.9.7067-final
- From Squeakmap load the latest published version of AST
- From Squeakmap load the latest published version of NewCompiler
- Preferences enable: #compileUseNewCompiler
- Object compileAll
- Boom.
Besides, when I tried to load NewCompiler yesterday evening via Universes in 3.10 it failed to load entirely. Not sure why (and I wasn't in a mood of trying to find out).
Cheers,
- Andreas
Not so. I attach pict. If this is a good to have, then all experts in this field should request for 3.11.
Edgar
Well so propose us your solution. At least the new compiler exist .
On Dec 19, 2007, at 6:12 PM, Andreas Raab wrote:
Philippe Marschall wrote:
there is really no reason whatsoever why a working closure solution couldn't be adopted to the current compiler (which would mitigate the risk factors of adoption dramatically).
They why was the closure solution of the new compiler rejected?
"The closure solution rejected"? Not that I know of. I haven't even seen a thorough discussion about its merits (which should include benchmarks etc) - what I did see was some talk that led me to believe that there is still a ways to go here. To the best of my knowledge this solution isn't "final" by any means.
Or do you mean the compiler itself? In which case all I can say it's hard to recommend a compiler which can't even recompile Object ;-)
Cheers,
- Andreas
Mth
Mathieu Suen wrote:
Well so propose us your solution. At least the new compiler exist .
You are missing my point entirely. I wasn't talking about inventing a solution - I was talking about adopting one, such as from NewCompiler if it's a viable, robust solution. But I don't know if it is, and I find little of a discussion about it and I couldn't try for myself. So it's kinda hard to tell.
Cheers, - Andreas
On Dec 19, 2007, at 6:12 PM, Andreas Raab wrote:
Philippe Marschall wrote:
there is really no reason whatsoever why a working closure solution couldn't be adopted to the current compiler (which would mitigate the risk factors of adoption dramatically).
They why was the closure solution of the new compiler rejected?
"The closure solution rejected"? Not that I know of. I haven't even seen a thorough discussion about its merits (which should include benchmarks etc) - what I did see was some talk that led me to believe that there is still a ways to go here. To the best of my knowledge this solution isn't "final" by any means.
Or do you mean the compiler itself? In which case all I can say it's hard to recommend a compiler which can't even recompile Object ;-)
Cheers,
- Andreas
Mth
Mathieu Suen wrote:
Well so propose us your solution.
Okay, I promised not to invent a solution but it's one of "those" problems that I've thought about for a while in the past so let's see. Here is how I would address the problem of implementing full closures in Squeak:
1) First, I would fix the "attempt to evaluate a block that is already evaluated" problem. That's about ten lines of code in the VM - if you look at primitiveValue[withArgs] you'll find that we fail if the CallerIndex is non-nil. Instead of failing, I'd clone the original context and continue as usual.
At this point you'd be able to execute code like:
| v | fac := [v > 0 ifTrue:[1] ifFalse:[v := v-1. fac value * (v+1)]]. fac value: 10.
which allows recursive evaluation but without block args (since they get hacked in some terrible ways).
2) Next, I'd deal with block variables (temps and args). Instead of sharing the method contexts, blocks would store their own temps and access them via tempAt:. So that for example code like here:
self do:[:v| | v2 | v2 := v squared. v2].
would be compiled into to:
self do:[:v| thisContext push: nil. "allocate v2" thisContext tempAt: 2 "v2" put: (thisContext tempAt: 1 "v") squared. thisContext tempAt: 2. ].
This would be slower than what we have today but obviously, the pattern "thisContext tempAt:" is a prime candidate for optimization. At this point we'd be able to run the following correctly:
fac [:v| v = 0 ifTrue:[1] ifFalse:[ v * (fac value: v-1)].
The implementation would be pretty straightforward too, since all you'd have to do is to add BlockVariableNode that generates tempAt: messages (plus some initialization). Very, very straightforward.
3) Next, I would deal with "outer" state like here:
self do:[:x| x do:[:y| x * y] ].
This is probably the hardest part since dealing with "non-method closed-over state" can be tricky due to life-time issues. Anyway, for starters I would just add a hidden temporary "outerContext" for all blocks that is just a temp like any other such that the compiler can generate code like:
self do:[:x| x do:[:y| ((thisContext tempAt: 1) "outerContext" tempAt: 1 "x") * (thisContext tempAt: 2) "y" ]. ].
At this point (I think - please correct me if I'm missing something) we have *complete* closure semantics in a straightforward way. What remains is optimization.
4) Lastly, optimization. I would start with the obvious push/popBlockTemp bytecodes which would bring all of the above up to par with current Squeak (except (3) which could still be slower). Finally, one might look at the life-time of closed-over state - it is certainly not necessary to preserve the entire outer context so there may be other things that could be done to fix it.
In any case, the above looks pretty simple and straightforward. Wouldn't you agree?
Cheers, - Andreas
On 19-Dec-07, at 8:19 PM, Andreas Raab wrote:
Mathieu Suen wrote:
Well so propose us your solution.
Okay, I promised not to invent a solution but it's one of "those" problems that I've thought about for a while in the past so let's see. Here is how I would address the problem of implementing full closures in Squeak:
Well at a quick look it appears plausible - but I have a much simpler idea. Eliot solved this problems in 1987 - in fact I *think* he was the first so to do but wouldn't bet any large amount of money - and the basics are described in his seminal paper from the 87 OOPSLa proceedings.
Since he is currently employed to do work in Squeak it seems entirely reasonable to me to see if he would (re)solve the problem for us.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Decafalon (n.): The grueling event of getting through the day consuming only things that are good for you.
tim Rowledge wrote:
On 19-Dec-07, at 8:19 PM, Andreas Raab wrote:
Mathieu Suen wrote:
Well so propose us your solution.
Okay, I promised not to invent a solution but it's one of "those" problems that I've thought about for a while in the past so let's see. Here is how I would address the problem of implementing full closures in Squeak:
Well at a quick look it appears plausible - but I have a much simpler idea. Eliot solved this problems in 1987 - in fact I *think* he was the first so to do but wouldn't bet any large amount of money - and the basics are described in his seminal paper from the 87 OOPSLa proceedings.
Note that Scheme interpreters have done this earlier than 1987 probably; Eliot was the first to bring the full power of lambdas to Smalltalk.
GNU Smalltalk's closures more or less follow the same style that Andreas outlined, except that as far as I understood he doesn't need BlockClosure objects because he keeps the bytecodes in the CompiledMethod rather than extracting them into a CompiledBlock. This makes sense given Squeak's Smalltalk-80 legacy, and it looks like BrouHaHa did the same.
A very useful optimization is to implement copying blocks. I've never found the time to implement it in GNU Smalltalk, but it should give interesting speedups.
Paolo
On 20-Dec-07, at 12:13 AM, Paolo Bonzini wrote:
Note that Scheme interpreters have done this earlier than 1987 probably; Eliot was the first to bring the full power of lambdas to Smalltalk.
Well yes, we *are* talking about Smalltalk here, after all.
I think BHH kept the bytecodes all in the CM but used a Closure object in place of using the copyBlock bytecode and then a blockcontext was created each time the closure was evaluated. But it is a very long time since I last did any active development of that VM, most likely 1991. Too long to remember details!
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: EIV: Erase IPL Volume
Tim Rowledge wrote:
Well at a quick look it appears plausible - but I have a much simpler idea. Eliot solved this problems in 1987 - in fact I *think* he was the first so to do but wouldn't bet any large amount of money - and the basics are described in his seminal paper from the 87 OOPSLa proceedings.
Well, I solved this problem by accident in 1984:
http://www.lsi.usp.br/~jecel/st84.txt
I didn't pay enough attention to the blue book and never considered the possibility that Smalltalk-80 didn't have full closures. Of course, since nobody was able to use my design Eliot's should be considered as the first. Though the text in the above link says this was just a "paper design", the main ideas were actually tested in a program writetn in TI99/4A Extended BASIC.
-- Jecel
On Thu, 20 Dec 2007 05:19:33 +0100, Andreas Raab wrote:
Mathieu Suen wrote:
Well so propose us your solution.
Okay, I promised not to invent a solution but it's one of "those" problems that I've thought about for a while in the past so let's see. Here is how I would address the problem of implementing full closures in Squeak:
- First, I would fix the "attempt to evaluate a block that is already
evaluated" problem. That's about ten lines of code in the VM - if you look at primitiveValue[withArgs] you'll find that we fail if the CallerIndex is non-nil. Instead of failing, I'd clone the original context and continue as usual.
At this point you'd be able to execute code like:
| v | fac := [v > 0 ifTrue:[1] ifFalse:[v := v-1. fac value * (v+1)]]. fac value: 10.
which allows recursive evaluation but without block args
Did you mean
| v | v := 10. fac := [v > 0 ifFalse:[1] ifTrue:[v := v-1. fac fixTemps clone value * (v+1)]]. fac clone value
I confess that I copied the idea from Self's view on method activation (a clone of a template :)
But then, how'd that be on Smalltalks other than Squeak, i.e. if a Seasider creates+tests such code with Squeak and later runs it on another Seaside-capable VM? Looks like opening the cross-Smalltalk Hunting Season :(
(since they get hacked in some terrible ways).
- Next, I'd deal with block variables (temps and args). Instead of
sharing the method contexts, blocks would store their own temps and access them via tempAt:. So that for example code like here:
self do:[:v| | v2 | v2 := v squared. v2].
would be compiled into to:
self do:[:v| thisContext push: nil. "allocate v2" thisContext tempAt: 2 "v2" put: (thisContext tempAt: 1 "v") squared. thisContext tempAt: 2. ].
Sure, that'd be an easy one for the compiler+decompiler guild.
This would be slower than what we have today but obviously, the pattern "thisContext tempAt:" is a prime candidate for optimization.
AFAIK there's no bytecode which pushes/pops temps in *activeContext*, only in *homeContext* (and so the main reason behind this discussion). That looks like a major change to VM.
At this point we'd be able to run the following correctly:
fac [:v| v = 0 ifTrue:[1] ifFalse:[ v * (fac value: v-1)].
This statement does not parse. Can you reformulate without syntax error.
The implementation would be pretty straightforward too, since all you'd have to do is to add BlockVariableNode that generates tempAt: messages (plus some initialization). Very, very straightforward.
Next, I would deal with "outer" state like here:
self do:[:x| x do:[:y| x * y] ].
This is probably the hardest part since dealing with "non-method closed-over state" can be tricky due to life-time issues. Anyway, for starters I would just add a hidden temporary "outerContext" for all blocks that is just a temp like any other such that the compiler can generate code like:
self do:[:x| x do:[:y| ((thisContext tempAt: 1) "outerContext" tempAt: 1 "x") * (thisContext tempAt: 2) "y" ]. ].
Sort of "outerContext" is already part of the BlockClosure handling with NewCompiler, there it's an instance of ClosureEnvironment.
At this point (I think - please correct me if I'm missing something) we have *complete* closure semantics in a straightforward way.
I would like to see comment(s) on the above from Marcus, but he's currently not subscribed to squeak-dev (cc'ed him anyways).
What remains is optimization.
Bryce & Marcus & Math & my discussed optimizing BlockClosure+bytecodes some time ago, if that's of interest
- http://www.google.com/search?&q=Newcompiler+bytecode+recycling+closure
- Lastly, optimization.
During the discussion we came to the point that context recycling, which is interrupted by explicit and implicit #pushActiveContext, would no longer be hindered because BlockClosure seems to be a complete replacement for BlockContext. By that time it looked like a very promising speed-up.
/Klaus
I would start with the obvious push/popBlockTemp bytecodes which would bring all of the above up to par with current Squeak (except (3) which could still be slower). Finally, one might look at the life-time of closed-over state - it is certainly not necessary to preserve the entire outer context so there may be other things that could be done to fix it.
In any case, the above looks pretty simple and straightforward. Wouldn't you agree?
Cheers,
- Andreas
Hi,
I am busy with writing my thesis right now... so it might take a while till I can answer.
I will try to write something over christmas to explain why I actually like Anthony Hannan's design of the NewCompiler (that it is split in SmaCC-arser, RB- AST,..., IRBuilder) even though it is slower in compiling code then the old compiler.
Sadly do not think that I can take part in any lengthy email-list- discussion right now, this has to wait untill March 08.
Marcus
On 20.12.2007, at 11:53, Klaus D. Witzel wrote:
On Thu, 20 Dec 2007 05:19:33 +0100, Andreas Raab wrote:
Mathieu Suen wrote:
Well so propose us your solution.
Okay, I promised not to invent a solution but it's one of "those" problems that I've thought about for a while in the past so let's see. Here is how I would address the problem of implementing full closures in Squeak:
- First, I would fix the "attempt to evaluate a block that is
already evaluated" problem. That's about ten lines of code in the VM - if you look at primitiveValue[withArgs] you'll find that we fail if the CallerIndex is non-nil. Instead of failing, I'd clone the original context and continue as usual.
At this point you'd be able to execute code like:
| v | fac := [v > 0 ifTrue:[1] ifFalse:[v := v-1. fac value * (v+1)]]. fac value: 10.
which allows recursive evaluation but without block args
Did you mean
| v | v := 10. fac := [v > 0 ifFalse:[1] ifTrue:[v := v-1. fac fixTemps clone value * (v+1)]]. fac clone value
I confess that I copied the idea from Self's view on method activation (a clone of a template :)
But then, how'd that be on Smalltalks other than Squeak, i.e. if a Seasider creates+tests such code with Squeak and later runs it on another Seaside-capable VM? Looks like opening the cross-Smalltalk Hunting Season :(
(since they get hacked in some terrible ways).
- Next, I'd deal with block variables (temps and args). Instead of
sharing the method contexts, blocks would store their own temps and access them via tempAt:. So that for example code like here:
self do:[:v| | v2 | v2 := v squared. v2].
would be compiled into to:
self do:[:v| thisContext push: nil. "allocate v2" thisContext tempAt: 2 "v2" put: (thisContext tempAt: 1 "v") squared. thisContext tempAt: 2. ].
Sure, that'd be an easy one for the compiler+decompiler guild.
This would be slower than what we have today but obviously, the pattern "thisContext tempAt:" is a prime candidate for optimization.
AFAIK there's no bytecode which pushes/pops temps in *activeContext*, only in *homeContext* (and so the main reason behind this discussion). That looks like a major change to VM.
At this point we'd be able to run the following correctly:
fac [:v| v = 0 ifTrue:[1] ifFalse:[ v * (fac value: v-1)].
This statement does not parse. Can you reformulate without syntax error.
The implementation would be pretty straightforward too, since all you'd have to do is to add BlockVariableNode that generates tempAt: messages (plus some initialization). Very, very straightforward.
- Next, I would deal with "outer" state like here:
self do:[:x| x do:[:y| x * y] ].
This is probably the hardest part since dealing with "non-method closed-over state" can be tricky due to life-time issues. Anyway, for starters I would just add a hidden temporary "outerContext" for all blocks that is just a temp like any other such that the compiler can generate code like:
self do:[:x| x do:[:y| ((thisContext tempAt: 1) "outerContext" tempAt: 1 "x") * (thisContext tempAt: 2) "y" ]. ].
Sort of "outerContext" is already part of the BlockClosure handling with NewCompiler, there it's an instance of ClosureEnvironment.
At this point (I think - please correct me if I'm missing something) we have *complete* closure semantics in a straightforward way.
I would like to see comment(s) on the above from Marcus, but he's currently not subscribed to squeak-dev (cc'ed him anyways).
What remains is optimization.
Bryce & Marcus & Math & my discussed optimizing BlockClosure +bytecodes some time ago, if that's of interest
- Lastly, optimization.
During the discussion we came to the point that context recycling, which is interrupted by explicit and implicit #pushActiveContext, would no longer be hindered because BlockClosure seems to be a complete replacement for BlockContext. By that time it looked like a very promising speed-up.
/Klaus
I would start with the obvious push/popBlockTemp bytecodes which would bring all of the above up to par with current Squeak (except (3) which could still be slower). Finally, one might look at the life-time of closed-over state - it is certainly not necessary to preserve the entire outer context so there may be other things that could be done to fix it.
In any case, the above looks pretty simple and straightforward. Wouldn't you agree?
Cheers,
- Andreas
Marcus -- Marcus Denker -- denker@iam.unibe.ch http://www.iam.unibe.ch/~denker
Your idea is quite the same than the one from the NewCompiler. In your idea #tempsAt: will fetch temps inside an array of an instance variable of the BlockContext. In the NewCompiler we have the same and we call this array a ClosureEnvironment.
As you said storing or pushing from temp is prim candidate. I have implement 2 bytecodes pushing and storing in ClosureEnvironment. I also have implement some primitive for createBlock: and stuff like that. The benchmark I have made show that it run 2 time slower but some other optimization could be done.
We also don't share the bytecode with the home context. instead we create a compiled method that we put in the literal array.
Note that we only create this method when it is needed. For example if the block do not reference a outer temp we use the old block context.
Cheers,
On Dec 20, 2007, at 5:19 AM, Andreas Raab wrote:
Mathieu Suen wrote:
Well so propose us your solution.
Okay, I promised not to invent a solution but it's one of "those" problems that I've thought about for a while in the past so let's see. Here is how I would address the problem of implementing full closures in Squeak:
- First, I would fix the "attempt to evaluate a block that is
already evaluated" problem. That's about ten lines of code in the VM
- if you look at primitiveValue[withArgs] you'll find that we fail
if the CallerIndex is non-nil. Instead of failing, I'd clone the original context and continue as usual.
At this point you'd be able to execute code like:
| v | fac := [v > 0 ifTrue:[1] ifFalse:[v := v-1. fac value * (v+1)]]. fac value: 10.
which allows recursive evaluation but without block args (since they get hacked in some terrible ways).
- Next, I'd deal with block variables (temps and args). Instead of
sharing the method contexts, blocks would store their own temps and access them via tempAt:. So that for example code like here:
self do:[:v| | v2 | v2 := v squared. v2].
would be compiled into to:
self do:[:v| thisContext push: nil. "allocate v2" thisContext tempAt: 2 "v2" put: (thisContext tempAt: 1 "v") squared. thisContext tempAt: 2. ].
This would be slower than what we have today but obviously, the pattern "thisContext tempAt:" is a prime candidate for optimization. At this point we'd be able to run the following correctly:
fac [:v| v = 0 ifTrue:[1] ifFalse:[ v * (fac value: v-1)].
The implementation would be pretty straightforward too, since all you'd have to do is to add BlockVariableNode that generates tempAt: messages (plus some initialization). Very, very straightforward.
- Next, I would deal with "outer" state like here:
self do:[:x| x do:[:y| x * y] ].
This is probably the hardest part since dealing with "non-method closed-over state" can be tricky due to life-time issues. Anyway, for starters I would just add a hidden temporary "outerContext" for all blocks that is just a temp like any other such that the compiler can generate code like:
self do:[:x| x do:[:y| ((thisContext tempAt: 1) "outerContext" tempAt: 1 "x") * (thisContext tempAt: 2) "y" ]. ].
At this point (I think - please correct me if I'm missing something) we have *complete* closure semantics in a straightforward way. What remains is optimization.
- Lastly, optimization. I would start with the obvious push/
popBlockTemp bytecodes which would bring all of the above up to par with current Squeak (except (3) which could still be slower). Finally, one might look at the life-time of closed-over state - it is certainly not necessary to preserve the entire outer context so there may be other things that could be done to fix it.
In any case, the above looks pretty simple and straightforward. Wouldn't you agree?
Cheers,
- Andreas
Mth
On Dec 19, 2007, at 9:12 AM, Andreas Raab wrote:
Philippe Marschall wrote:
Concerning speed: When doing an [Object2 compileAll] timeToRun (Object2 being the partial copy of class Object that NewCompiler was able to hand me) I ended up with 1444 msecs for Compiler and 4020 msecs for NewCompiler. Unless I'm mistaken this is an improvement over the last version I tried which was roughly 4x slower (this is still 2.8x slower but it's a definitive improvement).
I wasn't talking about compilation speed. What's really limiting for bigger projects is not the speed of the compiler but PackageInfo which makes Monticello slow. I was talking about execution speed.
I see. Has anyone measured the performance of the micro and macro benchmarks with the entire system compiled for using closures? I'm particularly interested in the macro benchmarks.
You should ask Marcus about what the current status exactly is. Or Klaus.
It would be a big step if it were possible to recompile Squeak successfully
The NewCompiler images from Marcus do this since a long time ago. See the Recompiler class.
Ah, good.
and perhaps even bytecode-identically.
Nah, NewCompiler in certain circumstances generates marginally more efficient bytecode.
This may actually be worthwhile "undoing" simply for being able to verify correctness.
And, do you know of any energy focused on improving the existing alternatives?
What alternatives?
The compiler that's actually being used. The one that's 3x faster ;-)
The one that's causing so much trouble in Seaside? The that gives you this cool C++ feeling bugs where you spend two hours trying to find out why your block arguments are all messed up and you think you went crazy? And then suddenly you add a lot of #fixTemps everywhere until it works?
Am I missing a smiley somewhere? You are describing properties that apply to NewCompiler just as well if you disable closure compilation and there is really no reason whatsoever why a working closure solution couldn't be adopted to the current compiler (which would mitigate the risk factors of adoption dramatically). Just like I was saying in the part you'd snipped away I'd be quite interested in adopting a working solution (which doesn't strike me as hard given the simplistic nature of the current compiler).
The NewCompiler status is:
Compiler: - Block closure - Work - Non block closure - Work (The old block style BlockContext)
Decompiler - Block closure - Don't Work (That should not take too long to fix it) - Non block closure - Work
Now still some job needs to be done for the integration in tools. Especially in the debugger there is some bug due to selection and to context. Debugger uses a complex interface to compile/decompile code (dunno why debugger needs to compile/decompile). Also when you try to debug an expression inside the debugger it crash. So far I haven't figure out how/why to solve this bug.
BTW, since you phrase it as such a big deal, is anyone using NewCompiler with Seaside? If it is really as problematic as you make it above I'd expect to see users of Seaside using it. Do they? If so, what's the practical experience?
Cheers,
- Andreas
Mth
"Mathieu Suen" mathk.sue@gmail.com wrote
The NewCompiler status is:
Compiler:
- Block closure - Work
I have not yet tried NewCompiler, but here is one that does not work in 3.10. At point 1>>, newModel is correct and non-nil. At point 2>> newModel is nil. I'm trying this on 3.10gamma.
Thanks -- Sophie
renderNewModelButton: html { M1. M2 } do: [ : modelClass | html submitButton callback: [ | newModel | self session magma commit: [ newModel := self repository models add: modelClass new. 1>> self halt] fixTemps. self halt. 2>> self call: (newModel viewerClass new model: newModel). selectedModel := nil ] fixTemps; value: #New , ' ' , modelClass name ]
Decompiler
- Block closure - Don't Work (That should not take too long to fix it)
That would unfortunately make it impossible for newbies like me to debug etc.
That would unfortunately make it impossible for newbies like me to debug etc.
Did you try to extract all the callback code to a separate method? I bet this will solve all problems for everybody.
Lukas
"Lukas Renggli" renggli@gmail.com wrote in message
That would unfortunately make it impossible for newbies like me to debug etc.
Did you try to extract all the callback code to a separate method? I bet this will solve all problems for everybody.
Yes, it gave me a work-around + better structured code, thanks! The underlying closure bug could still cause a lot of trouble, so I posted the example here as well.
Thanks - Sophie
Hi,
Have you try:
renderNewModelButton: html { M1. M2 } do: [ : modelClass | html submitButton callback: [ | newModel | self session magma commit: [ newModel := self repository models add: modelClass new.]
self call: (newModel viewerClass new model: newModel). selectedModel := nil ] fixTemps; value: #New , ' ' , modelClass name ]
On Dec 19, 2007, at 3:08 PM, itsme213 wrote:
"Mathieu Suen" mathk.sue@gmail.com wrote
The NewCompiler status is:
Compiler:
- Block closure - Work
I have not yet tried NewCompiler, but here is one that does not work in 3.10. At point 1>>, newModel is correct and non-nil. At point 2>> newModel
Of cours at pont 2 is nil cause: The #fixTemps of the commit block will create a copy of the newModel temp. Then when you affect a value to the newModel inside the block you are affecting the copy not the original one.
Will the second fix temp will prevent you having modelClass being the same for the commit of M1 and M2. It will copy the modelClass temps.
Ask if I haven't been clear.
Cheers,
is nil. I'm trying this on 3.10gamma.
Thanks -- Sophie
renderNewModelButton: html { M1. M2 } do: [ : modelClass | html submitButton callback: [ | newModel | self session magma commit: [ newModel := self repository models add: modelClass new. 1>> self halt] fixTemps. self halt. 2>> self call: (newModel viewerClass new model: newModel). selectedModel := nil ] fixTemps; value: #New , ' ' , modelClass name ]
Decompiler
- Block closure - Don't Work (That should not take too long to fix
it)
That would unfortunately make it impossible for newbies like me to debug etc.
Mth
On Wed, 19 Dec 2007 15:50:23 +0100, Mathieu Suen wrote:
Hi,
Have you try:
I would have attempted the same as Math did below since the first (now omitted) #fixTemps made a private copy of temp var newModel (just for use by the block given to #commit:).
renderNewModelButton: html { M1. M2 } do: [ : modelClass | html submitButton callback: [ | newModel | self session magma commit: [ newModel := self repository models add: modelClass new.]
self call: (newModel viewerClass new model: newModel). selectedModel := nil ] fixTemps; value: #New , ' ' ,
modelClass name ]
On Dec 19, 2007, at 3:08 PM, itsme213 wrote:
"Mathieu Suen" mathk.sue@gmail.com wrote
The NewCompiler status is:
Compiler:
- Block closure - Work
I have not yet tried NewCompiler, but here is one that does not work in 3.10. At point 1>>, newModel is correct and non-nil. At point 2>> newModel
Of cours at pont 2 is nil cause: The #fixTemps of the commit block will create a copy of the newModel temp. Then when you affect a value to the newModel inside the block you are affecting the copy not the original one.
Will the second fix temp will prevent you having modelClass being the same for the commit of M1 and M2. It will copy the modelClass temps.
Ask if I haven't been clear.
Cheers,
is nil. I'm trying this on 3.10gamma.
Thanks -- Sophie
renderNewModelButton: html { M1. M2 } do: [ : modelClass | html submitButton callback: [ | newModel | self session magma commit: [ newModel := self repository models add: modelClass new. 1>> self halt] fixTemps. self halt. 2>> self call: (newModel viewerClass new model: newModel). selectedModel := nil ] fixTemps; value: #New , ' ' , modelClass name ]
Decompiler
- Block closure - Don't Work (That should not take too long to fix it)
That would unfortunately make it impossible for newbies like me to debug etc.
Mth
I did not follow carefully but marcus and math are using the new compiler daily. Now I think that they do not use/publish on squeakmap. Mathieu fixed (all or most of the decompiling problems with nested blocks).
I asked him to reply. Our goal is to make sure that NewCompiler is working (they fixed the poor error handling and decompilation so that we can really use it). Normally marcus recompiled all the image with the NewCompiler as a simple tests and if I recall correctly it works.
Stef
On 19 déc. 07, at 08:06, Andreas Raab wrote:
Philippe Marschall wrote:
The compiler itself works and can be loaded into 3.9 without problems.
Unfortunately, I can't confirm the "working without problems" part in 3.9. For testing the speed I was copying class Object and it barfed halfways through in #copyTwoLevel in a perfectly valid expression:
class isVariable ifTrue:[ index _ self basicSize. [index > 0] whileTrue:[ newObject basicAt: index put: (self basicAt: index) shallowCopy. index _ index - 1]"xxxxx Argument expected ->xxxxx"].
(note: you can just select the expression in a workspace and doIt if you have compileUseNewCompiler enabled; it'll complain right away) This is very strange and the error message makes little sense to me - I tried adding a period because perhaps it was expecting a message to the block but that doesn't do it. Has anyone tried to simply recompile the entirety of 3.9 to see if that works or not? I've been using 3.9-final-7067 for my test with the latest published Squeakmap releases for AST and NewCompiler. If there is a better version to use, please let me know.
[BTW, there are many other obscure errors that happen when I simply remove the offending code and continue to compile. Ultimately it appears to die somewhere unrelated at first glance but since this only happens when using compileUseNewCompiler I'm pretty sure it's related to some NewCompiler issues]
Error messages are much improved and unlike your standard SmaCC error messages. Performance is lower. There is a lot of work going on in the decompiler and I don't know the exact state of this.
Nice to hear about the error messages. I hadn't tried the compiler in a while but the above really shows that error messages really have greatly improved.
Concerning speed: When doing an [Object2 compileAll] timeToRun (Object2 being the partial copy of class Object that NewCompiler was able to hand me) I ended up with 1444 msecs for Compiler and 4020 msecs for NewCompiler. Unless I'm mistaken this is an improvement over the last version I tried which was roughly 4x slower (this is still 2.8x slower but it's a definitive improvement).
You should ask Marcus about what the current status exactly is. Or Klaus.
It would be a big step if it were possible to recompile Squeak successfully and perhaps even bytecode-identically. Being able to generate an identical version of the method would allow people to trivially verify the correct functioning of NewCompiler.
And, do you know of any energy focused on improving the existing alternatives?
What alternatives?
The compiler that's actually being used. The one that's 3x faster ;-) If NewCompiler worked properly I'd be actually quite tempted to just adopt the closure solution to the current compiler since it's smaller, faster, and well-established.
Cheers,
- Andreas
may be you should ask in the compiler mailing-list since marcus does not read squeak-dev anymore.
stef On 19 déc. 07, at 00:34, Chris Muller wrote:
Would you mind sharing what details you know about the current state of closures and what the trade-offs are using the NewCompiler?
And, do you know of any energy focused on improving the existing alternatives?
On Dec 18, 2007 2:08 PM, Philippe Marschall philippe.marschall@gmail.com wrote:
2007/12/18, itsme213 itsme213@hotmail.com:
Do block closures work right in 3.9? 3.10?
No, block closures are broken in every Squeak. You have to load the NewCompiler and activate two compiler settings to get working block closures. You'll have to live with some other issues though.
Cheers Philippe
I am struggling with local variables used across nested blocks (Magma #commit: within Seaside #callback: or Scriptaculous #triggerInPlaceEditor etc.) and am finding it quite frustrating. I had some #fixTemps but am totally guessing at when and why to use it.
Thanks - Sophie
Hi
What #fixTemps do is copy the home context. for example
#(1 2 3) collect: [:each | [each < 3]] collect: [:block | block value] result is: #(false false false)
you then need a #fixTemps:
#(1 2 3) collect: [:each | [each < 3] fixTemps] collect: [:block | block value] result is: #(true true false)
Here the temp call each will exist in 3 different context. The 3 block [each < 3] will have 3 different parent.
HTH
On Dec 18, 2007, at 7:10 PM, itsme213 wrote:
Do block closures work right in 3.9? 3.10?
I am struggling with local variables used across nested blocks (Magma #commit: within Seaside #callback: or Scriptaculous #triggerInPlaceEditor etc.) and am finding it quite frustrating. I had some #fixTemps but am totally guessing at when and why to use it.
Thanks - Sophie
Mth
squeak-dev@lists.squeakfoundation.org