Let me summarize the points so far -
1. Missing socket options + "security stuff in file access" - Craig plans to implement within this version, before we start integrating. 2. Categorization imperfect - side effect of 3.3 transitions, will be corrected. 3. Initialization magie prevents loading Flow twice - not really a problem, so long as it's only the magic. We'll require other testing to make sure Flow does everything it's required to. 4. Backwards compatibility - - Flow changes a few names from X to OldX, otherwise everything is intact. - There are many applications refering to the old names, in the image and outside it. - This means that Flow does break everything not explicitly fixed, with fixes being just name changes (correct me if I got this wrong). - An alternative to the current situation is to make flow disjunct from all the existing code and then let people port as they wish. This requires some changes to names. - The old stuff probably should remain loadable as a package even after any transition is "over". 5. Proposals to take the primitive interfaces but not the stream refactoring - I can't summarize this because there were no specific proposal/problems raised (Michael? Goran?) - Inserting an idea here - *if* we agree the stream refactoring leads to a better state than streams are in right now, let's integrate it independently of the "better network primitives" issues. 6. Threads dependency - Flow as is requires OS threads. - Two options are to wait for platforms to aquire them or to work around missing threads in the Smalltalk code. - Some platform don't/shouldn't have preemptive threads (PS/2) - There are reasons to do the thread emulation inside the primitive and outside it.
1 + 2 are relevant to whether the code is actually better, and Craig agrees to fix them. 3 seems to be a non issue, since it's just initialization magic that prevents reloading (not lacking functionality). 4 is about the transition, which I agree we might have to hone a bit in order to prevent breakage, and make porting come after acceptance, and not vice versa. OTOH, these are only relevant if we clarify all the "should we do it at all issues". Which are:
5 - Michael, could you please clarify you intent here? I'm not sure whether your comments here are about transition or the desireability of the refactoring itself.
6 seems to hinge on whether Flow can have reasonable support in the image for emulating threading on platform that lack it, and on whether there's a reason that's not good enough. I think we agree that our goal is Squeak becomes more portable by this change. On the face of it, one implementation of platform agnostic thread emulation in the image is better than many different schemes in the platform specific primitives (our current situation). This is just my impression, and still doesn't prove that such in image emulation is reasonably doable.
Daniel
Craig Latta craig.latta@netjam.org wrote:
Hi Andreas--
From what I can see looking over the primitives, there are two things missing. One is support for socket options and one support for security stuff in file access. Both are not hard to add but can be painful if missing.
Fair enough; I plan to implement those things for version 2.
...it'd be nice to have a version that doesn't put 82 classes into "reorganize me" - makes it really hard to grok this stuff for someone who has no idea how to reorganize flow ;-)
Of course. This is an artifact of the way class categorization got, er, attacked in Squeak 3.3 (in which I was working for quite a while), before there was a stable 3.2. I think it was reasonable to leave things this way for an alpha release (the current Flow release is 2 alpha). I had planned proper categorization for 2 alpha 2. It'll be easier than I thought, now that categorization support has emerged unchanged. :)
Secondly, I am scared to death about the issue of backward compatibility.
I think the best thing I can say to assuage such fear is that I've already done "complete" installations of Flow into Squeak, in which I removed *all* the old stream and socket (and file!) support, in three days (24 hours' work), without incident.
Nevertheless...
I urge you to consider leaving the current streams and sockets basically intact and rather have flow be an _alternative_ to instead of a _replacement_ of the current classes.
Yeah, that's why I currently distribute Flow that way. After installing Flow, all Flow features and all old features work concurrently.
From what I can see there are only very few places affected where global names collide and if that's the entire problem someone ought to be able to come up with a few alternative names.
Actually, the current Flow release does this. Anywhere there's an existing class with a name I want to use, I rename it with "Old" prepended (e.g., "Stream" becomes "OldStream"). I also fixed all references to those classes (as of 3.2 final).
So...
With the current version of flow you will instantly break any code that - for example - uses "FileStream readOnlyFileNamed: 'foo.bar'" or "Socket newTCP" (and believe me, there are plenty of those in Croquet...
That's only true, possibly, for post-3.2final code; in the current release of Flow I've fixed all appropriate class references as of 3.2 final. E.g., (FileStream readOnlyFileNamed: 'foo.bar') became (OldFileStream readOnlyFileNamed: 'foo.bar'), and (Socket newTCP) became (OldSocket newTCP).
The new internal stream classes support the protocol of the old ones completely. The other "name-stealing" ones (e.g., FileStream and Socket) do this to a lesser degree. I decided it was easy enough to just fix all the class references.
...if you need another instructive example just try to install flow two times ;-)
That's not a fair example. :) There's some very appropriate but one-time magic in the Flow installer. Do you really need this capability? It can be done.
The entire situation here strikes me of being amazingly similar to the backward compatibility issues with modules - in both cases fundamental notions are being replaced instead of augmented. Let's try hard not to repeat this debacle.
I think this situation is very different, since the new and old stuff runs at the same time. We can be as gradual as we like in adapting current applications to the new framework. Also note that, as has been the case since Flow 1.2 (August 2000) no new VM is required (the new primitives are in a plugin). For me, this is all in marked contrast to the 3.3 modules experience; for example, class categories and pool dictionaries just broke. :) (This is not an invitation to reopen a discussion on modules. ;)
I'd be interested in finding out how feasable it is to implement the flow socket primitives in a non-threaded way. On PS/2 for example, or any *nix with broken pthreads. The code looks like it is relying heavily on a threaded implementation.
Yes, it is, and I think that's a very good thing, from simplicity and performance standpoints (see http://netjam.org/flow/advocacy.html). As I've said before, I don't think it's worth compromising on either of these points for the sake of supporting broken host platforms. I think it'd be better to support those platforms by augmenting Flow's current implementation. It seems quite feasible to me, by using Smalltalk threads that simulate some of what the host threads do in the current Flow implementation (not entirely unlike what happens in the mainstream Squeak system currently).
However, I do consider supporting the thread-capable platforms (e.g., win32, *BSD, Linux, Macintosh), which are overwhelmingly more widely used than the others (e.g., PS/2, Acorn), a much higher priority. Flow works on those popular platforms right now. And I think the work of supporting the thread-impaired platforms now may be rendered moot in the near future. I'm not familiar with the details of PS/2, but I know that there are, for example, people working on pthreads for the Acorn.
thanks,
-C
-- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org/resume Smalltalkers do: [:it | All with: Class, (And love: it)] _______________________________________________ Squeakfoundation mailing list Squeakfoundation@lists.squeakfoundation.org http://lists.squeakfoundation.org/listinfo/squeakfoundation
Daniel Vainsencher danielv@netvision.net.il wrote:
Let me summarize the points so far -
[Snipped a GOOD summary]
I still think we are maybe attacking this from the wrong angle. I like Colin's, Avi's and Cees' postings - why do we have to make these decisions now? Again - there should not be anything going *into* the image - we all agreed on that. Stuff should be going *out* from the image.
So make the Flow package able to coexist with the old stuff.
And not by changing the old stuff - that is again IMHO a very image centric approach. Just because all uses in the image have been fixed does NOT mean all breakage has been fixed. Just take Croquet, Seaside whatever is out there. There may even be secret proprietary code out there breaking.
Obviously this is doable and the arguments for it:
- No old code will break - Flow can coexist with the old code in the image
...seems IMHO to clearly outweigh the arguments against it. People can then easily choose which package to rely on.
This would make Flow a dependable package. Also break out the current networking into a package and have that be blessed to start with. Then if people in time migrate over to using Flow and Flow gets "battle tested" etc. - *then* we can start looking at blessing Flow instead of the old package. But that decision is a long way into the future.
regards, Göran
PS. Or am I missing something?
Goran,
PS. Or am I missing something?
No. Those are my points exactly. Thanks for summarizing.
Cheers, - Andreas
Daniel Vainsencher danielv@netvision.net.il wrote:
Let me summarize the points so far -
[Snipped a GOOD summary]
I still think we are maybe attacking this from the wrong angle. I like Colin's, Avi's and Cees' postings - why do we have to make these decisions now? Again - there should not be anything going *into* the image - we all agreed on that. Stuff should be going *out* from the image.
So make the Flow package able to coexist with the old stuff.
And not by changing the old stuff - that is again IMHO a very image centric approach. Just because all uses in the image have been fixed does NOT mean all breakage has been fixed. Just take Croquet, Seaside whatever is out there. There may even be secret proprietary code out there breaking.
Obviously this is doable and the arguments for it:
- No old code will break
- Flow can coexist with the old code in the image
...seems IMHO to clearly outweigh the arguments against it. People can then easily choose which package to rely on.
This would make Flow a dependable package. Also break out the current networking into a package and have that be blessed to start with. Then if people in time migrate over to using Flow and Flow gets "battle tested" etc. - *then* we can start looking at blessing Flow instead of the old package. But that decision is a long way into the future.
regards, Göran
PS. Or am I missing something?
Squeakfoundation mailing list Squeakfoundation@lists.squeakfoundation.org http://lists.squeakfoundation.org/listinfo/squeakfoundation
Hi Andreas--
Göran writes:
Or am I missing something?
You respond:
No. Those are my points exactly. Thanks for summarizing.
Not so fast... :) I'd still like a response to my previous message to you (http://lists.squeakfoundation.org/pipermail/squeakfoundation/2002-November/0..., and included below).
It still seems to me that you've missed significant points in what I wrote.
thanks,
-C
***
You fixed all references... you knew about.
That's not the end of the process. :) I think you're missing my intent that those changes be temporary. The full integration process would go like this:
1. Change some of the original class names ("Stream" becomes "OldStream", etc.) 2. Fix references to those classes. Everything keeps working during the rest of the integration process. 3. Install new classes (several of which use the original names, and provide backward-compatible protocol). 4. Incrementally change the above references to use the original class names again (references to "OldStream" become references to "Stream", which is the new but backward-compatible implementation, etc.) 5. The users of internal streams will see no cumulative change, since the Flow support for internal streams provides complete backward compatibility. The users of other support (e.g., sockets) can decide whether to use the new or old framework. 6. As we wish, we can incrementally remove the backward compatibility and old framework support (perhaps over a very long period of time).
The Flow 2a release implements steps 1 to 3, I'm proposing to finish the process. Given this process, I don't think your analogies to 3.3 modules or Croquet (or Microsoft :) are valid.
...why not just keep the two hierarchies just completely disjunct?
Because, e.g., I think what I call "Socket" should be called "Socket". :) As I see it, there is a genuine ontological conflict between the current mainstream implementation and mine. And after renaming the mainstream classes (which, for the reasons above, I think is no big deal), they *are* disjunct. :)
More seriously, I think it's very important to have the right names for things. Sure, one normally makes a file stream by evaluating an expression which doesn't refer to the actual instantiating class, but I think the system is easier to use if people run into appropriately evocative names for things in the system browsers, and this ease of use is critical.
Well if flow can't support those [thread-less] platforms at all...
Um, I just said I think it can...
...just for the record - I am still waiting for benchmarks which actually indicate that flow makes any noticable difference in speed ;-)
The reality of spare-time programming being what it is, if you want *me* to provide these benchmarks, you'll have to be prepared to wait until there are real applications I care about. :)
Please remember that my primary motivation in designing Flow the way I did was programming simplicity, not performance. It so happens that I also think that it will also be faster. But I don't start profiling things until they perform meaningful work. I haven't made time to write applications yet, mostly because the ones I want rely on yet more infrastructure that has little to do with streaming (how to store mail, how to represent system changes, etc.). (I don't mean this as a dig against, e.g., Celeste or Scamper, although I must admit I don't use them.)
But while you're waiting, please feel free to send tests that *you* would consider meaningful. :)
It seems quite feasible to me, by using Smalltalk threads that simulate some of what the host threads do in the current Flow implementation (not entirely unlike what happens in the mainstream Squeak system currently).
This isn't quite what I meant.
I know, but I stand by my answer. :)
The question was more like: Is it possible to write an implementation of the socket primitives that adheres exactly to the specification of the primitives which may not use a threaded implementation?
Sure, but I'd rather keep that simulation at the more malleable object-memory level.
Typically, it's best to stay compatible at the primitive level so that if there's a certain behavior expected by the ST code it should be able to rely on it.
Sure again, but Flow's design allows that behavior to be provided either by "auxilliary primitives" (which aren't called directly by Smalltalk code) or by yet more Smalltalk code. I'd rather go the Smalltalk route.
At the very least I wouldn't expect people running on HP-UX (which IIRC, has horribly broken pthreads) to expect that they have to "reprogram flow" in order to get something to run which works perfectly well on Linux.
Right, they wouldn't. Squeak would know what platform it's on, and act accordingly.
By the way, HP-UX has had perfectly fine pthreads since version 11.0 (two years ago). (Before that, it had DCE threads, which were indeed problematic.)
...there are lots of platforms that Squeak can run on and some of them already have and more will have networking capabilities [but not properly functioning threads]. And I don't want to arbitrarily limit those...
As I said, I think we can accomodate those platforms too.
thanks,
-C
***
-- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org/resume Smalltalkers do: [:it | All with: Class, (And love: it)]
Hi Craig,
No. Those are my points exactly. Thanks for summarizing.
Not so fast... :) I'd still like a response to my previous message to you
(http://lists.squeakfoundation.org/pipermail/squeakfoundation/2002-November/0..., and included below).
I wanted to see what others think.
You fixed all references... you knew about.
That's not the end of the process. :) I think you're missing my intent that those changes be temporary. The full integration process would go like this:
None of it changes the fact that if I load flow every last bit of the code that I wrote and that uses sockets or files will immediately stop working.
I might also add that I see Mikes rewrite in a different way since even though he's claiming that
There is no such thing when you replace seriously broken code. Period. Which means that I conciously made the decision to not care if my network rewrite breaks existing code (that is code currently not in the image).
the chances of breaking code from what he did are much smaller - all of the interfaces which used to be there in class Socket are still there so if someone really needs a quick way of porting things forward, this is doable by basically copying some code from the existing classes, or by providing a compatibility package - the proposal of (for example) returning strings from the exceptions raised is certainly not the Right Thing to do, yet it does provide an "easy way out" for people who just need their stuff working.
...why not just keep the two hierarchies just completely disjunct?
Because, e.g., I think what I call "Socket" should be called "Socket". :) As I see it, there is a genuine ontological conflict between the current mainstream implementation and mine. And after renaming the mainstream classes (which, for the reasons above, I think is no big deal), they *are* disjunct. :)
I tried to avoid the term "name space" in my original message because I agree with you here. It is important that things have the right names and with name spaces we'd be able of achieving this. I am still hoping that we'll get them before too long (since we're working on some of these aspects in Croquet), yet, up to the point where we have them, we need a workable solution. And given the current usage patterns it appears to me that it'd be unwise if we rename the things that are already there.
Well if flow can't support those [thread-less] platforms at all...
Um, I just said I think it can...
Sorry typo - this should have read "if flow COULD not support them".
But while you're waiting, please feel free to send tests that *you* would consider meaningful. :)
Hey, _you_ were claiming that flow would be more efficient (read the first sentence of your own comparison ;)
The question was more like: Is it possible to write an implementation of the socket primitives that adheres exactly to the specification of the primitives which may not use a threaded implementation?
Sure, but I'd rather keep that simulation at the more malleable object-memory level.
Typically, it's best to stay compatible at the primitive level so that if there's a certain behavior expected by the ST code it should be able to rely on it.
Sure again, but Flow's design allows that behavior to be provided either by "auxilliary primitives" (which aren't called directly by Smalltalk code) or by yet more Smalltalk code. I'd rather go the Smalltalk route.
But effectively this means a different set of primitives doesn't it? I'm not sure I understand correctly what you're trying to say here. My question was: Can I implement the primitives without using OS level threads. Your answer sounds like a no, as if you would have to have different primitives and then simulate the former OS-level threading in Squeak. Is this correct?!
By the way, HP-UX has had perfectly fine pthreads since version 11.0 (two years ago). (Before that, it had DCE threads, which were indeed problematic.)
Ah, well, it's been a while since I looked at HP-UX ;-)
Cheers, - Andreas
Hi Göran--
Or am I missing something?
Well, having done refactorings of this magnitude several times before, and having tried "both ways", I think that what I suggest would take a lot less time and energy, yielding better quality.
-C
-- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org/resume Smalltalkers do: [:it | All with: Class, (And love: it)]
Craig Latta craig.latta@netjam.org wrote:
Hi Göran--
Or am I missing something?
Well, having done refactorings of this magnitude several times before, and having tried "both ways", I think that what I suggest would take a lot less time and energy, yielding better quality.
Ok, I have too many things in the air right now, but IIRC you essentially want Flow to enter the base image as a big "refactoring", right?
Why is that better than letting it be a package that people can depend on if they like?
Some reasonable people may consider Flow to be much more than a "refactoring".
regards, Göran
Daniel Vainsencher wrote:
Let me summarize the points so far -
- Backwards compatibility -
There is no such thing when you replace seriously broken code. Period. Which means that I conciously made the decision to not care if my network rewrite breaks existing code (that is code currently not in the image). Having said that we should only break this code once. Meaning whatever rewritten code gets into the update stream (soon?) should define an API that then newer versions should be backwards compatible to.
- Proposals to take the primitive interfaces but not the stream
refactoring
- I can't summarize this because there were no specific
proposal/problems raised (Michael? Goran?)
I have to think more about this and tinker with some code to make a real proposal. It's more of a gut feeling that it shouldn't be too hard to exchange the current primitive layer with the Flow one without breaking other parts of the system. I just don't really know yet.
It would be different for the stream refactoring which is a whole can of worms for itself. Stephen Pair and I had a lengthy discussion on it and we finally agreed that refactoring the stream hierarchy only makes sense using traits. Unless you take up my proposal:
Stream (resource) InternalStream ExternalStream
;-)
5 - Michael, could you please clarify you intent here? I'm not sure whether your comments here are about transition or the desireability of the refactoring itself.
See above... I would like to see the network api and flow exchangeable (not co-existent, that's <censored>). The Stream refactoring should wait IMHO, although some of the flow ideas should be incorporated (NetStreams) to define a new compatible network/external resource API.
Michael
Hi Michael--
...we finally agreed that refactoring the stream hierarchy only makes sense using traits.
I'd be interested in substantiation there... it seems like overkill to me. I think what I did gets along just fine without them.
Unless you take up my proposal:
Stream (resource) InternalStream ExternalStream
That actually doesn't seem substantively different from what I did...
-C
-- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org/resume Smalltalkers do: [:it | All with: Class, (And love: it)]
squeakfoundation@lists.squeakfoundation.org
-
Andreas.Raab＠gmx.de -
Craig Latta -
Daniel Vainsencher -
goran.hultgren＠bluefish.se -
Michael Rueger