Nathanael,

> I didn't have the time for any debugging yet, so I don't know what is
> broken...

I actually mentioned the problem a while ago and it's not trivial. Here's
the relevant part of that message (quoted in full below):

> There's a single problem left at this point which is that you 
> can't recompile the entire class hierarchy - it will break at 
> Metaclass. This for two reasons. For one, Metaclass class is 
> an instance of Metaclass (e.g., "Metaclass class isMemberOf: 
> Metaclass" yields true) which means that when we convert the 
> _instances_ of Metaclass we also convert the _class_ of 
> Metaclass with possibly fatal consequences (and I have _no_ 
> idea why this has ever worked...) I think that just excluding 
> Metaclass' class from the update process should be enough 
> here but I'm not certain. The second problem with Metaclass 
> is that when you reshape Metaclass (e.g., creating 
> NewMetaclass) the meta class created for it (NewMetaclass 
> class) needs to be an instance of NewMetaclass for the same 
> reason. Not doing so will leave Metaclass with a broken 
> instance layout since after reshaping Metaclass its class 
> (Metaclass class) will still have the shape of OldMetaclass 
> (so to speak) while having the class pointer to NewMetaclass 
> (due to #become). I think that some creative use of a 
> "multiple new" (e.g., create NewMetaclass class <instance of: 
> OldMetaclass>, then create NewMetaclass <instanceOf: 
> NewMetaclass class>, then create NewMetaclass2 
> class<instanceOf: NewMetaclass>, then create <NewMetaclass2>) 
> should do the trick since the only thing that matters is that 
> the shape of both NewMetaclass and NewMetaclass class is 
> correct (everything else will fixed by #become later). 
> Alternatively, a well-placed #primitiveChangeClassTo: might 
> work miracles.
>
> If you want to twist your mind around this problem, please do!

Cheers,
  - Andreas

> -----Original Message-----
> From: Andreas Raab [mailto:Andreas.Raab at gmx.de] 
> Sent: Saturday, September 21, 2002 2:19 AM
> To: 'Nathanael Schärli'; 'schaerli at iam.unibe.ch'; 
> 'Dan at SqueakLand.org'; 'ian.piumarta at inria.fr'; 
> 'John.Maloney at SqueakLand.org'; 'Ted at SqueakLand.org'; 'Bill Cole'
> Subject: RE: More ClassBuilder trouble
> 
> 
> Hi Guys,
> 
> I actually got into thinking mode with the problem and it 
> turns out we've all been overlooking a couple of things for 
> way too long. First of all, some observations:
> 
> * Class and Metaclass reshapes: I think that not updating the 
> meta-classes when reshaping non-meta classes was a real bug. 
> When we create a new non-meta class we already create a 
> meta-class for it, and not updating the meta-class where we 
> update the non-meta class just brings trouble. The worst 
> thing is that it partly breaks the class/meta class 
> relationship since the becomeForward: of the old into the new 
> class will make the old meta class point to the new non-meta 
> class, and the old non-meta class an instance of the new meta 
> class. Clearly bad stuff, but in the grand scheme of things 
> actually just a minor quibble ;-)
> 
> * Obsolete subclasses: What are they?! They are classes that 
> still have instances when somebody wanted to remove a class 
> from the system. Since they may have instances we need to 
> record them just in case their superclass gets reshaped at 
> some point. However, from the point of ClassBuilder they are 
> just subclasses. There is nothing special about them they 
> have _exactly_ the same semantics as regular subclasses have. 
> So far so good.
> 
> Those of you who have been around ClassBuilder will know 
> quite a number of places where the obsolete subclasses were 
> managed. When I got into thinking mode one of my first 
> observations was that this is a pretty pointless exercise. 
> After all, ClassBuilder is *reshaping* classes, not removing 
> them, so from the point of updating an old into a new class 
> (using become) it simply _cannot_ introduce an obsolete 
> subclass. So why bother recording it?!
> 
> * Obsolete instances: We go to endless length in order to 
> make sure that there are no references to any old instances 
> during the reshape process. E.g., we run stuff 
> unpreemptively, use become on the instances, and, just at the 
> point where we can be _absolutely_ certain there are no more 
> references to it, we start remapping those (known to have no 
> references) instances into temp instances, use 
> #primitiveChangeClassTo: and dance all around the fact that a 
> nice fat GC would solve the entire problem.
> 
> * 'Super send' pointers: We 'fix' all all of the methods 
> while recompiling a class which may send to super in order to 
> get super-association right, we dance up and down with this 
> stuff, although we will later do a #become: of the old into 
> the new class which means that all of this 'fixing' is just a 
> plain pointless exercise. Since after the #becomeForward: 
> there can be no references to the old class whatsoever, and 
> any of those 'fixed' association will point to the new class 
> anyways. Which, of course, they would do equally well if they 
> would've pointed to the old class - hey that's why we use 
> #become, don't we?! ;-)
> 
> After I got the above straight down, I started to rip out 
> code and try to think what it is we *really* need to do. When 
> we mutate a class from old to new then we first mutate all of 
> the old class' subclasses (obsolete or not) into subclasses 
> of the new one. Fine. Let's assume we're going to do this in 
> a way that the old ones simply no longer exists. Then we take 
> the old class and want to update the class, its instances and 
> possibly its metaclass into the new class, instances and meta 
> class. So we need to do an #updateInstancesFrom: which 
> converts the instances and will guarantee that there's no 
> lasting pointer to any of the old instances (this is a 
> critical invariant). At this point, all we need to do is to 
> take the old class and meta class and just become them into 
> the new class and meta class.
> 
> That's it. That's all we ever need. After we've done the 
> final #become there are neither any pointers to the old 
> instances (known from #updateInstancesFrom:) nor can there be 
> any to the old class or meta class (since we just 
> forward-became them). So if we do a nice fat GC after 
> converting everything we are just done. Period. No need to do 
> any of that complex stuff we're doing right now. Neither in 
> reshaping nor in preparation of the new classes. Since we 
> know we're going to #become them later on we can just create 
> new classes, recompile them, update them.
> 
> So I thought, and I tried and instantly broke my system ;-) 
> Then I spent a day figuring out "what is wrong with #become" 
> (since I was absolutely certain that the above _must_ work) 
> ... only to find that I had stupidly broken the 
> superclass/subclass invariant which we work so hard to 
> preserve even for obsolete subclasses. Sigh. But as I said, 
> it _must_ work and it does, if you preserve the critical invariants.
> 
> Then I went on to remove the GC overhead since although the 
> above _greatly_ simplifies the entire reshape process 
> (removing four and heavily simplifying three more methods) it 
> is somewhat slower due to the extra GC after the #become. I 
> found two more possibilities to solve that problem besides 
> just that fat GC after each become: We could run the entire 
> update of the class hierarchy within a non-preemptive block. 
> Since ClassBuilder will not do any "bad stuff" it is known 
> that noone else can get a pointer to anything obsolete. But 
> still, I didn't like this very much since reshaping the 
> entire class hierarchy may run for minutes (even hours on 
> slower machines) and not being able to interact, even when 
> you absolutely have to, is kinda bad. So I went for the third 
> solution which is to assume that generally no other process 
> will do "bad stuff at high priority". The basic assumption is 
> that no process running above normal priority will use 
> reflection techniques to pick up, for example, #allInstances 
> or somesuch. Then, we can just put a single well-placed GC at 
> the end of reshaping the entire class hierarchy and everything's done.
> 
> [Note: If you think that the above 'heuristics' is not good 
> enough, we can still switch to any of the other two solution. 
> Or use a method that would avoid breaking the class pointers 
> - see below]
> 
> There's one exception of course. If we reshape the entire 
> class hierarchy we may have obsolete Metaclasses when we come 
> across Metaclass (since both may be in the tree, the 
> metaclasses as well as Metaclass itself). This problem is 
> easy to solve by adding "two extra GCs" for the single case 
> of reshaping Metaclass itself. That was a reasonable solution 
> for me, since how often do we reshape Metaclass in practice?! ;-)
> 
> Okay, I have attached the changes for you to play with. These 
> changes will work for everything you might try (one exception 
> below) IFF your system is in a consistent state to begin with 
> (this is important - we've done so many mistakes that there's 
> a good chance it may not). The changes simply must work - the 
> code is trivial (compared to what was there before almost 
> non-existent) and there's nothing that even eventually could 
> go wrong if you "play by the heuristics".
> 
> There's a single problem left at this point which is that you 
> can't recompile the entire class hierarchy - it will break at 
> Metaclass. This for two reasons. For one, Metaclass class is 
> an instance of Metaclass (e.g., "Metaclass class isMemberOf: 
> Metaclass" yields true) which means that when we convert the 
> _instances_ of Metaclass we also convert the _class_ of 
> Metaclass with possibly fatal consequences (and I have _no_ 
> idea why this has ever worked...) I think that just excluding 
> Metaclass' class from the update process should be enough 
> here but I'm not certain. The second problem with Metaclass 
> is that when you reshape Metaclass (e.g., creating 
> NewMetaclass) the meta class created for it (NewMetaclass 
> class) needs to be an instance of NewMetaclass for the same 
> reason. Not doing so will leave Metaclass with a broken 
> instance layout since after reshaping Metaclass its class 
> (Metaclass class) will still have the shape of OldMetaclass 
> (so to speak) while having the class pointer to NewMetaclass 
> (due to #become). I think that some creative use of a 
> "multiple new" (e.g., create NewMetaclass class <instance of: 
> OldMetaclass>, then create NewMetaclass <instanceOf: 
> NewMetaclass class>, then create NewMetaclass2 
> class<instanceOf: NewMetaclass>, then create <NewMetaclass2>) 
> should do the trick since the only thing that matters is that 
> the shape of both NewMetaclass and NewMetaclass class is 
> correct (everything else will fixed by #become later). 
> Alternatively, a well-placed #primitiveChangeClassTo: might 
> work miracles.
> 
> If you want to twist your mind around this problem, please do!
> 
> Ah, yes, and two more related notes: The issue with 
> MethodChangeRecord is almost non-existent after these 
> changes. It came mostly from not updating the meta-classes. 
> E.g., CompiledMethods may refer to any classes but will now 
> always refer to the "current" class if that class exists. So 
> only truly obsolete classes (e.g., those that were removed at 
> some point) could be held on by MCR's compiled method. Which 
> is good, since it means that the obsolete subclasses now 
> _really_ record only truly obsolete subclasses again.
> 
> The other note is about #become: changing class pointers. It 
> would still be _helpful_ if #become wouldn't change class 
> pointers since it would allow us not to worry about the need 
> for GC at all (the reason why we need the GC is that all of 
> the old class pointers will be broken after become). But it 
> is not strictly required. And, it would be _equally_ helpful 
> to make #becomeForward: automatically free the the forwarded 
> objects, e.g., free the chunks that they used. This might be 
> a simpler thing to do, equally effective (no broken 
> class-pointers; and we could use one-way become on instances, 
> too) and logically, after you're done with a becomeForward: 
> the forwarded object is garbage and could be "implicitly 
> collected" here.
> 
> Cheers,
>   - Andreas
>