[Vm-dev] VM Maker: VMMaker.oscog-cb.2376.mcz

Clément Bera bera.clement at gmail.com
Thu Apr 26 12:17:07 UTC 2018


Eliot, if you have any idea why there's no look-up in compactor classes and
how to fix it... I'm pretty sure it's just a class-side method
somewhere...  So I can remove all this duplication.

Anyway, I compiled VMs with PigCompactor, PlanningCompactor,
SelectiveCompactor & Sweep and they all seemed to work.

On Thu, Apr 26, 2018 at 2:12 PM, <commits at source.squeak.org> wrote:

>
> ClementBera uploaded a new version of VMMaker to project VM Maker:
> http://source.squeak.org/VMMaker/VMMaker.oscog-cb.2376.mcz
>
> ==================== Summary ====================
>
> Name: VMMaker.oscog-cb.2376
> Author: cb
> Time: 26 April 2018, 2:12:30.237019 pm
> UUID: bf5a9a98-571a-4a0e-985e-9e734854535a
> Ancestors: VMMaker.oscog-cb.2375
>
> Finally figured out that Slang is not able to do look-ups in compactor
> classes for some reason, hence only planning compactor was the only one
> compiling correctly since it defined everything inside...
>
> So I duplicated printTheBogon and postSwizzleAction to all subclasses.
>
> So I removed inheritance of SpurSelectiveCompactor to SpurSweeper, and
> duplicated used method.
>
> Eliot, if you have any idea why there's no look-up and how to fix it...
> I'm pretty sure it's just a class-side method somewhere...  So I can remove
> all this duplication.
>
> =============== Diff against VMMaker.oscog-cb.2375 ===============
>
> Item was changed:
>   CogClass subclass: #SpurCompactor
>         instanceVariableNames: 'manager scavenger coInterpreter'
>         classVariableNames: ''
>         poolDictionaries: 'SpurMemoryManagementConstants VMBasicConstants
> VMSpurObjectRepresentationConstants'
>         category: 'VMMaker-SpurMemoryManager'!
>
> + !SpurCompactor commentStamp: 'cb 4/26/2018 13:53' prior: 0!
> - !SpurCompactor commentStamp: 'cb 10/3/2017 10:49' prior: 0!
>   Common superclass of all compactors to define apis and simulation
> variables.
>
> + IMPORTANT: This defines APIs only, subclassing is prohibited on
> auxiliary classes (Slang compiler won't find inherited methods).
> +
>   The full GC in Spur is split in two, the marking phase and the compactor
> phase. The subclasses of SpurCompactor are implementations of the second
> phase, so they are called once the marking phase is finished. SpurCompactor
> is reponsible for:
>   - freeing unmarked objects
>   - compacting the live old space objects (though each subclass define
> what it does, some spurCompactor may not compact)
>   - unmarking all objects remaining live
>   - updating oops directly referred by the VM when they are moved
> (remapObj:/shouldRemapObj: thingy)
>
>   The main apis are the following:
>   - biasForGC/biasForSnapshot: tells the compactor if the GC is performed
> for snapshots or not, in general we want to compact more aggressively for
> snapshots to avoid saving large files with many unused space.
>   - compact: main API, should free the unmarked object, unmark the objects
> remaining live and potentially compact the heap
>   - remapObj:/shouldRemapObj: => Not really sure what this does, it seems
> it has to do with updating oops directly referred by the VM when they are
> moved.
>
>
>   Instance Variables
>         coInterpreter:                          <StackInterpreter>
>         compactedCopySpace:             <SpurNewSpaceSpace>
>         manager:
> <SpurMemoryManager>!
>
> Item was changed:
>   ----- Method: SpurCompactor>>postSwizzleAction (in category 'api') -----
>   postSwizzleAction
> +       self subclassResponsibility!
> -       "Do nothing by default"!
>
> Item was changed:
>   ----- Method: SpurCompactor>>printTheBogons: (in category 'debugging')
> -----
>   printTheBogons: aBogon
> +       self subclassResponsibility!
> -       <inline: true>
> -       coInterpreter
> -               print: 'bogon '; printHexnp: aBogon; cr!
>
> Item was added:
> + ----- Method: SpurPigCompactor>>postSwizzleAction (in category
> 'compatibility') -----
> + postSwizzleAction
> +       "do nothing"
> +       !
>
> Item was added:
> + ----- Method: SpurPigCompactor>>printTheBogons: (in category 'debug
> printing') -----
> + printTheBogons: aBogon
> +       <inline: true>
> +       coInterpreter
> +               print: 'bogon '; printHexnp: aBogon; cr!
>
> Item was added:
> + ----- Method: SpurPlanningCompactor>>postSwizzleAction (in category
> 'compatibility') -----
> + postSwizzleAction
> +       "do nothing"
> +       !
>
> Item was changed:
> + SpurCompactor subclass: #SpurSelectiveCompactor
> +       instanceVariableNames: 'biasForGC segmentToFill'
> - SpurSweeper subclass: #SpurSelectiveCompactor
> -       instanceVariableNames: 'segmentToFill'
>         classVariableNames: 'MaxOccupationForCompaction'
>         poolDictionaries: ''
>         category: 'VMMaker-SpurMemoryManager'!
>
> + !SpurSelectiveCompactor commentStamp: 'cb 4/26/2018 13:59' prior: 0!
> - !SpurSelectiveCompactor commentStamp: 'cb 4/26/2018 12:53' prior: 0!
>   SpurSelectiveCompactor compacts memory by selecting the memory segments
> with the most free space and compacting only those, to limit fragmentation
> while being really quick to perform. The algorithm is fast mostly because
> it does not update pointers: they are updated lazily during the next
> marking phase, so there is no need to read the fields of objects in other
> memory segments that the one compacted.
>
>   The algorithm works as follow. First, a global sweep pass iterates over
> the memory linearly, changing unmarked objects to free chunks and
> concatenating free chunks. During the global sweep phase, the segments of
> the heap are analysed to determine the percentage of occupation. Second,
> the least occupied segments are compacted by copying the remaining live
> objects into an entirely free segment, called regionToFill (we detail later
> in the paragraph where regionToFill comes from), changing their values to
> forwarding objects and marking the free chunks as unavailable (removed from
> free list and marked as data objects). Third, the next marking phase
> removes all forwarders. Fourth, at the beginning of the next compaction
> phase the compacted segments from the previous GC can be entirely marked as
> free space (No need to check anything inside, there were only forwarders
> and trash data). One of the compacted segment is then selected as the
> segmentToFill, others are just marked as free chunks.
>
>
>   The compaction is effectively partial, compacting only the most critical
> segments of the heap to limit fragmentation. Compaction time is crazy low,
> since a low number of objects are moved and pointer updated is lazily done
> during the next marking phase, while still preventing memory fragmentation.
>
>   Now this works well when biasForGC is true, but when performing a
> snapshot, the compactor is just total crap (we need to figure out a
> solution).
>
> + IMPORTANT: I could not figure out to make inheritance work so I copied
> methods from SpurSweeper here.
> +
>   segmentToFill <SegInfo> the segment that will be filled through the
> copying algorithm
>
>   !
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>assertNoSegmentBeingCompacted (in
> category 'compaction') -----
>   assertNoSegmentBeingCompacted
> -
>         "Assertion only - no segment is being claimed at this point"
> +       | segInfo |
> +       <var: 'segInfo' type: #'SpurSegmentInfo *'>
>         0 to: manager numSegments - 1 do:
> +               [:i|
> -               [:i| | segInfo |
>                  segInfo := self addressOf: (manager segmentManager
> segments at: i).
>                  self deny: (self isSegmentBeingCompacted: segInfo)].
>         !
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>biasForGC (in category 'api')
> -----
> + biasForGC
> +       biasForGC := true.!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>biasForSnapshot (in category
> 'api') -----
> + biasForSnapshot
> +       biasForGC := false.!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>bulkFreeChunkFrom: (in category
> 'sweep phase - copied from SpurSweeper') -----
> + bulkFreeChunkFrom: objOop
> +       "ObjOop is either a freeChunk or an object to free, always in old
> space. The old space entity before objOop is necessarily a marked object.
> +        Attempts to free as many byte from objOop, looking ahead for
> multiple freechunks / objects to free in a row"
> +       | bytes start next currentObj |
> +
> +       "Avoids pathological case, not point in dealing with non-mergeable
> free chunks, we would remove them and re-add them to the free list."
> +       (self isSingleFreeObject: objOop) ifTrue: [^0].
> +
> +       "We free unmarked objects and freechunks next to each others and
> merge them at the same time"
> +       start := manager startOfObject: objOop.
> +       currentObj := objOop.
> +       bytes := 0.
> +       [bytes := bytes + (manager bytesInObject: currentObj).
> +       self freeEntity: currentObj.
> +       next := manager objectStartingAt: start + bytes.
> +       self canUseNextEntityAsFreeSpace: next]
> +               whileTrue: [currentObj := next].
> +
> +       manager addFreeChunkWithBytes: bytes at: start.
> +
> +       ^ next!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>canUseAsFreeSpace: (in category
> 'sweep phase - copied from SpurSweeper') -----
> + canUseAsFreeSpace: objOop
> +       <inline: true>
> +       ^ (manager isFreeObject: objOop) or: [(manager isMarked: objOop)
> not]!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>canUseNextEntityAsFreeSpace: (in
> category 'sweep phase - copied from SpurSweeper') -----
> + canUseNextEntityAsFreeSpace: next
> +       <inline: true>
> +       ^ (manager oop: next isLessThan: manager endOfMemory) and: [self
> canUseAsFreeSpace: next]!
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>compactSegmentsToCompact (in
> category 'compaction') -----
>   compactSegmentsToCompact
>         "Forwards all objects in segments to compact and removes their
> freechunks"
> +       | segInfo fillStart |
> +       <var: 'segInfo' type: #'SpurSegmentInfo *'>
> -       | fillStart |
>         fillStart := segmentToFill segStart.
>
>          "Removes initial free chunk in segment to fill... (Segment is
> entirely free)"
>         manager detachFreeObject: (manager objectStartingAt: fillStart).
>
>          "Compact each segment to compact..."
>         0 to: manager numSegments - 1 do:
> +               [:i|
> -               [:i| | segInfo |
>                  segInfo := self addressOf: (manager segmentManager
> segments at: i).
>                 (self isSegmentBeingCompacted: segInfo)
>                         ifTrue: [fillStart := self compactSegment: segInfo
> freeStart: fillStart ]].
>
>          "Final free chunk in segment to fill..."
>          manager
>                 addFreeChunkWithBytes: segmentToFill segSize - manager
> bridgeSize + segmentToFill segStart - fillStart
>                 at: fillStart.
>
>          "Follow stack zone and caches..."
>         self postForwardingAction
>         !
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>computeSegmentsToCompact (in
> category 'compaction') -----
>   computeSegmentsToCompact
>         "Compute segments to claim: least occupied.
>          Answers true if at least 1 segment is being compacted."
> -
>         | canStillClaim aboutToClaim aboutToClaimSegment
> atLeastOneSegmentToCompact |
> +       <var: 'aboutToClaimSegment' type: #'SpurSegmentInfo *'>
>         atLeastOneSegmentToCompact := false.
>         aboutToClaimSegment := self findNextSegmentToCompact.
>         "Segment to fill is one of the segment compacted last GC.
>          If no segment were compacted last GC, and that there is
>          at least one segment to compact, allocate a new one."
>         aboutToClaimSegment ifNil: [^false].
>         segmentToFill ifNil: [self findOrAllocateSegmentToFill].
>         canStillClaim := segmentToFill segSize - manager bridgeSize.
>         [aboutToClaimSegment ifNil: [^atLeastOneSegmentToCompact].
>          aboutToClaim := aboutToClaimSegment segSize - manager bridgeSize
> * ((self occupationOf: aboutToClaimSegment) + 1) // 255. "+1 to round up,
> this is approx"
>          aboutToClaim < canStillClaim ] whileTrue:
>                 [self markSegmentAsBeingCompacted: aboutToClaimSegment.
>                  atLeastOneSegmentToCompact := true.
>                  canStillClaim := canStillClaim - aboutToClaim.
>                  aboutToClaimSegment := self findNextSegmentToCompact].
>         ^atLeastOneSegmentToCompact!
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>findAndSetSegmentToFill (in
> category 'freeing') -----
>   findAndSetSegmentToFill
> +       | segInfo firstEntity |
> +       <var: 'segInfo' type: #'SpurSegmentInfo *'>
>         0 to: manager numSegments - 1 do:
> +               [:i|
> -               [:i| | segInfo firstEntity |
>                  segInfo := self addressOf: (manager segmentManager
> segments at: i).
>                  firstEntity := manager objectStartingAt: segInfo segStart.
>                  ((manager isFreeObject: firstEntity) and: [(manager
> objectAfter: firstEntity limit: manager endOfMemory) = (manager
> segmentManager bridgeFor: segInfo)])
>                         ifTrue: [segmentToFill := segInfo. ^0]].
>         !
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>findNextSegmentToCompact (in
> category 'compaction') -----
>   findNextSegmentToCompact
>         "Answers the next segment to compact or nil if none.
>           The next segment to compact:
>          - cannot be segment 0 (Segment 0 has specific objects
>           (nil, true, etc.) and special size computed at start-up
>           that we don't want to deal with)
>          - cannot be be a segment already being compacted.
>          - cannot contain pinned object (since we're in a copying GC)
>          - cannot be entirely empty (no need to block that empty segment
> until next marking phase)
>          - cannot have a high occupation rate (>
> MaxOccupationForCompaction)"
> +       | leastOccupied leastOccupiedSegment tempOccupied segInfo |
> +       <var: 'segInfo' type: #'SpurSegmentInfo *'>
> -       | leastOccupied leastOccupiedSegment tempOccupied |
>         leastOccupied := 255.
>         1 to: manager numSegments - 1 do:
> +               [:i|
> -               [:i| | segInfo |
>                  segInfo := self addressOf: (manager segmentManager
> segments at: i).
>                  ((self isSegmentBeingCompacted: segInfo) or: [segInfo
> containsPinned or: [manager segmentManager isEmptySegment: segInfo] ])
>                         ifFalse:
>                                 [(tempOccupied := self occupationOf:
> segInfo) <= leastOccupied
>                                         ifTrue: [ leastOccupied :=
> tempOccupied.
>
>  leastOccupiedSegment := segInfo ]]].
>         leastOccupied > MaxOccupationForCompaction ifTrue: [^nil].
>         ^ leastOccupiedSegment!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>freeEntity: (in category 'sweep
> phase - copied from SpurSweeper') -----
> + freeEntity: entity
> +       <inline: true>
> +       (manager isFreeObject: entity)
> +               ifFalse: "Freed old space objects are removed from
> remembered table"
> +                       [(manager isRemembered: entity) ifTrue:
> +                               [scavenger forgetObject: entity]]
> +               ifTrue:  "Merged old space free chunks are removed from
> free list"
> +                       [manager detachFreeObject: entity]
> +       !
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>freePastSegmentsAndSetSegmentToFill
> (in category 'freeing') -----
>   freePastSegmentsAndSetSegmentToFill
>         "The first segment being claimed met becomes the segmentToFill.
> The others are just freed."
> +       | segInfo |
> +       <var: 'segInfo' type: #'SpurSegmentInfo *'>
>         segmentToFill := nil.
>         0 to: manager numSegments - 1 do:
> +               [:i|
> -               [:i| | segInfo |
>                  segInfo := self addressOf: (manager segmentManager
> segments at: i).
>                  (self isSegmentBeingCompacted: segInfo)
>                         ifTrue:
>                                 [self freeSegment: segInfo.
>                                  segmentToFill ifNil: [segmentToFill :=
> segInfo]]]!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>isSingleFreeObject: (in category
> 'sweep phase - copied from SpurSweeper') -----
> + isSingleFreeObject: objOop
> +       <inline: true>
> +       | next |
> +       ^ (manager isFreeObject: objOop) and:
> +               [next := manager objectAfter: objOop limit: manager
> endOfMemory.
> +               (manager oop: next isGreaterThanOrEqualTo: manager
> endOfMemory) or: [manager isMarked: next]]!
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>occupationOf: (in category
> 'segment access') -----
> + occupationOf: segInfo
> +       <var: 'segInfo' type: #'SpurSegmentInfo *'>
> - occupationOf: segment
> -       <var: 'segment' type: #'SpurSegmentInfo *'>
>         "Swizzle is abused bit 8 isClaimed bits 0-7 occupation"
> +       ^segInfo swizzle bitAnd: 16rFF!
> -       ^segment swizzle bitAnd: 16rFF!
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>postSwizzleAction (in category
> 'api') -----
>   postSwizzleAction
>         "Since the compact abuses the swizzle field of segment, it needs
> to be rest after start-up."
> +       | segInfo |
> +       <var: 'segInfo' type: #'SpurSegmentInfo *'>
>         0 to: manager numSegments - 1 do:
> +               [:i|
> -               [:i| | segInfo |
>                  segInfo := self addressOf: (manager segmentManager
> segments at: i).
>                  segInfo swizzle: 0 ]!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>printTheBogons: (in category
> 'debugging') -----
> + printTheBogons: aBogon
> +       <inline: true>
> +       coInterpreter
> +               print: 'bogon '; printHexnp: aBogon; cr!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>remapObj: (in category 'api')
> -----
> + remapObj: objOop
> +       <api>
> +       <inline: false>
> +       ^manager vanillaRemapObj: objOop!
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>setOccupation:used:unused: (in
> category 'segment access') -----
> + setOccupation: segInfo used: used unused: unused
> +       <var: 'segInfo' type: #'SpurSegmentInfo *'>
> - setOccupation: segment used: used unused: unused
> -       <var: 'segment' type: #'SpurSegmentInfo *'>
>         "Swizzle is abused bit 8 isClaimed bits 0-7 occupation
>          Setting occupation resets the claim bit"
>         | occupation |
>         occupation := used * 255 // (used + unused).
> +       segInfo swizzle: occupation!
> -       segment swizzle: occupation!
>
> Item was changed:
>   ----- Method: SpurSelectiveCompactor>>setOccupationAtIndex:used:unused:
> (in category 'segment access') -----
>   setOccupationAtIndex: segmentIndex used: used unused: unused
>         "Swizzle is abused bit 8 isClaimed bits 0-7 occupation
>          Setting occupation resets the claim bit"
>         | occupation segInfo |
> +       <var: 'segInfo' type: #'SpurSegmentInfo *'>
>         segInfo := self addressOf: (manager segmentManager segments at:
> segmentIndex).
>         occupation := used * 255 // (used + unused).
>         segInfo swizzle: occupation!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>shouldRemapObj: (in category
> 'api') -----
> + shouldRemapObj: objOop
> +       <api>
> +       ^manager vanillaShouldRemapObj: objOop!
>
> Item was added:
> + ----- Method: SpurSelectiveCompactor>>unmark: (in category 'sweep phase
> - copied from SpurSweeper') -----
> + unmark: objOop
> +       self assert: ((manager isMarked: objOop) and: [(manager
> isFreeObject: objOop) not]).
> +       (manager isSegmentBridge: objOop) ifFalse: [manager setIsMarkedOf:
> objOop to: false].
> +       (manager isPinned: objOop) ifTrue: [manager segmentManager
> notePinned: objOop]!
>
> Item was added:
> + ----- Method: SpurSweeper>>postSwizzleAction (in category 'api') -----
> + postSwizzleAction
> +       "do nothing"
> +       !
>
> Item was added:
> + ----- Method: SpurSweeper>>printTheBogons: (in category 'debugging')
> -----
> + printTheBogons: aBogon
> +       <inline: true>
> +       coInterpreter
> +               print: 'bogon '; printHexnp: aBogon; cr!
>
>


-- 
Clément Béra
https://clementbera.github.io/
https://clementbera.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20180426/22c86304/attachment-0001.html>


More information about the Vm-dev mailing list