<div dir="ltr">Hi All,<div><br></div><div> for those of you brave enough to want to try the new compactor on Mac OS in 32-bits you'll find unofficial Squeak and Pharo versions on <a href="mailto:cogftpuser@ftp.mirandabanda.org">cogftpuser@ftp.mirandabanda.org</a> (pw c0gg1ng) in Squeak-MacOSX-32bit-VMMaker.oscog-eem.2090-SpurPlanningCompactor.tar & Pharo-MacOSX-32bit-VMMaker.oscog-eem.2090-SpurPlanningCompactor.tar. The Squeak is uploaded, the Pharo is uploading now.</div><div><br></div><div>Both tars contain production, assert and debug VMs. If you see failures please try and create a reproducible case that shows an assert fail using either the assert or debug VMs.</div><div><br></div><div>And be sure to back up your images before hand. The compactor, being part of the GC, is run at snapshot time, so any bugs could leave you with a pile of bits in place of a previously launch able image.</div><div><br></div><div>I can see one issue; the snapshot files are not as small as expected, but haven't had time to investigate why yet. But I do see improved performance and know that some of you are eager to try it out. So let me know and please report back here with both positive and negative experiences.</div><div><br></div><div>Cheers!</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 13, 2017 at 3:14 PM, Eliot Miranda <span dir="ltr"><<a href="mailto:eliot.miranda@gmail.com" target="_blank">eliot.miranda@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi All,<div><br></div><div> here are some stats to go with the new compactor in use in my work VMMaker image running on a 2.3 GHz Intel Core i7 late 2012 MacMini running Yosemite 10.10.5</div><div><br></div><div><div>uptime<span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>0h6m2s</div><div>memory<span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>231,735,296 bytes</div><div><span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>old<span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>226,734,992 bytes (97.8%)</div><div><span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>young<span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>3,801,936 bytes (1.6%)</div><div><span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>used<span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>192,741,392 bytes (83.2%)</div><div><span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>free<span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>35,365,936 bytes (15.3%)</div><div>GCs<span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>6,992 (51.9 ms between GCs)</div><div><span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>full<span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>6 totalling 1,573 ms (0.4% uptime), avg 262.2 ms</div><div><span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>incr<span class="m_5073615157930592529gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>6,986 totalling 5,306 ms (1.5% uptime), avg 0.8 ms</div></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Fri, Jan 13, 2017 at 3:00 PM, <span dir="ltr"><<a href="mailto:commits@source.squeak.org" target="_blank">commits@source.squeak.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Eliot Miranda uploaded a new version of VMMaker to project VM Maker:<br>
<a href="http://source.squeak.org/VMMaker/VMMaker.oscog-eem.2090.mcz" rel="noreferrer" target="_blank">http://source.squeak.org/VMMak<wbr>er/VMMaker.oscog-eem.2090.mcz</a><br>
<br>
==================== Summary ====================<br>
<br>
Name: VMMaker.oscog-eem.2090<br>
Author: eem<br>
Time: 13 January 2017, 2:59:27.453633 pm<br>
UUID: 63a161b9-17e1-4911-a89a-1687d9<wbr>ba9a1a<br>
Ancestors: VMMaker.oscog-eem.2089<br>
<br>
SpurPlanningCompactor:<br>
Fix the freeing across segment boundaries at end of run bug (freeFrom:upTo:previousPin: must check for an intervening segment bridge).<br>
<br>
Attempt to write a test to catch this.<br>
<br>
SpurPlanningCompactor now ready for real-world testing.<br>
<br>
=============== Diff against VMMaker.oscog-eem.2089 ===============<br>
<br>
Item was changed:<br>
----- Method: SpurMemoryManager>>freeChunkWi<wbr>thBytes:at: (in category 'free space') -----<br>
freeChunkWithBytes: bytes at: address<br>
<inline: false><br>
| freeChunk |<br>
self assert: (self isInOldSpace: address).<br>
+ self assert: (segmentManager segmentContainingObj: address) = (segmentManager segmentContainingObj: address + bytes).<br>
freeChunk := self initFreeChunkWithBytes: bytes at: address.<br>
- self assert: (segmentManager segmentContainingObj: freeChunk) = (segmentManager segmentContainingObj: (self addressAfter: freeChunk)).<br>
self addToFreeList: freeChunk bytes: bytes.<br>
self assert: freeChunk = (self objectStartingAt: address).<br>
^freeChunk!<br>
<br>
Item was changed:<br>
----- Method: SpurPlanningCompactor>>freeFro<wbr>m:upTo:previousPin: (in category 'private') -----<br>
freeFrom: toFinger upTo: limit previousPin: previousPinOrNil<br>
"Free from toFinger up to limit, dealing with a possible intervening run of pinned objects starting at previousPinOrNil."<br>
<inline: false><br>
+ | effectiveToFinger pin nextUnpinned start seg |<br>
+ <var: #seg type: #'SpurSegmentInfo *'><br>
- | effectiveToFinger pin nextUnpinned start |<br>
self cCode: [] inSmalltalk:<br>
[coInterpreter cr; cr; print: 'freeing at '; printHexnp: toFinger; print: ' up to '; printHexnp: limit; print: ' pin '; printHexnp: previousPinOrNil; cr].<br>
effectiveToFinger := toFinger.<br>
pin := previousPinOrNil.<br>
+ "If the range toFinger to limit spans segments but there is no pin (as when freeing to the end of memory)<br>
+ segment boundaries must still be observed. So in this case use the nearest bridge above toFinger as the pin."<br>
+ pin ifNil:<br>
+ [seg := manager segmentManager segmentContainingObj: toFinger.<br>
+ seg segLimit < limit ifTrue:<br>
+ [pin := manager segmentManager bridgeFor: seg]].<br>
[pin notNil] whileTrue:<br>
[(start := manager startOfObject: pin) > toFinger ifTrue:<br>
[manager addFreeChunkWithBytes: start - effectiveToFinger at: effectiveToFinger].<br>
nextUnpinned := self unmarkPinnedObjectsAndFindFirs<wbr>tUnpinnedOrFreeEntityFollowing<wbr>: pin.<br>
nextUnpinned >= limit ifTrue:<br>
[^self].<br>
effectiveToFinger := manager startOfObject: nextUnpinned.<br>
pin := self findNextMarkedPinnedAfter: nextUnpinned].<br>
manager addFreeChunkWithBytes: limit - effectiveToFinger at: effectiveToFinger!<br>
<br>
Item was changed:<br>
----- Method: SpurPlanningCompactorTests>>te<wbr>stRandomAssortmentWithNewSegme<wbr>nt: (in category 'private') -----<br>
testRandomAssortmentWithNewSeg<wbr>ment: random<br>
"Test that the compactor can handle a random assortment of live, pinned, dead, and free chunks,<br>
with some allocation in a new segment. No live pinned objects are created in the new segment<br>
to obtain the situation that the last segment is entirely empty after compaction. This tests shrinkage."<br>
| om pig lastObj obj expectedFreeSpace liveFill pinFill liveCount pinCount totalLive totalPinned pinned |<br>
random reset. "random is a read stream on 3000 random numbers; for repeatability"<br>
om := self initializedVM objectMemory.<br>
om allOldSpaceObjectsDo: [:o| om setIsMarkedOf: o to: true. lastObj := o].<br>
<br>
pinFill := 16r99999900.<br>
liveFill := 16r55AA0000.<br>
liveCount := pinCount := expectedFreeSpace := 0.<br>
pinned := Set new.<br>
<br>
1000 timesRepeat:<br>
[| nSlots next newObj |<br>
nSlots := (random next * 300) rounded. "Make sure we stray into overflow size field territory."<br>
newObj := om allocateSlotsInOldSpace: nSlots format: om firstLongFormat classIndex: ClassBitmapCompactIndex.<br>
(next := random next) > 0.95<br>
ifTrue: "pinned"<br>
[om<br>
fillObj: newObj numSlots: nSlots with: pinFill + (pinCount := pinCount + 1);<br>
setIsPinnedOf: newObj to: true]<br>
ifFalse: "mobile"<br>
[om<br>
fillObj: newObj numSlots: nSlots with: liveFill + (liveCount := liveCount + 1)].<br>
(next := random next) >= 0.333<br>
ifTrue:<br>
[om setIsMarkedOf: newObj to: true.<br>
(om isPinned: newObj) ifTrue:<br>
[pinned add: newObj]]<br>
ifFalse: "dead or free"<br>
[expectedFreeSpace := expectedFreeSpace + (om bytesInObject: newObj).<br>
(om isPinned: newObj) "Must check /before/ setObjectFree: which clears all bits"<br>
ifTrue: [pinCount := pinCount - 1]<br>
ifFalse: [liveCount := liveCount - 1].<br>
next >= 0.2<br>
ifTrue: [om setIsMarkedOf: newObj to: false]<br>
ifFalse: [om setObjectFree: newObj]]].<br>
<br>
pig := om allocateSlotsInOldSpace: (om numSlotsOfAny: om findLargestFreeChunk) format: om firstLongFormat classIndex: ClassBitmapCompactIndex.<br>
self deny: pig isNil.<br>
self assert: 0 equals: om bytesLeftInOldSpace.<br>
om growOldSpaceByAtLeast: om growHeadroom // 2.<br>
self assert: om growHeadroom equals: om bytesLeftInOldSpace + om bridgeSize.<br>
expectedFreeSpace := expectedFreeSpace + (om bytesInObject: pig).<br>
<br>
1000 timesRepeat:<br>
[| nSlots next newObj |<br>
nSlots := (random next * 300) rounded. "Make sure we stray into overflow size field territory."<br>
newObj := om allocateSlotsInOldSpace: nSlots format: om firstLongFormat classIndex: ClassBitmapCompactIndex.<br>
"No pinned objects in second segment."<br>
om fillObj: newObj numSlots: nSlots with: liveFill + (liveCount := liveCount + 1).<br>
(next := random next) >= 0.333<br>
ifTrue:<br>
[om setIsMarkedOf: newObj to: true.<br>
(om isPinned: newObj) ifTrue:<br>
[pinned add: newObj]]<br>
ifFalse: "dead or free"<br>
[expectedFreeSpace := expectedFreeSpace + (om bytesInObject: newObj).<br>
liveCount := liveCount - 1.<br>
next >= 0.2<br>
ifTrue: [om setIsMarkedOf: newObj to: false]<br>
ifFalse: [om setObjectFree: newObj]]].<br>
<br>
totalPinned := pinCount.<br>
totalLive := liveCount.<br>
self assert: totalPinned < (totalPinned + totalLive / 20). "should average 2.5%"<br>
<br>
"useful pre-compaction printing:"<br>
false ifTrue:<br>
[liveCount := pinCount := 0.<br>
om allOldSpaceEntitiesFrom: (om objectAfter: lastObj) to: (om objectBefore: om endOfMemory) do:<br>
[:o|<br>
om coInterpreter print:<br>
((om isMarked: o)<br>
ifTrue: [(((om isPinned: o)<br>
ifTrue: [pinCount := pinCount + 1]<br>
ifFalse: [liveCount := liveCount + 1])<br>
printPaddedWith: Character space to: 3 base: 10), ' ']<br>
ifFalse: [' ']).<br>
om printEntity: o]].<br>
<br>
expectedFreeSpace := expectedFreeSpace + om bytesLeftInOldSpace.<br>
om compactor compact.<br>
self assert: expectedFreeSpace equals: om bytesLeftInOldSpace.<br>
self assert: om allObjectsUnmarked.<br>
<br>
"useful post-compaction printing:"<br>
false ifTrue:<br>
[liveCount := pinCount := 0.<br>
om allOldSpaceEntitiesFrom: (om objectAfter: lastObj) to: (om objectBefore: om endOfMemory) do:<br>
[:o|<br>
om coInterpreter print:<br>
((om isFreeObject: o)<br>
ifFalse: [(((om isPinned: o)<br>
ifTrue: [pinCount := pinCount + 1]<br>
ifFalse: [liveCount := liveCount + 1])<br>
printPaddedWith: Character space to: 3 base: 10), ' ']<br>
ifTrue: [' ']).<br>
om printEntity: o]].<br>
<br>
"First check and/or count populations..."<br>
liveCount := pinCount := 0.<br>
om allOldSpaceObjectsFrom: (om objectAfter: lastObj) do:<br>
[:o|<br>
(om isPinned: o)<br>
ifTrue:<br>
[pinCount := pinCount + 1.<br>
self assert: (pinned includes: o)]<br>
ifFalse: [liveCount := liveCount + 1]].<br>
self assert: totalPinned equals: pinCount.<br>
self assert: totalLive equals: liveCount.<br>
<br>
"Now check fills, which also tests update of first field on move..."<br>
liveCount := pinCount := 0.<br>
obj := lastObj.<br>
1 to: totalLive + totalPinned do:<br>
[:n| | expectedFill actualFill |<br>
[obj := om objectAfter: obj. (om isEnumerableObject: obj) or: [obj >= om endOfMemory]] whileFalse.<br>
expectedFill := (om isPinned: obj)<br>
ifTrue: [pinFill + (pinCount := pinCount + 1)]<br>
ifFalse: [liveFill + (liveCount := liveCount + 1)].<br>
1 to: (om numSlotsOf: obj) do:<br>
[:i| self assert: expectedFill equals: (actualFill := om fetchPointer: i - 1 ofObject: obj)]].<br>
"the Last segment should be empty"<br>
self assert: (om segmentManager isEmptySegment: (om segmentManager segments at: 1)).<br>
"They should be the last objects, followed by a free object to the end fo the first segment, a bridge, then an empty segment with a single free object in it."<br>
self assert: (om isFreeObject: (om objectAfter: obj)).<br>
self assert: (om isSegmentBridge: (om objectAfter: (om objectAfter: obj))).<br>
self assert: (om isFreeObject: (om objectAfter: (om objectAfter: (om objectAfter: obj)))).<br>
self assert: om endOfMemory equals: (om addressAfter: (om objectAfter: (om objectAfter: (om objectAfter: obj)))).<br>
<br>
"And the memory should shrink if the shrinkThreshold is low enough"<br>
om shrinkThreshold: om growHeadroom.<br>
om attemptToShrink.<br>
+ self assert: om segmentManager numSegments = 1!<br>
- self assert: om segmentManager numSegments = 1.!<br>
<br>
Item was added:<br>
+ ----- Method: SpurPlanningCompactorTests>>te<wbr>stRunOfObjectsWithExtraSegment (in category 'tests') -----<br>
+ testRunOfObjectsWithExtraSegme<wbr>nt<br>
+ "Test that the compactor can handle compacting more than one segment and shortening the memory."<br>
+ | om expectedFreeSpace pig gapObj obj |<br>
+ om := self initializedVM objectMemory.<br>
+ om allOldSpaceObjectsDo: [:o| om setIsMarkedOf: o to: true].<br>
+ "First create a gap"<br>
+ gapObj := om allocateSlotsInOldSpace: 100 format: om firstLongFormat classIndex: ClassArrayCompactIndex.<br>
+ om fillObj: gapObj numSlots: 100 with: om falseObject.<br>
+ self deny: (om isMarked: gapObj).<br>
+ expectedFreeSpace := om bytesInObject: gapObj.<br>
+ "Now some objects, a gap to a new segment and another run of objects."<br>
+ 1 to: 2 do:<br>
+ [:i|<br>
+ 10 timesRepeat:<br>
+ [obj := om allocateSlotsInOldSpace: 50 format: om firstLongFormat classIndex: ClassBitmapCompactIndex.<br>
+ om fillObj: obj numSlots: 50 with: 16r55AA55AA;<br>
+ setIsMarkedOf: obj to: true.<br>
+ obj := om allocateSlotsInOldSpace: 260 format: om firstLongFormat classIndex: ClassBitmapCompactIndex.<br>
+ om fillObj: obj numSlots: 260 with: 16rAA55AA55;<br>
+ setIsMarkedOf: obj to: true].<br>
+ i = 1 ifTrue:<br>
+ [pig := om allocateSlotsInOldSpace: (om numSlotsOfAny: om findLargestFreeChunk) format: om firstLongFormat classIndex: ClassBitmapCompactIndex.<br>
+ self deny: pig isNil.<br>
+ self assert: 0 equals: om bytesLeftInOldSpace.<br>
+ om growOldSpaceByAtLeast: om growHeadroom // 2.<br>
+ self assert: om growHeadroom equals: om bytesLeftInOldSpace + om bridgeSize.<br>
+ expectedFreeSpace := expectedFreeSpace + (om bytesInObject: pig)]].<br>
+<br>
+ "useful debugging:""om printOopsFrom: gapObj to: om endOfMemory"<br>
+ expectedFreeSpace := expectedFreeSpace + om bytesLeftInOldSpace.<br>
+ om compactor compact.<br>
+ self assert: expectedFreeSpace equals: om bytesLeftInOldSpace.<br>
+ self assert: om allObjectsUnmarked.<br>
+<br>
+ "The first mobile object past the pinned objects should have moved."<br>
+ self assert: ClassBitmapCompactIndex equals: (om classIndexOf: gapObj).<br>
+ obj := gapObj.<br>
+ "The objects have moved."<br>
+ 20 timesRepeat:<br>
+ [self assert: ClassBitmapCompactIndex equals: (om classIndexOf: obj).<br>
+ 0 to: (om numSlotsOf: obj) - 1 do: [:i| self assert: 16r55AA55AA equals: (om fetchPointer: i ofObject: obj)].<br>
+ obj := om objectAfter: obj.<br>
+ self assert: ClassBitmapCompactIndex equals: (om classIndexOf: obj).<br>
+ 0 to: (om numSlotsOf: obj) - 1 do: [:i| self assert: 16rAA55AA55 equals: (om fetchPointer: i ofObject: obj)].<br>
+ obj := om objectAfter: obj].<br>
+ "the Last segment should be empty"<br>
+ self assert: (om segmentManager isEmptySegment: (om segmentManager segments at: 1)).<br>
+ "They should be the last objects, followed by a free object to the end fo the first segment, a bridge, then an empty segment with a single free object in it."<br>
+ self assert: (om isFreeObject: obj).<br>
+ self assert: (om isSegmentBridge: (om objectAfter: obj)).<br>
+ self assert: (om isFreeObject: (om objectAfter: (om objectAfter: obj))).<br>
+ self assert: om endOfMemory equals: (om addressAfter: (om objectAfter: (om objectAfter: obj))).<br>
+<br>
+ "And the memory should shrink if the shrinkThreshold is low enough"<br>
+ om shrinkThreshold: om growHeadroom.<br>
+ om attemptToShrink.<br>
+ self assert: om segmentManager numSegments = 1!<br>
<br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div class="m_5073615157930592529gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span style="font-size:small;border-collapse:separate"><div>_,,,^..^,,,_<br></div><div>best, Eliot</div></span></div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span style="font-size:small;border-collapse:separate"><div>_,,,^..^,,,_<br></div><div>best, Eliot</div></span></div></div></div>
</div>