Vm-dev August 2023

vm-dev@lists.squeakfoundation.org

5 participants
10 discussions

[OpenSmalltalk/opensmalltalk-vm] mmap() sqUnixSpurMemory patch (Issue #665)
by David Stes 03 Mar '24

03 Mar '24

Hi, I have sent in the past email to Eliot Miranda about an issue which I discovered in August 2021 (two years ago), and I thought to raise an issue here. It is a Linux/UNIX patch issue with the behavior of the mmap() system call or library call as used by the OpenSmalltalk COG vm. When way back in 2021 some changes were made for Linux ARM8 to the file saUnixSpurMemory.c I reported that this was breaking the Solaris 11.3 build. Curiously the Solaris 11.4 build was still working fine. I think (I don't know) that perhaps the behavior of mmap() on Solaris 11.4 is slightly diferent than on 11.3, to match (perhaps) the Linux kernel behavior of mmap(). An idea that I have is to ask some UNIX guru's whether there exist a autoconf-archive (configure script) test or script to test the behavior of mmap() for portability. However here I will just report what the issue is : since august 2021 I am building succesfully on Solaris with the following patch ``` --- opensmalltalk-vm-sun-v5.0.34/platforms/unix/vm/sqUnixSpurMemory.c Fri Aug 13 19:54:42 2021 +++ p0/opensmalltalk-vm-sun-v5.0.34/platforms/unix/vm/sqUnixSpurMemory.c Sun Aug 15 14:11:58 2021 @@ -277,16 +277,21 @@ { void *hint = sbrk(0); // a hint of the lowest possible address for mmap void *result; + char *address; + unsigned long alignment; pageSize = getpagesize(); pageMask = ~(pageSize - 1); + alignment = max(pageSize,1024*1024); + address = (char *)(((usqInt)hint + alignment - 1) & ~(alignment - 1)); + #if !defined(MAP_JIT) # define MAP_JIT 0 #endif *desiredSize = roundUpToPage(*desiredSize); - result = mmap(hint, *desiredSize, + result = mmap(address, *desiredSize, #if DUAL_MAPPED_CODE_ZONE PROT_READ | PROT_EXEC, #else ``` I really need Eliot's help on this. Eliot : what I did was try to understand your code and you seem to make some alignment calculations on the desired memory location that you request through mmap(). I believe that the traditional UNIX mmap() or the Solaris 10 and Solaris 11.3 mmap() is reacting slightly differently to the requested memory "hint". By some experimentatoin and debugging I found the above patch, which I wrote based on looking at your code and try to mirror the calculations that you are doing, which are not fully symmetric. By copying some code, I discovered that with the above patch the build for both Solaris 11.3 and Solaris 11.4 are working fine again. I think this is a general UNIX issue that could be of interest to all UNIX software. If an autoconfigure autoconf-archive test would exist to test the behavior of mmap() then that could be dealt in all software that uses mmap() with anonymous memory mapping. As far as I know the history of the mmap() call is that it originated in the world of Sun and Solaris and is since a very long time present in Linux as well, and I think some improvements in Linux where then backported to Solaris 11.4. That would explain why the current OpenSmalltalk-VM code works fine on Solaris 11.4. For reasons of portability to all flavors of UNIX and Linux, I think a configure build test for the code could benefit portability of your Cog VM. Let us know please what you think. I could simply submit my patch as a patch to mainstream upstream OpenSmalltalk but I don't know your opinion about it. And the way I have been using it as a platform specific patch (that I apply prior to the build on Solaris) is fine for me as well. Regards, David Stes -- Reply to this email directly or view it on GitHub: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/665 You are receiving this because you are subscribed to this thread. Message ID: <OpenSmalltalk/opensmalltalk-vm/issues/665(a)github.com>

4 19

[OpenSmalltalk/opensmalltalk-vm] BitBlt rgbMul is broken for depth 1 and not correctly rounded above depth 4 (Issue #651)
by Nicolas Cellier 24 Nov '23

24 Nov '23

In depth 1, the resulting bits will always be 0. It's not a big problem because rgbMul is just a bitAnd operation at this depth. So a quick workaround would be to detect the case in BitBltSimulation destDepth = 1 ifTrue: [^self bitAnd: sourceWord with: destinationWord]. That would also accelerate the Bit BLock Transfer operation, so it's a good hack. But there is more. What we want is multiply ratios in interval [0,1]. dstRatio * srcRatio Our implementation is scaled ratio (scaled by `1 << nBits - 1`): src := (srcRatio * scale) rounded. dst := (dstRatio * scale) rounded. So what we want is: ((dst/scale) * (src/scale) * scale) rounded that is: (dst*src / (1<<nBits-1)) rounded Unfortunately, that's the other grief with the current implementation used for rounding: (dst+1)*(src+1) - 1 >> nBits It only equals correctly rounded operation for depths 2 and 4. For rounding we might use: (((dst/scale) * (src/scale) + 0.5) * scale) truncated. that is expressed with truncated division: dst*src + (scale+1//2) // scale So here is a nicer formulation for doing the job at any depth (including 5bits rgb channels for 16 bits depth) with correctly rounded division: aux := src * dst + (1 << (nBits - 1)). "add mid-scale for rounding" result := aux << (nBits - 1) + aux << (nBits -1). "divide by scale" This is because instead of dividing by scale, we can multiply by shifted inverse (sort of double precision), then shift right. (2 to: 32) allSatisfy: [:nBits | (1 << (nBits * 2) / (1 << nBits - 1)) rounded = (1 << nBits + 1)]. Multiplying by this inverse is easy and cheap: x * (1 << nBits + 1) = (x << nBits + x). And then applying the right shift `>> (2 * nBits)` is equivalent to: x >> nBits + x >> nBits. We must first add 0.5 (scaled), that is `src * dst + (1 << (nBits -1))` - our formulation of aux, and we're done. We verify: { (0 to: 1<<20-1) allSatisfy: [:i | (1<<9+i)>>10+ (1<<9+i)>>10 = (i/1023) rounded]. (0 to: 1<<18-1) allSatisfy: [:i | (1<<8+i)>>9+ (1<<8+i)>>9 = (i/511) rounded]. (0 to: 1<<16-1) allSatisfy: [:i | (1<<7+i)>>8+ (1<<7+i)>>8 = (i/255) rounded]. (0 to: 1<<14-1) allSatisfy: [:i | (1<<6+i)>>7+ (1<<6+i)>>7 = (i/127) rounded]. (0 to: 1<<12-1) allSatisfy: [:i | (1<<5+i)>>6+ (1<<5+i)>>6 = (i/63) rounded]. (0 to: 1<<10-1) allSatisfy: [:i | (1<<4+i)>>5+ (1<<4+i)>>5 = (i/31) rounded]. (0 to: 1<<8-1) allSatisfy: [:i | (1<<3+i)>>4+ (1<<3+i)>>4 = (i/15) rounded]. (0 to: 1<<6-1) allSatisfy: [:i | (1<<2+i)>>3+ (1<<2+i)>>3 = (i/7) rounded]. (0 to: 1<<4-1) allSatisfy: [:i | (1<<1+i)>>2+ (1<<1+i)>>2 = (i/3) rounded]. } allSatisfy: #yourself. The nice thing is that above down-scaling operation can be multiplexed. Suppose that we have p groups of 2*nBits `M` holding square-scale multiplication of each channel concatenated in a double-Word-Mul. doubleWordMul = Mp .... M5 M3 M1 Note we arrange to have odd channels in low word, and even channels in high word. We first form a `groupMask` on a word with (p+1)/2 groups of nBits alternating all one `i` and all zero `o`, `oioi...ioi`. channelMask := 1 << nBits - 1. groupMask := 0. 0 to: wordBits // (2 * nBits) do: [:i | groupMask = groupMask << (2 * nBits) + channelMask]. Where wordBits is the number of bits in a word (usually we want to operate on 32 bits words in BitBlt). We form the `doubleGroupMask` on a double-word with p groups of 2*nBits `oi`: doubleGroupMask := groupMask >> nBits. doubleGroupMask := doubleGroupMask << wordBits + groupMask. And we perform the division by scale: doubleWordMul := (doubleWordMul >> nBits bitAnd: doubleGroupMask) + doubleWord >> nBits bitAnd: doubleGroupMask. At this stage we obtain a double word containing scaled multiplicands interleaved with groups of nBits zeros: o mp ... o m3 o m1 Now the final result can be obtained by shifting back: doubleWordMul >> (wordBits - nBits) + (doubleWordMul bitAnd: groupMask) The only problem remaining is how to obtain the squared-scale multiplicands. It would be easy to form the alternate even-odd channels for each src and dst operands: doubleWordSrc := src >> nBits bitAnd: groupMask. doubleWordSrc := doubleWordSrc << wordBits + (src bitAnd: groupMask). doubleWordDst := dst >> nBits bitAnd: groupMask. doubleWordDst := doubleWordDst << wordBits + (dst bitAnd: groupMask). we now get `o sp ... o s3 o s1` and `0 dp ... o d3 o d1`, but we would now need a SIMD integer multiplication operating on groups of 2*nBits in parallel... We don't have that, at least in portable C code. So we still have to emulate it with a loop. half := 1 << (nBits - 1). shift := 0. doubleWordMul := 0 0 to: nChannels - 1 do: [:i | doubleWordMul := doubleWordMul + (((doubleWordSrc >> shift bitAnd: channelMask) * (doubleWordSrc >> shift bitAnd: channelMask) + half) << shift). shift := shift + nBits + nBits]. We know that each operation cannot overflow on upper neighbour group of 2*nBits, because the maximum value is: (1<<nBits-1) squared + (1 << (nBits-1)) = 1 << (2*nBits) - (2*(1<<nBits)) + (1 << (nBits-1)) - 1 < (1 << (2*nBits) - 1) It remains the odd case of 16 bits depth, which has 3 groups of 5 bits and a leading zero. I believe that above algorithm works without splitting in two half-words... To be tested. We have gathered the pieces for a correctly rounded almost-multiplexed rgbMul. Somehow have our cake and eat it too. -- Reply to this email directly or view it on GitHub: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/651 You are receiving this because you are subscribed to this thread. Message ID: <OpenSmalltalk/opensmalltalk-vm/issues/651(a)github.com>

2 4

Re: [Newbies] FreeBSD VM
by Tobias Pape 30 Sep '23

30 Sep '23

Hi David > On 30. Mar 2023, at 16:09, David Gray <dgray(a)iesl.forth.gr> wrote: > > I’ve been trying to get a working VM for FreeBSD. However even though I manage to get the system to start and update itself it gives a segmentation error when I open a workspace and type any character at all. I used the latest linux64x64 cog.spur. > Both the 6.0 and 5.3 images behaved the same. To get it to compile I had the comment out the Joystick plugin and edit the baud rates inside the serial plugin. I’m a complete beginner with squeak so I don’t know how to proceed from here. Based on your comments, I went and tried to verify that the VM still builds. I have made similar adjustments and was able to build a VM with the recents commits to the Cog branch of the VM. Can you please build a debug vm and try to recreate the error? I am very interested in that. Best regards -Tobias

4 5

VM Maker: VMMaker.oscog-mt.3336.mcz
by commits＠source.squeak.org 31 Aug '23

31 Aug '23

Marcel Taeumel uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-mt.3336.mcz ==================== Summary ==================== Name: VMMaker.oscog-mt.3336 Author: mt Time: 31 August 2023, 2:43:31.734117 pm UUID: 3c2286bf-77b6-7843-abc3-8ceac625b86c Ancestors: VMMaker.oscog-eem.3335 Adds commentary for path-lookup edge case on Windows platform. See OSVM commit 330d6779ad2ecbce1f07131f53d75cd168165f0b. =============== Diff against VMMaker.oscog-eem.3335 =============== Item was changed: InterpreterPlugin subclass: #FilePlugin instanceVariableNames: 'sCCPfn sCDPfn sCGFTfn sCLPfn sCSFTfn sDFAfn sCDFfn sCOFfn sCRFfn sHFAfn' classVariableNames: 'DirBadPath DirEntryFound DirNoMoreEntries' poolDictionaries: '' category: 'VMMaker-Plugins'! + !FilePlugin commentStamp: 'ct 11/30/2022 15:03' prior: 0! + Provide access to the host machine file system. Requires both the Cross platform support files from platforms - Cross - plugins - FilePlugin (or some suitable replacement) and the platform specific fils from platforms - {your platform} - plugins - FilePlugin. + + Note that in general, file paths must not contain syntactic sugar for the current platform (e.g., '.' or '..', or on Windows, forward slashes instead of backslashes). These conventions are only fully supported on Unix platforms; on Windows, they are only supported for short non-UNC file paths containing max 260 characters. See the comments in sqWin32Directory.c for more details.! - !FilePlugin commentStamp: 'tpr 5/5/2003 12:01' prior: 0! - Provide access to the host machine file system. Requires both the Cross platform support files from platforms - Cross - plugins - FilePlugin (or some suitable replacement) and the platform specific fils from platforms - {your platform} - plugins - FilePlugin.! Item was changed: ----- Method: FilePlugin>>primitiveDirectoryEntry (in category 'directory primitives') ----- primitiveDirectoryEntry "Two arguments - directory path, and simple file name; returns an array (see primitiveDirectoryLookup) describing the file or directory, or nil if it does not exist. + + Note that in general, the directory path name must not contain syntactic + sugar for the current platform (e.g., '.' or '..', or on Windows, forward slashes + instead of backslashes). These conventions are only fully supported on Unix + platforms; on Windows, they are only supported for short non-UNC file paths + containing max 260 characters (for the full path concatenated from the + directory path, a backslash, and the file name): + DON'T: primitiveDirectoryEntry '<very long path>\foo\.' ... + DON'T: primitiveDirectoryEntry '<very long path>\foo\..' ... + DON'T (on Windows): primitiveDirectoryEntry '<very long path>\foo/bar' ... + See the comment in sqWin32Directory.c for more details. + Primitive fails if the outer path does not identify a readable directory. (This is a lookup-by-name variant of primitiveDirectoryLookup.)" | requestedName pathName pathNameIndex pathNameSize status entryName entryNameSize createDate modifiedDate dirFlag posixPermissions symlinkFlag fileSize okToList reqNameIndex reqNameSize | <var: 'entryName' declareC: 'char entryName[256]'> <var: 'pathNameIndex' type: #'char *'> <var: 'reqNameIndex' type: #'char *'> <var: 'fileSize' type: #squeakFileOffsetType> <export: true> requestedName := interpreterProxy stackValue: 0. pathName := interpreterProxy stackValue: 1. (interpreterProxy isBytes: pathName) ifFalse: [^interpreterProxy primitiveFail]. "Outbound string parameters" pathNameIndex := interpreterProxy firstIndexableField: pathName. pathNameSize := interpreterProxy byteSizeOf: pathName. reqNameIndex := interpreterProxy firstIndexableField: requestedName. reqNameSize := interpreterProxy byteSizeOf: requestedName. self cCode: '' inSmalltalk: [entryName := ByteString new: 256. entryNameSize := createDate := modifiedDate := dirFlag := fileSize := posixPermissions := symlinkFlag := nil]. "If the security plugin can be loaded, use it to check for permission. If not, assume it's ok" okToList := sCLPfn ~= 0 ifTrue: [self cCode: '((sqInt (*)(char *, sqInt))sCLPfn)(pathNameIndex, pathNameSize)' inSmalltalk: [true]] ifFalse: [true]. status := okToList ifTrue: [self dir_EntryLookup: pathNameIndex _: pathNameSize _: reqNameIndex _: reqNameSize _: entryName _: (self addressOf: entryNameSize put: [:v| entryNameSize := v]) _: (self addressOf: createDate put: [:v| createDate := v]) _: (self addressOf: modifiedDate put: [:v| modifiedDate := v]) _: (self addressOf: dirFlag put: [:v| dirFlag := v]) _: (self addressOf: fileSize put: [:v| fileSize := v]) _: (self addressOf: posixPermissions put: [:v| posixPermissions := v]) _: (self addressOf: symlinkFlag put: [:v| symlinkFlag := v])] ifFalse: [DirNoMoreEntries]. interpreterProxy failed ifTrue: [^nil]. status = DirNoMoreEntries ifTrue: "no entry; return nil" [interpreterProxy "pop pathName, index, rcvr" pop: 3 thenPush: interpreterProxy nilObject. ^nil]. status = DirBadPath ifTrue: [^interpreterProxy primitiveFail]."bad path" interpreterProxy pop: 3 "pop pathName, index, rcvr" thenPush: (self cppIf: PharoVM ifTrue: [self makeDirEntryName: entryName size: entryNameSize createDate: createDate modDate: modifiedDate isDir: dirFlag fileSize: fileSize posixPermissions: posixPermissions isSymlink: symlinkFlag] ifFalse: [self makeDirEntryName: entryName size: entryNameSize createDate: createDate modDate: modifiedDate isDir: dirFlag fileSize: fileSize])! Item was changed: ----- Method: FilePlugin>>primitiveDirectoryLookup (in category 'directory primitives') ----- primitiveDirectoryLookup + "Two arguments - directory path, and an index to an item; returns an array (see primitiveDirectoryLookup) describing the file or directory, or nil if it does not exist. + + Note that in general, the directory path must not contain syntactic sugar for the current platform (e.g., '.' or '..', or on Windows, forward slashes instead of backslashes). These conventions are only fully supported on Unix platforms; on Windows, they are only supported for short non-UNC file paths containing max 260 characters: + DON'T: primitiveDirectoryLookup '<very long path>' 'foo\.' + DON'T: primitiveDirectoryLookup '<very long path>' foo\..' + DON'T (on Windows): primitiveDirectoryLookup '<very long path>' 'foo/bar' + DO: primitiveDirectoryLookup '<very long path>' '.' + See the comment in sqWin32Directory.c for more details. + + Primitive fails if the outer path does not identify a readable directory. (For a lookup-by-name variant, see primitiveDirectoryEntry.)" | index pathName pathNameIndex pathNameSize status entryName entryNameSize createDate modifiedDate dirFlag symlinkFlag posixPermissions fileSize okToList | <var: 'entryName' declareC: 'char entryName[256]'> <var: 'pathNameIndex' type: #'char *'> <var: 'fileSize' type: #squeakFileOffsetType> <export: true> index := interpreterProxy stackIntegerValue: 0. pathName := interpreterProxy stackValue: 1. (interpreterProxy isBytes: pathName) ifFalse: [^interpreterProxy primitiveFail]. pathNameIndex := interpreterProxy firstIndexableField: pathName. pathNameSize := interpreterProxy byteSizeOf: pathName. self cCode: '' inSmalltalk: [entryName := ByteString new: 256. entryNameSize := createDate := modifiedDate := dirFlag := fileSize := posixPermissions := symlinkFlag := nil]. "If the security plugin can be loaded, use it to check for permission. If not, assume it's ok" okToList := sCLPfn ~= 0 ifTrue: [self cCode: '((sqInt (*)(char *, sqInt))sCLPfn)(pathNameIndex, pathNameSize)' inSmalltalk: [true]] ifFalse: [true]. status := okToList ifTrue: [self dir_Lookup: pathNameIndex _: pathNameSize _: index _: entryName _: (self addressOf: entryNameSize put: [:v| entryNameSize := v]) _: (self addressOf: createDate put: [:v| createDate := v]) _: (self addressOf: modifiedDate put: [:v| modifiedDate := v]) _: (self addressOf: dirFlag put: [:v| dirFlag := v]) _: (self addressOf: fileSize put: [:v| fileSize := v]) _: (self addressOf: posixPermissions put: [:v| posixPermissions := v]) _: (self addressOf: symlinkFlag put: [:v| symlinkFlag := v])] ifFalse: [DirNoMoreEntries]. interpreterProxy failed ifTrue: [^nil]. status = DirNoMoreEntries ifTrue: "no more entries; return nil" [interpreterProxy "pop pathName, index, rcvr" pop: 3 thenPush: interpreterProxy nilObject. ^nil]. status = DirBadPath ifTrue: [^interpreterProxy primitiveFail]."bad path" interpreterProxy pop: 3 "pop pathName, index, rcvr" thenPush: (self cppIf: PharoVM ifTrue: [self makeDirEntryName: entryName size: entryNameSize createDate: createDate modDate: modifiedDate isDir: dirFlag fileSize: fileSize posixPermissions: posixPermissions isSymlink: symlinkFlag] ifFalse: [self makeDirEntryName: entryName size: entryNameSize createDate: createDate modDate: modifiedDate isDir: dirFlag fileSize: fileSize])!

1 0

[OpenSmalltalk/opensmalltalk-vm] 330d67: Repairs FileDirectory>>#exists for long paths (> 2...
by Christoph Thiede 31 Aug '23

31 Aug '23

Branch: refs/heads/Cog Home: https://github.com/OpenSmalltalk/opensmalltalk-vm Commit: 330d6779ad2ecbce1f07131f53d75cd168165f0b https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/330d6779ad2ecbce1f… Author: Christoph Thiede <christoph.thiede(a)student.hpi.de> Date: 2023-08-31 (Thu, 31 Aug 2023) Changed paths: M platforms/win32/vm/sqWin32Directory.c Log Message: ----------- Repairs FileDirectory>>#exists for long paths (> 260 characters) on Windows by well-defining an edge case in the Windows implementation of the FilePlugin primitiveDirectoryEntry. Documents the present limitations to syntactic sugar in long file paths in the platform code. An alternative consideration was to rewrite FileDirectory>>#exists to pass an empty string as file name to the primitive instead of modifying the VM, but strictly speaking, even this would have exploited an undefined behavior in the VM plugin, and an empty file name would be less idiomatic than a single dot. For the original bug report, see: https://github.com/hpi-swa-teaching/Morphic-Testing-Framework/issues/13 Thanks to Marcel (mt) for his support! Revision (4): Added example paths to comments and fixed erratum in comment of FilePlugin>>#primitiveDirectoryEntry.

1 0

In reference to: Minifying Woes
by Tom Braun 28 Aug '23

28 Aug '23

Hi all, I am new to the list, so I couldn’t answer directly to Minifying Woes (or at least didn’t know how to). I identified 75 VM intern objects in Tom’s (tobe) image, by loading it in the simulator and setting a halt after loading the image. With the following code: | collection | collection := OrderedCollection new. self allOldSpaceEntitiesDo: [:obj | (((self classIndexOf: obj) < self lastClassIndexPun) and: (self isImmediate: obj) not) ifTrue: [ collection add: obj] ]. (collection sorted: [:a :b | (self bytesInBody: a) > (self bytesInBody: b)]) collect: [:ea | ea hex -> (self bytesInBody: ea)] I get the following 75 objects: 1. A free chunk after all other objects. Can be ignored for the sake of minimising image size 2. Remembered set -> 1048592 byte 3. hiddenRootsObj -> 32848 byte 4 - 61. Pages of the mark and weakling stack -> respectively 32752 byte 62 - 74. arrays of the class table -> respectively 8208 byte 75. specialObjectsOop -> 520 byte As far as I can judge the stack pages and remembered set could be removed from the image to minimize it further. If I understand correctly the StackPages could be removed by using the SpurImagePreener. The remembered set Could be removed in the SpurImagePreener too (when the VM initialises the memory it initialises a new remembered set too, if it is nil). After a quick read of the SpurImagePreener I didn’t see that the remembered set gets removed. When I tried using the preener it resulted in an unusable image (both simulator and compiled VM couldn’t load it). I tried both: SpurImagePreener new preenImage: '/Users/tombraun/Desktop/Squeak6.0-22104-64bit copy.image' SpurImagePreener new writeDefaultHeader: true; savedWindowSize: 1@1; preenImage: '/Users/tombraun/Desktop/Squeak6.0-22104-64bit copy.image' Could be a me problem, as I did some changes to the memory management in my VMMaker image, although this shouldn’t influence the preener…. On this note @Eliot: why the decision to make the object stacks and the remembered set VM managed objects instead of allocating them separately? 1. We don’t need to keep them in a snapshot. All object stacks are empty after GC and as we flushed the new space pre snapshot The remembered set shouldn't need to be persisted too. 2. During GC we simply mark all stack pages and keep them alive. When we at least freed empty pages (after a limit, to prevent the running VM from having to allocate too many pages every GC?) I would see the value. What did I overlook or does it simply have historical reasons? Best, Tom (WoC)

5 5

VM Maker: VMMaker.oscog-eem.3335.mcz
by commits＠source.squeak.org 22 Aug '23

22 Aug '23

Eliot Miranda uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-eem.3335.mcz ==================== Summary ==================== Name: VMMaker.oscog-eem.3335 Author: eem Time: 22 August 2023, 1:29:56.927443 pm UUID: 1a04daa4-b43d-4e1f-9dc1-e9dd95701ce3 Ancestors: VMMaker.oscog-LM.3334 Rename CoInterpreterMT's processHasThreadId inst var to processHasThreadAffinity in the wake of Kernel-eem.1523. =============== Diff against VMMaker.oscog-LM.3334 =============== Item was changed: ----- Method: CoInterpreter>>setImageHeaderFlags: (in category 'internal interpreter access') ----- setImageHeaderFlags: flags "Set an array of flags indicating various properties of the saved image, responded to on image load. These are the same as the image header flags shifted right two bits, omitting the fullScreenFlag and float byte order flag. + Bit 0: if set, implies the image's Process class has threadAffinity as its 3rd inst var (zero relative) (meaningful to the MT VM only) - Bit 0: if set, implies the image's Process class has threadId as its 3rd inst var (zero relative) (meaningful to the MT VM only) Bit 1: if set, methods that are interpreted will have the flag bit set in their header Bit 2: if set, implies preempting a process does not put it to the back of its run queue Bit 3: if set, implies a threaded VM will not disown the VM if owned by the GUI thread (meaningful to the MT VM only) Bit 4: if set, implies the new finalization scheme where WeakArrays are queued Bit 5: if set, implies wheel events will be delivered as such and not mapped to arrow key events Bit 6: if set, implies arithmetic primitives will fail if given arguments of different types (float vs int) Bit 7: if set, implies file primitives (FilePlugin, FileAttributesPlugin) will answer file times in UTC not local times Bit 8: if set, implies the VM will not upscale the display on high DPI monitors; older VMs did this by default." flags asUnsignedInteger > 511 ifTrue: [^self primitiveFailFor: PrimErrUnsupported]. + "processHasThreadAffinity := flags anyMask: 1. specific to CoInterpreterMT" - "processHasThreadId := flags anyMask: 1. specific to CoInterpreterMT" imageHeaderFlags := (flags anyMask: 1) ifTrue: [imageHeaderFlags bitOr: 4] ifFalse: [imageHeaderFlags bitClear: 4]. flagInterpretedMethods := flags anyMask: 2. preemptionYields := flags noMask: 4. "noThreadingOfGUIThread := flags anyMask: 8.. specific to CoInterpreterMT" imageHeaderFlags := (flags anyMask: 8) ifTrue: [imageHeaderFlags bitOr: 32] ifFalse: [imageHeaderFlags bitClear: 32]. newFinalization := flags anyMask: 16. sendWheelEvents := flags anyMask: 32. primitiveDoMixedArithmetic := flags noMask: 64. imageHeaderFlags := (flags anyMask: 128) ifTrue: [imageHeaderFlags bitOr: 512] ifFalse: [imageHeaderFlags bitClear: 512]. upscaleDisplayIfHighDPI := flags noMask: 256! Item was changed: CoInterpreterPrimitives subclass: #CoInterpreterMT + instanceVariableNames: 'cogThreadManager checkThreadActivation maxWaitingPriority foreignCallbackPriority deferThreadSwitch disowningVMThread disownCount foreignCallbackProcessSlot activeProcessAffined relinquishing reenterThreadSchedulingLoop willNotThreadWarnCount processHasThreadAffinity' - instanceVariableNames: 'cogThreadManager checkThreadActivation maxWaitingPriority foreignCallbackPriority deferThreadSwitch disowningVMThread disownCount foreignCallbackProcessSlot activeProcessAffined relinquishing processHasThreadId reenterThreadSchedulingLoop willNotThreadWarnCount' classVariableNames: 'DisownFlagsShift DisownVMForProcessorRelinquish OwnVMForeignThreadFlag PerThreadRumpCStackSize PrimNumberRelinquishProcessor ProcessUnaffinedOnDisown ReturnToThreadSchedulingLoop VMAlreadyOwnedHenceDoNotDisown' poolDictionaries: 'VMThreadingConstants' category: 'VMMaker-Multithreading'! Item was changed: ----- Method: CoInterpreterMT>>bindProcess:toId: (in category 'process primitive support') ----- bindProcess: aProcess toId: newId "Change a Process's thread binding and answer 0, otherwise answer a suitable error code. Cases: process is unbound & unaffined + affinity 0 nothing to do + affinity non-zero ensure thread and bind - id 0 nothing to do - id non-zero ensure thread and bind process is affined (temporarily bound to a thread for the duration of a surrender of ownership) + affinity = affined index nothing to do + affinity = 0 nothing to do + affinity ~= 0 && affinity ~= affined index fail - id = affined index nothing to do - id = 0 nothing to do - id ~= 0 && id ~= affined index fail process is bound (permanently bound to a thread) + affinity = bound index nothing to do + affinity ~= bound index set bound index" - id = bound index nothing to do - id ~= bound index set bound index" | threadIdField ownerIndex affinedId | + processHasThreadAffinity ifFalse: - processHasThreadId ifFalse: [^PrimErrUnsupported]. + threadIdField := self threadAffinityFieldOf: aProcess. - threadIdField := self threadIdFieldOf: aProcess. ownerIndex := self ownerIndexOfThreadId: threadIdField. "If aProcess is affined (temporarily bound to) a thread then the operation can only succeed if the newId is the same as that aProcess is affined to, or is zero (is unbinding)." (self isAffinedThreadId: threadIdField) ifTrue: [affinedId := self affinedThreadId: threadIdField. (newId = 0 or: [newId = affinedId]) ifTrue: [self setThreadIdFieldOfProcess: aProcess to: newId << ThreadIdShift + affinedId. ^nil]. ^PrimErrInappropriate]. ownerIndex > 0 ifTrue: [(self startThreadForThreadIndex: ownerIndex) ifFalse: [^PrimErrLimitExceeded]]. self setThreadIdFieldOfProcess: aProcess to: newId << ThreadIdShift. ^nil! Item was changed: ----- Method: CoInterpreterMT>>checkVMOwnershipFromHeartbeat (in category 'process primitive support') ----- checkVMOwnershipFromHeartbeat "Check whether the VM is unowned and needs to set a thread running to try and own it. + Do not attempt this if the image doesn't have a threadAffinity inst var in Process; the VM - Do not attempt this if the image doesn't have a threadId inst var in Process; the VM can't thread these images." <inline: false> self sqLowLevelMFence. + (processHasThreadAffinity - (processHasThreadId and: [cogThreadManager vmIsUnowned]) ifTrue: [cogThreadManager ensureRunningVMThread: relinquishing]! Item was changed: ----- Method: CoInterpreterMT>>disownVM: (in category 'vm scheduling') ----- disownVM: flags "Release the VM to other threads and answer the current thread's index. Currently valid flags: DisownVMForFFICall - informs the VM that it is entering an FFI call DisownVMForThreading - informs the VM that it is entering code during which threading should be permitted OwnVMForeignThreadFlag - indicates lowest-level entry from a foreign thread - not to be used explicitly by clients - only set by ownVMFromUnidentifiedThread VMAlreadyOwnedHenceDoNotDisown - indicates an ownVM from a callback was made when the vm was still owned. - not to be used explicitly by clients - only set by ownVMFromUnidentifiedThread This is the entry-point for plugins and primitives that wish to release the VM while performing some operation that may potentially block, and for callbacks returning back to some blocking operation. If this thread does not reclaim the VM before- hand then when the next heartbeat occurs the thread manager will schedule a thread to acquire the VM which may start running the VM in place of this thread. N.B. Most of the state needed to resume after preemption is set in preemptDisowningThread." <public> <inline: false> | vmThread result | self assert: (flags >= 0 and: [flags < (1 bitShift: DisownFlagsShift)]). self assert: self successful. cogit recordEventTrace ifTrue: [self recordTrace: TraceDisownVM thing: (objectMemory integerObjectOf: flags) source: 0]. + processHasThreadAffinity ifFalse: - processHasThreadId ifFalse: [willNotThreadWarnCount < 10 ifTrue: [self print: 'warning: VM parameter 48 indicates Process doesn''t have threadId; VM will not thread'; cr. willNotThreadWarnCount := willNotThreadWarnCount + 1]]. vmThread := cogThreadManager currentVMThread. (flags anyMask: VMAlreadyOwnedHenceDoNotDisown) ifTrue: [disowningVMThread := vmThread. vmThread state: CTMUnavailable. ^0]. self assertCStackPointersBelongToCurrentThread. self assertValidNewMethodPropertyFlags. self cCode: '' inSmalltalk: [cogThreadManager saveRegisterStateForCurrentProcess. cogThreadManager clearRegisterStates]. (flags anyMask: DisownVMForProcessorRelinquish) ifTrue: [| proc | (proc := objectMemory splObj: foreignCallbackProcessSlot) ~= objectMemory nilObject ifTrue: [foreignCallbackPriority := self quickFetchInteger: PriorityIndex ofObject: proc]. relinquishing := true. self sqLowLevelMFence]. disownCount := disownCount + 1. disowningVMThread := vmThread. "self cr; cr; print: 'disownVM Csp: '; printHex: vmThread cStackPointer; cr. (0 to: 16 by: 4) do: [:offset| self print: ' *(esp+'; printNum: offset; print: ': '; printHex: (stackPages longAt: cogit processor sp + offset); cr]. cogit processor printIntegerRegistersOn: Transcript." "OwnVMForeignThreadFlag indicates lowest-level of entry by a foreign thread. If that's where we are then release the vmThread. Otherwise indicate the vmThread is off doing something outside of the VM." (flags anyMask: OwnVMForeignThreadFlag) ifTrue: ["I don't think this is quite right. Josh's use case is creating some foreign thread and then registering it with the VM. That's not the same as binding a process to a foreign thread given that the foreign callback process is about to terminate anyway (it is returning from a callback here). So do we need an additional concept, that of a vmThread being either of the set known to the VM or floating?" self flag: 'issue with registering foreign threads with the VM'. (self isBoundProcess: self activeProcess) ifFalse: [cogThreadManager unregisterVMThread: vmThread]] ifFalse: [vmThread state: CTMUnavailable]. result := ((vmThread index bitShift: DisownFlagsShift) bitOr: (activeProcessAffined ifTrue: [0] ifFalse: [ProcessUnaffinedOnDisown])) bitOr: flags. cogThreadManager releaseVM. ^result! Item was changed: ----- Method: CoInterpreterMT>>getImageHeaderFlags (in category 'image save/restore') ----- getImageHeaderFlags "Answer the flags that are contained in the 7th long of the image header." ^fullScreenFlag "0 or 1" + (VMBIGENDIAN ifTrue: [0] ifFalse: [2]) "this is the imageFloatsLittleEndian flag" + + (processHasThreadAffinity ifTrue: [4] ifFalse: [0]) - + (processHasThreadId ifTrue: [4] ifFalse: [0]) + (flagInterpretedMethods ifTrue: [8] ifFalse: [0]) + (preemptionYields ifTrue: [0] ifFalse: [16r10]) "was: noThreadingOfGUIThread ifTrue: [16r20] ifFalse: [0]); a broken idea" + (newFinalization ifTrue: [16r40] ifFalse: [0]) + (sendWheelEvents ifTrue: [16r80] ifFalse: [0]) + (primitiveDoMixedArithmetic ifTrue: [0] ifFalse: [16r100]) "N.B. flag mask 16r200 is fileTimesInUTC, responded to by the FilePlugin & FileAttributesPlugin" + (upscaleDisplayIfHighDPI ifTrue: [0] ifFalse: [16r400]) + (imageHeaderFlags bitClear: 16r7FF) "these are any flags we do not recognize"! Item was changed: ----- Method: CoInterpreterMT>>isAffinedProcess: (in category 'process primitive support') ----- isAffinedProcess: aProcess + ^self isAffinedThreadId: (self threadAffinityFieldValueOf: aProcess)! - ^self isAffinedThreadId: (self threadIdFieldValueOf: aProcess)! Item was changed: ----- Method: CoInterpreterMT>>ownerIndexOfProcess: (in category 'process primitive support') ----- ownerIndexOfProcess: aProcess + ^self ownerIndexOfThreadId: (self threadAffinityFieldOf: aProcess)! - ^self ownerIndexOfThreadId: (self threadIdFieldOf: aProcess)! Item was changed: ----- Method: CoInterpreterMT>>primitiveProcessBindToThreadId (in category 'process primitives') ----- primitiveProcessBindToThreadId "Attempt to bind the receiver to the thread with the id of the argument or nil, where the receiver is a Process. If successful the VM will ensure that there are at least id many threads active." | aProcess id | <export: true> self cCode: [] inSmalltalk: [cogThreadManager isNil ifTrue: [^self primitiveFail]]. + processHasThreadAffinity ifFalse: - processHasThreadId ifFalse: [^self primitiveFailFor: PrimErrUnsupported]. id := self stackTop. aProcess := self stackValue: 1. ((id = objectMemory nilObject or: [(objectMemory isIntegerObject: id) and: [id ~= (objectMemory integerObjectOf: 0)]]) and: [(objectMemory isPointers: aProcess) and: [(objectMemory slotSizeOf: aProcess) >= (ThreadIdIndex + 1)]]) ifFalse: [^self primitiveFailFor: PrimErrBadArgument]. id := id = objectMemory nilObject ifTrue: [0] ifFalse: [objectMemory integerValueOf: id]. id abs >= cogThreadManager maxNumThreads ifTrue: [^self primitiveFailFor: PrimErrLimitExceeded]. (self bindProcess: aProcess toId: id) ifNotNil: [:ec| ^self primitiveFailFor: ec]. self methodReturnReceiver. id := self ownerIndexOfProcess: aProcess. (aProcess = self activeProcess and: [(activeProcessAffined := id ~= 0) and: [(cogThreadManager vmOwnerIsCompatibleWith: id) not]]) ifTrue: [(self quickFetchInteger: PriorityIndex ofObject: aProcess) < maxWaitingPriority ifTrue: [maxWaitingPriority = self quickFetchInteger: PriorityIndex ofObject: aProcess]. self threadSwitchIfNecessary: aProcess from: CSThreadBind]! Item was changed: ----- Method: CoInterpreterMT>>primitiveProcessBoundThreadId (in category 'process primitives') ----- primitiveProcessBoundThreadId + "Answer the receiver's current threadAffinity or nil, where the receiver is a Process. + If the threadAffinity is positive then the receiver is bound to the thread with that id. + If the threadAffinity is negative then the receiver is excluded from running on the thread with that id." - "Answer the receiver's current thread Id or nil, where the receiver is a Process. - If the threadId is positive then the receiver is bound to the thread with that id. - If the threadId is negative then the receiver is excluded from running on the thread with that id." | aProcess id | <export: true> self cCode: [] inSmalltalk: [cogThreadManager isNil ifTrue: [^self primitiveFail]]. + processHasThreadAffinity ifFalse: - processHasThreadId ifFalse: [^self primitiveFailFor: PrimErrUnsupported]. aProcess := self stackTop. id := self ownerIndexOfProcess: aProcess. self methodReturnValue: (id = 0 ifTrue: [objectMemory nilObject] ifFalse: [objectMemory integerObjectOf: id])! Item was changed: ----- Method: CoInterpreterMT>>setImageHeaderFlags: (in category 'internal interpreter access') ----- setImageHeaderFlags: flags "Set an array of flags indicating various properties of the saved image, responded to on image load. These are the same as the image header flags shifted right two bits, omitting the fullScreenFlag and float byte order flag. + Bit 0: if set, implies the image's Process class has threadAffinity as its 3rd inst var (zero relative) - Bit 0: if set, implies the image's Process class has threadId as its 3rd inst var (zero relative) Bit 1: if set, methods that are interpreted will have the flag bit set in their header Bit 2: if set, implies preempting a process does not put it to the back of its run queue Bit 3: unassigned; used to mean if set, implies a threaded VM will not disown the VM if owned by the GUI thread; a broken idea Bit 4: if set, implies the new finalization scheme where WeakArrays are queued Bit 5: if set, implies wheel events will be delivered as such and not mapped to arrow key events Bit 6: if set, implies arithmetic primitives will fail if given arguments of different types (float vs int) Bit 7: if set, implies file primitives (FilePlugin, FileAttributesPlugin) will answer file times in UTC not local times Bit 8: if set, implies the VM will not upscale the display on high DPI monitors; older VMs did this by default." flags asUnsignedInteger > 511 ifTrue: [^self primitiveFailFor: PrimErrUnsupported]. (flags anyMask: 8) ifTrue: [^self primitiveFailFor: PrimErrInappropriate]. + processHasThreadAffinity := flags anyMask: 1. - processHasThreadId := flags anyMask: 1. flagInterpretedMethods := flags anyMask: 2. preemptionYields := flags noMask: 4. "was: noThreadingOfGUIThread := flags anyMask: 8. a broken idea" newFinalization := flags anyMask: 16. sendWheelEvents := flags anyMask: 32. primitiveDoMixedArithmetic := flags noMask: 64. imageHeaderFlags := (flags anyMask: 128) ifTrue: [imageHeaderFlags bitOr: 512] ifFalse: [imageHeaderFlags bitClear: 512]. upscaleDisplayIfHighDPI := flags noMask: 256! Item was changed: ----- Method: CoInterpreterMT>>setImageHeaderFlagsFrom: (in category 'image save/restore') ----- setImageHeaderFlagsFrom: headerFlags "Set the flags that are contained in the 7th long of the image header." imageHeaderFlags := headerFlags. "so as to preserve unrecognised flags." fullScreenFlag := headerFlags bitAnd: 1. imageFloatsBigEndian := (headerFlags noMask: 2) ifTrue: [1] ifFalse: [0]. + processHasThreadAffinity := headerFlags anyMask: 4. - processHasThreadId := headerFlags anyMask: 4. flagInterpretedMethods := headerFlags anyMask: 8. preemptionYields := headerFlags noMask: 16. "was: noThreadingOfGUIThread := headerFlags anyMask: 32. a broken idea" newFinalization := headerFlags anyMask: 64. sendWheelEvents := headerFlags anyMask: 128. primitiveDoMixedArithmetic < 0 ifTrue: "i.e. has it not been set on the command line?" [primitiveDoMixedArithmetic := headerFlags noMask: 256]. "N.B. flag mask 512 is responded to by the FilePlugin & FileAttributesPlugin" upscaleDisplayIfHighDPI < 0 ifTrue: "i.e. has it not been set on the command line?" [upscaleDisplayIfHighDPI := headerFlags noMask: 1024]. + processHasThreadAffinity ifFalse: + [self print: 'warning, processHasThreadAffinity flag is unset; cannot function as a threaded VM if so.'; cr]! - processHasThreadId ifFalse: - [self print: 'warning, processHasThreadId flag is unset; cannot function as a threaded VM if so.'; cr]! Item was added: + ----- Method: CoInterpreterMT>>threadAffinityFieldOf: (in category 'process primitive support') ----- + threadAffinityFieldOf: aProcess + ^processHasThreadAffinity + ifTrue: [objectMemory fetchPointer: ThreadIdIndex ofObject: aProcess] + ifFalse: [objectMemory nilObject]! Item was added: + ----- Method: CoInterpreterMT>>threadAffinityFieldValueOf: (in category 'process primitive support') ----- + threadAffinityFieldValueOf: aProcess + ^processHasThreadAffinity + ifTrue: + [| field | + field := objectMemory fetchPointer: ThreadIdIndex ofObject: aProcess. + field = objectMemory nilObject + ifTrue: [0] + ifFalse: [objectMemory integerValueOf: field]] + ifFalse: [0]! Item was removed: - ----- Method: CoInterpreterMT>>threadIdFieldOf: (in category 'process primitive support') ----- - threadIdFieldOf: aProcess - ^processHasThreadId - ifTrue: [objectMemory fetchPointer: ThreadIdIndex ofObject: aProcess] - ifFalse: [objectMemory nilObject]! Item was removed: - ----- Method: CoInterpreterMT>>threadIdFieldValueOf: (in category 'process primitive support') ----- - threadIdFieldValueOf: aProcess - ^processHasThreadId - ifTrue: - [| field | - field := objectMemory fetchPointer: ThreadIdIndex ofObject: aProcess. - field = objectMemory nilObject - ifTrue: [0] - ifFalse: [objectMemory integerValueOf: field]] - ifFalse: [0]! Item was added: + ----- Method: CogVMSimulator>>threadAffinityFieldOf: (in category 'multi-threading simulation switch') ----- + threadAffinityFieldOf: aProcess + "This method includes or excludes CoInterpreterMT methods as required. + Auto-generated by CogVMSimulator>>ensureMultiThreadingOverridesAreUpToDate" + + ^self perform: #threadAffinityFieldOf: + withArguments: {aProcess} + inSuperclass: (cogThreadManager ifNil: [CoInterpreterPrimitives] ifNotNil: [CoInterpreterMT])! Item was removed: - ----- Method: CogVMSimulator>>threadIdFieldOf: (in category 'multi-threading simulation switch') ----- - threadIdFieldOf: aProcess - "This method includes or excludes CoInterpreterMT methods as required. - Auto-generated by CogVMSimulator>>ensureMultiThreadingOverridesAreUpToDate" - - ^self perform: #threadIdFieldOf: - withArguments: {aProcess} - inSuperclass: (cogThreadManager ifNil: [CoInterpreterPrimitives] ifNotNil: [CoInterpreterMT])! Item was changed: ----- Method: StackInterpreter>>setImageHeaderFlags: (in category 'internal interpreter access') ----- setImageHeaderFlags: flags "Set an array of flags indicating various properties of the saved image, responded to on image load. These are the same as the image header flags shifted right two bits, omitting the fullScreenFlag and float byte order flag. + Bit 0: if set, implies the image's Process class has threadAffinity as its 3rd inst var (zero relative) (meaningful to the MT VM only) - Bit 0: if set, implies the image's Process class has threadId as its 3rd inst var (zero relative) (meaningful to the MT VM only) Bit 1: if set, methods that are interpreted will have the flag bit set in their header (meaningful to the Cog VM only) Bit 2: if set, implies preempting a process does not put it to the back of its run queue Bit 3: if set, implies a threaded VM will not disown the VM if owned by the GUI thread (meaningful to the MT VM only) Bit 4: if set, implies the new finalization scheme where WeakArrays are queued Bit 5: if set, implies wheel events will be delivered as such and not mapped to arrow key events Bit 6: if set, implies arithmetic primitives will fail if given arguments of different types (float vs int) Bit 7: if set, implies file primitives (FilePlugin, FileAttributesPlugin) will answer file times in UTC not local times Bit 8: if set, implies the VM will not upscale the display on high DPI monitors; older VMs did this by default." flags asUnsignedInteger > 511 ifTrue: [^self primitiveFailFor: PrimErrUnsupported]. + "threadAffinity := flags anyMask: 1. specific to CoInterpreterMT" - "processHasThreadId := flags anyMask: 1. specific to CoInterpreterMT" imageHeaderFlags := (flags anyMask: 1) ifTrue: [imageHeaderFlags bitOr: 4] ifFalse: [imageHeaderFlags bitClear: 4]. "flagInterpretedMethods := flags anyMask: 2. specific to CoInterpreter" imageHeaderFlags := (flags anyMask: 2) ifTrue: [imageHeaderFlags bitOr: 8] ifFalse: [imageHeaderFlags bitClear: 8]. preemptionYields := flags noMask: 4. "noThreadingOfGUIThread := flags anyMask: 8.. specific to CoInterpreterMT" imageHeaderFlags := (flags anyMask: 8) ifTrue: [imageHeaderFlags bitOr: 32] ifFalse: [imageHeaderFlags bitClear: 32]. newFinalization := flags anyMask: 16. sendWheelEvents := flags anyMask: 32. primitiveDoMixedArithmetic := flags noMask: 64. imageHeaderFlags := (flags anyMask: 128) ifTrue: [imageHeaderFlags bitOr: 512] ifFalse: [imageHeaderFlags bitClear: 512]. upscaleDisplayIfHighDPI := flags noMask: 256!

1 0

VM Maker: VMMaker.oscog-LM.3334.mcz
by commits＠source.squeak.org 22 Aug '23

22 Aug '23

Eliot Miranda uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-LM.3334.mcz ==================== Summary ==================== Name: VMMaker.oscog-LM.3334 Author: eem Time: 22 August 2023, 1:09:54.627877 pm UUID: 82305b00-ff38-47b9-978b-a25dc44258c8 Ancestors: VMMaker.oscog-eem.3333 First commit of Leon Matthes' work to revive the threaded FFI (committed by eem on behalf of LM cuz of Monticello permissions on source.squeak.org). Fix simulation so that the current processor register state reflects the current Smalltalk process. Eliot's first attempt was a bit broken. This approach of manually switching register states in tryLockVMOwnerTo: preserves fast simulation because we're not changing register state on every send to the processor object as was the case with the original MultiProcessor wrapper. Check for a missing register state by setting the register state for stack, frame & pc pointers to zero. Label the processes spawned by the simulation so they show up nicely in the process browser. releaseVM was used several places where disownVM: should have been used. Simplify cedeToHigherPriorityThreads; the VM can't be unowned when invoked. Nuke some unneeded halts. =============== Diff against VMMaker.oscog-eem.3333 =============== Item was changed: ----- Method: CoInterpreterMT>>cedeToHigherPriorityThreads (in category 'process primitive support') ----- cedeToHigherPriorityThreads "Invoked from checkForEventsMayContextSwitch: to switch threads if a thread wanting to acquire the VM has higher priority than the active process." + | activeProc processAffinity activeContext activePriority activeThread vmThread | - | activeProc ownerIndex activeContext activePriority activeThread vmThread | <var: #activeThread type: #'CogVMThread *'> <var: #vmThread type: #'CogVMThread *'> <inline: false> activeProc := self activeProcess. activePriority := self quickFetchInteger: PriorityIndex ofObject: activeProc. + processAffinity := self ownerIndexOfProcess: activeProc. + activeThread := cogThreadManager currentVMThread. + self assert: (cogThreadManager threadIndex: activeThread index isCompatibleWith: processAffinity). + - ownerIndex := self ownerIndexOfProcess: activeProc. - ownerIndex = 0 - ifTrue: [activeThread := cogThreadManager currentVMThread] - ifFalse: [activeThread := cogThreadManager vmThreadAt: ownerIndex]. activeThread priority: activePriority. vmThread := cogThreadManager highestPriorityThreadIfHigherThan: activePriority expectedMax: maxWaitingPriority. (vmThread isNil "no waiting thread of sufficiently high priority. Do not switch." or: [vmThread = activeThread]) "The activeProcess needs to run on a different thread. Leave this to threadSwitchIfNecessary:from: in checkForEventsMayContextSwitch:" ifTrue: [maxWaitingPriority > activePriority ifTrue: [maxWaitingPriority := activePriority]. ^self]. self assert: vmThread priority > activePriority. self assert: vmThread ~= cogThreadManager currentVMThread. self assertValidExecutionPointe: instructionPointer r: framePointer s: stackPointer. maxWaitingPriority > vmThread priority ifTrue: [maxWaitingPriority := vmThread priority]. statProcessSwitch := statProcessSwitch + 1. activeContext := self ensureFrameIsMarried: framePointer SP: stackPointer. objectMemory storePointer: SuspendedContextIndex ofObject: activeProc withValue: activeContext. self ensurePushedInstructionPointer. self externalWriteBackHeadFramePointers. self putToSleep: activeProc yieldingIf: preemptionYields. "Transcript cr; print: #cedeToHighestPriorityThreadIfHigherThan:; cr. self printExternalHeadFrame. self print: 'ip: '; printHex: self instructionPointer. Transcript cr; flush." self returnToSchedulingLoopAndReleaseVMOrWakeThread: vmThread source: CSCheckEvents! Item was changed: ----- Method: CoInterpreterMT>>disownVM: (in category 'vm scheduling') ----- disownVM: flags "Release the VM to other threads and answer the current thread's index. Currently valid flags: DisownVMForFFICall - informs the VM that it is entering an FFI call DisownVMForThreading - informs the VM that it is entering code during which threading should be permitted OwnVMForeignThreadFlag - indicates lowest-level entry from a foreign thread - not to be used explicitly by clients - only set by ownVMFromUnidentifiedThread VMAlreadyOwnedHenceDoNotDisown - indicates an ownVM from a callback was made when the vm was still owned. - not to be used explicitly by clients - only set by ownVMFromUnidentifiedThread This is the entry-point for plugins and primitives that wish to release the VM while performing some operation that may potentially block, and for callbacks returning back to some blocking operation. If this thread does not reclaim the VM before- hand then when the next heartbeat occurs the thread manager will schedule a thread to acquire the VM which may start running the VM in place of this thread. N.B. Most of the state needed to resume after preemption is set in preemptDisowningThread." <public> <inline: false> | vmThread result | self assert: (flags >= 0 and: [flags < (1 bitShift: DisownFlagsShift)]). self assert: self successful. cogit recordEventTrace ifTrue: [self recordTrace: TraceDisownVM thing: (objectMemory integerObjectOf: flags) source: 0]. processHasThreadId ifFalse: [willNotThreadWarnCount < 10 ifTrue: [self print: 'warning: VM parameter 48 indicates Process doesn''t have threadId; VM will not thread'; cr. willNotThreadWarnCount := willNotThreadWarnCount + 1]]. vmThread := cogThreadManager currentVMThread. (flags anyMask: VMAlreadyOwnedHenceDoNotDisown) ifTrue: [disowningVMThread := vmThread. vmThread state: CTMUnavailable. ^0]. self assertCStackPointersBelongToCurrentThread. self assertValidNewMethodPropertyFlags. self cCode: '' inSmalltalk: + [cogThreadManager saveRegisterStateForCurrentProcess. + cogThreadManager clearRegisterStates]. - [cogThreadManager saveRegisterStateForCurrentProcess]. (flags anyMask: DisownVMForProcessorRelinquish) ifTrue: [| proc | (proc := objectMemory splObj: foreignCallbackProcessSlot) ~= objectMemory nilObject ifTrue: [foreignCallbackPriority := self quickFetchInteger: PriorityIndex ofObject: proc]. relinquishing := true. self sqLowLevelMFence]. disownCount := disownCount + 1. disowningVMThread := vmThread. "self cr; cr; print: 'disownVM Csp: '; printHex: vmThread cStackPointer; cr. (0 to: 16 by: 4) do: [:offset| self print: ' *(esp+'; printNum: offset; print: ': '; printHex: (stackPages longAt: cogit processor sp + offset); cr]. cogit processor printIntegerRegistersOn: Transcript." "OwnVMForeignThreadFlag indicates lowest-level of entry by a foreign thread. If that's where we are then release the vmThread. Otherwise indicate the vmThread is off doing something outside of the VM." (flags anyMask: OwnVMForeignThreadFlag) ifTrue: ["I don't think this is quite right. Josh's use case is creating some foreign thread and then registering it with the VM. That's not the same as binding a process to a foreign thread given that the foreign callback process is about to terminate anyway (it is returning from a callback here). So do we need an additional concept, that of a vmThread being either of the set known to the VM or floating?" self flag: 'issue with registering foreign threads with the VM'. (self isBoundProcess: self activeProcess) ifFalse: [cogThreadManager unregisterVMThread: vmThread]] ifFalse: [vmThread state: CTMUnavailable]. result := ((vmThread index bitShift: DisownFlagsShift) bitOr: (activeProcessAffined ifTrue: [0] ifFalse: [ProcessUnaffinedOnDisown])) bitOr: flags. cogThreadManager releaseVM. ^result! Item was changed: ----- Method: CoInterpreterMT>>ownVMFromUnidentifiedThread (in category 'vm scheduling') ----- ownVMFromUnidentifiedThread "Attempt to take ownership from a thread that as yet doesn't know its index. This supports callbacks where the callback could originate from any thread. Answer 0 if the owning thread is known to the VM. Answer 1 if the owning thread is unknown to the VM and now owns the VM. Answer -1 if the owning thread is unknown to the VM and fails to own the VM. Answer -2 if the owning thread is unknown to the VM and there is no foreign callback process installed." | count threadIndex vmThread | <var: #vmThread type: #'CogVMThread *'> <inline: false> + + self cCode: [] inSmalltalk: [self halt: 'TODO: Implement processor register switching']. (threadIndex := cogThreadManager ioGetThreadLocalThreadIndex) ~= 0 ifTrue: [ "this is a callback from a known thread" (cogThreadManager vmOwnerIs: threadIndex) ifTrue: "the VM has not been disowned" [self assert: (disowningVMThread isNil or: [disowningVMThread = self currentVMThread]). disowningVMThread := nil. self currentVMThread state: CTMAssignableOrInVM. ^VMAlreadyOwnedHenceDoNotDisown]. ^self ownVM: threadIndex]. foreignCallbackPriority = 0 ifTrue: [^-2]. count := 0. "If the current thread doesn't have an index it's new to the vm and we need to allocate a new threadInfo, failing if we can't. We also need a process in the foreignCallbackProcessSlot upon which to run the thread's eventual callback." [[cogThreadManager tryLockVMOwnerTo: cogThreadManager ioCurrentOSThread asUnsignedInteger] whileFalse: [self waitingPriorityIsAtLeast: foreignCallbackPriority. cogThreadManager ioTransferTimeslice]. (objectMemory splObj: foreignCallbackProcessSlot) ~= objectMemory nilObject] whileFalse: [cogThreadManager releaseVM. (count := count + 1) > 1000 ifTrue: [^-2]. cogThreadManager ioMilliSleep: 1]. vmThread := cogThreadManager unusedThreadInfo. "N.B. Keep the VM locked anonymously so that we reserve the non-nil ForeignCallbackProcess for this thread, avoiding the race between competing foreign callbacks. The acquireVMFor: in ownVM: will set the vmOwner to the actual index. So only unlock on failure." vmThread ifNil: [cogThreadManager releaseVM. ^-1]. cogThreadManager setVMOwner: vmThread index. vmThread state: CTMWantingOwnership; priority: foreignCallbackPriority. cogThreadManager registerVMThread: vmThread. ^self ownVM: vmThread index + OwnVMForeignThreadFlag! Item was changed: ----- Method: CoInterpreterMT>>primitiveProcessBindToThreadId (in category 'process primitives') ----- primitiveProcessBindToThreadId "Attempt to bind the receiver to the thread with the id of the argument or nil, where the receiver is a Process. If successful the VM will ensure that there are at least id many threads active." | aProcess id | <export: true> self cCode: [] inSmalltalk: [cogThreadManager isNil ifTrue: [^self primitiveFail]]. processHasThreadId ifFalse: [^self primitiveFailFor: PrimErrUnsupported]. id := self stackTop. aProcess := self stackValue: 1. ((id = objectMemory nilObject or: [(objectMemory isIntegerObject: id) and: [id ~= (objectMemory integerObjectOf: 0)]]) and: [(objectMemory isPointers: aProcess) and: [(objectMemory slotSizeOf: aProcess) >= (ThreadIdIndex + 1)]]) ifFalse: [^self primitiveFailFor: PrimErrBadArgument]. id := id = objectMemory nilObject ifTrue: [0] ifFalse: [objectMemory integerValueOf: id]. id abs >= cogThreadManager maxNumThreads ifTrue: [^self primitiveFailFor: PrimErrLimitExceeded]. (self bindProcess: aProcess toId: id) ifNotNil: [:ec| ^self primitiveFailFor: ec]. self methodReturnReceiver. - self halt. id := self ownerIndexOfProcess: aProcess. (aProcess = self activeProcess and: [(activeProcessAffined := id ~= 0) and: [(cogThreadManager vmOwnerIsCompatibleWith: id) not]]) ifTrue: [(self quickFetchInteger: PriorityIndex ofObject: aProcess) < maxWaitingPriority ifTrue: [maxWaitingPriority = self quickFetchInteger: PriorityIndex ofObject: aProcess]. self threadSwitchIfNecessary: aProcess from: CSThreadBind]! Item was changed: ----- Method: CoInterpreterMT>>returnToSchedulingLoopAndReleaseVMOrWakeThread:source: (in category 'process primitive support') ----- returnToSchedulingLoopAndReleaseVMOrWakeThread: vmThread source: source <var: #vmThread type: #'CogVMThread *'> <inline: false> self recordThreadSwitchTo: (vmThread ifNotNil: [vmThread index] ifNil: [0]) source: source. vmThread ifNotNil: [cogThreadManager wakeVMThreadFor: vmThread index] + ifNil: [cogit disownVM: DisownVMForThreading]. - ifNil: [cogThreadManager releaseVM]. "I am not frightened of flying. Any value will do. I don't mind. Why should I be frightened of flying? There's no reason for it." self _longjmp: reenterThreadSchedulingLoop _: 1 ! Item was changed: ----- Method: CoInterpreterMT>>threadSchedulingLoopImplementation: (in category 'vm scheduling') ----- threadSchedulingLoopImplementation: vmThread "Enter a loop attempting to run the VM with the highest priority process and blocking on the thread's OS semaphore when unable to run that process. We will return to this via threadSwitchIfNecessary:from: which is called in the middle of transferTo:from: once the active process has been stored in the scheduler." <var: #vmThread type: #'CogVMThread *'> <inline: false> self _setjmp: reenterThreadSchedulingLoop. [self assert: vmThread state = CTMAssignableOrInVM. (cogThreadManager tryLockVMOwnerTo: vmThread index) ifTrue: ["If relinquishing is true, then primitiveRelinquishProcessor has disowned the VM and only a returning call or callback should take ownership in that case." relinquishing + ifTrue: [cogThreadManager disownVM: DisownVMForThreading] - ifTrue: [cogThreadManager releaseVM] ifFalse: [self tryToExecuteSmalltalk: vmThread]]. (cogThreadManager vmOwnerIs: vmThread index) ifFalse: [cogThreadManager waitForWork: vmThread]. true] whileTrue! Item was changed: ----- Method: CoInterpreterMT>>tryToExecuteSmalltalk: (in category 'vm scheduling') ----- tryToExecuteSmalltalk: vmThread "Attempt to run the current process, if it exists, on the given vmThread." <var: #vmThread type: #'CogVMThread *'> | dvmt activeProc ownerIndex | <var: #dvmt type: #'CogVMThread *'> self assert: (cogThreadManager vmOwnerIs: vmThread index). self assert: cogThreadManager ioGetThreadLocalThreadIndex = vmThread index. dvmt := disowningVMThread. disowningVMThread ifNil: [activeProc := self activeProcess] ifNotNil: [self preemptDisowningThread. activeProc := self wakeHighestPriority. activeProc ifNil: [activeProc := objectMemory nilObject] ifNotNil: [objectMemory storePointerUnchecked: MyListIndex ofObject: activeProc withValue: objectMemory nilObject]. objectMemory storePointer: ActiveProcessIndex ofObject: self schedulerPointer withValue: activeProc]. activeProc = objectMemory nilObject ifTrue: + [cogThreadManager disownVM: DisownVMForThreading. - [cogThreadManager releaseVM. ^nil]. ownerIndex := self ownerIndexOfProcess: activeProc. (ownerIndex = 0 or: [cogThreadManager vmOwnerIsCompatibleWith: ownerIndex]) ifTrue: [self assert: (objectMemory fetchPointer: MyListIndex ofObject: self activeProcess) = objectMemory nilObject. (objectMemory fetchPointer: SuspendedContextIndex ofObject: activeProc) ~= objectMemory nilObject ifTrue: [self externalSetStackPageAndPointersForSuspendedContextOfProcess: activeProc]. instructionPointer = cogit ceReturnToInterpreterPC ifTrue: [self deny: (self isMachineCodeFrame: framePointer). instructionPointer := self iframeSavedIP: framePointer]. self enterSmalltalkExecutive. "When we return here we should have already given up the VM and so we cannot touch any interpreter state." "NOTREACHED"]. cogThreadManager wakeVMThreadFor: ownerIndex! Item was added: + ----- Method: CogThreadManager>>assertEmptyRegisterStates: (in category 'simulation') ----- + assertEmptyRegisterStates: registerState + + self assert: (registerState at: cogit processor registerStateFPIndex) == 0. + self assert: (registerState at: cogit processor registerStateSPIndex) == 0. + self assert: (registerState at: cogit processor registerStatePCIndex) == 0.! Item was added: + ----- Method: CogThreadManager>>clearRegisterStates (in category 'simulation') ----- + clearRegisterStates + <doNotGenerate> + |setters| + setters := cogit processor registerStateSetters. + cogit processor perform: (setters at: cogit processor registerStateFPIndex) with: 0. + cogit processor perform: (setters at: cogit processor registerStateSPIndex) with: 0. + cogit processor perform: (setters at: cogit processor registerStatePCIndex) with: 0. + self assertEmptyRegisterStates: cogit processor registerState.! Item was changed: ----- Method: CogThreadManager>>ioNewOS:Thread: (in category 'simulation') ----- ioNewOS: function Thread: vmThread "Fork a new thread to evaluate function with the supplied argument. Return 0 on success and non-zero on error. The only errors equate to running out of resources, ie. memory. See platforms/Cross/vm/sq.h for the real definition." <doNotGenerate> + [self perform: function with: vmThread] forkNamed: 'CogVMThread (', vmThread index, ')'. - [self perform: function with: vmThread] fork. ^0! Item was changed: ----- Method: CogThreadManager>>ioTransferTimeslice (in category 'simulation') ----- ioTransferTimeslice <doNotGenerate> "See platforms/Cross/vm/sq.h for the real definition." - self saveRegisterStateForCurrentProcess. Processor yield! Item was changed: ----- Method: CogThreadManager>>ioWaitOnOSSemaphore: (in category 'simulation') ----- ioWaitOnOSSemaphore: aSemaphorePtr <var: #anOSSemaphore type: #'sqOSSemaphore *'> <returnTypeC: #void> <doNotGenerate> "See platforms/Cross/vm/sq.h for the real definition." "Simulate the VM's heartbeat by calling checkVMOwnershipFromHeartbeat if the wait times-out." - self saveRegisterStateForCurrentProcess. - [aSemaphorePtr value waitTimeoutMSecs: 1000] whileTrue: [coInterpreter checkVMOwnershipFromHeartbeat]! Item was added: + ----- Method: CogThreadManager>>loadOrInitializeRegisterStateFor: (in category 'simulation') ----- + loadOrInitializeRegisterStateFor: threadIndex + <doNotGenerate> + |processor| + processor := cogit processor. + processor setRegisterState: (registerStates + at: threadIndex + ifAbsentPut: + [self initializeProcessor: processor forThreadIndex: threadIndex. + processor registerState]).! Item was changed: ----- Method: CogThreadManager>>registerVMThread: (in category 'scheduling') ----- registerVMThread: vmThread "Register the current thread in the set of threads, initializing the supplied vmThread and setting the thread's thread block index there-to." <var: #vmThread type: #'CogVMThread *'> <returnTypeC: #'CogVMThread *'> + self assert: ((vmThread state = CTMInitializing + or: [vmThread state = CTMWantingOwnership]) and: [vmThread osThread isNil]). - self assert: (vmThread state = CTMInitializing - or: [vmThread state = CTMWantingOwnership]). vmThread osThread: self ioCurrentOSThread. self ioSetThreadLocalThreadIndex: vmThread index. self assert: self ioGetThreadLocalThreadIndex = vmThread index. ^vmThread! Item was changed: ----- Method: CogThreadManager>>saveRegisterStateForCurrentProcess (in category 'simulation') ----- saveRegisterStateForCurrentProcess "On switching osProcess we have to both - SAVE old process's register state - RESTORE new process's register state So what are the transitions? The transitions out (where processor state must be saved) are the opposites of tryLockVMOwnerTo:. Hence - waitOnOSSemaphore: - disownVM: - ioTransferTimeslice i.e. the continuations from here, disownVM:, and ioTransferTimeslice, will use tryLockVMOwnerTo: to regain control of the VM. So the register state to be restored at that point (if tryLockVMOwnerTo: succeeds) is the register state saved in one of the three places. But the processor is initialized in tryLockVMOwnerTo:, so only save the state if state is already present." <doNotGenerate> cogit withProcessorHaltedDo: [| currentVMThread state | currentVMThread := self vmThreadForCurrentProcess. + state := cogit processor registerState. + self assertValidStackPointersInState: state forIndex: currentVMThread index. + registerStates at: currentVMThread index put: state]! - (registerStates includesKey: currentVMThread index) ifTrue: - [state := cogit processor registerState. - self assertValidStackPointersInState: state forIndex: currentVMThread index. - registerStates at: currentVMThread index put: state]]! Item was changed: ----- Method: CogThreadManager>>setVMOwner: (in category 'public api') ----- setVMOwner: indexOrZero "An ugly accessor used in only three cases: 1. by ownVMFromUnidentifiedThread when the VM is first locked to the thread id of the unidentified thread, and then, once identified, to the thread's index. 2. by wakeVMThreadFor: used by the two-level scheduler to switch threads when a Smalltalk process switch occurs to a process affined to another thread. 3. to release the VM (set the owner to zero)" <inline: #always> + self assert: (vmOwner ~= 0 and: [vmOwner ~= indexOrZero]). self cCode: '' inSmalltalk: [coInterpreter transcript ensureCr; f: 'setVMOwner: %d -> %d (%s)\n' printf: { vmOwner. indexOrZero. thisContext home sender selector }]. + + "With the assignment followed by a LowLevelMFence this is equivalent to a + Fetch-Assign with an C++ Memory ordering of Squentially-Consistent. + TODO: We could make this a `release` ordering, which may perform better on ARM." vmOwner := indexOrZero. self sqLowLevelMFence! Item was changed: ----- Method: CogThreadManager>>threadIndex:isCompatibleWith: (in category 'public api-testing') ----- threadIndex: aThreadIndex isCompatibleWith: processThreadId "Test if processThreadId is ok to run on a thread with the given index." <inline: true> self assert: aThreadIndex > 0. + ^processThreadId = 0 + or: [processThreadId >= 0 + ifTrue: [aThreadIndex = processThreadId] + ifFalse: [aThreadIndex ~= processThreadId negated]]! - ^processThreadId >= 0 - ifTrue: [aThreadIndex = processThreadId] - ifFalse: [aThreadIndex ~= processThreadId negated]! Item was changed: ----- Method: CogThreadManager>>tryLockVMOwnerTo: (in category 'simulation') ----- tryLockVMOwnerTo: threadIndex "In the real VM this is a direct call of Cogit>>#tryLockVMOwnerTo:/ceTryLockVMOwner. In the simulation this is where register state is restored, simulating a thread switch. State is stored in saveRegisterStateForCurrentProcess (sent by disownVM:, ioWaitOnOSSemaphore: and ioTransferTimeslice). The code here and in saveRegisterStateForCurrentProcess allow us to avoid the expensive and complex MultiProcessor hack. The idea here is to save the register state around the invocation of tryLockVMOwnerTo:, and set the register state to that for the owner, changing the state if ownership has changed, restoring it if ownership has not changed." <doNotGenerate> self deny: threadIndex = 0. ^cogit withProcessorHaltedDo: + [| previousOwner currentOwner processor result priorState | - [| currentOwner prior processor result | processor := cogit processor. + "After switching, the 'current' owner will be the 'previous' owner. + Though the value will be the same, let's still introduce a second variable that we + can use after the switch to make it more clear what's going on." + previousOwner := currentOwner := vmOwner. + + priorState := processor registerState. + "If we currently have a VM owner, the register state should be + valid for that owner, otherwise it should be empty." + vmOwner ~= 0 + ifTrue: [self assertValidStackPointersInState: priorState forIndex: vmOwner] + ifFalse: [self assertEmptyRegisterStates: priorState]. + + vmOwner ~= threadIndex ifTrue: + ["There can be two cases here: + 1. There's currently a VM owner, and the processor has a non-empty state. + 2. There is currently no VM owner and the processor has an empty state. + In both cases, we need to restore our own register state into the processor, + as cogit>>#tryLockVMOwnerTo: call will use the processor, so the processor + cannot be empty and we also don't want the processor modify the current + owners state." + self loadOrInitializeRegisterStateFor: threadIndex]. + - currentOwner := vmOwner. - vmOwner ~= 0 ifTrue: - [prior := processor registerState. - self assertValidStackPointersInState: prior forIndex: vmOwner]. - processor setRegisterState: (registerStates - at: threadIndex - ifAbsentPut: - [self initializeProcessor: processor forThreadIndex: threadIndex. - processor registerState]). result := cogit tryLockVMOwnerTo: threadIndex. self assert: result = (threadIndex = vmOwner). + result + ifTrue: ["If we did actually change owners, assert that previously the processor was emtpy." + previousOwner ~= vmOwner ifTrue: [self assertEmptyRegisterStates: priorState]] + ifFalse: [processor setRegisterState: priorState.]. + - ifTrue: [registerStates at: threadIndex put: processor registerState] - ifFalse: [prior ifNotNil: [processor setRegisterState: prior]]. coInterpreter transcript ensureCr; f: (result ifTrue: ['tryLockVMOwner %d -> %d (%s) ok\n'] ifFalse: ['tryLockVMOwner %d -> %d (%s) FAILED\n']) + printf: { previousOwner. threadIndex. thisContext home sender selector }. - printf: { vmOwner. threadIndex. thisContext home sender selector }. self assertValidProcessorStackPointersForIndex: vmOwner. result]! Item was changed: ----- Method: CogThreadManager>>wakeVMThreadFor: (in category 'public api') ----- wakeVMThreadFor: index "Transfer the VM to the thread with index. Called from a thread that finds the highest priority runnable process is bound to the thread with index index." <returnTypeC: #void> | vmThread | self assert: (self vmIsOwned and: [(self vmOwnerIs: index) not]). self assert: (index between: 1 and: numThreads). - self setVMOwner: index. vmThread := threads at: index. + + "Instead of going through a #disownVM: call, directly set the new VM owner. + This has the advantage of avoiding a race for the different threads to become the new + VM owner. + In Simulation, this means we need to simulate a thread-switch." + self cCode: [] inSmalltalk: + [self saveRegisterStateForCurrentProcess. + self loadOrInitializeRegisterStateFor: index]. + self setVMOwner: index. + vmThread state ifNil: [self startThreadForThreadInfo: vmThread] ifNotNil: [self assert: (vmThread state = CTMWantingOwnership or: [vmThread state = CTMAssignableOrInVM]). self ioSignalOSSemaphore: (self addressOf: vmThread osSemaphore)]. self ioTransferTimeslice!

1 0

VM Maker: VMMaker.oscog-eem.3333.mcz
by commits＠source.squeak.org 12 Aug '23

12 Aug '23

Eliot Miranda uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-eem.3333.mcz ==================== Summary ==================== Name: VMMaker.oscog-eem.3333 Author: eem Time: 11 August 2023, 5:59:12.558349 pm UUID: 9113291a-a570-49e7-bbef-52142bb19946 Ancestors: VMMaker.oscog-eem.3332 minor clean-up/commentary. Help myself by documenting what the snapshotPrimitive answers (false for snapshot, true for resume). Simplify SpurMemoryManager>>#sufficientSpaceAfterGC:. Nuke an unused link reg related utility in the Cogit. Add the CogVMSimulator's fklush method to the STackInterpreterSimulator to ensure that GC progress is printed immediately while simulating. =============== Diff against VMMaker.oscog-eem.3332 =============== Item was changed: ----- Method: CoInterpreterPrimitives>>primitiveSnapshot (in category 'system control primitives') ----- primitiveSnapshot "Save a normal snapshot under the same name as it was loaded unless it has been renamed by the last primitiveImageName. + Note that when executed this primitive answers false, but when the resulting image is run afresh, the primitive answers true. Override to jump to the interpreter because the machine code zone is now void." <inline: false> self snapshot: false. (self iframeMethod: framePointer) = newMethod ifTrue: ["snapshot: has reached the end and built a frame. In the JIT we need to back-up the pc before reentering the interpreter." instructionPointer := instructionPointer - 1]. cogit ceInvokeInterpret "NOTREACHED"! Item was changed: ----- Method: CoInterpreterPrimitives>>primitiveSnapshotEmbedded (in category 'system control primitives') ----- primitiveSnapshotEmbedded "Save an embedded snapshot. + Note that when executed this primitive answers false, but when the resulting image is run afresh, the primitive answers true. Override to jump to the interpreter because the machine code zone is now void." <inline: false> self snapshot: true. (self iframeMethod: framePointer) = newMethod ifTrue: ["snapshot: has reached the end and built a frame. In the JIT we need to back-up the pc before reentering the interpreter." instructionPointer := instructionPointer - 1]. cogit ceInvokeInterpret "NOTREACHED"! Item was removed: - ----- Method: CogARMCompiler>>saveAndRestoreLinkRegUsingCalleeSavedRegNotLiveAtPointOfSendAround: (in category 'abi') ----- - saveAndRestoreLinkRegUsingCalleeSavedRegNotLiveAtPointOfSendAround: aBlock - "Extra1Reg is callee-saved and not live at point of send." - <inline: #always> - | inst | - inst := cogit MoveR: LinkReg R: Extra1Reg. - aBlock value. - cogit MoveR: Extra1Reg R: LinkReg. - ^inst! Item was removed: - ----- Method: CogARMv8Compiler>>saveAndRestoreLinkRegUsingCalleeSavedRegNotLiveAtPointOfSendAround: (in category 'abi') ----- - saveAndRestoreLinkRegUsingCalleeSavedRegNotLiveAtPointOfSendAround: aBlock - "Extra7Reg is callee-saved and not live at point of send." - <inline: #always> - | inst | - inst := cogit MoveR: LinkReg R: Extra8Reg. - aBlock value. - cogit MoveR: Extra8Reg R: LinkReg! Item was removed: - ----- Method: CogAbstractInstruction>>saveAndRestoreLinkRegUsingCalleeSavedRegNotLiveAtPointOfSendAround: (in category 'abi') ----- - saveAndRestoreLinkRegUsingCalleeSavedRegNotLiveAtPointOfSendAround: aBlock - "If the processor's ABI includes a link register, generate instructions - to save and restore it in a callee-saved register (if available) or on the stack around aBlock, which is assumed to generate code. - By default, do nothing. RISCs override." - <inline: #always> - ^aBlock value! Item was changed: ----- Method: InterpreterPrimitives>>primitiveSnapshot (in category 'system control primitives') ----- primitiveSnapshot + "Save a normal snapshot under the same name as it was loaded unless it has been renamed by the last primitiveImageName. + Note that when executed this primitive answers false, but when the resulting image is run afresh, the primitive answers true." - "save a normal snapshot under the same name as it was loaded unless it has been renamed by the last primitiveImageName" <inline: false> self snapshot: false ! Item was added: + ----- Method: SpurMemoryManager>>lowSpaceThreshold (in category 'accessing') ----- + lowSpaceThreshold + ^lowSpaceThreshold! Item was changed: ----- Method: SpurMemoryManager>>sufficientSpaceAfterGC: (in category 'gc - scavenging') ----- sufficientSpaceAfterGC: numBytes "This is ObjectMemory's funky entry-point into its incremental GC, which is a stop-the-world a young generation reclaimer. In Spur we run the scavenger. Answer if space is not low." | heapSizePostGC | self assert: numBytes = 0. self scavengingGCTenuringIf: TenureByAge. heapSizePostGC := segmentManager totalOldSpaceCapacity - totalFreeOldSpace. (heapSizePostGC - heapSizeAtPreviousGC) asFloat / heapSizeAtPreviousGC >= heapGrowthToSizeGCRatio ifTrue: [self fullGC] "fullGC will attempt to shrink" ifFalse: "Also attempt to shrink if there is plenty of free space and no need to GC" [totalFreeOldSpace > (shrinkThreshold * 2) ifTrue: [self attemptToShrink. ^true]]. [totalFreeOldSpace < growHeadroom and: [(self growOldSpaceByAtLeast: 0) notNil]] whileTrue: [totalFreeOldSpace >= growHeadroom ifTrue: [^true]]. + "Answer false if space is low" + ^lowSpaceThreshold <= totalFreeOldSpace! - lowSpaceThreshold > totalFreeOldSpace ifTrue: "space is low" - [lowSpaceThreshold := 0. "avoid signalling low space twice" - ^false]. - ^true! Item was added: + ----- Method: StackInterpreterSimulator>>flush (in category 'debug printing') ----- + flush + transcript flush. + "We *always* want to make transcript visible on flush" + TranscriptStream forceUpdate ifFalse: + [transcript changed: #appendEntry]!

1 0

Let's discuss the purpose of low-space detection...
by Marcel Taeumel 10 Aug '23

10 Aug '23

Hi all -- Let's discuss some realistic scenarios for an in-image low-space detection and handling mechanism. I think that, in many situations, users just expect some magic to the resolve a low-space issue. In reality, however, it can be rather tricky to recover and keep on working within such a too-big-to-handle image. Is it a rogue workspace do-it that needs to be aborted and GC'ed? Is it a server process that needs to cut off some erroneous client-request processing? Do we have to save precious data to an external, out-of-image storage? Do we expect some handling to then keep on working within the image? ... When thinking about the interactive tooling that is required to handle a low-space situation, I find it hard to estimate the kind of memory we actually need for such interactivity... As long as the VM does not grow further, the host environment is fine, I suppose. This is the status quo. The main issue seem to be our expectations of how the image behaves in such a situation... We might even need a 75% warning besides a critical low-space detection... Best, Marcel

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Vm-dev August 2023