<div id="__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: #000000">
Hi Dave, hi Eliot, hi Vanessa!<div><br></div><div>Thank you very much. Those answers are very helpful!</div><div><br></div><div>I am trying to learn about the differences of talking to a C library between from a C program and from within Squeak through FFI. I am especially interested in the existing safety nets to rely on or common patterns to follow when using FFI. </div><div><br></div><div>This includes:</div><div><span style="font-size: 13.3333px">- how to care for external structures created through FFI calls</span><br></div><div><span style="font-size: 13.3333px">- how to care for external structures created in-image, then passed to FFI calls</span></div><div><span style="font-size: 13.3333px">- when to use #new or #externalNew (+#free)</span></div><div>- differences between a handle being a ByteArray or an ExternalAddress</div><div>- what happens to all by external structures when (re-)starting the image</div><div>- ...</div><div><br></div><div>In this learning process, I want to double-check whether more clues can be offered through Squeak's tools. Especially if an action would crash the VM.</div><div><br></div><div>Latest thing -- that's why this question about ByteArrays -- was how to re-think this code:</div><div><br></div><div>MyStruct foo;</div><div>someFunctionFillsMyStruct(&foo);</div><div><br></div><div>Into this code:</div><div><br></div><div>foo := MyStruct new. "handle is ByteArray"</div><div>self apiSomeFunctionFillsMyStruct: foo.</div><div><br></div><div>Meaning, what whould be on the stack in C, can conveniently be hold in Squeak's object memory to be shared across Squeak processes and applications. No need to use malloc() and free():</div><div><br></div><div>MyStruct *foo = malloc(sizeof(MyStruct));</div><div><span style="font-size: 13.3333px">someFunctionFillsMyStruct(foo);</span><br></div><div><span style="font-size: 13.3333px">...</span></div><div><span style="font-size: 13.3333px">free(foo);</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">Which I can translate to Squeak FFI:</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">foo := MyStruct externalNew. "handle is ExternalAddress"</span></div><div><span style="font-size: 13.3333px">self apiSomeFunctionFillsMyStruct: foo. "same method as above! :-)"</span><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">...</span></div><div><span style="font-size: 13.3333px">foo free.</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">While there is n</span><span style="font-size: 13.3333px">o need to change #apiSomeFunctionFillsMyStruct: for this, </span><span style="font-size: 13.3333px">Squeak FFI conveniently copies structs from C stack memory to object memory anyway. No need to address the heap from within Squeak. Or this there? Performance?</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">:-)</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">Best,</span></div><div><span style="font-size: 13.3333px">Marcel</span></div><div class="mb_sig"></div>
<blockquote class="history_container" type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left: 10px;min-width: 500px">
<p style="color: #AAAAAA; margin-top: 10px;">Am 21.05.2020 04:08:18 schrieb David T. Lewis <lewis@mail.msen.com>:</p><div style="font-family:Arial,Helvetica,sans-serif">On Wed, May 20, 2020 at 06:53:37PM +0200, Marcel Taeumel wrote:<br>> Hi, all!<br>> <br>> How can I figure out, whether the handle of an external object -- if it is a byte array -- was fabricated in the image or whether it is actually something creating in the external library? For actual instances of ExternalAddress, it is obvious that those are meant to point to external memory. But what about instances of ByteArray being stored in the "handle" instVar?<br>> <br>> I tried Object >> #isPinned. :-D Did not work. I am looking at all the primitives in ByteArray.<br>> <br>> ... is there actually a difference? Or are all byte arrays that I find in the image actually in the object memory?<br>> <br>> Best,<br>> Marcel<br>> <br><br>There is no simple answer.<br><br>A good illustration is this:<br><br> SourceFiles first fileID<br><br>On a 64-bit VM, you will see a ByteArray of size 24. On a 32-bit VM, it<br>is a shorter ByteArray. In either case, the ByteArray instance exists<br>entirely within the object memory.<br><br>The byte values within that ByteArray happen to be the value of a C pointer,<br>which is the address in the process virtual memory of a data structure that<br>lives in FilePlugin within the VM. That data structure contains various things,<br>including (on a Unix VM) another pointer to a FILE struct that lives in the<br>C runtime library.<br><br>None of those pointers or internal things have any meaning within the image<br>or within the object memory itself. It is best to think of the fileID field<br>as an opaque handle to something in the external world, and the fact that<br>the bytes just happen to be a C pointer is something that you are supposed<br>to not notice.<br><br>I really wish that Andreas could be here to comment. I clearly recall his<br>shock and dismay on finding out that I was using the actual byte contents of<br>a fileID to do things in the OSProcess plugin. We had slightly different<br>perspectives on that topic, but if you were at all interested in issues of<br>security for the Squeak execution environment (as Andreas was), then you would<br>want to hear his perspective.<br><br>So in some sense, you should not really be able to know if a ByteArray contains<br>a pointer to something elsewhere in the virtual memory of the VM. On the<br>other hand, if you already know that you are doing something dangerous and<br>insecure, then it would be really convenient to be able to answer the question<br>that you are asking - does this ByteArray object in the object memory contain<br>a reference to some external thing outside of the object memory, and if so<br>is it safe for me to use it?<br><br>I don't know that there could ever be a safe answer to that question. The<br>image and the VM have no way of knowing what happens to things at the other<br>end of that C pointer. So for example in the case of fileID, you really need<br>to keep track of when a FileStream refers to invalid addresses. Thus the<br>data structure is:<br><br> /* squeak file record; see sqFilePrims.c for details */<br> typedef struct {<br> int sessionID; /* ikp: must be first */<br> void *file;<br> squeakFileOffsetType fileSize; /* 64-bits we hope. */<br> #if defined(ACORN)<br> // ACORN has to have 'lastOp' as at least a 32 bit field in order to work<br> int lastOp; // actually used to save file position<br> char writable;<br> char lastChar;<br> char isStdioStream;<br> #else<br> char writable;<br> char lastOp; /* 0 = uncommitted, 1 = read, 2 = write */<br> char lastChar;<br> char isStdioStream;<br> #endif<br> } SQFile;<br><br>The first field of the struture is sessionID, which is a value associated with<br>the currently running VM program. If you save your image and start it again,<br>the sessionID in the new VM instance will now be different, which allows the<br>FilePlugin to figure out that the pointer to the FILE struct (or to a HANDLE<br>on Windows) is not valid, and therefore it should not attempt to dereference<br>that pointer (VM crash).<br><br>This is just one example, but it illustrates that general case, which is that<br>the VM cannot be expected to keep track of what people are doing on the other<br>end of those C pointers, and the image in turn cannot be expected to know if<br>a ByteArray that contains a C pointer is referring to anything useful or<br>safe on the other end of the pointer that was saved in the ByteArray.<br><br>In specific cases, you can consider handling this by keeping track of the<br>known valid external references. If you look at the Windows FilePlugin, you<br>will see that Andreas did this by maintaining a registry of known valid<br>HANDLE values, and failing the primitives when an unregistered HANDLE was<br>passed, e.g. by my WindowsOSProcessPlugin which attempted to pass unregistered<br>HANDLE values for anonymous pipes.<br><br>This was an annoyance for me because I could not pursue my OSProcess hacks<br>on Windows (and I abandoned the effort). But from a security and system<br>integrity point of view, Andreas was right. To this day, I do not have<br>any good answer for how to handle this.<br><br>So it is not a easy problem.<br><br>Dave<br><br><br></div></blockquote></div>