[Vm-dev] Cog: A question about: setInterruptCheckChain()

[Eliot Miranda]

Hi Andreas,

    excuse me for flogging the odd limb of a dead horse...  I agree with
many of your points below but a few I still need to flail away at...


On Wed, Sep 29, 2010 at 4:22 PM, Andreas Raab <andreas.raab at gmx.de> wrote:

>
> On 9/29/2010 1:03 PM, Eliot Miranda wrote:
>
>> I think it /is/, and you shouldn't allow your disinclination for the FFI
>> approach blind you to the fact that one way of keeping the VM small is
>> to produce a really strong flexible FFI that supports threading and
>> callbacks and to implement interfaces above the line.  Do you seriously
>> believe things like the ODBC connect should be implemented in plugins?
>>
>
> Absolutely. Memory safety being the prime reason.


I don't see that C provides any more or less memory safety than the FFI,
unless the FFI can't access accurate type information for signatures and
parameter types, and can't easily manage unmoving objects.  I know this is
an issue, requiring good header parsing a C compiler modelling tools, or use
of things like gcc-xml, but it is possible, and there are clunky solutions
to unmoving objects (ExternalData, Alien) and more elegant facilities in
gestation (a pinning garbage collector).  So if grant me that the FFI can
access accurate type information and provide unmoving objects then there's
no difference since C allows one to mismanage memory (e.g. free memory still
in use, cast a pointer inappropriately, etc, etc) just as much, if not more
than, the FFI.  Memory safety errors in both C and Smalltalk domains can
cause hard-to-dagnose crashes that happen long after the fact.  The one
advantage that C has is that one has to use only one domain while debugging,
C, whereas if one is using the FFI it's likely one has to debug in both.


In the case of event callbacks its a different issue.  The current event
>> queue abstraction is broken in that it prevents Smalltalk code from
>> answering questions asked through events such as WM_QUERYENDSESSION
>> because the VM is forced to queue events, and hence can only provide a
>> default answer in the window proc.  So there are areas where having
>> proper callbacks is required to be able to integrate properly with the
>> host operating system.
>>
>
> Please. The issue here is that our event mechanism is a bit simplistic but
> the Android VM is a great example of how to deal with these issues properly.
> Sure, callbacks could address this too, but claiming *only* callbacks can
> address this is simply wrong.


I don't see this in my specific example.  One can provide a hack such as
specifying a default response before the fact for the VM to respond with
to WM_QUERYENDSESSION but that doesn't give the image a chance to act, e.g.
to flush data or to query the user.  So humour me, how would you solve this
specific case without callbacks?  The requirement is that the image gets a
chance to query the user in response to WM_QUERYENDSESSION so that they can
abort the reboot if desired.


I don't disagree with what you say about plugins, or about maintaining
>> cross-platform abstractions in the VM for core functionality that is
>> common across platforms, but I think its clear one needs true callbacks,
>> and that at least for integrating with the GUI this is the best way to
>> implement it.
>>
>
> A solution that inherently implies exposure of raw pointers to users isn't
> what I would call the "best" solution. That said, I'm not arguing against
> callbacks - when you need them you need them. And there are practical
> situations where it's incredibly handy to have them. And yet, would I design
> the system so that it depends at its core on such facilities? Hell, no,
> because in addition to being memory-unsafe, porting an FFI is *much* harder
> than porting a bit of C support code.


But porting the FFI is /not/ harder than porting a JIT (believe me), and as
an FFI grows in sophistication it gets easier to port because techniques and
solutions used on one platform are applicable to others (again, I've been
there and done that either with discarding the bulk of the assembler in an
existing implementation or a new port in the VisualWorks FFI implementations
on Alpha AXP, MIPS, PowerPC, SPARC (32-bit & 64-bit) and x86-64).


   It really is a matter of where you want to deal with certain aspects
>>    of the system. I prefer to deal with that level of stuff in a plugin
>>    via C because:
>>    + it provides a way to do ensure memory safety
>>
>> there's nothing in plugins per-se to maintain memory safety.  A mistake
>> in a plugin is as destructive as a mistake in an FFI call.  In either
>> case a wrapper around a particular call can validate and provide safety.
>>
>
> In theory, yes. In practice such wrappers *do* exist in plugins and they do
> *not* exist in FFI calls. Why? Good question. Probably because people
> understand that in C land there are more constraints and they are mentally
> "set up" to keep an eye on the constraints and to make sure the incoming
> stuff is okay.


I don't agree that they don't exist in Smalltalk code.  Well-written
Smalltalk code that uses an FFI typically does provide an
abstraction-boundary and manage memory behind it.  The ODBC connect is a
good example.

   + it provides a way to provide abstractions
>>
>> - so does Smalltalk, and t least in SMalltalk we have inheritance to
>> enable us to express cross-platform APIs in abstract classes and map
>> these down onto particular platforms in concrete subclasses.  This is
>> /far better/ than the sprawling mess that is the platforms tree in the VM.
>>
>
> In theory yes. In practice, who knows. Having looked at some Smalltalks
> that expose all the platform details (like Dolphin) it seems to me that the
> equation of "platform FFI code * number of platforms" will be /far more/ of
> a sprawling mess. Which is probably why (unless I'm mistaken in which case
> I'll appreciate a correction) precisely none of the cross-platform
> Smalltalks do that.


Two great advantages that Smalltalk has here are browsability and
inheritance. It is straight-forward to use inheritance to group platform
fan-out and the IDE supports comparing the various platform-specific
implementations.  Further, inheritance encourages code-sharing and keeping
platform fan-out to a minimum.  In contrast the platform tree in the Squeak
VM has very little shared code and IME is quite badly organized.

   + the code is faster to write and debug in C
>>
>> this is debatable.
>>
>
> Hardly. Let's start with "#include <windows.h>" (or whatever platform stuff
> you happen to need). You're losing right there. The problem is that the
> system isn't set up to seamlessly interact with C. Even if you've got some
> auto-magic header file importer that creates structs and defines and stuff,
> as long as it takes more than, i.e.,
>
> winMessageBox: contents title: title
>        <include: 'windows.h'>
>        MessageBox(NULL, contents, title, MB_OK + MB_ICONWARNING).
>
> you're going to lose because it takes so much longer to get to the point of
> actually writing the code you're trying to write.
>

That's a strong point.   It was hard in VisualWorks to find the effort to
keep the C preprocessor up-to-date.  However, I can at least imagine a
well-maintained interface to windows.h and know how I would implement it
(I've discussed this a lot before; I won't bore you).


> Secondly, you *really* need to try the latest Visual Studio and play with
> its debugger - I was blown away while debugging SqueakSSL when I hit a
> typical NULL pointer, fixed the code in my plugin DLL, continued in VS and
> the underlying Squeak KEPT RUNNING! Holy crap. This stuff has come a
> looooong way.


True.  But it breaks my heart.  I don't want to be embraced and then
extended ;)  They're playing catch-up and have probably caught up, but I
would still like to compete.



>  In the VW FFI with the ability to catch exceptions
>> and pass them back up (doesn't always work, a bad emory corruption may
>> crash the entire system) makes thing easier.  But low-level debugging is
>> painful in general.
>>
>>    + access to facilities (threads, interrupts, atomicity) not
>>    available otherwise
>>
>> Again, a strong FFI provides at least some of these facilities.  Not
>> everything can be done through the FFI, but a lot can be done, and
>> elegantly and extensibly, by Smalltalk programmers, not a few VM
>> specialists.
>>
>
> I call BS. The reality is that one needs to know three kinds of things,
> only one of which would be helped by using the FFI:
>
> 1) The "mechanics" of writing a plugin, i.e., slang and compilation. This
> is the part where the FFI would help, but so would better APIs for
> marshaling. While it's "cool" that one can run slang plugins in Squeak it is
> also completely and utterly *useless* when it comes to integrating platform
> specific stuff and having to deal with Slang, VMMaker and the awful build
> setups, and then -on top of that- the actual work you're trying to do is
> just a bit too much.
>
> 2) Understanding the rules of interfacing the external world, i.e., when
> can pointers be kept, what does GC do to your objects etc. None of this is
> helped by using the FFI.
>
> 3) Understanding the actual domain code. Again, none of this is helped by
> the FFI (to the contrary because nobody will find examples for the weird
> Smalltalk selectors we use on the web and people will in turn not know where
> to find the actual value for MB_OK or somesuch).
>
> Basically, the assumption that *all* the difficulty is in writing the
> marshaling code is nonsense. There is some of it, true, but that is entirely
> our fault for providing poor integration APIs.


Agreed.


>
>
>     On the downside, there is:
>>    - it's harder to modify for people who don't know C and are not set
>>    up to build a plugin
>>    - it's often aimed at the smallest common denominator (though not
>>    necessarily)
>>
>> - it is a far more fixed interface that is far harder to evolve
>>
>
> Yes, but when you call it "stable" instead of "fixed" and "robust" instead
> of "harder to evolve" it becomes a plus :-)
>

There's plenty of examples of stable interfaces above the FFI.  VisualWorks
has depended on its database connects; they are its bread if not its butter
and are implemented on the FFI.      The Windows registry is accessed solely
through the FFI, etc.  These interfaces are certainly stable and have
evolved.

best
Eliot


> Cheers,
>  - Andreas
>
>
>     Cheers,
>>      - Andreas
>>
>>
>>        On the other hand, a VM providing a larger de facto immutable API:
>>        - somehow is more secure (a guaranty we can run very old images in
>>        very new OSes).
>>        - can maintain the illusion that despite the efforts of OS and
>>        hardware designers to make it ever more complex, a single person
>> can
>>        still understand (almost) the whole system.
>>        - avoid the necessity to embed knowledge of dozens of different
>>        variants of different OSes in your image.
>>
>>        Of course, the complexity still exists under the carpet... When I
>>        contemplate all the unecessary complex knowledge from those
>>        beautiful
>>        configure/cmake scripts and macros, I'm not convinced all this
>> cruft
>>        will be easier to modify in Smalltalk than it is in C world.
>>
>>        I perfectly understand too the desire of a VM maintainer to lighten
>>        the burden ;) though.
>>
>>        Nicolas
>>
>>        2010/9/29 Eliot Miranda<eliot.miranda at gmail.com
>>        <mailto:eliot.miranda at gmail.com>>:
>>
>>
>>
>>
>>
>>            On Wed, Sep 29, 2010 at 10:22 AM, Andreas
>>            Raab<andreas.raab at gmx.de <mailto:andreas.raab at gmx.de>>  wrote:
>>
>>
>>
>>                On 9/29/2010 9:36 AM, Eliot Miranda wrote:
>>
>>
>>                        See the recordMouseEvent() and its friends
>>                    (sqNextEventPut,
>>                        eventBuffer[1024] ...) in sqWin32Window, for
>>                    better understanding my
>>                        point :)
>>
>>
>>                    Which makes my point well.  These are simply
>>                    maintaining a queue.  If
>>                    one has callbacks then the bulk of the Windows event
>>                    handling system can
>>                    be lifted up into the image.  This is what Vassili
>>                    Bykov did in Newspeak
>>                    above my Alien callbacks.  In Newspeak the Windows
>>                    MainWndProc is a
>>                    callback:
>>
>>
>>                Thus losing any level of platform abstraction. A bad
>>                trade-off, IMO.
>>
>>
>>            The platform abstraction merely moves from the VM to classes
>>            in the image.  But it does mean that one can properly
>>            implement Windows event semantics (things like query quit
>>            events) that the queue prevents.  And it means that the VM
>>            gets simpler and complexity is handled by better facilities
>>            (in Smalltalk with its many advantages for creating and
>>            maintaining abstractions with concrete variant
>> implementations).
>>            But you and I have had this discussion a number of times and
>>            I take your point about the VM maintaining a black-box
>>            abstraction boundary whereas the in-image implementation is
>>            in theory porous.  As I've said before this doesn't strike
>>            me as that important when Smalltalk has had boundary
>>            violators such as instVarAt:[put:] for ever and they are
>>            rarely abused.  If one were to implement a native GUI event
>>            interface in Smalltalk using callbacks one would have to
>>            police the abstraction boundary.  But that's easily done,
>>            and there are other benefits
>>            - the VM, being simpler, gets more longevity since changing
>>            the event interface doesn't imply a VM change
>>            - the system, having native interfaces, can support each
>>            platform's facilities in full instead of providing the
>>            lowest common denominator
>>            So IMO it is a good trade-off.
>>            best
>>            Eliot
>>
>>
>>                Cheers,
>>                  - Andreas
>>
>>
>>
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20101001/9723258b/attachment-0001.htm