[squeak-dev] WindowEvents and new SystemLaunchEvent/#launchDrop from singleton VM (was: Changeset: Enhanced integration of drag'n'drop from host)

Thiede, Christoph Christoph.Thiede at student.hpi.uni-potsdam.de
Wed Nov 4 19:57:14 UTC 2020


Hi Jakob, thanks for your reply! :-)


So you are proposing to build a separate dispatching mechanism for system launch events, if necessary, which sounds very reasonable with regard to possible performance issues. In this case, there would be no need for an extra event class and I could couple #generateDropFilesEvent: just a little bit stronger to the world.


> If the world morph were to dispatch the event, what about non-Morphic applications?


<http://www.hpi.de/>
Now you're touching a sore spot on event generation in general. No skiving, event generation is very closely coupled to Morphic, just because all the event buffer decoding happens in the HandMorph class! If Morphic is unloaded, the new DnD features cannot be used, just like they were never implemented in MVC, and just like window events which neither ever worked in MVC, too. If we wanted to decouple these aspects, we probably would need to extract most of the event generation methods into another class (maybe EventSensor? maybe an intermediary abstraction level?) and provide some callbacks for things like launchDrops. But I believe this is enough stuff for a follow-up project - which I personally would not give a high priority just because I'm not aware of any concrete use case for a non-Morphic event generation framework at the moment. :-)

Any other opinions on this question? Otherwise, I will proceed with eliminating the SystemLaunchEvent soon.

Best,
Christoph
________________________________
Von: Jakob Reschke <forums.jakob at resfarm.de>
Gesendet: Dienstag, 3. November 2020 11:23:24
An: The general-purpose Squeak developers list; Thiede, Christoph
Cc: Taeumel, Marcel
Betreff: Re: [squeak-dev] WindowEvents and new SystemLaunchEvent/#launchDrop from singleton VM (was: Changeset: Enhanced integration of drag'n'drop from host)

I haven't thought this through, but what I first thought of was something like SystemChangeNotifier. A central instance where objects can register for such global events.

It sounds odd to me to deliver the event to every single morph in the system. Most of them are probably not interested. So the interested ones should ask for notifications.
If the world morph were to dispatch the event, what about non-Morphic applications?

Am Mo., 2. Nov. 2020 um 19:28 Uhr schrieb Thiede, Christoph <Christoph.Thiede at student.hpi.uni-potsdam.de<mailto:Christoph.Thiede at student.hpi.uni-potsdam.de>>:

Hi Marcel, thanks for the feedback & merge! :-)

<http://www.hpi.de/>

One part of the patch remains to be reviewed that you did not yet merge because it requires additional discussion; I'm speaking about the SystemLaunchEvent feature. I'd like to ask everyone interested in this proposal to have a look at my original changeset, but because I know its easier just to read this email without downloading and trying out anything, I'm citing the relevant diff here again:
<http://www.hpi.de/>


MorphicEvent subclass: #SystemLaunchEvent
        instanceVariableNames: 'arguments wasHandled'
        classVariableNames: ''
        poolDictionaries: ''
        category: 'Morphic-Events'!



!SystemLaunchEvent commentStamp: 'ct 1/18/586455 02:22' prior: 0! I am signaled by the VM that is configured as a singleton application when it has been invoked with a new image file. This feature is also known as #launchDrop as well as under several platform-specific names: Unix - runAsSingleInstance Win32 - RunSingleApp Instance Variables arguments: A collection of FileStreams. wasHandled: A Boolean.!


!Morph methodsFor: 'event handling' stamp: 'ct 10/15/2020 19:31'!
handlesSystemLaunch: evt
        "Do I want to receive #systemLaunch: when the VM instance is invoked with a second image file? See #handleSystemLaunch:."

        ^ false! !

!Morph methodsFor: 'event handling' stamp: 'ct 10/15/2020 19:36'!
systemLaunch: evt
        "Handle a systemLaunch event, signaled when the VM instance is invoked with a second image file. See #handleSystemLaunch:. Hook for subclasses."! !



!Morph methodsFor: 'events-processing' stamp: 'ct 10/15/2020 19:32'!
handleSystemLaunch: anEvent
        "Handle a system launch event from the OS. Signaled if them VM is configured as a singleton application when it has been invoked again with a second image file. See SystemLaunchEvent comment."

        anEvent wasHandled ifTrue: [^ self]. "not interested"
        (self handlesSystemLaunch: anEvent) ifFalse: [^ self].
        anEvent wasHandled: true.
        self systemLaunch: anEvent.! !



!PasteUpMorph methodsFor: 'event handling' stamp: 'ct 10/15/2020 19:32'!
handlesSystemLaunch: anEvent

        ^ self isWorldMorph! !

!PasteUpMorph methodsFor: 'event handling' stamp: 'ct 10/15/2020 19:57'!
systemLaunch: anEvent
        "Handle a number of files the singleton VM was called with again. See SystemLaunchEvent comment."

        self flag: #todo. "Do something more useful with the image here, e. g. tell the VM to load it."
        ^ self dropFiles: anEvent arguments event: anEvent! !



!HandMorph methodsFor: 'private events' stamp: 'ct 1/18/586455 10:01'! generateDropFilesEvent: evtBuf "Generate the appropriate mouse event for the given raw event buffer." | position buttons modifiers stamp numFiles dragType | stamp := evtBuf second. stamp = 0 ifTrue: [stamp := Time eventMillisecondClock]. dragType := evtBuf third. position := evtBuf fourth @ evtBuf fifth. buttons := 1. "red" "hacked because necessary for correct mouseMoveDragging handling" modifiers := evtBuf sixth. buttons := buttons bitOr: (modifiers bitShift: 3). numFiles := evtBuf seventh. dragType caseOf: { ... [4] -> [ "dragDrop" | oldButtons | externalDropMorph ifNil: [ "dragDrop has been sent without prior dragging. This happens if the VM is configured as a singleton application and has been called again (aka #launchDrop)." ^ SystemLaunchEvent new setArguments: (self collectDropFilesAndDirectories: numFiles) source: self stamp: stamp]. ...

So the question is: Do we need an additional event class for this, and if yes, how do we want to propagate these events?
I have already spent some thoughts on this before submitting the changeset, and I did not want to make this event exclusive for the world only but accessible for every morph. However, was this a good idea, or would this be a rather different approach from how "system events" work (or should work)?

A good precedent may be WindowEvent which is indeed dispatched to the world morph only - at least at the moment. I'm trying to construct an example where another morph could be interested in these events, too. For example, a tool might have opened a session to some critical external system, maybe an SSH shell, and this tool would like to output an extra warning to the user when a #windowClose event is received. However, should it get access to WindowEvents for this purpose, or should they be hidden behind the PasteUpMorph facade and we would prefer to design some high-level hook on the world instead which the SSH tool then could register itself against as a "critical operation"? (NB: The shutDown list would not be the right solution in this fictive example - when #shutDown: is called, the shutdown operation is already in progress which would be "too late" for our tool to reject the shutdown.)

I'm pretty sure we can conduct more or less the same debate for the new proposed SystemLaunchEvent because a SystemLaunchEvent is basically very similar to a WindowEvent, just without a position but with one or multiple file paths (internally: file streams).
Should you be able to watch an event like this using HandMorph >> #showEvents:? I think so.
Should you be able to receive an event like this using an event listener? Probably?
Should you be able to intercept an event like this using an event filter? I don't know, but could it harm?
Should every morph be able to subscribe to events like this by overriding #handlesSystemLaunch:? I have no idea ...

So I'm looking forward to your opinions on this issue ... Do we need a new event class for this, or does it suffice to tell the PasteUpMorph a "private message" only? My versions browser for #generateDropFilesEvent: contains this old version, too, which would be pretty much our alternative:

generateDropFilesEvent: evtBuf
"Generate the appropriate mouse event for the given raw event buffer."

...
dragType caseOf: {
...
[4] -> [ "dragDrop"
| oldButtons |
externalDropMorph ifNil: [
| filesAndDirectories |
"dragDrop has been sent without prior dragging. This happens if the VM is configured as a singleton application and has been called again (aka #launchDrop)."
filesAndDirectories := self collectDropFilesAndDirectories: numFiles.
self flag: #todo. "ct: Do we need an extra event class for this?"
self world launchDropFiles: filesAndDirectories event: (MorphicUnknownEvent new
setType: #launchDrop argument: filesAndDirectories hand: self stamp: stamp).
^ nil].

Your two cents please! :-)

Best,
Christoph

________________________________
Von: Taeumel, Marcel
Gesendet: Montag, 2. November 2020 16:40 Uhr
An: Thiede, Christoph; squeak-dev at mail.squeak.org<mailto:squeak-dev at mail.squeak.org>
Betreff: Re: Changeset: Enhanced integration of drag'n'drop from host

Done and merged into Trunk.

Best,
Marcel

Am 26.10.2020 07:47:02 schrieb Marcel Taeumel <marcel.taeumel at hpi.de<mailto:marcel.taeumel at hpi.de>>:

Hi Christoph!

This is great news! I will take a look at it and test it for the Windows platform.

Best,
Marcel

Am 23.10.2020 14:11:28 schrieb Thiede, Christoph <christoph.thiede at student.hpi.uni-potsdam.de<mailto:christoph.thiede at student.hpi.uni-potsdam.de>>:

Hi all! :-) I'm glad to officially submit my dropFiles changeset today, finally. It enhances the integration of drag'n'drop (DnD) events delivered by the VM from the host operating system to the image, following one of Marcel's ideas: Instead of generating separate DropFilesEvent instances, every DnD event from the VM is translated into a TransferMorph that can be dragged into every morph or tool like every other Morph of TransferMorph, too.


[cid:1758d9b7373f456b1e51]


What is the goal of this changeset?

This design change entails two key advantages: First, every dragged host element is represented by a first-class object (which is a TransferMorph) instance. Second, as a consequence, applications do no longer need to implement a separate protocol for handling host DnDs (which was #dropFiles:/#wantsDropFiles:) but only need to provide one single proper implementation of #acceptDroppingMorph:event:/#wantsDroppedTransferMorph:. In particular, this also enables developers who use the ToolBuilder(Spec) framework to handle contents dropped from the host system since the old dropFiles protocol was never forwarded to the ToolBuilder framework.


How does it work?

There have already been complete implementations of the DropPlugin on the VM side for all important platforms (Windows, Linux, and macOS) for a long time (almost 20 years ...) which record four different types of drag events, namely: #dragEnter, #dragMove, #dragLeave, and #dragDrop. However, until now, only the last of them, #dragDrop, has been handled on the image side (see HandMorph >> #generateDropFilesEvent:). The main work of this changeset can be found in just this method where I added support for the other drag types in order to create, move, and release a TransferMorph accordingly. Other changes include:

  *   Deprecated, but kept working the old DropFilesEvent protocol (these events are still processed, but a DeprecationWarning is signaled if any morph actually handles them).
  *
Migrated the only implementation of the old dropFiles protocol, which is PasteUpMorph, to the unified TransferMorph protocol.
  *
Introduced a new event class + protocol for SystemLaunchEvent (aka #launchDrop) which is generated by a singleton VM when it has been invoked with a new image file (see class comment). That #launchDrop is a curious hack on the VM side to reuse the DnD protocol but has completely different semantics that now are separated on the image side.
  *   Implemented the SystemLaunchEvent protocol on PasteUpMorph.

Besides the changeset, I have also fixed a few minor inconsistencies with several VM-specific implementations in the OpenSmalltalk-VM repository (see #508<https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/508>, #514<https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/514>, and #518<https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/518>) that have already been merged (thanks to all reviewers!).

What's the current state of the changeset?
I have successfully tested the changeset on Windows (Win 2004) and Ubuntu/X11. Also, I made sure it works with Squeak.js and TruffleSqueak. However, I do not have any possibility to test it on the OSVM implementations for macOS and iOS, so your help and feedback especially for these platforms will be greatly appreciated!
Please see the attached demo video on how to test the changeset: Simply drag some files into an inspector field or a workspace and you should receive a list of FileStreams and/or FileDirectories. Also, aborting a drag operation should not fail, i.e. drag a file over the image but then drop the file in another window.

What's next?
There is much more potential for a comfortable host DnD integration! Concretely, I am having two downstream goals in mind: First, dropping other content types besides files and/or directories in your image, such as texts or images (currently, the DropPlugin implementations that I could try out are restricted to the former two types). Second, dragging objects out of the VM to save them as a file, or even better, to drop them in a second image! Vanessa has told me<https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/508#issuecomment-655021654> that this was even possible in the past for certain platforms, but unfortunately, the relevant code appears to be dead nowadays. Nevertheless, I have stumbled upon a lot of stubs for both features in the DropPlugin and HandMorph code, and I'm looking forward to reviving them in some interesting follow-up projects!

Which related contributions/discussions are there else?

  *   System-ct.1156 (Inbox, fixes double prompt for handling a dropped file when the first dialog was canceled; not included in the changeset)
  *   [squeak-dev] Re: Please try out | Inspector Refactoring =)<http://forum.world.st/Please-try-out-Inspector-Refactoring-tp5114974p5116422.html> (the thread contains some interesting brainstorming about MIME type support for dragging inside the image)

Please test and review! And many thanks to Marcel for his helpful tips!

Best,
Christoph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20201104/37fc9d5d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 133740 bytes
Desc: pastedImage.png
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20201104/37fc9d5d/attachment-0001.png>


More information about the Squeak-dev mailing list