[Vm-dev] NativeBoost / Callbacks and Multithreading (Eliot: This is not just NB, we need you here :P)

[Nicolai Hess]

2015-04-20 16:20 GMT+02:00 Eliot Miranda <eliot.miranda at gmail.com>:

>
> Hi Nicolai,
>
> On Apr 19, 2015, at 3:36 AM, Nicolai Hess <nicolaihess at web.de> wrote:
>
>
>
> 2015-04-18 18:35 GMT+02:00 Eliot Miranda <eliot.miranda at gmail.com>:
>
>>
>> Hi Nicolai,
>>
>>     I'm rather excited that you're interested in this.  Perhaps we (you,
>> Esteban, Clément and I) could meet in a google hangout to discuss soon?
>>
>> As Esteban said I was able to demonstrate threaded calls on Mac.
>> CoInterpreterMT is the subclass of CoInterpreter that adds support for
>> threading. But the code is out-of-date.  So some work is needed to get back
>> to where I was.  But to have someone focused on this is the key.  I don't
>> think we have someone with the cycles yet and if you're that person (and
>> can talk to Esteban, Clément and I, and maybe Ronie and Doug McPherson
>> and Thierry) then I'm sure we can get it working soon.  Spur has the
>> necessary pinning support for example.
>>
>
> Well, I am not sure I can be a help :)
> And I spend my main time for fixing issues in the core image.
> But I would like to help if I can.
>
>
> Good, thanks!
>
> About the CoInterpreterMT, is it possible to get this working with the NB
> additions?
>
>
> Of course, but NB has several issues
> - it isn't cross-platform
> - it isn't secure (it makes the entire heap executable)
> - it is yet another JIT while we have a cross-platform one which can more
> easily integrate FFI marshaling code with Smalltalk code
>

I had some hard time fighting with old doc about FFI / uncertain state of
current FFI (which version works with what VM ). And finally NB was
the one I find that works good with Pharo and has some very well examples
and existing working projects.
I'll switch to whatever new FFI (without the above issues) will come after
NB. But right now, I don't know what are my options, is
there anything else I can use?



>
> It would be great but it sounds too simple, but would it work if we use a
> MT VM with NB additions and
> change the NB callback entry/leave code to make calls to ownVM and
> disownVM
> (for callbacks defined in the main vm thread, but called by some native
> code in another thread) ?
>
>
> Hmmm are NB callbacks different from Alien callbacks?  IIRC, NB uses the
> Alien callback machinery.  Igor, did you reimplement callback entry fir NB
> or reuse my code?  If not, what was the motivation to not use the existing
> infrastructure?
>
>
> I am not sure if we need spurs pinning support here, because the current
> NB implementation works without it.
>
>
> What would happen if one made an FFI call that included a ByteArray passed
> by reference which is used by the called code as a buffer and that code
> called back and in the callback the GC ran and moved the buffer then?
>
> What would happen if one made an FFI call that included a ByteArray
> passed by reference which is used by the called code as a buffer and a
> thread switch occurred and the system ran the GC which moved the buffer
> while the first thread was using the buffer?
>

Sure, but arent NB objects marked as ExternalObjects?


>
> Eliot (phone)
>
>
>> Eliot (phone)
>>
>> On Apr 18, 2015, at 9:24 AM, Nicolai Hess <nicolaihess at web.de> wrote:
>>
>> Thank you Esteban,
>>
>> 2015-04-18 9:36 GMT+02:00 Esteban Lorenzano <estebanlm at gmail.com>:
>>
>>>
>>> Hi,
>>>
>>> So… “multithread” is a complicated issue. Pharo as most Smalltalks is
>>> designed thinking is as monolithic so if you are trying to make the image
>>> to work in different processes… it will not be easy at all :)
>>> Time ago Eliot did a prototype (more than a prototype in fact, but still
>>> far from “production ready”) to have Threaded FFI and I’m quite sure this
>>> is the way to go, at least as a 1st milestone.
>>>
>>> I already have a process building this experimental VM:
>>>
>>> https://ci.inria.fr/pharo/view/VM/job/CogMTVM/
>>>
>>
>> But this one does not include NativeBoost support. Do we have a build
>> with MT and NB?
>>
>>
>>>
>>> (No idea why windows build failed last two times… this can be a random
>>> fail).
>>>
>>> I do not remember exactly the changes needed in the image to take
>>> advantage of it… it was just an instVar in Process, I think, but then FFI
>>> package was adapted to take advantage of this… no idea where it is now (if
>>> is already incorporated in latest FFI, which we have).
>>>
>>> Then, after this… I imagine the callback mechanism can be adapted to
>>> work in multithread environments but probably we will need something like
>>> isolates from Dart to provide some degree of multithread processing without
>>> killing the image (but I’m just thinking in loud here, this can be a really
>>> bad idea, and probably there are other ways to do this better… I will let
>>> Eliot to explain better).
>>>
>>> *TL;DR: *Start for the CogMTVM, is the way to go.
>>>
>>> cheers,
>>> Esteban
>>>
>>> On 17 Apr 2015, at 09:06, Nicolai Hess <nicolaihess at web.de> wrote:
>>>
>>> Hi,
>>>
>>> I've tried to build a pharovm with multithread support. I changed the
>>> config in the
>>> pharo generator.image to use the cogmt config and was able to build a vm.
>>> But this vm crashes on startup. (windows7)
>>>
>>> Is this the right way and did anyone got this to work (windows or linux)?
>>>
>>> And can this work to make NB callbacks working for multithreaded
>>> libraries.
>>>
>>> (I made some simple bindings for the gstreamer lib with NB, this works
>>> for simple calls (create element/change state). But I guess this won't
>>> work
>>> for any gstreamer function that requests a callback that may b e called
>>> from
>>> a different thread)
>>>
>>> Would this be the right way to do:
>>> - build a vm with MT support
>>> - guard the NB callback entry/leave code with the ownVM()/disownVM()
>>> call.
>>>
>>> Or is there more to do.
>>>
>>> Thanks in advance
>>>
>>>
>>> nicolai
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20150420/dbe8aa54/attachment-0001.htm