[Pharo-dev] [Vm-dev] Re: Random forest in Pharo
Robert Withers
robert.w.withers at gmail.com
Fri Dec 4 01:13:27 UTC 2015
Hi Eliot,
On 10/19/2015 10:10 AM, Eliot Miranda wrote:
>
> On Oct 19, 2015, at 5:03 AM, Robert Withers
> <robert.w.withers at gmail.com <mailto:robert.w.withers at gmail.com>> wrote:
>
>> Hi Esteban,
>>
>>
>> On 10/19/2015 05:10 AM, Esteban Lorenzano wrote:
>>>
>>> Hi,
>>>
>>> just to be clear.
>>> When we talk about MTVM, we talk about a MT-FFI, *not* a MTVM in
>>> general.
>>> In general, a “common” approach to MT cannot be applied in Pharo (or
>>> Smalltalk in general) and to get a VM *and* an image working
>>> properly is an effort that makes what I called massive some mails
>>> above like a small stone compared to a mountain.
>>
>> Could you please help me by talking further about the different
>> models and scopes of what is meant by MT?
>>
>> a) MT-FFI I believe gives the developer a way to call and be invoked
>> on callback, asynchronously. Is it so?
>
> It's a bit more than that. It is the sharing of the VM between
> different threads, but only allowing one thread to own the VM at any
> one time, changing ownership in call out or Smalltalk process switch
> time. This approach provides interleaved concurrency but not
> parallelism in your Smalltalk code and it means the Smalltalk class
> library doesn't have to be made thread-safe, which as Esteban said is
> a huge task.
>
> See
> http://lists.gforge.inria.fr/pipermail/pharo-project/2011-January/038943.html
>
> and google "eliot Miranda Simmons own thread" to find more messages.
>
>
>> b) General MTVM means other system services are threaded, like I/O
>> events and scheduling and heartbeat.
>
> No; at least not in my opinion. In the standard single-threaded VM the
> heartbeat is ideally a thread (it can be an interval timer, but that's
> problematic; system calls get interrupted), and maybe an incremental
> global GC could be in its own thread.
>
> So I'm defining the MTVM to be the sharing of the VM between threads,
> and /not/ just the use of threads to implement non-Smalltalk sub tasks
> of the VM, and /not/ a full-blown multithreaded Smalltalk VM providing
> concurrent execution of Smalltalk processes in parallel.
I've been thinking about your definition of a concurrent, interleaved VM
threading model and I get it and why its the desirable approach. This
concurrent interleaved multi-threading approach combines underlying
liveness with overlying safety.
Hopefully you aren't minding that I proceeded to look at an event-loop
mechanism and I have three changes to a "standard" event-loop to
function in the vm:
1) a thread carousel to regulate thread switching between event
executions (stack), fed by the queued event frames holding special
threads other than the standard event thread.
2) a pool for unscheduled events pending a signal to schedule them into
the queue. Sepaphore>>#wait would interrupt the running process and
unexecute and unschedule them into the pool until a signal on that
semaphore.
3) a bank of FFI threads, each in a specialized, dedicated pool-event-loop
Here is a diagram I drew, trying to show this model. I am not sure it's
feasible or desirable, but I hope it adds rather than subtracts to your
thoughts.
Best,
Robert
>
>> I think that the right model (my stack/priQueue/pool intuition?) will
>> change a Herculean task into a fairly straightfoward task and
>> achievable. Change the problem, to get better answers.
>
> This has been well thought through and discussed. The definition
> above is very useful. It provides a system that can inter operate
> with concurrent code without having to implement a system that
> provides parallelism. It is used in David Simmons' VMs for S# etc and
> a similar (but less performant) scheme is available in Python VMs.
>
> Please, let's get this scheme working first. I'm not at all happy
> (read, extremely unhappy) that there is not much focus on working
> together to get our current VM to an excellent state and instead lots
> of work on other VMs that is speculative and a long way away from
> being production ready. We have a huge amount of work to do on Cog:
>
> - event-driven VM (that hence costs 0% processor time at idle)
> - 64-bits (x64 and ARM and...?)
> - Sista adaptive optimizer
> - FFI via dynamic generation of marshaling code, as required for
> efficient and correct call outs on x64
> - MTVM as defined above
> - an incremental global mark-sweep GC for Spur
> - running on Xen/Unikernels/containers
> - providing a JavaScript plugin to proved rendering and events so we
> can run an efficient VM in a web browser
> - a port of the Interpreter/Context VM to Spur
>
> IMO, things that can /and should/ wait are
> - throwing away Slang and providing a true written-in-pure-Smalltalk
> VM that is self-bootstrapped a la Gerardo Richarte and Xavier Burroni
> - a truly parallel multi/threaded VM
>
> and things we shouldn't go anywhere near are
> - using libffi
> - targeting JavaScript, Java or any other dynamic language de jour
> that happens to run in a web browser but either provides abysmal
> performance or doesn't support full Smalltalk semantics
> - implementing the VM in other VM frameworks such as PyPy which simply
> strengthens that community and weakens our own
>
> Right now there are only a handful of people who make commits to the
> VM and three who are "full time", and we're all overloaded. But the
> VM is the base of the pillar and if we want to provide high-quality
> solutions that people will pay money to use we have to have a
> high-quality VM. In Spur we have a VM that is significantly faster
> that VW, and very reliable. In Sista we will have a system that is
> much faster and can be improved upon for years to come and a system
> that can migrate to future VMs (because it is mostly Smalltalk), and
> useful support for a high quality FFI. People like have stepped up
> and made significant contributions to give us what is a respectable VM
> that is on an arc to providing a really high-quality production
> Smalltalk VM written in Smalltalk produced by a very small community.
> But it is now 2015 and Cog started 7 years ago. All the work on other
> VMs, deployment platforms etc, IMO dilutes and delays in delivering to
> our community a truly world-class VM that we can compete with against
> Java HotSpot, node.js v8, lua luajit, factor, swift et al. Please get
> on board. We'd love the help and we can guarantee you'll have fun and
> you can guarantee you'll have an impact.
>
>>
>> I appreciate you and this MT discussion.
>>
>>
>>> Said that:
>>>
>>> - What is in plans is MT-FFI, and that will be available eventually.
>>> - There is an approach I want to re-work, that would allow us profit
>>> of multicores without going multithread: the “hydra” experiment made
>>> some years ago by Igor creates a good basis to this. But is also a
>>> lot of of work (but a lot less than a complete MT), and not a real
>>> priority for now… I hope to resume work on that area some day… just
>>> not anytime soon.
>>
>> Yes, please. I recall those discussions. Hydra is cosmological.
>>
>> Regards,
>> Robert
>>
>>>
>>> Esteban
>>>
>>>> On 18 Oct 2015, at 17:56, Ben Coman <btc at openInWorld.com
>>>> <mailto:btc at openinworld.com>> wrote:
>>>>
>>>>
>>>> On Sat, Oct 17, 2015 at 2:25 AM, Robert Withers
>>>> <robert.w.withers at gmail.com <mailto:robert.w.withers at gmail.com>> wrote:
>>>>> Yes, exactly. I do realize I was consciously changing that effort
>>>>> synchronization order.
>>>>
>>>> I see 64-bit being higher priority than multi-threaded for the wider
>>>> community. Dealing with larger in-Image data opens the door to more
>>>> corporate project/funding opportunities. Also simplifying the install
>>>> on modern Linux platforms without requiring additional 386 libraries
>>>> will help acceptance there.
>>>>
>>>>> It is my humble opinion, without really knowing, that 64-bit would
>>>>> have to be redone after the MTVM completes.
>>>>
>>>> I would assume it was the other way around. Presuming that Eliot has
>>>> sponsors influencing his priorities, it seems given that 64-bits will
>>>> happen first. I would guess any MTVM development on the old vm would
>>>> then need to be reworked.
>>>>
>>>>> I was doing so with the idea in mind that I and others
>>>>> might dig into working on the VM, for threading support, while Eliot
>>>>> maintains focus on 64-bits...a tall order, I know.
>>>>
>>>> The usual downside of splitting resources applies. There are not that
>>>> many "others" and maybe they would be drawn away from helping with the
>>>> 64-bit vm. If the 64-bit vm goes slower for lack of resources then
>>>> your footing for MTVM will shifting for a longer time. You may
>>>> ultimately get where you want to go faster by helping with the 64-bit
>>>> vm. The rapport built with other vm devs from working on 64-bit might
>>>> could then be applied to MTVM. (Of course, its your free time, so you
>>>> should pursue what interests you.)
>>>>
>>>>> I was barely familiar with the VM, slang, interpreter, it years ago...
>>>>> I'm totally unfamiliar with cog.
>>>>
>>>> The experience you gain from working beside Esteban and Eliot on
>>>> 64-bit Cog/Spur could then be applied to a MTVM.
>>>>
>>>> btw, you may find these threads interesting...
>>>> *
>>>> http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/2015-April/108648.html
>>>> *
>>>> http://forum.world.st/Copy-on-write-for-a-multithreaded-VM-td4837905.html
>>>>
>>>> cheers -ben
>>>>
>>>>> I believe another item on that list ought to be modernizing slang. So
>>>>> many big items!
>>>>>
>>>>> Robert
>>>>>
>>>>>
>>>>>
>>>>> On 10/16/2015 12:48 PM, Stephan Eggermont wrote:
>>>>>>
>>>>>>
>>>>>> On 16-10-15 14:05, Robert Withers wrote:
>>>>>>>
>>>>>>> Because of that assumption I've made and without the
>>>>>>> responsibilities
>>>>>>> you have, Esteban, but recognizing modernizing NB to FFI, my desired
>>>>>>> list is:
>>>>>>
>>>>>>
>>>>>> I would expect the least total effort to be needed by keeping the
>>>>>> work
>>>>>> of Esteban and Eliot as much as possible aligned. That is what
>>>>>> Esteban's
>>>>>> list achieves.
>>>>>>
>>>>>> Stephan
>>>>>>
>>>>>>
>>>>>
>>>
-------------- next part --------------
Skipped content of type multipart/related
More information about the Vm-dev
mailing list