<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Hi Fabio,</div><div><br>On Jan 6, 2017, at 1:47 PM, Fabio Niephaus <<a href="mailto:lists@fniephaus.com">lists@fniephaus.com</a>> wrote:<br><br></div><blockquote type="cite"><div><span></span></div></blockquote><blockquote type="cite"><div><div dir="ltr">Hi Eliot,<br><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 6, 2017 at 8:23 PM Eliot Miranda <<a href="mailto:eliot.miranda@gmail.com">eliot.miranda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr" class="gmail_msg">Hi Fabio, Hi Guille,<div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg">On Fri, Jan 6, 2017 at 9:44 AM, Fabio Niephaus <span dir="ltr" class="gmail_msg"><<a href="mailto:lists@fniephaus.com" class="gmail_msg" target="_blank">lists@fniephaus.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> <br class="gmail_msg"><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="m_-304522989685937264gmail-m_3886557463388821969GmSign gmail_msg">On Fri, Jan 6, 2017 at 6:33 PM Eliot Miranda <<a href="mailto:eliot.miranda@gmail.com" class="gmail_msg" target="_blank">eliot.miranda@gmail.com</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
Hi Guille,<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
> On Jan 6, 2017, at 6:44 AM, Guillermo Polito <<a href="mailto:guillermopolito@gmail.com" class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg" target="_blank">guillermopolito@gmail.com</a>> wrote:<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
><br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
> Hi,<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
><br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
> I was checking the code in sqUnixHeartbeat.c to see how the heartbeat thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
><br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
> What do you think about having a VM that manages that as an argument provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
I think it's a fine idea but it isn't really the issue. The issue is that the itimer mechanism is problematic, especially for foreign code, and is therefore a stop gap. The itimer interrupts long-running system calls, which means that things like sound libraries break (at Qwaq I had to fix ALSA to get it to work with the itimer heartbeat). Since Pharo is becoming more reliant on external code it may impact us more going forward.<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
The real issue is that linux's requirement that thread priorities be set in per-application file in /etc/security/limits.d (IIRC) is a big. Neither Windows nor Mac OS X requires such nonsense, and a threaded heartbeat is used on those systems without any issue at all. Why linux erected this mess in the first place is something I don't understand.<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
I had to implement the itimer heartbeat to get Qwaq forums running on Linux running pre 2.6 kernels, but had many other problems to solve as a result (ALSA, database connects).<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
Were it that the vm merely had to detect whether it could use the threaded heartbeat then things would be easy. Instead one can only use the thing if one has superuser permissions to install a file in /etc, just to use a thread of higher priority than the main one.<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Thanks for the explanation, Eliot. I had no idea how bad the issues are with the itimer, but I'm glad you also see the user-facing issue with the heartbeat.</div><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">An alternative might be to lower the priority of the main thread. Then the file installation would be unnecessary.<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Could you elaborate a little bit more on this idea? How could this impact the vm? What could be the drawbacks here?</div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">First of all, for the heartbeat thread to work reliably it must run at higher priority than the thread running Smalltalk code. This is because its job is to cause Smalltalk code to break out at regular intervals to check for events. If the Smalltalk code is compute-intensive then it will prevent the heartbeat thread from running unless the heartbeat thread is running at a higher priority, and so it will be impossible to receive input keys, etc. (Note that if event collection was in a separate thread it would suffer the same issue; compute intensive code would block the event collection thread unless it was running at higher priority).</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Right now, Linux restricts creating threads with priority higher than the default to those programs that have a /etc/security/limits.d/program.conf file that specifies the highest priority thread the program can create. And prior to the 2.6.12 kernel only superuser processes could create higher-priority threads. I do know that prior to 2.6.12 one couldn't create threads of *lower* priority than the default either (I would have used this if I could).</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">If 2.6.12 allows a program to create threads with lower priorities *without* needing a /etc/security/limits.d/program.conf, or more conveniently to allow a thread's priority to be lowered, then the idea is:</div><div class="gmail_msg">1. at start-up create a heartbeat thread at the normal priority</div><div class="gmail_msg">2. lower the priority of the main VM thread below the heartbeat thread.</div><div class="gmail_msg">Alternatively, one could spawn a new lower-priority thread to run Smalltalk code, but this may be be much more work.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">The draw-back is that running Smalltalk in a thread whose priority is lower than the default *might* impact performance with lots of other processes running. This depends on whether the scheduler conflates thread priorities with process priorities (which was the default with old linux threads, which were akin to processes).</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Sop there are some tests to perform:</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">a) see if one can lower the priority of a thread without having a /etc/security/limits.d/program.conf in place</div><div class="gmail_msg">b) write a simple performance test (nfib?) in a program that can be run either with its thread having normal or lower priority, and run two instances of the program at the same time and see if they take significantly different times to compute their result</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">If a) is possible and b) shows no significant difference in the wall-times of the two programs then we can modify the linux heartbeat code to *lower* the priority of the main Smalltalk thread if it finds it can't create a heartbeat thread with higher priority.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I hope this answers your questions.</div></div></div></div></blockquote><div><br></div><div>Yes, it does. Thanks!</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">As a footnote let me describe why we use a heartbeat at all. When I started working on the VisualWorks VM (HPS) in the '90s it had no heartbeat (IIRC, it might have only been the Windows VM that worked like this). Instead there was a counter decremented in every frame-building send (i.e. in the jitted machine code that activated a Smalltalk send), and when this counter went to zero the VM broke out and checked for events. This counter was initialized to 256 (IIRC). Consequently there was an enormous frequency of event checks, until, that is, ione did something that reduced the frequency of frame-building sends. One day I was doing something which invoked lots of long-running large integer primitives and I noticed that when I tried to interrupt the program it took many seconds before the system stopped. What was happening was that the large integer primitives were taking so long that the counter took many seconds to count down to 0. The system didn't check for events very often. So the problems with a counter are that</div><div class="gmail_msg">a) a read-modify-write cycle for a counter is in itself very expensive in a high-frequency operation like building a frame</div><div class="gmail_msg">b) in normal operation the counter causes far too many check-fore-event calls</div><div class="gmail_msg">c) in abnormal operation the counter causes infrequent check-fore-event calls</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">One solution on Unix is an interval timer (which my old BrouHaHa VMs used, but it did;t have much of an FFI so the problems it caused weren't pressing).</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">The natural solution is a heartbeat thread, and this is used in a number of VMs. One gets a regular event check frequency at very low cost. In Smalltalk VMs which do context-to-stack mapping it is natural to organize the stack as a set of pages and hence to have frame building sends check a stack limit (guarding the end of the page). The heartbeat simply sets the stack limit to the highest possible address to cause a stack limit check failure on the next send, and the stack check failure code checks if the stack limit has been set to the highest dress and calls the event check instead of handling the stack page overflow. In the HotSpot Java VM, if the platform supports it, a frame building send writes a byte to a guard page. Modern professors have write buffers so the write has very low cost (because it is never read) and is effectively free. So the heartbeat changes the guard page's permissions to take away write permission and cause an exception. The exception handler then checks and causes the VM to check for events. For this to work, all of writes, removing and setting page write permissions and handling exceptions must be sufficiently cheap. Anyone looking for a low-level project for the Cog VM could take a look at this mechanism. I've chosen to stick with the simple stack limit approach.</div></div></div></div></blockquote><div><br></div><div>And thanks for all this info. I didn't really know much about the itimer vs heartbeat thread topic TBH. I was just surprised that this is so complicated, because I thought that event handling would be relatively easy. However, I still don't completely understand why other applications (e.g. games) don't have these event problems, but that's something I can look into myself this weekend :)</div></div></div></div></blockquote><div><br></div>Forgive me; I missed this excellent question. I don't know if I'm correct but I presume the the reason a game application has no difficulties like this is that it can include an explicit poll for events as part of its main loop. While a game may be quite complex with many moving parts it's overall architecture is quite simple. It is a loop, advancing a model, rendering the model, and polling for events.<div><br></div><div>A Smalltalk vm, like a real processor, is a compute engineer me that goes whatever it is told, which may include being told to do nothing but execute code, but it must still be able to receive interrupts, even when running at full tilt performing nothing but calculations. </div><div><br></div><div>If one wants the Smalltalk vm to run suboptimally one will arrange that as part of its normal execution (e.g. on method activations) it will burn some cycles deciding if it should poll for events, e.g. decrenentung a counter. But as I explained elsewhere this has problems. If method invocation frequency falls, e.g. as a side effect of invoking long-running primitives, then it may poll for events too infrequently.</div><div><br></div><div>If however, one wants the Smalltalk vm to run optimally, one wants the event check to occur at regular intervals, without slowly Ng down Smalltalk execution, hence either the itimer interrupt or the heartbeat thread.</div><div><br></div><div>Does this make things clearer?</div><div><br><blockquote type="cite"><div dir="ltr"><div class="gmail_quote"><div>Fabio</div></div></div></blockquote><span style="background-color: rgba(255, 255, 255, 0);"><div><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div>Eliot<br>_,,,^..^,,,_ (phone)</span><div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Fabio<br class="gmail_msg"></div><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
To summarize, the itimer heartbeat is to be avoided as much as possible. It causes hard to debug issues with external code, has to be turned off and on around fork. It's a stop gap. Having to install a file in /etc just to be able to use a thread is insane (and AFAICT unique to linux). Whatever you do in the short term to deal with these problems I'll support, but in the long term we simply want a threaded heartbeat without needing to install anything.<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
><br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
> The code in sqUnixHeartbeat.c is not a lot nor very complex, it should not be difficult to do...<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
><br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
> Also, what would be the drawbacks besides an increase on the vm size?<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
I hope I've explained above that I expect the drawbacks will be intermittent failures of external code.<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
<br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
><br class="m_-304522989685937264gmail-m_3886557463388821969gmail_msg gmail_msg">
> Guille</blockquote></div></div></blockquote></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"></div><br class="gmail_msg"><div class="m_-304522989685937264gmail_signature gmail_msg"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><span style="font-size:small;border-collapse:separate" class="gmail_msg"><div class="gmail_msg">_,,,^..^,,,_<br class="gmail_msg"></div><div class="gmail_msg">best, Eliot</div></span></div></div></div>
</div></div>
</blockquote></div></div>
</div></blockquote></div></div></body></html>