[Vm-dev] Unix heartbeat thread vs itimer

[Ben Coman]

On Wed, Jan 11, 2017 at 4:54 AM, Eliot Miranda <eliot.miranda at gmail.com>
wrote:

>
> Hi Denis,
>
> On Tue, Jan 10, 2017 at 12:08 PM, Denis Kudriashov <dionisiydk at gmail.com>
> wrote:
>
>>
>> Hi.
>>
>> Will event-driven VM fix this problem completely? Or heartbeat will be
>> needed anyway?
>>
>
> As I understand it, the heartbeat us always needed.  In an event-driven VM
> it may be that the Smalltalk executive is called form the event loop, but
> the Smalltalk executive still has to break out of executing Smalltalk to
> return to the event loop to receive new events.
>
> The thing to understand about the JIT VM is that Smalltalk is executing in
> machine code just like simple code in some low level language.  That code
> can respond to Smalltalk events such as attempting to wait on a Semaphore
> with no outstanding signals, which may cause it to switch processes.  But
> it cannot respond to external asynchronous events unless it is informed of
> those events.  And it is very difficult to construct a VM that can accept
> interrupts at arbitrary times that activate Smalltalk code (imagine
> receiving an interrupt in the middle of a GC, or mid-way through looking up
> a send not found in the cache, etc, etc; essentially interrupts can only be
> accepted at limited times).  So the VM needs to check for events (and
> indeed interrupts) at safe points.  The efficient implementation of safe
> points is checking on the next frame-building send.  But if every
> frame-building send checked for interrupts and/or events, frame build would
> be very slow and the entire VM would crawl.  The function of the heart beat
> is to cause frame building sends to check for interrupts and/or events at
> regular, but (relative to frame-builling send frequency) infrequent
> occasions.
>
> Compare that to interrupts in a real processor.  The processor /also/ only
> tests e.g. the interrupt request pin at a safe point (perhaps at the end of
> each interaction decode cycle), and also provides means to disable
> interrupts for critical sections of code.  It's just that the quanta are
> much smaller than in a Smalltalk vm.
>
> HTH
>
> 2017-01-06 20:23 GMT+01:00 Eliot Miranda <eliot.miranda at gmail.com>:
>>
>>>
>>> Hi Fabio, Hi Guille,
>>>
>>> On Fri, Jan 6, 2017 at 9:44 AM, Fabio Niephaus <lists at fniephaus.com>
>>> wrote:
>>>
>>>>
>>>> On Fri, Jan 6, 2017 at 6:33 PM Eliot Miranda <eliot.miranda at gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Hi Guille,
>>>>>
>>>>> > On Jan 6, 2017, at 6:44 AM, Guillermo Polito <
>>>>> guillermopolito at gmail.com> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > I was checking the code in sqUnixHeartbeat.c to see how the
>>>>> heartbeat thread/itimer worked. It somehow bothers me that there are
>>>>> different compiled artifacts, one per option.
>>>>> >
>>>>> > What do you think about having a VM that manages that as an argument
>>>>> provided when we launch the VM? This would add some flexibility that we
>>>>> don't have right now because we make the decision at compile time.
>>>>>
>>>>> I think it's a fine idea but it isn't really the issue.  The issue is
>>>>> that the itimer mechanism is problematic, especially for foreign code, and
>>>>> is therefore a stop gap.  The itimer interrupts long-running system calls,
>>>>> which means that things like sound libraries break (at Qwaq I had to fix
>>>>> ALSA to get it to work with the itimer heartbeat).  Since Pharo is becoming
>>>>> more reliant on external code it may impact us more going forward.
>>>>>
>>>>> The real issue is that linux's requirement that thread priorities be
>>>>> set in per-application file in /etc/security/limits.d (IIRC) is a big.
>>>>> Neither Windows nor Mac OS X requires such nonsense, and a threaded
>>>>> heartbeat is used on those systems without any issue at all.  Why linux
>>>>> erected this mess in the first place is something I don't understand.
>>>>>
>>>>> I had to implement the itimer heartbeat to get Qwaq forums running on
>>>>> Linux running pre 2.6 kernels, but had many other problems to solve as a
>>>>> result (ALSA, database connects).
>>>>>
>>>>> Were it that the vm merely had to detect whether it could use the
>>>>> threaded heartbeat then things would be easy.  Instead one can only use the
>>>>> thing if one has superuser permissions to install a file in /etc, just to
>>>>> use a thread of higher priority than the main one.
>>>>>
>>>>
>>>> Thanks for the explanation, Eliot. I had no idea how bad the issues are
>>>> with the itimer, but I'm glad you also see the user-facing issue with the
>>>> heartbeat.
>>>>
>>>>
>>>>> An alternative might be to lower the priority of the main thread.
>>>>> Then the file installation would be unnecessary.
>>>>>
>>>>
>>>> Could you elaborate a little bit more on this idea? How could this
>>>> impact the vm? What could be the drawbacks here?
>>>>
>>>
>>> First of all, for the heartbeat thread to work reliably it must run at
>>> higher priority than the thread running Smalltalk code.  This is because
>>> its job is to cause Smalltalk code to break out at regular intervals to
>>> check for events.  If the Smalltalk code is compute-intensive then it will
>>> prevent the heartbeat thread from running unless the heartbeat thread is
>>> running at a higher priority, and so it will be impossible to receive input
>>> keys, etc. (Note that if event collection was in a separate thread it would
>>> suffer the same issue; compute intensive code would block the event
>>> collection thread unless it was running at higher priority).
>>>
>>> Right now, Linux restricts creating threads with priority higher than
>>> the default to those programs that have a /etc/security/limits.d/program.conf
>>> file that specifies the highest priority thread the program can create.
>>> And prior to the 2.6.12 kernel only superuser processes could create
>>> higher-priority threads.  I do know that prior to 2.6.12 one couldn't
>>> create threads of *lower* priority than the default either (I would have
>>> used this if I could).
>>>
>>> If 2.6.12 allows a program to create threads with lower priorities
>>> *without* needing a /etc/security/limits.d/program.conf, or more
>>> conveniently to allow a thread's priority to be lowered, then the idea is:
>>> 1. at start-up create a heartbeat thread at the normal priority
>>> 2. lower the priority of the main VM thread below the heartbeat thread.
>>> Alternatively, one could spawn a new lower-priority thread to run
>>> Smalltalk code, but this may be be much more work.
>>>
>>> The draw-back is that running Smalltalk in a thread whose priority is
>>> lower than the default *might* impact performance with lots of other
>>> processes running.  This depends on whether the scheduler conflates thread
>>> priorities with process priorities (which was the default with old linux
>>> threads, which were akin to processes).
>>>
>>> Sop there are some tests to perform:
>>>
>>> a) see if one can lower the priority of a thread without having a
>>> /etc/security/limits.d/program.conf in place
>>> b) write a simple performance test (nfib?) in a program that can be run
>>> either with its thread having normal or lower priority, and run two
>>> instances of the program at the same time and see if they take
>>> significantly different times to compute their result
>>>
>>> If a) is possible and b) shows no significant difference in the
>>> wall-times of the two programs then we can modify the linux heartbeat code
>>> to *lower* the priority of the main Smalltalk thread if it finds it can't
>>> create a heartbeat thread with higher priority.
>>>
>>>
With recent Pharo discussion on heatbeat  itimer versus threaded

http://forum.world.st/Esteban-s-ChangeLog-week-of-6-March-2017-td4938349.html

I got curious....

$ vi test.c
#include <stdio.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <stdlib.h>
#include <time.h>

void fib(int n, int pid)
{
   int first = 0, second = 1, next, c;
   for ( c = 0 ; c < n ; c++ )
   {
      if ( c <= 1 )
         next = c;
      else
      {
         next = first + second;
         first = second;
         second = next;
      }
      //printf("%d - fib=%d\n", pid, next);
   }
}


struct timespec timediff(struct timespec start, struct timespec end)
{
        struct timespec temp;
        temp.tv_sec = end.tv_sec-start.tv_sec;
        temp.tv_nsec = end.tv_nsec-start.tv_nsec;
        if (temp.tv_nsec < 0)
        {       temp.tv_nsec += 1000000000;
                temp.tv_sec -= 1;
        }
        return temp;
}

int main(int argc, char *argv[])
{
        int which = PRIO_PROCESS;
        id_t pid;
        int priority, fibN, ret;
        struct timespec start, stop, diff;

        if(argc<2)
        {       printf("Usage: %s ProcessPriority FibN\n", argv[0]);
                exit(1);
        }
        priority = atoi(argv[1]);
        fibN = atoi(argv[2]);

        pid = getpid();
        printf("%d\t ==> %d original priority\n", pid, getpriority(which,
pid));

        setpriority(which, pid, priority);
        priority = getpriority(which, pid);
        printf("%d\t ==> %d new priority\n", pid, priority);

        clock_gettime( CLOCK_REALTIME, &start);
        sleep(1); // allow all threads to be scheduled
        fib(fibN, pid);
        clock_gettime( CLOCK_REALTIME, &stop);
        diff = timediff(start, stop);
        printf("\n%d @ %d\t ==> execution time %d:%d\n", pid, priority,
diff.tv_sec - 1, diff.tv_nsec);
}

///////////////////////

$ gcc test.c

$ uname -a
Linux dom0 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) i686
GNU/Linux

$ nproc --all
4

$ N=1000000000 ; for NPROC in 1 ; do (./a.out 19 $N &) && (./a.out 1 $N &)
&& (./a.out 0 $N &) ; done
28175 @ 0 ==> execution time 5:137392274
28171 @ 19 ==> execution time 5:493271222
28173 @ 1 ==> execution time 5:678498982

...for NPROC in 1 2 ; do...
28339 @ 0 ==> execution time 5:891516242
28333 @ 0 ==> execution time 6:101871486
28331 @ 1 ==> execution time 6:197583303
28337 @ 1 ==> execution time 6:473926938
28335 @ 19 ==> execution time 11:19093473
28329 @ 19 ==> execution time 11:109494611

...for NPROC in 1 2 3 ; do...
28370 @ 0 ==> execution time 8:286661748
28364 @ 0 ==> execution time 8:346971535
28376 @ 0 ==> execution time 8:919760746
28362 @ 1 ==> execution time 9:943310436
28368 @ 1 ==> execution time 10:43977329
28374 @ 1 ==> execution time 10:251189507
28372 @ 19 ==> execution time 14:807238482
28360 @ 19 ==> execution time 15:48684466
28366 @ 19 ==> execution time 15:392610447

...for NPROC in 1 2 3 4 ; do...
28401 @ 0 ==> execution time 10:808863373
28407 @ 0 ==> execution time 11:144571568
28395 @ 0 ==> execution time 11:311897577
28389 @ 0 ==> execution time 12:49899167
28399 @ 1 ==> execution time 12:391939682
28387 @ 1 ==> execution time 12:922309497
28393 @ 1 ==> execution time 12:997908723
28405 @ 1 ==> execution time 13:116623935
28385 @ 19 ==> execution time 18:710195627
28391 @ 19 ==> execution time 19:160867082
28397 @ 19 ==> execution time 19:306339215
28403 @ 19 ==> execution time 19:340283641

...for NPROC in 1 2 3 4 5; do...
28431 @ 0 ==> execution time 13:512767819
28437 @ 0 ==> execution time 13:682814129
28449 @ 0 ==> execution time 13:695500843
28443 @ 0 ==> execution time 14:212788768
28425 @ 0 ==> execution time 14:384973010
28435 @ 1 ==> execution time 15:356897222
28423 @ 1 ==> execution time 15:638388578
28441 @ 1 ==> execution time 15:692913828
28429 @ 1 ==> execution time 15:822047741
28447 @ 1 ==> execution time 15:845915440
28445 @ 19 ==> execution time 23:221173838
28433 @ 19 ==> execution time 23:349862424
28421 @ 19 ==> execution time 23:388930822
28439 @ 19 ==> execution time 23:374234661
28427 @ 19 ==> execution time 23:517771887

cheers -ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20170314/af27c5f7/attachment-0001.html>