[Vm-dev] [Pharo-users] macOS VM builds with AddressSanitizer/LeakSanitizer

Guillermo Polito guillermopolito at gmail.com
Wed Feb 27 11:05:56 UTC 2019

Hi Manuel,

Disclaimer: I'm not an objectiveC expert either :)

When I did that (and compiled, and tested), I remember having seen that it
was using the automatic reference counting from objectiveC.
Actually, putting the [release] was producing for me a segmentation fault.
Did you try it yourself and did it fix the problem for you?

On Wed, Feb 27, 2019 at 10:52 AM Manuel Leuenberger <
leuenberger at inf.unibe.ch> wrote:

> Not sure if I found something, but looking
> at sqSqueakOSXApplication+events.m I find
> - (void) pumpRunLoopEventSendAndSignal:(BOOL)signal {
>        NSEvent *event;
>        NSMutableArray *alienEventQueue = [[NSMutableArray alloc] init];
>        while ((event = [NSApp nextEventMatchingMask:NSEventMaskAny
>                              untilDate:nil
>                              inMode:NSEventTrackingRunLoopMode
>                              dequeue:YES])) {
>          // If the event is not a system event or an event of *this*
> window, queue the event
>          // Otherwise treat the event normally and send it to the app
>          if (event.window && event.window != gDelegateApp.window){
>            [alienEventQueue addObject: event];
>          }else{
>            [NSApp sendEvent: event];
>            if (signal) {
>  interpreterProxy->signalSemaphoreWithIndex(gDelegateApp.squeakApplication.inputSemaphoreIndex);
>            }
>          }
>      }
>      // Put back in the queue all events that did not belong to this window
>      // They will be managed by other windowing systems
>      // (or by a subsequent invocation to this function)
>      while ((event = [alienEventQueue firstObject])) {
>        [NSApp postEvent: event atStart: NO];
>        [alienEventQueue removeObject: event];
>      }
> }
> I have no clue about Objective-C, but I think
>  NSMutableArray *alienEventQueue = [[NSMutableArray alloc] init];
> makes me own the queue on the heap, isn't there a
> [alienEventQueue release];
> missing at the end?
> Cheers,
> Manuel
> On 26 Feb 2019, at 23:31, Manuel Leuenberger <leuenberger at inf.unibe.ch>
> wrote:
> Alright, back in the game, this time using gperftools.
> I finally managed to reproduce a leaking run, but it took me hours to get
> there, randomly rendering GtCoders and messing around. gperftools
> instruments malloc and friends, capturing heap profiles every 30 seconds.
> Now I have profiles from a freshly started, small image, before it went
> berserk on memory hogging, and when it allocates over 3GB. Now, I have
> visualizations of allocation sites at these instants, and diffs between
> them, as well as some thread dumps on the berserk state. I managed to rule
> out the in-image objects to contribute to memory by cleaning images with
> garbage collect and cache resets. Some handy resets are these:
> NECController reset.
> FreeTypeCache current removeAll.
> ASTCache reset.
> Smalltalk garbageCollect.
> These get rid of a leak in NECController, resets the font cache (creates a
> lot of bitmaps), and ASTs, created by the coders. With
> SpaceTally printSpaceAnalysis.
> I get objects stats from within the image, and I managed to keep them in
> the same ballpark for all instants I took profiles of, by inspecting
> over-populated instances, like this
> instances := WeakOrderedCollection withAll: RBMessageNode allInstances.
> ReferenceFinder findPathTo: instances atRandom.
> With
> FFIExternalResourceManager uniqueInstance instVarNamed: #registry
> I also made sure that external objects are clean.
> Now, I am pretty confident that I can clean images, so the memory-overload
> should be sourced in the VM. Still, I cannot reliably reproduce the leak.
> It just happens after running an image for hours, doing stuff, idling,
> going to sleep, wake up.
> Unfortunately, I did not run it on a debug VM, so nearly no symbols, but I
> prepared a summary of the analysis anyway, as it already gives some hints
> that some of you can maybe interpret better than me. Have look at the PDFs
> in particular, they show the allocation sites at the three interesting
> instants, (clean, before berserk, after berserk), as well as the diffs in
> between.
> https://www.dropbox.com/s/zusnfn5mhyw3rmp/memory-analysis.zip?dl=0
> I find that berserk mode is caused by a huge pile of Objective-C objects.
> The thread samples show that while idling, it seems to be busy processing
> OS events. But I do not know what kind of objects they actually are.
> My current hypothesis is:
> The leak is caused by processing a huge pile of OS events being processed
> (reprocessed?). For some reason, this only happens after a long run time.
> Now, how can I accept/reject this hypothesis? Is there a way to log events
> received by the VM? I also have a core dump of the 3GB process, but I have
> no idea how I could check it for Objective-C objects grouped by type. If
> anybody knows how to do that, I might be able to come up with a
> comprehensive list of objects that lead to the berserk mode. Maybe it is
> not even are real leak, but just accumulating objects that should be thrown
> away at some point. Also, I can still not fully reject that the image is
> involved, maybe it forgets external resources it should not forget at some
> point.
> As I now know again that I can trigger berserk mode, I will switch to the
> debug VM/Moz2D again with ASan. More symbols, leak reports, might reveal
> better information.
> Anyway, if anybody made it through the whole email and maybe even looked
> at the ZIP, I would appreciate ideas on how to proceed, what to test, what
> to inspect. I recently had a few incidents where I realized that one of my
> implicit assumptions/axioms does not hold, and I kept digging in the wrong
> direction for too long, so any feedback on this is welcome.
> Cheers,
> Manuel
> On 30 Jan 2019, at 18:40, Manuel Leuenberger <leuenberger at inf.unibe.ch>
> wrote:
> I thought about adding a flag for all ASan magic in the build scripts,
> then document the procedure in the readme, maybe also more extensive
> somewhere else. You probably have a better idea where a good place would be
> than I do.
> I dug into this today again and wrote a parser for the reports (
> https://github.com/maenu/leak-reporter). It has a not-so-good
> visualization of merged stacks showing heavy frames as big and solid, it
> also includes example reports (
> https://github.com/maenu/leak-reporter/tree/master/res). The reported
> leaks only amount for about 1% of leaked memory, not too many objects
> leaking, but I am not sure if ASan just counts object layout size or
> objects graphs (probably not). What I notice is that OS events are leaking,
> as they are copied. Might be an issue. But I am not entirely sure if this
> explains everything. Maybe the VM just allocates too many objects of some
> type, but frees them on shutdown, so they might not even appear in the
> reports. Can a C heap analyzer give me stats about struct instances per
> struct type? Might be useful to analyze core dumps then.
> Cheers,
> Manuel
> <Screen Shot 2019-01-30 at 18.27.22.png>
> Direct leak of 219512 byte(s) in 1193 object(s) allocated from:
>     #0 0x10bb1fffc in __sanitizer_mz_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5dffc)
>     #1 0x7fffdd224281 in malloc_zone_malloc (libsystem_malloc.dylib:x86_64+0x2281)
>     #2 0x7fffc6de33d5 in AllocateAndInitializeEvent(__CFAllocator const*) (HIToolbox:x86_64+0x33d5)
>     #3 0x7fffc6e3fc4d in CopyEventInternal(__CFAllocator const*, OpaqueEventRef*) (HIToolbox:x86_64+0x5fc4d)
>     #4 0x7fffc6e49228 in CopyEventAs (HIToolbox:x86_64+0x69228)
>     #5 0x7fffc6e1d349 in CreateEventWithCGEvent (HIToolbox:x86_64+0x3d349)
>     #6 0x7fffc6e1b0e2 in CreateAndPostEventWithCGEvent(__CGEvent*, unsigned int, unsigned char, __CFMachPortBoost*) (HIToolbox:x86_64+0x3b0e2)
>     #7 0x7fffc6e275cd in Convert1CGEvent(unsigned char) (HIToolbox:x86_64+0x475cd)
>     #8 0x7fffc6e2745e in MainLoopObserver(unsigned int, OpaqueEventRef*, void*) (HIToolbox:x86_64+0x4745e)
>     #9 0x7fffc6de8368 in _NotifyEventLoopObservers (HIToolbox:x86_64+0x8368)
>     #10 0x7fffc6e10ea5 in RunCurrentEventLoopInMode (HIToolbox:x86_64+0x30ea5)
>     #11 0x7fffc6e10bf8 in ReceiveNextEventCommon (HIToolbox:x86_64+0x30bf8)
>     #12 0x7fffc6e10b25 in _BlockUntilNextEventMatchingListInModeWithFilter (HIToolbox:x86_64+0x30b25)
>     #13 0x7fffc53a5a53 in _DPSNextEvent (AppKit:x86_64+0x46a53)
>     #14 0x7fffc5b217ed in -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] (AppKit:x86_64+0x7c27ed)
>     #15 0x10b7ce07c in -[sqSqueakOSXApplication(events) pumpRunLoopEventSendAndSignal:] sqSqueakOSXApplication+events.m:78
>     #16 0x10b7ce7d9 in -[sqSqueakOSXApplication(events) pumpRunLoop] sqSqueakOSXApplication+events.m:105
>     #17 0x10b7e4898 in vmIOProcessEvents sqSqueakEventsAPI.m:80
>     #18 0x10b7e49c7 in ioProcessEvents sqSqueakEventsAPI.m:103
>     #19 0x10b5b5653 in checkForEventsMayContextSwitch gcc3x-cointerp.c:62680
>     #20 0x10b5c17ed in ceCheckForInterrupts gcc3x-cointerp.c:15188
>     #21 0x119a75794  (<unknown module>)
>     #22 0x10b566712 in interpret gcc3x-cointerp.c:2754
>     #23 0x10b7ebada in -[sqSqueakMainApplication runSqueak] sqSqueakMainApplication.m:201
>     #24 0x7fffc93786fc in __NSFirePerformWithOrder (Foundation:x86_64+0xd76fc)
>     #25 0x7fffc78cfc56 in __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ (CoreFoundation:x86_64h+0xa6c56)
>     #26 0x7fffc78cfbc6 in __CFRunLoopDoObservers (CoreFoundation:x86_64h+0xa6bc6)
>     #27 0x7fffc78b05f8 in __CFRunLoopRun (CoreFoundation:x86_64h+0x875f8)
>     #28 0x7fffc78b0033 in CFRunLoopRunSpecific (CoreFoundation:x86_64h+0x87033)
>     #29 0x7fffc6e10ebb in RunCurrentEventLoopInMode (HIToolbox:x86_64+0x30ebb)
> On 29 Jan 2019, at 20:19, Alistair Grant <akgrant0710 at gmail.com> wrote:
> On Tue, 29 Jan 2019 at 15:32, Eliot Miranda <eliot.miranda at gmail.com>
> wrote:
> ...
> So this is great.  The final step is seeing which FFI method invokes
> moz2d_draw_target_create_for_data_type and then the storage leak can be
> fixed.
> What would be wonderful is to capture your experience in diagnosing this
> storage leak as a guide to others.  But in what form?  One would be a blog
> post.  I’m nervous though because blogs sometimes disappear.  One would be
> some section in one of the READMEs in the source tree.  One would be a
> carefully written email to this list recapitulating or summarizing the
> experience.
> The advantage of the mailing list is that it won’t go away any time soon.
> So how about writing a new message in this thread which summarizes?  It
> could be written as more of a guide “Storage leaks come in more than one
> variety...  To diagnose a leak in the C heap... To link against a C leak
> checker...” and include links to the other messages in this thread on the
> mail list serve for details “Specifics on how to modify the Mac build to
> link in the leak checker are ...”.
> What do you think?  What to others think?  How do we best capture this for
> posterity?
> I would split this in two:
> - The basic instructions on compiling with the leak checker should go
> in the HowToBuild.
> - The description you're suggesting above should go in
> http://opensmalltalk.org/ as Ben suggested, or a wiki page in the
> OpenSmalltalk-VM repository.
> Each should provide a link to the other.
> I'm not a big fan of github wiki pages, but they have two advantages:
> - They keep the content with the code.
> - They're relatively easy to download and back-up offline (in the
> event that we decide to move the code elsewhere).
> My preference would be http://opensmalltalk.org/, but I'm not sure
> what the access rights on it are.  Actually, I didn't even know it
> existed, thanks Ben!
> As Tim said, email lists are hard to search, and can't be easily
> updated.  Blog posts don't really lend themselves to being updated
> either.
> My 2c...
> Cheers,
> Alistair


Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20190227/c8a6314f/attachment-0001.html>

More information about the Vm-dev mailing list