<div dir="ltr">Alistair,<div><br></div><div>Indeed I also had an issue with workingDirectoryPath. This is the complete list of what I changed in the Pharo 7 image, </div><div>it's not clear to me if everything is required (Like the zodiac change, I ended up using the global instead):</div><div><br></div><div>!FreeTypeSystemSettings class methodsFor: 'settings' stamp: 'cb 11/15/2019 21:22' prior: 25673466!<br>ft2LibraryVersion<br> ^ Smalltalk ui theme <br> newLabelIn: World <br> for: self <br> label: 'Available version: None'<br> getEnabled: nil.! !<br>!FreeTypeSettings class methodsFor: 'system startup' stamp: 'cb 11/15/2019 21:24' prior: 25665613!<br>startUp: resuming<br> "resuming ifTrue:[ self updateFreeType ]"! !<br>!LGitLibrary class methodsFor: 'system startup' stamp: 'cb 11/15/2019 21:24' prior: 31291584!<br>startUp: isImageStarting<br> "isImageStarting ifTrue: [ self uniqueInstance initializeLibGit2 ]"! !<br>!OSPlatform methodsFor: 'accessing' stamp: 'cb 11/15/2019 21:36' prior: 53154052!<br>currentWorkingDirectoryPath<br> "This method calls the method getPwdViaFFI with arugement of a buffer size. By default it uses the defaultMaximumPathLength of each subclass as the buffer size."<br> ^ '/' "self currentWorkingDirectoryPathWithBufferSize: self defaultMaximumPathLength"<br>! !<br>!ZdcPluginSSLSession methodsFor: 'initialization' stamp: 'cb 11/15/2019 21:39' prior: 66961330!<br>initialize<br> "Initialize the receiver"<br> "[ handle := self primitiveSSLCreate ]<br> on: PrimitiveFailed <br> do: [ :exception |<br> ZdcPluginMissing signal ].<br> self logging: false"<br>! !<br>!PharoFilesOpener methodsFor: 'public' stamp: 'cb 11/27/2019 22:56' prior: 54156796!<br>changesFileOrNilReadOnly: readOnly silent: silent<br> | changesFile |<br> changesFile := self openChanges: self changesName readOnly: readOnly.<br> (changesFile isNil and: [ silent not ])<br> ifTrue: [ self informProblemInChanges: self cannotLocateMsg ].<br> ^ changesFile .<br> ! !<br><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 29, 2019 at 11:53 AM Clément Béra <<a href="mailto:bera.clement@gmail.com">bera.clement@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi,</div><div dir="auto"><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 29, 2019, 10:21 Alistair Grant <<a href="mailto:akgrant0710@gmail.com" target="_blank">akgrant0710@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Clément,<br>
<br>
<br>
On Thu, 28 Nov 2019 at 22:36, Clément Béra <<a href="mailto:bera.clement@gmail.com" rel="noreferrer" target="_blank">bera.clement@gmail.com</a>> wrote:<br>
><br>
> Hi Alistair,<br>
><br>
> I've just investigated the bug tonight and fixed it in VMMaker.oscog-cb.2595. I compiled a new VM from 2595 and I was able to run the 400 iterations of your script without any crashes.<br>
> Thanks for the easy reproduction! Last year when I used the GC benchmarks provided by Feenk, with ~10Gb workloads, for the DLS paper [1], I initially had an image crashing 9 times out of 10<br>
> when going to 10Gb. I fixed a few bugs on the production GC back then (mainly on segment management) which led the benchmarks to run successfully 99% of the times. But it was still crashing<br>
> on 1%, since I was benchmarking on experimental GCs with various changes I thought the bug did not happen in the production GC, but it turns out I was wrong. And you found a reliable way to<br>
> reproduce :-). So I could investigate. It's so fun to do lemming debugging in the simulator.<br>
<br>
We need to thank Juraj here, he was the one who produced the initial<br>
version of the script which made all of this possible.<br>
<br></blockquote><div>Thanks Juraj. Are you both Feenk people?</div><div>Are you mainly working on the VM Alistair? Or just having fun?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> The GC bug was basically that when Planning Compactor (Production Full GC compactor) decided to do a multiple pass compaction, if it managed to compact everything in one go then it would<br>
> get confused and attempt to compact objects upward instead of downward (address wise) on the second attempt, and that's broken and corrupts memory.<br>
><br>
> I started from this script:<br>
><br>
> | aJson anArray |<br>
> aJson := ZnEasy get: '<a href="https://data.nasa.gov/resource/y77d-th95.json" rel="noreferrer noreferrer" target="_blank">https://data.nasa.gov/resource/y77d-th95.json</a>' asZnUrl.<br>
> Array streamContents: [ :aStream |<br>
> 400 timesRepeat: [<br>
> aStream nextPutAll: (STON fromString: aJson contents).<br>
> Smalltalk saveSession ] ].<br>
><br>
><br>
> It makes me however very sad that you were not able to use the simulator to debug this issue, I used it and that's how I tracked down the bug in only a few hours. Tracking things down in lldb would have taken me weeks, and I would not have been able to do it since I work during the week :-).<br>
><br>
> Therefore I'm going to explain you my process to reproduce the bug in the simulator and to understand where the issue comes from. The mail is quite long, but it would be nice if you could track the bug quickly on your own next time using the simulator. Of course you can skip if you're not interested. @Eliot you may read since I explain how I set-up a Pharo 7 image for simulator debugging, that might come handy for you at some point.<br>
><br>
> 1] The first thing I did was to reproduce your bug, based on the script, both on Cog and and Stack vm compiled from OpenSmalltalk-VM repository. I initially started with Pharo 8, but for some reason that image is quite broken (formatter issue? Integrator gone wild?).<br>
<br>
That was unlucky timing, there was a bad commit made. I think it's<br>
largely tidied up now, still, using the current stable version isn't<br>
necessarily bad :-)<br>
<br>
Just for future reference: the first thing I tried was reproducing it<br>
on the Pharo 8 minimal image (I did this before the formatter bug<br>
appeared and kept the same image). The minimal image has a few<br>
advantages:<br>
<br>
- It's smaller, 14M vs. 54M, so less memory to keep track of (and the<br>
simulator will be a bit faster)<br>
- It doesn't have FreeType loaded, so that quickly ruled it out as an issue.<br>
- I wasn't sure if there would be other FFI calls, so this just<br>
reduced the chances.<br>
<br></blockquote><div>Not having FreeType and LibGit would be nice indeed. The difference between simulation performance 14Mb-54Mb is not really an issue for me, the bug happened on > 100Mb heap and simulation is still fairly fast. </div><div>The problem is more to find a reliable way to crash soon after start-up, in some cases I start the simulator, go to sleep, but if the next morning it hasn't crashed, well, too bad :-(.</div><div><br></div><div>In most cases we reproduce bugs using the Squeak REPL image. See:</div><div><a href="https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/buildspurtrunkreaderimage.sh" target="_blank">https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/buildspurtrunkreaderimage.sh</a><br></div><div>I suggest you try using the simulator on the squeak repl, it's convenient you can run a few things and see what is going on. The REPL support chunk format (Put a ! after each do it).</div><div>You can build something similar from the minimal Pharo if you want to, but I doubt you'll catch bugs that you can't catch from the Squeak one.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> So I switched to Pharo 7 stable. It crashes on both VMs, so I knew the bug was unrelated to the JIT. Most bugs on the core VM (besides people mis-using FFI, which is by far the most common VM bug reported) is either JIT or GC. So we're tracking a GC bug.<br>
> I then built an image which runs your script at start-up (Smalltalk snapshot: true andQuit: true followed by your script, I select all and run do-it).<br>
><br>
> 2] Then I started the image in the simulator. First thing I noticed is that Pharo 7 is using FFI calls in FreeType, from start-up, and even if you're not using text or if you disable FreeType from the setting browser, Pharo performs in the backgrounds FFI calls for freetype. FreeType FFI calls are incorrectly implemented (the C stack references heap object which are not pinned), therefore these calls corrupts the heap. Running a corrupted heap on the VM has undefined behavior, therefore any usage of Pharo 7 right now, wether you actually text or not, wether freetype is enabled or not in the settings, is undefined behavior. I saw in the thread Nicolas/Eliot complaining that this is not a VM bug, indeed, pinning objects is image-side responsibility and it's not a VM bug. In addition, most reported bug comes from people mis-using FFI, so I understand their answer. There was however another bug in the GC, but it's very hard for us to debug it if it's hidden after image corrupting bugs like the FreeType one here.<br>
> So for that I made that change:<br>
> FreeTypeSettings>>startUp: resuming<br>
> "resuming ifTrue:[ self updateFreeType ]"<br>
> saved, restarted the image, and ensured it was not corrupted (leak checker + swizzling in simulation).<br>
><br>
> 3] Then I started the image in the simulator. Turns out the image start-up raises error if libgit cannot be loaded, and then the start-up script is not executed due to the exception. So I made that change:<br>
> LibGitLibrary>>startUp: isImageStarting<br>
> "isImageStarting ifTrue: [ self uniqueInstance initializeLibGit2 ]"<br>
<br>
Also for future reference, I'm surprised you didn't hit an FFI call<br>
trying to get the current working directory. Making the following<br>
change in OSPlatform removes the FFI call:<br>
<br>
currentWorkingDirectoryPathWithBuffer: aByteString<br>
<primitive: 'primitiveGetCurrentWorkingDirectory' module:<br>
'UnixOSProcessPlugin' error: ec><br>
^self primitiveFailed<br>
<br>
(if on windows you need to use WinOSProcessorPlugin).<br><br><br></blockquote><div>Err. Maybe I forgot to write down a few steps here and commented a few other methods... I fixed it and then wrote the mail, I don't remember it all. </div><div>I think indeed there was something accessing source or change files and I commented something in there.</div><div>I'll try to check the change file later on.</div><div>I don't have access to my laptop right now I'm at work so I cannot check.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
> 4] Turns out ZnEasy does not work well in the simulator. So I preloaded this line aJson := ZnEasy get: '<a href="https://data.nasa.gov/resource/y77d-th95.json" rel="noreferrer noreferrer" target="_blank">https://data.nasa.gov/resource/y77d-th95.json</a>' asZnUrl. into a Global variable. The rest of the script remains the same. I can finally run your script in the simulator! Usually we simulate Squeak image and all these preliminary steps are not required. But! It is still easier to reproduce this bug that most bugs I have to deal with for Android at work, at least I don't need to buy an uncommon device from an obscure chinese vendor to reproduce :-).<br>
<br>
I put the data in to a file and loaded it :-)<br>
<br>
> 5] To shortcut simulation time, since the bug happened around the 60th save for me, I build a different script which snapshots the image to different image names.<br>
<br>
We also updated the script to save to different files.<br>
<br>
But did you actually get it to save the image in the simulator? I'm<br>
just reproducing your work now but couldn't save an image due to a bug<br>
in the FileAttributesPluginSimulator. I've got a fix and will commit<br>
a bit later.<br>
<br></blockquote><div>Yes, running the script in the simulator generated me around 30 images (Save57.image to Save90.image). I frequently use saving from the simulator (usually Squeak image though). Should work.</div><div>Then running the script again to 400 iterations from the VM I generated filled my local SSD :-).</div><div>I don't remember which API I used though to save, maybe we used different ones? I try to use snapshot:andQuit: as much as possible to avoid unexpected errors, but this time I renamed, I don't remember how. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> With a crash at snapshot 59 (only change file written to disk), image 57 was the latest non corrupted image. I then started the simulator (The StackSimulator since we are debugging a GC bug, not the Cog simulator, simulation is faster and simpler). I used the standard script available in the workspace of the Cog dev image built from the guidelines. [2]<br>
> | sis |<br>
> sis := StackInterpreterSimulator newWithOptions: #(ObjectMemory Spur64BitMemoryManager).<br>
> "sis desiredNumStackPages: 8." "Speeds up scavenging when simulating. Set to e.g. 64 for something like the real VM."<br>
> "sis assertValidExecutionPointersAtEachStep: false." "Set this to true to turn on an expensive assert that checks for valid stack, frame pointers etc on each bytecode. Useful when you're adding new bytecodes or exotic execution primitives."<br>
> sis openOn: 'Save57.image'.<br>
> sis openAsMorph; run<br>
> I then let the simulator simulate, went swimming for 1h, and came back 1h30 later (with commute time). The bug happened in the simulator at save 90, I don't know how long it took to reproduce, but < 1h30. Then I had an assertion failure in the compactor:<br>
> self assert: (self validRelocationPlanInPass: finalPass) = 0.<br>
> Good! From there I debugged using lemming debugging (technique described in [3], Section 3.2). When the assertion has failed, simulation is the clone. I went up in the debugger to the point where the clone was made, and restarted the same GC approximately 40 times during debugging because once the heap is corrupted you cannot know anymore what the problem is, but you need to trigger the problem to understand. 40 lemmings over that cliff :-) Good lemmings.<br>
><br>
> Then I quickly figured out that the GC was performing two successive compactions, and that the second compaction is broken right at the start (tries to move objects upward). Then I looked at the glue code in-between the 2 compactions, and yeah, in the case where the first compaction has compacted everything, the variables are incorrectly set for the second compaction. I tried fixing the variables but it's not that easy, so instead I just aborted compaction in that case (See VMMaker.oscog-cb.2595).<br>
><br>
> 6] I then compiled a VM from the sources to check Slang translator would not complain, it did not. I then built a stack VM (Cog VM seems to be broken on tip of tree due on-going work for ARMv8 support) and run your script again. I was able to run the 400 iterations without crash. Bug seems to be fixed!<br>
><br>
> @Eliot now needs to fix tip of tree, generate the code and produce new VMs. ARMv8 support is quite exciting though, giving that MacBooks do not support 32 bits any more and that the next Macbooks are rumoured to be on ARMv8. One wouldn't want to run the VM in a virtual box intel image :-).<br>
><br>
> Alistair, let me know if you have questions. I hope you can work with the simulator as efficiently as we can. If you've not seen it, there's this screencast where I showed how I used the simulator to debug JIT bugs [4]. Audio is not very good because my spoken English sucks, but it shows the main ideas.<br>
><br>
> [1] <a href="https://www.researchgate.net/publication/336422106_Lazy_pointer_update_for_low_heap_compaction_pause_times" rel="noreferrer noreferrer" target="_blank">https://www.researchgate.net/publication/336422106_Lazy_pointer_update_for_low_heap_compaction_pause_times</a><br>
> [2] <a href="http://www.mirandabanda.org/cogblog/build-image/" rel="noreferrer noreferrer" target="_blank">http://www.mirandabanda.org/cogblog/build-image/</a><br>
> [3] <a href="https://www.researchgate.net/publication/328509577_Two_Decades_of_Smalltalk_VM_Development_Live_VM_Development_through_Simulation_Tools" rel="noreferrer noreferrer" target="_blank">https://www.researchgate.net/publication/328509577_Two_Decades_of_Smalltalk_VM_Development_Live_VM_Development_through_Simulation_Tools</a><br>
> [4] <a href="https://clementbera.wordpress.com/2018/03/07/sista-vm-screencast/" rel="noreferrer noreferrer" target="_blank">https://clementbera.wordpress.com/2018/03/07/sista-vm-screencast/</a><br>
<br>
You wrote in [3]:<br>
<br>
"the slightest change in the heap<br>
might change the bug; any variability in timing or user input<br>
can result in a different heap and hence in the bug morphing<br>
or going into hiding."<br>
<br>
This was evident in this issue. While the script (fortunately) would<br>
always produce a crash, small changes, such as how the initial JSON is<br>
loaded, or the name of the image that it is saved to, caused fairly<br>
large changes in the number of loops to trigger the crash.<br></blockquote><div><br></div><div>Yeah that's the main problem when debugging GC in general. Pharo is less deterministic than Squeak for some reason (things are happening in the background doing FFI calls). In both environment user events is a problem.</div><div>That's why lemming debugging is very handy. And that's why OpenSmalltalk-VM development tools are far superior to other VMs I've dealt with. The back-in-time features that I used in C++ recently are very good though, in OpenSmalltalk-VM </div><div>I guess the circular buffer of JIT simulation has a better time spent on tools/productivity ratio and is enough for now.</div><div><br></div><div>And this is a crash. Performance pitfalls issue are even harder to track down IMO.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Also, while trying to reproduce your debug steps above, the image I<br>
have already has memory leaks, so it isn't hitting the "self assert:<br>
(self validRelocationPlanInPass: finalPass) = 0" assertion.<br></blockquote><div> </div><div>You have to start simulation on a non already corrupted image. Did you make sure to comment the startUp: method in FreeTypeSettings? Disabling FreeType in the setting browser is not enough. Then you need to save and restart the image, and verifies it is not already corrupted.</div><div>If you're talking about starting simulation from the saved images from the script, I did not take the latest which crashed because it was already corrupted, I used 57 while 58 was saved and 59 only changes were saved. You can see at start-up if swizzling and the initial GC find leaks.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks for the links, I'll keep reading.<br>
<br>
Thanks again!<br>
Alistair<br>
<br>
> --<br>
> Clément Béra<br>
> <a href="https://clementbera.github.io/" rel="noreferrer noreferrer" target="_blank">https://clementbera.github.io/</a><br>
> <a href="https://clementbera.wordpress.com/" rel="noreferrer noreferrer" target="_blank">https://clementbera.wordpress.com/</a><br>
</blockquote></div></div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><span style="font-size:12.8px">Clément Béra<br></span><span style="color:rgb(0,0,238)"><a href="https://clementbera.github.io/" target="_blank">https://clementbera.github.io/</a></span><div style="font-size:12.8px"><a href="https://clementbera.wordpress.com/" target="_blank">https://clementbera.wordpress.com/</a></div></div></div></div></div></div>